United States Government Accountability Office

GAO Report to Congressional Committees

BUSINESS SYSTEMS 

MODERNIZATION 

DOD Needs to Fully 

Define Policies and 

Procedures for 

Institutionally 

Managing Investments

May 2007 

 

GAO-07-538 
What GAO Found

United States Government Accountability Office

Why GAO Did This Study

Highlights

Accountability Integrity Reliability

May 2007

BUSINESS SYSTEMS MODERNIZATION

DOD Needs to Fully Define Policies and 

Procedures for Institutionally Managing 

Investments 

Highlights of GAO-07-538, a report to 

congressional committees 

In 1995, GAO first designated the 

Department of Defense’s (DOD)

business systems modernization 

program as “high-risk,” and 

continues to do so today. In 2004, 

Congress passed legislation 

reflecting prior GAO 

recommendations for DOD to 

adopt a corporate approach to

information technology (IT)

business system investment 

management. To support GAO’s 

legislative mandate to review

DOD’s efforts, GAO assessed 

whether the department’s 

corporate investment management 

approach comports with relevant 

federal guidance. In doing so, GAO 

applied its IT Investment 

Management framework and 

associated methodology, focusing

on the framework’s stages related 

to the investment management

provisions of the Clinger-Cohen Act 

of 1996. 

What GAO Recommends

GAO recommends that DOD fully

define the project and portfolio 

management policies and 

procedures discussed in GAO’s 

framework. DOD agreed with

GAO’s overall conclusions and

partially agreed with five of GAO’s 

recommendations. However, DOD 

disagreed with the remaining four 

recommendations, stating that the 

department is, among other things,

already meeting the intent of these

recommendations. GAO does not

agree; its recommendations focus

on fully defining policies and 

procedures that satisfy key 

practices in its framework.

DOD has established the management structures needed to effectively 

manage its business system investments, but it has not fully defined many of 

the related policies and procedures that GAO’s IT Investment Management 

framework defines. Specifically, the department has defined four of nine 

practices that call for project-level policies and procedures, and one of the 

five practices that call for portfolio-level policies and procedures (see 

below). For example, DOD has established an enterprisewide IT investment

board responsible for defining and implementing its business system 

investment governance process, documented policies and procedures for 

ensuring that systems support ongoing and future business needs, developed 

procedures for identifying and collecting information about these systems to

support investment selection and control, and assigned responsibility to an 

individual or a group for managing the development and modification of the 

business system portfolio selection criteria. However, DOD has not fully 

documented business system investment policies and procedures for 

directing investment board operations, selecting new investments, 

reselecting ongoing investments, integrating the investment funding and the 

investment selection processes, and developing and maintaining a complete 

business system investment portfolio(s). 

Regarding project-level investment management practices, DOD officials 

said that these are performed at the component level, and that departmental 

policies and procedures established for overseeing components’ execution 

of these practices are sufficient. For portfolio-level practices, however, these 

officials stated that they intend to improve departmental policies and 

procedures for business system investments by, for example, establishing a 

single governance structure, but plans or time frames for doing so have not 

been established. Until DOD fully defines departmentwide policies and 

procedures for both individual projects and portfolios of projects, it risks 

selecting and controlling these business system investments in an 

inconsistent, incomplete, and ad hoc manner, which in turn reduces the 

chances that these investments will meet mission needs in the most costeffective manner. 

Policies and Procedures for Project-Level and Portfolio-Level Management

Stage 2: Building the 

investment foundation 

Key practices 

executed 

Stage 3: Developing a 

complete investment 

portfolio

Key practices 

executed 

Instituting the investment 

board 

1/2 Defining the portfolio criteria 1/2 

Meeting business needs 1/1 Creating the portfolio 0/1 

Selecting an investment 0/3 Evaluating the portfolio 0/1 

Providing investment 

oversight 

0/1 Conducting postimplementation

reviews 

0/1 

Capturing investment 

information 

2/2 

Overall 4/9 1/5 

Source: GAO. 

www.gao.gov/cgi-bin/getrpt?GAO-07-538.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Randolph C. 

Hite at (202) 512-3439 or hiter@gao.gov. 
Contents 

Letter 1

Results in Brief 3

Background 5

DOD Has Established the Structures Needed to Effectively Manage 

Business System Investments, but Has Not Fully Defined Many 

of the Related Policies and Procedures 22 

Conclusions 32

Recommendations for Executive Action 32 

Agency Comments and Our Evaluation 33 

Appendix I Objective, Scope, and Methodology 40 

Appendix II Comments from the Department of Defense 41 

Appendix III GAO Contact and Staff Acknowledgments 51 

Tables 

Table 1: DOD Business Investment Management System Entities’ 

Roles, Responsibilities, and Composition 16 

Table 2: DOD’s Investment Tiers 18 

Table 3: Stage 2 Critical Processes—Building the Investment 

Foundation 23

Table 4: Summary of Policies and Procedures for Stage 2 Critical 

Processes—Building the Investment Foundation 26 

Table 5: Stage 3 Critical Processes—Developing a Complete 

Investment Portfolio 29 

Table 6: Summary of Policies and Procedures for Stage 3 Critical 

Processes—Developing a Complete Investment Portfolio 31 

Figures 

Figure 1: Simplified DOD Organizational Structure 6 

Figure 2: The Five ITIM Stages of Maturity with Critical Processes 11 

Figure 3: Working Relationships among DOD Business Investment 

Management System Governance Entities 17 

Page i GAO-07-538 Business Systems Modernization
Figure 4: Simplified Process Flow of Certification Reviews and 

Approvals 20

Figure 5: Simplified Process Flow of Annual Reviews 21 

Abbreviations 

ASD(NII)/CIO Assistant Secretary of Defense (Networks and Information 

Integration)/Chief Information Officer 

BEA business enterprise architecture 

BMA business mission area 

BTA Business Transformation Agency 

DAS Defense Acquisition System 

DBSAE Defense Business Systems Acquisition Executive 

DBSMC Defense Business Systems Management Committee 

DITPR DOD Information Technology Portfolio Repository 

DOD Department of Defense 

IRB Investment Review Board 

IT information technology 

ITIM Information Technology Investment Management 

framework 

JCIDS Joint Capabilities Integration and Development System 

MAIS Major Automated Information System 

MDAP Major Defense Acquisition Programs 

OMB Office of Management and Budget 

OSD Office of the Secretary of Defense 

PCA pre-certification authority 

PPBE Planning, Programming, Budgeting, and Execution

USD(AT&L) Under Secretary of Defense (Acquisition, Technology, and 

Logistics) 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 

Page ii GAO-07-538 Business Systems Modernization
United States Government Accountability Office

Washington, DC 20548 

May 11, 2007

Congressional Committees 

For decades, the Department of Defense (DOD) has been challenged in 

modernizing its timeworn business systems.1 In 1995, we designated DOD’s 

business systems modernization program as high risk, and we continue to 

designate it as such today.2

 As our research on public and private sector 

organizations shows, one essential ingredient to a successful systems 

modernization program is having an effective institutional approach to 

managing information technology (IT) investments. 

In May 2001, we recommended that the department establish a corporate 

approach to investment control and decision making.3

 Between 2001 and 

2005, we reported that the department’s business systems modernization 

program was still not being effectively managed,4 and we made additional 

investment-related recommendations. Congress subsequently included 

provisions in the Ronald W. Reagan National Defense Authorization Act 

 1

Business systems are information systems that include financial and nonfinancial systems

and support DOD’s business operations, such as civilian personnel, finance, health, 

logistics, military personnel, procurement, and transportation. 

2

GAO, High-Risk Series: An Update, GAO-07-310 (Washington, D.C.: January 2007). 

3

GAO, Information Technology: Architecture Needed to Guide Modernization of DOD’s 

Financial Operations, GAO-01-525 (Washington, D.C.: May 17, 2001).

4

See, for example, GAO, DOD Business Systems Modernization: Long-standing 

Weaknesses in Enterprise Architecture Development Need to Be Addressed, GAO-05-702 

(Washington, D.C.: July 22, 2005); DOD Business Systems Modernization: Billions Being

Invested without Adequate Oversight, GAO-05-381 (Washington, D.C.: Apr. 29, 2005); DOD

Business Systems Modernization: Limited Progress in Development of Business 

Enterprise Architecture and Oversight of Information Technology Investments, 

GAO-04-731R (Washington, D.C.: May 17, 2004); DOD Business Systems Modernization: 

Important Progress Made to Develop Business Enterprise Architecture, but Much Work 

Remains, GAO-03-1018 (Washington, D.C.: Sept. 19, 2003); Business Systems 

Modernization: Summary of GAO’s Assessment of the Department of Defense’s Initial

Business Enterprise Architecture, GAO-03-877R (Washington, D.C.: July 7, 2003);

Information Technology: Observations on Department of Defense’s Draft Enterprise

Architecture, GAO-03-571R (Washington, D.C.: Mar. 28, 2003); DOD Business Systems 

Modernization: Improvements to Enterprise Architecture Development and

Implementation Efforts Needed, GAO-03-458 (Washington, D.C.: Feb. 28, 2003); and

GAO-01-525. 

Page 1 GAO-07-538 Business Systems Modernization
for Fiscal Year 20055

 that reflected our recommendations, including those 

for establishing and implementing effective business system investment 

management structures and processes. 

Between 2005 and 2006,6

 we reported that DOD had made important 

progress in establishing and implementing these structures and processes, 

but that much remained to be accomplished relative to the act’s

requirements. For example, we reported that the department’s business 

system investment approach was not institutionalized at all levels of the 

department. 

To support GAO’s legislative mandate to review DOD’s annual report on 

its business systems modernization program, and as agreed with your 

offices, the objective of this review was to determine whether DOD’s 

corporate investment management approach comports with relevant 

federal guidance. To accomplish our objective, we analyzed documents 

and interviewed agency officials to determine whether DOD has developed 

the structures, policies, and procedures associated with executing those 

key practices in our IT Investment Management (ITIM) framework that

assist organizations in complying with the investment management 

provisions of the Clinger-Cohen Act of 1996.7

 This framework provides a 

hierarchical maturity model for IT investment management and a method

for evaluating and assessing the maturity of an agency’s investment 

management. We performed our work at DOD headquarters in Arlington, 

Virginia, from August 2006 through April 2007 in accordance with 

generally accepted government auditing standards. Details on our 

objective, scope, and methodology are contained in appendix I. 

 5

Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005, Pub. L. No. 

108-375, § 332, 118 Stat. 1811, 1851-1856 (Oct. 28, 2004) (codified in part at 10 U.S.C. §

2222). 

6

GAO, Defense Business Transformation: A Comprehensive Plan, Integrated Efforts, and 

Sustained Leadership Are Needed to Assure Success, GAO-07-229T (Washington, D.C.: 

Nov. 16, 2006); Business Systems Modernization: DOD Continues to Improve 

Institutional Approach, but Further Steps Needed, GAO-06-658 (Washington, D.C.: May 15,

2006); and DOD Business Systems Modernization: Important Progress Made in

Establishing Foundational Architecture Products and Investment Management 

Practices, but Much Work Remains, GAO-06-219 (Washington, D.C.: Nov. 23, 2005).

7

GAO, Information Technology Investment Management: A Framework for Assessing and

Improving Process Maturity, GAO-04-394G (Washington, D.C.: March 2004). 

Page 2 GAO-07-538 Business Systems Modernization
DOD has established the management structures needed to effectively 

manage its business system investments, but it has not fully defined many

of the related policies and procedures that our framework defines. 

Specifically, DOD has fully defined four of nine key practices that call for 

project-level policies and procedures, and one of the five practices that 

call for portfolio-level policies and procedures. For example, regarding 

project-level investment, the department has (1) established an 

enterprisewide investment board and subordinate boards that are 

responsible for business system investment governance, (2) documented 

policies and procedures for ensuring that systems support ongoing and 

future business needs, (3) developed procedures for identifying and 

collecting information about these systems to support investment 

selection and control, and (4) assigned responsibility for ensuring that the 

information collected during project identification meets the needs of the

investment management process. Regarding portfolio-based investment, 

DOD has assigned responsibility to the Under Secretary of Defense for 

Acquisition, Technology, and Logistics for managing business system 

portfolio selection criteria. 

Results in Brief 

However, DOD has not fully documented business system investment 

policies and procedures related to five key project-level management 

practices. For example, policies and procedures do not (1) define how the 

investment selection, acquisition, and funding processes are coordinated; 

(2) specify how the full range of cost, schedule, and benefit data accessible 

by the Investment Review Boards (IRB) are to be used in making selection 

(i.e., certification) decisions; (3) specify how reselection decisions at the 

corporate level (i.e., annual review decisions) consider investments that 

are in operations and maintenance; (4) describe how funding decisions are 

integrated with the process of selecting an investment at the corporate 

level; and (5) provide sufficient oversight and visibility into componentlevel investment management activities, including component reviews of 

systems in operations and maintenance. Furthermore, DOD does not have 

documented policies and procedures for (1) defining the portfolio criteria,

(2) creating the portfolio, (3) evaluating the portfolio, and (4) conducting 

postimplementation reviews for all business systems. 

Regarding project-level investment management practices, DOD officials 

stated that these are performed at the component level, and that 

departmental policies and procedures established for overseeing 

execution of these practices by components are sufficient. Regarding 

portfolio-level practices, however, these officials stated that they intend to

improve departmental policies and procedures for business system 

investments by, for example, establishing a single governance structure, 

Page 3 GAO-07-538 Business Systems Modernization
but plans or time frames for doing so have not been established. According

to our ITIM framework, adequately documenting both the policies and the 

associated procedures that govern how an organization manages its IT 

investment portfolio(s) is important because doing so provides the basis 

for having rigor, discipline, and repeatability in how investments are 

selected and controlled across the entire organization. Until DOD fully 

defines departmentwide policies and procedures for both individual 

projects and portfolios of projects, it risks selecting and controlling these 

business system investments in an inconsistent, incomplete, and ad hoc 

manner, which in turn reduces the chances that these investments will 

meet mission needs in the most cost-effective manner. 

To strengthen DOD’s business system investment management capability, 

we are recommending that the department fully define the policies and 

procedures associated with project-level and portfolio-level investment 

management as discussed in our guidance for IT investment management.8

In written comments on a draft of this report, signed by the Deputy Under

Secretary of Defense (Business Transformation) and reprinted in appendix

II, the department stated that it agreed with the report’s overall 

conclusions, and it described efforts under way and planned that it said 

would address many of the gaps identified in the report. In this regard, the

department partially concurred with five of the report’s recommendations, 

adding that our recommendations and feedback are helpful in guiding 

DOD’s business transformation and related improvement efforts. 

However, the department disagreed with the remaining four 

recommendations for two primary reasons. First, it stated that its existing 

investment management structure already satisfies the intent of these 

recommendations. For example, it stated that its policies already require

the provision of cost, schedule, and funding data as part of investment 

certifications and annual reviews, and that a linkage currently exists 

among the investment selection, acquisition, and funding processes. We do

not agree with this reasoning. Our recommendations are not intended to 

address whether existing policies or guidance provide for the use of cost, 

schedule, and funding data, or whether they state that investment 

selection, acquisition, and funding decision making are linked. Rather, our 

recommendations address the definitions of policy, guidance, and 

supporting procedures that fall short of satisfying the best practices 

 8

GAO-04-394G. 

Page 4 GAO-07-538 Business Systems Modernization
embodied in our ITIM framework. In the case of the above examples, 

while we do not question whether investment data are provided to 

investment decision-making bodies, the department’s policies and 

procedures do not include specific decision criteria that explain how these 

data are to be used to make consistent, repeatable selection and 

reselection decisions across all investments. Furthermore, while we do not 

question that existing guidance contains an illustration depicting a link 

between investment certification and review and other DOD decision 

support processes, including the funding process, neither this guidance 

nor supporting procedures define how this linkage is executed (i.e., how 

investment funding decisions are in fact integrated with investment 

selection decisions). 

Second, DOD stated that our recommendations contradict the 

department’s “tiered accountability” approach to investment management, 

in which responsibility and accountability for business system investment 

management is allocated between the Office of the Secretary of Defense 

(corporate level) and DOD components (subsidiary levels) on the basis of 

investment size and significance. We do not agree with the department’s 

reasoning. We support DOD’s tiered accountability concept because it is

consistent with the hierarchical investment structures described in our 

ITIM framework. Under the department’s current policies and guidance, 

however, most DOD investments are not subject to corporate visibility and 

oversight, either because they do not involve development/modernization 

(i.e., they are in operations and maintenance) or because they do not 

exceed a certain dollar threshold. Our framework recognizes that effective 

implementation of this concept should include appropriate corporate 

visibility into and oversight of investments, either through review and 

approval of those investments that meet certain criteria or through 

awareness of a subordinate board’s investment management activities. 

Moreover, this visibility and oversight should extend to the entire portfolio 

of investments, including those that are in operations and maintenance. To

ensure that this occurs, applicable policies and procedures need to 

explicitly cover all such investments and need to define how this is to be 

accomplished. 

DOD is a massive and complex organization. To illustrate, the department 

reported that its fiscal year 2006 operations involved approximately $1.4 

trillion in assets and $2.0 trillion in liabilities, more than 2.9 million 

military and civilian personnel, and $581 billion in net cost of operations. 

To date, for fiscal year 2007, the department received appropriations of 

about $501 billion. Organizationally, the department includes the Office of 

Background 

Page 5 GAO-07-538 Business Systems Modernization
the Secretary of Defense (OSD), the Chairman of the Joint Chiefs of Staff, 

the military departments, numerous defense agencies and field activities, 

and various unified combatant commands that are responsible for either 

specific geographic regions or specific functions. (See fig. 1 for a 

simplified depiction of DOD’s organizational structure.) 

Figure 1: Simplified DOD Organizational Structure

Secretary of Defense

Department of

the Army

Department of

the Navy

Department of

the Air Force

Office of the 

Secretary of

Defense

DOD Field

Activities

Combatant

Commands

Defense

Agencies

Inspector

General

Joint Chiefs

of Staff

Source: GAO based on DOD documentation.

Deputy Secretary of Defense

a

a

The Chairman of the Joint Chiefs of Staff serves as the spokesman for the commanders of the 

combatant commands, especially on the administrative requirements of their commands. 

In support of its military operations, the department performs an 

assortment of interrelated and interdependent business functions, 

including logistics management, procurement, health care management, 

and financial management. As we have previously reported,9

 the systems 

environment that supports these business functions is overly complex and 

error-prone, and is characterized by (1) little standardization across the 

department, (2) multiple systems performing the same tasks, (3) the same 

data stored in multiple systems, and (4) the need for data to be entered 

manually into multiple systems. Moreover, according to DOD, this systems 

environment is comprised of approximately 3,100 separate business 

systems. For fiscal year 2007, Congress appropriated approximately $15.7

billion to DOD, and for fiscal year 2008, DOD has requested about $15.9 

billion in appropriated funds to operate, maintain, and modernize these 

business systems and the associated infrastructures. 

 9

GAO-06-658. 

Page 6 GAO-07-538 Business Systems Modernization
As we have previously reported,10 the department’s nonintegrated and 

duplicative systems impair DOD’s ability to combat fraud, waste, and 

abuse. In fact, DOD currently bears responsibility, in whole or in part, for 

15 of our 27 high-risk areas.11 Eight of these areas are specific to DOD,12

and the department shares responsibility for 7 other governmentwide highrisk areas.13 DOD’s business systems modernization is one of the high-risk 

areas, and it is an essential enabler to addressing many of the department’s 

other high-risk areas. For example, modernized business systems are 

integral to the department’s efforts to address its financial, supply chain, 

and information security management high-risk areas. 

IT Investment 

Management Is Critical to 

Achieving Successful 

Systems Modernization 

A corporate approach to IT investment management is characteristic of 

successful public and private organizations. Recognizing this, Congress 

enacted the Clinger-Cohen Act of 1996,14 which requires the Office of 

Management and Budget (OMB) to establish processes to analyze, track, 

and evaluate the risks and results of major capital investments in IT 

systems made by executive agencies.15 In response to the Clinger-Cohen 

Act and other statutes, OMB has developed policy and issued guidance for 

the planning, budgeting, acquisition, and management of federal capital

 10See, for example, GAO, DOD Travel Cards: Control Weaknesses Resulted in Millions of

Dollars of Improper Payments, GAO-04-576 (Washington, D.C.: June 9, 2004); Military

Pay: Army National Guard Personnel Mobilized to Active Duty Experienced Significant 

Pay Problems, GAO-04-89 (Washington, D.C.: Nov. 13, 2003); and Defense Inventory:

Opportunities Exist to Improve Spare Parts Support Aboard Deployed Navy Ships, 

GAO-03-887 (Washington, D.C.: Aug. 29, 2003). 

11GAO-07-310. 

12These 8 high-risk areas include DOD’s (1) overall approach to business transformation, 

(2) business systems modernization, (3) financial management, (4) personnel security 

clearance program, (5) supply chain management, (6) support infrastructure management, 

(7) weapon systems acquisition, and (8) contract management. 

13The 7 governmentwide high-risk areas are (1) disability programs, (2) ensuring the 

effective protection of technologies critical to U.S. national security interests, 

(3) interagency contracting, (4) information systems and critical infrastructure, 

(5) information-sharing for homeland security, (6) human capital, and (7) real property.

14The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11101-11704. This act expanded the 

responsibilities of OMB and the agencies that had been set under the Paperwork Reduction 

Act with regard to IT management. See 44 U.S.C. 3504(a)(1)(B)(vi) (OMB); and 44 U.S.C.

3506(h)(5) (agencies). 

15We have made recommendations to improve OMB’s process for monitoring high-risk IT

investments; see GAO, Information Technology: OMB Can Make More Effective Use of Its 

Investment Reviews, GAO-05-276 (Washington, D.C.: Apr. 15, 2005). 

Page 7 GAO-07-538 Business Systems Modernization
assets.16 We have also issued guidance in this area,17 which defines 

institutional structures, such as the IRBs; processes for developing 

information on investments (such as costs and benefits); and practices to 

inform management decisions (such as whether a given investment is 

aligned with an enterprise architecture). 

IT investment management is a process for linking IT investment decisions 

to an organization’s strategic objectives and business plans. Consistent 

with this, the federal approach to IT investment management focuses on 

selecting, controlling, and evaluating investments in a manner that 

minimize risks while maximizing the return of investment.18

IT Investment Management: A 

Brief Description 

• During the selection phase, the organization (1) identifies and analyzes 

each project’s risks and returns before committing significant funds to any 

project and (2) selects those IT projects that will best support its mission 

needs. 

• During the control phase, the organization ensures that projects, as they 

develop and investment expenditures continue, meet mission needs at the

expected levels of cost and risk. If the project is not meeting expectations

or if problems arise, steps are quickly taken to address the deficiencies.

• During the evaluation phase, expected results are compared with actual 

results after a project has been fully implemented. This comparison is 

done to (1) assess the project’s impact on mission performance, 

(2) identify any changes or modifications to the project that may be 

 16This policy is set forth and guidance is provided in OMB Circular A-11 (Nov. 2, 2005) 

(section 300), and in OMB’s Capital Programming Guide, which directs agencies to 

develop, implement, and use a capital programming process to build their capital asset 

portfolios. 

17See, for example, GAO-04-394G; GAO, Information Technology: A Framework for

Assessing and Improving Enterprise Architecture Management (Version 1.1), 

GAO-03-584G (Washington, D.C.: April 2003); and Assessing Risks and Returns: A Guide

for Evaluating Federal Agencies’ IT Investment Decision-making, GAO/AIMD-10.1.13

(Washington, D.C.: February 1997). 

18GAO-04-394G; GAO/AIMD-10.1.13; GAO, Executive Guide: Improving Mission 

Performance Through Strategic Information Management and Technology, GAO/AIMD94-115 (Washington, D.C.: May 1994); and Office of Management and Budget, Evaluating 

Information Technology Investments, A Practical Guide (Washington, D.C.: November 

1995). 

Page 8 GAO-07-538 Business Systems Modernization
needed, and (3) revise the investment management process based on 

lessons learned. 

Our ITIM framework consists of five progressive stages of maturity for any 

given agency relative to selecting, controlling, and evaluating its 

investment management capabilities.19 (See fig. 2 for the five ITIM stages 

of maturity.) This framework is grounded in our research of IT investment 

management practices of leading private and public sector organizations. 

The maturity stages are cumulative; that is, to attain a higher stage, an 

agency must institutionalize all of the critical processes at the lower 

stages, in addition to the higher stage critical processes. 

The framework can be used to assess the maturity of an agency’s 

investment management processes and as a tool for organizational 

improvement. The overriding purpose of the framework is to encourage 

investment selection and control and to evaluate processes that promote 

business value and mission performance, reduce risk, and increase 

accountability and transparency. We have used the framework in several

of our evaluations,20 and a number of agencies have adopted it.

With the exception of the first stage, each maturity stage is composed of 

“critical processes” that must be implemented and institutionalized for the 

organization to achieve that stage. Each ITIM critical process consists of 

“key practices”—to include organizational structures, policies, and 

procedures—that must be executed to implement the critical process. It is 

not unusual for an organization to perform key practices from more than 

one maturity stage at the same time. However, our research shows that 

agency efforts to improve investment management capabilities should 

Overview of GAO’s ITIM 

Maturity Framework 

 19GAO-04-394G. 

20GAO, Information Technology: Centers for Medicare & Medicaid Services Needs to 

Establish Critical Investment Management Capabilities, GAO-06-12 (Washington, D.C.: 

Oct. 28, 2005); Information Technology: HHS Has Several Investment Management 

Capabilities in Place, but Needs to Address Key Weaknesses, GAO-06-11 (Washington, 

D.C.: Oct. 28, 2005); Information Technology: FAA Has Many Investment Management 

Capabilities in Place, but More Oversight of Operational Systems Is Needed, GAO-04-822 

(Washington, D.C.: Aug. 20, 2004); Bureau of Land Management: Plan Needed to Sustain

Progress in Establishing IT Investment Management Capabilities, GAO-03-1025 

(Washington, D.C.: Sept. 12, 2003); Information Technology: Departmental Leadership 

Crucial to Success of Investment Reforms at Interior, GAO-03-1028 (Washington, D.C.: 

Sept. 12, 2003); United States Postal Service: Opportunities to Strengthen IT Investment 

Management Capabilities, GAO-03-3 (Washington, D.C.: Oct. 15, 2002); and Information 

Technology: DLA Needs to Strengthen Its Investment Management Capability, 

GAO-02-314 (Washington, D.C.: Mar. 15, 2002). 

Page 9 GAO-07-538 Business Systems Modernization
focus on implementing all lower-stage practices before addressing higherstage practices. 

In the ITIM framework, Stage 2 critical processes lay the foundation by 

establishing successful, predictable, and repeatable investment control 

processes at the project level. At this stage, the emphasis is on establishing 

basic capabilities for selecting new IT projects; controlling projects so that 

they finish predictably within the established cost, schedule, and 

performance expectations; and identifying and mitigating exposure to risk. 

Stage 3 is where the agency moves from project-centric processes to 

portfolio-based processes and evaluates potential investments according

to how well they support the agency’s missions, strategies, and goals. This

stage focuses on continually assessing both proposed and ongoing 

projects as part of complete investment portfolios—integrated and 

competing sets of investment options. It also focuses on maintaining 

mature, integrated selection (and reselection); control; and 

postimplementation evaluation processes. This portfolio perspective 

allows decision makers to consider the interaction among investments and 

the contributions to organizational mission goals and strategies that could 

be made by alternative portfolio selections, rather than to focus 

exclusively on the balance between the costs and benefits of individual 

investments. Organizations implementing Stages 2 and 3 practices have in

place capabilities that assist in establishing selection, control, and 

evaluation structures, policies, procedures, and practices that are required 

by the investment management provisions of the Clinger-Cohen Act.21

Stages 4 and 5 require the use of evaluation techniques to continuously 

improve both investment processes and portfolios to better achieve 

strategic outcomes. At Stage 4, an organization has the capacity to conduct

IT succession activities and, therefore, can plan and implement the 

deselection of obsolete, high-risk, or low-value IT investments. An 

organization with Stage 5 maturity conducts proactive monitoring for 

breakthrough technologies that will enable it to change and improve its 

business performance. 

 21The Clinger-Cohen Act of 1996, 40 U.S.C. §§ 11311-11313. 

Page 10 GAO-07-538 Business Systems Modernization
Figure 2: The Five ITIM Stages of Maturity with Critical Processes 

DOD’s major system investments (i.e., weapon and business systems) are 

governed by three management systems—the Joint Capabilities 

Integration and Development System (JCIDS); the Planning, Programming, 

Budgeting, and Execution (PPBE) system; and the Defense Acquisition 

System (DAS). 

Source: GAO. 

- Optimizing the investment process

- Using IT to drive strategic business change

- Improving the portfolio's performance 

- Managing the succession of information systems

- Defining the portfolio criteria

- Creating the portfolio 

- Evaluating the portfolio 

- Conducting postimplementation reviews

- Instituting the investment board

- Meeting business needs

- Selecting an investment

- Providing investment oversight

- Capturing investment information

Stage 5: Leveraging IT for 

 strategic outcomes

Maturity stages Critical processes

Stage 4: Improving the

 investment process

Stage 3: Developing a complete

 investment portfolio

Stage 2: Building the investment

 foundation

Stage 1: Creating investment awareness IT spending without disciplined investment processes

Overview of DOD’s 

Corporate Approach for 

Identifying, Funding, and 

Acquiring All System 

Investments 

• JCIDS is a need-driven, capabilities-based approach to identify warfighting 

needs and meet future joint forces challenges. It is intended to identify 

future capabilities for DOD; address capability gaps and mission needs 

recognized by the Joint Chiefs of Staff or derived from strategic guidance, 

such as the National Security Strategy Report22 or Quadrennial Defense 

Review;23 and identify alternative solutions by considering a range of 

doctrine, organization, training, materiel, leadership and education, 

personnel, and facilities solutions. According to DOD, the Joint Chiefs of 

 22The National Security Strategy Report required by 50 U.S.C. 404a is a comprehensive 

report on the national security strategy of the United States submitted by the President to

Congress.

23See 10 U.S.C. 118. The Quadrennial Defense Review is a comprehensive examination of 

the national defense strategy, force structure, force modernization plans, infrastructure, 

budget plan, and other elements of the defense program and policies of the United States 

with a view toward determining and expressing the defense strategy of the United States

and establishing a defense program for the next 20 years. 

Page 11 GAO-07-538 Business Systems Modernization
Staff, through the Joint Requirements Oversight Council, has primary 

responsibility for defining and implementing JCIDS. 

• PPBE is a calendar-driven approach that is composed of four phases that 

occur over a moving 2-year cycle. The four phases—planning, 

programming, budgeting, and executing—define how budgets for each 

DOD component and the department as a whole are created, vetted, and 

executed. As recently reported,24 the components start programming and 

budgeting for addressing a JCIDS-identified capability gap or mission need 

several years before actual product development under DAS begins, and

before OSD formally reviews the components’ programming and 

budgeting proposals (i.e., Program Objective Memorandums). Once 

reviewed and approved, the financial details in the Program Objective

Memorandums become part of the President’s budget request to Congress. 

During budget execution, components may submit program change 

proposals or budget change proposals, or both (e.g., program cost 

increases or schedule delays). According to DOD, the OSD Under 

Secretary of Defense (Policy), the Director for Program Analysis and 

Evaluation,25 and the Under Secretary of Defense (Comptroller) have 

primary responsibility for defining and implementing the PPBE system.

• DAS is described in the DOD Directive 5000.1 and the DOD Instruction 

5000.226 and establishes the procedures for the Defense Acquisition 

Management Framework, which consists of three event-based milestones 

associated with five key program life-cycle phases. These five phases are 

as follows: 

1. Concept Refinement: Intended to refine the initial JCIDS-validated 

system solution (concept) and create a strategy for acquiring the 

investment solution. A decision is made at the end of this phase 

(milestone A decision) regarding whether to move to the next phase 

(Technology Development). 

 24GAO, Best Practices: An Integrated Portfolio Management Approach to Weapon System 

Investments Could Improve DOD’s Acquisition Outcomes, GAO-07-388 (Washington, D.C.: 

Mar. 30, 2007). 

25The Director for Program Analysis and Evaluation is the principal staff assistant who 

conducts independent analysis for, and provides independent advice on, all DOD program 

and evaluation matters to the Secretary and Deputy Secretary of Defense. 

26DOD Directive 5000.1, May 12, 2003 and DOD Instruction 5000.2, May 12, 2003. 

Page 12 GAO-07-538 Business Systems Modernization
2. Technology Development: Intended to determine the appropriate set of 

technologies to be integrated into the investment solution by 

iteratively assessing the viability of various technologies while 

simultaneously refining user requirements. Once the technology has 

been demonstrated in a relevant environment, a decision is made at 

the end of this phase (milestone B decision) regarding whether to 

move to the next phase (System Development and Demonstration). 

3. System Development and Demonstration: Intended to develop a 

system or a system increment and demonstrate through developer 

testing that the system/system increment can function in its target 

environment. A decision is made at the end of this phase (milestone C 

decision) regarding whether to move to the next phase (Production 

and Deployment). 

4. Production and Deployment: Intended to achieve an operational 

capability that satisfies the mission needs, as verified through 

independent operational test and evaluation, and ensures that the 

system is implemented at all applicable locations. 

5. Operations and Support: Intended to operationally sustain the system 

in the most cost-effective manner over its life cycle. 

A key principle of DAS is that investments are assigned a category, where 

programs of increasing dollar value and management interest are subject

to more stringent oversight. For example, Major Defense Acquisition 

Programs (MDAP)

27 and Major Automated Information Systems (MAIS)28

are large, expensive programs subject to the most extensive statutory and 

regulatory reporting requirements and, unless delegated, are reviewed by

acquisition boards at the DOD corporate level. Smaller and less risky 

acquisitions are generally reviewed at the component executive or lower 

levels. Another key principle is that DAS requires acquisition management 

 27A MDAP is an acquisition program that is estimated by the Under Secretary of Defense for

Acquisition, Technology, and Logistics to require an eventual total expenditure for 

research, development, and test and evaluation of more than $365 million (fiscal year 2000 

constant dollars) or, for procurement, of more than $2.190 billion (fiscal year 2000 constant

dollars). 

28A MAIS is a program or initiative that is so designated by the Assistant Secretary of

Defense (Networks and Information Integration)/Chief Information Officer or that is 

estimated to require program costs in any single year in excess of $32 million (fiscal year 

2000 constant dollars), total program costs in excess of $126 million (fiscal year 2000 

constant dollars), or total life-cycle costs in excess of $378 million (fiscal year 2000 

constant dollars). 

Page 13 GAO-07-538 Business Systems Modernization
under the direction of a milestone decision authority.29 The milestone 

decision authority—with support from the program manager and advisory 

boards, such as the Defense Acquisition Board30 and the IT Acquisition 

Board31—determines the project’s baseline cost, schedule, and 

performance commitments. The Under Secretary of Defense for 

Acquisition, Technology, and Logistics (USD(AT&L)) has primary 

responsibility for defining and implementing DAS. 

DOD’s business system investments are also governed by a fourth 

management system that addresses how these investments are reviewed, 

certified, and approved for compliance with the business enterprise 

priorities and activities outlined by the business enterprise architecture 

(BEA). For the purposes of this report, we refer to this fourth management 

system as the Business Investment Management System. This fourth 

management system is described in the following text in terms of 

governance entities, tiered accountability, and business system investment 

certification reviews and approvals. According to DOD, these four 

management systems are the means by which DOD selects, controls, and 

evaluates its business system investments. 

In 2005, the department reassigned responsibility for providing executive 

leadership for the direction, oversight, and execution of its business 

systems modernization efforts to several entities. These entities and their 

responsibilities include the following: 

DOD Business System 

Investments Are Subject to 

a Fourth Management 

System 

Business System Investment 

Roles and Responsibilities 

 29According to DOD, the milestone decision authority is the designated individual who has

overall responsibility for an investment. This person has the authority to approve an

investment’s progression in the acquisition process and is responsible for reporting cost,

schedule, and performance results. For example, the milestone decision authority for a

MDAP program, when not delegated to the component level, is the Under Secretary of 

Defense for Acquisition, Technology, and Logistics, and the milestone decision authority

for a MAIS system is the Assistant Secretary of Defense (Networks and Information 

Integration)/Chief Information Officer or a designee. 

30The Defense Acquisition Board, chaired by the Under Secretary of Defense for 

Acquisition, Technology, and Logistics, conducts reviews for MDAPs at major program

milestones and documents the decision(s) resulting from the review in an Acquisition 

Decision Memorandum. 

31The IT Acquisition Board, chaired by the Assistant Secretary of Defense (Networks and 

Information Integration)/Chief Information Officer, conducts reviews for MAIS at major 

program milestones and documents the decision(s) resulting from the review in an

Acquisition Decision Memorandum. 

Page 14 GAO-07-538 Business Systems Modernization
• The Defense Business Systems Management Committee (DBSMC) serves 

as the highest-ranking governance body for business systems 

modernization activities. 

• The Principal Staff Assistants serve as the certification authorities for 

business system modernizations in their respective core business 

missions. 

• The IRBs are chartered by the Principal Staff Assistants and are the review

and decision-making bodies for business system investments in their 

respective areas of responsibility.32 

• The component pre-certification authority (PCA) is accountable for the 

component’s business system investments and acts as the component’s 

principal point of contact for communication with the IRBs. 

• The Business Transformation Agency (BTA) is responsible for leading and 

coordinating business transformation efforts across the department. The 

BTA is organized into seven directorates, one of which is the Defense 

Business Systems Acquisition Executive (DBSAE)—the component 

acquisition executive for DOD enterprise-level (DOD-wide) business 

systems and initiatives. This directorate is responsible for developing,

coordinating, and integrating enterprise-level projects, programs, systems, 

and initiatives—including managing resources such as fiscal, personnel, 

and contracts for assigned systems and programs. 

Table 1 lists these entities and provides greater detail on their roles, 

responsibilities, and composition. Figure 3 provides a simplified 

illustration of the relationships among these entities. 

 32The four IRBs are for (1) Financial Management, established by the Deputy Under 

Secretary of Defense for Financial Management; (2) Weapon Systems Lifecycle 

Management and Materiel Supply and Services Management; (3) Real Property and 

Installations Lifecycle Management, both established by the USD(AT&L); and (4) Human 

Resources Management, established by the Under Secretary of Defense for Personnel and 

Readiness. 

Page 15 GAO-07-538 Business Systems Modernization
Table 1: DOD Business Investment Management System Entities’ Roles, Responsibilities, and Composition 

Entity Roles and responsibilities Composition 

DBSMC • Serves as approving authority for business system 

certifications. 

• Establishes policies and approves the business mission 

area (BMA)a

 strategic plan, the transition plan for 

implementation for business systems modernization, the 

transformation program baseline, and the BEA. 

Chaired by the Deputy Secretary of 

Defense; vice chair is the USD(AT&L). 

Includes senior leadership in OSD; the 

military departments’ secretaries; and 

defense agencies’ heads, such as the 

Assistant Secretary of Defense (Networks 

and Information Integration)/Chief 

Information Officer (ASD(NII)/CIO), the 

Vice Chairman of the Joint Chiefs of Staff, 

and the commanders of the U.S. 

Transportation Command and the Joint 

Forces Command. 

Principal Staff Assistants/ 

Certification Authorities 

• Support the DBSMC’s management of enterprise business

IT investments. 

• Serve as the certification authorities accountable for the 

obligation of funds for respective business systems 

modernization within designated core business missions.b

• Provide the DBSMC with recommendations for system 

investment approval. 

Under Secretaries of Defense for 

Acquisition, Technology, and Logistics; 

Comptroller; and Personnel and 

Readiness. 

IRBs • Serve as the oversight and investment decision-making 

bodies for those business capabilities that support 

activities under their designated areas of responsibility. 

• Recommend certification for all business system 

investments costing more than $1 million that are 

integrated and compliant with the BEA. 

Includes the Principal Staff Assistants, 

Joint Staff, ASD(NII)/CIO, core business 

mission area representatives, military 

departments, defense agencies, and 

combatant commands. 

Component PCA • Ensures that component-level investment review 

processes integrate with the investment management 

system. 

• Identifies those component systems that require IRB 

certification and prepares, reviews, approves, validates, 

and transfers investment documentation as required. 

• Assesses and precertifies architecture compliance of 

component systems submitted for certification and annual 

review. 

• Acts as the component’s principal point of contact for 

communication with the IRBs. 

Includes the Chief Information Officer from 

the Air Force; the Principal Director of 

Governance, Acquisition, and Chief 

Knowledge Office from the Army; the Chief 

Information Officer from the Navy; and 

comparable representatives from other 

defense agencies. 

BTA • Serves as the day-to-day management entity of the 

business transformation effort at the DOD enterprise level. 

• Provides support to the DBSMC and the IRBs. 

• Operates under the authority of the USD(AT&L) under the 

direction of the Deputy Under Secretary of Defense for 

Business Transformation and the Deputy Under Secretary 

of Defense for Financial Management. 

Comprised of seven directorates (DBSAE, 

Enterprise Integration, Transformation 

Planning and Performance, Transformation 

Priorities and Requirements, Investment 

Management, Warfighter Support Office, 

and Chief of Staff). 

Source: GAO based on DOD documentation.

Page 16 GAO-07-538 Business Systems Modernization
a

According to DOD, the BMA is responsible for ensuring that capabilities, resources, and materiel are 

reliably delivered to the warfighter. Specifically, the BMA addresses areas such as real property and 

human resources management. 

b

DOD has five core business missions: Human Resources Management, Weapon System Lifecycle 

Management, Materiel Supply and Services Management, Real Property and Installations Lifecycle 

Management, and Financial Management. 

Figure 3: Working Relationships among DOD Business Investment Management System Governance Entities 

DBSMC

Financial 

Management

BTA

Weapon Systems

Lifecycle Management

and Materiel Supply and

Services Management

Real Property and

Installations Lifecycle

Management

Human Resources

Management

IRBs

Component PCA Component PCA Component PCA

Source: GAO based on DOD documentation. 

Principal Staff 

Assistant 

Certification 

Authorities

Under Secretary 

of Defense (Personnel

and Readiness)

USD 

(AT&L) Under Secretary 

of Defense 

(Comptroller)

According to DOD, in 2005 it adopted a tiered accountability approach to 

business transformation. Under this approach, responsibility and 

accountability for business investment management is allocated between 

the DOD corporate (i.e., OSD) and the components on the basis of the 

amount of development/modernization funding involved and the 

investment’s “tier.” DOD corporate is responsible for ensuring that all 

business systems with a development/modernization investment in excess 

of $1 million are reviewed by the IRBs for compliance with the BEA, 

certified by the Principal Staff Assistants, and approved by the DBSMC. 

Components are responsible for certifying development/modernization 

investments with total costs of $1 million or less. All DOD development 

Tiered Accountability

Page 17 GAO-07-538 Business Systems Modernization
and modernization efforts are also assigned a tier on the basis of the 

acquisition category or the size of the financial investment, or both. 

According to DOD, a system is given a tier designation when it passes 

through the certification process. Table 2 describes the four investment 

tiers and identifies the associated reviewing and approving entities. 

Table 2: DOD’s Investment Tiers 

Tier description Reviewing/Approving entities 

Tier 1 MAIS and MDAPs IRB and DBSMC 

Tier 2 Exceeding $10 million in total 

development/modernization costs, 

but not designated MAIS or MDAPs 

IRB and DBSMC 

Tier 3 Exceeding $1 million and up to $10 

million in total 

development/modernization costs 

IRB and DBSMC 

Tier 4 Investment funding required up to $1 

million 

Component-level review only (unless the 

system or line of business it supports is

designated as special interest by the 

Certification Authority) 

Source: DOD. 

DOD’s business investment management system includes two types of 

reviews for business systems: certification and annual reviews. 

Certification reviews apply to new modernization projects with total cost 

over $1 million. This review focuses on program alignment with the BEA 

and must be completed before components obligate funds for programs. 

The annual review applies to all business programs. The focus for the 

annual review is to determine whether the system development effort is 

meeting its milestones and addressing its IRB certification conditions. 

Business Investment 

Certification Reviews and 

Approvals 

Certification reviews and approvals: Tiers 1 through 3 business system 

investments are certified at two levels—component-level precertification 

and corporate-level certification and approval. At the component level, 

program managers prepare, enter, maintain, and update information about 

their investments in the DOD IT Portfolio Repository (DITPR),33 such as 

regulatory compliance reporting, an architectural profile, and

requirements for investment certification and annual reviews. The 

 33DITPR is DOD’s authoritative repository for certain information about DOD’s business 

systems, such as system names and the responsible DOD components, that are required for 

the certification, approval, and annual reviews of these business system investments. 

Page 18 GAO-07-538 Business Systems Modernization
component PCA validates that the system information is complete and 

accessible on the IRB Portal, reviews system compliance with the BEA 

and enterprise transition plan, and verifies the economic viability analysis. 

The PCA asserts the status and validity of the investment information by 

submitting a component precertification letter to the appropriate IRB for 

its review. 

At the corporate level, the IRB reviews the system information and 

precertification letter submitted by the PCA to determine whether to 

recommend investment certification. On completion of its review, a 

certification memorandum is prepared and signed by the designated 

certification authority34 that documents the IRB’s system certification 

decisions and any related conditions. The memorandum is then forwarded 

to the DBSMC, which either approves or disapproves the IRB’s decisions 

and issues a memorandum containing its decisions. If the DBSMC 

disapproves a system investment, it is up to the component PCA to decide 

whether to resubmit the investment after it has resolved the relevant

issues. Figure 4 provides a simplified overview of the process flow of 

certification reviews and approvals. 

 34The certification authority is the designated Principal Staff Assistant with responsibility 

for review, approval, and oversight of the planning, design, acquisition, deployment,

operation, maintenance, and modernization of defense business systems.

Page 19 GAO-07-538 Business Systems Modernization
Figure 4: Simplified Process Flow of Certification Reviews and Approvals 

Reviews and approves

IRB certification

DBSMC

Reviews program information, 

submits precertification letter, 

updates certification information in

DITPR, and uploads all information

to the IRB Portal 

Enters and maintains program

information in DITPR

Program Manager

CorporateLevel

Certification

and

Approval

Not 

approved

Source: GAO based on DOD documentation.

Reviews and certifies

investments

IRB/Certification Authority

Approved

Approved

Submits

Business System Investments

Component PCA

ComponentLevel

Precertification

Not 

approved

Not 

approved

Annual reviews: Tiers 1 through 4 business system investments are 

annually reviewed at two levels—the component level and the corporate 

level. At the component level, program managers review and update 

information on all tiers of investments, both in modernization and 

operations and maintenance, on an annual basis in DITPR. The updates for 

Tiers 1 through 3 with system development/modernization include cost, 

milestone, and risk variances and actions or issues related to certification 

conditions. The PCA then verifies and submits the information for Tiers 1 

through 3 systems in development/modernization for IRB review in an 

annual review assertion letter. The letter addresses system compliance 

Page 20 GAO-07-538 Business Systems Modernization
with the BEA and the enterprise transition plan, and includes investment 

cost, schedule, and performance information.35

At the corporate level, the IRBs annually review certified Tiers 1 through 3 

investments in development/modernization. These reviews focus on 

program compliance with the BEA, program performance against cost and 

milestone baselines, and progress in meeting certification conditions. The 

IRBs can revoke an investment’s certification when the system has 

significantly failed to achieve performance commitments (i.e., capabilities 

and costs). When this occurs, the component must address the IRB’s 

concerns and resubmit the investment for certification. Figure 5 shows a 

simplified overview of the process flow of annual reviews. 

Figure 5: Simplified Process Flow of Annual Reviews 

 

Reviews investments previously 

certified by the DBSMC 

IRB/Certification Authority

Reviews investment information, 

submits annual review 

assertion letter, and updates

supporting documentation in

DITPR and the IRB Portal 

Updates DITPR

Program Manager

CorporateLevel Annual 

Review

Not 

approved

Not 

approved

Source: GAO based on DOD documentation.

Approved

Submits

Component PCA

ComponentLevel Annual 

Review

Annual Review

of Investments

35In addition, each component PCA submits a list of system names to the IRBs on a

semiannual basis, to include Tier 4 systems and systems in operations and maintenance 

that have been reviewed at the component level. 

Page 21 GAO-07-538 Business Systems Modernization
According to our ITIM framework, organizations should establish the 

management structures needed to manage their investments and build an

investment foundation by having defined policies and procedures for 

selecting and controlling individual projects (Stage 2 capabilities), and 

organizations also should manage projects as a portfolio of investments 

according to defined policies and procedures, treating them as an 

integrated package of competing investment options and pursuing those 

that best meet the strategic goals, objectives, and mission of the agency

(Stage 3 capabilities). These Stages 2 and 3 capabilities assist agencies in

complying with the investment management provisions of the ClingerCohen Act. 

The department has defined four of nine practices that call for projectlevel policies and procedures (see table 4) and one of the five practices 

that call for portfolio-level policies and procedures (see table 6). 

Specifically, it has established the management structures contained in our

ITIM framework, but it has not fully defined many of the related policies 

and procedures. 

DOD Has Established 

the Structures Needed 

to Effectively Manage 

Business System 

Investments, but Has 

Not Fully Defined 

Many of the Related 

Policies and 

Procedures 

With respect to project-level investment management practices, DOD 

officials stated that these are performed at the component level, and that

departmental policies and procedures established for overseeing 

components’ execution of these practices are sufficient. With respect to 

portfolio-level practices, however, these officials stated that they intend to

improve departmental policies and procedures for business system 

investments by, for example, establishing a single governance structure, 

but plans or time frames for doing so have not been established. According

to our ITIM framework, adequately documenting both the policies and the 

associated procedures that govern how an organization manages its IT 

investment portfolio(s) is important because doing so provides the basis 

for having rigor, discipline, and repeatability in how investments are 

selected and controlled across the entire organization. Until DOD fully 

defines departmentwide policies and procedures for both individual 

projects and the portfolios of projects, it risks selecting and controlling 

these business system investments in an inconsistent, incomplete, and ad 

hoc manner, which in turn reduces the chances that these investments will 

meet mission needs in the most cost-effective manner. 

Page 22 GAO-07-538 Business Systems Modernization
At ITIM Stage 2, an organization has attained repeatable and successful IT 

project-level investment control and basic selection processes. Through 

these processes, the organization can identify project expectation gaps 

early and take the appropriate steps to address them. ITIM Stage 2 critical 

processes include (1) defining investment board operations, 

(2) identifying the business needs for each investment, (3) developing a 

basic process for selecting new proposals and reselecting ongoing 

investments, (4) developing project-level investment control processes, 

and (5) collecting information about existing investments to inform 

investment management decisions. Table 3 describes the purpose of each

of these Stage 2 critical processes. 

DOD Has Begun to Build a 

Foundation for ProjectLevel Investment 

Management, but Key 

Policies and Procedures 

Are Not Fully Defined 

Table 3: Stage 2 Critical Processes—Building the Investment Foundation 

Critical process Purpose 

Instituting the investment board To define and establish an appropriate investment management structure and the processes for 

selecting, controlling, and evaluating investments. 

Meeting business needs To ensure that investments support the organization’s business needs and meet users’ needs. 

Selecting an investment To ensure that a well-defined and disciplined process is used to select new proposals and

reselect ongoing investments. 

Providing investment oversight To review the progress of investments, using predefined criteria and checkpoints, in meeting 

cost, schedule, risk, and benefit expectations and to take corrective action when these 

expectations are not being met. 

Capturing investment information To make available to decision makers information to evaluate the impacts and opportunities 

created by proposed (or continuing) investments. 

Source: GAO.

Within these five critical processes are nine key practices that call for 

policies and procedures associated with effective project-level 

management. DOD has fully defined the policies and procedures needed to

ensure that four of these nine practices are performed in a consistent and 

repeatable manner. Specifically, DOD has established the management 

structures by instituting an enterprisewide investment board—the 

DBSMC—composed of senior executives, including the Deputy Secretary 

of Defense, with final approval authority over associated subsidiary 

investment boards. These lower-level investment boards include 

representatives from combatant commands, components, and the Joint 

Chiefs of Staff. In addition, DOD’s business transformation and IRB 

guidance define a process for ensuring that programs support the 

department’s ongoing and future business needs. DOD also has policies 

and procedures for submitting, updating, and maintaining investment 

information in DITPR and the IRB Portal. Furthermore, the department 

has assigned the component’s PCA the responsibility to ensure that 

Page 23 GAO-07-538 Business Systems Modernization
specific investment information contained in the portfolio repository and 

the IRB Portal is accurate and complete. 

However, the policies and procedures associated with the remaining five

project-level management practices are missing critical elements needed 

to effectively carry out essential investment management activities. For 

example: 

• Policies and procedures for instituting the investment board do not 

address how investments that are past the development/modernization 

stage (i.e., in operations and maintenance) are to be governed. Given that 

DOD invests billions of dollars annually in operating and maintaining 

business systems, this is significant. While DOD officials stated that 

component-level policies and procedures address systems outside of 

development/modernization, our ITIM framework emphasizes that the 

corporate investment boards should continue to review important 

information about an investment, such as cost and performance baselines, 

throughout the investment’s life cycle. In addition, the IRB Concept of 

Operations and other IRB documentation do not explicitly outline how the 

business investment management system is coordinated with JCIDS, 

PPBE, and DAS. Without clearly defined visibility into all investments with 

an understanding of decisions reached through other management 

systems, inconsistent decisions may result. 

• Procedures do not specify how the full range of cost, schedule, and benefit 

data is used by the IRBs in making selection (i.e., certification) decisions. 

According to BTA officials, each IRB decides how to ensure compliance 

and determines additional factors to consider when making certification 

decisions. However, DOD did not provide us with any supplemental 

policies or procedures for any of the four IRBs. Without documenting how 

IRBs consider factors such as cost, schedule, and benefits when making 

selection decisions, the department cannot ensure that the IRBs and the 

DBSMC consistently and objectively select proposals that best meet the 

department’s needs and priorities. Furthermore, while the procedures 

specify decision criteria that address statutory requirements for alignment 

to the BEA, the criteria allow programs to postpone demonstrating full 

compliance with several BEA artifacts until the final phases of the 

acquisition process. As a result, programs risk beginning production and 

deployment before ensuring that a business system is fully aligned to the 

BEA. 

• Policies and procedures do not specify how reselection decisions at the 

corporate level (i.e., annual review decisions) consider investments that 

are in operations and maintenance. Without an understanding of how the 

Page 24 GAO-07-538 Business Systems Modernization
IRBs are to consider these investments when making reselection 

decisions, their ability to make informed and consistent reselection and 

termination decisions is limited. 

• Policies and procedures do not specify how funding decisions are 

integrated with the process of selecting an investment at the corporate 

level. Without considering component and corporate budget constraints 

and opportunities, the IRBs risk making investment decisions that do not 

effectively consider the relative merits of various projects and systems 

when funding limitations exist. 

• Policies and procedures do not exist that provide for sufficient oversight 

and visibility into component-level investment management activities, 

including component reviews of systems in operations and maintenance 

and Tier 4 investments. According to DOD officials, investment oversight 

is implemented through tiered accountability, which, among other things, 

allocates responsibility and accountability for business system 

investments with total costs of $1 million or less and those in operations 

and maintenance to the components. However, the department did not 

provide policies and procedures defining how the DBSMC and the IRBs 

ensure visibility into these component processes. This is particularly 

important because, according to DOD’s March 15, 2007, annual report to

Congress, only 285 of approximately 3,100 total business systems have 

completed the IRB certification process and have been approved by the 

DBSMC. DOD officials also stated that the remaining business systems 

have not been through the certification process and have not been given a 

tier designation. Without policies and procedures defining how the 

DBSMC and the IRBs have visibility into and oversight of all business 

system investments, DOD risks components continuing to invest in 

systems that are duplicative, stovepiped, nonintegrated, and unnecessarily 

costly to manage, maintain, and operate. 

Table 4 summarizes our findings relative to DOD’s execution of the nine 

practices that call for the policies and procedures needed to manage IT 

investments at the project level. 

Page 25 GAO-07-538 Business Systems Modernization
Table 4: Summary of Policies and Procedures for Stage 2 Critical Processes—Building the Investment Foundation 

Critical 

process Key practice Rating Summary of evidence 

1. An enterprisewide IT

investment board composed of

senior executives from IT and 

business units is responsible for 

defining and implementing the

organization’s IT investment 

governance process. 

Executed DOD has instituted an enterprisewide business system investment board—

the DBSMC—composed of senior executives, including the Deputy 

Secretary of Defense and the ASD(NII)/CIO. This board is responsible for 

establishing and implementing policies governing the organization’s 

investment process and approving lower-level investment board processes 

and procedures. 

Instituting 

the 

investment 

board 

2. The organization has a 

documented IT investment 

process directing each 

investment board’s operations. 

Not 

executed

DOD’s IRB Concept of Operations directs its IRBs and includes the roles 

and responsibilities of the boards and individuals involved. However, the 

concept of operations does not assign the boards accountability for 

programs throughout the investment life cycle (i.e., investments that are 

past the development/modernization stage and in operations and 

maintenance). In addition, according to our ITIM guidance, the 

department’s investment process should specify the manner in which 

investment-related processes will be coordinated with other organizational 

plans, processes, and documents. However, DOD’s concept of operations 

does not specify how the business investment management system is 

coordinated with JCIDS, PPBE, and DAS. 

Meeting 

business 

needs 

1. The organization has 

documented policies and 

procedures for identifying IT

projects or systems that support 

the organization’s ongoing and

future business needs. 

Executed DOD's Business Transformation Guidance and the Investment Certification

and Annual Review Process User Guidance define a process for ensuring

that IT business system investments support the department’s ongoing and

future business needs. 

Page 26 GAO-07-538 Business Systems Modernization
Critical 

process Key practice Rating Summary of evidence 

1. The organization has 

documented policies and 

procedures for selecting a new 

investment. 

Not 

executed

DOD has a two-stage selection process. The first stage involves selection 

of systems using the JCIDS, DAS, and PPBE management systems. At this 

level, proposals and alternatives are viewed and prioritized for system 

selection. The second stage of selection involves (1) certifying and 

approving Tiers 1 through 3 investments and (2) elevating certain 

component investments to an enterprisewide status using the business 

investment management system. 

While DOD’s IRB Concept of Operations and its Investment Certification 

and Annual Review Process User Guidance define the department’s 

corporate approach for certifying and approving investments, they do not 

contain a structured method defining how certification decisions are 

reached. For example, the guidance does not specify how cost, schedule, 

and benefit data are to be used in making certification decisions. According

to our ITIM guidance, a structured selection method should provide 

investment boards, business units, and IT developers with a common 

understanding of the selection process, including the cost, schedule, and 

benefit data used to compare and select projects. In addition, neither the 

IRB Concept of Operations nor the Investment Certification and Annual 

Review Process User Guidance define the selection criteria used to elevate 

these investments to an enterprisewide status. 

Furthermore, the BEA Compliance Guidance allows programs to postpone 

demonstrating full compliance with several BEA artifacts until the final 

phases of the acquisition process. In addition, criteria for certifying 

compliance with the BEA are inconsistently described in DOD 

documentation. For example, the BEA Compliance Guidance provides 

different checkpoints for assessing compliance during the life cycle of a 

program than the Business Transformation Guidance. 

2. The organization has 

documented policies and 

procedures for reselecting 

ongoing investments. 

Not 

executed

DOD’s IRB Concept of Operations and the Investment Certification and 

Annual Review Process User Guidance define the department’s corporate 

approach for annually reviewing investments. However, these documents 

do not include specific criteria that describe how the IRBs make reselection 

decisions. For example, while DOD officials stated that a program’s risk 

areas (i.e., cost, schedule, and performance) are identified and discussed 

by the IRB during the annual reviews, the guidance does not specify how 

this information is used in making annual review decisions. In addition, the 

guidance does not provide for the reselection of investments that are in 

operations and maintenance. Our ITIM guidance states that consistent 

qualitative and quantitative measures are needed for analyzing a project for 

reselection or, if necessary, termination. According to ITIM, the results of 

this analysis can help the investment board determine the potential risk and 

return of continuing to fund an ongoing project and to prioritize projects on 

the basis of decision criteria. 

Selecting 

an 

investment 

3. The organization has 

documented policies and 

procedures for integrating 

investment funding with 

investment selection. 

Not 

executed

According to DOD officials and the Investment Certification and Annual 

Review Process User Guidance, the IRBs are aware of the amount of 

funding components have requested for a program. However, this guidance 

does not specify how funding decisions are integrated with the process of 

selecting an investment, and does not specify how the DBSMC and the 

IRBs use this information in carrying out decisions on system certification 

and approvals. 

Page 27 GAO-07-538 Business Systems Modernization
Critical 

process Key practice Rating Summary of evidence 

Providing 

investment 

oversight 

1. The organization has 

documented policies and 

procedures for management 

oversight of IT projects and 

systems. 

Not 

executed

DOD’s IRB Concept of Operations and the Investment Certification and 

Annual Review Process User Guidance do not provide sufficient oversight 

and visibility into component-level investment management activities, 

including component reviews of systems in operations and maintenance 

and Tier 4 investments. For example, while the components submit a list of 

systems reviewed at their levels, the list lacks important project information, 

including adherence to cost, schedule, and risk criteria. According to ITIM, 

to maintain adequate oversight, the investment board should have visibility 

into each project’s performance and progress toward predefined cost and 

schedule expectations as well as each project’s anticipated benefits and 

risk exposure. In addition, IRB policies and procedures do not define how 

the department’s management systems, JCIDS, PPBE, and DAS, are 

related. 

1. The organization has 

documented policies and 

procedures for identifying and 

collecting information about IT

projects and systems to support 

the investment management 

process. 

Executed DOD’s Investment Certification and Annual Review Process User Guidance 

describes the procedures for submitting, updating, and maintaining 

information in DITPR and the IRB Portal, both of which support the 

business investment management system. 

Capturing 

investment 

information

2. An official is assigned 

responsibility for ensuring that the 

information collected during 

project and systems identification 

meets the needs of the 

investment management process. 

Executed DOD’s Investment Certification and Annual Review Process User Guidance 

assigns the component PCA the responsibility to ensure investment 

information contained in DITPR and the IRB Portal is accurate and 

complete. The guidance also assigns IRB staff responsibility for verifying 

these data. 

Source: GAO.

According to BTA officials, the IRB Concept of Operations and the 

Investment Certification and Annual Review Process User Guidance are 

not intended to describe the detailed approach that each IRB will use 

when making certification decisions, adding that the components are 

responsible for selection, annual review, budgeting, and acquisition. While 

the ITIM framework does allow for multiple entities to carry out 

investment selection, control, and evaluation, building a sound investment 

foundation requires that the enterprisewide investment review board has 

documented criteria and decision-making procedures, clear integration 

among investment decision-support systems, and policies to ensure board 

access to system information throughout the life cycle for all investments. 

Until DOD’s documented IT investment management policies and 

procedures include fully defined policies and procedures for Stage 2 

activities, specify the linkages between the various related processes, and

describe how investments are to be governed in the operations and 

maintenance phase, DOD risks that investment management activities will 

not be carried out consistently and in a disciplined manner. Moreover, 

Page 28 GAO-07-538 Business Systems Modernization
DOD also risks selecting investments that will not cost-effectively meet its 

mission needs. 

DOD Has Assigned 

Responsibility, but Has Not 

Defined the Policies and 

Procedures Associated 

with Effective PortfolioLevel Management 

At Stage 3, an organization has defined critical processes for managing its

investments as a portfolio or set of portfolios.36 Portfolio management is a 

conscious, continuous, and proactive approach to allocating limited 

resources among competing initiatives in light of the investments’ relative 

benefits. Taking an agencywide perspective enables an organization to 

consider its investments comprehensively, so that collectively the 

investments optimally address the organization’s missions, strategic goals, 

and objectives. Managing IT investments as portfolios also allows an 

organization to determine its priorities and make decisions about which 

projects to fund on the basis of analyses of the relative organizational 

value and risks of all projects, including projects that are proposed, under 

development, and in operation. Although investments may initially be 

organized into subordinate portfolios—on the basis of, for example, 

business lines or life-cycle stages—and managed by subordinate 

investment boards, they should ultimately be aggregated into enterpriselevel portfolios. 

According to ITIM, Stage 3 involves (1) defining the portfolio criteria; 

(2) creating the portfolio; (3) evaluating (i.e., overseeing) the portfolio; and 

(4) conducting postimplementation reviews. Table 5 summarizes the 

purpose of each of these activities. 

Table 5: Stage 3 Critical Processes—Developing a Complete Investment Portfolio

Critical process Purpose 

Defining the portfolio criteria To ensure that the organization develops and maintains portfolio selection criteria that 

support its mission, organizational strategies, and business priorities. 

Creating the portfolio To ensure that investments are analyzed according to the organization’s portfolio selection

criteria, and to ensure that an optimal investment portfolio with manageable risks and 

returns is selected and funded.

Evaluating the portfolio To review the performance of the organization’s investment portfolio(s) at agreed-upon 

intervals, and to adjust the allocation of resources among investments as necessary. 

Conducting postimplementation reviews To compare the results of recently implemented investments with the expectations that 

were set for them, and to develop a set of lessons learned from these reviews. 

Source: GAO.

 36Investment portfolios are integrated agencywide collections of investments that are

assessed and managed collectively on the basis of common criteria. 

Page 29 GAO-07-538 Business Systems Modernization
DOD is executing one of the five practices within these four critical 

processes that call for policies and procedures associated with effective 

portfolio-level management. Specifically, DOD has issued departmentwide

guidance37 that assigns responsibilities to the USD(AT&L) for managing 

and establishing business system investment portfolios, including 

leveraging or establishing a governance forum to oversee these business 

system investment portfolio activities. 

However, DOD has not fully defined the policies and procedures needed to

effectively execute the remaining four portfolio management practices 

relative to business system investments. Specifically, DOD does not have 

policies and procedures for defining the portfolio criteria or for creating 

and evaluating the portfolio. In addition, while DOD has policies and 

procedures for conducting postimplementation reviews as part of DAS, 

these reviews do not address systems at all tier levels. Furthermore, there 

are no procedures detailing how lessons learned from these reviews are 

used during investment review as the basis for management and process 

improvements. 

Table 6 summarizes the rating for each critical process required to manage

investment as a portfolio and summarizes the evidence that supports these 

ratings. 

 37DOD Directive 8115.01, Information Technology Portfolio Management, and DOD

Instruction 8115.02, Information Technology Portfolio Management Implementation.

Page 30 GAO-07-538 Business Systems Modernization
Table 6: Summary of Policies and Procedures for Stage 3 Critical Processes—Developing a Complete Investment Portfolio 

Critical process Key practice Rating Summary of evidence 

Defining the portfolio 

criteria 

1. The organization has documented 

policies and procedures for creating 

and modifying IT portfolio selection 

criteria. 

Not 

executed

DOD’s IT Portfolio Management Implementation states 

that the USD(AT&L) is responsible for creating and 

modifying portfolio criteria (e.g., prioritization and 

investment tradeoffs) for business system investments. 

However, the USD(AT&L) has not documented the 

related policies and procedures. 

2. Responsibility is assigned to an 

individual or group for managing the 

development and modification of the 

IT portfolio selection criteria. 

Executed DOD’s IT Portfolio Management assigns responsibility 

for the business mission area portfolio management to 

the USD(AT&L), who leads and manages business 

system investments in coordination with the 

ASD(NII)/CIO, the Under Secretary of Defense 

(Comptroller), and the Under Secretary of Defense 

(Personnel and Readiness). 

Creating the portfolio 1. The organization has documented 

policies and procedures for 

analyzing, selecting, and maintaining 

the investment portfolios. 

Not 

executed

DOD does not have policies and procedures for 

analyzing, selecting, and maintaining business system 

investment portfolios. 

Evaluating the portfolio 1. The organization has documented 

policies and procedures for 

reviewing, evaluating, and improving 

the performance of its portfolio(s). 

Not 

executed

While the IRB Concept of Operations states that the 

IRBs are responsible for reviewing factors associated

with portfolio management, such as architecture 

alignment and capability delivery, there are no policies 

and procedures indicating how the IRBs should use 

these factors and project indicators—such as cost, 

schedule, and risk—to review, evaluate, and improve 

their portfolios. According to our ITIM guidance for 

Stage 3, IRBs should use actual investment data, such 

as project cost and adherence to schedule, as the basis 

for reviewing and evaluating its portfolio(s) to ensure 

that the overall portfolio provides the maximum benefits 

at a desired cost and at an acceptable level of risk. 

Conducting

postimplementation 

reviews 

1. The organization has documented 

policies and procedures for 

conducting postimplementation

reviews. 

Not 

executed

While DOD requires postimplementation reviews for 

Tier 1 systems as part of DAS, there are no policies or 

procedures for conducting them for Tiers 2 or 3 

systems. Moreover, there are no policies or procedures 

directing the DBSMC or IRBs, or both, which are 

accountable for corporate business system 

investments, to consider information gathered and to 

develop lessons learned from these postimplementation 

reviews. According to ITIM, an effective 

postimplementation review includes, among other 

things, how conclusions, lessons learned, and 

recommended management action steps are to be 

disseminated to executives and others. 

Source: GAO.

According to BTA officials, while portfolio management is primarily a 

component responsibility, they are working toward developing more 

effective departmentwide portfolio management processes, but plans or 

Page 31 GAO-07-538 Business Systems Modernization
time frames for doing so have not been established. Without defining 

corporate policies and procedures for managing business system 

investment portfolios, DOD is at risk of not consistently selecting the mix 

of investments that best supports the departmentwide mission needs and 

ensuring that investment-related lessons learned are shared and applied 

departmentwide. 

Given the importance of business systems modernization to DOD’s 

mission, performance, and outcomes, it is vital for the department to adopt 

and employ an effective institutional approach to managing business 

system investments. While the department has established aspects of such 

an approach and, thus, has a foundation on which to build, it is lacking 

other important elements, such as specific policies and procedures needed 

for project-level and portfolio-level investment management, including

integration with DOD’s other key management systems and sufficient 

oversight and visibility into operations and maintenance investments and 

Tier 4 investments. This means that DOD lacks an institutional capability 

to ensure that it is investing in business systems that best support its 

strategic needs, and that ongoing projects meet cost, schedule, and 

performance expectations. Until DOD develops this capability, the 

department will be impaired in its ability to optimize business mission area 

performance and accountability. 

To strengthen DOD’s business system investment management capability 

and address the weaknesses discussed in this report, we recommend that 

the Secretary of Defense direct the Deputy Secretary of Defense, as the 

chair of the DBSMC, to ensure that well-defined and disciplined business 

system investment management policies and procedures are developed 

and issued. At a minimum, this should include project-level management 

policies and procedures that address the following five areas:

Conclusions 

Recommendations for 

Executive Action 

• instituting the investment boards, including assigning the investment 

boards responsibility, authority, and accountability for programs 

throughout the investment life cycle and specifying how the business 

investment management system is coordinated with JCIDS, PPBE, and 

DAS; 

Page 32 GAO-07-538 Business Systems Modernization
• selecting new investments, including specifying how cost, schedule, and 

benefit data are to be used in making certification decisions; defining the 

criteria used to select investments as enterprisewide; and establishing 

consistent and effective guidance for BEA compliance; 

• reselecting ongoing investments, including specifying how cost, schedule, 

and performance data are to be used in the annual review process and 

providing for the reselection of investments that are in operations and 

maintenance; 

• integrating funding with the process of selecting an investment, including 

specifying how the DBSMC and the IRBs use funding information in 

carrying out decisions on system certification and approvals; and 

• overseeing IT projects and systems, including providing sufficient 

oversight and visibility into component-level investment management 

activities. 

These well-defined and disciplined business system investment 

management policies and procedures should also include portfolio-level 

management policies and procedures that address the following four 

areas: 

• creating and modifying IT portfolio selection criteria for business system 

investments; 

• analyzing, selecting, and maintaining business system investment 

portfolios; 

• reviewing, evaluating, and improving the performance of its portfolio(s) by

using project indicators, such as cost, schedule, and risk; and

• conducting postimplementation reviews for all investment tiers and 

directing the investment boards, which are accountable for corporate 

business system investments, to consider the information gathered and to

develop lessons learned from these reviews. 

In written comments on a draft of this report, signed by the Deputy Under

Secretary of Defense (Business Transformation) and reprinted in appendix

II, the department stated that it agreed with the report’s overall 

conclusions, and it described efforts under way and planned that it said 

would address many of the gaps identified in the report. In this regard, the

department partially concurred with five of the report’s recommendations, 

Agency Comments 

and Our Evaluation 

Page 33 GAO-07-538 Business Systems Modernization
adding that our recommendations and feedback are helpful in guiding 

DOD’s business transformation and related improvement efforts. 

Nevertheless, the department disagreed with the remaining four 

recommendations on the grounds that their intent had already been met 

through DOD’s existing business system investment management 

structure and processes, or that they contradicted the tiered accountability 

concept embedded in this structure and processes. The department’s 

comments relative to each of our project-level and portfolio-level 

recommendations, along with our responses to its comments, are provided

below. 

With respect to our five project-level recommendations, the department 

stated that it partially agreed with two and disagreed with three. 

• DOD partially agreed with our recommendation to define and implement 

policies and procedures that assign the investment boards responsibility 

for programs throughout the investment life cycle and specify how the 

business investment management system is coordinated with JCIDS, 

PPBE, and DAS. In particular, it stated that under its tiered accountability 

approach to business systems investment management, the components 

are currently required to review all programs throughout their investment 

life cycles. We do not question this requirement, and we recognize it in our 

report. However, consistent with our ITIM framework, the corporate 

investment boards should continue to review investments that meet the 

defined threshold criteria throughout their life cycles (i.e., when they are 

in operations and maintenance). In contrast, DOD’s corporate boards 

focus only on those investments that are in the

development/modernization stage. The department also stated that a 

linkage is currently depicted in existing guidance among its investment 

selection, acquisition, and funding processes. While we do not question 

that this guidance contains an illustration depicting such a link, neither 

this guidance nor supporting procedures define how this linkage is 

executed (e.g., how investment funding decisions are in fact integrated 

with investment selection decisions). DOD’s comments appear to 

acknowledge this point by stating that the department has begun to define

and implement a Business Capability Lifecycle concept, which is intended 

to integrate the investment selection and acquisition management 

processes for Tier 1 and enterprise systems into a single oversight process 

that leverages the existing IRB and DBSMC oversight framework. 

• DOD partially agreed with our recommendation to define and implement 

policies and procedures that specify how cost, schedule, and benefit data

are to be used in making certification and annual review decisions; define 

the criteria used to select investments as enterprisewide; and establish 

Page 34 GAO-07-538 Business Systems Modernization
consistent and effective guidance for BEA compliance. In particular, the 

department agreed that additional criteria are required for selecting 

enterprisewide investments, noting that initial criteria have been defined 

and will be incorporated in the investment management process. However, 

the department did not agree that cost, schedule, and BEA compliance 

information are not sufficiently used for certification and annual review 

decisions, adding that such information is required in its current policies. 

We do not agree. Specifically, while we do not question whether 

investment data are provided to the DBSMC and the IRBs, the 

department’s policies and procedures do not include specific decision 

criteria that explain how these data are to be used to make consistent, 

repeatable selection and reselection decisions across all investments. In 

addition, while BEA compliance policies have been developed and are 

being used, the guidance is not fully defined. For example, the guidance

allows programs to defer demonstrating full compliance with important 

BEA artifacts until the final phases of the acquisition process, at which 

time addressing instances of noncompliance would be more expensive and 

difficult. Furthermore, the compliance criteria are not consistently 

described in different guidance documentation. As a result, DOD risks 

beginning system production and deployment before ensuring that a 

system is sufficiently aligned to the BEA. 

• DOD did not agree with our recommendation to define and implement 

policies and procedures that provide for the reselection of investments 

that are in operations and maintenance. According to DOD, components 

are required by policy to annually review all business systems, including 

investments for which there is no planned development or modernization 

spending. We agree that the annual review process does require this. 

However, consistent with our ITIM framework, the corporate investment

boards should continue to reselect investments that meet the defined 

threshold criteria throughout their life cycles (i.e., when they are in 

operations and maintenance). In contrast, DOD’s corporate boards focus 

only on reselecting those investments that are in the 

development/modernization stage. 

• DOD did not agree with our recommendation to define and implement 

policies and procedures that specify how the corporate boards use funding 

information in carrying out decisions on system certification and 

approvals. In this regard, it stated that such information is required in its 

current policies and considered during board deliberations. We do not 

agree. Our recommendation does not address whether existing policies or

guidance provide for the collection of this information; our 

recommendation addresses the definition of policy, guidance, and 

supporting procedures that fall short of satisfying the best practices 

Page 35 GAO-07-538 Business Systems Modernization
embodied in our ITIM framework. Specifically, while we do not question 

whether funding data are provided to investment decision-making bodies, 

the department’s policies and procedures do not include specific decision 

criteria that explain how these data are to be used to make consistent, 

repeatable selection and reselection decisions across all investments. 

• DOD did not agree with our recommendation to define and implement 

policies and procedures that provide for sufficient oversight and visibility 

into component-level investment management activities. In particular, it 

stated that this recommendation contradicts the department’s “tiered 

accountability” approach to investment management. We do not agree. 

Under the department’s current policies and guidance, most DOD 

investments are not subject to corporate visibility and oversight, either 

because they do not involve development/modernization (i.e., they are in 

operations and maintenance) or because they do not exceed a certain 

dollar threshold. Our framework recognizes that effective implementation 

of a tiered accountability concept should include appropriate corporate 

visibility into and oversight of investments, either through review and 

approval of those investments that meet certain criteria or through 

awareness of a subordinate board’s investment management activities. 

Moreover, this visibility and oversight should extend to the entire portfolio 

of investments, including those that are in operations and maintenance. To

ensure that this occurs, applicable policies and procedures need to 

explicitly cover all such investments and need to define how this is to be 

accomplished. 

With respect to our four portfolio-level recommendations, the department

stated that it partially agreed with three and disagreed with one. 

• DOD partially agreed with our recommendation to define and implement 

policies and procedures for creating and modifying portfolio selection 

criteria for business system investments. In particular, it stated that while 

components are responsible for developing and managing their own 

portfolio management processes, upcoming initiatives, such as the 

Business Capability Lifecycle concept, will lead to revisions in the 

department’s investment review policies and procedures, such as 

including portfolio selection criteria for enterprise systems that span 

components. However, while these are important steps, the concept, as 

defined by the department, does not apply to the thousands of investments 

that are not enterprisewide. 

• DOD partially agreed with our recommendation to define and implement 

policies and procedures that address analyzing, selecting, and maintaining 

business system investment portfolios. In particular, it stated that the 

Page 36 GAO-07-538 Business Systems Modernization
implementation of the Business Capability Lifecyle concept will provide 

the corporate boards with improved visibility into all investments in a 

given portfolio and a broader set of criteria for analyzing, selecting, and 

maintaining business system investment portfolios. 

• DOD partially agreed with our recommendation to define and implement 

policies and procedures that address reviewing, evaluating, and improving 

the performance of its portfolio(s) by using cost, schedule, and risk 

indicators. In particular, it stated that while such indicators are part of the 

investment certification and review processes, efforts are now under way 

to better understand the nature and impact of program risks through 

application of an Enterprise Risk Assessment Methodology. While we 

recognize the role and value of such tools in understanding and addressing 

program risks, this tool is program-specific and not portfolio-focused. 

• DOD did not agree with our recommendation to define and implement 

policies and procedures that address conducting postimplementation 

reviews and having the corporate investment boards consider the review 

results and develop lessons learned from them. In particular, it stated that

this process should not be managed by the Deputy Secretary of Defense, 

and also stated that our recommendation is redundant with 

postimplementation reviews currently required under OMB Circular A130.38 We do not agree with DOD’s statements. Our recommendation does 

not call for the Deputy Secretary to manage the postimplementation 

review process. Rather, it provides for developing policies and procedures

for performing postimplementation reviews for all tiers of business 

systems and having the DBSMC and IRBs, which are the corporate 

investment boards, consider the information gathered from these reviews 

and develop lessons learned. 

We are sending copies of this report to interested congressional 

committees; the Director, Office of Management and Budget; the Secretary

of Defense; the Deputy Secretary of Defense; the Under Secretary of 

Defense for Acquisition, Technology, and Logistics; the Under Secretary of 

Defense (Comptroller); the Assistant Secretary of Defense (Networks and 

Information Integration)/Chief Information Officer; the Under Secretary of 

 38According to OMB Circular A-130, which establishes policy for the management of federal

information resources, as part of the capital planning process, an agency must, among

other things, conduct postimplementation reviews of information systems and information

resource management processes to validate estimated benefits and costs; document 

effective management practices for broader use; and document lessons learned from the

postimplementation reviews. 

Page 37 GAO-07-538 Business Systems Modernization
Defense (Personnel and Readiness); and the Director, Defense Finance 

and Accounting Service. Copies of this report will be made available to 

other interested parties upon request. This report will also be available at 

no charge on our Web site at http://www.gao.gov. 

If you or your staffs have any questions on matters discussed in this 

report, please contact me at (202) 512-3439 or hiter@gao.gov. Contact 

points for our Offices of Congressional Relations and Public Affairs may 

be found on the last page of this report. GAO staff who made major 

contributions to this report are listed in appendix III. 

Randolph C. Hite 

Randolph C. Hite 

Director, Information Technology Architecture 

 and Systems Issues 

Page 38 GAO-07-538 Business Systems Modernization
List of Committees 

The Honorable Carl Levin 

Chairman 

The Honorable John McCain 

Ranking Member 

Committee on Armed Services 

United States Senate 

The Honorable Daniel Inouye 

Chairman 

The Honorable Ted Stevens 

Ranking Member 

Committee on Appropriations 

United States Senate 

The Honorable Ike Skelton 

Chairman 

The Honorable Duncan Hunter 

Ranking Member 

Committee on Armed Services 

House of Representatives 

The Honorable John P. Murtha 

Chairman 

The Honorable C.W. Bill Young 

Ranking Member 

Committee on Appropriations 

House of Representatives 

Page 39 GAO-07-538 Business Systems Modernization
Appendix I: Objective, Scope, and Appendix I: Objective, Scope, and Methodology 

Methodology 

Our objective was to determine whether the Department of Defense’s 

(DOD) corporate investment management approach comports with 

relevant federal guidance. Our analysis was based on the best practices 

contained in GAO’s Information Technology Investment Management 

(ITIM) framework, and the framework’s associated evaluation

methodology, and focused on DOD’s establishment of departmental-level

policies and procedures for business system investments needed to assist 

organizations in complying with the investment management provisions of 

the Clinger-Cohen Act of 1996 (Stages 2 and 3). It did not include case 

studies to verify the implementation of established policies and 

procedures. 

To address our objective, we asked DOD to complete a self-assessment of 

its corporate investment management process and provide the supporting 

documentation. We then reviewed the results of the department’s selfassessment of Stages 2 and 3 organizational commitment practices—

meaning those practices related to structures, policies, and procedures—

and compared them against our ITIM framework. We also validated and

updated the results of the self-assessment through document reviews and 

interviews with officials, such as the Director of Investment Management 

and the Defense Business Systems Acquisition Executive. In doing so, we 

reviewed written policies, procedures, and guidance and other 

documentation providing evidence of executed practices, including the 

Defense Acquisition System guidance, the Investment Review Board (IRB) 

Concept of Operations and Guidance, the Business Enterprise 

Architecture Compliance Guidance, IRB charters and meeting minutes, 

and the Business Transformation Guidance. 

We compared the evidence collected from our document reviews and 

interviews with the key practices in ITIM. We rated the key practices as 

“executed” on the basis of whether the agency demonstrated (by providing 

evidence of performance) that it had met all of the criteria of the key

practice. A key practice was rated as “not executed” when we found 

insufficient evidence of all elements of a practice being fully performed or 

when we determined that there were significant weaknesses in DOD’s 

execution of the key practice. In addition, we provided DOD with the 

opportunity to produce evidence for the key practices rated as “not 

executed.” 

We conducted our work at DOD headquarters offices in Arlington, 

Virginia, from August 2006 through April 2007 in accordance with 

generally accepted government auditing standards. 

Page 40 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department Appendix II: Comments from the Department of Defense 

of Defense 

Page 41 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 42 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 43 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 44 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 45 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 46 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 47 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 48 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 49 GAO-07-538 Business Systems Modernization
Appendix II: Comments from the Department 

of Defense 

Page 50 GAO-07-538 Business Systems Modernization
Appendix III: GAO

A

 Contact and Staff 

cknowledgments 

Page 51 GAO-07-538

Appendix III: GAO Contact and Staff 

Acknowledgments 

GAO Contact Randolph C. Hite, (202) 512-3439 or hiter@gao.gov 

In addition to the contact person named above, key contributors to this 

report were Neil Doherty, Nalani Fraser, Nancy Glover, Michael Holland, 

Neelaxi Lakhmani (Assistant Director), Jacqueline Mai, Sabine Paul, Niti

Tandon, and Jennifer Stavros-Turner. 

Business Systems Modernization

Staff 

Acknowledgments 

(310636) 
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Mail or Phone 

To Report Fraud,

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER