Information Technology: Centers for Medicare &amp; Medicaid Services Needs to Establish Critical Investment Management Capabilities
Released: 2005-11-28T00:00:00

Letter

Background

CMS has become the largest purchaser of health care in the United States, serving nearly 83 million Medicare and Medicaid beneficiaries. The agency administers the Medicare program, enacted in 1965, which provides health insurance to people who are aged 65 years and over and to some people with disabilities who are under aged 65 years. The agency also works with the states to administer the Medicaid program, enacted in 1965 as a jointly funded program in which the federal government matches state spending according to a formula to provide medical and health-related services to low-income Americans.

In fiscal year 2005, CMS will reportedly spend about $519 billion: 63 percent for Medicare, 35 percent for Medicaid and Medicaid administration, and the remaining 2 percent for the State Children’s Health Insurance Program and other administrative costs. CMS estimates that its total budget in fiscal year 2006 will be $622 billion.

The agency carries out its responsibilities from its national headquarters located in Baltimore, Maryland, and its 10 regional offices located throughout the nation. It is organized around three centers (to support its key functions): the Center for Medicare Management, the Center for Beneficiary Choices, and the Center for Medicaid and State Operations. Numerous other offices throughout the agency support these centers.

CMS’s Use of Information Technology

IT systems play a vital role in helping CMS to fulfill its responsibilities in carrying out the Medicare and Medicaid programs. These systems help to maintain Medicare information on the millions of beneficiaries, providers, and medical services provided. For example, CMS’s Medicare Fee-for- Service claims processing systems process more than 1 billion claims annually and make benefit payments for the 41 million elderly and disabled beneficiaries. In fiscal year 2004, the Medicare program had estimated outlays of $297 billion in health care benefits.

Similarly, IT systems are relied on to manage the Medicaid program. In fiscal year 2003, this program provided benefits totaling about $261 billion to nearly 54 million people. Of this amount, the federal share was about $153 billion.

To assist the states in developing and operating MMISs used to process Medicaid claims and administer the program, CMS provides funding assistance through grants. In fiscal year 2005, about $1.79 billion, or 70 percent, of CMS’s nearly $2.55 billion total appropriations for IT went to support Medicaid state investments. The remaining approximately $0.76 billion, or 30 percent, was used for CMS’s internal investments. Figure 1 shows the breakdown of this funding between CMS’s internal IT investments and Medicaid state IT investments.

Weaknesses Previously Identified in CMS’s IT Investment Management Processes

In September 2001, we reported that CMS’s processes for managing its IT investments omitted key review, approval, and evaluation steps. We recognized that the agency was making efforts to strengthen its IT planning and had developed guidance for an improved management process, but stated that it would need to make considerable progress in implementing these changes to ensure that its ongoing modernization efforts stayed on track. To improve its investment management processes, we made several recommendations to the CMS Administrator, including establishing sufficient and written criteria to ensure a consistent process for funding IT projects agencywide, and establishing a systematic process for evaluating completed IT projects that included cost, milestone, and performance data.

CMS’s Approach to Investment Management

Several groups and individuals play a role in CMS’s process to manage its internal IT investments, including an investment board for establishing the IT investment governance principles. However, a different process is used to oversee the Medicaid IT systems that the agency jointly funds with the states. This process is carried out by CMS’s Center for Medicaid and State Operations and 10 regional offices. Both of these processes, along with the roles and responsibilities of the groups and individuals involved, are described below.

Process for Managing Internal Investments

The groups and individuals who play a role in CMS’s internal IT investment management process include the Information Technology Investment Review Board, Executive Steering Committees, Enterprise Architecture Group, and Component Leads.

Information Technology Investment Review Board (ITIRB). This board was established in January of 2004 to provide a corporate perspective in evaluating IT investments against CMS’s business priorities. Its members consist of senior leadership from CMS centers, offices, and regional offices, and it is chaired by the agency’s Chief Information Officer (CIO). Initially, the primary ITIRB responsibility was overseeing investments associated with the Medicare Presciption Drug, Improvement, and Modernization Act of 2003 (MMA) and with CMS’s revitalization initiative. These investments made up about one-third of CMS’s fiscal year 2005 Operating Plan for internal systems. In the spring of 2005, the role of the board was expanded to include all internal IT investments. To assist the ITIRB in its activities, CMS staff from the Office of Information Services and the Office of Financial Management provide administrative support. According to its charter, the board is responsible for establishing the criteria for the selection, control, and evaluation of CMS’s portfolio of IT projects; developing the agency’s IT operation plan and responding to the President’s budget request; reviewing the performance of IT investments using the criteria and checkpoints in meeting cost, schedule, risk, and benefit expectations and taking corrective actions when expectations are not being met; ensuring that IT investments in operation are periodically evaluated to determine whether they should be retained, modified, replaced, or terminated; and comparing the results of implemented investments with the expectations that were set for them and developing a set of lessons learned for future process improvement.

Executive Steering Committees (ESC). The ESCs were established to support the ITIRB in carrying out its responsibilities. Each ESC is responsible for managing IT projects (or investments) that are grouped together into a portfolio for each of CMS’s business components. This responsibility includes maintaining the appropriate mix of IT investments in its portfolio, managing the investments in its portfolio, and providing funding recommendations to the ITIRB for these investments. The membership of each ESC depends on the IT investments contained in the portfolio, but, at a minimum, every CMS component that sponsors a project is to have a representative on the ESC.

Enterprise Architecture Group. This group, formally known as the IT Architecture Planning Staff, supports the IT investment management process, by, among other things, reviewing business case analyses for new investments and major enhancements to ensure that they are consistent with the enterprise architecture, by making recommendations based upon that review that are aimed at the optimal leveraging of assets.

Component Leads. These individuals provide support in the IT investment management process by serving as liaisons between the Office of Information Services and individual project managers. Component Leads are to assist project managers in understanding CMS’s investment management process and other operational policies and processes. They can also provide project managers with key contacts for various IT services that project owners may require during implementation of a project.

In the spring of 2005, CMS implemented a new budget formulation process and used it to select its IT investments. This process begins with an information request from the CIO asking that each component submit information on all of its investments, both new and ongoing. This information is to include (1) a score sheet for each investment that shows how it compared with prescribed criteria, such as alignment with business drivers and IT strategic goals, and (2) a prioritized list of all investments for the component. For new investments, the components also are to submit an IT Fact Sheet (an investment proposal) that the ITIRB support staff; the Enterprise Architecture Group; and, ultimately, the board review to determine if the need for the new investment is justified. If the need is found to be justified, project managers receive funding to develop a Business Case Analysis (smaller projects may not require such a document), which goes through the same review process as the IT Fact Sheet.

The ITIRB support staff review all information submitted in response to the information request and prepare it for the ESCs’ review. The ESCs reevaluate the investments against the criteria, making adjustments to the scoring if necessary, and make funding recommendations to the board. The ITIRB makes strategic and funding recommendations regarding CMS’s IT capital investment portfolio to CMS’s Chief Operating Officer who, in turn, provides recommendations to the CMS’s Office of Financial Management for integration into the agency’s overall budget. Figure 2 illustrates CMS’s process for selecting its internal IT investments.

To date, the ITIRB’s role in controlling (overseeing) IT investments has been primarily limited to those associated with the MMA and revitalization initiatives. According to CMS officials, efforts to define procedures for the board to control all internal investments, in accordance with the responsibilities described in its charter, are currently under way.

Process for Approving and Monitoring State Medicaid IT Investments

The ITIRB plays no role in approving and monitoring state Medicaid IT investments. Instead, the process for approving states’ requests for matching funds for MMIS activities—including the design, development, and installation of new MMISs, and the operations, maintenance, and enhancement of existing MMISs—is the shared responsibility of CMS’s Center for Medicaid and State Operations (hereafter referred to as the central office) and its 10 regional offices. According to regulations, the State Medicaid Manual, and officials we interviewed at CMS’s central office and 5 regional offices, CMS’s process for approving states’ requests generally consists of the activities discussed below: To request federal funds for state MMIS activities, states must prepare an advance planning document (APD), which identifies, among other things, the purpose, scope, benefits, and preliminary cost estimates for the activities they want to undertake. States submit this document to the regional office, which reviews the APD for completeness and technical content. Regional office staff generally ensure that requests support the Medicaid program, are in compliance with federal requirements, and represent cost-effective solutions. Also, the regional office may have suggestions for the states to improve their APDs. Some of the officials we interviewed told us that they work with the states to complete the APDs to expedite the review and approval process.

Once regional office staff determine that an APD adequately justifies the request for funding and the request is approved by that regional office’s Associate Regional Administrator for Medicaid, the CMS central office and HHS are notified of the approval through a process referred to as the Office of the Secretary Notice process. Once the central office concurs, the regional office can send an approval letter to the state.

The states typically hire contractors to perform the MMIS activities.

With the approval of an APD, a state is given the clearance to develop the request-for-proposals for soliciting contractor proposals. While the APD is a high-level justification for funding, the request-for-proposals is to contain the more detailed requirements of the MMIS activities. Before it is issued, the request-for-proposals must be approved by the CMS regional office through a process similar to that used for the APD.

The states review the proposals received and evaluate them in order to make the final selection. While regional office staff do not formally approve a state’s evaluation process, they do review the process to ensure that it allows for open and free competition, to the maximum extent practicable.

The states draft a contract for the MMIS activities. Prior to its award, the contract is reviewed by regional office staff and approved by the Associate Regional Administrator for Medicaid. The state then makes an award to the contractor whose bid or offer is responsive to the solicitation and most advantageous to the state—considering price, quality, and other factors.

When the contracted MMIS activities start, regional office staff begin monitoring the status of these activities through a variety of mechanisms, including reviews of status reports; on-site visits; and meetings with external groups, such as industry associations, provider groups, and vendors.

Once MMISs are built and become operational, CMS establishes a team consisting of headquarters and regional office staff with expertise in relevant areas to do on-site reviews, referred to as certification reviews. During these reviews, which are to be conducted about 6 months after a system has been in operation, the team makes sure that the system satisfies the terms of the state’s APD, meets minimal federal requirements, and complies with current regulations and policy. CMS has written guidance for conducting these reviews, which it is in the process of revising.

Regional office staff are to continue monitoring MMIS activities through the previously mentioned mechanisms.

Information Technology Investment Management Maturity Framework

The Information Technology Investment Management (ITIM) framework is a maturity model comprising five progressive stages of maturity that an agency can achieve in its investment management capabilities. The ITIM framework was developed on the basis of our research into the IT investment management practices of leading private- and public-sector organizations. It identifies critical processes for making successful IT investments, organized into the five increasingly mature stages. These maturity stages are cumulative; that is, in order to attain a higher stage of maturity, the agency must have institutionalized all of the requirements for all of the lower stages in addition to the higher stage.

The ITIM framework can be used to assess the maturity of an agency’s investment management processes and as a tool for organizational improvement. The overriding purpose of the framework is to encourage investment processes that increase business value and mission performance, reduce risk, and increase accountability and transparency in the decision process. We have used the framework in several of our evaluations, and a number of agencies have adopted it. These agencies have used ITIM for purposes ranging from self-assessment to the redesign of their IT investment management processes.

The ITIM framework’s five maturity stages represent steps toward achieving stable and mature processes for managing IT investments. Each stage builds on the lower stages; the successful attainment of each stage leads to improvement in the organization’s ability to manage its investments. With the exception of the first stage, each maturity stage is composed of “critical processes” that must be implemented and institutionalized in order for the organization to achieve that stage. These critical processes are further broken down into key practices that describe the types of activities that an organization should be performing to successfully implement each critical process. An organization may be performing key practices from more than one maturity stage at the same time. This is not unusual, but efforts to improve investment management capabilities should focus on becoming compliant with lower-stage practices before addressing higher-stage practices.

Stage 2 of the ITIM framework encompasses building a sound investment management process by establishing basic capabilities for selecting new IT projects. It also involves developing the capability to control projects so that they finish predictably within established cost and schedule expectations and the capability to identify potential exposures to risk and put in place strategies to mitigate that risk. The basic selection processes established in Stage 2 lays the foundation for more mature selection capabilities in Stage 3.

Stage 3 requires that an organization continually assess both proposed and ongoing projects as parts of a complete investment portfolio—an integrated and competing set of investment options. It focuses on establishing a consistent, well-defined perspective on the IT investment portfolio and maintaining mature, integrated selection (and reselection), control, and evaluation processes that can be evaluated during postimplementation reviews. This portfolio perspective allows decision makers to consider the interaction among investments and the contributions to organizational mission goals and strategies that could be made by alternative portfolio selections, rather than focusing exclusively on the balance between the costs and benefits of individual investments. Organizations implementing Stages 2 and 3 have in place the selection, control, and evaluation processes that are required by the Clinger-Cohen Act of 1996.

Stages 4 and 5 require the use of evaluation techniques to continuously improve both the investment portfolio and the investment processes in order to better achieve strategic outcomes. At Stage 4 maturity, an organization has the capacity to conduct IT succession activities and, therefore, can plan and implement the deselection of obsolete, high-risk, or low-value IT investments. An organization with Stage 5 maturity conducts proactive monitoring for breakthrough information technologies that will enable it to change and improve its business performance. Stages 4 and 5 define key attributes that are associated with the most capable organizations.

Figure 3 shows the five ITIM stages of maturity and the critical processes associated with each stage.

As defined by the model, each critical process consists of “key practices” that must be executed to implement the critical process.

CMS’s Capabilities to Manage Its Internal Investments Are Limited

In order to have the capabilities to effectively manage IT investments, an agency, at a minimum, should (1) build an investment foundation by putting basic, project-level control and selection practices in place (Stage 2 capabilities) and (2) manage its projects as a portfolio of investments, treating them as an integrated package of competing investment options and pursuing those that best meet the strategic goals, objectives, and mission of the agency (Stage 3 capabilities).

CMS has executed 20 of the 38 key practices that are required to build a foundation for IT investment management. In addition, because CMS has focused primarily on establishing the Stage 2 practices, it has executed only 2 of the 27 Stage 3 key practices. Until CMS implements all of the key practices associated with building the investment foundation and managing its investments as a portfolio, the agency will not have much assurance that it has selected the mix of investments that best supports its strategic goals, or that it will be able to manage the investments to successful completion.

CMS Has Established about Half of the Foundational Practices for Investment Management

At the ITIM Stage 2 level of maturity, an organization has attained repeatable, successful IT project-level investment control processes and basic selection processes. Through these processes, the organization can identify expectation gaps early and take the appropriate steps to address them. According to the ITIM framework, critical processes at Stage 2 include (1) defining IT investment board operations, (2) identifying the business needs for each IT investment, (3) developing a basic process for selecting new IT proposals and reselecting ongoing investments, (4) developing project-level investment control processes, and (5) collecting information about existing investments to inform investment management decisions. Table 1 describes the purpose of each of these Stage 2 critical processes.

Because IT investment management has only recently become an area of management attention, CMS has put in place 20 of the 38 Stage 2 key practices required for basic project-level selection and control. The agency has satisfied the majority of the key practices associated with establishing an IT investment review board, capturing investment information, and meeting business needs. CMS also has recently established a process for selecting investments, but it has not yet established a process for the IT investment review board to provide investment oversight. Figure 4 summarizes the status of CMS’s critical processes for Stage 2, showing how many key practices CMS has executed in managing its internal IT investments.

The creation of decision-making bodies or boards is central to the IT investment management process. At the Stage 2 level of maturity, organizations define one or more boards, provide resources to support their operations, and appoint members who have expertise in both operational and technical aspects of the proposed investments. The boards operate according to a written IT investment process guide that is tailored to the organization’s unique characteristics, thus ensuring that consistent and effective management practices are implemented across the organization. Once board members are selected, the organization ensures that they are knowledgeable about policies and procedures for managing investments. Organizations at the Stage 2 level of maturity also take steps to ensure that executives and line managers support and carry out the decisions of the investment board. According to the ITIM framework, an IT investment management process guide should (1) be a key authoritative document that the organization uses to initiate and manage IT investment processes and (2) provide a comprehensive foundation for policies and procedures developed for all other related processes. (The complete list of key practices is provided in table 2.)

CMS has executed 5 of the 8 key practices for this critical process. For example, in January 2004, the agency established the ITIRB to manage internal investments and provide business-driven leadership to its operations and development. While the ITIRB was initially only responsible for overseeing MMA and revitalization initiatives, its responsibilities were expanded this past spring to include management and oversight responsibilities for all internal investments. ITIRB members are senior- level officials from both business and IT areas who understand board policies and procedures.

The ITIRB is adequately resourced to maintain its operations. For example, the Program Management and Support Group within the Office of Information Services assists the board in such ways as coordinating and integrating the investment management process. This group serves as the principal contact and entry point for all new and proposed IT projects. In addition, nine Executive Steering Committees were recently established to support the work of the ITIRB by managing a subset of investments grouped together according to business function. Their responsibilities include, among other things, scoring and ranking IT investments, and recommending investments to the ITIRB for funding.

Notwithstanding these strengths, CMS does not have an IT investment process guide that reflects the agency’s current investment management practices. For example, the agency uses ESCs to work with the board on specific areas of IT investments, but its process guide does not identify this critical group. Moreover, the process guide does not mention the agency’s move to classify its IT investments in line with the department’s classification scheme. (The new classification scheme consists of three levels in which projects are rated as major, supporting, or tactical.) Instead, the process guide outlines a four-level classification scheme that identifies investments as A, B, C, or D, depending on the nature and sensitivity of the project. According to CMS officials, the guide has not yet been updated because the agency has made a priority of fully defining its processes before documenting them. Documenting the process, however, does not preclude it from future revisions or improvements, but does provide a basis for consistent implementation across the agency. Until CMS’s documented IT investment process guidance is updated, executives are at risk of inconsistently performing key investment decision-making activities and inaccurately communicating management practices. Such updated guidance would also provide a process that could lead to greater accountability about future IT investment outcomes, which would be helpful to new members joining the board.

Another key weakness is that CMS’s ITIRB has not operated in accordance with its assigned roles and responsibilities. For example, the ITIRB has not yet been involved in systematically controlling investments nor has it actively maintained the documented investment management process. Until the ITIRB fully carries out its assigned roles and responsibilities, executives will not have assurance that the whole IT investment management process is functioning smoothly and effectively as intended.

Table 2 shows the rating for each key practice required to implement the critical process for instituting the investment board at the Stage 2 level of maturity. Each of the “executed” ratings shown below represents instances where, on the basis of the evidence provided by CMS officials, we concluded that the specific key practices were executed by the agency.

Defining business needs for each IT project helps to ensure that projects and systems support the organization’s business needs and meet users’ needs. This critical process ensures that a link exists between the organization’s business objectives and its IT management strategy. According to the ITIM, effectively meeting business needs requires, among other things, (1) documenting business needs with stated goals and objectives, (2) identifying specific users and other beneficiaries of IT projects and systems, (3) providing adequate resources to ensure that projects and systems support the organization’s business needs and meet users’ needs, and (4) periodically evaluating the alignment of IT projects and systems with the organization’s strategic goals and objectives. (The complete list of key practices is provided in table 3.)

CMS has in place 5 of the 7 key practices for meeting business needs. The agency’s IT Investment Management Process Guide and Business Case Analysis Development Guide require business needs for both proposed and ongoing IT projects and systems to be identified in an IT fact sheet and, in some instances, a business case analysis document. The agency also has detailed procedures for developing these documents that call for identifying users. We verified that the four projects we reviewed identified specific users and also documented how the projects linked back to CMS business needs. Resources for ensuring that IT projects and systems support the organization’s business needs and meet users’ needs include Component Leads, the Enterprise Architecture Group, and detailed procedures and associated templates for developing the IT fact sheet and business case analysis document.

Although CMS has performed most of the key practices associated with meeting business needs, a few weaknesses remain. Specifically, officials told us they rely on the HHS strategic plan to guide their efforts because CMS’s strategic plan documenting the agency’s business mission, goals, and objectives is outdated. However, the primary tool used to justify funding for investments does not tie into the HHS plan but provides high-level business drivers for aligning these investments with business needs. While, according to agency officials, these business drivers reflect a common understanding of the agency’s goals and objectives, they are not descriptive enough to drive IT investments. Until CMS develops a current strategic plan or other detailed statement of business mission with supporting goals and objectives, the agency is at risk of not being able to thoroughly communicate critical information on its goals and objectives or to provide clear and transparent direction for its IT investment management process.

Finally, CMS’s budget formulation process serves as a mechanism to reevaluate the alignment of projects and systems with the organization’s goals and objectives. However, the ITIRB selected investments for the first time this past spring and, therefore, has not yet had to reevaluate projects’ and systems’ alignment with organizational goals and objectives. When CMS executes all key practices associated with this critical process, it will have greater assurance that its projects effectively meet the agency’s business needs.

Table 3 shows the rating for each key practice required to implement the critical process for meeting business needs at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

Selecting new IT proposals and reselecting ongoing investments require a well-defined and disciplined process to provide the agency’s investment board, business units, and developers with a common understanding of the process and the cost, benefit, schedule, and risk criteria that will be used both to select new projects and to reselect ongoing projects for continued funding. According to the ITIM, this critical process requires, among other things, (1) making funding decisions for new proposals according to an established process; (2) providing adequate resources for investment selection activities; (3) using a defined selection process to select new investments and reselect ongoing investments; (4) establishing criteria for analyzing, prioritizing, and selecting new IT investments and for reselecting ongoing investments; and (5) creating a process for ensuring that the criteria change as organizational objectives change. (The complete list of key practices is provided in table 4.)

CMS has executed 4 of the 10 key practices associated with selecting an investment. Specifically, CMS used a process it defined in February 2005—its budget formulation process—to select new investments and reselect existing investments using a set of limited criteria. We confirmed that the four projects we reviewed were reselected using this new process. In addition, by using the budget formulation process to select investments, executives had assurance that funding decisions were aligned with selection decisions. Officials indicated that adequate resources were provided for identifying and selecting investments.

However, weaknesses remain in the selection area. Although CMS has a number of documents that address investment selection and reselection, these documents are not linked to provide a clear understanding of the selection and reselection process. In addition, they do not define (1) the roles and responsibilities for each participating unit involved in the project selection process and (2) the decision-making procedures. CMS officials told us they chose to first implement the selection process and then go back to document it. Another key weakness in the selection area is that, although selection and reselection criteria have been defined, they do not include cost, benefit, schedule, and risk factors. Officials indicated that because the Executive Steering Committees and the ITIRB had a short amount of time to perform selection activities this year, they defined a limited set of criteria to evaluate projects. Further, CMS does not have a mechanism to ensure that its selection criteria continue to reflect organizational objectives.

Until CMS implements all of the key practices associated with selecting investments, executives will not be adequately assured that they are consistently and objectively selecting projects that meet the needs and priorities of the agency in a cost-effective and risk-insured manner.

Table 4 shows the rating for each key practice required to implement the critical process for selecting an investment at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

An organization should provide effective oversight for its IT projects throughout all phases of their life cycles. Its investment board should maintain adequate oversight and observe each project’s performance and progress toward predefined cost and schedule expectations as well as each project’s anticipated benefits and risk exposure. The investment board should also employ early warning systems that enable it to take corrective action at the first sign of cost, schedule, or performance slippages. This board has ultimate responsibility for the activities within this critical process. According to the ITIM framework, effective project oversight requires, among other things, (1) having written policies and procedures for management oversight; (2) developing and maintaining an approved management plan for each IT project; (3) making up-to-date cost and schedule data for each project available to the oversight boards; (4) having regular reviews by each investment board of each project’s performance against stated expectations; and (5) ensuring that corrective actions for each underperforming project are documented, agreed to, implemented, and tracked until the desired outcome is achieved. (The complete list of key practices is provided in table 5.)

CMS has only executed 1 of the 7 key practices associated with effective project oversight. While CMS’s IT Investment Management Process Guide addresses management oversight of IT projects and systems, it does not include specific procedures for the ITIRB’s oversight of IT projects and systems. In addition, while the board is receiving performance data for some investments, including revitalization investments, it is not yet performing oversight of projects on a systematic basis. CMS officials indicated that the ITIRB’s involvement in overseeing investments to date has been limited because the board was first focusing on selecting investments. However, CMS recognizes the importance of the ITIRB’s involvement in oversight of IT investments, and, according to officials, the agency is currently developing an approach to address this issue.

While CMS is in the process of developing a structured process for the ITIRB to oversee investments, other entities are involved in the oversight of projects. For example, performance information for one of the projects we reviewed was not provided to CMS’s ITIRB, but instead was provided to senior-level management, such as the Chief Technology Officer and directors from some CMS components. Until the ITIRB systematically oversees CMS’s investments, the oversight process will not benefit from the corporate perspective that is gained by having an enterprisewide board. As a result, executives may not be able to easily determine the impact individual project decisions may have on other projects and on the attainment of organizational goals and objectives.

Table 5 shows the rating for each key practice that is required to implement the critical process for project oversight at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

To make good IT investment decisions, an organization must be able to acquire pertinent information about each investment and store that information in a retrievable format. During this critical process, an organization identifies its IT assets and creates a comprehensive repository of investment information. This repository provides information to investment decision makers to help them evaluate the impacts and opportunities that would be created by proposed or continuing investments. It can provide insights and trends about major IT cost and management drivers. The repository can take many forms and does not have to be centrally located, but the collection method should identify each IT investment and its associated components. This critical process may be satisfied by the information contained in the organization’s current enterprise architecture, augmented by additional information—such as financial information and information on risk and benefits—that the investment board may require to ensure that informed decisions are being made. According to the ITIM framework, effectively managing this repository requires, among other things, (1) developing written policies and procedures for identifying and collecting the information, (2) assigning responsibility for ensuring that the information being collected meets the needs of the investment management process, (3) identifying IT projects and systems and collecting relevant information to support decisions about them, and (4) making the information easily accessible to decision makers and others. (The complete list of key practices is provided in table 6.)

CMS has in place 5 of the 6 key practices associated with capturing investment information. For example, CMS’s IT Investment Management Process Guide identifies specific information that is needed in the investment management process, such as how each IT project relates to the business needs of CMS. According to officials, adequate resources are provided to support the collection of investment information, such as the agency’s IT Investment Tracking Database, and an individual assigned responsibility for ensuring that the necessary information is collected to meet the needs of the investment management process.

CMS is collecting specific information about IT investments to support decisions about these investments, including projects’ scores against selection criteria and earned value management information. We verified that this information was collected for the four projects we reviewed. However, although the ITIRB has used investment information to support selection decisions, it has not yet used it to systematically oversee projects. According to CMS officials, specific procedures for the ITIRB’s oversight of IT projects and systems are currently being defined.

Table 6 shows the rating for each key practice required to implement the critical process for capturing investment information at the Stage 2 level of maturity and summarizes the evidence that supports these ratings.

CMS Lacks the Key Capabilities Needed to Manage Its Investments as a Portfolio

During Stage 3, the investment board enhances the investment management process by developing a complete investment portfolio. An investment portfolio is an integrated, agencywide collection of investments that are assessed and managed collectively on the basis of common criteria. Managing investments within the context of such a portfolio is a conscious, continuous, and proactive approach to expending limited resources on an organization’s competing initiatives in light of the relative benefits expected from these investments. Taking an agencywide perspective enables an organization to consider its investments comprehensively, so that, collectively, the investments optimally address the organization’s missions, strategic goals, and objectives. Managing investments with a portfolio approach also allows an organization to determine priorities and make decisions about which projects to fund, and continue to fund, on the basis of analyses of the relative organizational value and risks of all projects, including projects that are proposed, under development, and in operation. For an organization to reap the full benefits of the portfolio process, it should collect all of its investments into an enterprise-level portfolio that is overseen by its senior investment board. Although investments may initially be selected into subordinate portfolios—on the basis of, for example, the lines of business or life-cycle stages-—and managed by subordinate investment boards, they should ultimately be aggregated into this enterprise-level portfolio.

According to our ITIM framework, critical processes performed by Stage 3 organizations include (1) defining the portfolio criteria, (2) creating the portfolio, (3) evaluating the portfolio, and (4) conducting postimplementation reviews. Table 7 shows the purpose of each critical process in Stage 3.

CMS has executed very few key practices—2 of 27—associated with Stage 3 critical processes. Specifically, under the critical process for defining the portfolio criteria, CMS provided evidence that it had designated a working group to be responsible for developing and modifying portfolio selection criteria. Under the critical process for creating the portfolio, CMS provided evidence that it was capturing and maintaining investment information for future reference. In its self-assessment, the agency stated that it was not executing any other Stage 3 key practices. According to officials, CMS has not concentrated on implementing Stage 3 key practices because the agency is first focusing its resources on establishing the practices associated with Stage 2. Until CMS fully implements the critical processes associated with managing its investments as a complete portfolio, it will not have the data or enterprisewide perspective it needs to make informed decisions about its collection of investments.

CMS Does Not Have a Comprehensive Plan to Coordinate and Guide Its Improvement Efforts

CMS has initiated efforts to improve its investment management process. While these efforts do not fully address any of the weaknesses we identify in this report, they enhance the agency’s ability to perform key activities. Specifically: CMS has begun to implement a tool for capturing project information.

According to officials, the tool will bring together investment information currently residing in various locations, including project description information captured in its IT Investment Tracking Database, information such as project scores collected to support project selection activities, and earned value management data.

Although information to support investment decisions does not have to be in one location, doing so will improve accessibility and facilitate its use by decision makers.

CMS recently established Executive Steering Committees to support the ITIRB in carrying out its investment management responsibilities. These groups played a key role in selecting investments for the fiscal year 2007 budget by reviewing investment information and making recommendations for funding to the investment board. They are currently determining procedures for overseeing investments. According to officials, once procedures for the Executive Steering Committee oversight have been determined, CMS plans to focus on defining procedures for determining how and when to involve the investment board in oversight—a key weakness identified in this report.

Although CMS has initiated these improvement efforts, it has not coordinated them with the additional efforts needed to address the weaknesses identified in this report in a comprehensive plan that (1) specifies measurable goals, objectives, and milestones; (2) specifies needed resources; (3) assigns clear responsibility and accountability for accomplishing tasks; and (4) is approved by senior-level management. We have previously reported that such a plan is instrumental in helping agencies coordinate and guide improvement efforts.

CMS officials recognize the value of having a comprehensive plan and told us they have begun to develop one; however, a time frame for completing the plan has not been established. Until CMS develops this plan, the agency risks not being able to put in place an effective management process that will provide appropriate executive-level oversight for minimizing risks and maximizing returns.

Process for Monitoring MMISs Could Benefit from Standard Procedures, Guidance, and Reporting Requirements

As we previously noted, the responsibility for approving and monitoring MMISs that CMS funds jointly with the states falls to CMS’s central office and its 10 regional offices, with the bulk of the activities being performed by the regional offices. Although the process for approving states’ funding requests for MMIS activities is characterized by (1) standard procedures performed consistently across the regional offices, (2) guidance that staff can rely on in carrying out their duties, and (3) requirements for reporting information to the central office, the process for monitoring MMIS activities is not.

Standard Procedures, Guidance, and Reporting Requirements Exist for the Approval Process

The process for approving states’ requests for federal funding of MMISs is characterized by a defined set of activities that are performed consistently across the regional offices. These activities include regional office staff review and approval of the standard documentation (i.e., the APDs, request-for-proposals, and contracts) that the states prepare to justify their requests. Specifically, as we previously described: States prepare an APD to request funding for MMISs. Regional office staff review the document to ensure that states’ requests support the Medicaid program, are in compliance with federal requirements, and represent cost-effective solutions. Once regional office staff determine that the APD adequately justifies the request, they issue a formal approval letter to the states (with concurrence from CMS’s central office).

The request-for-proposals that the states prepare to solicit contractor bids for MMIS activities, including development and operations, is reviewed and approved by regional office staff through a process similar to that used to approve the APDs.

Regional office staff review the states’ process for reviewing contractors’ proposed bids.

Regional office staff review and approve the contract, after which the state makes an award to the contractor whose bid or offer is responsive to the solicitation and is most advantageous to the state—considering price, quality, and other factors.

Regional office staff told us that they rely on the State Medicaid Manual and the Code of Federal Regulation for guidance in performing activities for approving states’ requests for federal funding. Regional staff are also required to inform the CMS central office of all approval actions through the Office of the Secretary Notice process previously mentioned.

Process for Monitoring State MMISs Lacks Standard Procedures, Guidance, and Reporting Requirements

In contrast to the approval process, the process for monitoring MMIS activities lacks (1) standard procedures regional office staff must perform to carry out their responsibilities, (2) guidance for staff to rely on, and (3) requirements for staff to report on the results of their monitoring efforts to the central office.

First, regional office staff use a variety of mechanisms to monitor MMIS activities. These mechanisms include reviews of project status reports; site visits; telephone calls; and meetings with external groups, such as industry associations, provider groups, and vendors. In addition, regional office staff determine if and when to use these mechanisms. Table 8 shows the different mechanisms used by the regional office staff we interviewed and the number of regional offices who used them.

Second, CMS has no guidance for regional office staff to use in monitoring MMIS activities. While CMS has a Regional Office Manual that includes guidance for monitoring MMIS activities, this manual is not used by regional office staff because, according to officials, it has not been maintained throughout the years, and it no longer reflects current processes.

Third, there are no requirements for regional office staff to report to CMS’s central office on their monitoring of states’ federally funded MMISs activities. Monthly teleconferences are conducted between the central office and regional offices to discuss activities performed by these offices, including activities to monitor state MMISs. According to CMS officials, there is some communication outside of the scheduled teleconferences to discuss any issues that might arise regarding the status of state MMISs. In addition, according to officials, the certification reviews performed about 6 months after the MMISs have become operational provide opportunities to determine firsthand how systems are performing. Despite these mechanisms, the central office has no requirements for regional office staff to regularly report on the results of their efforts to monitor MMIS activities.

According to CMS officials, the central office has traditionally placed greater emphasis on the front-end approval of requests for federal funding. The central office, however, now recognizes the need for and value of adopting an approach for maintaining the visibility of MMISs from beginning to end. To address this need, central office staff told us that they plan to ask the regional offices to provide them with quarterly reports on the status of MMIS activities in their states as part of a broader effort that is currently under way to improve the administration of the Medicaid program. Central office staff stated this effort would also result in standard procedures and guidance to support regional office staff’s monitoring efforts. While these activities would strengthen the monitoring process, during our review central office staff did not yet have specific plans or time frames for implementing them.

Until CMS defines standard procedures for monitoring MMIS activities, guidance for staff to rely on, and reporting requirements, CMS’s central office may not be able to easily determine whether state MMISs are facilitating the delivery of Medicaid benefits in the most effective and beneficial manner.

Conclusions

Because IT investment management has only recently become an area of management focus, CMS capabilities to manage its internal investments are limited. Specifically, the agency has established about half of the practices for building the investment foundation, but few practices to manage its investments as a portfolio. Although the foundational practices have equipped CMS with the capabilities it needs to improve its management of individual investments, the agency is hampered in its ability to manage them as a portfolio because it has not implemented the practices for doing so. Until CMS fully establishes the key practices required to build the investment foundation and manage its investments as a portfolio, it will not have the capabilities it needs to ensure that investments supporting its multibillion-dollar Medicare and Medicaid programs are being managed to minimize risks and maximize returns.

Critical to CMS’s success in going forward will be the development of an implementation plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior-level management. Although the agency has initiated improvement efforts, it has not developed a comprehensive plan to guide these and other efforts needed to improve its investment management process. Without such a plan and procedures for implementing it, CMS will be challenged in sustaining the commitment it needs to fully establish its investment management process.

Finally, the process for approving states’ funding requests for MMIS activities is characterized by standard procedures that are performed consistently across the regional offices, guidance, and requirements for informing the central office of regional office staff activities. The process for monitoring the development and operations of state MMIS, on the other hand, has no standard procedures for regional office staff, no guidance, and no requirement to report information to the central office. Without these elements for monitoring MMIS activities, CMS’s central office may not be able to easily determine whether state MMISs are facilitating the delivery of Medicaid benefits in the most effective and beneficial manner.

Recommendations for Executive Action

To strengthen CMS’s capability to manage its internal IT investments and address the related weaknesses addressed in this report, we recommend that the Secretary of the Department of Health and Human Services direct the CMS Administrator to develop and implement a plan for improving CMS’s IT investment management processes. The plan should address the weaknesses described in this report. The plan should (1) first focus on correcting the weaknesses in Stage 2 critical processes, and next focus on the Stage 3 critical processes, and (2) at a minimum, provide for accomplishing the following 12 actions: Update the agency’s investment management guide to reflect current investment management processes.

Establish a process for the board to actively maintain the agency’s documented investment management process.

Use an updated strategic plan or other detailed statement of business mission with supporting goals and objectives to align investments with business needs.

Ensure that the board periodically evaluates the alignment of IT projects and systems with strategic goals and objectives and take corrective actions when misalignment occurs.

Fully document procedures that address investment selection and reselection and (1) provide a clear understanding of the selection and reselection process, (2) define the roles and responsibilities for each participating unit involved in the project reselection process, and (3) define the decision-making procedures.

Document procedures for integrating funding with investment selection.

Revise the ITIRB’s selection and reselection criteria to include cost, benefit, schedule, and risk factors, and establish a mechanism to ensure these criteria continue to reflect organizational objectives.

Define, document, and implement procedures for the ITIRB’s oversight of projects and systems.

Implement processes to use investment information to fully support investment management decisions.

Implement the Stage 3 critical processes for defining portfolio criteria, creating the portfolio, evaluating the portfolio, and conducting postimplementation reviews, which are necessary for portfolio management.

We also recommend that the Secretary for Health and Human Services direct the CMS Administrator to ensure that the plan draw together ongoing efforts and additional efforts that are needed to address the weaknesses identified in this report. The plan should also (1) specify measurable goals, objectives, and milestones; (2) specify needed resources; (3) assign clear responsibility and accountability for accomplishing tasks; and (4) be approved by senior-level management. In implementing the plan, the Administrator should ensure that progress is measured and reported periodically to the Secretary of Health and Human Services.

To improve CMS’s process for monitoring states’ progress in developing and maintaining Medicaid management information systems, we are recommending that the Secretary of the Department of Health and Human Services direct the CMS Administrator to take the following two actions: Define standard procedures and supporting guidance for regional offices to monitor MMIS activities.

Require regional offices to regularly report on their MMIS monitoring activities to CMS’s central office.

Agency Comments and Our Evaluation

The Administrator of CMS provided written comments on a draft of this report (reprinted in app. II). In these comments, CMS identified actions it is taking or plans to take to address our recommendations and stated that effective management of IT investments is a critical priority at the agency. CMS contended that many of the agency’s improvements to its IT investment management process were not fully reflected in the report, and took exception with the need for up-to-date, documented processes to ensure consistency.

Concerning our description of progress in implementing investment management processes, CMS commented that the report indicates that the agency has only established 2 out of 27 key practices needed to manage investments as a portfolio. CMS stated that this is misleading since the report also indicates that the agency has accomplished 20 of 38 foundational IT investment management practices. CMS also provided examples of the practices it has implemented, such as establishing an investment review board. In our report, we make a distinction between foundational practices, which are the Stage 2 key practices for establishing basic project-level selection and control capabilities, and portfolio-level practices, which are the Stage 3 key practices for managing investments as an integrated set of competing options. We also note that both of these sets of key practices are needed to implement the processes required by the Clinger-Cohen Act of 1996. On the basis of this, we state that CMS does not have the full suite of capabilities to manage its internal investments because it has only established a little over half of the foundational practices and 2 of 27 portfolio-level key practices, and we reiterate the need to fully establish both sets of practices to increase assurance that executives are selecting and managing the mix of investments that best meets the agency’s needs and priorities. In our report, the sections in which we discuss the implementation of specific key practices associated with critical processes from our IT investment management framework each describe CMS’s efforts and accomplishments to improve its IT investment management processes. These include all of the examples of accomplishments CMS provided in its comments.

In its comments, CMS took issue with our reporting that its IT investment management guide did not reflect the current process, and that its procedures for selecting and reselecting IT investments were not fully documented. Although the agency fully agreed that an up-to-date guide would constitute a piece of an effective process, it commented that the emphasis should be on strengthening the process first and updating documentation later. CMS made three points: (1) it is not practical to publish an updated guide without having the effective and repeatable underlying process in place and noted that it is not provided the latitude to do this; (2) in the section of the report discussing instituting the investment board, the noted successful execution of key practices appears to be negated by the statement that the investment management processes are not documented; and (3) in the same section of the report, we are implying that an updated guide would improve rather than explain the process. We disagree with CMS that the process needs to be repeatable and strengthened before it can be documented. Documented procedures could actually serve to strengthen and improve the process by ensuring it is performed consistently. Finally, we are not negating the successful implementation of key practices to institute the investment board. We are simply emphasizing the importance of having documentation to drive the investment management process.

In its comments, CMS also noted actions it is taking to (1) develop a plan to implement key practices in Stages 2 and 3; (2) revise existing documentation to reflect processes in place that are not formally documented; and (3) develop a plan that will be approved by senior management that will incorporate goals, objectives, and milestones required to further close the gaps between existing processes and our ITIM framework. Regarding our recommendation to improve its process for monitoring state MMIS activities and reporting to the central office, CMS stated that it is developing standard procedures and supporting guidance for the regional office(s) for monitoring these systems activities and reporting to the central office. We agree with CMS that these actions would address many of the weaknesses we identify in this report.

CMS also provided some technical comments, which we have incorporated into the report as appropriate.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to other interested congressional committees, the Secretary of Health and Human Services, the CMS Administrator, the CMS Chief Information Officer, and other interested parties. Copies will also be made available at no charge on our Web site at www.gao.gov. If you have any questions on matters discussed in this report, please contact David A. Powner at (202) 512-9286, or at pownerd@gao.gov, or Leslie G. Aronovitz at (312) 220-7600, or at aronovitzl@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Objectives, Scope, and Methodology

The objectives of our review were to (1) evaluate Centers for Medicare & Medicaid Services (CMS) capabilities for managing its internal information technology (IT) investments, (2) determine any plans the agency might have for improving these capabilities, and (3) examine CMS’s process for approving and monitoring the state Medicaid management systems it funds.

To address our first objective, we reviewed the results of the agency’s self- assessment of Stages 2 and 3 practices using our Information Technology Investment Management framework (ITIM) and validated and updated the results of the self-assessment through document reviews and interviews with officials. We reviewed written policies, procedures, and guidance and other documentation providing evidence of executed practices, including CMS’s IT Investment Management Process Guide, CMS’s Policy for IT Investment Management, and CMS’s fiscal year 2006/2007 budget process. We also reviewed the CMS Information Technology Investment Review Board (ITIRB) meeting minutes. We did not assess CMS’s progress in establishing the capabilities found in Stages 4 and 5 of the ITIM framework because CMS acknowledged that it had not executed any of the key practices in these higher maturity stages. In addition, we conducted interviews with officials from the Office of Information Services who have responsibility for the development and implementation of CMS’s IT investment management process.

We compared the evidence collected from our document reviews and interviews with the key practices in our ITIM framework. We rated the key practices as “executed” on the basis of whether the agency demonstrated (by providing evidence of performance) that it had met the criteria of the key practice. A key practice was rated as “not executed” when we found insufficient evidence of a practice being executed or when we determined that there were significant weaknesses in CMS’s execution of the key practice. In addition, CMS was provided with the opportunity to produce evidence for key practices rated as “not executed.”

As part of our analysis, we selected four CMS IT projects as case studies to verify that the critical processes and key practices were being applied. The projects were selected because they (1) supported different functional areas, (2) were in various life-cycle phases, and (3) required different levels of funding. The four projects are described below: Healthcare Integrated General Ledger Accounting System—The project is intended to standardize the collection, recording, and reporting of Medicare financial information by contractors. It is to replace the cumbersome ad hoc spreadsheets and “cuff” systems being used by Medicare contractors to accumulate and report financial information to CMS. The project’s life-cycle cost is estimated at about $567 million.

Medicare Claims Processing Redesign—This project is intended to integrate and modernize the Common Working File system and Redesign and the Medicare Shared Systems enterprise claims processing applications and data systems. The modernization and unification of these systems is to allow CMS to significantly enhance program capabilities, integrity, performance, efficiencies, and maintainability; reduce program change implementation time frames; improve accuracy, timeliness, and quality of Medicare transaction processing; reduce system exposure to security risks; and facilitate use of the Internet. The project’s life-cycle cost is estimated at nearly $494 million.

Medicare Managed Care System—This project is intended to cover the redesign of CMS’s managed care family of systems, including the legacy Group Health Plan system. It is to provide the platform for implementing requirements under the MMA. The project is intended to replace aging operations and to continue to support the agency’s managed care business needs until all functions are migrated to a new system. Its life- cycle cost is estimated at about $111 million.

National Plan and Provider Enumeration System—The project is intended to implement a Health Insurance Portability and Accountability Act requirement to issue a unique identifier to each covered health care provider in the United States. It is expected to result in administrative savings by simplifying a complicated, multifaceted enumeration scheme, whereby a provider is issued different identifiers for electronic transactions by each health plan with which it does business, and sometimes multiple identifiers from a single plan. It will impact several million providers and health plans in the nation. The project’s life-cycle cost is estimated at about $38 million.

For these projects, we reviewed project management documentation, such as project plans, business cases, status reports, and documentation on how these projects were selected by the ITIRB. We also interviewed the project managers for these projects.

To address our second objective, we examined documentation on what management actions had been taken and what initiatives had been planned by the agency. This documentation included a requirements document for a tool CMS is currently implementing that is to help the agency with IT investment management, among other things. We also interviewed officials from the Office of Information Services to determine what efforts CMS had undertaken to improve IT investment management processes.

To address our third objective, we reviewed documentation supporting CMS’s implementation of processes for (1) approving states’ requests for funding their Medicaid Management Information Systems (MMIS) and (2) monitoring these MMISs, including related legislation, policy, and implementing guidance. We also interviewed officials at CMS headquarters and at the 5 CMS regional offices with the highest fiscal year 2004 expenditures for administrative services, which includes MMISs.

We conducted our work at CMS headquarters in Washington, D.C., and at 5 CMS regional offices located in New York, New York; Philadelphia, Pennsylvania; Chicago, Illinois; San Francisco, California; and Atlanta, Georgia, from January 2005 through September 2005 in accordance with generally accepted government auditing standards.

Comments from the Centers for Medicare & Medicaid Services

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to the persons named above, William G. Barrick, Shaunessye Curry, Mary Beth McClanahan, Sabine R. Paul, and Amos Tevelow made key contributions to this report.