Information Technology: HHS Has Several Investment Management Capabilities in Place, but Needs to Address Key Weaknesses
Published: 2005-10-28T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Health and Human Services (HHS) is one of the largest federal agencies, the nation's largest health insurer, and the largest grant- making agency in the federal government. The department manages over 300 programs that serve to improve the health and well-being of the American public and is comprised of several component agencies covering a wide range of activities including conducting and sponsoring medical and social science research, guarding against the outbreak of infectious diseases, assuring the safety of food and drugs, and providing health care services and insurance. It also manages and funds a variety of information technology (IT) initiatives ranging from those facilitating the payment of claims for Medicare and Medicaid services to those supporting health surveillance and communications. In fiscal year 2006, the department plans to spend over $5 billion on information technology--the third largest IT expenditure in the federal budget. As we agreed with Congress, our objectives were to (1) assess the department's capabilities for managing its IT investments and (2)determine any plans the department might have for improving those capabilities. To address these objectives, we analyzed documents and interviewed agency officials to (1)validate and update HHS's self-assessments of key practices in the framework and (2)evaluate HHS's plans for improving its capabilities.
section_title: What GAO Found
paragraphs: Because of the management attention that has been given to IT investment management, HHS has established over half of the foundational practices needed to manage its IT investments individually and about 30 percent of the key practices needed to effectively manage its portfolio of investments. For example, HHS has implemented many of the practices required to ensure that (1) projects support business needs and meet users' requirements, (2) a well-defined and disciplined process is used to select IT investments, (3) investment information is captured in a repository for decision makers, and (4) IT portfolio selection criteria are developed and maintained. However, critical weaknesses remain in several areas. Specifically, HHS lacks: (1) business representation on its senior IT investment review board of component agencies to carry out its full scope of responsibilities, (2) an established process for the IT investment board to regularly review a defined set of the component agencies' IT investments and maintain visibility of other investments, (3) criteria for assessing portfolio performance or regularly reviewing the performance of the organization's investment portfolio, and (4) processes for conducting post-implementation reviews (PIR) of its IT investments. The department also does not have a structured mechanism in place for ensuring that component agencies define and implement investment management processes that are aligned with those of the department. Until the department fully establishes all foundational and portfolio-level practices and establishes a mechanism to ensure that component agencies define and implement processes that are aligned with those of the department, executives cannot be assured that they are appropriately selecting, managing, and evaluating the mix of investments that will maximize returns to the organization, taking into account the appropriate level of risk. HHS has initiated steps to improve its investment management process; however, these steps do not fully address the weaknesses we identify in this report, nor are they coordinated along with other needed improvement efforts into a plan that (1) is based on an assessment of strengths and weaknesses; (2) specifies measurable goals, objectives, and milestones; (3) specifies needed resources; (4) assigns clear responsibility and accountability for accomplishing tasks; and (5) is approved by senior management. Without such a plan and procedures for implementing it, the department risks being unable to effectively establish mature investment management capabilities. As a result, executives may not be able to make informed and prudent investment decisions in managing the department's annual multibillion-dollar IT budget.

section_title: Letter
paragraphs: The Department of Health and Human Services (HHS) is one of the largest  federal agencies, the nation’s largest health insurer, and the largest grant- making agency in the federal government. The department manages over  300 programs that serve to improve the health and well-being of the  American public and is comprised of several component agencies covering  a wide range of activities including conducting and sponsoring medical and  social science research, guarding against the outbreak of infectious  diseases, assuring the safety of food and drugs, and providing health care  services and insurance. It also manages and funds a variety of information  technology (IT) initiatives ranging from those facilitating the payment of  claims for Medicare and Medicaid services to those supporting health  surveillance and communications. In fiscal year 2006, the department plans  to spend over $5 billion on information technology—the third largest IT  expenditure in the federal budget.
This report is one of two we prepared in response to your request that we  evaluate HHS’s information technology investment management  capabilities. It focuses on HHS’s processes for making IT investment  management decisions and evaluates how well these processes compare  with the accepted practices presented in our IT investment management  (ITIM) framework. This framework provides a method for evaluating and  assessing how well an agency is selecting and managing its IT resources. As  we agreed with your office, our objectives were to (1) assess the  department’s capabilities for managing its IT investments and  (2) determine any plans the department might have for improving those  capabilities. To address these objectives, we analyzed documents and  interviewed agency officials to (1) validate and update HHS’s self- assessments of key practices in the framework and (2) evaluate HHS’s  plans for improving its capabilities. We performed our work from January  through September 2005, in accordance with generally accepted  government auditing standards. Appendix I contains details about our  objectives, scope, and methodology.
subsections: section_title: Background
subsections: section_title: HHS’s Mission, Organizational Structure, and Use of IT
paragraphs: HHS is the primary organization within the federal government that is  devoted to protecting the health of Americans. It provides essential human  services, such as ensuring food and drug safety and assisting needy  families. HHS administers more grant dollars than all other federal  agencies combined, providing over $200 billion of the more than $350  billion in federal funds that were awarded to states and other entities in  fiscal year 2002, the most recent year for which these data are available.  For fiscal year 2005, HHS had a budget of $581 billion and a workforce of  over 67,000 employees.
To accomplish its mission, HHS is comprised of 12 component agencies  and several staff offices that cover a wide range of activities—including  conducting and sponsoring medical and social science research, guarding  against the outbreak of infectious diseases, assuring the safety of food and  drugs, and providing health care services and insurance. The Office of the  Secretary consists of several staff divisions and offices, including the Office  of the Assistant Secretary for Budget, Technology, and Finance. The HHS  Office of the Chief Information Officer (CIO) is located within this staff  office (see fig. 1).
Information technology investments play a critical role in helping HHS  carry out its diverse mission. According to the President’s most recent  budget, HHS expects to spend about $5 billion in IT in fiscal year 2006,  making the department’s IT investment budget the third largest in the  federal government. As figure 2 illustrates, approximately $3 billion is  designated as grants to states for investments for Medicaid programs and  other purposes, such as child support enforcement systems. Approximately  $2 billion is for discretionary investment spending, of which 89 percent is  used to fund IT investments for component agencies; 7 percent is invested  in HHS enterprisewide initiatives; and 4 percent is used to fund other  initiatives, including Office of the Inspector General IT investments.
Table 1 provides additional information about the component agencies and  their estimated IT budget for fiscal year 2006.
HHS’ investments reflect the diversity of the department’s missions and  operating environments. For example, HHS currently has several  enterprisewide IT initiatives that enable stakeholders to advance the  causes of better health, safety, and well-being for American people. These  initiatives include:  Unified Financial Management System, a new core financial system, to  help management monitor budgets, conduct operations, evaluate  program performance, and make financial and programmatic decisions.  As a core financial system, it will interface with an estimated 110 other  HHS information systems.
The Office of the Assistant Secretary for Public Health Emergency  Preparedness maintains a command center where it can coordinate the  response to public health emergencies from one centralized location.  This center is equipped with satellite teleconferencing capability,  broadband Internet hookups, and analysis and tracking software.
In addition, HHS’s component agencies have several projects and systems  that are critical to the effective implementation of HHS’s mission, including  the following:   The Food and Drug Administration’s Automated Drug Information  Management System is to be developed as a fully electronic information  management system that will receive, evaluate, and disseminate  information about investigational and marketing submissions for human  drugs and therapeutic biologics.
The National Institutes of Health’s major IT initiative, the Clinical  Research Information System, is a comprehensive effort to modernize  the systems that support clinical care and the agency’s collection of  research data for the intramural clinical research programs.
The Centers for Disease Control and Prevention’s major IT initiative,  Public Health Information Network, is a national initiative to implement  a multiorganizational business and technical architecture for public  health information systems.
section_title: Prior Reviews Identified Weaknesses in HHS’s IT Investment Management Process
paragraphs: In January 2004, we reported on a broad view of the government’s  implementation of investment management practices at 26 major  departments and agencies, including HHS. We also reported—and HHS  acknowledged—that there were serious weaknesses in investment  management. Notably, the department had not yet established selection  criteria for project investments or a requirement that investments support  work processes that have been simplified or redesigned. In addition, the  department did not have decision-making rules to guide oversight of IT  investments, review projects at major milestones, or systematically track  corrective actions. Accordingly, we made several recommendations,  including that HHS revise its investment management policy and require  PIRs to address validating benefits and costs. In response to our  recommendations, the department has been modifying several of its  investment management policies, including its capital planning and  investment control guidance and its governance policies.
More recently, in June 2005, we reported that the HHS IT Investment  Review Board had conducted only budgetary reviews of the Centers for  Disease Control and Prevention’s Public Health Information Network and  some of its initiatives, until this past February, when HHS initiated steps for  better monitoring of system development projects. We concluded that until  management implements a systematic method for IT investment reviews, it  will have difficulty minimizing risks while maximizing returns on these  critical public health investments.
section_title: HHS’s Approach to Investment Management
paragraphs: HHS has several groups and individuals involved in managing both the  enterprisewide and component agency IT investments. They are involved  from reviewing and approving a proposed IT project, through the process  of budgeting for it, monitoring it through implementation, and evaluating it  at its conclusion. The composition, roles, and responsibilities of these  individuals and groups are described below: Information Technology Investment Review Board (ITIRB)—Chaired by  HHS’s CIO, this board is responsible for selecting, controlling, and  evaluating all departmental IT investments. Members include the Deputy  Assistant Secretary for Budget, Finance, Performance and Planning; the  Directors for Acquisition Management Policy and Human Resources; and  the component agency CIOs. The board is supported by an executive  secretary who is responsible for, among other things, managing the flow of  IT investment documentation, scheduling meetings, and assisting the  members in preparing for their meetings. Currently, this board reviews all  enterprisewide investments and delegates responsibilities for component  agency investments to each individual component agencies investment  review boards in accordance with departmental policies and procedures.
CIO Council—Also chaired by the HHS CIO and comprised of component  agency CIOs, this board advises the HHS ITIRB on the technical soundness  of all IT investments that require departmental review and provides  recommendations regarding, among other things, technical aspects of  affordability, soundness of design, risk, and compliance with architectural  and security standards.
Critical Partners—Comprised of departmental officials from various  functional areas, including enterprise architecture, security and privacy,  acquisition management, finance, budget, human resources, and e-government; this group is responsible for ensuring that most  investments comply with the HHS policy in each of the functional areas  and for advising the HHS ITIRB and individual IT investment managers on  issues in their areas of expertise. Each review results in a determination  whether the investment is approved, conditionally approved, or not  approved. A not approved result is flagged for executive review.
Business Case Quality Review Team—Comprised of component agency  officials, this group evaluates the justifications for IT investments—both  formal business cases and information documented in the department’s  portfolio management tool’s Select forms—against the criteria used by the  Office of Management and Budget’s to evaluate business cases agencies  submit to the office as part of the formulation of the federal budget and  provides recommendations for improving these justifications.
Capital Planning and Investment Control (CPIC)  Reengineering/Portfolio Management Tool (PMT) Implementation  Team—Chaired by the Office of the CIO officials with representatives from  the Critical Partners and the Business Case Quality Review Team, this  group advises the board on issues regarding investment management  policies and procedures and the implementation of the department’s  portfolio management tool.
Investment Managers—Responsible for managing investments in  accordance with approved cost, schedule, and performance baselines, and  for maintaining information on project status, control, performance, risk,  and corrective actions.
section_title: Process for Managing Investments
paragraphs: The department has defined a three-phase process for managing  investments that involves selecting proposed projects and reselecting  ongoing projects (select phase), controlling ongoing projects through  development (control phase), and evaluating projects that have been  deployed (evaluate phase). The department retains direct management of  HHS enterprisewide IT investments and delegates considerable authority  for other investments to component agencies. Specifically, the department  selects ongoing and new component agency investments through the  process for selecting enterprisewide IT investments described below.  Controlling and evaluating component agency IT investments are delegated  to the component agencies, which are required by the department to follow  a process similar to the one described below.
Each phase of the process for enterprisewide investments is comprised of  multiple steps that set out requirements needed for the HHS ITIRB to make  the decision to move forward with the project.
The purpose of the select phase is to ensure that HHS chooses the projects  that best support its mission and applies resources to the most important  and valuable investments. The select phase is also intended to help the  department justify budget requests by demonstrating sound business cases  and project plans. To select investments, HHS has established two separate  components—investment screening for new investment proposals and  investment scoring and screening for ongoing investments.
During the new investment screening, the investment manager is expected  to develop a project prospectus, which identifies a specific business need  and preliminary, high-level system requirements. A high-level  determination of resource and schedule requirements is also to be  conducted as part of the business need identification activities. Approval of  the project prospectus by the HHS ITIRB signifies that the agency agrees  that the need is critical enough to proceed to the next step in which the  business case is developed. During business case development, the  investment manager is required to develop the business case, which  establishes the lifecycle cost, schedule, benefits, and performance  baselines and includes an analysis for each investment to identify  alternatives that may satisfy the needs of the department. In addition, the  investment managers sign a document called the accountability agreement  form to accept responsibility for reporting on the project status in  achieving performance baselines throughout the remaining phases of the  investment management process.
After the project is initially approved by the HHS ITIRB, the business cases  and Select forms for most IT investments are updated annually as part of  the budget formulation process. (The Select forms are a collection of forms  with HHS’s portfolio management tool that capture investment data to  justify funding and ensure adequate project planning during the select  phase.) The first step within the annual budget formulation process  requires that all component agencies use the Select forms to report the  project cost estimates that best represent the level of funding required to  meet program or business needs. At this point, the Critical Partners and the  Business Case Quality Review Team score and rank the Select forms using  the department’s portfolio management tool to create a single HHS  portfolio as well as component agency portfolios to provide  recommendations to the component agencies for making final adjustments  to their portfolio ranking.
Once the component agencies have made the appropriate changes, the  Office of the CIO develops prioritized IT portfolios for HHS as a whole as  well as each component agency to present to the HHS ITIRB. The  departmental board and CIO Council review and comment on the  prioritized portfolio and submit it to the Secretary’s Budget Council for  input into their budget deliberations. The Secretary’s Budget Council then  makes recommendations to the Secretary regarding HHS and component  agencies’ budgets. Finally, the department submits its approved Secretary’s  IT budget to the Office of Management and Budget for inclusion in the  President’s Budget.
Once selected for inclusion in the department’s IT portfolio, each project is  to be managed by an investment manager and reviewed by the ITIRB on a  quarterly basis throughout the end of development. The board performs  reviews of projects that deviate from predetermined budget, schedule, or  performance milestones established in the business case and works with  the investment managers to develop a correction action plan. The ITIRB  must also decide whether to continue to fund the project; rebaseline the  scope, schedule, or budget; or to terminate the project.
Once a project has been fully implemented, the HHS ITIRB is to conduct  annual reviews of all HHS enterprisewide steady state investments—that  is, investments in operations and maintenance—to determine whether they  continue to meet the business needs. In addition, investments that have  recently completed implementation or a significant phase are to undergo  PIRs to evaluate actual development events against project management  plans and to identify lessons learned that can be applied to current and  future investments.
Figure 3 illustrates HHS’s investment management process phases and  steps. The highlighted steps represent the activities that the department  conducts for both enterprisewide and component agency investments.
section_title: ITIM Maturity Framework
paragraphs: The ITIM framework is a maturity model composed of five progressive  stages of maturity that an agency can achieve in its investment  management capabilities. It was developed on the basis of our research  into the IT investment management practices of leading private- and public- sector organizations. In each of the five stages, the framework identifies  critical processes for making successful IT investments. The maturity  stages are cumulative; that is, in order to attain a higher stage the agency  must have institutionalized all of the critical processes at the lower stages,  in addition to the higher stage critical processes.
The framework can be used to assess the maturity of an agency’s  investment management processes and as a tool for organizational  improvement. The overriding purpose of the framework is to encourage  investment processes that increase business value and mission  performance, reduce risk, and increase accountability and transparency in  the decision process. We have used the framework in several of our  evaluations, and a number of agencies have adopted it. These agencies  have used ITIM for purposes ranging from self-assessment to redesign of  their IT investment management processes.
ITIM’s five maturity stages represent steps toward achieving stable and  mature processes for managing IT investments. Each stage builds on the  lower stages; the successful attainment of each stage leads to improvement  in the organization’s ability to manage its investments. With the exception  of the first stage, each maturity stage is composed of “critical processes”  that must be implemented and institutionalized in order for the  organization to achieve that stage. These critical processes are further  broken down into key practices that describe the types of activities that an  organization should be performing to successfully implement each critical  process. It is not unusual for an organization to be performing key  practices from more than one maturity stage at the same time, but efforts  to improve investment management capabilities should focus on  implementing all lower stage practices before addressing higher stage  practices.
In the ITIM framework, Stage 2 critical processes lay the foundation for  sound IT investment processes by helping the agency to attain successful,  predictable, and repeatable investment control processes at the project  level. Specifically, Stage 2 encompasses building a sound investment  management foundation by establishing basic capabilities for selecting new  IT projects. It also involves developing the capability to control projects so  that they finish predictably within established cost and schedule  expectations and the capability to identify potential exposures to risk and  put in place strategies to mitigate that risk. The basic selection processes  established in Stage 2 lays the foundation for more mature selection  capabilities in Stage 3, which represents a major step forward in maturity,  in which the agency moves from project-centric processes to a portfolio  approach, evaluating potential investments by how well they support the  agency’s missions, strategies, and goals.
Stage 3 requires that an organization continually assess both proposed and  ongoing projects as parts of a complete investment portfolio—an  integrated and competing set of investment options. It focuses on  establishing a consistent, well-defined perspective on the IT investment  portfolio and maintaining mature, integrated selection (and reselection),  control, and evaluation processes, which are to be evaluated during PIRs.  This portfolio perspective allows decision makers to consider the  interaction among investments and the contributions to organizational  mission goals and strategies that could be made by alternative portfolio  selections, rather than to focus exclusively on the balance between the  costs and benefits of individual investments.
Stages 4 and 5 require the use of evaluation techniques to continuously  improve both the investment portfolio and the investment processes in  order to better achieve strategic outcomes. At Stage 4 maturity, an  organization has the capacity to conduct IT succession activities and,  therefore, can plan and implement the deselection of obsolete, high-risk, or  low-value IT investments. An organization with Stage 5 maturity conducts  proactive monitoring for breakthrough information technologies that will  enable it to change and improve its business performance. Organizations  implementing Stages 2 and 3 have in place the selection, control, and evaluation processes that are required by the Clinger-Cohen Act of 1996.  Stages 4 and 5 define key attributes that are associated with the most  capable organizations.
Figure 4 shows the five ITIM stages of maturity and the critical processes  associated with each stage.
As defined by the model, each critical process consists of “key practices”  that must be executed to implement the critical process.
section_title: HHS Has Established Many Key Practices for Managing Its Investments, but Has Provided Limited Guidance and Oversight to Component Agencies Processes
paragraphs: In order to have the capabilities to effectively manage IT investments, an  agency, at a minimum, should, (1) build an investment foundation by  putting basic, project-level control and selection practices in place (Stage 2  capabilities) and (2) manage its projects as a portfolio of investments,  treating them as an integrated package of competing investment options  and pursuing those that best meet the strategic goals, objectives, and  mission of the agency (Stage 3 capabilities). These practices may be  executed at various organizational levels of the agency, including at the  component level. However, overall responsibility for their success remains  at the department level. Therefore, at a minimum, the department should  effectively oversee component agencies’ IT investment management  processes.
HHS has executed 24 of the 38 key practices that the ITIM framework  requires to build a foundation for IT investment management (Stage 2) and  8 of the 27 key practices required to manage investments as a portfolio  (Stage 3). However, the department has only provided limited oversight of  component agencies’ ITIM processes. Until HHS implements and oversees  a stable investment management process throughout the department, it  will lack essential management controls over all of its IT investments, and  it will be unable to ensure that it is appropriately selecting, managing, and  evaluating the mix of investments that will maximize returns to the  organization, taking into account the appropriate level of risk.
subsections: section_title: HHS Has Established Over Half of the Foundational Practices Needed to Manage Its Investments
paragraphs: At the ITIM Stage 2 level of maturity, an organization has attained  repeatable, successful IT project-level investment control processes and  basic selection processes. Through these processes, the organization can  identify expectation gaps early and take the appropriate steps to address  them. According to the ITIM, critical processes at Stage 2 include  (1) defining IT investment board operations, (2) identifying the business  needs for each IT investment, (3) developing a basic process for selecting  new IT proposals and reselecting ongoing investments, (4) developing  project-level investment control processes, and (5) collecting information  about existing investments to inform investment management decisions.  Table 2 describes the purpose of each of these Stage 2 critical processes.
In the federal government, the agency head and the CIO are responsible for  effectively managing information technology. The agency head, through  the department-level CIO, is responsible for providing leadership and  oversight for foundational critical processes by ensuring that written  policies and procedures are established, repositories of information are  created that support investment decision making, resources are allocated,  responsibilities are assigned, and all the activities are properly carried out  where they may be most effectively executed. In a large and diverse  organization such as HHS, it is especially critical that the CIO create this  structure and framework to ensure that the organization is effectively  managing its investments at every level. This means that the CIO must  ensure that component agencies have investment management processes  in place that adequately support the department’s investment management  process to make certain that funds are being expended on component  agency investments that will fulfill mission needs.
Because of the management attention that has been given to IT investment  management, the department has put in place over half of the key practices  needed to establish the investment foundation. The department has  satisfied all of the key practices associated with ensuring that projects and  systems support organizational needs and meet users’ needs. It has  satisfied most of the key practices associated with identifying and  collecting investment information, selecting new proposals and  reselecting ongoing investments, and instituting the department’s  investment review board.
However, because of its limited involvement in overseeing component  agency investments, the department has not executed any of the key  practices related to providing investment oversight.
Table 3 summarizes the status of HHS’s critical processes for Stage 2 and  shows how many key practices HHS has executed in managing its IT  investments.
section_title: HHS Has Established an Investment Review Board, but It Is Operating without a Comprehensive Process Guide
paragraphs: The establishment of decision-making bodies or boards is a key component  of the IT investment management process. At the Stage 2 level of maturity,  organizations define one or more boards, provide resources to support the  boards’ operations, and appoint members who have expertise in both  operational and technical aspects of proposed investments. The boards  should operate according to a written IT investment process guide that is  tailored to the organization’s unique characteristics, thus ensuring that  consistent and effective management practices are implemented across the  organization. The organization selects board members to ensure that they  are knowledgeable about policies and procedures for managing  investments. Organizations at the Stage 2 level of maturity also take steps  to ensure that executives and line managers support and carry out the  decisions of the investment board. According to the ITIM, organizations  should (1) use an investment management guide as an authoritative  document to initiate and manage investment processes and (2) provide a  comprehensive foundation for the policies and procedures that are  developed for all of the other related processes. (The complete list of key  practices is provided in table 4.)
The department has executed 5 of the 8 key practices for this critical  process. The department established an IT investment review board as its  corporate-level investment board that consists of senior officials, including  the CIO and the Deputy Assistant Secretaries for Budget, Finance, and  Performance & Planning. The board is adequately resourced, with most  support being provided by the Office of the CIO, whose responsibilities  include developing and modifying the department’s criteria for selecting,  controlling, and evaluating potential and existing IT investments. In  addition, the CIO Council reviews the enterprisewide investments for  technical soundness and provides its recommendations to the board. The  Critical Partners and Business Case Quality Review Team provide  additional support to the board by reviewing and scoring most of their IT  investments.
To ensure that the board’s decisions are carried out for enterprisewide  investments, the ITIRB approves an accountability agreement document  and business case that identify the benefits, costs, and schedule for the  approved investments. The board then monitors the investments through  the end of development. HHS requires the component agencies to follow a  similar process in accordance with departmental policies and procedures.  We verified that an accountability agreement document was signed and the  business case identified performance expectations for the two  enterprisewide IT investments we reviewed—Public Key Infrastructure  and Enterprise Architecture initiatives. Additionally, the board has  oversight of the development and maintenance of the documented IT  investment process through the CPIC Reengineering/PMT Implementation  Team, who provides investment management policy change  recommendations to the board for approval.
Although HHS has implemented these key practices, it does not have a  comprehensive organization-specific process guide to direct the operations  of the investment board. While the Information Resources Management  policy, guidelines, and standard operating procedures provide general  guidance on the organization’s investment management process, they do  not reflect the current investment management process. Moreover, they do  not constitute an IT investment process guide because they do not  sufficiently define the investment process. Specifically, the policies and  procedures do not include information on the roles of the key players such  as the CIO Council, Critical Partners, Business Case Quality Review Team,  or the component agency investment review boards. In addition, they do  not identify the manner in which investment board’s processes are to be  coordinated with other key organizational plans and processes (such as the  budget formulation process). HHS has recently drafted a revised  investment management policy addressing many of these weaknesses;  however, it has not been finalized, and HHS officials could not provide a  final issuance date. Without a comprehensive investment management  process guide, the department lacks the assurance that IT investment  activities will be coordinated and performed in a consistent and cost- effective manner.
Moreover, while HHS has established an IT investment board, the board  does not have business representation (that is, mission representation)  from component agencies. Instead, Chief Information Officers represent  the component agencies. According to HHS’s CIO, the membership of the  board is adequate for carrying out the investment activities it currently  performs—primarily focusing on enterprisewide IT investments. However,  because allocating resources among major IT investments may require  fundamental trade-offs among a multitude of business objectives, portfolio  management decisions are essentially business decisions, and therefore  require sufficient business representation on the board. Until the  department adjusts its board membership to include business  representation from component agencies, it will not have assurance that it  includes those executives who are in the best position to make the full  range of decisions needed to enable the agency to meet its mission most  effectively, particularly as it begins to execute its full range of  responsibility.
Finally, the HHS ITIRB is not operating according to its assigned authority  and responsibility. The department’s investment management policy and  the HHS ITIRB’s charter state that the board has oversight responsibility  for both enterprisewide and a defined set of component agency IT  investments, including projects that are high risk, crosscutting, and require  review by the Office of Management and Budget. However, the board  currently oversees only enterprisewide IT investments. According to HHS  officials, the department has delegated authority to the component  agencies to conduct investment reviews; however, the board does not have  a mechanism in place for ensuring that component agencies are conducting  such reviews in accordance with department policies and procedures. Until  the board operates according to its assigned authority, it cannot ensure that  component agency investments are properly aligned with the organization’s  objectives or reviewed by the appropriate board.
Table 4 shows the rating for each key practice required to institute the  investment board. Each of the “executed” ratings shown below represents  instances where, on the basis of the evidence provided by HHS officials, we  concluded that the specific key practices were executed by the  organization.
Defining business needs for each IT project helps to ensure that projects  and systems support an organization’s business needs and meet users’  needs. This critical process ensures that an organization’s business  objectives and its IT management strategy are linked. According to the  ITIM, effectively meeting business needs requires, among other things,  (1) documenting business needs with stated goals and objectives;  (2) identifying specific users and other beneficiaries of IT projects and  systems; (3) providing adequate resources to ensure that projects and  systems support the organization’s business needs and meet users’ needs;  and (4) periodically evaluating the alignment of IT projects and systems  with the organization’s strategic goals and objectives. (The complete list of  key practices is provided in table 5.)
The department has in place all of the key practices for meeting business  needs. Specifically, HHS has policy and procedures that call for business  needs to be identified in the business case or the portfolio management  tool’s Select forms for both proposed and ongoing enterprisewide and  component agency IT projects. Resources devoted to ensuring that IT  projects and systems support the organization’s business needs and meet  users’ needs include the Business Case Quality Review Team, the Critical  Partners, the portfolio management tool, and detailed procedures and  associated templates for developing business cases. HHS’s specific  business mission, with stated goals and objectives, is defined in the HHS  Strategic Plan for fiscal years 2004 through 2009.
Further, HHS defines and documents business needs for both proposed and  ongoing enterprisewide and component agency IT projects, and identifies  users and other beneficiaries during its selection activities. In addition,  according to HHS IT officials, end users participate in project management  throughout the IT project’s life cycle. For the four projects we reviewed, we  verified that business needs and specific users and other beneficiaries were  identified and documented in the business case or in the Select forms  within HHS’s portfolio management tool. In addition, end users are  involved in project management throughout the life cycle of the  enterprisewide investments. For example, users of HHS’s Public Key  Infrastructure and Enterprise Architecture initiatives participate in project  management through integrated project teams, which meet approximately  once a month and are comprised of representatives from the component  agencies. Because the department has executed all of the key practices  associated with identifying business needs, it has increased confidence that  its IT projects will meet both business needs and users’ needs.
Table 5 shows the rating for each key practice required to meet business  needs and summarizes the evidence that supports these ratings.
Selecting new IT proposals and reselecting ongoing investments require a  well-defined and disciplined process to provide the agency’s investment  boards, business units, and developers with a common understanding of  the process and the cost, benefit, schedule, and risk criteria that will be  used both to select new projects and to reselect ongoing projects for  continued funding. According to the ITIM, this critical process requires,  among other things, (1) making funding decisions for new proposals  according to an established process; (2) providing adequate resources for  investment selection activities; (3) using a defined selection process to  select new investments and reselect ongoing investments; (4) establishing  criteria for analyzing, prioritizing, and selecting new IT investments and for reselecting ongoing investments; and (5) creating a process for ensuring  that the criteria change as organizational objectives change. (The complete  list of key practices is provided in table 6.)
HHS has executed 7 of the 10 key practices associated with selecting an  investment. For example, resources devoted to selection activities include  the Critical Partners, Business Case Quality Review Team, and portfolio  management tool, which contains several forms for selecting IT projects  and systems. HHS also has detailed procedures for using its portfolio  management tool and developing business cases. The criteria for analyzing,  prioritizing, selecting and reselecting new and ongoing investments  address the President’s Management Agenda, HHS strategic goals, and IT  strategic goals, value, and risk. They are incorporated into the department’s  portfolio management tool and are reviewed by the investment review  board and adjusted within the tool annually at the beginning of each budget  cycle to reflect organizational objectives. This year, HHS added additional  criteria—a quality score.
HHS uses its annual budget formulation process to select both  enterprisewide and component agency proposed and ongoing IT  investments. We verified that the four projects we reviewed were  reselected by the department using the annual budget formulation process.
Although HHS has the above strengths, the department has not executed  any of the practices associated with documenting policies and procedures.  Specifically, HHS has not fully documented its process for selecting new IT  proposals and reselecting ongoing IT investments. Although a number of  documents address investment selection, they are not linked to provide  decision makers with a clear understanding of the selection and reselection  processes. In addition, they do not define the roles and responsibilities for  all key players involved in these processes. Moreover, although the HHS  Office of the CIO works directly with the department’s Office of the Budget,  HHS does not have policies and procedures documenting the integration of  funding with the process of selecting and reselecting investments. Until the  department fully documents policies and procedures for selecting new IT  proposals and reselecting ongoing IT investments, the department will not  be adequately certain that it is consistently and objectively selecting and  reselecting investments that best meet the needs and priorities of the  department.
Table 6 shows the rating for each key practice required to select an  investment and summarizes the evidence that supports these ratings.
An organization should effectively oversee its IT projects throughout all  phases of their life cycles. Its investment board should observe each  project’s performance and progress toward predefined cost and schedule  expectations as well as each project’s anticipated benefits and risk  exposure. This does not mean that a departmental board, such as the  ITIRB, should micromanage each project to provide effective oversight;  rather it means that the departmental board should be actively involved in  all IT investments and proposals that are high cost or high risk or have  significant scope and duration and at a minimum, should, have a  mechanism for maintaining visibility of other investments. The board  should also employ early warning systems that enable it to take corrective  actions at the first sign of cost, schedule, and performance slippages.  According to the ITIM, effective project oversight requires, among other  things, (1) having written policies and procedures for management  oversight; (2) developing and maintaining an approved management plan  for each IT project; (3) making up-to-date cost and schedule data for each  project available to the oversight boards; (4) having regular reviews by  each investment board of each project’s performance against stated  expectations; and (5) ensuring that corrective actions for each  underperforming project are documented, agreed to, implemented, and  tracked until the desired outcome is achieved. (The complete list of key  practices is provided in table 7.)
The department has not executed any of the seven key practices associated  with effective project oversight, primarily because of its limited role in  overseeing component agency IT investments. Specifically, while the  department has documented standard operating procedures and  instructional memorandums for oversight of enterprisewide IT  investments, they are not comprehensive in that they do not specify the  board’s responsibilities for investment oversight; procedural rules for the  ITIRB operations and decision making during project oversight; or policies  and procedures for overseeing component agency IT investments.
The HHS ITIRB is currently performing regular reviews of enterprisewide  IT projects and systems against stated expectations through reports that  are available to decision makers on the HHS Intranet. However, the  department is not regularly reviewing component agency investments that  are high risk, crosscutting, and require review by the Office of Management  and Budget, although their policy calls for it. The board also does not have  a mechanism for maintaining visibility of other component agency  investments.
The department delegates oversight of these investments to the component  agencies but believes it is nonetheless effectively overseeing component  agency investments through (1) reviews of these investments as part of the  annual Critical Partner and Business Case Quality reviews performed  during the annual selection process and the use of (2) earned value  management data. Although the annual reviews may provide insight into  the status of investments, they are not frequent enough to allow for timely  identification of problems. Moreover, while HHS officials told us that staff  responsible for collecting earned value management data on component  agency investments share significant concerns about the data with the  ITIRB, they did not have formal documentation clearly supporting this  issue. In addition, formal procedures for elevating issues to the board have  not been developed. In the absence of effective board oversight, HHS  executives will not have the information they need to determine whether  component agency projects are being developed on schedule and within  budget. In addition, the department will run the risk that underperforming  component agency projects will not be identified in time for corrective  actions to be taken.
We verified that HHS provided oversight for the two enterprisewide  investments, but had delegated oversight activities for the two component  agency investments we reviewed.
Table 7 shows the rating for each key practice required to provide  investment oversight and summarizes the evidence that supports these  ratings.
To make good IT investment decisions, an organization must be able to  acquire pertinent information about each investment and store that  information in a retrievable format. During this critical process, an  organization identifies its IT assets and creates a comprehensive repository  of investment information. This repository provides information to  investment decision makers to help them evaluate the potential impacts  and opportunities created by proposed or continuing investments. It can  provide insights into major IT cost and management drivers and trends.  The repository can take many forms and need not be centrally located, but  the collection method should, at a minimum, identify each IT investment  and its associated components. This critical process may be satisfied by the  information contained in the organization’s current enterprise architecture,  augmented by additional information—such as financial information and  information on risk and benefits—that the investment board may require to  ensure that informed decisions are being made. According to the ITIM,  effectively managing this repository requires, among other things,  (1) developing written policies and procedures for identifying and  collecting the information; (2) assigning responsibilities for ensuring that  the information being collected meets the needs of the investment  management process; (3) identifying IT projects and systems and collecting  relevant information to support decisions about them; and (4) making the  information easily accessible to decision makers and others. (The complete  list of key practices is provided in table 8.)
HHS has executed 5 of the 6 key practices for capturing investment  information. For example, the department has several documents that  define the policies and procedures for identifying and collecting investment  information in its repositories and also assign responsibility to the HHS  CIO for ensuring that the information collected during project and systems  identification meets the needs of the investment management process.  HHS maintains a portfolio management tool, which serves as the primary  repository for identifying and collecting information about both  department and component agency IT projects and systems. The  department’s portfolio management tool is easily accessible to decision  makers at both the department and component level and the Office of the  CIO has provided decision makers with various training manuals and  guidance memorandums. In addition, the department also identifies and  collects information about enterprisewide IT investments using its  Intranet. Further, the department recently began collecting earned value  information through spreadsheets on major HHS IT investments that  compares planned and actual cost and schedule information. These  repositories are easily accessible to the board members.
The key practice HHS has not executed has to do with the captured  investment information not yet being used by the HHS ITIRB to fully  support decisions about component agency investments. For example, the  earned value investment data received from each component agency has  not been used by the HHS ITIRB for control and evaluation decisions.  According to agency officials, the department has recently begun  monitoring the earned value data to identify investments that report cost  and schedule variances and these officials acknowledge a need to formalize  the process for doing so. Until HHS’s decision makers use the information  in the repository to fully support the investment management process, it  will be unable to effectively evaluate the impacts and opportunities created  by proposed or continuing investments.
Table 8 shows the rating for each key practice required to capture  investment information and summarizes the evidence that supports these  ratings.
Once an agency has attained Stage 2 maturity, it needs to implement critical  processes for managing its investments as a portfolio (Stage 3). An IT  investment portfolio is an integrated, agencywide collection of investments  that are assessed and managed collectively based on common criteria.  Managing investments as a portfolio is a conscious, continuous, and  proactive approach to allocating limited resources among an organization’s  competing initiatives in light of the relative benefits expected from these  investments. Taking an agencywide perspective enables an organization to  consider its investments comprehensively, so that collectively the  investments optimally address the organization’s missions, strategic goals,  and objectives. Managing IT investments as a portfolio also allows an  organization to determine its priorities and make decisions about which  projects to fund and continue to fund based on analyses of the relative  organizational value and risks of all projects, including projects that are  proposed, under development, and in operation. Although investments may  initially be organized into subordinate portfolios—based on, for example,  business lines or life cycle stages—and managed by subordinate  investment boards; they should ultimately be aggregated into this  enterprise-level portfolio.
According to the ITIM framework, Stage 3 maturity includes (1) defining  the portfolio criteria, (2) creating the portfolio, (3) evaluating the portfolio,  and (4) conducting postimplementation reviews. Table 9 summarizes the  purpose of each critical process in Stage 3.
HHS has executed 8 of the 27 key practices required by Stage 3. For  example, the department’s core IT portfolio selection criteria, including  cost, benefit, schedule, and risk are approved by the HHS ITIRB. In  addition, the investment board examines the mix of new and ongoing  investments and their respective data and analyses to select investments to  fund. However, many key practices still need to be executed before HHS  can effectively manage its IT investments from a portfolio perspective. For  example, HHS has not addressed any of the key practices related to  evaluating the portfolio or conducting PIRs. Until HHS fully implements the  critical processes associated with managing its investments as a complete  portfolio, it will not have the data it needs to make informed decisions  about competing investments.
Table 10 summarizes the status of HHS’s critical processes for Stage 3,  showing how many associated key practices it has executed.
To manage IT investments effectively, an organization needs to establish  rules or “portfolio selection criteria” for determining how to allocate scarce  funding to existing and proposed investments. Thus, developing an IT  investment portfolio requires defining appropriate cost, benefit, schedule,  and risk criteria with which to evaluate individual investments in the  context of all other investments. To ensure that the organization’s strategic  goals, objectives, and mission will be satisfied by its investments, the  criteria should have an enterprisewide perspective. Further, if an  organization’s mission or business needs and strategies change, criteria for  selecting investments should be reexamined and modified as appropriate.  Portfolio selection criteria should be disseminated throughout the  organization to ensure that decisions concerning investments are made in a  consistent manner and that this critical process is institutionalized. To  achieve this result, project management personnel and others should be  aware of the criteria and address the criteria in funding submissions for  projects. Resources required for this critical process typically include the  time and attention of executives involved in the process, adequate funding,  and supporting tools. (The complete list of key practices is provided in  table 11.)
The department has executed 5 of the 7 key practices for this critical  process. For example, responsibility has been assigned to the HHS Lead  Capital Planner for managing the development and modification of the IT  portfolio selection criteria, and adequate resources have been committed  for portfolio selection activities, including the Critical Partners, portfolio  management tool project manager, and the Office of the CIO staff.  Moreover, the project management personal and other stakeholders are  aware of the portfolio selection criteria that are embedded into the  department’s portfolio management tool and also contained within policies  and procedures.
Finally, the HHS ITIRB approves the core IT selection criteria, including  cost, benefit, schedule, and risk criteria, based on the organization’s  mission, goals, strategies, and priorities. Beginning in fiscal year 2004, HHS  began scoring and ranking approximately 80 percent of its IT investments  against alignment, value, and risk criteria in order to determine a priority  score, which is the sum of alignment, value, and risk criteria scores,  weighted for relative importance. Similarly, for the fiscal year 2007 budget  formulation process, HHS began collecting investment information on the  business case quality, Critical Partner reviews, and cost and schedule  variance to determine a quality score, which is the sum of the business case  quality, Critical Partner reviews, and cost and schedule variance scores,  weighted for relative importance. The HHS ITIRB evaluates and annually  adjusts its portfolio selection criteria within the portfolio management tool.
Despite these important steps in defining portfolio selection criteria,  weaknesses remain. The department has not developed policies or  procedures for modifying the portfolio selection criteria to reflect changes  to HHS mission, goals, strategies, and priorities. In addition, the HHS ITIRB  began reviewing the IT portfolio selection criteria this year. However, the  process for modifying portfolio selection criteria is not institutionalized  because the process to do so was only used once and there are no  documented policies and procedures to ensure that it will be used again.  Until HHS defines and implements the practices required for defining the  portfolio criteria definition, it will not have the tool it needs to select  investments that support its mission, organizational strategies, and  business priorities.
Table 11 shows the rating for each key practice required to define portfolio  selection criteria and summarizes the evidence that supports these ratings.
At Stage 3, organizations create a portfolio of IT investments to ensure that  IT investments are analyzed according to the organization’s portfolio  selection criteria and to ensure that an optimal IT investment portfolio with  manageable risks and returns is selected and funded. According to ITIM,  creating the portfolio requires organizations to, among other things,  document policies and procedures for analyzing, selecting, and maintaining  the portfolio; provide adequate resources, including people, funding, and  tools for creating the portfolio; and capture the information used to select,  control, and evaluate the portfolio and maintain it for future reference. In  creating the portfolio, the investment board must also (1) examine the mix  of new and ongoing investments, and their respective data and analyses  and select investments for funding and (2) approve or modify the  performance expectations for the IT investments they have selected. (The  complete list of key practices is provided in table 12.)
HHS has executed 3 of the 7 key practices associated with creating the  portfolio. Beginning in fiscal year 2004, the department began to create a  portfolio by using its portfolio management tool to collect cost, benefit,  schedule, risk, strategic alignment, and enterprise architecture information  on investments accounting for 80 percent of the dollar value of the HHS IT  investment portfolio. Each component agency’s IT portfolio is displayed in  priority order along with where each investment falls within the overall IT  portfolio. Further, according to HHS IT officials, the agency has adequate  resources for portfolio selection activities, including the Critical Partners,  the portfolio management tool project manager, and the Office of the CIO  staff. These officials also stated that HHS ITIRB members are also  knowledgeable about the process of creating a portfolio.
Nevertheless, HHS has a number of significant weaknesses in the way it  creates a portfolio. First, it does not have policies and procedures that  sufficiently address this critical process. Although the department has  policies and procedures for creating IT portfolio selection criteria, they  lack policies and procedures for using these criteria to analyze, select, and  maintain the investment portfolio. Second, even though the HHS ITIRB has  quarterly reviews to compare project and system performance with  expectations for enterprisewide IT investments, the board is not provided  with information comparing the performance of component agency  investments against expectations. In addition, the board approves or  modifies the performance expectations for the enterprisewide IT  investments it has selected, but does not regularly approve or modify the  performance expectations for component agency IT investments or ensure  that this is done. Moreover, as previously mentioned, investment  information has not been used to fully support control and evaluate  decisions for component agency investments. Unless HHS defines and  implements the practices for creating a comprehensive portfolio of IT  investments, it will not be able to determine whether it has selected the mix  of investments that best meets its needs considering resource and funding  constraints.
Table 12 shows the rating for each key practice required to create a  portfolio and summarizes the evidence that supports these ratings.
This critical process builds upon the Stage 2 critical process, Providing  Investment Oversight, by adding the elements of portfolio performance to  an organization’s investment control capacity. Compared with less mature  organizations, Stage 3 organizations will have the foundation they need to  control the risks faced by each investment and to deliver benefits that are  linked to mission performance. In addition, a Stage 3 organization will have  the benefit of performance data generated by Stage 2 processes. Executive- level oversight of risk management outcomes and incremental benefit  accumulation provides the organization with increased assurance that each  IT investment will achieve the desired results. (The complete list of key  practices is provided in table 13.)
HHS has not executed any of the seven key practices for evaluating a  portfolio. It has yet to develop policies and procedures that address  performance oversight from a portfolio perspective. Moreover, while the  department annually reviews its portfolio as part of its selection process, it  does not evaluate the investment portfolio on a continuing basis to assess  its performance. Finally, the results of Providing Investment Oversight  reviews from Stage 2 are important to this critical process. However, as  previously mentioned, while the HHS ITIRB has oversight of  enterprisewide investments, it does not regularly review a defined set of  component agencies’ investments and maintain visibility of other  investments. Although the department’s portfolio management tool has the  ability to summarize performance metrics for each investment and quickly  understand the status of each investment and any potential emerging  problem area, the tool is currently only being used on an ad hoc basis to  make portfolio oversight decisions. Defining and implementing processes  to evaluate the performance of its entire portfolio would provide HHS with  greater assurance that it is controlling the risks and achieving the benefits  associated with the mix of investments it has selected.
Table 13 shows the rating for each key practice required to evaluate the  portfolio and summarizes the evidence that supports these ratings.
The purpose of a PIR is to evaluate an investment after it has completed  development (that is, after its transition from the implementation phase to  the operations and maintenance phase) in order to validate actual  investment results. This review is conducted to (1) examine differences  between estimated and actual investment costs and benefits and possible  ramifications for unplanned funding needs in the future and (2) extract  “lessons learned” about the investment selection and control processes  that can be used as the basis for management improvements. Similarly,  PIRs should be conducted for investment projects that were terminated  before completion, to readily identify potential management and process  improvements. (The complete list of key practices is provided in table 14.)
HHS has not executed the six key practices for conducting PIRs. Although  its policy calls for postimplementation reviews of IT investments that have  recently completed implementation of the entire investment or a significant  phase of the investment, the department does not have specific procedures  for conducting such reviews, including specifying who conducts and  participates in the PIR, what information is presented in a PIR, or how  results are to be disseminated to decision makers. To date, HHS has  conducted closeout reviews of two enterprisewide investments following  their implementation; however, while these reports do cover investment  cost expectations, they cannot be considered PIRs because the reports do  not address general conclusions, lessons learned, or schedule deviations.  Unless PIRs are conducted on a regular basis, HHS will not be able to  effectively evaluate the results of its IT investments to determine whether  continuation, modification, or termination of an IT investment would be  necessary in order to meet stated HHS mission objectives.
Table 14 shows the rating for each key practice required to conduct PIRs  and summarizes the evidence that supports these ratings.
The ability of a department-level CIO to effectively oversee IT investment  management processes throughout the agency depends on the existence of  appropriate management structures with adequate authorities and  sufficient guidance. Under the Clinger-Cohen Act of 1996, the CIO of each  agency is responsible for effectively managing all of the agency’s IT  resources. To comply with the act, HHS designates its CIO to be  responsible for ensuring that the component agencies are defining and  implementing effective investment management processes that are  appropriately aligned with the department’s processes.
Although each component agency has staff responsible for gathering,  maintaining, and analyzing IT investment information, the HHS Office of  the CIO has the responsibility to define and implement overall HHS IT  investment management practices, and monitor component agency  investment management practices to ensure a cohesive departmental  process and the capability exists to carry out the process. In accordance  with this, the department’s investment management policies and guidelines  state that the component agencies are to establish and manage investment  management processes and governance structures that are aligned with the  department’s policies and procedures. However, as mentioned in previous  sections, the department’s investment management policies and  procedures have several weaknesses. For example, HHS does not have a  set of documented procedures that provide decision makers with a clear  understanding of the selection and reselection process.
Moreover, HHS currently has no structured mechanism in place to ensure  that the component agencies are adhering to the department’s policies and  procedures. According to HHS officials, the CIO has the authority to audit a  component agencies IT investment management process. However, they  were unable to provide us evidence of having performed any such audits.  These officials also stated that the department’s portfolio management tool  is another method that will enable HHS to oversee component-level  investment management processes. However, since not all component  agencies are using the portfolio management tool to individually make  select, control, and evaluate decisions, its usefulness in this regard is  limited. Until the department develops a mechanism for ensuring that  component agencies define and implement investment management  processes that align with those of the department, it is running the risk that  effective processes are being institutionalized at both the department and  the component agency level. In addition, the department will be unable to  ensure that it is optimizing its investments in IT and effectively assessing  and managing the risks of these investments.
section_title: HHS Does Not Have a Plan to Coordinate and Guide Improvement Efforts
paragraphs: HHS has initiated several efforts to improve its investment management  process. Specifically, it has drafted a revised investment management guide  that addresses the weaknesses with current guidance that we identify in  this report. In addition, in February 2005, HHS incorporated capabilities  into its portfolio management tool to enhance performance of control and  evaluate functions. Specifically, the tool now has the capabilities to  produce (1) scorecards to provide data for each investment in a portfolio,  allowing cross investment comparisons on data elements collected; (2)  investor maps to provide a graphical depiction of a portfolio in terms of up  to six data categories, with the ability to show target and actual values; and  (3) a workbook module to track the identification and resolution of issues  that may arise regarding the management of an investment or set of  investments.
Although HHS has initiated these efforts, they only fully address 2 of the 14  Stage 2 key practices the department did not execute.
The draft investment management guidance, when finalized, will  address weaknesses associated with one of the key practices for  instituting the investment board by reflecting the current management  process, including information on the roles of key working groups  involved in the organization’s IT investment processes, and identifying  the manner in which investments board’s processes are to be  coordinated with other key organizational plans and processes. The  guidance will also address the integration of the funding and selection  processes, a key practice the department has not executed that is  associated with selecting an investment.
The enhanced portfolio management tool capabilities will enhance the  department’s ability to oversee investments’ performance and position  the board to perform portfolio evaluation activities, but they will not  fully address any of the weaknesses we identify.
HHS has not coordinated these and additional efforts that would address  the weaknesses we identify in this report in a comprehensive plan that (1)  specifies measurable goals, objectives, and milestones; (2) specifies  needed resources; (3) assigns clear responsibility and accountability for  accomplishing tasks; and (4) is approved by senior management. We have  previously reported that such a plan is instrumental in helping agencies  coordinate and guide improvement efforts. Until HHS develops a plan that  would allow for the systematic prioritization, sequencing, and evaluation of  improvement efforts, the agency risks not being able to effectively  establish the mature investment management processes that result in  greater certainty about the outcomes of future IT investments.
section_title: Conclusions
paragraphs: Because of the attention that has been given to investment management,  HHS has established several of the practices needed to effectively manage  its investments. These practices have strengthened the department’s basic  capabilities for selecting and controlling projects and begun to equip the  department with the capabilities it needs to make informed decisions about  competing investments. However, several significant weaknesses remain in  the foundational practices needed to manage individual investments, the  portfolio-level investments needed to manage investments as a collection,  and in the level of guidance and oversight provided to component agency  investment management processes. These weaknesses hamper the  department’s ability to ensure that it is managing the mix of investments  that will maximize returns to the organization, taking into account the  appropriate level of risk.
Critical to HHS’s success, going forward will be the development of an  implementation plan that (1) is based on an assessment of strengths and  weaknesses; (2) specifies measurable goals, objectives, and milestones; (3)  specifies needed resources; (4) assigns clear responsibility and  accountability for accomplishing tasks; and (5) is approved by senior  management. Although the department has initiated improvement efforts,  it has not developed a comprehensive plan to guide these and other efforts  needed to improve its investment management process. Without such a  plan and procedures for implementing it, it is unlikely that the department  will effectively establish mature investment management capability. As a  result, HHS will continue to be challenged in its ability to make informed  and prudent investment decisions in managing its annual multibillion-dollar  IT budget.
section_title: Recommendations for Executive Action
paragraphs: To strengthen HHS’s investment management capability and address the  weaknesses discussed in this report, we recommend that the Secretary of  the Department of Health and Human Services direct the Chief Information  Officer to develop and implement a plan for improving the department’s IT  investment management processes. The plan should address the  weaknesses described in this report, beginning with those we identified in  our Stage 2 analysis and continuing with those we identified in our Stage 3  analysis. The plan should, at a minimum, provide for accomplishing the  following:  Develop comprehensive guidance and additional supporting guidance  that defines and describes the complete investment management  process, unifies existing processes enterprisewide, reflects changes in  processes as they occur; define the operations and decision-making  processes of the HHS investment review board and other management  entities, such as the component agencies, involved in managing IT  investments.
Ensure that HHS’s investment review board’s membership includes  business representation of its component agencies as it begins to  execute its full range of responsibilities.
Develop well-defined and disciplined written procedures that outline  the process for selecting new IT proposals, reselecting ongoing IT  investments, and integrating funding with the process of selecting an  investment.
Establish a process for the investment board to regularly review and  track the performance of a defined set of component agency IT systems  against expectations, and take corrective actions when these  expectations are not being met; and establish a mechanism for  maintaining visibility into other investments.
Develop and implement policies and procedures for modifying IT  portfolio selection criteria.
Develop policies and procedures for using the portfolio selection  criteria to create its portfolio.
Develop, review, and modify criteria for assessing portfolio performance  at regular intervals to reflect current performance expectations.
Define and implement processes for carrying out PIRs for all IT  investments.
We also recommend that the HHS Secretary direct the CIO to ensure that  the plan draws together ongoing efforts and additional efforts that are  needed to address the weaknesses identified in this report. The plan should  also (1) specify measurable goals, objectives, and milestones; (2) specify  needed resources; (3) assign clear responsibility and accountability for  accomplishing tasks; and (4) be approved by senior management.
Finally, to improve the department oversight of its component agency  investment management process, we are recommending that the HHS  Secretary direct the HHS CIO to establish a mechanism for ensuring  component agencies define and implement investment management  processes that are aligned with those of the department.
section_title: Agency Comments
paragraphs: The Department of Health and Human Services’s Inspector General  provided written comments on a draft of this report (reprinted in app. II).  In these comments, HHS generally agreed with our findings and  recommendations and stated that the report represented a fair assessment  of the department’s progress in IT investment management. The  department added that it will leverage the report in its efforts to improve its  investment management processes.
HHS expressed differing perspectives on the inclusion of component  agency business representation on the investment review board and the  performance of postimplementation reviews. Specifically, regarding  business representation on the board, the department commented that it  used a hierarchy of investment reviews (with the first review occurring at  the component agency) combined with ITIRB members representing  mission support areas, such as Finance, Acquisition, and Human  Resources, to provide a structure for making the business decisions  regarding the department’s investments. We disagree with the department  that this arrangement provides an adequate structure for managing the  department’s investments. Because allocating resources among major IT  investments may require fundamental trade-offs among a multitude of  business objectives, portfolio management decisions are essentially  business decisions, and therefore require sufficient business  representation on the board. CIOs and executives responsible for mission- support functions do not constitute sufficient business representation  because, by virtue of their responsibilities, they are not in the best position  to make business decisions. Portfolio management decisions are better  made by executives with business line decision-making authority.
Regarding PIRs, HHS commented that it was currently informally  performing them by conducting closeout reviews of recently implemented  investments and annual reviews of systems in operations and maintenance.  PIRs are conducted to determine whether cost, benefit, schedule, and risk  expectations that were set for investments were achieved and develop  lessons learned about the investment selection and control processes that  can be used as the basis for management improvements. However, neither  the closeout reviews, nor the reviews of systems in operations and  maintenance, are addressing all these elements. Specifically, as we stated in  our report, the closeout reviews do not address schedule deviations,  determine whether the benefits were achieved, or identify lessons learned.  In addition, the reviews of projects in operations and maintenance do not  capture the benefits realized or identify lessons learned.
Commenting on departmental-level oversight of component agency  investments, HHS stated that it agrees with our recommendation to  improve its oversight of component agency investments. It stated that it  would use a number of mechanisms to do this, including performing audits  to ensure alignment of component agency’s processes with those of the  department, using earned value management data to identify potential  performance problems with most investments, and directly reviewing  investments determined to be of high priority. We agree with HHS that  these steps would help address some of the weaknesses in project  oversight that we identify in this report.
As agreed with your office, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days from the  date of this report. At that time, we will send copies to other interested  congressional committees, the Secretary of Health and Human Services,  and other interested parties. We will also make copies available to others  upon request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
Should you or your offices have questions on matters discussed in this  report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact  points for our Offices of Congressional Relations and Public Affairs may be  found on the last page of this report. GAO staff who made major  contributions to this report are listed in appendix III.
section_title: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to (1) assess the Department of Health  and Human Services’s capabilities for managing its IT investments and (2)  determine any plans HHS might have for improving those capabilities.
To address our first objective, we reviewed the results of the department’s  self-assessment of Stages 2 and 3 practices using our ITIM framework and  validated and updated the results of the self-assessment through document  reviews and interviews with officials. We reviewed written policies,  procedures, and guidance and other documentation providing evidence of  executed practices, including HHS’s Capital Planning and Investment  Control Policy and Guidelines, standard operating procedures, portfolio  management tool training manuals, and various instructional  memorandums. We also reviewed the HHS ITIRB meeting materials,  including quarterly status reports, meeting minutes, and records of  decisions. We did not assess progress in establishing the capabilities found  in Stages 4 and 5 because the department acknowledged that it had not  executed any of the key practices in higher maturity stages. In addition, we  conducted interviews with officials from the Office of the CIO, whose main  responsibility is to oversee and ensure that HHS’s IT investment  management process is implemented and followed to determine the level  of oversight and guidance the department is providing to its component  agencies. We also interviewed the Centers for Medicare & Medicaid’s  Director for Investment Tracking and Assessment to determine the level of  investment management guidance and oversight that is provided by the  department.
As part of our analysis, we selected two HHS enterprisewide and two  component agency IT projects as case studies to verify that the critical  processes and key practices were being applied. The projects selected  (1) are recognized as major systems, (2) were in different life cycle phases,  (3) represent a mix of headquarters and component agency investments,  (4) support different functional areas, and (5) required different levels of  funding. The four projects are described below:   HHS Public Key Infrastructure—This project supports digital signatures  and other public key-enabled security services; it is intended to be the  underlying architecture to support secure transmissions of electronic  communication, such as encrypted email, by linking a digital key to a  specific person, and issues and manages digital certificates. The intent  of the project is to provide an identity proofing process that is both fast  and certificate authority neutral. It is an agencywide strategic initiative  that provides security services. The project is a major enterprisewide  investment and is in the operations and maintenance phase. The project  has a planned completion date of July 2011 and is estimated to spend  $7.7 million for fiscal year 2006.
HHS Enterprise Architecture Initiative—This initiative is to provide the  overall framework for planning and managing the technology-supported  information assets of HHS and give the department the ability to identify  data and process redundancies and inefficiencies in its information  systems. The program’s objectives focus on development of operational  policies and support that enable identification, analysis and ongoing  management of the business, and information and related technology  architectures. It is to provide leadership, direction, and support to HHS’s  component agencies in planning and implementing information systems  to support required business processes. As of fiscal year 2005, the  initiative is a major enterprisewide program investment and is estimated  to spend $15.0 million for fiscal year 2006.
National Institutes of Health’s Electronic Research Administration— This initiative is the National Institutes of Health’s infrastructure for  conducting interactive electronic transactions for the receipt, review,  monitoring, and administration of grant awards to biomedical  investigators worldwide. It is to provide the technology capabilities for  the agency to efficiently and effectively perform grants administration  functions. The system is to provide end-to-end support of the grants  administration process, including receipt of applications, review and  selection of grantees, financial and progress reporting, issuance of final  reports and grant dole-out, invention reporting, and interface with  accounting systems. It is a major component agency investment and is  expected to have a useful life of 13 years. The project is estimated to  spend $42.1 million for fiscal year 2006.
Food and Drug Administration’s Mission Accomplishment and  Regulatory Compliance Services—This program is a comprehensive  redesign and reengineering of core mission-critical systems at the  agency, including the Field Accomplishments and Compliance Tracking  System and the Operation and Administration Support System. The first  of these systems is to support the investigation, tracking of compliance,  and laboratory operations related to domestic operations under the  agency’s purview; the second is to primarily support the review and  decision-making process of products imported into the United States.  Both are legacy systems that execute on client-server platforms; while  currently viable, the current systems cannot address many of the  business needs due to the exponential growth in functionality on a rigid  platform that was not designed to support the extent of change that has  been required. The Mission Accomplishment and Regulatory  Compliance Services is a major component agency investment and is  expected to move to production in September 2007 and have a useful  life of 10 years. The project is estimated to spend $10.2 million for fiscal  year 2006.
For these projects, we reviewed project management documentation, such  as business cases, status reports, and meeting minutes. We also  interviewed officials from the Office of the CIO for the two component  agency investments and the project managers for the two HHS  enterprisewide projects.
We compared the evidence collected from our document reviews and  interviews to the key practices in ITIM. We rated the key practices as  “executed” on the basis of whether the agency demonstrated (by providing  evidence of performance) that it had met the criteria of the key practice. A  key practice was rated as “not executed” when we found insufficient  evidence of a practice during the review or when we determined that there  were significant weaknesses in HHS’s execution of the key practice. In  addition, HHS was provided the opportunity to produce evidence for key  practices rated as “not executed.”
To address our second objective, we obtained and evaluated documents  showing what management actions had been taken and what initiatives had  been planned by the agency. This documentation included the Policy  Advisory Board charter, draft investment management policies and  procedures, as well as procedures and guidance for control and evaluate  functionalities within HHS’s portfolio management tool. We also  interviewed officials from the Office of the CIO to determine efforts  undertaken to improve IT investment management processes.
We conducted our work at HHS headquarters in Washington, D.C., from  January through September 2005, in accordance with generally accepted  government auditing standards.
section_title: Comments from the Department of Health and Human Services
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the person named above, Neil Doherty, Joanne Fiorino,  Sabine Paul, Nik Rapelje, Niti Tandon, and Amos Tevelow made key  contributions to this report.