Information Technology: INS Needs to Strengthen Its Investment Management Capability
Released: 2000-12-29T00:00:00

Letter

Introduction

The mission of INS, an agency of the Department of Justice, is to administer and enforce the immigration laws of the United States. To accomplish this, INS is organized into three core business areas— enforcement, immigration services, and corporate services. Enforcement includes, among other things, conducting inspections of travelers entering the United States as they arrive at more than 300 land, sea, and air ports of entry; detecting and preventing the smuggling and illegal entry of aliens; and identifying and removing persons who have no lawful immigration status in the United States. Immigration services, which involve regulating permanent and temporary immigration to the United States, include granting legal permanent residence status, nonimmigrant status (e.g., tourists and students), and naturalization. Corporate services include records management, financial management, personnel management, and inventory management support for INS activities.

INS' IT assets play a significant role in (1) receiving and processing naturalization and other benefit applications, (2) processing immigrants and nonimmigrants entering and leaving the United States, and (3) identifying and removing people who have no lawful immigration status in the United States. For example, the Computer-Linked Application Information Management System (CLAIMS 4) is a centralized case management tracking system, that offers support for a variety of tasks associated with processing and adjudicating naturalization benefits. In addition, the Deportable Alien Control System (DACS) automates many of the functions associated with tracking the location and status of illegal aliens in removal proceedings, including detention status.

INS' Current IT Investment Efforts

INS has multiple efforts underway to develop and acquire new information systems and to maintain existing ones. According to INS, in fiscal year 2000, it obligated about $327 million on IT activities, including about $94 million for new development and the remaining amount, which includes enhancing existing systems, for operations and maintenance. For example, INS obligated $14.5 million in fiscal year 2000 to continue development of CLAIMS 4, which supports the processing of applications and petitions for immigrant benefits and is intended to fully replace CLAIMS 3. In addition, INS obligated about $18 million in fiscal year 2000 to further deploy its Integrated Surveillance Intelligence System (ISIS), which includes the deployment of intelligent computer aided detection systems, unattended ground sensors, and fixed cameras along the northern and southern borders to provide around-the-clock visual coverage of the border. For fiscal year 2001, INS plans to spend about $226 million on IT for operations and maintenance activities. INS funds most of its IT efforts with operation and maintenance funds and currently is developing or maintaining 74 information systems.

Recent Reviews Have Identified IT Project Management Weaknesses

Recent reviews have identified several weaknesses in INS' management of its IT projects. For example, in August 1998, the Logistics Management Institute (LMI) reported that INS' Office of Information Resources Management (OIRM) (1) did not maintain accurate cost estimates for the complete life cycle of projects and (2) did not track and manage projects to a set of cost, schedule, technical, and benefit baselines. Further, LMI noted that while INS' System Development Life Cycle (SDLC) manual provides a good model for systems development projects, OIRM did not consistently follow it, often bypassing key SDLC phases.

Similarly, in July 1999, the Justice Inspector General (IG) reported that (1) estimated completion dates for some INS IT projects had been delayed without explanation for the delays, (2) project costs continued to spiral upward with no justification for how funds are spent, and (3) projects were nearing completion with no assurance that they would meet performance and functional requirements.

Recognizing the need to address these weaknesses, INS established an Operational Assessment Team to analyze reported weaknesses and recommend specific actions to address them. The Operational Assessment Team validated the deficiencies identified in the LMI and Justice IG reports and identified additional ones. For example, the team found that system requirements were not consistently collected, recorded, documented, tracked, and controlled. To illustrate, of 105 projects reviewed by the team, fewer than 50 percent had documented requirements and most of the requirements that had been documented were not current.

Further, in August 2000, we reported that INS did not have an enterprise architecture to guide the development and evolution of its information systems. An enterprise architecture is an institutional systems blueprint that defines in both business and technological terms the organization's current and target operating environments and provides a road map for moving from one to the other. It is required by the Clinger-Cohen Act and is a recognized practice of successful public and private sector organizations. INS had initiated some limited efforts to document its current architecture, but it had not yet begun developing a target architecture or a plan to move from the current to the target environment. Moreover, INS had not yet established the management structure and controls to develop the architecture. The absence of such an enterprise architecture increases the risk that the hundreds of millions of dollars INS spends each year on information systems will not be well integrated or compatible and will not effectively support mission needs and priorities.

Overview of INS' Current Approach to IT Investment Management

In 1997, INS established an investment review board (IRB). The IRB consists of four voting members—the Deputy Commissioner (Chair) and INS' three Executive Associate Commissioners—and advisory or supporting members, including the Director of the Budget Office and the Acting Associate Commissioner of the Office of Information Resources Management. In November 1998, INS also established the Executive Steering Committee (ESC) to support the IRB. The ESC comprises portfolio managers and advisory members, which analyze investment proposals and make recommendations on these proposals to the IRB.

The IRB has established a process for selecting new IT proposals. According to INS officials, new proposals are developed throughout the year as business needs are identified and are forwarded to the appropriate portfolio manager for review. After reviewing the proposal, the portfolio manager forwards it to the ESC for consideration for funding. The ESC examines the proposals submitted and determines the appropriate funding for each project. Once funding is determined, the ESC forwards the proposed funding levels to the IRB, which makes the final investment selections and budget formulation decisions. See figure 2 for INS' new proposal selection process.

As part of INS' annual budget execution process, the IRB considers the funding requests of ongoing and new projects. Project managers define requirements for their ongoing projects, which they submit to the responsible portfolio managers for review. After reviewing the requirements and funding requests, each portfolio manager submits them to the ESC for review and to the IRB for approval. The approved funding is submitted to the Budget Office for inclusion into its budget execution process. According to INS officials, new proposals are considered for funding only after ongoing projects have been funded.

Framework for Assessing Agencies' IT Investment Management

Several recent management reforms—including the revision to the Paperwork Reduction Act and the passage of the Clinger-Cohen Act of 1996, the Government Performance and Results Act of 1993, and the Chief Financial Officers Act of 1990—have introduced requirements emphasizing the need for federal agencies to improve their management processes for selecting and managing IT resources. In particular, the Clinger-Cohen Act requires that the head of each agency implement a process for maximizing the value of the agency's IT investments and for assessing and managing the risks of its acquisitions. A key goal of the Clinger-Cohen Act is that agencies have processes and information in place to help ensure that projects are being implemented at acceptable costs within reasonable and expected time frames and that they are contributing to tangible, observable improvements in mission performance.

We and the Office of Management and Budget (OMB) have developed guidance to assist federal agencies in managing IT investments. One such guide, Assessing Risks and Returns: A Guide for Evaluating Federal Agencies' IT Investment Decision-making, incorporates our analysis of the management practices of leading private and public sector organizations as well as the provisions of major federal legislation (e.g., Clinger-Cohen Act) and executive branch guidance that address investment decision-making. The guide provides a method for determining how well a federal agency is selecting and managing its IT resources and identifies specific areas where improvements can be made.

To enhance this guidance, we issued an Information Technology Investment Management (ITIM) maturity framework in May 2000. ITIM provides a common framework for assessing IT capital planning and investment management practices by describing the organizational processes, and their interrelationships that are the tenets of good investment management. ITIM is based on the best-practices work done as part of our ongoing research into the IT management practices of leading organizations.

ITIM is a hierarchical model comprising five maturity stages. These maturity stages represent steps toward achieving stable and mature investment management processes. As agencies advance through the model's stages, their capability to manage IT increases. Each stage builds upon the lower stages and enhances the organization's ability to manage its investments. With the exception of the first stage, each maturity stage is composed of critical processes that must be implemented and institutionalized for the organization to satisfy the requirements of that stage. These critical processes are further broken down into key practices that describe the types of activities that an agency should be engaged in to successfully implement each critical process. An organization that has these critical processes in place is in a better position to successfully invest in IT. (See figure 3 for the five stages and associated critical processes).

As established by the model, each critical process contains five core elements that indicate whether the implementation and institutionalization of a process can be effective and repeated. The five core elements are:

Purpose: This is the primary reason for engaging in the critical process and states the desired outcome for the critical process.

Organizational commitment: This comprises management actions that ensure that the critical process is established and will endure. Key practices typically involve establishing organizational policies and engaging senior management sponsorship.

Prerequisites: These are the conditions that must exist within an organization to successfully implement a critical process. This typically involves allocating resources, establishing organizational structures, and providing training.

Activities: These are the key practices necessary to implement a critical process. An activity occurs over time and has recognizable results. Key practices typically involve establishing procedures, performing and tracking the work, and taking corrective actions as necessary.

Evidence of performance: This comprises artifacts, documents, or other evidence that supports a contention that the key practices within a critical process have or are being implemented. This core element typically consists of the collection and verification of physical, documentary, or testimonial evidence and typically involves reviews by objective parties.

With the exception of the purpose core element, each of the other core elements contains key practices. The key practices are the attributes and activities that contribute most to the effective implementation and institutionalization of a critical process. (Figure 4 shows the relationship between the various ITIM components.)

Objectives, Scope, and Methodology

Our objectives were to determine whether (1) INS is effectively managing its IT investments and (2) the Department of Justice is effectively promoting, guiding, and overseeing INS' investment management activities.

To determine whether INS is effectively managing its investments, we applied our ITIM framework and the associated assessment method. As part of the ITIM assessment method, INS conducted a self-assessment of its IT investment management activities using the ITIM framework. In its self- assessment, INS indicated whether it executed each of the key practices in stages two through five. INS asserted that it executed many of the key practices within stages two and three but only four key practices in all of stages four and five. Accordingly, we did not include ITIM stages four and five in the scope of our review. Also, we did not evaluate the key practices within stages two and three that INS stated it had not executed.

We evaluated INS against 9 of the 10 critical processes in stages two and three. We did not evaluate INS against the stage three critical process Authority Alignment of IT Investment Boards. This critical process is only relevant if an organization has more than one IT investment board and INS has only one. The nine critical processes we examined focus primarily on INS' ability to effectively select and control its IT investments.

To determine whether INS had implemented these nine critical processes, we evaluated policies, procedures, and guidance related to INS' IT investment management activities. In particular, we analyzed the following: organizational charters, INS' System Development Life Cycle manual, requirements management process guide, and administrative manuals (e.g., Personal Property Handbook). We also reviewed documentation associated with specific investment management activities, such as IRB and ESC meeting minutes, project management plans, system deployment plans, budget formulation and execution plans, quarterly reports to Justice, and contractor statements of work.

In addition, we reviewed four IT projects to verify the execution of INS- defined processes, procedures, and practices. The four projects were selected based on the following criteria: (1) the projects should represent different life cycle phases (e.g., requirements definition, design, operations and maintenance), (2) the projects should support different INS business areas (e.g., Examinations, Enforcement), (3) at least one project should be considered high risk, and (4) at least one project should have been reviewed by Justice's Information Technology Investment Board (ITIB). The projects we evaluated are:

Coordinated Interagency Partnership Regulating International Students (CIPRIS): CIPRIS is an Internet-based system that is intended to modernize and streamline the current process for collecting information relating to nonimmigrant foreign students and other exchange program participants. It is intended to enable U. S. universities, schools, and cultural exchange programs to report and share information electronically with INS and other government regulatory agencies. INS has implemented an operational prototype of CIPRIS at 21 educational institutions. CIPRIS is a concept exploration project that supports the Examinations business area within INS. INS has designated CIPRIS as a high-risk project and it has been reviewed by Justice's ITIB. According to INS, it obligated about $3.1 million for CIPRIS in fiscal year 2000.

Computer-Linked Application Information Management System (CLAIMS) 4.0: According to INS, CLAIMS 4 is intended to improve delivery of naturalization services by fully automating INS' case management system. According to INS, CLAIMS 4 supports the Immigration Services Program within INS and is currently operational at 59 sites. According to INS, it obligated $14.5 million for CLAIMS 4 in fiscal year 2000.

Integrated Surveillance Intelligence System (ISIS): ISIS was established to detect and deter illegal intruders and to safely apprehend illegal aliens on the U.S.-Mexico and U.S.-Canada borders. ISIS is designed to provide all-weather sensor and video surveillance of the U.S. borders 24 hours a day, 7 days a week. The major components of ISIS are the Intelligent Computer-Assisted Detection system, ground sensors, and the Remote Video Surveillance system. ISIS supports the Enforcement program area within INS and has been reviewed by Justice's ITIB. According to INS, it obligated about $18 million for ISIS in fiscal year 2000 to further deploy the system.

Central Index System (CIS): CIS provides INS with information about persons of interest to the INS. According to INS, CIS also interacts with various INS databases to provide the data necessary for INS operations. CIS currently maintains approximately 45 million detailed records on individuals of interest to INS. CIS supports the INS' Corporate business area and is in the operations and maintenance phase of its life cycle. According to INS, it obligated about $2.6 million for CIS in fiscal year 2000.

We did not validate INS' IT spending obligations for fiscal year 2000 and IT spending estimates for fiscal year 2001.

To supplement our document reviews, we interviewed senior INS officials, including the Deputy Commissioner, who chairs the IRB, and the Executive Associate Commissioner for Management, who is the Chief Information Officer (CIO) and an IRB member. We also interviewed the Acting Associate Commissioner for Information Resources Management, who chairs the ESC; the Director of INS' Investment Management Team; portfolio managers; the Director of the Office of Strategic Information and Technology Development; IT project managers; program managers; Office of Budget representatives; and officials involved with the development and maintenance of INS' asset tracking systems.

We compared the evidence collected from our document review and interviews to the key practices and critical processes in ITIM. Because ITIM is a hierarchical framework, the rating of each critical process is dependent on the key practices below it. Therefore, we first rated the key practices. In accordance with the ITIM assessment method, we rated a key practice as “executed” when we determined, by consensus, that INS was executing the key aspects of the practice. A key practice was rated as “not executed” when we determined that there were significant weaknesses in INS' execution of the key practice and INS offered no adequate alternative, or when the team found no evidence of a practice during the review.

Once the key practices were rated, we rated each of the nine critical processes we reviewed. A critical process was rated as “implemented” if all of the underlying key practices were rated as being executed. A critical process was rated as “not implemented, but improvements underway” if over half, but not all, of its underlying key practices were rated as being executed. A critical process was rated as “not implemented” when there were significant weaknesses (i.e., fewer than 50 percent of the key practices had been implemented) in INS' implementation of the underlying key practices and no adequate alternative was in place.

To determine whether the Department of Justice is effectively promoting, guiding, and overseeing INS' investment management activities, we interviewed officials within the Office of Information Management and Security Staff, the organization that plays a leading role in Justice's investment management activities. We also reviewed Justice's January 2000 investment management guidance, draft policy and guidance documents, INS project proposals, ITIB review and decision documentation, and quarterly briefing documents. We also discussed Justice's oversight activities with various officials within INS.

We conducted our work at INS and Justice headquarters in Washington, D.C., from May 2000 through October 2000 in accordance with generally accepted government auditing standards. Justice's Assistant Attorney General for Administration provided written comments of a draft of this report. These comments are presented in chapter 5 and are reprinted in appendix I.

INS Lacks Foundation Capabilities Upon Which to Build IT Investment Management Maturity

The primary purpose of ITIM stage two maturity is to attain repeatable, successful IT project-level investment control processes and basic selection processes. For an organization to develop an overall sound IT investment management process, it must first be able to control its investments so that it can identify expectation gaps early and correct them. According to ITIM, stage two maturity includes (1) defining IRB operations, (2) developing a basic process for selecting new IT proposals, (3) developing project-level investment control processes, (4) creating an IT asset inventory, and (5) identifying the business needs for each IT project.

INS has not fully implemented any of the critical processes associated with stage two; however, it has improvements underway and is close to fully implementing two of these processes. INS has (1) established an IRB, which comprises both IT and business senior executives and functions as INS' central decision-making body for IT projects, and (2) the IRB has followed a structured process for developing and selecting new IT proposals and making initial funding decisions for these proposals.

However, INS has not yet developed some of the capabilities necessary to build a sound IT investment management process. For example, INS has not (1) established basic project-level control processes to ensure that its IT projects are performing as expected, (2) created an IT asset inventory for investment management, and (3) defined business needs for all of its IT projects. According to INS, it lacks these critical investment capabilities because it has not yet made IT investment management an institutional priority. Table 1 summarizes INS' stage two maturity.

INS' capabilities for each of the stage two critical processes are discussed below.

INS Has Established an IRB, But Has Not Developed Policies and Procedures to Govern IRB Operations

The purpose of this critical process is to define and establish the governing board or boards responsible for selecting, controlling, and evaluating IT investments. This includes defining the membership, guiding policies, operations, roles and responsibilities, and authorities for the investment board and, if appropriate, each board's support staff. These policies, roles and responsibilities, and authorities also provide the basis for the board's investment selection, control, and evaluation activities.

According to ITIM, effective IT investment board operations require, among other things, that (1) the board membership include both IT and business knowledge, (2) the organization's executives and line managers support and carry out board decisions, (3) the organization create an organization-specific process guide that includes policies and procedures to direct the board's operations, and (4) the IRB operate according to these written policies and procedures.

INS is executing many of the practices in this critical process. For example, INS has an IRB that functions as a central decision-making body for IT investments and is composed of senior executives from both INS' IT and business areas. During our discussions with agency officials, we found broad support within the organization for the IRB's decisions. For example, three of the four program/project managers we interviewed acknowledged the IRB's role in investment decision-making. The IRB is chaired by the Deputy Commissioner and includes INS' three Executive Associate Commissioners. The IRB is supported by an ESC, which is comprised of senior representatives who manage INS' eight IT portfolios. The ESC reviews and analyzes IT investments and makes recommendations to the IRB for final approval. This senior level involvement and the breadth of representation help to demonstrate executive sponsorship of the process and support for the projects selected.

While INS has an IRB, it is not functioning according to written policies and procedures. Instead, the IRB operates according to undocumented procedures for selecting new IT proposals. According to the Director of INS' Investment Management Team, INS has begun developing written policies and procedures and plans to complete them about March 2001. However, until INS develops and implements these policies and procedures, key IT investment activities may not be done consistently, if at all. Table 2 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Is Not Effectively Overseeing Its Ongoing IT Projects

The purpose of project oversight is to ensure that the IRB provides effective oversight for its ongoing IT projects throughout all phases of their life cycle. Under stage 2 maturity, the IRB should review each project's progress toward predefined cost and schedule expectations, using established criteria, and take corrective actions when cost estimates and project milestones are not achieved. Implementing this critical process provides the basis for evolving the organization's IT investment control activities.

According to ITIM, effective project oversight requires, among other things, (1) having written polices and procedures for project management, (2) developing and maintaining an approved project management plan for each IT project, (3) having written policies and procedures for oversight of IT projects, (4) making up-to-date cost and schedule data for each project available to the IRB, (5) reviewing each project's performance by comparing actual cost and schedule data to expectations regularly, and (6) ensuring that corrective actions for each underperforming project are defined, implemented, and tracked until the desired outcome is achieved.

INS is not effectively overseeing its IT projects. While INS has documented policies and procedures for project management in its System Development Life Cycle (SDLC) manual, it is not following its own procedures. For example, INS has not developed and maintained project management plans that include cost and schedule controls for each of its IT projects, an SDLC requirement. In fact, only two of the four projects that we reviewed had current project management plans.

Furthermore, INS does not have written polices and procedures for oversight of its IT projects. Without written polices and procedures, INS increases the risk that project oversight activities will not be performed effectively. For example, the IRB does not (1) receive up-to-date cost and schedule data for each project, (2) oversee each project's performance regularly by comparing actual cost and schedule data to expectations, and (3) ensure that corrective actions are implemented and tracked for underperforming projects. In the absence of effective oversight, INS executives do not have adequate assurance that IT projects are being developed on schedule and within budget. Table 3 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Is Not Tracking and Using IT Asset Information for Investment Management Purposes

The purpose of the asset tracking critical process is to create and maintain an IT asset inventory to assist in managerial decision-making. To make good investment decisions, an organization must know where its IT assets (i.e., personnel, systems, applications, hardware, software licenses, etc.) are located and how funds are being expended toward acquiring, maintaining, and deploying them. This critical process identifies IT assets within the organization and creates a comprehensive inventory of them. This inventory can take many forms, but regardless of form, the inventory should identify each asset and its associated components.

Beyond identifying IT assets, this process is used to support other ITIM critical processes by serving as an investment information and data repository that contains such items as the list of systems and projects and data on each project's progress toward achieving its plans. To support investment decision-making, this inventory should also be accessible where it is of the most value to decisionmakers.

According to ITIM, effectively tracking IT assets requires, among other things, (1) making investment information available on demand to decisionmakers, (2) developing and maintaining an IT asset inventory according to written procedures, (3) overseeing the development and maintenance of the asset tracking process, and (4) assigning responsibility for managing this tracking process.

INS has not implemented an effective IT asset tracking process for investment management. While investment information from various sources has been available to the IRB on an ad hoc basis, it is not available on demand and INS has not developed and maintained an inventory for investment management purposes according to written policies and procedures. In addition, the IRB does not oversee IT asset tracking activities and has not assigned responsibility for managing this tracking process to support investment decision-making. In the absence of standard, documented procedures for developing and maintaining the inventory, INS executives do not have adequate assurance that timely, complete, and consistent asset data are available to them. Table 4 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Has Not Defined Business Needs for All Its IT Projects

The purpose of defining business needs for each IT project is to ensure that each project supports the organization's business needs and meets users' needs. Thus, this critical process creates the link between the organization's business objectives and its IT management strategy. According to ITIM, effectively identifying business needs requires, among other things, (1) defining the organization's business needs or stated mission goals, (2) identifying users for each project who will participate in the project ‘s development and implementation, (3) defining business needs for each project, and (4) training IT staff in business needs identification.

INS has executed some of the key practices associated with effectively defining business needs for IT projects. For example, INS has (1) defined its business needs and mission goals in its annual performance plan and (2) identified users for its projects who participate in the project ‘s development and implementation. However, INS has not clearly defined specific business needs for each project. In addition, only one of the four project managers that we interviewed stated that he or she had been trained in business needs identification. In the absence of documented business needs, the IRB cannot ensure that it is selecting IT investments that meet its mission needs and priorities. Table 5 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Has a Structured Process for Selecting New IT Proposals But Has Not Consistently Analyzed Them According to Established Criteria

The purpose of proposal selection is to establish a structured process for selecting new IT proposals. According to ITIM, effective proposal selection requires, among other things, (1) designating an official to manage the proposal selection process, (2) using a structured process to develop new IT proposals, (3) making funding decisions for new IT proposals according to an established selection process, and (4) analyzing and ranking new IT proposals according to established selection criteria, including cost and schedule criteria.

INS has established a structured process for selecting new IT proposals. The Deputy Commissioner, as the Chair of the IRB, is designated to manage INS' proposal selection process. In addition, INS uses a structured process to develop new proposals and makes initial funding decisions for these proposals. However, INS has not consistently analyzed and ranked these proposals according to established selection criteria. Established selection criteria would assist IT managers in creating proposals that best meet the needs and priorities of INS. Table 6 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Is Not Managing Its IT Investments as a Complete Portfolio

An IT investment portfolio is a collection of investments that are assessed and managed based on common criteria. While an organization may have more than one level of investment portfolios, it should always have an enterprisewide portfolio. Managing investments as a portfolio is a conscious, continuous, and proactive approach to expending limited resources on all competing initiatives in light of the relative beneficial effects of these investments. Taking an enterprisewide portfolio perspective enables an organization to consider its investments comprehensively so that the investments address its mission, strategic goals, and objectives. A portfolio approach also allows an organization to determine priorities and make decisions about which projects to fund based on analyses of the relative costs, benefits, and risks of all projects, including projects that are proposed, under development, and in operation.

The purpose of ITIM stage three maturity is to create and manage IT investments as a complete enterprise investment portfolio. Once ongoing projects can be implemented on schedule and within budget as is emphasized in stage two, the organization is capable of managing its projects as an investment portfolio. According to ITIM, stage three maturity includes (1) defining portfolio selection criteria, (2) engaging in project- level investment analysis, (3) developing a complete portfolio based on the investment analysis, and (4) maintaining oversight over the investment performance of the portfolio.

INS has not implemented any of the critical processes in stage three. In general, INS has not created the associated policies and procedures to initiate or perpetuate any of the critical processes, and as a result, it has not systematically collected and analyzed the data needed to make sound and informed decisions about competing investment choices, which consciously consider value and risk. In addition, while INS has established eight portfolio categories, it has not established an enterprisewide investment portfolio. Therefore, decisions may be made between competing investments within a business area, but INS cannot make trade- offs between investments across the enterprise to determine which projects contribute most to the agency mission and priorities. According to INS officials, INS has not yet made IT investment management an institutional priority. Table 7 summarizes INS' stage three maturity.

INS' capabilities for each of the stage three critical processes are discussed below.

INS Has Not Created Useful Portfolio Selection Criteria

Portfolio selection criteria make up a necessary part of an IT investment management process. Developing an enterprisewide investment portfolio involves defining appropriate investment cost, benefit, schedule, and risk criteria to ensure that the selected investments will best support the organization's strategic goals, objectives, and mission. Thus, portfolio selection criteria need to reflect the enterprisewide and strategic focus of the organization. In addition, the criteria should (1) include cost, benefit, schedule, and risk elements, which serve to create a common set of criteria that are used to compare projects of different types to one another and (2) be clearly communicated to project managers throughout the organization so that these managers can take the criteria into account when developing proposals. Without portfolio selection criteria, projects may be selected on the basis of isolated business needs, the type and availability of funds, or the receptivity of management to a specific project proposal.

Thus, according to ITIM, developing portfolio selection criteria requires, among other things, that (1) an investment board approve the criteria, including cost, benefit, schedule, and risk criteria; (2) the criteria be distributed throughout the organization; (3) adequate resources be provided for selection criteria definition activities; and (4) a working group be responsible for creating and modifying the criteria.

INS developed criteria for selecting new proposals; however, the criteria had not been approved by the IRB and did not consistently include cost, schedule, benefit, and risk criteria. Furthermore, INS had not distributed the criteria throughout INS. For example, none of the IT project and program managers that we interviewed were aware of the selection criteria that had been developed. In addition, while INS indicated that it has adequate resources to develop complete portfolio selection criteria, it has not designated a working group to create and modify the criteria. Without useful selection criteria, INS is missing a critical means of ensuring that selected investments best support the organization's mission and priorities. Table 8 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Does Not Analyze Its IT Investments Based on Cost, Benefit, Schedule, and Risk Data When Making Investment Decisions

The purpose of investment analysis is to ensure that all IT investments are consistently analyzed and prioritized according to the organization's portfolio selection criteria, which should include cost, benefit, schedule, and risk criteria. According to ITIM, effective investment analysis includes, among other things, that (1) portfolio selection criteria have been developed; (2) the IRB ensures that cost, benefit, schedule, and risk data are assessed and validated for each investment; (3) the IRB compares each investment against the organization's portfolio selection criteria; and (4) the IRB creates a ranked list of investments using the portfolio selection criteria.

INS' IRB does not analyze and rank proposed and ongoing investments based on their expected cost, benefit, schedule, and risk. As mentioned previously, INS has not developed selection criteria that include these elements, nor has it ensured that cost, benefit, schedule, and risk data are assessed and validated for each IT investment. For example, none of the four projects we reviewed provided cost, benefit, schedule, or risk data to INS' IRB for consideration during the selection process. Instead, the IRB focused on the near-term cost (e.g., annual budget dollars) of each project and the perceived importance of the project to INS' mission. In the absence of portfolio selection criteria and good investment-related data (i.e., cost, benefit, schedule, and risk data), the IRB cannot compare and analyze its investments based on their cost, benefit, schedule, and risk expectations and create a ranked list of investments that best align with mission improvement goals and organizational direction. As a result, INS is missing critical information for making sound IT investment decisions. Table 9 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Does Not Comparatively Assess All Its IT Projects When Making Selections for Funding

The purpose of the portfolio development process is to ensure that the IRB analyzes and compares all IT investments to select and fund those with manageable risks and returns and that best address the strategic business direction and priorities of the organization. Once this is accomplished, investments can be compared to one another within and across the portfolio categories and the best overall portfolio can then be selected for funding.

According to ITIM, portfolio development requires, among other things, (1) defining common portfolio categories and assigning each investment to a portfolio category; (2) ensuring that investments have been analyzed and their cost, benefit, schedule, and risk data validated; and (3) examining the mix of investments across the portfolio categories in making funding decisions.

INS does not assess all its IT projects in making selections for funding. While INS has defined common portfolio categories, it is not using them to manage its investments. INS has created eight portfolio categories and assigned all of its investments to one of the portfolios. However, the IRB has not analyzed these investments, including both proposed and ongoing projects, based on validated cost, benefit, schedule, and risk data. Without these meaningful data, the IRB cannot compare its investments across portfolio categories. As a result, the IRB cannot make trade-offs between investment alternatives, determine which projects contribute most to agency performance, or eliminate redundant systems. Table 10 summarizes the ratings for each key practice and the specific findings supporting the ratings.

INS Does Not Oversee IT Investments' Cost, Benefit, Schedule, and Risk Performance

The purpose of the portfolio performance oversight critical process is to ensure that each IT investment achieves its cost, benefit, schedule, and risk expectations. This critical process builds upon the IT Project Oversight critical process by adding the elements of benefit measurement and risk management to an organization's investment control capacity. Executive- level oversight of project-level risk and benefit management activities provides the organization with increased assurance that each investment will achieve the desired cost, benefit, schedule, and risk results.

According to ITIM, effective portfolio performance oversight requires, among other things, that the IRB (1) have access to up-to-date cost, benefit, schedule, and risk data; (2) monitor the performance of each investment in its portfolio by comparing actual project-level cost, benefit, schedule, and risk data to the predefined expectations for the project; and (3) correct poorly performing projects.

INS does not monitor its investments' performance to ensure that they are meeting cost, benefit, schedule, and risk performance expectations. As mentioned previously, up-to-date cost, benefit, schedule, and risk data are not available. Without these data, the IRB is unable to monitor the performance of its investments to ensure that they are achieving their cost, benefit, schedule, and risk expectations and to act when performance problems arise. Table 11 summarizes the ratings for each key practice and the specific findings supporting the ratings.

The Department of Justice Is Not Guiding and Overseeing INS' Investment Management Approach

The Clinger-Cohen Act of 1996 imposed rigor and structure on how agencies approach the selection and management of IT projects. Among other things, it requires the head of each agency to implement a process for maximizing the value of the agency's IT investments and assess and manage the risks of its IT investments. It also requires that the agency CIO work with the agency head in implementing this process. As such, Justice is responsible for ensuring that its bureaus and components, including INS, implement an effective IT investment management process.

Justice has not provided INS, or any other Justice component, sufficient direction, guidance, and oversight of IT investment management activities. While Justice issued guidance in January 2000 describing its high-level investment management process, the guidance does not address the need or requirement for Justice's components to implement an IT investment management process. Specifically, this guidance does not instruct the components to establish IT investment management processes nor does it establish expectations for doing so. According to Justice officials, Justice had not established these processes because of other competing department priorities, even though the department and its components spent about $3 billion on IT in fiscal years 1999 and 2000.

During the course of our work, Justice began drafting IT investment management policy and guidance documents in collaboration with an intercomponent working group. The draft policy directs Justice components to establish and use an IT investment management process and directs the Justice CIO to monitor the components' investment management processes through periodic briefings. A supplemental guidance document provides procedures for developing an investment management process. Justice officials stated that they plan to issue the final policy by the end of December 2000 and the guidance by March 2001. Until Justice issues its policy and guidance and begins monitoring its components' progress, it has no assurance that it has the necessary investment management processes in place to maximize the value of its IT investments and manage the risks associated with them.

Conclusions, Recommendations, and Agency Comments

IT is critical to INS' ability to provide vital services, such as granting naturalization benefits and detecting and preventing the illegal entry of aliens into the United States. Effectively and efficiently managing IT requires, among other things, a structured approach for minimizing the risk and maximizing the return on IT investments. However, INS executives are making investment decisions involving hundreds of millions of dollars without vital data about these investments' relative costs, benefits, and risks. As a result, INS cannot adequately know whether it is making the right investment decisions, whether it has selected the mix of investments that best meets its overall mission and business priorities, or whether these investments are living up to expectations.

INS has initiated efforts to establish an IT investment management foundation. However, it is lacking many important foundational investment management capabilities, particularly those relating to controlling projects against predetermined expectations and addressing variances. As a result, it runs the serious risk that its IT projects will be late, cost more than expected, and not perform as intended.

INS' use of portfolio categories and portfolio managers provides some structure to its portfolio development process and provides each business area the opportunity to identify the projects that it determines to be the most important to its performance. However, INS' lack of performance data from ongoing projects handicaps the IRB's ability to perform its portfolio oversight function. In addition, the absence of any project-to-project comparison limits the IRB's ability to judge whether its mix of investments best meets its mission needs and priorities. As a result, INS can have little confidence that its chosen mix of IT investments best meets mission goals and priorities and that these investments will be developed within an acceptable level of risk, on time, and within budget.

Further, Justice has a statutory role under the Clinger-Cohen Act to ensure that its component agencies, including INS, have effective investment management processes. Until Justice fulfills this role, it has little assurance that INS, or its other components, are investing the department's limited IT resources to maximize return on investment, minimize risk, and best support mission needs.

Recommendations for Executive Action

To strengthen INS' investment management capability and address the weaknesses discussed in this report, we recommend that you direct the Commissioner of the Immigration and Naturalization Service to designate development and implementation of effective IT investment management processes as an agencywide priority and manage it as such. Specifically, you should direct the Commissioner to do the following:

Develop a plan, within 9 months, for implementing IT investment management process improvements that is based on stages two and three critical processes and specifies measurable goals and time frames, ranks initiatives, defines a management structure for directing and controlling the improvements, establishes review milestones, and recognizes any direction and guidance that Justice issues. This plan should first focus on those critical processes in stage two of ITIM because, collectively, they provide the foundation for building a mature IT investment management process.

Submit the plan to the Justice CIO for review and approval.

Implement the approved plan and report to the Justice CIO, according to established review milestones, on progress made against the plan's goals and time frames.

Further, because the absence of effective investment management processes and an enterprise architecture severely limits INS' ability to effectively manage its IT investments, we recommend that until INS develops a complete enterprise architecture and implements the key practices associated with stages two and three critical processes, as described in this report, you direct the Commissioner to limit requests for future appropriations for IT only to efforts that support ongoing operations and maintenance, but not major enhancements, of existing systems; support INS efforts to develop and implement IT investment management processes and an enterprise architecture; are small, represent low technical risk, and can be delivered in a relatively short period of time; or are congressionally mandated.

Further, to improve Justice's guidance and oversight of components' IT investment management process activities, we also recommend that you direct the Justice CIO to follow through on the department's plans to issue an IT investment management policy and guidance to the components and to ensure that the policy and guidance:

Directs Justice components and bureaus, including INS, to develop and implement IT investment management processes.

Instructs Justice components and bureaus on how to develop an investment management process. This guidance should be based on the investment management guidance contained in this report and, at a minimum, should include component roles, responsibilities, authorities, and policies and procedures for developing an IT investment management process.

Directs the Justice CIO to monitor the components' progress in developing and establishing an IT investment management process and take appropriate action if they are not progressing sufficiently.

Agency Comments and Our Evaluation

In written comments on a draft of this report, Justice's Assistant Attorney General for Administration generally agreed with our recommendations, although he offered minor wording modifications on two recommendations that he said would increase Justice's ability to fully implement them. The Assistant Attorney General for Administration also disagreed with our finding that Justice is not guiding and directing INS' investment management approach.

Justice generally agreed with our recommendation that INS develop and submit to Justice a plan for implementing investment management process improvements. However, Justice suggested that the time frame for developing the plan be clarified such that INS has 6 months to develop and submit its plan to Justice once Justice issues its new IT investment management guidance. Because our recommendation directed INS to consider any Justice guidance and direction in developing its investment management process improvement plan, we modified the recommendation to include an additional 3 months to allow time for Justice to issue its guidance, which it plans to do in March 2001.

Justice also concurred with our recommendation that INS limit future appropriation requests for IT to certain investment categories because it lacks an enterprise architecture and effective investment management processes, but suggested that we specify that this recommendation is in effect until INS completes its architecture and implements investment management processes. Because this is the intent of our recommendation, we clarified the recommendation to make this explicit.

Also in its comments, Justice agreed that, while INS has some important investment management capabilities, INS still needs to develop effective investment management processes. Further, Justice agreed with our recommendation for Justice to issue an investment management policy and guidance to its components, including INS, that (1) directs components to develop and implement IT investment management processes, (2) instructs components on how to develop and implement these processes based on the investment management framework in our report, and (3) ensures that components' progress in doing so is monitored. Moreover, Justice stated, which we note in our report, that it is now working with its components to develop an IT investment management policy and process, and it has made this a department priority for this year.

However, Justice stated that our draft report fails to recognize the extent of Justice's oversight of INS' IT investment management process. Further, it disagreed with our finding that Justice is not guiding and directing INS' investment management approach. Justice stated that it has established guidance for all aspects of IT management that its components are expected to follow and has a process for overseeing components' management of their investments. Justice cited six examples to illustrate its point, such as Justice approval authority of all component IT investments with life-cycle cost over $1 million, Justice establishment of an IT investment board, Justice meetings with components, including Attorney General meetings with the INS Commissioner, and Justice forwarding of OMB budget requirements to components.

We do not agree with Justice's position. While we concur that the examples cited by Justice represent important IT management functions to be performed in providing management oversight of individual IT investments, such management oversight is not the focus of our findings, conclusions, and recommendations. Rather, our report addresses Justice's efforts to ensure that its components, including INS, have each defined and implemented effective IT investment management processes. As such, we sought evidence from Justice demonstrating that it has directed its components to establish such processes, provided guidance to its components on how to develop and implement these processes, and monitored its components' progress to determine whether they are implementing such processes. However, besides the steps that Justice initiated during the course of our inquiries and plans to take, which we have described in this report, we found no such evidence. Moreover, Justice stated in its written comments that it agreed with our recommendation for it to provide investment management process direction, guidance, and oversight to its components.

Justice's written comments and our evaluation of them are presented in appendix I.