Immigration Benefits System: Better Informed Decision Making Needed on Transformation Program
Released: 2015-06-17T00:00:00

Letter

Background

The mission of USCIS is to adjudicate applications and petitions for immigration benefits, and requests for other action by individuals seeking to become either a citizen of the United States or a lawful permanent resident or to temporarily study, live, or work in this country. To accomplish its mission, each year USCIS processes millions of applications, petitions, and requests for more than 50 types of immigrant and nonimmigrant-related benefits.

As we reported in 2007, the processing of millions of applications, petitions, and requests has been hindered by inefficient, paper-based processes, which has resulted in a backlog that peaked in 2004 at more than 3.8 million cases; tens of thousands of missing or misplaced files; difficulties in verifying the identity of applicants; problems in providing other government agencies with the information necessary to identify criminals and potential terrorists; and benefits being issued to unverified applicants.

USCIS Established a Transformation Program to Modernize Benefits Processing

Begun in 2005, the goals of the Transformation Program are to modernize the paper-based immigration benefits process to enhance national security and system integrity, and to improve customer service and operational efficiency. The program is comprised of many systems, each of which provides a service to facilitate operations, such as identity management and risk and fraud analytics. The main component of the program is the USCIS Electronic Immigration System (ELIS), which is to provide case management for the adjudication of immigration benefits and interface with other systems to achieve end-to-end processing of immigration benefits. Once the system has been implemented, USCIS expects that

Applicants will be able to establish an account with USCIS to file and track the status of the application, petition, or request online.

The system will apply risk-based rules automatically to incoming applications, petitions, and requests to identify potentially fraudulent applications and national security risks.

Adjudicators will have electronic access to applications, petitions, and requests, relevant policies and procedures, and external databases to aid in decision making.

USCIS will have management information to track and allocate workload.

The system will allow electronic linkages to other agencies, such as the Departments of Justice and State, for data sharing and security purposes. Appendix II lists systems internal and external to USCIS with which USCIS ELIS will interface.

Figure 1 provides an overview of how applicants are to file for immigration benefits and apply for citizenship using the new system.

Five core operational requirements are expected to form the foundation of USCIS ELIS, which is to process and manage all applications. Table 1 describes the five core operational requirements.

Each core operational requirement is broken down into a series of capabilities, features, and sub-features. For example, the Intake and Account Management operational requirement has five capabilities, such as the Establish/Authenticate portal account capability, which is broken down into four features, including Manage Account Access. This feature is further decomposed into five sub-features, including Provide Account Management for Customers and Representatives. Appendix III provides additional information on the planned capabilities for the Transformation Program.

Key Aspects of USCIS’ Original Transformation Program Strategy

USCIS began implementing the Transformation Program by awarding contracts for various phases, including for pilot projects and system architecture. In fiscal year 2006, USCIS initiated three pilot projects and one for a proof-of-concept effort.

Enumeration pilot: The pilot was a joint effort by USCIS and the then- United States Visitor and Immigrant Status Indicator Technology (US- VISIT) program to positively identify individuals through the generation of a unique identifier permanently associated with a person. The enumerator is created by the submission of fingerprint and biographic data. Each time the person returns to USCIS for a subsequent benefit application, biometric information is to be matched to determine if the individual is the same person. This process is intended to limit the number of times fingerprints are submitted for background checks.

Biometric Storage System pilot: The Biometric Storage System was to help streamline the established USCIS biometric and card production processes and become the centralized repository for all USCIS customer biometrics. The system receives the 10-print fingerprints taken at application support centers along with related biographic data, and submits them to the Federal Bureau of Investigation for a fingerprint check and then to the US-VISIT Automated Biometric Identification System for creation of the enumerator and for permanent storage. The 10-print fingerprints and photos are then associated with benefit cards that serve as travel documents. The system is to store the images only as long as needed to facilitate adjudication and production of benefits cards.

Digitization pilot: The Digitization pilot was comprised of the Enterprise Document Management System, which stores electronic documents along with metadata describing the documents, and the Records Digitization Facility, which scans documents into electronic format, creates the metadata, and transfers the document to the document management system. The pilot tested the use of images by adjudicators in their day-to-day work environment.

The proof-of-concept was intended to demonstrate the case processing capability of the case management system. Specifically, it was to demonstrate that the enumerator (unique identifier based on biometrics) could link an applicant to related filings for that individual and assess whether the case management system could be used to view digitized files. According to the Transformation Program Management Plan, a post- implementation review completed in October 2007 determined that the proof-of-concept met all of its primary objectives and that the Transformation Program should continue with this approach.

In November 2008, USCIS awarded a solutions architect contract for approximately $500 million to be allocated over a 5-year period to design, develop, test, deploy, and sustain the Transformation Program by November 2013. The overall strategy was to deliver the solution in two increments: Increment 1, to include releases A and B, and Increment 2, to include releases C, D, and E. Releases A and B included making USCIS ELIS available to applicants applying for nonimmigrant benefits and much of the functionality needed to operate the five core operational requirements. Table 2 shows the order in which USCIS ELIS’s five releases were to be deployed and the types of benefits to be made available to applicants in each release.

In July 2011, DHS officially approved the Transformation Program’s acquisition program baseline and supporting operational requirements. The baseline included approved objectives (targets that reflect the most likely cost and schedule) and approved thresholds (ceilings which, if exceeded, initiate official re-planning actions). Table 3 details the cost and schedule parameters of the program’s approved July 2011 baseline.

On May 22, 2012, USCIS launched the first release of USCIS ELIS (release A1). This release included capabilities associated with all of the core operational requirements, such as online account setup, case management, case acceptance, applicant evidence intake, and notice generation. Since May 2012, five primary releases, along with a series of maintenance releases, have been deployed to add functionality to USCIS ELIS, with new applicant processing added in two of the five releases. Table 4 shows the date of each functional release and a description of the capabilities in the release.

DHS Acquisition Management

USCIS ELIS is to be managed consistent with DHS’s acquisition management process. DHS’s Acquisition Management Directive 102-01 and its Instruction Manual 102-01-001 (the guidebook) establish the department’s policies and processes for managing major acquisition programs. Acquisition Management Directive 102-01 provides high-level direction to program managers to help determine funding needs, capability requirements, and schedule, and the guidebook provides specific direction to program managers on how to implement the directive, such as how to address a program breach.

DHS’s Deputy Secretary and Under Secretary for Management serve as the acquisition decision authorities for the department’s largest acquisition programs, those with life-cycle cost estimates of $1 billion or more. The Under Secretary for Management also serves as DHS’s Chief Acquisition Officer, and in this role is responsible for the management and oversight of the department’s acquisition policies and procedures.

To assist in the acquisition oversight process, the Under Secretary for Management is supported by two offices within the department. The Office of Program Accountability and Risk Management (PARM) is responsible for DHS’s overall acquisition governance process. The Office of the Chief Information Officer (OCIO) is responsible for, among other things, setting departmental information technology (IT) policies, processes, and standards. It is also responsible for ensuring that IT acquisitions comply with DHS IT management processes, technical requirements, and the approved enterprise architecture.

Within the OCIO, the Enterprise Business Management Office is to ensure that the department’s IT investments align with its missions and objectives. As part of its responsibilities, this office periodically assesses IT investments to gauge how well they are performing through a review of program risk, human capital, cost and schedule, and requirements. These assessments serve as the criteria for reporting to the IT Dashboard for oversight by the Office of Management and Budget (OMB).

In March 2015, we reported that DHS acquisition policy does not define the differences in the role of PARM and the OCIO’s Enterprise Business Management Office in oversight of major IT acquisitions. In particular, the functions of PARM and the Enterprise Business Management Office may overlap. Further, we reported that programs report to PARM and the Enterprise Business Management Office through two separate information systems, which further complicate the distinction. In order to ensure consistent, effective oversight of DHS’s acquisition programs, we recommended that the Secretary of DHS direct the Under Secretary for Management to develop written guidance to clarify the roles and responsibilities of PARM and OCIO Enterprise Business Management Office for conducting oversight of major acquisition programs. DHS concurred with our recommendations.

DHS Acquisition Life Cycle

DHS Acquisition Management Directive 102-01 establishes that a major acquisition program’s decision authority is responsible for reviewing and approving the movement of the program through four phases of the acquisition life cycle at a series of five acquisition decision events. These acquisition decision events, which can be more than 1 year apart, provide the acquisition decision authority an opportunity to assess whether a major program is ready to proceed through the life cycle phases. Following are the four phases of the acquisition life cycle, as established in DHS acquisition policy: 1. Need: Department officials identify that there is a need, consistent with DHS’s strategic plan, justifying an investment in a new capability and the establishment of an acquisition program to produce that capability. This phase concludes with the acquisition decision authority granting the acquisition program approval to proceed at acquisition decision event 1. 2. Analyze/Select: A designated program manager reviews alternative approaches to meeting the need and recommends a best option to the acquisition decision authority. This phase concludes with the acquisition decision authority granting the acquisition program approval to proceed at acquisition decision event 2A. 3. Obtain: The program manager develops, tests, and evaluates the selected option. The acquisition decision authority may review the acquisition program multiple times before granting the acquisition program approval to proceed with particular acquisition activities. Acquisition decision event 2B focuses on the cost, schedule, and performance parameters for each of the program’s projects. The program may also proceed through acquisition decision event 2C, which focuses on low rate initial production. The phase concludes with the acquisition decision authority granting the program approval to proceed at acquisition decision event 3. 4. Produce/Deploy/Support: DHS delivers the new capability to its operators, and maintains the capability until it is retired. This phase includes sustainment, which begins when a capability has been fielded for operational use; sustainment involves the supportability of fielded systems through disposal, including maintenance.

Figure 2 depicts the acquisition life cycle established in the directive.

Two important aspect of the acquisition decision events are the review and approval of key acquisition documents critical to establishing the need for a major program, its operational requirements, an acquisition baseline, and testing and support plans. Table 5 describes the key acquisition documents requiring department-level approval before a program can move to the next acquisition phase.

Figure 3 depicts the acquisition life cycle and associated program documentation requirements.

We have previously reported on the management and development of the USCIS Transformation Program. In July 2007, we evaluated the Transformation Program strategic and expenditure plans to determine the extent to which these plans had prepared USCIS to carry out its program. We reported that the agency’s plans partially or fully addressed most key practices but more attention was needed in certain areas such as performance measurement and IT management. We also reported that the plans provided some information on costs and revenues, but that USCIS had not finalized its acquisition strategy and, therefore, cost estimates were uncertain.

To improve its transformation strategy and fully address congressionally requested information, we recommended that the Director of USCIS address gaps in plans in the areas of performance measurement, strategic human capital management, communications, and IT management practices. DHS concurred with our recommendations.

Between September 2009 and September 2011, USCIS took steps to address the gaps identified in our report, such as finalizing a balanced set of four performance measures and establishing fiscal year 2012 targets that aligned with transformation goals for customer satisfaction, decisional accuracy, timeliness, and efficiency.

In November 2011, we assessed the extent to which USCIS had followed DHS acquisition policy in developing and managing the Transformation Program. We reported that the agency had not consistently followed the acquisition management approach that DHS outlined in management directives in developing and managing the program. For example, USCIS did not complete several acquisition planning documents required by DHS policy prior to moving forward with an acquisition approach, which contributed to schedule delays and increased program costs. To help ensure that USCIS used a comprehensive and cost-effective approach to the development and deployment of transformation efforts to meet the agency’s goals of improved adjudications and customer services processes, we recommended that the Director of USCIS develop and maintain an integrated master schedule consistent with best practices for the Transformation Program and ensure that the life-cycle cost estimate be informed by milestones and associated tasks from reliable schedules. The agency concurred with our recommendations and has begun work to address them.

Transformation’s Deployment Schedule Is Significantly Delayed, Causing Missed Cost Savings and Deferred Mission Benefits

The Transformation Program has changed its system acquisition strategy, which has contributed to significant delays in the program’s planned schedule. As of March 2015, USCIS ELIS functionality deployed in the program’s initial releases is still in operation. Moving forward, USCIS estimates that its Transformation Program will now cost up to $3.1 billion and be fully deployed no later than March 2019. This is an increase of approximately $1 billion and a delay of over 4 years from its initial approved baseline. In addition, several major changes were made to the Transformation Program acquisition strategy to help address concerns about delays and cost overruns, including changes to the software development methodology, contracting approach, and program architecture. However, the plans for this new approach have yet to be formally approved. Nevertheless, despite the lack of an updated and approved program baseline, USCIS has begun system acquisition by awarding contracts for planning and development. Moreover, the delay in the program’s planned schedule has in turn impacted USCIS’s ability to achieve cost savings, operational efficiencies, and other benefits.

Transformation Program’s Baseline Estimates and System Acquisition Strategy Have Significantly Changed

As of March 2015, the program had work underway to maintain existing operations as well as to transition to a new system architecture. More specifically, USCIS ELIS capabilities developed and implemented in releases A1 through A2.5 were still in operation. To maintain this functionality, USCIS extended its initial Solution Architect contract, originally scheduled to end in 2014, through May 2015. This functionality is expected to be replaced under a new architecture (as discussed later in this report). The program also had transition activities underway to test this new architectural environment. In particular, USCIS conducted a limited deployment for replacing and renewing permanent resident cards in November 2014 and fully deployed an initial set of capabilities in March 2015. Following this deployment, USCIS expects to begin decommissioning the existing capabilities.

With regard to program baseline estimates, the Transformation Program is expected to cost no more than $3.1 billion and to be deployed no later than March 2019. This is an increase of approximately $1 billion and delay of over 4 years from the initial baseline approved in July 2011. According to program officials, the Transformation Program is expected to deliver the same system capabilities as initially intended; however, the plan for deploying such capabilities has been revised.

Table 6 compares the Transformation program’s current draft baseline to its initial baseline plans.

In addition to the changes made to the program baseline estimates, several major changes were made to the Transformation Program acquisition strategy. These changes are summarized in table 7 and discussed in more detail below.

Software development methodology: The Transformation program transitioned from a traditional waterfall approach to an Agile one. Agile software development is consistent with industry best practices and existing incremental development requirements. Further, our work has shown that, if performed effectively, this approach can provide more flexibility to respond to changing agency priorities, and allow for easier incorporation of emerging technologies and termination of poorly performing investments with fewer sunk costs.

Under the previous approach, USCIS ELIS was to deploy the program under five major releases. The non-immigrant line of business was to be delivered across two releases, A and B. The three remaining lines of business—immigrant, humanitarian, and citizenship—were to be delivered through releases C, D, and E, respectively. Under the Agile approach, the program plans to deliver smaller sets of functionality every 4 weeks. These shorter-term releases are intended to culminate in a more complete release of a USCIS business line in generally no longer than 6 months. The program estimates it will move through 16 releases prior to achieving full operating capability.

Contracting approach: In October 2013, the DHS Deputy Chief Procurement Officer approved a revision to the program’s acquisition plan. This updated acquisition plan reflected a transition from a primary contract for a single solution architect to a series of contracts for tasks that had previously been provided by the solution architect (e.g., requirements development and testing).

The Office of Transformation Coordination, with support from the Office of Information Technology, has assumed responsibility for integrating and overseeing the work performed under at least nine different contracts. For example, training of USCIS personnel who will interact with USCIS ELIS is provided through one contract, while integration of software code from multiple development teams will be handled through a different contract. In addition, software development will be handled by multiple contractors, each operating on 6-month contracts that can be renewed depending on the performance of the contractor.

According to USCIS officials and briefings to senior leadership, this new acquisition plan is to allow the Transformation Program to transition away from a solution that delivered deficient software code and an overly complex solution. However, the new acquisition approach also introduces risk to the program. According to documentation supporting the Transformation Program acquisition plan, the primary risk of this approach is that the contract incentive structure may not lead to cooperative behavior on the part of the various contractors. Our prior work on approaches of this nature has also shown that as the number of contracts and contractual relationships increase, so does program complexity. Further, the schedule and performance risks that arise from higher program complexity typically result in greater costs.

Program architecture: In March 2013, the Transformation program was granted approval by its executive steering committee to move to a new, more modern, solution architecture. Design and development of this new architecture began in September 2012 and was intended to simplify the existing USCIS ELIS architecture by using fewer commercial products and including open source software; support enterprise solutions that could be reused across USCIS rather than limiting their use to USCIS ELIS; and allow for easier scalability to accommodate surges in the benefits processing by using cloud services.

According to the USCIS Chief Information Officer, none of the hardware or software from the existing architecture will be used as part of the new architecture, with the exception of some licenses associated with risk and fraud services. Accordingly, the program will rebuild functionality previously developed in releases A1 through A2.5, such as support for the I-526 form. The program plans to build this functionality in releases 5.1, 5.2, and 5.3. Appendix III provides additional information on the capabilities (releases) of USCIS ELIS as well as when they are to be made available.

To transition to and test the efficacy of the new architecture, USCIS planned a limited and full deployment for the replacement and renewal of permanent resident cards (form I-90) for the first release of system functionality. The Transformation Program conducted the limited deployment in November 2014. In May 2015, the Director of Operational Test and Evaluation signed off on an operational assessment of the limited deployment.

According to the USCIS Chief Information Officer, the three key changes made to the acquisition strategy were in conjunction with a March 2012 OMB-facilitated TechStat review with DHS of this program. As part of the root-cause analysis to inform the TechStat review, the USCIS Chief Information Officer noted that delays and cost overruns were partly the result of the solution architect delivering deficient software code and performing at an unacceptably low rate of productivity.

Despite a Lack of Formal DHS Approval of Current Plans, Transformation Has Proceeded with System Acquisition Work

The Transformation program’s current baseline estimates and plans have yet to be formally approved by the DHS Undersecretary for Management (the acquisition decision authority for this program); as a result, the program has not been in compliance with DHS acquisition policies and procedures since 2013. According to these policies and procedures, any breaches to the approved acquisition program baseline (cost, schedule, and performance parameters) require that, within 90 days, a new baseline be approved or a program review be conducted to review the proposed baseline revisions and recommendations to the acquisition decision authority.

For the Transformation Program, neither of these actions has taken place since 2013—when the program breached its approved acquisition program baseline (from July 2011). According to the component lead in PARM, a new baseline was not approved because supporting acquisition planning documentation was not yet mature. Despite the lack of an updated and approved program baseline, USCIS has begun to implement its current system acquisition plans. For example, in October 2013 and February 2014, the Acting Under Secretary for Management allowed USCIS to award a series of contracts to develop the pilot for the new architecture and re-build existing functionality, as well as to proceed with planning and development of future releases. According to the USCIS component lead for PARM, development of the USCIS ELIS system is limited under the approved contracts. However, according to USCIS planning documents, the releases currently being developed make up at least 66 percent of the total capabilities of USCIS ELIS.

According to program officials, an acquisition decision event is expected to occur at the beginning of April 2015, which should bring the Transformation Program back into compliance with DHS policy. Specifically, the acquisition decision (referred to as 2B in the DHS acquisition life cycle) is intended to obtain the DHS Under Secretary for Management’s approval of a new acquisition program baseline. However, as discussed later in this report, this acquisition decision event has already been rescheduled twice by the department—initially set to occur in September 2013 and then in September 2014. Figure 4 provides a timeline of past and current program events.

Until the Transformation program has a revised program baseline (reflecting the significant changes in strategy) that has been approved at the departmental level, program accountability for cost, schedule, and operational commitments will continue to be limited.

Changes to Transformation Program’s Acquisition Strategy Have Contributed to Missed Cost Savings and Deferral of Operational Efficiencies and Other Benefits

The changes in the Transformation Program’s acquisition strategy have significantly impacted the program’s planned schedule targets, which in turn have had negative effects on USCIS’s ability to achieve cost savings, operational efficiencies, and other benefits.

In justifying a transition to the new architecture, the program projected an accelerated completion date of August 2016. Since then, draft acquisition documents have pushed the completion date out to no later than March 2019. According to program officials, the delay in getting to full operational capability is due in part to unexpected or greater-than- expected challenges in implementing the software methodology, contracting, and architecture changes. For example:

The development and test environments took longer than expected due the complexity of standing up the environments for use.

A bid protest of the flexible Agile development services contract required the program to adjust the schedule and extend the solution architect contract and also contract for another team to continue development work until the bid protest was resolved and development work could be initiated under the contract. According to the Chief of the Office of Transformation Coordination, the temporary development teams that were used while the bid protest was ongoing performed more slowly than what was projected for permanent development teams.

The program determined that it would be unable to re-use existing software and hardware purchased by the solution architect and in use by USCIS. As a result, work performed under the solution architect contract would need to be redeveloped as part of future releases. This additional work was not initially planned, and therefore expected to require additional effort.

The schedule delays in system delivery have in turn hampered USCIS’s ability to realize cost savings associated with the Transformation Program. For example, the business case for the program highlighted cost savings that would be realized from decommissioning of legacy systems upon full deployment of USCIS ELIS. Each of these legacy systems must remain operational to allow USCIS to perform its mission until an alternative option is available. In fiscal year 2014, the total cost of maintaining systems that could have been decommissioned if USCIS ELIS had been fully operational was approximately $71 million. In addition, the business case for the Transformation Program also identified cost savings from reducing data entry and mail handling costs. USCIS will continue to incur such costs while the program awaits full implementation.

In addition, the schedule delays have deferred USCIS’s ability to realize operational improvements tied to the program. For example, the Transformation Program is expected to implement organizational and business process changes to better use IT. According to USCIS, this increased use of IT should help achieve goals such as reducing the immigration benefit backlog through business process change; improving customer service through expanded electronic filing; and enhancing national security by authenticating users and integrating with external agency databases. Due to delays in the program, these improvements have yet to be achieved.

Also, we have previously found weaknesses with existing processes and systems by which USCIS processes benefits. The agency has cited the Transformation Program as one effort that will help to address these concerns. For example, in our December 2014 report, we found that collection and maintenance of data for the K nonimmigrant visa process (form I-129F) were not reliable or were not collected or maintained in a reportable format. However, according to USCIS Service Center Operations officials, the agency will be able to collect and maintain more complete data through the deployment of the electronic I-129F petition in USCIS ELIS. Thus, until the new system is deployed, weaknesses in these existing processes may continue.

DHS Oversight Bodies Need to Improve Governance and Oversight of the Transformation Program

DHS acquisition policy identifies a process by which governance bodies should evaluate cost, schedule, and performance to make decisions about individual programs. Based on this process, multiple bodies govern the program, requiring corrective actions when needed. In addition, the department’s OCIO and PARM offices are tasked with conducting evaluations of individual programs, which inform congressional and OMB oversight. However, a lack of reliable information against which to monitor the Transformation Program has inhibited the ability of these governing and oversight bodies to make informed decisions.

Governance Bodies Have Mostly Taken Actions Aligned with Leading Practices but Decisions Are Based on Unreliable Information

The department’s acquisition policy requires that the DHS Acquisition Review Board support the Under Secretary for Management by reviewing major acquisition programs for proper management, oversight, accountability, and alignment to the department’s strategic functions at key decision points and other meetings, as needed. DHS acquisition policy establishes that the review board be chaired by the acquisition decision authority and consist of individuals that manage DHS’s mission objectives, resources, and contracts. Table 8 identifies the members of the Transformation Program’s Acquisition Review Board.

In May 2012, the Under Secretary for Management chartered the Executive Steering Committee for the Transformation Program to help improve program governance. In contrast to the Acquisition Review Board, this committee assumed authority to oversee all aspects of the execution of the Transformation Program between key decision points. The Executive Steering Committee includes voting and non-voting members from DHS and USCIS. Table 9 provides a list of the committee’s voting and non-voting members.

Leading practices that we and others have identified note that oversight is a critical element of an investment’s life cycle. To be effective, governance bodies should, among other things:

Monitor a project’s performance and progress toward predefined cost and schedule expectations: Governance bodies should measure the actual value of project planning parameters, compare actual values to estimates in the plan, and identify significant deviations.

Ensure that corrective actions are identified and assigned to the appropriate parties at the first sign of cost, schedule, and/or performance problems: Governance bodies should collect and analyze issues based on predefined expectations and determine corrective actions to address them.

Ensure that these corrective actions are tracked until the desired outcomes are achieved: Governance bodies should track the implementation of all corrective actions until the desired outcomes occur.

Rely on complete and accurate data to review the performance of IT projects and systems against stated expectations: Governance bodies should integrate, measure, and analyze activities into the processes of the project, including tracking actual progress and performance against established plans and objectives. Analysis should account for the quality (e.g., age, reliability) of all data that are used for the analysis, regardless of the source of the data. The quality of data should be considered to help select the appropriate analysis procedure and evaluate the results of the analysis.

Table 10 shows the extent to which each of the governance bodies met the leading practices for performing oversight.

As shown, the two governance bodies have implemented three of the leading practices:

Acquisition Review Board: The board implemented the leading practice associated with ensuring corrective actions are identified. For example, during a program review on February 16, 2012, the board discussed a request to provide the program with an additional $21.5 million to cover unplanned costs resulting from delays in the program schedule. Based on this discussion, the review board identified a series of action items, including a request for USCIS to return for an additional review after delivery of the first release to request approval to commit additional funding for development and delivery of the next release. The Acquisition Decision Authority committed additional funding in May 2012.

Executive Steering Committee: This body implemented two of the leading practices. In particular, the committee fully ensured that corrective actions were identified, and that corrective actions were tracked until the desired outcomes were achieved. As part of its reviews, the committee assigned corrective actions to specific individuals and entities and tracked them until they had been addressed. For example, in July 2013, the Executive Steering Committee tasked the Office of Transformation Coordination with forming a working group to propose potential options for sequencing of forms and associated business processes to be included in releases 10.0 through 16.0. This corrective action was tracked and subsequently completed in September 2013.

The governance bodies partially implemented three leading practices:

Acquisition Review Board: The board partially addressed two practices associated with monitoring cost and schedule against predefined parameters and following corrective actions through to completion. For example, with respect to monitoring against predefined parameters, in February 2012, the review board evaluated the planned costs of release A against the approved baseline amount to determine if a request for additional funding was justified. However, in February 2014, the program approved awarding a series of contracts but did so without an approved acquisition program baseline.

With respect to corrective actions, in a July 2013 acquisition decision memorandum, the Transformation Program was directed to take the corrective action of returning to the Acquisition Review Board for a re- baseline decision by September 2013 in order to establish new cost and schedule parameters against which to evaluate the program. A February 2014 acquisition decision memorandum again directed the program to return for a re-baseline decision by September 2014. However, as of February 2015, a re-baseline decision had not occurred.

As another example, in a May 2012 acquisition decision memorandum, the Transformation Program was directed to submit a Systems Engineering Life Cycle Tailoring Plan by June 2012 that was to document the Agile approach. According to the component lead in PARM, this document was not submitted for approval.

Executing Steering Committee: The committee partially addressed one practice associated with monitoring cost and schedule against predefined parameters. For example, in April 2013, the committee reviewed and approved the funding of approximately $31 million for IT to support the Transformation Program. However, this evaluation did not include an assessment of these costs against an approved baseline for IT funding. Officials from the Office of Transformation Coordination explained that the approval of funds was based on a DHS-approved life-cycle cost estimate from April 2013. However, this cost estimate was not supported by an approved schedule, current requirements, and the other documentation required for a complete understanding of the program, which would be provided by an approved baseline.

Finally, the program’s two governance bodies did not implement two leading practices:

Acquisition Review Board: The board did not make decisions based on reliable cost, schedule, and performance information. For example, in October 2013, the Acquisition Decision Authority approved a new contract for continued development of the new architecture. In addition, in February 2014, the authority approved a bridge contract for the solutions architect to, among other things, assist in developing the new architecture. However, operational requirements along with an integrated master schedule and cost estimates had not been approved to support these decisions. This is consistent with our previous findings. For example, in November 2011, we found that USCIS managed the program without specific acquisition management controls, such as reliable schedules. As such, the review board did not have reasonable assurance that it could meet its future milestones or that it is developing a system that will achieve performance goals. A summary of the corresponding recommendations we made and their status is discussed earlier in this report.

Executive Steering Committee: The committee did not make decisions based on reliable cost, schedule, and performance information. For example, in March 2013, the committee voted unanimously to migrate to a new architecture for the Transformation Program. This approval was based, in part, on the cost analysis reported by USCIS. However, this analysis included cost savings that did not fully account for the added costs for merging and migrating data from the old architecture. In addition to excluding costs for data migration, the cost analysis also projected significant cost savings to occur based on an accelerated date for full operating capability. Specifically, that analysis was based on an August 2016 date, but the full operating capability date in the draft baseline is no later than March 2019. Officials from the Office of Transformation Coordination added that the Executive Steering Committee decision was also based on how the simplification of the architecture would reduce risks to performance of the system. Nevertheless, the decision was not supported by reliable cost and schedule information.

Further, the committee approved capabilities to develop and deliver in each release. However, the program did not have approved documentation tying approved operating requirements to capabilities and features on which such decisions could be made. Additionally, as with the Acquisition Review Board, the committee relied on cost and schedule information that had not been approved.

Until these governance bodies can base their reviews of performance on timely, complete, and accurate data, they will be limited in their ability to make timely decisions and to provide effective oversight.

Key External Reporting to Support Program Oversight is Based on Unreliable Information

In addition to governance from the decision-making bodies previously discussed, two DHS offices—PARM and OCIO—assist in the oversight of the Transformation Program. In particular, these offices perform periodic reviews of the Transformation Program to assess general program health and risk. These assessments are shared with external stakeholders with oversight responsibilities, including OMB, Congress, and, in the case of the OCIO, reported publicly on the Federal IT Dashboard. However, these assessments of the Transformation Program reflected unreliable and, in some cases, inaccurate information.

PARM develops periodic reports to ensure that various DHS programs and their components within the agency satisfy compliance-related mandates and improve investment management. To support these efforts, PARM primarily draws on information reported to DHS’s official system of record for acquisition program reporting. PARM’s assessments are also informed by its participation in oversight meetings. For the Transformation Program, the PARM component lead for USCIS attended a majority of the Executive Steering Committee meetings from 2012 through 2014, among other things.

PARM reported assessments of the Transformation Program in its annual Comprehensive Acquisition Status Reports for calendar year 2011 and fiscal years 2013 and 2014, but these assessments were not always based on reliable information. In its most recent status report, dated June 2014, PARM reported that the program was shown to be on target for cost and schedule performance targets. The key events for the next 12 months included software development for releases 7 and 8, with completion dates planned for December 17, 2014, and March 31, 2015, respectively. However, as of June 2014, completion dates were revised with only release 5.0 scheduled for deployment in early 2015. Release 7 was re-scheduled for December 2015, and release 8 for April 2016. According to the USCIS PARM component lead, the report pulled data from March 2014 and the program had not yet updated its acquisition program reporting system with revised deployment dates. Further, as previously discussed, revised cost and schedule baselines were not established for the program after 2011 so any targets reported in the status report reflected unapproved parameters reported by the program.

We reported in March 2015 on data issues and reporting limitations affecting the accuracy of information contained in DHS’s acquisition program reporting system and the reliability of acquisition status reports departmentwide. To help address these concerns, we recommended that DHS determine a mechanism to hold programs accountable for entering data in its reporting system consistently and accurately, and to adjust reporting requirements to include, among other things, reasons for any significant changes in a program’s acquisition cost, quantity, or schedule from the previous report. DHS concurred with the recommendations and, if they are appropriately implemented, the reliability and utility of these reports for decision making purposes should be improved.

OCIO’s Enterprise Business Management Office also assists in overseeing the Transformation program. In particular, it performs periodic reviews that serve as the basis for program ratings that are publicly reported on the Federal IT Dashboard. To support its assessments, OCIO primarily draws on information reported to the Investment Management System—DHS’s official system of record for reporting business cases to the Office of Management and Budget. For the Transformation Program, Enterprise Business Management Office representatives also participate in the Executive Steering Committee and Program Management Review meetings.

The Enterprise Business Management Office performed four assessments of the Transformation Program from June 2013 through June 2014, but these assessments were based on unreliable or conflicting data. During that period, the program was evaluated as either a moderately high or medium risk IT program. The most recent assessment from June 2014 assessed the program as a category 3, medium risk investment, which is an improvement from the February 2014 rating of 2, moderately high risk. The assessment states that the program underwent a re-baseline for release 5.0 and, as a result, reported an acceptable schedule variance and positive cost performance. Further, it stated that an operational analysis of USCIS ELIS was completed in August 2013. However, as discussed previously in this report, the program has experienced over a four-year delay in its schedule and has not performed a re-baseline to bring it back within cost and schedule thresholds.

In addition, the operational analysis performed in August 2013 pertained to the old USCIS ELIS architecture, which is scheduled for decommissioning, and did not reflect the new USCIS ELIS architecture. Instead, the Transformation Program has not yet been able to successfully test to any of its key performance parameters.

Until the OCIO bases its review of performance on timely, complete, and accurate data, its ability to effectively provide oversight and to make timely decisions may be limited. In addition, it risks reporting information on the Federal IT Dashboard that is inaccurate or otherwise misleading, thus limiting OMB and congressional oversight.

Conclusions

The Transformation Program presents a significant opportunity for USCIS to address long-standing challenges by reengineering the manner in which it conducts business. However, the program has experienced delays of over four years and a projected cost overrun of approximately $1 billion since it first established its initial baseline. Each delay means that more time will pass until USCIS is fully prepared to provide better service to those applying for immigration and citizenship benefits and until USCIS realizes the many benefits that the program is intended to provide, such as improved customer service through expanded electronic filing and enhanced national security by authenticating users and integrating with external agency databases.

Changes made to the program’s acquisition strategy were intended to help mitigate past technical and programmatic challenges; however, the current plans have yet to be approved in accordance with departmental policy. Among other things, the Transformation Program is proceeding with a substantial amount of system development work without a current and approved acquisition program baseline. As we have previously concluded, developing a system without first establishing parameters around which the system will be developed is risky and can result in additional cost overruns and schedule delays. Moreover, program accountability will continue to be limited in the absence of such parameters.

While Transformation’s various oversight bodies are active in their respective roles, decisions about the program had been made with incomplete and inaccurate data. In addition, key program health assessment reports shared with OMB and Congress are also unreliable. The ability of USCIS, DHS, Congress, and OMB to effectively monitor program performance and make informed program decisions will continue to be limited until department-level governance and oversight bodies more effectively use reliable program information to inform their program evaluations.

Recommendations for Executive Action

To more fully understand the impact of recent changes to the Transformation Program and help ensure improved Transformation Program governance and oversight, we are making four recommendations for the Secretary of DHS to direct to components within DHS.

To help ensure that progress made by the Transformation Program can be monitored against established and approved parameters, we recommend that the Secretary of DHS direct the department’s Under Secretary for Management to re-baseline cost, schedule, and performance expectations for the remainder of the Transformation Program.

To improve Transformation Program governance, we recommend that the Secretary of DHS direct the Under Secretary for Management to ensure that the Acquisition Review Board is effectively monitoring the Transformation Program’s performance and progress toward a predefined cost and schedule; ensuring that corrective actions are tracked until the desired outcomes are achieved; and relying on complete and accurate program data to review the performance of the Transformation Program against stated expectations.

To improve Transformation Program governance, we further recommend that the Secretary of DHS direct the DHS Under Secretary for Management, in coordination with the Director of US Citizenship and Immigration Services, to ensure that the Executive Steering Committee is effectively monitoring the Transformation Program’s performance and progress toward a predefined cost and schedule and relying on complete and accurate program data to review the performance of the Transformation Program against stated expectations.

To help ensure that assessments prepared by OCIO in support of the department’s updates to the federal IT Dashboard more fully reflect the current status of the Transformation Program, we recommend that the Secretary of DHS direct the department’s Chief Information Officer to use accurate and reliable information, such as operational assessments of the new architecture and cost and schedule parameters approved by the Under Secretary of Management.

Agency Comments and Our Evaluation

We provided a draft of this product to DHS for comment. In its written comments, reproduced in appendix IV, DHS disagreed with our conclusion about the impact of the changes made to the Transformation acquisition strategy on the program’s cost and schedule, but concurred with all four of our recommendations. DHS fully addressed one of our recommendations and provided plans of action for the remaining three. DHS also provided technical comments that we have incorporated into the report as appropriate.

Regarding our conclusion, the department disagreed that changes to the acquisition strategy delayed the program and added $1 billion to the overall cost, citing changes in the time period covered by each program cost estimate. Further, DHS did not believe we adequately considered the potential benefits and lower risks that resulted from revising the acquisition strategy for the program, stating that the original approach would have far exceeded initial cost estimates and schedule.

Our report did not assess how much the program would have cost or when it would have delivered a complete solution if USCIS continued to pursue its original approach. Nevertheless, we maintain, as stated earlier in this report, the acquisition program baseline approved in May 2015 reflects delays of nearly four years and approximately $1 billion in additional cost when compared to the program’s July 2011 program baseline. Our report also documents that cost increases and delays in achieving full operational capability were due, in part, to unexpected or greater-than-expected challenges in implementing the program’s new approach. In response to the department’s comment about the time period covered by the new cost estimate, we added additional detail to the report to reflect that the newly-approved program baseline covers an additional 11 years beyond what was addressed in the initial program baseline. Even taking this into account, these additional years only account for about $536 million of the increased costs.

Concerning the statement that we did not adequately consider the potential benefits and lower risks of revising the acquisition strategy, we disagree and stand by our analysis and findings. With respect to potential benefits, we reported that some of the changes have the potential to result in improvements. For example, our work has shown that, if performed effectively, an Agile approach to software development can provide more flexibility to respond to changing agency priorities and allow for easier incorporation of emerging technologies. However, other changes may introduce additional program risk. For example, our prior work has shown that program complexity increases as the number of contracts and contractual relationships increase. Further, the schedule and performance risks that arise from higher program complexity can result in greater program costs.

Regarding our recommendation to re-baseline cost, schedule, and performance expectations for the remainder of the Transformation Program, DHS provided evidence that it has been fully implemented. Specifically, DHS provided an approved acquisition program baseline and supporting documents. These documents demonstrate that the department has approved a re-baseline of cost, schedule, and performance expectations for the remainder of the Transformation Program.

DHS concurred with our recommendation that the Acquisition Review Board improve its governance of the program and described specific actions taken that they believe will fully address it. For example, DHS cited a signed memorandum of agreement outlining specific metrics related to cost, schedule, and technical performance. It also cited procedures to ensure acquisition decision memorandum actions are tracked until desired outcomes are achieved and approval of all required acquisition documentation prior to a re-baseline. If implemented effectively, these actions have the potential to fully address our recommendation.

DHS concurred with our recommendation on improving the governance of the Transformation Program’s Executive Steering Committee. The department also described planned actions to address the recommendation, including ensuring that cost and schedule data are presented and evaluated against the Acquisition Program Baseline. The department estimated that it will be able to demonstrate successful resolution of this recommendation by December 31, 2015. If implemented effectively, these actions have the potential to fully address our recommendation.

DHS also concurred with our recommendation to ensure that the department’s Chief Information Officer updates the federal IT Dashboard using accurate and reliable information about the program. The department described actions it has taken and plans to take to address this recommendation. For example, the DHS Chief Information Officer has established an Integrated Product Team with PARM to identify gaps in its assessment processes and establish better coordination to ensure the timely availability of updated Acquisition Program Board information. In addition, the department stated it would take steps to improve oversight and data quality by consolidating two automated tools into a single enterprise information management and repository system. The department estimated that it will be able to demonstrate successful resolution of this recommendation by December 31, 2015. If implemented effectively, these actions have the potential to fully address our recommendation.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to interested congressional committees and the Secretary of Homeland Security. This report will also be available at no charge on our website at http://www.gao.gov.

If you or your staffs have any questions on matters discussed in this report, please contact me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) determine the status of the Transformation Program, including the impact of changes made to the acquisition strategy and (2) assess the extent to which the Department of Homeland Security (DHS) and the U.S. Citizenship and Immigration Services (USCIS) are executing effective program oversight and governance.

To determine the status of the Transformation Program and the impact of changes made to the acquisition strategy, we reviewed and analyzed recent program planning documentation and compared it to past documentation and previously reported issues. Specifically, we reviewed recent documentation such as a draft acquisition program baseline, test and evaluation master plan and operational requirements, the acquisition plan, annual budget figures, budget justification reviews, the annual operating plan, capabilities and constraints documents, contracts and associated statement of work, life-cycle cost estimate, privacy impact assessments, system design documents, team process agreement, and the Transformation road map.

To identify differences in past and current program expectations, we reviewed past program documentation, such as the acquisition program baseline, acquisition plan, business case analysis for process alternatives, concept of operations, cost estimating baseline document, exhibit 300 submissions to the Office of Management and Budget, life- cycle cost estimate, mission needs statement, operational assessments and the associated operational requirements document, program management plan, and solution architect contract and statement of work. For example, we compared the operational requirements document approved in 2011 to a draft operational requirements document from 2014 to determine changes in operating requirements and measures used in operational testing. We also compared the cost savings and benefits in the business case of process alternatives to release-specific capabilities and constraints documents along with other planning documentation to determine if recent changes to the program would impact cost savings and benefits intended for the program.

In addition, we reviewed information from DHS and USCIS oversight entities, such as meeting minutes and slide decks from the Executive Steering Committee to further understand program changes. We also reviewed related GAO and DHS Inspector General reports to capture previously identified issues encountered by the Transformation Program.

To determine the extent to which DHS and USCIS are executing effective program oversight and governance, we reviewed DHS acquisition management policy, analyzed roles and responsibilities, and reviewed the execution of these roles and responsibilities against relevant policy, guidance, and leading practices. Specifically, we identified DHS and USCIS policy for acquisition management such as Acquisition Management Directive 102-01 to understand the DHS and USCIS program oversight entities and expectations. We also identified criteria for effective governance and oversight based on practices captured in the IT Investment Management Maturity Framework and the Capability Maturity Model Integration for Development. We reviewed and analyzed charters of oversight bodies, meeting minutes, presentation slides and supporting materials, and after action reports of various oversight entities, including the Acquisition Review Board, Executive Steering Committee, the Office of the Chief Information Officer (OCIO), the Office of Program Accountability and Risk Management (PARM), the Component Acquisition Review Board and the Product Management Team.

Based on this information, we assessed the extent to which the Acquisition Review Board and Executive Steering Committee had executed key governance and oversight practices. We assessed a governance body as having implemented a practice if the practice was shown to have been consistently applied (at least 80 percent of the time), partially implemented if the practice was applied but on an inconsistent basis (at least a quarter of the time), and not implemented if the practice was not applied.

To determine the reliability of assessments produced by PARM, we reviewed policy for the comprehensive acquisition status reports to understand the criteria for assessing major acquisition programs. We reviewed the master acquisition oversight list from 2010 through 2014 to confirm that the Transformation Program was properly included in such reviews and included in that list. We analyzed comprehensive acquisition status reports in which the Transformation Program was identified. These reports include calendar year 2011, fiscal years 2013 and 2014. We compared these reports against current acquisition planning documentation to gauge the accuracy of the information. For example, we compared the tasks reported on the comprehensive acquisition status reports that were planned for the next 12 months to the program schedule presented to the Executive Steering Committee for that same time period.

We interviewed PARM officials to discuss any gaps. We also reviewed a prior GAO report covering PARM oversight and reporting for the comprehensive acquisition status report and the reliability of information in the system supporting the process. In that report, GAO determined that the data in the system were not sufficiently reliable for our purposes, as also discussed in this report. Moreover, PARM officials acknowledged that there were data accuracy issues with the system.

To determine the reliability of assessments produced by OCIO, we reviewed the office’s program assessment scoring and Federal IT Dashboard reporting guides to understand the criteria for assessing major IT programs. We analyzed all reports and underlying scorecards for the Transformation Program. These reports covered from June 2013 through June 2014. We compared these reports against current acquisition planning documentation to gauge the accuracy of the information. For example, we compared information contained in the assessment narrative to presentations made before oversight bodies for that same time period. We interviewed OCIO officials to discuss any gaps. We determined these assessments were unreliable as discussed in the report.

We conducted this performance audit from August 2014 to May 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: USCIS ELIS Interfaces

The Transformation Program is to link developed systems, enterprise services, and existing systems and capabilities to enable end-to-end processing. The central project in this portfolio is the USCIS Electronic Immigration System (USCIS ELIS), which is to enable electronic filing and adjudicative case management. USCIS ELIS is to function as a component of a larger architecture consisting of multiple systems, services and interfaces. In particular, according to USCIS, USCIS ELIS is to interface with existing systems, some of which are to be decommissioned as the program is fully deployed. These components must all function together as a whole to enable USCIS to operate in a person-centric, paperless environment. Table 11 describes systems and services with which USCIS ELIS is planned to interface.

Appendix III: Planned Transformation Program Capabilities to be Delivered by USCIS

The Transformation Program plans to be fully operational no later than March 2019. In order to be fully operational, the program intends to develop and deploy capabilities incrementally across various releases, beginning with release 5.0. Table 12 describes the planned capabilities to be delivered by release according to the Transformation Program road map, as of December 2014.

Appendix IV: Comments from the Department of Homeland Security

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, individuals making contributions to this report included Michael Holland (assistant director), Mathew Bader, Kathryn Bernet, Nancy Glover, Martin Skorczynski, Nathan Tranquilli, and Johnathan Wall.