Immigration Benefits System: Significant Risks in USCIS's Efforts to Develop its Adjudication and Case Management System
Released: 2017-03-16T00:00:00

Letter

Background

The goals of the Transformation Program are to modernize the paper- based immigration benefits process, enhance national security and system integrity, and improve customer service and operational efficiency. Established in 2006, the program comprises many systems, each of which provides a service to facilitate operations, such as identity management and risk and fraud analytics. The objectives of the Transformation Program are to allow applicants to establish an account with USCIS to file and track the status of the application, petition, or request online; the USCIS Electronic Immigration System (USCIS ELIS), which is the main component of the program, to apply risk-based rules automatically to incoming applications, petitions, and requests to identify potentially fraudulent applications and national security risks; adjudicators to have electronic access to applications, petitions, and requests, relevant policies and procedures, and external databases;

USCIS to have management information to track and allocate

USCIS ELIS to have electronic linkages to other agencies, such as the Departments of Justice and State, for data sharing and security purposes.

As the main component of the program, USCIS ELIS is to provide case management for adjudicating immigration benefits. USCIS ELIS relies on and interfaces with other systems that provide additional capabilities, such as user authentication and scheduling, to deliver end-to-end processing. In particular, as of July 2016, the system was expected to interface with at least 30 other systems, ranging in function from fraud detection to law enforcement and to online payment.

The program expects to achieve its goals and objectives through the delivery of five core operating requirements. These core requirements, which are expected to collectively deliver the program’s mission needs, are (1) intake and account management; (2) benefits case management; (3) electronic content management; (4) agency and knowledge management; and (5) risk and fraud management. Table 1 describes the five core operational requirements.

Program Governance and Oversight

The USCIS Transformation Program is governed by multiple bodies within DHS and the program. Specifically, DHS governance bodies, such as the Acquisition Review Board and Executive Steering Committee, evaluate cost, schedule, and performance of a program and provide corrective actions when needed. In addition, the department’s Office of the Chief Information Officer and Office of Program Accountability and Risk Management conduct oversight of individual programs, which in turn, informs congressional and Office of Management and Budget oversight. For example, the Office of Program Accountability and Risk Management develops periodic reports to ensure that various DHS programs and their components within the agency satisfy compliance-related mandates and improve investment management. In addition, the Office of the Chief Information Officer performs periodic reviews that serve as the basis for publicly reported program ratings.

Within USCIS, directorates govern divisions and program offices govern specific functions, such as the Transformation Program. One program office, the Office of Transformation Coordination, has managed and overseen the development of USCIS ELIS. Other USCIS directorates and program offices, including the Office of Information Technology and the Customer Service and Public Engagement directorate, have supported the management and oversight of the larger USCIS Transformation Program.

Software Development Approach

In April 2015, USCIS officially changed its software development approach. In particular, USCIS transitioned from a waterfall approach to develop, test, and deliver USCIS ELIS functionality to an incremental approach. In an incremental approach, software is developed, tested, and delivered in smaller components or phases, rather than in the typically long, sequential phases of a traditional waterfall approach. Incremental software development is consistent with the Office of Management and Budget’s IT Reform Plan and the law commonly referred to as the Federal Information Technology Acquisition Reform Act (FITARA). The incremental approach chosen by USCIS, called Agile, is intended to allow subject matter experts to validate requirements, processes, and system functionality in increments, and deliver the functionality to users in short cycles.

Transformation Program Has Experienced Significant Cost Increases and Schedule Delays

Since USCIS began implementation of the Transformation Program in 2006, the effort has experienced significant cost increases and schedule delays. In particular, the program’s most recent baseline, approved in April 2015, indicates that the Transformation Program will cost up to $3.1 billion and be fully deployed no later than March 2019.This is an increase of approximately $1 billion with a delay of more than 4 years from its initial July 2011 acquisition program baseline. In November 2008, USCIS awarded a solutions architect contract for approximately $500 million over a 5-year period to design, develop, test, deploy, and sustain the Transformation Program by November 2013. In July 2011, DHS officially approved the Transformation Program’s acquisition program baseline and supporting operational requirements. This baseline estimate was for about $2.1 billion and the program was projected to reach full operating capability no later than June 2014.

In May 2012, USCIS launched the first release of USCIS ELIS. This release included capabilities associated with all of the system’s core operational requirements, such as online account setup, case management, case acceptance, applicant evidence intake, and notice generation. Subsequent to this first release, USCIS deployed additional releases to add functionality. This initial USCIS ELIS was used to process nonimmigrant requests to change or extend their status, USCIS immigrant fees, and immigrant petitions for foreign nationals who make investments in U.S. commercial enterprises.

Beginning in 2013, the program was operating beyond its approved baseline, a situation DHS referred to as being “in breach.” As a result, we reported in 2015 that the program was not in compliance with DHS acquisition policies and procedures. To address the breach, DHS acquisition policies and procedures required that, within 90 days, a new baseline be approved or a program review be conducted to review the proposed baseline revisions. We reported that neither of these actions had taken place since 2013.

However, despite exceeding its approved baseline and operating without a DHS-approved revised acquisition strategy and baseline, the Transformation program continued with its system development efforts. As part of the continued development, USCIS pursued a new acquisition approach. As we reported in May 2015, changes in its approach included changes to the software development methodology, contracting approach, and program architecture and were intended to help address concerns about delays and cost overruns. See table 2 for a description of these changes.

The shift in acquisition approach also resulted in the proposed development of a new USCIS ELIS system. The new system was to be a separate and distinct system from the previously-developed USCIS ELIS. In November 2014, after a year and a half of planning and development, USCIS deployed an initial version of the new USCIS ELIS system to allow for limited processing of permanent resident card renewals and replacements. This initial deployment also tested all core processing capabilities, such as the ability to electronically file or schedule appointments for collecting biometric information.

Between November 2014 and February 2015, the functionality of the new USCIS ELIS was enhanced to allow full processing of permanent resident card renewals and replacements. Nevertheless, during this time, the Transformation Program continued to operate in breach status, without a DHS-approved revised acquisition strategy and baseline. In 2015, we reported that this breach had impacted DHS’ ability to effectively oversee and govern the program because oversight was no longer being conducted relative to a current and approved program schedule.

The program’s revised acquisition baseline and strategy were formally approved in April 2015, over 2 years after the breach. Under its new acquisition baseline, USCIS estimated the program to cost up to $3.1 billion and to be fully deployed no later than March 2019. This was an increase of approximately $1 billion and a delay of over 4 years from the program’s initial approved baseline in July 2011.

The new acquisition strategy represented official DHS approval of the program’s updated acquisition approach and also formally required that the old USCIS ELIS be decommissioned and that USCIS continue to pursue the new USCIS ELIS. In May 2015, we reported that changes in the Transformation Program acquisition strategy intended, in part, to address the breach had contributed to significant delays in the program’s planned schedule.

In July 2016, we once again expressed concerns over program performance, focusing on the reliability of the program’s cost, schedule, and scope measurements. For example, we reported that delays in the schedule could increase the risk that the program would proceed with future USCIS ELIS development or deployment before it was ready in order to meet committed dates. In addition, we reported that the program was at risk of future schedule delays in later USCIS ELIS releases, which might result in the program exceeding its revised cost or schedule thresholds. We made twelve recommendations to improve the ability of USCIS to manage the program. USCIS concurred with our recommendations and has initiated work to implement them.

USCIS continues to expand the ability of the new USCIS ELIS to process citizenship and immigration benefits. Specifically, after April 2016, USCIS deployed the additional capability to process applications for temporary protected status and additional naturalization forms. However, in August 2016, the program reverted back to the legacy system, called the Computer-Linked Application Information Management System 4, for processing the same naturalization form that had been deployed in the new USCIS ELIS only 4 months earlier. As a result of the switchover and other technical issues with the new system, the program did not complete deployment of system functionality associated with its Citizenship line of business by its September 2016 deadline, resulting in another schedule breach.

In November 2016, USCIS submitted a breach remediation plan to the Deputy Under Secretary for Management that identified several root causes for the breach. These causes included that: the program’s schedule did not allow time to gather user feedback or address complexities discovered during development; new requirements were added; and,

USCIS leadership did not provide any consistent performance requirements on what the program was supposed to accomplish for specific business lines.

USCIS had planned to re-baseline the program in February 2017 to account for the September 2016 breach. However, in December 2016, DHS leadership directed USCIS to stop planning and development for new lines of business, update its breach remediation plan and acquisition documentation, and brief leadership on the program’s revised approach by February 2017. The program submitted a revised remediation plan on February 1, 2017. The revised plan was subsequently accepted by the Acting Under Secretary for Management on February 14, 2017.

The revised remediation plan set an expectation that the program would submit revised acquisition documentation for review including a new baseline by February 28, 2017. However, according to the Deputy Chief of the Resource Management Division within the USCIS Office of Information Technology, the program was granted an extension. According to this official, the program expects to discuss the revised acquisition documentation before the Acquisition Review Board in March 2017.

Delays are Impacting the Ability of USCIS to Realize the Benefits of Transformation

The Transformation Program’s delays in delivering system functionality have limited USCIS’s ability to realize its planned cost savings and operational improvements. With respect to cost savings, the program’s business case highlighted cost savings that would be realized from decommissioning legacy systems on full deployment of USCIS ELIS.

However, these legacy systems must remain operational to allow USCIS to perform its mission until an alternative option is available—thus, preventing the associated savings from being realized. For example, in fiscal year 2014, the total cost of maintaining systems that could have been decommissioned if USCIS ELIS had been fully operational was approximately $71 million. Further, the business case for the Transformation Program identified anticipated cost savings from reducing data entry and mail handling costs. With the continuing delays, however, USCIS will continue to incur such costs while the program awaits full implementation.

In addition, the schedule delays have deferred USCIS’s ability to realize operational improvements tied to the program and intended to resolve issues we’ve previously reported. For example, the Transformation Program is expected to implement organizational and business process changes to better use the new electronic system. According to USCIS, this increased use of IT should help achieve goals such as reducing the immigration benefit backlogs through business process change, improving customer service through expanded electronic filing, and enhancing national security by authenticating users and integrating with external agency databases. However, once again, the delays in delivery of the program mean that these improvements have yet to be achieved.

Program Management Challenges Contribute to System Performance Risks

Prior to the Transformation Program’s change in acquisition strategy, USCIS spent more than 8 years and approximately $475 million on developing the old USCIS ELIS. According to program officials, this system was decommissioned in April 2016 due to its instability. As we have reported, the old USCIS ELIS proceeded through development despite challenges in program management and limited oversight. Given this history and the subsequent commitment of additional resources for a new USCIS ELIS, it is more important than ever that USCIS consistently follow key practices associated with software development, systems integration and testing, and contract management and execute effective program oversight and governance.

Inconsistent Software Development Practices Risk Further Program Costs and Delays In July 2016, we reported that the program had at least partially adhered to 7 of 8 key practices for effectively managing Agile software development in producing USCIS ELIS. We reported that the program deviated from key software development practices for various reasons. For example, we reported that the program was not always completing planning for software releases prior to initiating development as required in agency policy. The USCIS Chief Information Officer explained that, although policy requires a program to obtain approval for the scope of each release prior to proceeding with development, this was no longer the practice for the Transformation Program. Instead, approval was granted for six months of development and the scope of that approval was revisited as needed. In contrast, time frames of individual releases were expected to vary depending on the scope of the release.

We also found that the program was not consistently following other key practices and that controls were not always in place to ensure the program adhered to them. For example, with respect to monitoring and reporting on program performance through the collection of reliable metrics, we were unable to track monthly reports on program scope back to the associated software release backlog. In addressing this matter, the Business Integration Division within the Office of Transformation Coordination acknowledged issues regarding traceability. The division subsequently determined that its process for tracking monthly reports on program scope back to the associated software release backlog was not effective since it relied solely on the review of the user stories. The division acknowledged that requirements traceability is critical to avoid scope creep and to demonstrate that the features implemented addressed the mission needs.

With respect to the practice of setting outcomes for Agile software development, which is the only key practice that the program did not fully or partially address, the program did not define Agile software development outcomes. The program provided various reasons for not addressing the practice. For example, the Chief of OTC stated that the outcome or goal for the program is to deploy a product line within baseline cost, schedule, and performance parameters. Nevertheless, we reported that the program had not established a well understood goal, or set of goals, for its transition to Agile development. Table 3 describes the program’s satisfaction of key practices in developing USCIS ELIS.

In not addressing key practices for Agile software development, the program faces added risks in deploying a system that does not meet the cost, schedule, or the performance needs of USCIS. We recommended that the program, with assistance from the department, take seven actions to provide reasonable assurance that it executes Agile software development for USCIS ELIS consistent with its own policies and guidance and follows applicable leading practices. At present, all of these recommendations remain open.

Testing Practices Need Improvement to Address System Performance Risks In this same report, we found that the program was not consistently following key system integration and testing practices. Specifically, the program had fully implemented 1 and partially implemented 2 key practices. For example, the program had established an environment and procedures for continuous integration and was conducting unit and integration, functional acceptance, interoperability, and end user tests, as well as performing code inspection. However, we also found that the program was not consistently adhering to its policies and guidance or meeting benchmarks for unit and integration, and functional acceptance tests, and code inspection. Moreover, test plans, cases, and results were not fully developed for interoperability and end user testing.

We reported that the implementation of systems integration and testing deviated from key practices in part because policy and guidance were not being updated to reflect changes in the approach. For example, with respect to performing continuous testing, the program did not meet its stated goals for continuous testing because, according to the USCIS Chief Information Officer, certain program goals were unrealistic. Table 4 describes the program’s satisfaction of key practices in USCIS ELIS integration and testing.

In addition, program policy and guidance did not always align with key practices in systems integration and testing, nor were controls always in place to ensure adherence to policy and guidance. For example, before a developer could integrate code with that of other developers or development teams, peer inspection by another individual was expected to occur to help ensure that the code met program standards. However, the program had not established controls to monitor the extent to which peer inspection had occurred.

As a result of these findings on systems integration and testing, we reported that the program risked poor system performance after being released to the public. At that time, this risk was already being realized. Specifically, the program had reported experiencing issues with USCIS ELIS as a result of deploying software that had not been fully tested. For example: In June 2015, the Quality Assurance Team reported that production issues, such as bugs and defects, had increased noticeably after the February 2015 deployment.

In July 2015, the Quality Assurance Team reported that defects (originating from either production or development) were becoming a significant part of USCIS ELIS iteration work.

On September 22, 2015, in addition to prior and subsequent outages of the system, the Quality Assurance Team reported that USCIS ELIS was unavailable for approximately 15 hours due to issues with code quality.

On September 24, 2015, USCIS ELIS encountered performance problems that impacted nearly 5,000 cases. Approximately 2,600 of these cases had to be abandoned.

In November and December 2015, the Quality Assurance Team reported that code quality had become a major issue.

In January 2016, the program reported more than 800 minutes in unplanned network outages.

In February 2016, the program reported missing the threshold for USCIS ELIS reliability (e.g. mean time between failure), for two straight months and four of the last six months.

In March 2016, program metrics indicated that production tickets were increasing faster than they could be addressed.

Based on these findings associated with systems integration and testing, we made two recommendations to the program. These recommendations were associated with updating existing policy and guidance for systems integration and testing and considering additional controls. At present, these recommendations remain open.

Contract Oversight Practices Limit Contractor Accountability In the same report, based on a review of six contracts, we found that USCIS had mixed success in implementing selected key contract management internal controls. Specifically, we found that contracting officer representatives were meeting training requirements and USCIS had documented its rationale for not pursuing selected contracts as performance based. However, we also reported that the program could improve contract monitoring to provide reasonable assurance that contractors were meeting program needs. For example, we reported that

The Agile development services contract contained appropriate performance criteria that linked to the program goals, but the program did not clearly define measures against which to analyze differences between services expected and those delivered. Because oversight of contractor performance resides with the Office of Transformation Coordination and management of the contract is the responsibility of the Office of Information Technology, we reported that the need for clearly-defined performance measures was particularly important.

The program maintained some required documentation in contract files and used contractor performance assessments. However, more clearly defined success measures and evaluation against those measures could have alerted the program to issues in systems integration and testing.

Based on these findings, we reported that the agency lacked information for measuring contractor performance and determining if the Office of Information Technology was meeting its objectives in supporting the program. To help improve oversight of these selected contracts, we made three recommendations focused on contract administration and supporting controls. Since then, the agency has taken steps to close these recommendations, successfully closing two of the three. If effectively implemented, these actions should contribute to improved administration of the contracts that we assessed in our review.

DHS Oversight Bodies Need to Improve Governance and Oversight of the Transformation Program We also reported in May 2015 that the program’s two key governance bodies were mostly taking actions aligned with leading practices, but that their decisions were made based on unreliable information. For example, we reported that the Acquisition Review Board ensured that corrective actions were identified when cost, schedule, or performance issues arose. However, the board was not always monitoring performance and progress toward a predefined cost and schedule or ensuring that corrective actions were tracked until the desired outcomes were achieved.

In addition, we found that two DHS offices assisting in overseeing the program—the Office of Program Accountability and Risk Management and the Office of the Chief Information Officer— developed program assessments that reflected unreliable and, in some cases, inaccurate information. For example, the Office of the Chief Information Officer performed four assessments of the Transformation Program from June 2013 through June 2014. The most recent assessment available at the time, from June 2014, stated that the program underwent a re-baseline for release 5.0 and, as a result, reported an acceptable schedule variance and positive cost performance. However, at that time, the program had experienced over a 4-year delay in its schedule and had not performed a re-baseline to bring it back within cost and schedule thresholds.

Based on these findings, we reported that the ability of governance bodies to make timely decisions and provide effective oversight was limited. To help improve program governance, we made four recommendations. Subsequently, the program addressed one recommendation by establishing a new baseline, although that baseline has again been breached. The program has taken steps to close the remaining recommendations, although the recommendations remain open.

In summary, the USCIS Transformation Program began officially pursuing a new acquisition strategy in April 2015 to mitigate risks encountered in developing the original system. However, this new strategy reflects a higher cost estimate and a longer amount of time before the system is fully implemented than the program’s previously approved strategy. In addition, the program is again encountering issues in development and production. If the agency does not address issues in its efforts to develop and test software, oversee contractors, and govern the program it risks additional cost increases, schedule delays, and performance shortfalls. In addition, continued delays limit the program’s ability to achieve critical goals, such as delivering system functionality to enhance customer service and enhancing national security.

Chairman Perry, Ranking Member Correa, and Members of the subcommittee, this completes my prepared statement. I would be pleased to respond to any questions that you may have at this time.

Contacts and Acknowledgments

If you or your staffs have any questions about this testimony, please contact Carol C. Harris, Director, Information Technology Acquisition Management Issues, at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this testimony statement. GAO staff who made key contributions to this testimony are Michael Holland (Assistant Director), Mathew Bader (Analyst in Charge), and Nancy Glover.

Appendix I: Related GAO Products

Immigrant Investor Program: Progress Made to Detect and Prevent Fraud, but Additional Actions Could Further Agency Efforts, GAO-16-828 (Washington, D.C.: Sept. 13, 2016).

Immigration Benefits System: U.S. Citizenship and Immigration Services Can Improve Program Management, GAO-16-467 (Washington, D.C.: July 7, 2016).

Asylum: Additional Actions Needed to Assess and Address Fraud Risks, GAO-16-50 (Washington, D.C.: Dec. 2, 2015).

Immigrant Investor Program: Additional Actions Needed to Better Assess Fraud Risks and Report Economic Benefits, GAO-15-696 (Washington, D.C.: Aug. 12, 2015).

Immigration Benefits System: Better Informed Decision Making Needed on Transformation Program, GAO-15-415 (Washington, D.C.: May 18, 2015).

H-2A and H-2B Visa Programs: Increased Protections Needed for Foreign Workers, GAO-15-154 (Washington, D.C.: Mar. 6, 2015).

Immigration Benefits: Improvements Needed to Fully Implement the International Marriage Broker Regulation Act, GAO-15-3 (Washington, D.C.: Dec. 10, 2014).

H-2A Visa Program: Modernization and Improved Guidance Could Reduce Employer Application Burden, GAO-12-706 (Washington, D.C.: Sept. 12, 2012).

Immigration Benefits: Consistent Adherence to DHS’s Acquisition Policy Could Help Improve Transformation Program Outcomes, GAO-12-66 (Washington, D.C.: Nov. 22, 2011).

Department of Homeland Security: Progress Made and Work Remaining in Implementing Homeland Security Missions 10 Years after 9/11, GAO-11-881 (Washington, D.C.: Sept. 7, 2011).

Immigration Benefits: Actions Needed to Address Vulnerabilities in Process for Granting Permanent Residency, GAO-09-55 (Washington, D.C.: Dec. 5, 2008).

USCIS Transformation: Improvements to Performance, Human Capital, and Information Technology Management Needed as Modernization Proceeds, GAO-07-1013R (Washington, D.C.: July 17, 2007).

Appendix II: Status of GAO Recommendations to Improve Transformation Program Performance

Appendix II: Status of GAO Recommendations to Improve Transformation Program Performance Recommendation 1. Document specific performance measures and targets for the pilots, increments, and the transformed organization that are outcome-oriented, objective, reliable, balanced, limited to the vital-few, measurable, and aligned with organizational goals.

Increase coordination between program office and the Office of Human Capital to ensure transformation and human capital change initiatives are aligned. 3. Plan for the number and types of human resources required in the program office to carry the 4. Plan for obtaining and developing the IT human capital necessary to support the 5. Determine the critical skills and competencies that will be needed to achieve future programmatic results as well as strategies to address gaps in employee numbers, deployment, and skills and competencies. 6. Address continuity in key transformation leadership positions and address impacts to time 7. Use performance expectations and competencies to hold USCIS executives and employees accountable for achieving the goals of the transformation. 8. Continue to develop an enterprise architecture that sufficiently guides and constrains the transformation plans, as DHS works to address limitations in its own enterprise architecture and alignment processes. 9. Complete a comprehensive communication strategy that involves communicating early and often to build trust, ensuring consistency of message, and encouraging two-way communication. Further, the communication strategy should address plans for communicating implementation goals and timelines to demonstrate progress. 10. Complete a comprehensive communication strategy that addresses plans for formally engaging internal and external stakeholders throughout the transformation, and tailors information to meet these stakeholders’ specific needs. 11. Complete a comprehensive communication strategy that addresses plans for a long-term, detailed strategy to share information with employees and stakeholders over the course of the transformation. 12. Document specific performance measures and targets for the pilots, increments, and the transformed organization that are outcome-oriented, objective, reliable, balanced, limited to the vital-few, measurable, and aligned with organizational goals. 13. Develop and maintain an Integrated Master Schedule consistent with these same best practices for the Transformation Program. 14. Ensure that the life-cycle cost estimate is informed by milestones and associated tasks from reliable schedules that are developed in accordance with the nine best practices we identified. 15. Re-baseline cost, schedule, and performance expectations for the remainder of the 16. Ensure that the Acquisition Review Board is effectively monitoring the Transformation Program’s performance and progress toward a predefined cost and schedule; ensuring that corrective actions are tracked until the desired outcomes are achieved; and relying on complete and accurate program data to review the performance of the Transformation Program against stated expectations.

Recommendation 17. Ensure that the Executive Steering Committee is effectively monitoring the Transformation Program’s performance and progress toward a predefined cost and schedule and relying on complete and accurate program data to review the performance of the Transformation Program against stated expectations. 18. Direct the department’s Chief Information Officer to use accurate and reliable information, such as operational assessments of the new architecture and cost and schedule parameters approved by the Under Secretary of Management. 19. Complete planning for software releases prior to initiating development and ensure software meets business expectations prior to deployment. 20. Consistently implement the principles of the framework adopted for Agile software 21. Define and consistently execute appropriate roles and responsibilities for individuals responsible for development activities consistent with its selected development framework. 22. Identify all system users and involve them in release planning activities. 23. Write user stories that identify user roles, include estimates of complexity, take no longer than one sprint to complete, and describe business value. 24. Establish outcomes for Agile software development. 25. Monitor program performance and report to appropriate entities through the collection of 26. Conduct unit and integration, and functional acceptance tests, and code inspection consistent 27. Develop complete test plans and cases for interoperability and end user testing, as defined in the USCIS Transformation Program Test and Evaluation Master Plan, and document the results. 28. Clearly define measures against which to analyze differences between services expected 29. Ensure contracting officer representatives are maintaining complete contract files. 30. Ensure quality assurance surveillance plans are developed when appropriate.

Closed This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.