IMMIGRATION 

BENEFITS SYSTEM

U.S. Citizenship and 

Immigration Services 

Can Improve Program 

Management

Report to Congressional Requestors

July 2016

GAO-16-467

United States Government Accountability Office
United States Government Accountability Office

Highlights of GAO-16-467, a report to 

congressional requesters

July 2016

IMMIGRATION BENEFITS SYSTEM

U.S. Citizenship and Immigration Services Can 

Improve Program Management

Why GAO Did This Study

Each year, USCIS processes millions 

of applications for persons seeking to 

study, work, visit, or live in the United 

States, and for persons seeking to 

become a U.S. citizen. In 2006, USCIS

began the Transformation Program to 

enable electronic adjudication and 

case management tools that would 

allow users to apply and track their 

applications online. In 2012, to address 

performance concerns, USCIS 

changed its acquisition strategy to

improve system development. 

In May 2015, GAO reported that 

USCIS expected the program to cost 

up to $3.1 billion and be fully 

operational by March 2019. This 

includes more than $475 million that 

was invested in the initial version of the 

program’s key case management 

component, USCIS ELIS, which has 

since been decommissioned. 

This report evaluates the extent to 

which the program is using information 

technology program management

leading practices. To perform this 

work, GAO identified agency policy 

and guidance and leading practices in, 

among other things, cost estimation, 

Agile software development, and 

systems integration and testing, and 

compared these with practices being 

used by the program.

What GAO Recommends

GAO is making 12 recommendations

to improve Transformation Program 

management, including ensuring

alignment among policy, guidance, and 

leading practices in areas such as 

Agile software development and 

systems integration and testing. DHS 

concurred with the recommendations.

What GAO Found

U.S. Citizenship and Immigration Services (USCIS) created a reliable updated 

estimate to project the Transformation Program’s cost, but has experienced 

program management challenges. In particular, the program’s cost estimate was 

well-documented and substantially comprehensive, accurate, and credible. 

However, among other things, software development and systems integration 

and testing for USCIS’s Electronic Immigration System (USCIS ELIS) have not 

consistently been managed in line with the program’s policies and guidance or 

with leading practices. 

Regarding software development, the Transformation Program has produced 

some software increments, but is not consistently following its own guidance and 

leading practices. The software development model (Agile) adopted by the 

USCIS Transformation Program in 2012 includes practices aimed at continuous, 

incremental release of segments of software. Important practices for Agile 

defined in program policies, guidance, and leading practices include ensuring 

that the software meets expectations prior to being deployed, teams adhere to 

development principles, and development outcomes are defined. The table below 

lists the program’s status in addressing eight key Agile development practices. 

For example, the program has committed to a specific framework for software 

development, referred to as Scrum, but has deviated from the underlying 

practices and principles of this framework. 

Table: Implementation of Key Agile Practices for USCIS ELIS

Practice Rating

Completing planning for software releases prior to initiating development and ensuring 

software meets business expectations prior to deployment

◐

Adhering to the principles of the framework adopted for implementing Agile software 

development

◐

Defining and consistently executing appropriate roles and responsibilities for individuals 

responsible for development activities 

◐

Identifying users of the system and involving them in release planning activities ◐

Writing user stories that identify user roles, include estimates of complexity, take no 

longer than one sprint to complete, and describe business value

◐

Prioritizing user stories to maximize the value of each development cycle ●

Setting outcomes for Agile software development ○

Monitoring and reporting on program performance through the collection of reliable 

metrics

◐

Source: GAO analysis of USCIS documentation. I GAO-16-467.

Note: ● yes ◐ partial ○ no

The Transformation Program has established an environment that allows for 

effective systems integration and testing and has planned for and performed 

some system testing. However, the program needs to improve its approach to 

system testing to help ensure that USCIS ELIS meets its intended goals and is 

consistent with agency guidance and leading practices. Among other things, the 

program needs to improve testing of the software code that comprises USCIS 

ELIS and ensure its approaches to interoperability and end user testing, 

respectively, meet leading practices. Collectively, these limitations have 

contributed to issues with USCIS ELIS after new software is released into 

production.

View GAO-16-467. For more information, 

contact Carol C. Harris at (202) 512-4456 or 

chac@gao.gov.
Page i GAO-16-467 Immigration Benefits System

Letter 1

Background 4

USCIS Transformation Program Has Not Fully Implemented 

Leading IT Management Practices 21

Conclusions 43

Recommendations for Executive Action 44

Agency Comments and Our Evaluation 45

Appendix I Objectives, Scope, and Methodology 48

Appendix II Management of Agile Software Development for USCIS ELIS 63

Appendix III Management of Systems Integration and Testing for USCIS ELIS 97

Appendix IV USCIS ELIS Interfaces 113

Appendix V Comments from the Department of Homeland Security 116

Appendix VI GAO Contact and Staff Acknowledgments 122

Tables

Table 1: Key Changes to the Transformation Program’s 

Acquisition Strategy 10

Table 2: Key DHS Acquisition Documents Updated Prior to ReBaseline of the USCIS Transformation Program 11

Table 3: Planned Product Lines by Release, as of February 2016 17

Table 4: Adherence to Key Agile Development Management 

Practices 23

Table 5: Adherence to Key Integration and Testing Practices 28

Contents
Page ii GAO-16-467 Immigration Benefits System

Table 6: Dollar Amount for USCIS Contractors for the 

Transformation Program, as reflected in the USCIS 

Transformation Program Acquisition Plan as of March 

2015 58

Table 7: Release Planning and Development Dates 65

Table 8: Planned versus Actual Release Cycles for Major Program 

Releases, as of April 2016 92

Table 9: Transformation Program Sub-Features Delivered by 

Near-Term Releases, as reported to DHS in February 

2016 94

Table 10: Summary of Management of Continuous Integration and 

Testing Tools and Practices 98

Table 11: USCIS ELIS End User Test Plan Evaluation 110

Table 12: USCIS ELIS End User Test Case Evaluation 111

Table 13: USCIS ELIS End User Test Summary Results 

Evaluation 111

Table 14: Interfaces Planned for USCIS ELIS 113

Figures

Figure 1: USCIS Organizational Chart 5

Figure 2: Organizational Structure for the Office of Transformation 

Coordination 6

Figure 3: Organizational Structure for the Office of Information 

Technology 7

Figure 4: Planned Transformation Program Benefit Application 

and Adjudication Process (simplified) 9

Figure 5: Comparison of Agile and Waterfall Software 

Development 13

Figure 6: USCIS Process for Agile Software Development 15

Figure 7: Traceability from Operational Requirements to User 

Stories 16

Figure 8: Scrum Process for USCIS ELIS Release Process 71

Figure 9: Continuous Integration Build Process for USCIS ELIS 99

Figure 10: Unit and Integration Testing Process 101
Page iii GAO-16-467 Immigration Benefits System

Abbreviations 

CIO Chief Information Officer

COR contracting officer’s representative

DHS Department of Homeland Security

ELIS Electronic Immigration System

FADS Flexible Agile Development Services

IT information technology

OIT Office of Information Technology

OTC Office of Transformation Coordination

USCIS U.S. Citizenship and Immigration Services

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. The published product may be reproduced and distributed in its entirety 

without further permission from GAO. However, because this work may contain 

copyrighted images or other material, permission from the copyright holder may be 

necessary if you wish to reproduce this material separately.
Page 1 GAO-16-467 Immigration Benefits System

441 G St. N.W.

Washington, DC 20548

July 7, 2016

Congressional Requesters

Each year, the Department of Homeland Security’s (DHS) U.S. 

Citizenship and Immigration Services (USCIS) processes millions of 

applications for more than 50 types of immigrant and nonimmigrantrelated benefits for persons seeking to study, work, visit, or live in the 

United States, and for persons seeking to become a U.S. citizen.1 Having 

a system to facilitate the processing of immigration benefits and 

citizenship applications in a timely manner and identify fraudulent and 

criminal activity is essential for ensuring the integrity of the immigration 

process.

We have previously reported2 on inefficiencies in USCIS’s current 

benefits processing systems. These systems employ both electronic and 

paper records, and therefore make sharing the information with other 

government agencies (such as those for monitoring criminal or terrorist 

activity) difficult and time-consuming.

 1

Immigrant benefits are for foreign nationals (citizens of another country) who seek to live 

or work in the United States permanently. Nonimmigrant benefits are for individuals 

seeking to enter the United States for a specific purpose, such as tourism or temporary 

employment. Citizenship benefits can be granted to a noncitizen who meets certain 

eligibility requirements and seeks to become a United States citizen. Humanitarian 

benefits are for persons who are brought to the United States or are currently in the United 

States and who are fleeing persecution, require temporary protection from removal, or 

need an extended stay due to emergent circumstances (e.g., those placed in temporary 

protected status, seeking asylum or entering as a refugee, or granted significant public 

benefit parole) as a form of humanitarian aid, such as people in need of shelter or aid from 

disaster, oppression, or other specific circumstances.

2

GAO, Asylum: Additional Actions Needed to Assess and Address Fraud Risks, 

GAO-16-50 (Washington, D.C.: Dec. 2, 2015); Immigrant Investor Program: Additional 

Actions Needed to Better Assess Fraud Risks and Report Economic Benefits, 

GAO-15-696 (Washington, D.C.: Aug. 12, 2015); Immigration Benefits System: Better 

Informed Decision Making Needed on Transformation Program, GAO-15-415 

(Washington, D.C.: May 18, 2015); Immigration Benefits: Improvements Needed to Fully 

Implement the International Marriage Broker Regulation Act, GAO-15-3 (Washington, 

D.C.: Dec. 10, 2014); and H-1B Visa Program: Reforms Are Needed to Minimize the Risks 

and Costs of the Current Program, GAO-11-26 (Washington, D.C.: Jan. 14, 2011). 

Letter
Page 2 GAO-16-467 Immigration Benefits System

In 2006, USCIS embarked on a major initiative—the Transformation 

Program—to enable electronic adjudication and case management tools 

that would allow applicants to apply and track the progress of their 

application online. USCIS planned to complete this initial effort no later 

than June 2014 at a cost of up to $2.1 billion. In May 2015, we reported 

that USCIS expected the program to cost up to $3.1 billion and be fully 

operational no later than March 2019.3 The program reported 

expenditures of nearly $1.2 billion (in then-year dollars) through the end 

of fiscal year 2014. Since that time, the program’s monthly performance 

reports to DHS reported additional expenditures of $192 million through 

February 2016.

According to USCIS, as of April 2016, the program had deployed 

functionality to support the processing of approximately 2.3 of the 7.3 

million applications processed by USCIS each year, or about 31% of the 

total workload. The USCIS Electronic Immigration System (USCIS ELIS) 

is the main component of the Transformation Program.

You asked us to examine USCIS’s effort to modernize its paper-based 

application, case management, and adjudication systems and processes. 

Our objective was to assess the extent to which the USCIS 

Transformation Program is using information technology (IT) program 

management leading practices. To do so, we assessed the extent to 

which USCIS is implementing Agile software development4 for USCIS 

ELIS; adhering to leading practices and agency guidance5 in system 

integration and testing; following leading practices in monitoring the 

largest Transformation Program contractors; and developing a life cycle 

cost estimate for the Transformation Program consistent with our cost 

estimating guidance.

 3

GAO-15-415.

4

Agile software development calls for delivering software in small, short increments rather 

than in the typically long, sequential phases used in a traditional waterfall approach. More 

a philosophy than a methodology, Agile emphasizes early and continuous software 

delivery, as well as using collaborative teams and measuring progress with working 

software. 

5

For the purposes of this report, guidance refers to a variety of USCIS and program-level 

documentation, such as plans, guidelines, and team artifacts. 
Page 3 GAO-16-467 Immigration Benefits System

To assess the extent to which the program is implementing Agile software 

development and is adhering to leading practices and agency guidance in 

system integration and testing, we identified applicable leading practices 

and program policies and guidance. Based on this information, we 

identified eight practices for Agile software development and three 

practices for system integration and testing. We then reviewed relevant 

plans and program documentation and interviewed USCIS staff and 

determined the extent to which the program addressed the practices.

To assess the extent to which the program followed leading practices in 

monitoring its largest contractors, we selected the two largest contracts 

managed by the Office of Transformation Coordination (OTC) and the 

four largest contracts managed by the USCIS Office of Information 

Technology (OIT). We also identified four practices based on applicable 

department and agency policies and procedures and leading practices. 

We then reviewed relevant plans and contract documentation, such as 

monthly performance statements and contract deliverables, and 

interviewed USCIS staff and determined the extent to which the program 

addressed the key practice areas.

To examine the extent to which the cost estimate for the program was 

consistent with leading practices, we relied on the leading practices 

contained in GAO’s Cost Estimating and Assessment Guide.

6 We 

obtained and reviewed documentation and met with key program officials 

and evaluated the program’s practices relative to our cost guide.

We conducted this performance audit from May 2015 to July 2016, in 

accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives. More details about the 

objectives, scope, and methodology can be found in appendix I.

 6

GAO, GAO Cost Estimating and Assessment Guide: Best Practices for Developing and 

Managing Capital Program Costs, GAO-09-3SP (Washington, D,C,: March 2009). 
Page 4 GAO-16-467 Immigration Benefits System

USCIS’ mission is to provide accurate and useful information to its 

applicants, adjudicate immigration and U.S. citizenship benefits, promote 

an awareness and understanding of citizenship, and ensure the integrity 

of the immigration system. According to DHS, on an average day, USCIS 

completes 23,000 applications for various immigration benefits; welcomes 

3,200 new citizens; answers 44,000 phone calls to its toll-free customer 

service line; serves 9,500 customers at 84 local offices; fingerprints and 

photographs 15,000 applicants at 136 application support centers; 

conducts 148,000 national security background checks; and processes 

2,040 petitions filed by employers to bring foreign national workers to the 

United States.

To administer its mission, USCIS is divided into directorates and program 

offices. Directorates are director-led departments in charge of multiple 

divisions. Program offices have a specific function and are led by a chief. 

Figure 1 provides an overview of the USCIS organizational structure.

Background
Page 5 GAO-16-467 Immigration Benefits System

Figure 1: USCIS Organizational Chart

OTC manages and oversees the development of USCIS ELIS, the 

account-based system that allows electronic submission, tracking, and 
Page 6 GAO-16-467 Immigration Benefits System

viewing of case status updates. OTC is supported in managing and 

overseeing the larger USCIS Transformation Program by other USCIS 

directorates and program offices, including OIT and the Customer Service 

and Public Engagement directorate.7

OTC is composed of four divisions: Business Integration, Stakeholder 

Engagement, Program and Resource Management, and Capability 

Delivery. Figure 2 provides an overview of the OTC organizational 

structure.

Figure 2: Organizational Structure for the Office of Transformation Coordination

OIT is composed of six divisions. Within these, the Applied Technology 

Division is further made up of five branches: Strategic Vendor 

Management, Architecture, Delivery Assurance, Program Support, and 

Agile Coaching and Testing. In particular,

The Delivery Assurance branch is expected to support software 

application development, such as USCIS ELIS and supporting systems, 

 7

According to the Chief of OTC, the Customer Service and Public Engagement 

Directorate is responsible for myUSCIS, which will provide a public interface to USCIS 

ELIS.
Page 7 GAO-16-467 Immigration Benefits System

along with infrastructure upgrades through, among other things, quality 

assurance and software release management.

The Agile Coaching and Testing branch is expected to provide support to 

USCIS by establishing and monitoring an automated test framework that 

allows for continuous delivery of software capabilities and providing 

coaching for development and business teams. As part of its coaching 

responsibilities, this branch supports project teams with identification of 

Agile engineering best practices and methodologies, and promotes use of 

such practices. Figure 3 provides an overview of the OIT organizational 

structure.

Figure 3: Organizational Structure for the Office of Information Technology

The goals of the Transformation Program are to modernize the paperbased immigration benefits process, enhance national security and 

system integrity, and improve customer service and operational 

efficiency. Established in 2006, the program comprises many systems, 

each of which provides a service to facilitate operations, such as identity 

management and risk and fraud analytics. The objectives of the 

Transformation Program are to allow

USCIS Intends to 

Modernize Immigration 

Benefits and Adjudication 

Processing
Page 8 GAO-16-467 Immigration Benefits System

• applicants to establish an account with USCIS to file and track the 

status of the application, petition, or request online;

• USCIS ELIS to apply risk-based rules automatically to incoming 

applications, petitions, and requests to identify potentially fraudulent 

applications and national security risks;

• adjudicators to have electronic access to applications, petitions, and 

requests, relevant policies and procedures, and external databases;

• USCIS to have management information to track and allocate 

workload; and

• USCIS ELIS to have electronic linkages to other agencies, such as 

the Departments of Justice and State, for data sharing and security 

purposes.

The main component of the program is USCIS ELIS, which is to provide 

case management for adjudicating immigration benefits. USCIS ELIS 

relies on and interfaces with other systems that provide additional 

capabilities, such as user authentication and scheduling, to deliver endto-end processing. Applications are ingested into the system either 

through the Lockbox8 for paper applications, or an online interface9 for 

electronic applications. Figure 4 provides a simplified depiction of the 

steps involved in the filing and processing of immigration benefits and 

citizenship applications under the Transformation Program.

 8

Historically, USCIS has required applicants, petitioners, or benefit requestors to submit a 

paper submission for immigration applications, petitions, or benefit requests to one of its 

Lockbox facilities. Lockbox facilities are operated by U.S. Department of Treasury financial 

agents on behalf of USCIS to receive paper requests, process payments, and forward the 

requests to USCIS Service Centers in paper and electronic format for further processing. 

As USCIS ELIS expands to new immigration benefit types, the applicable Lockbox facility 

will adjust its legacy delivery process to transmit filings to USCIS ELIS rather than to the 

legacy system. It will transmit the data to USCIS ELIS and the scanned images of the 

paper filings to USCIS ELIS electronic storage.

9

Recently, the program pursued development of a new external customer interface called 

myUSCIS, which is intended to allow customers to directly file applications for immigration 

benefits electronically and access an online account to view their case status, receive 

updates and notices from USCIS, and communicate online with USCIS for assistance with 

or information about their cases. It is managed by the Customer Service and Public 

Engagement Directorate within USCIS, which is separate from OTC.
Page 9 GAO-16-467 Immigration Benefits System

Figure 4: Planned Transformation Program Benefit Application and Adjudication Process (simplified)

In November 2008, USCIS awarded a solutions architect contract to 

International Business Machines Corporation for approximately $500 

million to be allocated over a 5-year period to design, develop, test, 

deploy, and sustain the Transformation Program by November 2013. In 

July 2011, DHS officially approved the Transformation Program’s 

acquisition program baseline. The baseline estimate was for about $2.1 

billion and projected that the program would reach full operating capability 

no later than June 2014.

An initial version of USCIS ELIS was launched in May 2012. This release 

included capabilities associated with all of the core operational 

requirements for the processing of non-immigrant requests to change or 

Transformation Program 

Has Experienced Key 

Changes
Page 10 GAO-16-467 Immigration Benefits System

extend status, such as online account setup, case management, case 

acceptance, applicant evidence intake, and notice generation.

However, leading up to and following deployment, the program office 

encountered a number of challenges with the system. For example, 

according to the program acquisition plan, automation of policies, rules, 

and processes for decision making were later determined to not be 

necessary or cost effective to maintain in the long term; the program had 

used a development approach that did not allow problems to show up 

early enough in the process to remedy; and the system was overly 

complex.

As a result, beginning in 2012, major changes were made to the 

Transformation Program’s acquisition strategy. While transitioning to the 

new strategy, USCIS continued to fund development and enhancement of 

the legacy USCIS ELIS. Changes in the acquisition strategy are 

summarized in table 1.

Table 1: Key Changes to the Transformation Program’s Acquisition Strategy

Key change Previous approach New approach

Contracting approach One contractor that served as the sole 

solution architect and system integrator.

Multiple contractors to provide various services, with 

USCIS serving as the system integrator.

Software development 

approach

Waterfall development, an approach that 

uses long, sequential phases, resulting in 

product delivery years after program 

initiation.

Agile software development, an approach that delivers 

software in small, short increments, resulting in software 

released in phases.

Program architecture Included a large number of proprietary 

commercial off-the-shelf software products, 

which are ready-made and available for 

sale.

Includes open source software, which is publicly 

available for use, study, reuse, modification, 

enhancement, and redistribution by the software’s 

users. This software is to be used in combination with 

fewer commercial off-the-shelf products in a cloud 

computing environment.

Source: GAO-15-415. | GAO-16-467

In April 2015, USCIS updated the acquisition program baseline to reflect 

changes in the underlying acquisition strategy. This new baseline shows 
Page 11 GAO-16-467 Immigration Benefits System

a cost of no more than $3.1 billion10 and full operating capability no later 

than March 2019. As a result of these changes, the program office 

determined that it could no longer maintain the original USCIS ELIS 

system deployed in May 2012, and reported that it had been 

decommissioned in April 2016, even after having invested more than 

$475 million in its development.

As part of the re-baseline, the program also updated supporting 

documentation.11 Table 2 lists the acquisition documents that were 

updated prior to re-baseline approval from DHS and the purpose of each 

document.

Table 2: Key DHS Acquisition Documents Updated Prior to Re-Baseline of the USCIS Transformation Program

Document Purpose

Life cycle cost estimate Provides an exhaustive and structured accounting of all resources and associated cost 

elements required to develop, produce, deploy, and sustain a particular program.

Test and evaluation master plan Documents the overarching test and evaluation approach for the acquisition program. 

Describes the developmental and operational test and evaluation needed to determine a 

system’s technical performance, operational effectiveness/suitability, and limitations.

Acquisition plan Provides a top-level plan for the overall acquisition approach. Describes why the 

solution is in the government’s best interest and why it is the most likely to succeed in 

delivering capabilities to operators.

Operational requirements document Provides a number of performance parameters that must be met by a program to 

provide useful capability to the operator by closing the capability gaps identified in the 

mission need statement.

Integrated logistics support plan Defines the strategy for ensuring the supportability and sustainability of a future 

capability. Provides critical insight into the approach, schedule, and funding 

requirements for integrating supportability requirements into the systems engineering 

process.

 10The program baseline approved in 2011 was based on costs from fiscal year 2006 to 

fiscal year 2022. The program baseline approved in April 2015 was based on costs from 

fiscal years 2006 to 2033. According to the baseline documentation, these additional 

years contribute about $500 million (then-year) to the program’s anticipated costs.

11As part of the re-baseline, the program updated the USCIS Office of Transformation 

Coordination Lifecycle Cost Estimate, USCIS Transformation Program Test and 

Evaluation Master Plan: Electronic Immigration System, Office of Transformation 

Coordination Program Acquisition Plan Update, USCIS Transformation Program 

Operational Requirements Document, and USCIS Transformation Program Integrated 

Logistics Support Plan. It also created the USCIS Transformation Program System 

Engineering Life Cycle Tailoring Plan.
Page 12 GAO-16-467 Immigration Benefits System

Document Purpose

Systems engineering life cycle tailoring plan Documents the selection of the applicable development methodology and applicable 

tailoring of the systems engineering life cycle activities, artifacts, and reviews based on 

the specific characteristics of the program/project. Represents the agreement between 

the program/project and applicable decision authorities concerning the basic technical 

approach for the project.

Source: GAO analysis of USCIS documentation and DHS policy. | GAO-16-467

USCIS transitioned its approach to software development from waterfall 

development and is now using an incremental software development 

approach to develop, test, and deliver USCIS ELIS functionality to its 

users. Such an approach is consistent with OMB’s IT Reform Plan12 and 

the law commonly referred to as the Federal IT Acquisition Reform Act.

13

The incremental approach chosen by USCIS, called Agile, allows subject 

matter experts to validate requirements, processes, and system 

functionality in increments, and deliver the functionality to users in shorter 

cycles. See figure 5 for a depiction of software development using the 

Agile approach compared to a waterfall approach.

 12OMB, 25 Point Implementation Plan to Reform Federal Information Technology 

Management (Washington, D.C.: Dec. 9, 2010).

1340 U.S.C. § 11319(b)(1)(B)(ii). The Federal Information Technology Reform provisions 

of the Carl Levin and Howard P. ‘Buck’ McKeon National Defense Authorization Act for 

Fiscal Year 2015, Pub. L. No. 113-291, div. A, title VIII, subtitle D, 128 Stat. 3292, 3438-

3450 (Dec. 19, 2014) are commonly referred to as the Federal IT Acquisition Reform Act.

USCIS Is Developing Its 

Transformation Program 

Incrementally
Page 13 GAO-16-467 Immigration Benefits System

Figure 5: Comparison of Agile and Waterfall Software Development

The Agile approach emphasizes early and continuous software delivery, 

the use of collaborative teams, and measurement of progress by the 

development of software in increments.
Page 14 GAO-16-467 Immigration Benefits System

USCIS policy14 defines various stages by which programs plan for, 

develop, and deploy a release. In preparation for a release, a program is 

expected to undergo a release planning phase that includes a release 

planning review. The release planning review allows the USCIS Chief 

Information Officer (CIO) or a designee to assess the project team’s 

readiness to proceed with the release cycle based on whether the 

investment is justified by the expected results. After approval from this 

official, the project then proceeds to development activities. Development 

takes place in iterations, whereby a development team delivers a piece of 

functionality with each iteration. Collectively, these iterations will comprise 

the capabilities or features committed to in the release planning review, or 

a revised set of capabilities if priorities shift during the release.

Once the program has determined that all of the needed functionality has 

been developed, the release undergoes a release readiness phase, 

which includes a release readiness review. The release readiness review 

allows the USCIS CIO or a designee to assess whether the current 

increment is ready to be deployed based on whether the release has 

been adequately tested, reviewed by the product owner and users, and is 

compliant with internal standards. The official approving the release is to 

verify that release activities have been coordinated and determine 

whether USCIS is prepared for the impact of the release, among other 

things. After obtaining approval, the release is then deployed and 

becomes accessible both to internal USCIS users responsible for 

processing of applications, such as an adjudicator, and customers outside 

USCIS, such as applicants or Department of State officials with access 

permission.15 Figure 6 depicts the Agile software development cycle 

within USCIS.

 14USCIS, Management Instruction CIS-OIT-001, Office of Information Technology: Agile 

Development Policy (Apr. 10, 2013).

15Federal partners and other DHS components obtain information from USCIS ELIS

through the Enterprise Service Bus Person Centric Query System.
Page 15 GAO-16-467 Immigration Benefits System

Figure 6: USCIS Process for Agile Software Development

To help guide its system development efforts, OTC updated and DHS 

approved the USCIS Transformation Program Operational Requirements 

Document in April 2015. This document identifies the program’s high-level 

objectives and intended outcomes in a set of five core operating 

requirements. The core program operating requirements are further 

described in terms of capabilities and features.16 Capabilities plan to 

deliver business value over a number of releases, while features 

generally fulfill one or more stakeholder needs and are delivered within a 

release. Although not reflected in the operational requirements document, 

the program further decomposes features into a set of sub-features that 

are then elaborated on in a series of system or business requirements 

known as user stories.

As of February 2016, OTC reported that the five operating requirements 

are composed of 20 capabilities, 62 features, and 235 sub-features. This 

represents an overall decrease of 2 features and increase of 11 sub-

 16The Operational Requirements Document defines requirements as long-term, portfoliolevel initiatives that deliver Transformation across the enterprise (not limited solely to 

USCIS ELIS). It defines capabilities as groups of features that work together to deliver 

business value. Capabilities can span multiple releases and can involve multiple systems 

and organizations. It defines features as services provided by the system that fulfill a 

specific need and that can be developed (mostly) within a single system release.
Page 16 GAO-16-467 Immigration Benefits System

features since OTC first began reporting this metric.17 Figure 7 shows the 

relationship between the various levels of requirements by release.

Figure 7: Traceability from Operational Requirements to User Stories 

According to USCIS’s policy for Agile software development,

18 the 

Transformation Program, including USCIS ELIS, is to be developed and 

implemented in a series of releases that are to occur every 3-6 months. In 

addition to major releases, the Chief of OTC and USCIS CIO stated that 

the program deploys weekly minor releases into production. Minor 

releases are to enhance, or build on, existing product lines. To plan for 

and track development and delivery of prospective capabilities by release, 

the program created and regularly revises a program road map.

As shown in table 3, USCIS plans to deliver the program over a series of 

16 releases, with each release generally corresponding to a new product 

line. As part of each release, USCIS ELIS is the primary focus, and other 

related systems are updated as needed. Each product line contributes to 

processing one of four lines of business.19

 17As part of continuous monitoring since the program re-baseline, the Office of 

Transformation Coordination submits an agreed upon set of monthly metrics to the DHS 

Office of Program Accountability and Risk Management and the DHS Enterprise Business 

Management Office within the Office of the CIO. The program began including the number 

of capabilities, features, and sub-features as part of this presentation in September 2015. 

18CIS-OIT-001. 

19According to the Acquisition Plan, USCIS processes more than 50 types of benefit 

requests comprised of four lines of business, with approximately 15 product lines.
Page 17 GAO-16-467 Immigration Benefits System

Table 3: Planned Product Lines by Release, as of February 2016

Release Product line Line of business

5.0.0 Permanent resident card (I-90 soft launch) Immigrant

5.0.1 Permanent resident card (I-90 hard launch) Immigrant

5.0.2 Permanent resident card (I-90 Lockbox) Immigrant

5.1 (Rebuild) Immigrant visa processing Immigrant

6.1 Deferred action for childhood arrivals Humanitarian

6.2 Temporary protected status Humanitarian

7.1 Naturalization (N-400) Citizenship

8.1 Citizenship Citizenship

9.0 Family-based adjustment of status Immigrant

10.0 Other (non-employment based) adjustment of 

status and EB-5

Immigrant

11.0 Employment-based adjustment of status Immigrant

12.0 Employment-based nonimmigrants Non-Immigrant

13.0 Employment-based nonimmigrants Non-Immigrant

14.0 Refugee/asylum Humanitarian

15.0 Miscellaneous Humanitarian

16.0 International adoptions Humanitarian

Source: Transformation Program Road Map. | GAO-16-467.

From May 2013 through November 2014, USCIS planned, developed, 

and deployed a temporary launch of the new USCIS ELIS to 

accommodate the limited processing20 of one product line that processes 

the renewal and replacement of permanent resident cards.21 This phase 

included development of core processing capabilities, such as the ability 

to electronically file or schedule appointments for the collection of 

 20The program permitted the public to file applications for a 72-hour period in November 

2014. Between November 2014 and February 2015, the program processed applications 

submitted during the 72-hour period using the new USCIS ELIS. During this period, the 

program also continued developing additional functionality and addressing bugs or defects 

in the process. In February 2015, filing of I-90 applications via USCIS ELIS was made fully 

available to the public.

21Applicants file the I-90 form, “Application to Replace Permanent Resident Card,” in order 

to replace a permanent resident card due to its being lost, stolen, destroyed, or meeting 

other conditions.
Page 18 GAO-16-467 Immigration Benefits System

biometrics information. Development of this product line was completed 

and fully deployed in February 2015.

From February 2015 through the present, OTC has continued to add new 

functionality to USCIS ELIS, along with enhancements to existing 

functionality. For example, the program enhanced the functionality 

deployed in February 2015 by adding the ability to accept paper 

submissions for processing renewals and replacements of permanent 

resident cards through the Lockbox. The program also proceeded with 

new product lines that included parallel development of up to four product 

lines at once (releases 5.1, 6.1, 6.2, and 7.1). By March 2016, OTC had 

completed the roll out of two of these product lines (releases 5.1 and 6.1) 

to allow for processing of immigrant visa payments22 and applications for 

deferred action for childhood arrivals.23

Subsequently, USCIS continued incremental system development and 

enhancement of USCIS ELIS with focus, in part, on processing 

applications for naturalization and temporary protected status.24

According to the program road map, product lines deployed as of April 

2016 (releases 5.0, 5.1, 6.1, and 7.1) represent approximately 2.3 million 

 22An individual seeking to immigrate to the United States as a lawful permanent resident 

must pay the $165 USCIS immigrant fee online unless he/she meets certain exemption 

conditions. USCIS uses this fee to process an immigrant visa packet and produce a 

permanent resident card. 

23On June 15, 2012, the Secretary of Homeland Security announced that certain people 

who came to the United States as children and met several key guidelines could request 

consideration of deferred action for a period of two years, subject to renewal, and would 

then be eligible for work authorization. Individuals who could demonstrate through 

verifiable documentation that they met these guidelines would be considered for deferred 

action. Determinations would be made on a case-by-case basis under the guidelines in 

the Secretary of Homeland Security’s memorandum. In October 2015, the program made 

available limited processing of applications. Development of the product line was 

completed and fully available in February 2016. 

24The Secretary of Homeland Security may designate a foreign country for temporary 

protected status due to conditions in the country that temporarily prevent the country’s 

nationals from returning safely, or in certain circumstances, where the country is unable to 

handle the return of its nationals adequately. USCIS may grant temporary protected status 

to eligible nationals of certain countries (or parts of countries), who are already in the 

United States. Temporary protected status is a temporary benefit that does not lead to 

lawful permanent resident status or give any other immigration status.
Page 19 GAO-16-467 Immigration Benefits System

of the 7.3 million applications processed each year, or approximately 31% 

of the total workload.

USCIS assumed responsibility for managing multiple contractors as part 

of the Transformation Program’s change in acquisition strategy. As part of 

this new approach, contract management responsibilities are shared 

between OTC and OIT for 12 active contracts:25 seven to be managed by 

OIT, and 5 to be managed by OTC.26

Contract oversight is largely the responsibility of the contracting officer 

and contracting officer’s representative (COR) assigned to a particular 

contract. The contracting officer has authority to enter into, administer, 

and terminate contracts and has the overall responsibility for ensuring the 

contractor complies with the terms of the contract. As part of his or her 

responsibilities, the contracting officer selects a COR, who assists in the 

technical monitoring and day-to-day administration of a contract.

Within DHS, the COR administers and directs daily operations within the 

scope of the contract, monitors contractor performance, ensures that 

requirements meet the terms of the contract, and partners with the 

Contracting Officer. The COR is authorized to monitor the contract on 

behalf of the Contracting Officer. However, the COR is not authorized to 

make any contractual commitments or changes that may affect the 

contract price, terms, or conditions without the approval of the Contracting 

Officer.

 25Two additional contracts were active as of the date the Acquisition Plan was approved. 

These contracts provided continued development services until the Flexible Agile 

Development Services contract was awarded and independent verification and validation 

services.

26A number of additional contracts have since been awarded by USCIS in support of the 

Transformation Program. The acquisition plan does not reflect these new contracts nor 

has it been updated since approval in March 2015 to include these contracts.

Program Contract 

Management 

Responsibilities Are 

Shared Across USCIS
Page 20 GAO-16-467 Immigration Benefits System

We have previously reported on the management and development of the 

USCIS Transformation Program. In July 2007,27 we evaluated the 

Transformation Program’s strategic and expenditure plans to determine 

the extent to which these plans had prepared USCIS to carry out its 

program. We reported that the agency’s plans partially or fully addressed 

most key practices, but more attention was needed in certain areas such 

as performance measurement and IT management. We also reported that 

the plans provided some information on costs and revenues, but that 

USCIS had not finalized its acquisition strategy and, therefore, cost 

estimates were uncertain. To improve its strategy and to fully address 

congressionally requested information, we recommended that the 

Director of USCIS address gaps in plans in the areas of performance 

measurement, strategic human capital management, communications, 

and IT management practices. DHS concurred with our 

recommendations. Between September 2009 and September 2011, 

USCIS took steps to address the gaps identified in our report, such as 

finalizing a balanced set of four performance measures and establishing 

fiscal year 2012 targets that aligned with transformation goals for 

customer satisfaction, decisional accuracy, timeliness, and efficiency.

In November 2011,28 we assessed the extent to which USCIS had 

followed DHS acquisition policy in developing and managing the 

Transformation Program. We reported that the agency had not 

consistently followed the acquisition management approach that DHS had 

outlined in management directives for developing and managing the 

program. For example, USCIS did not complete several acquisition 

planning documents required by DHS policy prior to moving forward with 

an acquisition approach, which contributed to schedule delays and 

increased program costs. To help ensure that USCIS used a 

comprehensive and cost-effective approach to the development and 

deployment of transformation efforts to meet the agency’s goals of 

improved adjudication and customer services processes, we 

recommended that the Director of USCIS develop and maintain an 

 27GAO, USCIS Transformation: Improvements to Performance, Human Capital, and 

Information Technology Management Needed as Modernization Proceeds, 

GAO-07-1013R (Washington, D.C.: July 17, 2007).

28GAO, Immigration Benefits: Consistent Adherence to DHS’s Acquisition Policy Could 

Help Improve Transformation Program Outcomes, GAO-12-66 (Washington, D.C.: Nov. 

22, 2011).

GAO Has Previously 

Reported on 

Transformation Program
Page 21 GAO-16-467 Immigration Benefits System

integrated master schedule consistent with leading practices for the 

Transformation Program and ensure that the life-cycle cost estimate be 

informed by milestones and associated tasks from reliable schedules. 

Between November 2011 and May 2016, USCIS took steps to address 

the gaps identified in our report, such as developing an integrated master 

schedule that was generally consistent with leading practices.

In May 2015,29 we reviewed the Transformation Program’s current status, 

including the impact of recent changes made, and assessed the extent to 

which DHS and USCIS are executing effective program oversight and 

governance. We noted that these changes have significantly delayed the 

program’s planned schedule. We added that, while the program’s two key 

governance bodies had taken actions aligned with leading IT 

management practices, neither had used reliable information to make 

decisions. To improve the ability of USCIS and DHS to monitor program 

performance, we recommended that these bodies use reliable information 

to inform their program evaluations. USCIS concurred with our 

recommendations and has begun work to address them.

USCIS created a reliable updated estimate to project the program’s cost 

but has experienced challenges in managing software development, 

testing, and contract management for USCIS ELIS, the key component of 

the program. In particular, software development, testing, and contract 

monitoring have not consistently been managed in line with the program’s 

policies and guidance or with leading practices.

USCIS officials cited various reasons for the inconsistences between 

program management activities and the practices described in program 

policies, guidance, and leading practices. For example, while USCIS 

documentation indicates that the program is being managed according to 

a software development framework that calls for development teams to 

follow specific practices, USCIS officials stated that the program is 

adjusting these practices based on lessons learned. However, the 

program is not ensuring that such changes are communicated broadly to 

development teams by updating its guidance to reflect these adjustments. 

These management weaknesses—which at times mirror those previously 

 29GAO-15-415.

USCIS 

Transformation 

Program Has Not 

Fully Implemented 

Leading IT 

Management 

Practices
Page 22 GAO-16-467 Immigration Benefits System

reported—increase the risk that the system will continue to exceed cost 

and schedule goals and that it may not meet performance expectations.

As noted, USCIS transitioned from a waterfall approach for software 

development to an Agile approach in order to increase the speed with 

which software is released into production and provide better control over 

each segment as it is released. Agencies can accomplish this objective 

by designing and implementing key management practices that are 

intended to maximize the value of software development and meet the 

values and principles of Agile software development. Based on the 

approach followed by the program, along with DHS and USCIS policy, 

procedures, and guidance30 as well as leading practices,31 key 

management practices for effective Agile software development include:

1. completing planning for software releases prior to initiating 

development and ensuring software meets business expectations 

prior to deployment;

 30Department of Homeland Security, Instruction Number: 102-01-004, Agile Development 

and Delivery for Information Technology, Revision #0(Apr. 11, 2016); Pre-Decisional Draft,

DHS Agile Development Guidebook (Sept. 2, 2014); Draft, Agile Methodologies in DHS:

Agile Working Group White Paper (no date); U.S. Citizenship and Immigration Services, 

Management Instruction CIS-OIT-001, Office of Information Technology: Agile 

Development Policy (Apr. 10, 2013); Agile Processes and Practices Principles and 

Guidelines, version 4.0 (Dec. 6, 2013); USCIS Transformation Program: System 

Engineering Life Cycle Tailoring Plan, version 1.0 (March 2015); USCIS ELIS Team 

Process Agreement for the April 7, 2015 to September 30, 2015 Development Period; and 

USCIS ELIS Team Process Agreement for the July 22, 2015 to January 30, 2016 

Development Period. 

31GAO, Software Development: Effective Practices and Federal Challenges in Applying 

Agile Methods, GAO-12-681 (Washington, D.C.: Jul. 27, 2012); Office of Management and 

Budget, U.S. Digital Services, Playbook (version pulled Dec. 18, 2015); TechFAR: 

Handbook for Procuring Digital Services Using Agile Processes (draft version pulled Nov. 

6, 2014); Software Engineering Institute, Agile Metrics: Progress Monitoring of Agile 

Contractors, CMU/SEI-2013-TN-029 (Jan. 2014); CMMI® for Development, Version 1.3, 

CMU/SEI-2010-TR-033 (Nov. 2010); MITRE, Defense Agile Acquisition Guide: Tailoring 

DoD IT Acquisition Program Structures and Processes to Rapidly Deliver Capabilities

(March 2014); Handbook for Implementing Agile in Department of Defense Information 

Technology Acquisition (Dec. 15, 2010); ISO/IEC/IEEE, Systems and software 

engineering —Developing user documentation in an agile environment, Corrected 

Version, ISO/IEC/IEEE 26515:2012(E) (Mar. 15, 2012); http://agilemanifesto.org.; Ken 

Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: The 

Rules of the Game (Jul. 2013).

Program Has Not 

Implemented Its Defined 

Software Development

Approach Consistent with 

Its Policies and Associated 

Leading Practices
Page 23 GAO-16-467 Immigration Benefits System

2. adhering to the principles of the framework adopted for implementing 

Agile software development;

3. defining and consistently executing appropriate roles and 

responsibilities for individuals responsible for development activities;

4. identifying users of the system and involving them in release planning 

activities;

5. writing user stories that identify user roles, include estimates of 

complexity, take no longer than one sprint to complete, and describe 

business value;

6. prioritizing user stories to maximize the value of each development 

cycle;

7. setting outcomes for Agile software development; and

8. monitoring and reporting on program performance through the 

collection of reliable metrics.

The program has defined and is at least partially adhering to most 

practices for effectively managing Agile software development in 

producing USCIS ELIS. However, the program is not fully adhering to 

most practices. Specifically, the program has fully implemented one key 

practice, partially implemented six practices, and did not implement the 

other practice. Table 4 describes the program’s adherence to these 

practices in developing USCIS ELIS. See appendix II for a more detailed 

analysis of each of the eight management practices.

Table 4: Adherence to Key Agile Development Management Practices

Key practice Rating Adherence to key practice

Completing planning for software 

releases prior to initiating 

development and ensuring software 

meets business expectations prior to 

deployment

◐ The program created and provided evidence of having completed some aspects of 

planning for its software releases. For example, the program reviewed the core 

artifacts stipulated in USCIS policy in planning for each release of USCIS ELIS. 

These artifacts include a definition of the initial scope of the software release. The 

program also provided evidence that management discussed the work completed 

during the release and associated risks prior to deployment of each major release.

However, the program did not complete other aspects of planning for its software 

releases. For example, the program did not provide evidence that the acquisition plan 

and program cost estimate were reviewed or updated as part of each release 

planning review. In addition, teams initiated development of major releases prior to 

completing planning for the release. Specifically, development across four of the five 

major releases began one week to one month prior to obtaining approval to complete 

release planning.

The program also did not demonstrate that it took all planned steps for ensuring 

software meets business expectations prior to deploying software increments. For 

example, the program demonstrated that the USCIS CIO reviewed documents such 

as system design documentation as part of release readiness reviews. However, the 
Page 24 GAO-16-467 Immigration Benefits System

Key practice Rating Adherence to key practice

program did not demonstrate that other documents, such as automated and manual 

test scripts, were reviewed before the CIO approved individual releases. 

Adhering to the principles of the 

framework adopted for implementing 

Agile software development

◐ The program incorporated some of the principles of the Scrum framework, but did not 

follow others. For example, stakeholders were included in the sprint review and 

acceptance criteria was defined and discussed by the development team as part of 

sprint planning, as prescribed in the Scrum Guide. a

However, teams did not hold a retrospective meeting at the end of every sprint and, 

when held, development teams were not consistently implementing improvements 

identified during these retrospectives. 

Defining and consistently executing 

appropriate roles and responsibilities 

for individuals responsible for 

development activities 

◐ The program at times deviated from both the Scrum Guidea and its internal 

management approach. The program’s Scrum teams included both a product owner 

and development team, but did not include an individual explicitly designated to serve 

as a Scrum master. Moreover, one product owner was responsible for 12 

development teams at one time during a release and another was responsible for 8 

teams. This was in contrast to the program’s management approach that limited 

product owner responsibility to 4 development teams or less.

Following our discussions with OTC about product owner responsibilities, the 

program removed the 4-team restriction on the product owner and created a new 

position, product manager, to help the product owner in performing responsibilities. 

However, without also assuming the authority of the product owner, this new position 

adds an additional layer of communication and coordination, which may impact team 

performance.

Identifying users of the system and 

involving them in release planning 

activities

◐ The program has defined user roles within USCIS and roles associated with external 

customers. The program provided evidence that these USCIS users were involved in 

release planning and involvement of this group has increased over time. The program 

also provided additional evidence that external customers, while not involved in 

release planning, were included as part of ongoing planning and development efforts. 

For example, in October 2015, the program held two virtual focus groups to, among 

other things, understand customer perception of USCIS ELIS. However, the program 

did not identify users in external agencies, or other components within DHS, and did 

not include these users in release planning.

Writing user stories that identify user 

roles, include estimates of 

complexity, take no longer than one 

sprint to complete, and describe 

business value

◐ User stories we evaluated identified a user role and contained an estimate of 

complexity at least 90 percent of the time. However, of the approximately 1,000 user 

stories tracked by the Quality Assurance Team, over 30 percent took over a full sprint 

to develop. Further, most of the user stories did not define the associated business 

value. 

Prioritizing user stories to maximize 

the value of each development cycle

● The backlogs for releases 6.1, 6.2, and 7.1 were ordered and clearly reflect the sprint 

in which a team expects to develop the story. According to the product owners for 

releases 6.1, 6.2, and 7.1, user story priorities are based on the need to develop 

specific features associated with an upcoming release.

According to product owners, software development for multiple features occurs 

during each sprint. For example, a product owner might assign a development team 

some of the functionality associated with validating a customer identification number 

along with some functionality associated with scheduling a biometrics appointment 

during a single two-week development cycle. According to the product owners, work 

is organized in this fashion to address dependencies among features that impact the 

timing of needed functionality. 

Setting outcomes for Agile software 

development 

○ The program has not established intended outcomes for Agile software development.
Page 25 GAO-16-467 Immigration Benefits System

Key practice Rating Adherence to key practice

Monitoring and reporting on program 

performance through the collection 

of reliable metrics

◐ Some metrics were reliable and addressed their intended purpose. For example, the 

program provided evidence of collecting reliable metrics associated with code quality. 

In addition, the program collects external customer satisfaction through a survey. The 

results from this survey are then reported to oversight bodies and in monthly program 

management review meetings.

However, other metrics were either unreliable or were not collected. For example, the 

program does not monitor internal USCIS user satisfaction with USCIS ELIS. 

Therefore, it cannot measure the level of satisfaction of adjudicators or others using 

the system to facilitate the processing of applications. Moreover, metrics intended to 

demonstrate if the scope of each release is consistent with plans are not fully 

traceable back to the intended functionality.

Source: GAO analysis of USCIS documentation. I GAO-16-467.

Note: ● yes ◐ partial ○ no

An element was determined to be a “no” if USCIS provided no evidence to satisfy any portion of the 

practice; “partial” if USCIS provided evidence that satisfied some, but not all, of the practice; and 

“yes” if USCIS provided evidence that it substantially satisfied all elements of the practice.

a

Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: The Rules 

of the Game (Jul. 2013).

The implementation of software development deviated from key practices 

in part because USCIS policy and guidance were not being updated to 

reflect changes in program implementation. For example, the USCIS CIO 

explained that, although policy requires a program to obtain approval for 

the scope of a release prior to proceeding with development, this is no 

longer the practice for the Transformation Program. Instead, approval is 

granted for six months of development and the scope of that approval is 

revisited as needed. Moreover, program-level procedures and 

assessments documented that teams committed to using the Scrum 

framework to guide their Agile development activities. Program officials 

stated that this framework is no longer followed as practices are 

continuously evolving to reflect lessons learned in each iteration. 

However, program-level procedures and assessments were not updated 

to reflect these changes.

In addition, based on our analysis, the program is not consistently 

following other key practices because USCIS policy and guidance do not 

align with leading practices. For example, policy, procedures, and 

guidance provided by the program do not define all of the key roles and 

responsibilities for developing USCIS ELIS. As a result, various 

individuals within the program reported different opinions about who was 

serving in critical roles, such as that of the Scrum master.

Further, controls were not always in place to ensure the program adhered 

to key practices. For example, we were unable to track monthly reports 
Page 26 GAO-16-467 Immigration Benefits System

on program scope back to the associated release backlog. In a written 

response, the Business Integration Division within OTC acknowledged 

issues in traceability of user stories to sub-features. This division stated 

that the process that was used to verify the number of sub-features 

implemented against planned was based on verbal confirmation from the 

product owner. The division subsequently determined that this process 

was not effective since it relied solely on the review of the user stories 

and was not as exact and reliable as expected. The division noted that 

requirements traceability is critical to avoid scope creep and to 

demonstrate that the user stories implemented addressed the mission 

needs.

Until the program addresses the key practices for Agile software 

development, it risks deploying a system that does not meet the cost, 

schedule, or performance needs of USCIS.

In addition to management of Agile software development, a program 

needs to manage the quality of the software being developed through 

ongoing systems integration and testing. DHS and USCIS policy, 

procedures, and guidance,32 as well as leading practices,33 identify key 

management practices for systems integration and testing. These 

practices include:

1. establishing an environment and procedures for continuous 

integration34 and testing of code;

 32Department of Homeland Security, Pre-Decisional Draft, DHS Agile Development 

Guidebook (Sept. 2, 2014); U.S. Citizenship and Immigration Services, USCIS Test and 

Evaluation Master Plan: Electronic Immigration System, version 1.0 (February 2015); 

USCIS ELIS Team Process Agreement for the April 7, 2015 to September 30, 2015 

Development Period; and USCIS ELIS Team Process Agreement for the July 22, 2015 to 

January 30, 2016 Development Period; Agile Processes and Practices Principles and 

Guidelines, version 4.0 (Dec. 6, 2013); USCIS Test Plan for Release 5.1-6.0 (10/15/2014-

4/15/2015); USCIS Test Plan (updated July 2015). 

33IEEE, IEEE Standard 29119, Software and System Engineering – Software Testing, 

Part 1: Concepts and Definitions, Part 2: Test Process, Part 3: Test Documentation (New 

York, NY, September 2013).

34Continuous integration is a practice of frequently merging an individual piece of software 

code with the main code repository, so new changes are tested continuously. 

Testing Efforts Are Not 

Fully Consistent with 

Agency Guidance or 

Leading Practices
Page 27 GAO-16-467 Immigration Benefits System

2. performing continuous testing35 through the use of

• Unit tests: Small, automated tests used to verify that each 

individual unit of code written by the developer works as intended.

• Integration tests: Tests that are intended to verify that individual 

modules or components function together as expected when 

combined.

• Functional acceptance tests: Tests to verify that all user story 

acceptance criteria have been met and should be executed as 

capabilities are added or updated.

• Code inspection: Process of inspecting code automatically and 

manually and comparing it to internal coding guidelines.

3. developing complete test plans and cases for interoperability and end 

user testing and documenting those results. Interoperability testing is 

an independent test in a production-like environment that is intended 

to verify the capability of system components and external work flows, 

communication, and information sharing in a controlled condition. The 

USCIS Transformation Program Test and Evaluation Master Plan

requires execution of interoperability tests following any change to 

USCIS ELIS interfaces. End user testing is conducted to verify that 

business value is delivered and user story acceptance criteria are 

met. The USCIS ELIS End User Test Strategic Plan describes end 

user testing as involving USCIS ELIS users to test functionalities and 

usability using real-world scenarios prior to implementing a release 

into production.

The program has fully implemented one of these key practices and 

partially implemented the other two. Specifically, the program has 

established an environment and procedures for continuous integration 

and is conducting unit and integration, functional acceptance, 

interoperability, and end user tests, and code inspection.

However, the program is not consistently adhering to its policies and 

guidance or meeting benchmarks for unit and integration, and functional 

 35Continuous testing is a practice of testing software code throughout a sprint and 

providing feedback to the developers so that they can address issues and develop 

software that meets all required quality criteria. Using this approach, code is tested in 

various environments, from the development environment to a production-like 

environment. 
Page 28 GAO-16-467 Immigration Benefits System

acceptance tests, and code inspection. Moreover, test plans, cases, and 

results were not fully developed for interoperability and end user testing. 

Table 5 describes our findings in more detail. See appendix III for a more 

detailed analysis of each of the three key practices.

Table 5: Adherence to Key Integration and Testing Practices

Key practice Rating Adherence to key practice

Establishing an environment and 

procedures for continuous integration 

and testing of code

● The program has established an environment for continuous integration and testing 

for USCIS ELIS that is consistent with program planning documentation. For 

example, the program has created a process for the automated build of software 

code and automated testing of that code during the build process. This environment 

also includes configuration management, daily code integration, code build stops 

with test failure, and the capacity to provide rapid feedback on defects.

Performing continuous testing through 

the use of unit and integration tests, 

functional acceptance tests, and code 

inspection

◐ The program demonstrated that it conducted unit and integration testing, functional 

acceptance testing, and code inspection.

However, the program did not demonstrate that it consistently met stated goals. For 

example, none of the internal applications that collectively make up USCIS ELIS 

have achieved 100 percent code coverage, which is the program’s stated goal. In 

particular, agency reports show that internal application code coverage ranged from 

31.2 percent to 96.9 percent. In addition, 63 of the 74 user stories we reviewed did 

not include evidence that functional acceptance tests were run and passed. 

Moreover, 13 of the 53 user stories we evaluated and to which peer inspection was 

applicable did not include evidence of peer inspection, which is an important 

component of inspecting code.

Developing complete test plans and 

cases for interoperability and end user 

testing and documenting those results

◐ The program developed nearly complete test plans and cases and documented the 

results for end user testing from June 2015 through March 2016. However, the 

program did not provide evidence of test plans, cases, or results for end user testing 

between March 2015 and June 2015 and the test plan is a draft document. Further, 

test plans and cases were not developed for interoperability testing, as the term is 

described in the USCIS Transformation Program Test and Evaluation Master Plan, 

and documentation of the results of such testing was incomplete. 

Source: GAO analysis of USCIS documentation. I GAO-16-467.

Note: ● yes ◐ partial ○ no

An element was determined to be a “no” if USCIS provided no evidence to satisfy any portion of the 

practice; “partial” if USCIS provided evidence that satisfied some, but not all, of the practice; and 

“yes” if USCIS provided evidence that it substantially satisfied all elements of the practice.

The implementation of systems integration and testing deviated from key 

practices in part because policy and guidance were not being updated to 

reflect changes in the approach. For example, the program did not meet 

its stated goals for continuous testing because, according to the USCIS 

CIO, certain program goals are unrealistic. For example, the USCIS 

Transformation Program Test and Evaluation Master Plan stated that unit 

tests will reach 100% code coverage for the individual developed units. 

However, a member of the U.S. Digital Services team who is assisting the 

program stated that he and others conducting testing were unaware of 
Page 29 GAO-16-467 Immigration Benefits System

the plan and that information contained in the plan does not reflect the 

practice of USCIS ELIS testers. Moreover, this individual agreed with the 

USCIS CIO’s assessment that the goal of 100% code coverage is 

unrealistic.

In addition, the program did not always have controls in place to ensure 

developers adhered to policy and guidance. For example, before a 

developer can integrate code with that of other developers or 

development teams, peer inspection by another individual is expected to 

occur to help ensure that the code meets program standards. However, 

the program has not established controls to monitor the extent to which 

peer inspection occurs.

Further, program policy and guidance did not always align with best 

practices in systems integration and testing. For example, with respect to 

interoperability and end user testing, these tests are performed outside of 

the continuous integration pipeline and are therefore not automated. The 

program provided tester notes and testimonial evidence to support the 

manual execution of interoperability (e.g. interface) tests, as called for by 

program policy and guidance. However, these tester notes and 

testimonial do not meet leading practices for planning or documenting 

results that demonstrate the ability of USCIS ELIS to interface with other 

systems internal and external to USCIS.

Until the program fully addresses the key practices for systems

integration and testing by ensuring the consistent implementation of 

policy, procedures, and guidance consistent with leading practices, the 

program risks poor system performance after it has been released to the 

public. This risk is of particular concern due to system performance 

issues that have already been realized. Specifically, the program has 

reported experiencing issues with USCIS ELIS as a result of deploying 

software that had not been fully tested. For example:

• In June 2015, the Quality Assurance Team reported that production 

issues, such as bugs and defects, had increased noticeably after the 

February 2015 deployment.

• In July 2015, the Quality Assurance Team reported that defects 

(originating from either production or development) were becoming a

significant part of USCIS ELIS iteration work.

• On September 22, 2015, in addition to prior and subsequent outages 

of the system, the Quality Assurance Team reported that USCIS ELIS 
Page 30 GAO-16-467 Immigration Benefits System

was unavailable for approximately 15 hours due to issues with code 

quality.

• On September 24, 2015, USCIS ELIS encountered issues that 

impacted nearly 5,000 cases. Approximately 2,600 of these cases had 

to be abandoned.

• In November and December 2015, the Quality Assurance Team 

reported that code quality had become a major issue.

• In January 2016, the program reported more than 800 minutes in 

unplanned network outages.

• In February 2016, the program reported missing the threshold for 

USCIS ELIS reliability (e.g. mean time between failure), for two 

straight months and four of the last six months.

• In March 2016, program metrics indicated that production tickets were 

increasing faster than they could be addressed.

Key internal controls, as defined in agency policy and guidance36 as well 

as other federal guidance,37 should be in place to manage oversight of 

contractors and provide reasonable assurance that performance aligns 

with expectations. Key internal controls include assigning CORs who 

have met agency training requirements, documenting the rationale if a 

performance-based contract is not awarded, defining and assessing 

contractors against appropriate performance measures if a performancebased contract is awarded, and conducting ongoing monitoring activities.

 36Department of Homeland Security, Department of Homeland Security Acquisition 

Regulation (June 2006); Department of Homeland Security Acquisition Manual (October 

2009); Federal Acquisition Certification for Contracting Officer’s Representatives and 

Appointment and Revocation, Acquisition Workforce Policy Number 064-04-003, Revision 

02 (Aug. 8, 2012); DHS, Contracting Officer’s Representative (COR) Essential Element 

Guidebook, Spiral 1 (September 2012); Office of Procurement Operations: Acquisition 

Manual, version 1.0 (May 1, 2015). 

37GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: November 1999); Office of Federal Procurement Policy, A Guide to 

Best Practices for Contract Administration (Washington, D.C.: October 1994). 

Contract Management 

Policies Are Not Being 

Implemented Consistently
Page 31 GAO-16-467 Immigration Benefits System

We reviewed management and oversight of six contracts supporting the 

Transformation Program. Two of these were firm-fixed-price38 task 

orders39 and managed by OTC—Program Management Support Services 

and Training and Instructional Design Services—and the other four were 

cost plus fixed fee40 task orders managed by OIT.41 These four OITmanaged task orders were placed against a single contract for Flexible 

Agile Development Services (FADS) and share a common performance 

work statement, tasks, and deliverables.42 Similarly, one individual within 

OIT assumed COR responsibilities for all four task orders. We determined 

that the COR managed each of the four task orders consistently, 

regardless of contractor.

USCIS had mixed success in implementing selected key internal controls 

on the contracts we reviewed. For example, while CORs are meeting 

training requirements, the program can improve monitoring of contracts to 

 38A firm-fixed-price contract provides for a price that is not subject to any adjustment on 

the basis of the contractor’s cost experience in performing the contract. This contract type 

places maximum risk and full responsibility for all costs and resulting profit or loss on the 

contractor. A firm-fixed-price contract is suitable for acquiring commercial items or for 

acquiring other supplies or services on the basis of reasonably definite functional or 

detailed specifications when the contracting officer can establish fair and reasonable 

prices at the outset. 48 C.F.R. 16.202.

39A task order is an order for services placed against an established contract or with 

government sources. 48 C.F.R. 2.101(b).

40A cost-plus-fixed-fee contract is a cost reimbursement contract that provides for 

payment to the contractor of a negotiated fee that is fixed at the inception of the contract. 

The fixed fee does not vary with actual cost, but may be adjusted as a result of changes in 

the work to be performed under the contract. This contract type permits contracting for 

efforts that might otherwise present too great a risk to contractors, but it provides the 

contractor only a minimum incentive to control costs. A cost-plus-fixed-fee contract is 

suitable for use when, among other things, the contract is for development and test, and 

using a cost-plus-incentive-fee contract is not practical. A cost-plus-fixed-fee contract 

normally should not be used in development of major systems once preliminary 

exploration, studies, and risk reduction have indicated a high degree of probability that the 

development is achievable and the government has established reasonably firm 

performance objectives and schedules. 48 C.F.R. 16.306.

41The Program Management Support Services and Training and Instructional Design 

Services task orders represent approximately 90 percent of the total funding managed by 

OTC, as defined in the acquisition plan.

42The four Flexible Agile Development Services task orders represent approximately 77 

percent of the total funding managed by OIT, as defined in the acquisition plan. 
Page 32 GAO-16-467 Immigration Benefits System

provide reasonable assurance that they are meeting program needs. In 

addition, the program can improve its management and monitoring of 

performance-based contracts.

Initial and ongoing training requirements for USCIS CORs are clearly 

detailed in DHS guidance.43 Because of the size and complexity of the 

contracts, the CORs were required to maintain the highest level of 

certification called for in DHS guidance.44 Specifically, requirements 

include 60 hours of training when assuming basic COR responsibilities 

and 40 hours of refresher training every two years after the initial training

in order to maintain certification. They must also take specific COR ethics 

training initially as well as subsequent annual ethics training. In addition, 

federal regulations and DHS procedures and guidance require CORs to 

file a financial disclosure form.

Each of the CORs assessed had completed training requirements at the 

time of appointment and had also documented their completed refresher 

training requirements for the contracts we reviewed. In addition, the 

CORs had fulfilled their ethics and financial disclosure requirements.

The Federal Acquisition Regulation and OMB guidance direct federal 

agencies to use a performance-based contract—that is, a contract with 

measurable performance standards and a method for assessing a 

contractor’s performance against those performance standards—to the 

maximum extent practicable when acquiring services.45 DHS policy46 also 

 43Department of Homeland Security, Acquisition Workforce Policy Number 064-04-003, 

Revision 02 and Contracting Officer’s Representative (COR) Essential Elements 

Guidebook, Spiral 1.

44Because of the complexity of the DHS contract portfolio, CORs can only be certified as 

level II or level III when training is completed and experience is gained. Level II CORs can 

manage contracts other than high risk or major investment contracts. Only level III CORs 

can manage high risk or major investment contracts.

45OMB, Office of Federal Procurement Policy, Memorandum for Chief Acquisition Officers 

and Senior Procurement Executives; Subject: Use of Performance-Based Acquisitions 

(Washington, D.C.: July 21, 2006). The Federal Acquisition Regulation refers to 

performance-based contracting as a contract method; for ease of reporting and for 

purposes of this review, we include performance-based contracts in our discussion of 

contract types.

46DHS, Office of Procurement Operations, Acquisition Manual Version 1.0.

CORs are meeting training 

requirements

USCIS documented rationale 

for not pursuing selected 

contracts as performancebased
Page 33 GAO-16-467 Immigration Benefits System

encourages the use of performance-based acquisition to the maximum

extent practicable. However, if the contract is not performance-based, the 

acquisition plan must include the rationale.47

The two contracts managed by OTC are not performance-based, but the 

rationale for this decision is documented in the contract file. Specifically, 

the acquisition plan for program management support services states that 

it is not performance-based because staffing resources were not sufficient 

to support this type of acquisition. In addition, it states that tasks listed in 

the statement of work cannot be converted to a performance-based task. 

The contract for training and instructional design services included a 

memo to the contract file that explained the rationale. According to this 

memo, the contract was a new requirement focused heavily on course 

development and training on an ongoing basis. As such, there were no 

established measurable performance standards to draw on. The memo 

notes that a performance-based acquisition was considered but a 

performance work statement was ultimately deemed inappropriate. We 

did not evaluate the rationales provided by the program.

According to federal government internal control standards, appropriate 

and relevant performance measures and indicators must be established 

and continually compared against program goals and objectives.48 The 

standards also recommend incorporating performance data into operating 

reports that inform management of inaccuracies or internal control 

problems.

The Agile development services contract contains appropriate 

performance criteria that are linked to the program goals. Specifically, the 

primary goal of this contract is developing deployable application code 

that supports the USCIS mission to move towards an innovative, 

electronic, customer-centric architecture. To achieve this goal, the 

contractors provide services such as software development using Agile 

methodologies and testing and integrating code. The performance criteria 

used to measure contractor performance address code quality and 

standards adherence; business satisfaction; test quality and test 

coverage; collaboration; productivity; innovation; and process and 

 47Federal Acquisition Regulation, 48 C.F.R. 7.105.

48 GAO/AIMD-00-21.3.1.

Agile development services 

contract is performance-based, 

but expectations are not 

defined
Page 34 GAO-16-467 Immigration Benefits System

continuous improvement. The contract also requires the contractors to 

submit monthly performance and expenditure reports, status briefings, 

and other program management documentation on a timely basis. 

According to the Contracting Officer, a contractor that does not meet a 

performance requirement might have its annual target fee reduced and 

might not be contracted to provide additional Agile development teams.49

USCIS tracks performance data for these contracts through monthly 

operating reports, referred to as a “performance scorecard.” The 

scorecards track the accomplishments of most of the performance areas 

established in the contract. For example, the scorecard assesses the 

quality of code developed by the contractor and discipline in contractor 

adherence to program defined processes. The monthly scorecards are 

prepared by the product manager, project manager, and release manager 

in OTC and then submitted to the COR, who resides in OIT. Performance 

is assessed using program records of activities and interactions from the 

reporting month, metrics provided by the Quality Assurance Team, as 

well as from self-evaluations submitted by the contractor. The COR does 

not submit any financial or other status reports to the Contracting Officer 

regarding ongoing contractor performance, financial and budgetary data, 

the status of the contract’s schedule, or risk issues. As such, the 

Contracting Officer relies on the COR, along with separate monthly 

performance summaries submitted by the contractor, to verbally advise 

them on any issues.

Although performance criteria were defined in the contracts, the program 

did not clearly define measures against which to analyze differences 

between services expected and those delivered. Guidance for completing 

the performance scorecard identifies possible measures to use in 

evaluation. For example, this guidance states that contractor code quality 

should be evaluated, in part, based on the length of time to fix a broken 

build, the number of code check-ins, number and severity of defects 

found, and automated test code coverage. However, the measures do not 

clearly define program expectations, such as the length of time to fix a 

broken build. Further, contractors were not consistently evaluated against 

 49The Agile development services contracts are structured to allow the program to add 

additional development teams during the period of performance as needed. Each 

development team can only provide up to a specified level-of-effort and collectively the 

development teams cannot exceed the cost of the contract. 
Page 35 GAO-16-467 Immigration Benefits System

the proposed measures in the guidance. For example, in July 2015, 

performance measures associated with defects, broken builds, and 

automated test coverage were not included in evaluating the code quality 

for three contractors.

In addition, performance scorecards do not measure contractor 

performance against all of the criteria specified in the contracts. The 

contracts articulate seven performance measures, while the performance 

scorecard evaluates contractors against only five performance criteria. 

Specifically, the scorecard does not evaluate the contractor based on test 

quality and coverage, or productivity. Further, while the scorecard does 

assess code quality, it does not fully assess against the associated 

performance criteria, as it does not assess adherence to coding 

standards. In an interview, neither the Contracting Officer nor the COR 

provided a rationale for discrepancies in performance measurement 

between the contract and the scorecard, although the Deputy Chief of the 

USCIS Contracting Office stated that the government can change the 

methodology it uses to measure performance. However, the contract files 

included no evidence that a change in performance measures was 

discussed or the reason for omitting certain performance measures 

specified in the contract.

Some of the performance measures omitted in the performance 

scorecard are measures that would help provide indicators associated 

with concerns raised by the Quality Assurance Team in that team’s 

monthly status reports on program performance. For example, in August 

2015, the Quality Assurance Team identified testing as a major concern, 

as it reported that testing before the end of a sprint was inadequate. The 

Quality Assurance Team reiterated in October 2015 the concern that 

development teams were not completing basic testing. If test quality and 

test coverage were considered in these monthly assessments, 

contractors could be further held accountable for the quality and coverage 

of tests.

Because oversight of contractor performance resides with OTC and 

management of the contract is the responsibility of OIT, the need for 

clearly-defined performance measures is particularly important. Until the 

program clearly defines appropriate performance measures and 

indicators for Agile development service contracts, and continually 

compares contractor performance with expected or planned goals, the 

agency lacks important information for measuring contractor performance 

and determining if OIT is meeting its objectives in supporting the program.
Page 36 GAO-16-467 Immigration Benefits System

According to leading practices,50 ongoing monitoring activities include 

maintaining a complete contract file, creating a quality assurance 

surveillance plan, and completing and submitting a contractor 

performance assessment. The program has at least partially addressed 

some of these activities, but has not addressed other activities. 

Specifically, the program uses contractor performance assessments to 

assess and report performance, and maintains some, but not all, required 

documentation in its contract files. However, the program is not creating 

quality assurance surveillance plans to monitor contract performance.

Program maintained some documentation in COR files

Contract monitoring leading practices recommend that agencies conduct 

ongoing monitoring by maintaining contract files. In addition, the DHS 

COR guidebook states that complete and orderly COR files are vital in 

administering a contract to ensure that the government and the contractor 

meet their obligations. When disagreements or questions of interpretation 

arise, the COR file plays a critical role in resolving a dispute. A proper, 

well-documented file will also facilitate the transfer of responsibility if the 

COR is replaced during the contract. The COR must ensure that this file 

includes records documenting all correspondence, such as e-mails, 

telephone calls, or formal and informal meetings, between and among the 

COR, the contractor, the Contracting Officer, and other personnel relating 

to contractor performance. The file should also include a log of any 

actions taken as a result of communications with the contractor and other 

relevant documents, as appropriate, such as the COR appointment letter, 

approved and accepted deliverables, and copies of all inspection and 

acceptance documents and invoices.

CORs maintained proper documentation for some contract actions such 

as contractor onboarding, certifications and associated appointment 

responsibilities, materials such as market research or resumes of 

contractor personnel, and had fully documented modifications made to 

 50GAO, GAO/AIMD-00-21.3.1; National Flood Insurance Program: Progress Made on 

Contract Management but Monitoring and Reporting Could be Improved, GAO-14-160 

(Washington, D.C.: Jan. 15, 2014): Office of Federal Procurement Policy, A Guide to Best 

Practices for Contract Administration; Department of Homeland Security, Acquisition 

Workforce Policy Number 064-04-003, Revision 02; Contracting Officer’s Representative 

(COR) Essential Elements Guidebook, Spiral 1; Office of Procurement Operations, 

Acquisition Manual Version 1.0.

Monitoring of selected program 

contracts needs improvement
Page 37 GAO-16-467 Immigration Benefits System

the contracts. For example, the program management support contract 

included nineteen modifications that included changes such as the 

assignment of a new contracting officer, the revision of the statement of 

work, and extension of the terms of the contract.

However, COR files were not complete. Specifically, the file for the 

program management support services contract did not document 

correspondence, meetings, or other communication; excluded a 

deliverable required in the contract; and inspection and acceptance 

documents were omitted. The training and instructional design contract 

excluded correspondence, meetings, or other communication, and all 

inspection and acceptance documents were omitted. The FADS contract 

files included correspondence, but excluded other communication, such 

as monthly performance scorecard review meetings, all inspection and 

acceptance documents, and some deliverables. For example, contractors 

submitted a quality management plan, but the plan was not always 

updated, as specified in guidance. In addition, the file did not include 

updated plans for each contractor.

Contracting officers stated that they did not review the COR files during 

performance of the contracts. As a result, contracting officers for the 

contracts we reviewed relied on the COR to maintain a file and to 

communicate with the Contracting Officer when the COR deemed it 

necessary. For example, if there was a discrepancy over performance or 

the quality of a deliverable, the matter might be escalated to the 

Contracting Officer. The COR for the program management support and 

training contracts stated that they assumed various communications 

should only be documented when a discrepancy arose. Since no 

discrepancy reports were necessary, the COR believed that 

correspondence or other communication did not need to be documented. 

In addition, this official stated that inspection procedures and the 

acceptability or unacceptability of deliverables were not documented 

because all parties involved in these processes were aware of the 

process. The COR for the FADS contracts added that updated quality 

management plans should have been provided and that this was an 

oversight.

USCIS provided a policy update that is intended to increase contracting 

officer oversight of the COR. According to a Contracting Officer, there are 

new directives regarding the expectations on how the contracting officer 

reviews COR performance. This official explained that, previously, policy 

stated that the contracting officer could provide evaluative information on 

the COR during a performance review. This individual stated that, now, 
Page 38 GAO-16-467 Immigration Benefits System

the contracting officer is required to provide mid- and end-of-the-year 

feedback to the COR supervisor regarding performance. The contracting 

officer stated that additional improvements in the area are likely going 

forward.

Although the CORs may be well informed regarding contractor 

performance and the procedures for inspection and acceptance, a fully 

documented file should be maintained. Until representatives responsible 

for day-to-day monitoring of Transformation Program contracts fully 

document the contract file, the program increases the risk of 

misunderstandings when disagreements or questions of interpretation 

arise. Moreover, without a fully documented file, the program may also be 

unable to effectively transfer responsibility if the COR is replaced during 

the contract.

Quality assurance surveillance plans were not developed

As discussed earlier, a leading practice in contract monitoring is the 

development of a quality assurance surveillance plan for performancebased contracts. According to the DHS COR guidebook, the plan should 

be prepared in conjunction with the performance work statement and 

should include, among other things, the performance standards, the 

methods of assessment, an assessment schedule, and a description of 

corrective actions to be taken if the required performance standards are 

not met. Development of a plan is important, as it provides a systematic, 

structured method for the evaluation of services and products that 

contractors are required to furnish. The quality assurance surveillance 

plan should focus on the quality delivered by the contractor, not on the 

steps taken or procedures used.

Contractors developed quality management plans, but these plans did not 

meet the requirements of a quality surveillance plan as articulated in DHS 

guidance.51 The COR and Contracting Officer for the FADS contract 

stated that they rely on quality management plans and performance 

scorecard guidance to guide contract surveillance activities. Quality 

management plans submitted by three of the four FADS contractors 

 51The program management support services and training and instructional design 

services contracts are not performance-based. As such, we did not expect a quality 

assurance surveillance plan.
Page 39 GAO-16-467 Immigration Benefits System

discuss proposed or planned metrics by which to evaluate each 

performance requirement, along with potential methods for selfinspection. Moreover, guidance for completing the monthly performance 

scorecard discusses observable behavior, possible metrics, and 

additional questions for OTC staff to consider. However, these documents 

do not reflect committed metrics against which to measure contractor 

performance; instead, they reflect potential metrics. Moreover, 

documentation does not define the methods of assessment the COR will 

undertake, how often an assessment will be performed, or actions the 

COR will take if performance is not met.

The FADS Contracting Officer acknowledged the need to develop a 

quality assurance surveillance plan for performance-based contracts, and 

did not know why one had never been developed. The Contracting Officer 

explained that, originally, USCIS intended to develop a quality assurance 

surveillance plan for the FADS contract. However, no one followed 

through on this action and, since then, the Contracting Officer has 

changed. The current Contracting Officer stated that various actions are 

under consideration that may help ensure that quality is more closely 

monitored for future FADS contracts. For example, the Contracting Officer 

is actively involved in a re-compete of the original FADS contract, which 

will expire in late 2016, and expects to include a quality assurance 

surveillance plan as part of the new contract. The Deputy Chief of the 

USCIS Contracting Office provided a draft quality assurance surveillance 

plan for OIT’s Agile development contract in May 2016, but did not 

provide a date for when the document will be finalized.

Until USCIS incorporates the quality assurance surveillance plan to 

measure contractor performance for FADS, the USCIS Transformation 

Program risks deploying software code that does not meet the quality 

standards for development or testing intended for the contract and 

unplanned expenditures in the future to repair the software.

Program used contractor performance assessments

DHS uses the Contractor Performance Assessment Reporting System to 

assess and report performance on previous contracts, and applicable 

regulations and contract management guidance require entry of contract 

performance assessments in the system. According to DHS’s COR 

guidebook, the COR should prepare an assessment of contractor 

performance at least annually.
Page 40 GAO-16-467 Immigration Benefits System

USCIS documented performance assessments for all six contracts we 

reviewed. For example, the program management support services 

contract we reviewed was active in fiscal years 2011 through 2015. The 

COR submitted the performance assessments as required during this 

time. Contractor performance assessments reflected positive evaluations 

in the areas of quality, schedule, and management, which is consistent 

with the COR and Contracting Officer statements.

USCIS also documented a performance assessment for the training and 

instructional design contract. The contract was awarded in April 2014 and 

active in fiscal years 2014 and 2015. The COR submitted one 

performance assessment covering contractor performance from July 

2014 through July 2015. As with the program management support 

contract, monitored by the same COR as the training contract, the 

contractor performance assessment reflected positive evaluations in the 

areas of quality, schedule, and management, which is consistent with the 

testimony of the COR and Contracting Officer responsible for 

management of this contract.

Moreover, USCIS also documented performance assessments for its 

FADS contracts. The contracts we reviewed were awarded in September 

2014 and active in fiscal year 2015. The COR reported performance 

assessments for all four contractors in March 2016. The FADS contracts 

operate under a 6-month period of performance, rather than full-year 

option periods, and therefore performance assessments were not 

completed prior to exercising the next option. The Contracting Officer 

explained that the requirement in USCIS is for contractor performance to 

be assessed annually and, as such, this condition was not changed to 

accommodate the reduction in period of performance. To mitigate this 

fact, the Contracting Officer stated that they used the monthly 

performance scorecard in considering whether to exercise the next option 

for each contractor.

Although performance assessments were submitted for FADS contractors 

on time, these assessments did not always align with testimony of the 

COR and Contracting Officer or documentation in the contract file. For 

example, two of the FADS contractors were rated as being unsatisfactory 

in the area of cost control, leading one of the contractors to not concur 

with the assessment and request a re-evaluation. In prior interviews, the 

COR responsible for monitoring the FADS contracts stated that contractor 

performance was satisfactory and that no discrepancy reports had been 

filed. Since then, the Contracting Officer provided correspondence that 

was not included as part of the COR file, which indicated that USCIS 
Page 41 GAO-16-467 Immigration Benefits System

communicated with both contractors regarding a deviation from the 

agreed level-of-effort for the contract.

Until USCIS takes steps to improve the consistency with which 

contractors are evaluated in the Contractor Performance Assessment 

Reporting System and documents those steps, it and the Transformation 

Program will risk disputes in the annual assessment of contractor 

performance.

GAO’s Cost Estimating and Assessment Guide52 identifies a number of 

best practices that are the basis of effective program cost estimating and 

should result in reliable and valid cost estimates that management can 

use for making informed decisions. Specifically, a reliable cost estimate 

should be comprehensive (costs are neither omitted nor double counted); 

well-documented (the estimate is thoroughly documented, including 

source data and significance, clearly detailed calculations and results, 

and explanations for choosing a particular method or reference); accurate 

(the estimate is unbiased, not overly conservative or overly optimistic, and 

based on an assessment of most likely costs); and credible (the estimate 

discusses any limitations of the analysis from uncertainty or biases 

surrounding data or assumptions).

In February 2015, OTC updated the life cycle cost estimate for the 

Transformation Program, with an expected cost of no more than $3.1 

billion. DHS subsequently approved the revised estimate in April 2015 as 

part of the program re-baseline. Changes in the cost estimate accounted 

for, among other things, new contracts and a revised schedule to support 

the Agile software development strategy. Such changes represented an 

additional $275 million in acquisition cost.

We determined that OTC’s February 2015 cost estimate met the 

characteristics of a reliable estimate. Specifically:

• The program life cycle cost estimate was well documented. For 

example, USCIS officials demonstrated the estimate source data, the 

calculations performed, and estimating methodology used to derive 

 52GAO-09-3SP.

Program’s Life Cycle Cost 

Estimate Was Reliable
Page 42 GAO-16-467 Immigration Benefits System

costs, and step-by-step tracing of the basis of estimate for each cost 

element. Further, the documentation reflected the information in the 

technical baseline description and the program demonstrated that the 

cost estimate had been reviewed and accepted by management, 

including a recent briefing to management in March 2015 that clearly 

explained the cost estimate.

• The program estimate was substantially comprehensive. For example, 

the estimate tracked well to the technical scope defined in the cost 

estimating baseline document, had a work breakdown structure that 

was product-oriented and at an appropriate level of detail, and 

appeared to document major ground rules and assumptions. The 

estimate also included all life cycle costs, with the rationale of the 

program extending through 2033, assuming 15 years of operations 

after achieving full operational capacity. However, there were no costs 

for retirement or decommissioning documented in the life cycle cost 

estimate. Nevertheless, we found the estimate to be sufficiently 

comprehensive.

• The program estimate was substantially accurate. Specifically, the 

estimate was based on comparable historic programs, contained few 

or minor mistakes, and the main estimating technique of extrapolating 

costs from actual contractual data was used to estimate almost all of 

the remaining costs. Inflation was also applied correctly and was 

embedded within the cost model. Transformation officials consistently 

review USCIS planned versus actual costs, as the life cycle cost 

estimate is required to be updated annually by DHS. As part of this 

update, actual costs for prior and current years are documented in the 

program’s cost model and then reviewed and approved by 

management for the next life cycle cost estimate revision.

However, variances between planned and actual costs were not 

always tracked. In addition, the cost estimate is funded to the 80 

percent confidence level, which is conservative. Nevertheless, we 

found the estimate to be sufficiently accurate at the time it was 

developed.

• The program estimate was substantially credible. For example, 

USCIS conducted sensitivity analysis to test the impacts of changes 

to program cost drivers. In addition, the estimating team performed a 

cost risk uncertainty analysis on associated elements of the cost 

model to determine the probability that a specific cost target would be 

exceeded. The results of this analysis showed that the initial cost 

estimate was at the 22 percent confidence level. However, the 

program added contingency funding to increase the level of 

confidence to the 80th percentile. This high level of confidence 
Page 43 GAO-16-467 Immigration Benefits System

assumes most program risks will occur and cannot be successfully 

mitigated, which is conservative. In addition, program officials 

reported that the current version of the life cycle cost estimate does 

not include any crosschecks, as this was done for an earlier estimate 

and the results were still considered valid. Finally, while DHS 

reviewed the USCIS life cycle cost estimate and found it to be 

reasonable, no independent cost estimate was developed to check 

whether other estimating methods produced similar results. 

Nevertheless, we found the estimate to be sufficiently credible.

Prior to the change in acquisition strategy, USCIS spent more than 8 

years and more than $475 million in developing a system that proved to 

be unstable and decommissioned it in April 2016. This previous system 

proceeded through development despite limited oversight and challenges 

in program management. Given this history and the subsequent 

commitment of additional resources for a new system, it is more important 

than ever that USCIS consistently follow leading practices and agency 

policy and guidance in its system development efforts. Consistent 

implementation can provide reasonable assurance that the program will 

deliver a quality system that meets user needs in a timely and costeffective manner.

The USCIS Transformation Program has at least partially addressed 

many of the practices we identified for implementing Agile software 

development, adhering to leading practices and agency guidance in 

systems integration and testing, and monitoring the largest 

Transformation program contractors. Moreover, it developed a reliable 

estimate against which to monitor program cost performance on an 

annual basis. Partial adherence to the relevant practices shows the 

program has taken some important steps to help ensure effective 

development of its new system. The program has also shown 

improvement in addressing specific practices over time during the course 

of this review.

Nevertheless, the program is still inconsistently following leading 

practices and agency policies and guidance in Agile software 

development, systems integration and testing, and contract management. 

By not fully implementing documented policies and guidance, some of 

which are limited and do not address important leading practices, the 

program is once again increasing the likelihood that obstacles similar to 

those encountered with the original deployment of USCIS ELIS will be 

encountered again. In addition, the program runs the risks that 

Conclusions
Page 44 GAO-16-467 Immigration Benefits System

deployment of the system will continue to face delays, that the 

functionality deployed will be of limited quality, and that production issues 

will delay performance of the system and the processing of filings for 

citizenship and immigration benefits.

The USCIS Transformation Program is in a relatively early stage of 

development. However, as additional major releases are deployed, the 

complexity of the effort will increase, as will the risks associated with not 

having addressed management deficiencies. By implementing software 

development consistent with its policy and guidance, and following 

leading practices, along with introducing additional internal controls, 

USCIS will increase the likelihood that the Transformation Program will 

ultimately meet the expectations and needs of USCIS, DHS, and the 

people that the program intends to serve.

We are making 12 recommendations aimed at improving information 

technology program management:

To provide reasonable assurance that the program executes Agile 

software development for USCIS ELIS consistent with its own policies 

and guidance and follows applicable leading practices, we recommend 

that the Secretary of DHS direct the Director of USCIS to direct the 

USCIS CIO, in coordination with the DHS CIO and the Chief of OTC, to 

review and update, as needed, existing policies and guidance and 

consider additional controls to:

• Complete planning for software releases prior to initiating 

development and ensure software meets business expectations prior 

to deployment.

• Consistently implement the principles of the framework adopted for 

Agile software development.

• Define and consistently execute appropriate roles and responsibilities 

for individuals responsible for development activities consistent with 

its selected development framework.

• Identify all system users and involve them in release planning 

activities.

• Write user stories that identify user roles, include estimates of 

complexity, take no longer than one sprint to complete, and describe 

business value.

• Establish outcomes for Agile software development.

Recommendations for 

Executive Action
Page 45 GAO-16-467 Immigration Benefits System

• Monitor program performance and report to appropriate entities 

through the collection of reliable metrics.

To help manage the USCIS ELIS system, we recommend that the 

Secretary of DHS direct the Director of USCIS to direct the USCIS CIO, in 

coordination with the DHS CIO and the Chief of OTC, to review and 

update existing policies and guidance and consider additional controls to

• Conduct unit and integration, and functional acceptance tests, and 

code inspection consistent with stated program goals.

• Develop complete test plans and cases for interoperability and end 

user testing, as defined in the USCIS Transformation Program Test

and Evaluation Master Plan, and document the results.

To help improve oversight of selected Transformation Program contracts, 

we recommend that the Secretary of DHS direct the Director of USCIS to 

direct the Chief of the USCIS Contracting Office, in coordination with the 

appropriate contracting officer, to consider inconsistencies between policy 

and leading practices in contract administration and, as needed, institute 

controls to

• Clearly define measures against which to analyze differences 

between services expected and those delivered.

• Ensure CORs are maintaining complete contract files.

• Ensure quality assurance surveillance plans are developed when 

appropriate.

We received written comments on a draft of this report from DHS. The 

comments are reprinted in appendix V.

In the comments, DHS concurred with our recommendations and also 

described ongoing and planned actions to address 11 of them, including

milestones for completion. The planned steps the department described 

will likely be sufficient to address these 11 recommendations if they are 

effectively implemented. For example, for the recommendation aimed at 

monitoring and reporting on program performance, DHS stated that it is 

updating guidance for collecting Transformation Program metrics and 

expanding the number of metrics to include a better representation of 

actions. DHS intends to complete this update, along with others aimed at 

improving Agile software development, by January 2017. In addition, for 

two recommendations aimed at improving contract management, the 

department stated that the USCIS Office of Contracting will conduct a 

Agency Comments 

and Our Evaluation
Page 46 GAO-16-467 Immigration Benefits System

variety of training to increase awareness and conduct an internal audit to 

further ensure compliance. The department stated that it plans to 

complete actions associated with all eleven of our recommendations by 

July 2017.

Regarding our remaining recommendation that DHS write user stories 

that identify user roles, include estimates of complexity, take no longer 

than one sprint to complete, and describe business value, the department 

requested that GAO consider the recommendation resolved and closed, 

as implemented. The department stated that USCIS OIT and OTC 

currently follow a user story process that was implemented in April 2013 

where both written and oral communication ensures that the information 

necessary to understand user roles and business value is shared. 

Nevertheless, as noted in our report, we found that the process 

implemented in April 2013 resulted in user stories taking longer than one 

sprint to complete and that user stories did not adequately describe 

business value. As such, we do not believe that the recommendation can 

be closed as implemented until the program takes steps to improve its 

practices in this area. DHS also provided technical comments which we

have incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of 

this report earlier, we plan no further distribution until 8 days from the 

report date. At that time, we will send copies to the appropriate 

congressional committees, the Secretary of Homeland Security, and other 

interested parties. In addition, the report is available at no charge on the 

GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact 

me at (202) 512-4456 or chac@gao.gov. Contact points for our Offices of 

Congressional Relations and Public Affairs may be found on the last page 

of this report. GAO staff who made key contributions to this report are 

listed in appendix VI.

Carol C. Harris

Director

Information Technology Acquisition Management Issues
Page 47 GAO-16-467 Immigration Benefits System

List of Requesters

The Honorable Ron Johnson 

Chairman 

Committee on Homeland Security and Government Affairs

United States Senate

The Honorable Michael T. McCaul 

Chairman 

The Honorable Bennie G. Thompson 

Ranking Member 

Committee on Homeland Security 

House of Representatives

The Honorable Scott Perry 

Chairman 

The Honorable Bonnie Watson Coleman 

Ranking Member 

Subcommittee on Oversight and Management Efficiency 

Committee on Homeland Security 

House of Representatives

The Honorable Candice S. Miller 

House of Representatives

The Honorable Jeff Duncan 

House of Representatives
Appendix I: Objectives, Scope, and 

Methodology

Page 48 GAO-16-467 Immigration Benefits System

Our objective was to assess the extent to which the U.S. Citizenship and 

Immigration (USCIS) Transformation Program is using information 

technology (IT) program management leading practices. To do so, we 

assessed the extent to which USCIS is implementing Agile software 

development for USCIS ELIS; adhering to leading practices and agency 

guidance1 in system integration and testing; following leading practices in 

monitoring the largest Transformation Program contractors; and 

developing a life cycle cost estimate for the Transformation Program 

consistent with our cost estimating guidance.

To assess the extent to which the program is implementing Agile software 

development for USCIS ELIS, we identified Agile software development 

leading practices and guidance outlined in the following sources:

• Software Development: Effective Practices and Federal Challenges in 

Applying Agile Methods2

• U.S. Digital Services Playbook3

• TechFAR handbook4

• Department of Homeland Security (DHS), Agile Management 

Instruction5

• DHS, Agile Development Guidebook6

 1

For the purposes of this report, guidance refers to a variety of USCIS and program-level 

documentation, such as plans, guidelines, and team artifacts. 

2

GAO, Software Development: Effective Practices and Federal Challenges in Applying 

Agile Methods, GAO-12-681 (Washington, D.C.: Jul. 27, 2012).

3

Office of Management and Budget, U.S. Digital Services, Playbook (version pulled Dec. 

18, 2015). 

4

Office of Management and Budget, TechFAR: Handbook for Procuring Digital Services 

Using Agile Processes (draft version pulled Nov. 6, 2014).

5

Department of Homeland Security, Instruction Number: 102-01-004, Agile Development 

and Delivery for Information Technology, Revision #0, (Apr. 11, 2016). 

6

Department of Homeland Security, Draft, DHS Agile Development Guidebook (Sept. 2, 

2014). 

Appendix I: Objectives, Scope, and 

Methodology
Appendix I: Objectives, Scope, and 

Methodology

Page 49 GAO-16-467 Immigration Benefits System

• DHS, Agile White Paper7

• USCIS, Agile Development Policy8

• USCIS, Agile Processes and Practices Principles and Guidelines, 

version 4.09

• USCIS, USCIS Transformation Program: System Engineering Life 

Cycle Tailoring Plan, version 1.010

• USCIS, Team Process Agreements11

• Agile Metrics: Progress Monitoring of Agile Contractors12

• Principles Behind the Agile Manifesto13

• The Scrum Guide14

• Standard 26515 for Agile User Documentation15

• CMMI for Development, version 1.316

 7

Department of Homeland Security, Draft, Agile Methodologies in DHS: Agile Working 

Group White Paper (no date). 

8

U.S. Citizenship and Immigration Services, Management Instruction CIS-OIT-001, Office 

of Information Technology: Agile Development Policy (Apr. 10, 2013). 

9

U.S. Citizenship and Immigration Services, Agile Processes and Practices Principles and 

Guidelines, version 4.0 (Dec. 6, 2013).

10U.S. Citizenship and Immigration Services, USCIS Transformation Program: System 

Engineering Life Cycle Tailoring Plan, version 1.0 (March 2015).

11U.S. Citizenship and Immigration Services, USCIS Team Process Agreement for ELIS, 

version 1.1 (Jun. 3, 2014); USCIS Team Process Agreement for ELIS2, updated for 

releases 5.1, 5.2, and 6.0 (10/15/14 to 04/15/15); USCIS ELIS Team Process Agreement 

for the April 7, 2015 to September 30, 2015 Development Period; and USCIS ELIS Team 

Process Agreement for the July 22, 2015 to January 30, 2016 Development Period.

12SEI, Agile Metrics: Progress Monitoring of Agile Contractors, CMU/SEI-2013-TN-029 

(Jan. 2014). 

13http://agilemanifesto.org. 

14Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 

15ISO/IEC/IEEE, Systems and Software Engineering —Developing User Documentation in 

an Agile Environment, corrected version, ISO/IEC/IEEE 26515:2012(E) (Mar. 15, 2012).

16SEI, CMMI® for Development, Version 1.3, CMU/SEI-2010-TR-033 (Nov. 2010). 
Appendix I: Objectives, Scope, and 

Methodology

Page 50 GAO-16-467 Immigration Benefits System

• Defense Acquisition Agile Guide17

• Handbook for Implementing Agile in DOD IT Acquisition18

After reviewing the sources listed, we grouped key Agile software 

development processes into eight key areas: software release practices, 

software development events and principles, software development roles 

and responsibilities, user involvement in release planning, user story 

definition and development, user story prioritization, establishing 

outcomes for Agile software development, and tracking and reporting of 

program metrics. As part of our assessment of these eight areas, we 

reviewed relevant plans and documentation and interviewed USCIS staff. 

We determined whether the information was fully, partially, or not present, 

and recorded the presence of this information by recording our 

evaluations as “yes,” “partial,” or “no.” An element was a “no” if the 

program provided no evidence to satisfy any portion of the practice; 

“partial” if the program provided evidence that satisfied some, but not all, 

of the practice; and “yes” if the program provided evidence that it 

substantially satisfied all elements of the practice.

To obtain a better understanding of the program’s Agile software 

development approach, we performed on-site observations over a 3-week 

period. During these observations, we attended development team 

meetings including 2 sprint planning sessions, about 55 daily stand-ups, 

about 10 cross-team meetings, 1 user story demonstration, and 2 sprint 

reviews. We did not attend a retrospective because no retrospectives 

were held during the period we observed. In addition to observing team 

meetings, we documented program planning artifacts posted on the walls 

of team common areas and individual team rooms. We compiled our 

observations and noted potential issues.

To evaluate compliance with software release practices, we first reviewed 

USCIS Agile software development policy and guidelines to define the 

requirements for release planning and readiness reviews. We reviewed 

required documentation associated with planning for major releases of 

 17MITRE, Defense Agile Acquisition Guide: Tailoring DOD IT Acquisition Program 

Structures and Processes to Rapidly Deliver Capabilities (Mar. 2014). 

18MITRE, Handbook for Implementing Agile in Department of Defense Information 

Technology Acquisition (Dec. 15, 2010).
Appendix I: Objectives, Scope, and 

Methodology

Page 51 GAO-16-467 Immigration Benefits System

USCIS ELIS, which were releases 5.0, 5.1, 6.1, 6.2, and 7.1. This 

included reviewing, among other documentation, capabilities and 

constraints, team process agreements and project oversight plans. We 

also reviewed approval memos associated with each release to 

determine the date when approval was granted and to analyze the level 

of review performed prior to approval. We then compared the approval 

date in the memo to the first date in which development occurred for a 

release to determine if development occurred prior to approval. We also 

reviewed all required documentation associated with deployment (i.e. 

readiness) of all major releases of USCIS ELIS as of March 2016. These 

releases were releases 5.0, 5.1, and 6.1. This included a review of, 

among other documentation, automated test scripts and security reports. 

As with release planning, we also reviewed approval memos associated 

with deployment of each major release to analyze the level of review 

performed prior to approval.

To evaluate adherence to software development events and principles 

described in the selected framework, we reviewed team process 

agreements governing development of USCIS ELIS, Quality Assurance 

Team assessments of development team adherence to USCIS Agile 

software development policy and contract documentation for the 

development of USCIS ELIS to determine the framework within which 

development teams were operating. Based on this framework, we 

reviewed related artifacts, such as

• capabilities and constraints documents reflecting sub-features 

commitments made for each release;

• team process agreements; and

• consolidated release assessments prepared by the Quality Assurance 

Team.

This documentation allowed us to track development teams and the 

program through the development of each major software release of 

USCIS ELIS. To evaluate the reliability of data contained in the 

supporting documentation, we submitted questions and interviewed 

USCIS staff regarding internal controls of the consolidated release 

assessments completed by quality assurance analysts to monitor various 

software development events. We determined that data contained in 

these assessments were sufficiently reliable for our purposes. We then 

interviewed USCIS staff to reconcile any gaps identified in our findings.
Appendix I: Objectives, Scope, and 

Methodology

Page 52 GAO-16-467 Immigration Benefits System

To evaluate software development roles and responsibilities, we reviewed 

the Office of Transformation Coordination (OTC) management model, 

which describes the program’s approach to managing USCIS ELIS, and 

associated descriptions provided by OTC, general roles and 

responsibilities defined for the framework referenced for development of 

USCIS ELIS (e.g. Scrum), OTC organizational chart, and team rosters for 

developing and maintaining USCIS ELIS. We also interviewed USCIS 

staff, including government project managers and product owners, and 

contractor development team technical leads to further understand roles 

and responsibilities. To validate testimony, we compared statements 

against program documentation, such as contractor performance work 

statements, quality management plans, consolidated release 

assessments, and team process agreements.

To evaluate user involvement in release planning, we first reviewed 

artifacts provided by the Business Integration Division within OTC, 

including process scenarios, participation lists, and meeting minutes. We 

confirmed participation lists with representatives from the USCIS Service 

Center Operations and Field Operations directorates. We then reviewed 

memoranda of agreement governing the assignment of subject matter 

experts from other USCIS directorates or program offices in support of 

the program. We interviewed USCIS staff to determine the relative 

involvement of groups in the development of user stories during release 

planning.

To evaluate user story definition and development and user story 

prioritization, we reviewed versions of the backlogs for releases 5.0, 5.1, 

6.1, 6.2, and 7.1 provided to us between November 2014 as part of a 

prior audit19 and March 2016. To evaluate the reliability of data contained 

in the backlogs, we submitted questions and interviewed USCIS staff 

regarding internal controls of the Agile life cycle management software 

tools used by the program at various stages of development. We 

determined that user story data contained in the backlogs were 

sufficiently reliable for our purposes. We further interviewed product 

owners responsible for overseeing development of USCIS ELIS and 

maintenance of the release backlogs, along with other staff facilitating 

 19GAO, Immigration Benefits System: Better Informed Decision Making Needed on 

Transformation Program, GAO-15-415 (Washington, D.C.: May 18, 2015). 
Appendix I: Objectives, Scope, and 

Methodology

Page 53 GAO-16-467 Immigration Benefits System

product owners, to determine how user stories were ordered and 

prioritized to maximize the value of each development cycle. We 

validated product owner statements by comparing those statements 

against the sequence of development, as reflected in the release 

backlogs.

To evaluate the identification, tracking, and reporting of program metrics, 

we interviewed USCIS staff, including the Quality Assurance Team, who 

are responsible for the collection of program metrics, to understand the 

metrics collected by the program and the purpose for each metric. Based 

on these responses, we analyzed documentation provided by the 

program to show evidence of data collection and application, including but 

not limited to:

• work in progress spreadsheets for user story and incident tracking;

• consolidated release assessments;

• development progress metrics such as cumulative flow diagrams;

• external customer satisfaction surveys; and

• deployment cycle times.

To evaluate the reliability of data contained in the supporting 

documentation, we submitted questions and interviewed USCIS staff on 

internal controls for systems or manual collection on which metrics are 

based. We determined that data used to compile program metrics for 

reporting and internal monitoring were sufficiently reliable for our 

purposes. Any concerns with data reliability are raised in the report; 

however, we determined that these concerns did not impact our findings 

or conclusions.

To evaluate the program’s monitoring and reporting against reliable Agile 

software development, cost, schedule, and performance outcomes, we 

first interviewed USCIS staff, submitted written questions, and reviewed 

USCIS Agile software development policy and guidelines to understand 

the program’s goals specific to Agile software development. We then 

reviewed the USCIS Transformation Program Acquisition Program 

Baseline (program level) and USCIS Office of Transformation 

Coordination Lifecycle Cost Estimate, which define the approved cost and 

schedule goals for the program. We compared these goals against 

progress reflected in program reports to management, such as program 

management reviews and Executive Steering Committee presentations, 

and the integrated master schedule maintained by the program to 
Appendix I: Objectives, Scope, and 

Methodology

Page 54 GAO-16-467 Immigration Benefits System

determine the extent to which the program was meeting cost and 

schedule goals. We also reviewed the backlogs for releases 6.1, 6.2, and 

7.1 to determine the reliability and traceability between sub-feature 

commitments reported to the DHS Office of Program Accountability and 

Risk Management and user stories developed in those releases. As with 

our reporting on tracking of other program metrics (i.e. customer 

satisfaction), any concerns with data reliability are raised in the report; 

however, these concerns did not impact our findings or conclusions.

To assess the extent to which the program is adhering to leading 

practices and agency guidance in USCIS ELIS integration and testing, we 

began by reviewing the following sources to determine areas appropriate 

to select for this review:

• DHS, Agile Development Guidebook20

• USCIS, Test and Evaluation Master Plan21

• USCIS, Team Process Agreement22

• USCIS Test Plan for Release 5.1-6.0 (10/15/2014-4/15/2015)

• USCIS Test Plan (updated July 2015)

• USCIS, Agile Development Policy23

• USCIS, Agile Processes and Practices Principles and Guidelines, 

version 4.024

• Software Testing Standards25

 20Department of Homeland Security, Pre-Decisional Draft, DHS Agile Development 

Guidebook. 

21U.S. Citizenship and Immigration Services, USCIS Test and Evaluation Master Plan: 

Electronic Immigration System, version 1.0. 

22U.S. Citizenship and Immigration Services, USCIS ELIS Team Process Agreement for 

the April 7, 2015 to September 30, 2015 Development Period; and USCIS ELIS Team 

Process Agreement for the July 22, 2015 to January 30, 2016 Development Period. 

23CIS-OIT-001.

24U.S. Citizenship and Immigration Services, Agile Processes and Practices Principles 

and Guidelines, version 4.0. 
Appendix I: Objectives, Scope, and 

Methodology

Page 55 GAO-16-467 Immigration Benefits System

The areas we selected for review during this audit reflected USCIS’s 

implementation of continuous integration development and continuous 

testing practices (such as unit and integration tests), automating of those 

practices as applicable, delivering business value to the customer (such 

as functional acceptance and end user tests), and ensuring that the 

software performs as intended in a larger software ecosystem 

(interoperability tests).

More specifically, we selected the following areas for review:

• establishing an environment and procedures for continuous 

integration and testing of code;

• performing continuous testing through the use of unit and integration 

tests, functional acceptance tests, and code inspection; and

• developing complete test plans and cases for interoperability and end 

user testing and documenting those results.

The scope of our assessments spanned the development period for 

USCIS ELIS releases 6.1 and 6.2, which began in April 2015.

After reviewing the documents listed above, we identified processes, 

tools, and practices necessary to support continuous integration of Agile 

software development, such as configuration management, frequent code 

submission, automation of builds and tests, the ability to detect and stop 

defective code from progressing in the development process, and the 

ability to provide rapid feedback to developers and stakeholders about the 

health of the development process. With respect to continuous 

integration, we evaluated the existence or absence of the practices 

applied to the USCIS ELIS development process as required by USCIS 

policy and guidelines. We determined whether a given practice or process 

was implemented, partially implemented, or not implemented and applied 

corresponding “yes”, “partial”, or “no” ratings.

To evaluate the environment and procedures established by USCIS for 

continuous integration and testing of the newly developed or modified 

code, we reviewed the team process agreement for releases 6.1 and 6.2, 

test plan, and Quality Assurance Team documentation. This allowed us to 

 25IEEE, IEEE Standard 29119, Software and System Engineering – Software Testing, 

Part 1: Concepts and Definitions, Part 2: Test Process, Part 3: Test Documentation (New 

York, NY: September 2013). 
Appendix I: Objectives, Scope, and 

Methodology

Page 56 GAO-16-467 Immigration Benefits System

determine what processes and tools are employed in USCIS ELIS 

production, how newly developed code flows through various quality 

gates and development stages, and whether the process is automated. 

Based on document review, we requested script files from the tools used 

in code development to verify implementation of the continuous 

integration and testing. We also requested outputs from the automated 

tools that indicated successful code build as well as unsuccessful outputs 

to verify the existence of the feedback mechanism. In addition, we 

observed live demonstrations of the tools used in developing USCIS ELIS 

and obtained artifacts from those tools, such as build and deployment 

scripts, logs, and screenshots. We also interviewed technical leads from 

each of the Flexible Agile Development Support contracts as well as the 

Senior Technical Lead responsible for the continuous integration and 

deployment of USCIS ELIS about the implementation of continuous 

integration and testing practices during the USCIS ELIS development life 

cycle.

To evaluate continuous testing, we examined the program’s efforts to use 

unit and integration tests, functional acceptance tests, and code 

inspection. For unit and integration testing, we interviewed agency 

officials, including team technical leads and the Quality Assurance Team 

lead. We also reviewed the USCIS Transformation Program Test and 

Evaluation Master Plan, which established code coverage standards for 

unit and integration tests. In addition, we reviewed team process 

agreements and the USCIS ELIS Test Plan. We requested and reviewed 

test summary reports and Quality Assurance Team monthly and weekly 

reports. Based on the document review, we were able to determine 

whether unit or integration tests were being executed. Based on the 

review of the Quality Assurance Team reports, we identified deficiencies 

listed in the reports and, to the extent possible, whether the problem 

persisted over time. With respect to code coverage, we evaluated 

documentation of code coverage provided by the program. This 

documentation was based on data from an automated tool used in 

developing USCIS ELIS. We determined the data to be reliable for the 

purpose of our audit.

To evaluate whether USCIS delivered intended functionality for each user 

story to the user, we evaluated whether functional acceptance tests were 

executed. We requested artifacts associated with a random sample of 80 

user stories drawn from the release 6.1 and 6.2 backlog submitted to us 

in October 2015. We requested documentation and participated in an onsite evaluation of the results captured in the software development tool. 

We evaluated whether each of the user stories had associated 
Appendix I: Objectives, Scope, and 

Methodology

Page 57 GAO-16-467 Immigration Benefits System

acceptance criteria, since those are the basis for the acceptance tests. 

We also verified whether test results were attached with the user story. 

Based on the evaluation of the documentation, we determined whether 

the program met this requirement (“yes”), met the requirement partially 

(“partial”), or did not meet the requirement (“no”). To fully meet the 

requirement, we expected the program to provide documentation of 

acceptance criteria and acceptance tests for all user stories in the 

sample. A partial rating indicates that some, but not all, of the expected 

artifacts were captured in the software development tool.

To evaluate code inspection, we began by reviewing leading practices26

and USCIS policy. Based on the documentation provided and testimonial 

evidence, we determined that the program uses both automated and 

manual code inspection in developing USCIS ELIS. In particular, we 

determined that USCIS had integrated automated code inspection into its 

continuous integration process and that code is inspected manually by 

the Quality Assurance Team and by developers during the peer review 

process. To evaluate the code inspection process, we interviewed 

technical team leads and Quality Assurance Team staff. We also 

observed a demonstration of the tools used for automated code 

inspection and how those tools are used in the software development life 

cycle. In addition, we reviewed coding standards used by the automated 

tools to identify code quality issues. With respect to peer review manual 

inspection, we reviewed logs from the version control software and 

verified whether the code was peer reviewed prior to being merged with 

the code used for deployment into production for a random sample of 80 

user stories selected from the backlog for releases 6.1 and 6.2.

We also examined the extent to which USCIS has developed complete 

test plans and cases for interoperability and end user testing and 

documented those results. To evaluate interoperability and end user 

testing, we identified leading practices27 related to test plans, test cases 

developed and based on the test plan, and test completion reports that 

are compiled after execution of the test cases. We requested and 

reviewed interoperability and end user test plans, test cases, and test 

completion reports. We also interviewed testers from the interoperability 

 26IEEE, IEEE Standard 29119-1. 

27IEEE, IEEE Standard 29119–3. 
Appendix I: Objectives, Scope, and 

Methodology

Page 58 GAO-16-467 Immigration Benefits System

test team and live interface test team and agency officials responsible for 

end user testing. In addition, we submitted questions about end user 

testing practices and processes to the independent testing team. We then 

determined whether test documentation provided by the program met 

industry practices by noting if the documentation addressed all of the 

leading practices (“yes”), a subset of leading practices (“partial”) or none 

of the practices (“no”).

To assess the extent to which the USCIS Transformation Program 

followed leading practices in monitoring the largest Transformation 

Program contracts, we selected the two largest Transformation Program 

contracts managed by OTC—the program management and support 

services contract and the training and instructional design services 

contract—and the four largest contracts managed by the Office of 

Information Technology (OIT)—four Flexible Agile Development Services 

(FADS) contracts. Specifically, using the acquisition plan approved for the 

Transformation Program in March 2015, we identified all of the active 

contracts identified in the plan as of July 2015 and the total amount of 

funds to be obligated to Transformation Program contractors. To narrow 

the number of contractors selected for review, we selected contracts 

managed by both OTC and OIT. As shown in table 6, the total contract 

dollars for the two OTC managed contractors we selected represented 

approximately 90 percent ($50 million out of a total of $56 million) of the 

overall active contracting dollars planned for OTC contracts. The total 

contract dollars for the four OIT managed contractors we selected 

represented approximately 77 percent ($235 million out of a total of $307 

million) of the overall active contracting dollars planned for OIT contracts.

Table 6: Dollar Amount for USCIS Contractors for the Transformation Program, as reflected in the USCIS Transformation 

Program Acquisition Plan as of March 2015

Contract name Description

Total value 

(millions) Ownership

Front End Visualization Services Software visualization tool that provides the capability for 

incorporating visualized requirements into the requirements 

generation and management process.

$0.24 OTC

Requirements Development and 

Support Services

Services to assist and support the requirements management 

process to create and maintain the requirements/product 

backlog for USCIS ELIS.

2.60 OTC

Operational Test and Evaluation 

Services

Operational test and evaluation services 2.80 OTC

Training and Instructional Design 

Services

Allows creation and implementation of instructor-led and 

Web-based training.

11.36 OTC
Appendix I: Objectives, Scope, and 

Methodology

Page 59 GAO-16-467 Immigration Benefits System

Contract name Description

Total value 

(millions) Ownership

Program Management and Support 

Services

Program management support to OTC. 38.60 OTC

Transformation Integration & 

Configuration Services

Integrates source code received from different sources, 

prepares it for integrated testing, and promotes builds 

between test, stage, and production environments. Activities 

pertain to USCIS ELIS and some of the systems supporting 

USCIS ELIS.

11.90 OIT

Certification and Accreditation Certification and accreditation of the Solution Architect 

systems.

28.32 OIT

Architecture and Design Services Services for the definition and governance of architecture and 

design of all USCIS systems. 

31.74 OIT

Flexible Agile Development Services Sources Agile development teams to support USCIS ELIS 

development efforts.

52.99 OIT

Flexible Agile Development Services Same description. 58.58 OIT

Flexible Agile Development Services Same description. 61.09 OIT

Flexible Agile Development Services Same description. 62.24 OIT

Source: GAO analysis of USCIS Transformation Program Acquisition Plan I GAO-16-467.

Note: Two additional contracts were current as of the date the Acquisition Plan was approved. These 

contracts provided continued development services, until the Flexible Agile Development Services 

contract was awarded, and independent verification and validation services. Both contracts were set 

to expire in May 2015 and were therefore not considered during our selection of contractors for 

review.

We compared USCIS’ contract management practices to criteria outlined 

in the following sources:

• The Federal Acquisition Regulation

• Standards for Internal Control in the Federal Government28

• A Guide to Best Practices for Contract Administration,

29 and

• DHS, Contracting Officer’s Representative Essential Element 

Guidebook, Spiral 130

 28GAO/AIMD-00-21.3.1.

29Office of Federal Procurement Policy, A Guide to Best Practices for Contract 

Administration (Washington, D.C.: October 1994).

30Department of Homeland Security, Contracting Officer’s Representative (COR) 

Essential Element Guidebook, Spiral 1 (September 2012).
Appendix I: Objectives, Scope, and 

Methodology

Page 60 GAO-16-467 Immigration Benefits System

• DHS, Department of Homeland Security Acquisition Regulation31

• DHS, Department of Homeland Security Acquisition Manual32

• DHS, Acquisition Workforce Policy Number 064-04-003, Revision 0233

• DHS, Office of Procurement Operations: Acquisition Manual, version 

1.034

We grouped key contract management processes into four key areas 

using the sources listed: training and ethics requirements for contracting 

officer’s representative (COR), documentation required for contracts that 

are not performance-based, definition of performance measures for 

contracts that are performance-based, and ongoing monitoring and 

reporting. We then assessed these areas based on six key sub-areas 

described in the body of this report: COR training and ethics, nonperformance based contracting, performance measures, COR file 

maintenance, quality assurance surveillance plans, and contractor 

performance assessments. As part of our assessment, we reviewed 

relevant plans and documentation and interviewed USCIS staff. For each 

contract, we determined whether the information was fully, partially, not 

present, or not applicable for each key sub-area. For each contract, we 

recorded our evaluations as “yes,” “partial,” “no,” or “not applicable.” An 

element was a “no” if USCIS provided no evidence that satisfied any 

portion of the practice; “partial” if USCIS provided evidence that satisfied 

some, but not all, of the practice; and “yes” if USCIS provided complete 

evidence that it fully satisfied the practice. We consolidated our evaluation 

of the four FADS contracts as the results of our assessment were 

consistent across each contract.

 31Department of Homeland Security, Department of Homeland Security Acquisition 

Regulation (June 2006).

32Department of Homeland Security, Department of Homeland Security Acquisition 

Manual (October 2009).

33Department of Homeland Security , Federal Acquisition Certification for Contracting 

Officer’s Representatives and Appointment and Revocation, Acquisition Workforce Policy 

Number 064-04-003, Revision 02 (Aug. 8, 2012).

34Department of Homeland Security, Office of Procurement Operations: Acquisition 

Manual, version 1.0 (May 1, 2015).
Appendix I: Objectives, Scope, and 

Methodology

Page 61 GAO-16-467 Immigration Benefits System

To conduct our assessment for each of the selected contracts, we 

reviewed the statements of work or performance work statements, 

monthly contractor performance summaries, resource expenditure and 

monitoring reports, contract deliverables, other COR file documentation, 

and annual assessments. We collected data and conducted interviews 

with representatives from USCIS on their contract management roles and 

responsibilities, including the COR and contracting officer for each 

contract, and other representatives from the contracting office for further 

insight on the extent to which the program had followed monitoring 

policies and procedures.

To examine the extent to which the USCIS cost estimate for the 

Transformation Program was consistent with leading practices, we relied 

on the leading practices contained in our cost guide.35 We evaluated the 

USCIS Office of Transformation Coordination Life Cycle Cost Estimate, 

version 2.0, dated February 2015, including assumptions and results, to 

determine whether the cost estimate was comprehensive, welldocumented, accurate, and credible. We obtained and reviewed 

documentation, including the program office estimate, software cost 

model, and risk and uncertainty analysis. We also met with key program 

officials, such as USCIS staff and contractors responsible for preparing 

and maintaining the program cost estimate, to present the preliminary 

results of our assessment of the program’s cost estimate and obtained 

explanations and clarifications.

To analyze the program cost estimate against the leading practices 

documented in our cost guide, we reviewed program documentation and 

conducted interviews with program officials to understand the 

methodology USCIS had used to create its estimates. We developed an 

overall assessment rating for each leading practice using the following 

definitions:

• Not met: USCIS provided no evidence that satisfied any portion of the 

practice.

• Minimally met: USCIS provided some evidence that satisfied a small 

portion of the practice.

 35GAO-09-3SP.
Appendix I: Objectives, Scope, and 

Methodology

Page 62 GAO-16-467 Immigration Benefits System

• Partially met: USCIS provided evidence that satisfied about half of the 

practice.

• Substantially met: USCIS provided evidence that satisfied a large 

portion of the practice.

• Met: USCIS provided evidence that satisfied all of the practice.

We shared our initial assessment with USCIS, obtained feedback and 

additional supporting documentation, and then summarized the final 

results for our report.

We conducted this performance audit from May 2015 through July 2016, 

in accordance with generally accepted government auditing standards. 

Those standards require that we plan and perform the audit to obtain 

sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that 

the evidence obtained provides a reasonable basis for our findings and 

conclusions based on our audit objectives.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 63 GAO-16-467 Immigration Benefits System

This appendix describes in more detail our evaluation of the eight key 

practices for managing and implementing Agile software development. It 

does not present new findings. Rather, the information detailed in this 

appendix is intended to assist USCIS in implementing the 

recommendations described in our report.

For the Transformation Program, USCIS has modified the Department of 

Homeland Security’s (DHS) traditional system development life cycle to 

accommodate the requirements of a program implementing an Agile 

software development approach. This modified approach includes two 

primary points of review that take place at the beginning and end of the 

development of a portion of software, called a release, before it is made 

available to users. These points are the release planning review, during 

which the program plans for the software release, and the release 

readiness review, during which the program ensures that the software 

that has been developed meets business expectations prior to 

deployment.

Release planning review

The USCIS Transformation Program Systems Engineering Life Cycle 

Tailoring Plan tailors DHS’s project planning review, systems definition 

review, and preliminary design review out of the development life cycle 

and replaces it with the release planning review. According to USCIS 

Agile Development Policy,

1 each release cycle will include a release 

planning review. This review is intended to communicate what is planned 

for the release and ensure that all information technology (IT) and other 

resources are committed to support it. According to the plan, to ensure a 

business decision is made as to whether the investment in the release 

cycle is justified by the expected results (i.e. capabilities to be produced), 

a release should not proceed to release activities (e.g. development or 

testing) until it has secured release planning review approval.

To support this business decision, USCIS Agile Development Policy

requires the development and approval of a set of core release planning 

review documents. The core documents for release planning are:

 1

CIS-OIT-001. 

Appendix II: Management of Agile Software 

Development for USCIS ELIS

Completing planning for 

software releases prior to 

initiating development and 

ensuring software meets 

business expectations prior to 

deployment
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 64 GAO-16-467 Immigration Benefits System

• release capabilities and constraints;2

• team process agreement;3

• definition of done;

• project oversight plan;4

• critical task schedule;

• test and evaluation plan; and

• privacy impact assessment, if required.

The program’s tailoring plan identifies additional artifacts required for the 

release planning review. These additional artifacts include an update to 

the acquisition plan and life cycle cost estimate.

The program prepared the required documents articulated in USCIS Agile 

Development Policy to support its approval to proceed with development 

of USCIS ELIS for system releases 5.0 through 7.0. Specifically, for the 

five major releases planned as of February 2016, the program had 

developed and reviewed capabilities and constraints, a team process 

 2

The capabilities and constraints document defines the mission capabilities to be 

developed for a release in support of a specific product line (i.e. Deferred Actions for 

Childhood Arrivals). Typically, capabilities are documented via a traceability matrix that 

specifies the specific sub-features for development and whether the sub-feature is new 

development, re-use of existing functionality, similar to existing functionality but requiring 

some re-work, or not applicable/not to be developed in the release. This document also 

defines constraints such as standards (i.e. enterprise architecture compliance), time 

frames, architecture and design, performance expectations (i.e. system response time), 

and other areas (i.e. reliance on existing program and technical management structures 

that were put in place to support legacy USCIS ELIS development).

3

USCIS policy does not define the purpose or provide guidelines for completing a team 

process agreement. However, according to the Chief of the Applied Technology Division, 

the team process agreement is generally the agreement within the team of how they will 

work with each other. Among other things, the team process agreements we reviewed 

defined the development methodology to be used (i.e. Scrum), length and number of 

iterations, number of teams and team structure, foundational USCIS Agile practices, and 

Agile tools available to teams.

4

USCIS policy does not define the purpose or provide guidelines for completing a project 

oversight plan. However, according to the Chief of the Applied Technology Division, the 

project oversight plan generally addresses how the team will interact with the oversight 

groups. Among other things, the project oversight plans we reviewed defined the planned 

costs for each release, general risk mitigation procedures, and information radiators, such 

as physical task boards or feature progress charts.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 65 GAO-16-467 Immigration Benefits System

agreement that included the definition of done, the project oversight plan, 

and the critical task schedule. The test and evaluation plan and privacy

impact assessment are documents that are updated with each release, as 

needed, and therefore do not require a unique submission as part of each 

release.

In addition to the required documentation, the program office also 

provided evidence for approval from the Chief Information Officer (CIO) to 

proceed with development of all planned releases. Specifically, the 

program office provided evidence of approval for release 5.0 in May 2013 

and subsequent extensions, release 5.1 in November 2014, releases 6.1 

and 6.2 in April 2015, and release 7.1 in July 2015. As part of this review 

and approval, a quality assurance analyst reviewed the required 

documents cited.

However, system development for individual releases began prior to 

completion of the release planning review. Specifically, for four of the five 

major releases, development began prior to the designated authority’s 

approval to begin development. Table 7 describes the date on which 

approval was granted relative to the start of development.

Table 7: Release Planning and Development Dates

Release

Release planning review 

approval date

Date on which 

development began

5.0 5/28/2013 6/12/2013

5.1 11/5/2014 10/1/2014

6.1 4/7/2015 3/18/2015

6.2 4/7/2015 3/4/2015

7.1 7/30/2015 7/22/2015

Source: GAO analysis of USCIS documentation I GAO-16-467.

According to the Chief of the Applied Technology Division in the USCIS 

Office of Information Technology (OIT), a release planning review 

provides authority for the program to work for the next 6 months. He 

added that it is not for a specific scope because scope can change. 

Moreover, this official stated that a release planning review is used to 

execute contracts and any delays in the review would impact the ability of 

development teams to continue developing software code. This official 

stated that this partly contributed to development activities occurring prior 

to release planning review approval.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 66 GAO-16-467 Immigration Benefits System

However, we found that the release planning review period did not always 

align with a six-month period of development. For example, release 

planning review approval for releases 6.1 and 6.2 was granted in April 

2015. Release planning review approval was then granted for release 7.1 

only three, rather than six, months later in July 2015. Further, the 

intention of the release planning review, as articulated in USCIS policy 

and the program tailoring plan, is to approve functionality for a specific 

product line, not for general six-month development periods.

In addition to the sequencing of development, the program did not provide 

evidence that the life cycle cost estimate or acquisition plan were updated 

during the release planning review. The Chief of the Program and 

Resource Management division in OTC stated that the program 

acquisition plan is updated annually rather than as a part of the release 

planning review. Moreover, in written responses, the program stated that 

the life cycle cost estimate is also only submitted as a formal update on 

an annual basis. Although the program stated that the life cycle cost 

estimate is routinely updated with actual costs, USCIS did not provide 

evidence to support this statement. Release planning review memos also 

included no evidence that these additional documents were updated or 

discussed.

Collectively, the practices being followed by the program for a release 

planning review and the policy and guidelines governing this process do 

not align. This includes the artifacts produced and updated as part of this 

process. The USCIS CIO concurred with our assessment that USCIS 

policy and guidance needs to be updated to reflect such inconsistencies 

between policy and practice.

Release readiness review

The USCIS Transformation Program Systems Engineering Life Cycle 

Tailoring Plan tailors the production readiness review out of DHS’s 

system development life cycle and replaces it with the release readiness 

review. According to this plan, release readiness reviews are conducted 

as needed and are focused on ensuring all elements of the release are 

complete including testing, documentation, approvals, accreditations, and 

provisions for sustainment. The plan designates the release manager, in 

consultation with other stakeholders, to approve that a release meets 

business expectations prior to deployment (e.g. be made available for 

use).
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 67 GAO-16-467 Immigration Benefits System

To support this decision, USCIS Agile Development Policy also requires 

the development and approval of a set of core release readiness review 

documents. The core documents for release readiness are:

• system design document;

• automated and manual build and installation scripts;

• automated and manual test scripts;

• automated and manual deployment scripts;

• internal change control board or change control board package

• security plan; and

• security assessment report.

The program documented some artifacts required for the release 

readiness review and was granted approval to proceed with deployment 

for all major releases for USCIS ELIS. As of February 2016, major 

releases of USCIS in production included releases 5.0, 5.1 and 6.1. In 

support of these reviews, the program provided evidence that system 

design documentation and automated and manual scripts were submitted 

as part of release readiness reviews.

While the program documented some artifacts required for release 

readiness review, it did not provide evidence of having reviewed or 

updated other required artifacts. Specifically, the program did not provide 

an updated security assessment report, security plan, or change control 

documentation. Moreover, although the program provided manual and 

automated scripts, it provided no evidence that the actual scripts were 

assessed as part of the release readiness review. For example, as part of 

the approval granted for limited deployment of functionality in support of 

release 6.1, officials assessed a general test summary report, but the 

program did not provide evidence that actual automated and manual 

build, installation, test, or deployment scripts were reviewed. Instead, 

officials discussed testing in general prior to deployment, such as whether 

performance or end user testing had been completed.

The USCIS CIO stated that policy and guidance reflect an old process. 

This official explained that the program is no longer holding a release 

readiness review for each release because deployments (e.g. 

enhancements) occur weekly. Instead, the decision to hold a release 

readiness review is based on his discretion as USCIS CIO, although a 

review is still held prior to deploying a major release. The Chief of the 

Applied Technology Division added that approval to complete a release 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 68 GAO-16-467 Immigration Benefits System

readiness review and deploy a major release is a business, not a 

technical, decision. Both this official and the USCIS CIO explained that, 

from their perspective, all of the technical work associated with testing the 

system has taken place well in advance of a release readiness review. 

For example, the USCIS CIO stated that the continuous integration 

pipeline allows for a regular build of the code, in which tests are 

constantly run to ensure the software works as intended.

As with the release planning review, we found that collectively the 

practices being followed by the program for a release readiness review 

and the policy and guidelines governing that process do not align. This 

includes the artifacts produced and updated as part of this process. The 

USCIS CIO concurred with our assessment that USCIS policy and 

guidance needs to be updated to reflect such inconsistencies between 

policy and practice.

The USCIS Transformation Program Systems Engineering Life Cycle 

Tailoring Plan does not identify the specific Agile methodology or 

document the basis for selecting the specific methods. Rather, the 

tailoring plan states that the USCIS CIO (e.g. Lead Technical Authority) is 

responsible for providing guidance and direction related to all technical 

matters and IT process issues, including Agile and lean methodologies, 

the systems engineering life cycle processes, and continuous integration 

and continuous delivery of code to meet planned capabilities in USCIS 

ELIS. Rather than identifying a specific methodology, this plan discusses 

the tailored review approach under an Agile software development 

methodology.

USCIS Agile Development Policy also does not prescribe a particular 

methodology for Agile software development. Specifically, USCIS policy 

states that current policy and standards are silent on the specific Agile 

methodologies to be used and on the appropriate guidelines and controls 

for Agile projects. This policy instead is intended to provide more detailed 

direction to USCIS on the use of Agile methods. In particular, the policy 

considers projects to be Agile if they meet select requirements such as 

frequent delivery and the assignment of a product owner (see the next 

section on roles and responsibilities for a further discussion of the product 

owner). In the absence of a defined Agile software development 

methodology, USCIS policy and guidance calls for the creation of a team 

process agreement during release planning to articulate the methodology 

agreed upon between the development teams and the business.

Adhering to the principles of 

the framework adopted for 

implementing Agile software 

development
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 69 GAO-16-467 Immigration Benefits System

The team process agreements for planned major releases (e.g. 5.0, 5.1, 

6.1, 6.2, and 7.1) stipulate that teams will adhere to the Scrum 

framework. A commitment to this framework is further reinforced in 

consolidated release assessments prepared by the Quality Assurance 

Team.

Scrum is one approach for implementing Agile software development. 

Scrum’s roles, events, artifacts, and rules, which can collectively be 

considered its principles, are defined in the Scrum framework.5 The 

framework identifies basic rules to guide a team in the application of 

various events that are organized into a series of “sprints.”6 These events 

(e.g. sprint planning, daily Scrum, sprint review, and sprint retrospective) 

are to occur once each sprint and the daily Scrum is to occur each day. 

Specifically, the Scrum Guide defines the following events:

• Sprint planning: Sprint planning defines what will be delivered in the 

software increment7 resulting from the upcoming sprint and how the 

work needed to deliver the increment will be achieved. The 

development team8 defines the functionality that will be developed 

during the sprint.

• Daily Scrum: The daily Scrum is a 15-minute meeting held at the

same time and place each day by the development team to 

synchronize activities and create a plan for the next 24 hours. This is 

accomplished by inspecting the work completed since the last day’s 

Scrum and forecasting the work that can be done before the next one.

 5

Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (July 2013).

6

A sprint is a set period of time, for example, two weeks, during which the development 

team is expected to complete tasks related to the development of an increment of 

software.

7

An increment is a piece of working software that adds to previously created increments. 

A collective set of increments comprise a software release.

8

Development team is the role within a Scrum team accountable for managing, 

organizing, and doing all development work required to create a releasable increment of 

product every sprint.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 70 GAO-16-467 Immigration Benefits System

• Sprint review: A sprint review is an informal meeting held at the end of 

the sprint to inspect the software increment and adapt the backlog9 if 

needed. The Scrum team10 and stakeholders collaborate about what 

was accomplished in the sprint and decide on what will be 

accomplished next.

• Sprint retrospective: A sprint retrospective occurs after the sprint 

review and prior to the next sprint planning meeting. The retrospective 

is an opportunity to discuss how the last sprint went with regard to 

people, relationships, process, and tools; identify and order the major 

items that went well and potential improvements; and create a plan for 

implementing improvements to implement in the next sprint.

See figure 8 for a depiction of a traditional Scrum process flow within the 

release planning and readiness review processes and time frames 

specific to the development of USCIS ELIS.

 9

The product backlog is an ordered list of the work to be accomplished in order to create, 

maintain, and sustain a product. In the case of USCIS, the product backlog is broken into 

a series of major and minor release backlogs. A release backlog includes all user stories 

and the associated features to be delivered in a release. A sprint backlog includes stories 

pulled from the release backlog to be delivered in a sprint.

10The Scrum team is a self-organizing group consisting of a product owner, development 

team, and Scrum master. Product owner is the role in Scrum accountable for maximizing 

the value of a product, primarily by incrementally managing and expressing business and 

functional expectations for a product to the development team(s). A Scrum master is the 

role within a Scrum team accountable for guiding, coaching, teaching and assisting the 

team and its environments in a proper understanding and use of Scrum.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 71 GAO-16-467 Immigration Benefits System

Figure 8: Scrum Process for USCIS ELIS Release Process 

USCIS ELIS development teams consistently held daily Scrum sessions. 

Specifically, according to consolidated release assessments produced by 

the Quality Assurance Team, a daily Scrum took place nearly every day.

However, while daily Scrums occur regularly, consolidated release 

assessments show that development teams are not consistently holding 

sprint planning, sprint review, or sprint retrospective discussions. Starting 

with the development of releases 6.1 and 6.2, the program changed sprint 

review procedures so that a review is held every other sprint, instead of 

each sprint, as prescribed by Scrum. Further, consolidated release 

assessments show that retrospectives are being conducted in an erratic 

manner, instead of at the end of each sprint, as prescribed by the Scrum 

framework and in team process agreements. For example, one 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 72 GAO-16-467 Immigration Benefits System

development team went nearly five months without conducting a 

retrospective. In another case, a development team held a retrospective 

on August 25, 2015, and then not again until October 21, 2015. 

Collectively, only two of the 12 development teams working on USCIS 

ELIS held a retrospective each sprint. In the case of those two teams, a 

retrospective was only held each sprint starting in July 2015. Prior to July 

2015, they were held less frequently.

In addition, within each event, teams are not consistently adhering to the 

basic principles of the event. The consolidated release assessments 

indicate that teams adhere to certain principles, such as involving 

stakeholders and obtaining feedback during the sprint review. Further, 

acceptance criteria is defined and discussed by the development team as 

part of sprint planning. However, other principles are not consistently 

performed. For example, development teams are not implementing 

improvements identified during the team retrospective at least 30 percent 

of the time. Moreover, as discussed in the next subsection addressing 

roles and responsibilities, product owners are not always available to the 

team for events, such as sprint planning. Consolidated release 

assessments noted that the product owner is absent from sprint planning 

over 20% of the time.11

The program did not adhere to principles of the Scrum framework 

because USCIS policy and guidance does not consistently align with 

those principles. According to the Chief of the Capability Delivery Division 

within OTC, the program uses some aspects of the Scrum framework, but 

also follows the principles of Agile software development by continuously 

tailoring practices to align with the concept of lean software development. 

He added that the program should not be held accountable to the Scrum 

 11In answering the question of whether the product owner attended planning, consolidated 

release assessments found 57 percent of the time they did, 21 percent of the time they did 

not, and 21 percent of the time this question was not applicable. Guidance for completion 

of the consolidated release assessment notes that this response can be marked as a 

“yes” if a delegate attends as long as delegate explicitly states that he can make 

decisions. In a written response, the Chief of the Delivery Assurance branch in OIT stated 

that the team has monitored sufficient USCIS ELIS releases to state that product owner 

involvement is a general rule of behavior for all of the development teams, but there are 

exceptions to every rule. He added that, given the size of the meetings and number of 

attendees involved, it is easy to make errors regarding roles. 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 73 GAO-16-467 Immigration Benefits System

Guide12 principles because this framework is not meant to provide major 

government investments with guidance on navigating through acquisition 

oversight and regulatory requirements. The USCIS CIO added that Scrum 

is not a USCIS requirement and some teams can use other methods, 

such as Kanban or a hybrid process such as Scrum and Kanban.13 This 

official stated that USCIS has defined the characteristics that the team’s 

process has to follow in order to be considered Agile, but that teams are 

encouraged to find their own process that works best. However, the 

USCIS CIO added that each Agile framework includes its own rules that 

make it successful and that teams should be more or less following the 

rules for the framework.

While we recognize that USCIS policy allows for team flexibility in Agile 

software development practices and the need for continuous 

improvement of those practices, Scrum is the framework teams have 

adopted for development of USCIS ELIS and it is reflected throughout 

various program documents. This was reiterated by the Quality 

Assurance Team and is reflected in the consolidated release 

assessments.14 The Scrum Guide15 is clear that Scrum’s overall roles, 

artifacts, events, and rules are unchangeable and although it is possible 

to implement only parts of Scrum, the result is not Scrum. Scrum exists 

only in its entirety and functions as a container for other techniques, 

 12Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 

13Agilealliance.org, a nonprofit organization committed to advancing Agile development 

principles and practices, defines the Kanban method as an approach to continuous 

improvement that relies on visualizing the current system of work scheduling, managing 

“flow” as the primary measure of performance, and whole-system optimization. As a 

process improvement approach, it does not prescribe any particular practices. The focus 

of Kanban is to optimize the throughput of work by visualizing its flow of work through the 

process, limiting work in progress, and explicitly identifying policies for the flow of work. 

Kanban has distinct differences from other popular Agile methodologies, primarily the fact 

that it is not based on timeboxed iterations, but rather allows for continuous prioritization 

and delivery of work. 

14Consolidated release assessments allow for a quality assurance analyst to document 

whether a team is adhering to the practices of Scrum, Kanban, or a combination of the 

two. The consolidated release assessment for each team developing USCIS ELIS 

document if the team is adhering to Scrum practices.

15Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 74 GAO-16-467 Immigration Benefits System

methodologies, and practices. While the program office may find it 

inappropriate to be held accountable to the Scrum Guide,

16 this is the 

authoritative source that defines the practices and principles that govern 

Scrum. Allowing ad hoc deviations from Scrum framework events, 

whereby some are honored and others are not, results in reduced 

transparency and lost opportunities for learning and adaptation. For 

example, without consistent retrospectives during each sprint, the 

development team will not be able to discuss and commit to potential 

improvements to implement in the upcoming sprint, thereby impacting 

team performance.

According to the Scrum Guide, the Scrum team consists of a 

development team, a product owner, and a Scrum master to optimize 

flexibility, creativity, and productivity. Scrum team roles and their related 

responsibilities consist of, among other things:

• Development team: The development team is responsible for 

delivering a software increment each sprint. The only position title in a 

development team is to be that of the developer—there are no 

exceptions to this rule. Optimal development team size is between 

three and nine team members.

• Product owner: The product owner is to be the person responsible for 

maximizing the value of the product, the work of the development 

team, and the backlog. How this is accomplished may vary widely 

across organizations, Scrum teams, and individuals.

• Scrum master: The Scrum master is to be responsible for ensuring 

that the principles of Scrum are understood and enacted by ensuring 

that the Scrum team adheres to Scrum theory, practices, and rules. In 

addition, the Scrum master is a go-between for the Scrum team with 

those outside the team. The Scrum master does not manage or 

oversee the Scrum team.

 16Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013).

Defining and consistently 

executing appropriate roles 

and responsibilities for 

individuals responsible for 

development activities
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 75 GAO-16-467 Immigration Benefits System

Development team

Based on our review, we found that the program did not consistently 

execute two basic principles as defined in the Scrum Guide17 when 

standing up development teams: the team should be between 3 and 9 

people and team members should be referred to solely as ‘developers.’

According to the backlogs for releases 6.1, 6.2, and 7.1, along with the 

supporting team rosters, there are 12 development teams working on 

USCIS ELIS. According to the team rosters provided as of August 2015, 4 

of the 12 development teams include, or plan to include, more than 9 

team members. The program office also stipulates that management and 

technical leads are members of the development teams. With the addition 

of these team members, an additional 5 development teams (9 in total) 

include, or plan to include, more than 9 team members. This finding was 

further substantiated in resource expenditure reports. For example, in 

April 2015, 14 team members charged hours for one development team.

The Chief of OTC stated that the resources (i.e. staff) for the contractor 

development teams are divided into teams in order to meet contractual 

obligations. The program manager in OIT stated that the contracts were 

written with the expectation that the standard team size would be 

between 7 and 9 staff. This official explained that some people might be 

moving across teams, so the count of the development team size might 

change over time. We requested additional evidence from the program to 

help reconcile this matter, but the program did not provide the requested 

information.

In addition to team size, contracts and the program team roster define 

specific positions for the development team members. For example, 

contracts identify team members as a system architect, test engineer, 

application programmer, database specialist, business process 

reengineering, and other such titles. Development team rosters further 

classify members as developer, tester, 508 tester, business analyst, 

management lead, and technical lead.

 17Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 76 GAO-16-467 Immigration Benefits System

The program manager in OIT stated that all individuals in the 

development teams are first considered developers, then each individual 

has an area of focus. This official reiterated that, while development team 

members may carry certain titles, each team member is performing all of 

the functions necessary for the team rather than limiting their work to their 

predefined role. The Chief of the Applied Technology Division in OIT 

added that each development team member has different skills and may 

specialize in an area in order to meet the different needs of the 

development team. However, this official confirmed that development 

team members are referred to by specific titles beyond simply ‘developer.’ 

As a result, in delineating team roles, the program may not fully embrace 

the ideas underlying the new approach.

Product owner

The Scrum Guide18 and U.S. Digital Services Playbook recommend 

assigning a product owner and holding that person accountable. To 

effectively perform this role, there must be a single product owner who 

has appropriate authority and responsibility to assign tasks and work 

elements; make business, product, and technical decisions; and be 

accountable for the success or failure of the overall service. The product 

owner is ultimately responsible for how well the service meets the needs 

of its users.

To supplement the traditional product owner role, OTC initially drafted a 

management model that further defined the product owner role and 

responsibilities. According to this management model, which describes 

the program’s approach for managing USCIS ELIS, the product owners 

should be assigned responsibility for no more than four development 

teams.

Product owners are granted the necessary authority and responsibility to 

perform their duties. The two product owners assigned to oversee 

releases 6.1, 6.2, and 7.1 stated that they retain responsibility for the 

release backlog and user stories within that backlog, including accepting 

and prioritizing user stories. The program management model reinforces 

this assertion and states that the product owner is responsible for 

 18Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 77 GAO-16-467 Immigration Benefits System

ensuring the development of a USCIS ELIS release that meets the vision 

and business requirements of USCIS. It also notes that the purpose of 

this position is to serve as a single point of contact for the development 

teams.

However, while product owners assume authority for each release, 

product owners are responsible for over 4 development teams. 

Specifically, the product owner for release 7.1 was at times responsible 

for work being conducted by all 12 development teams working on USCIS 

ELIS. The other product owner, responsible for releases 6.1 and 6.2, was 

at times responsible for work being performed by 10 development teams.

The two product owners stated that the number of development teams 

they work with has not presented any challenges, but the results of the 

Quality Assurance Team’s consolidated release assessments, as well as 

prior product owner testimony and our own observations, indicate 

otherwise. During development of release 5.0, one product owner stated 

that it was a challenge to accommodate more than one team and she had 

to stagger her time between the teams to support sprint planning and 

maintain meaningful dialogue between the product owner and the team. 

In addition, the consolidated release assessments indicate that product 

owners did not attend 21 percent of sprint planning meetings (26 of 122 

responses). This idea was reinforced in our own observations of daily 

stand-up meetings and sprint planning. Our observations covered sprint 

56 only and therefore are not a reliable indicator of broader concerns, but 

product owner availability was an issue voiced by development team 

members and observed by our team during that time period.

The Chief of OTC stated that, when the product owner role and 

responsibilities were first defined, the program office may have felt that 

only 4 development teams were needed to develop a product line. 

However, in reality, more development teams are necessary to complete 

a product line. As such, the span of control for the product owner has 

increased. To accommodate and support this increased span of control, 

the Chief added that, in December 2015, OTC revised its reporting 

structure such that product managers now report to the product owners to 

help extend the product owners’ reach (previously product owners had no 

staff reporting to them). The Chief of OTC noted that the product 

managers assume more of the work of the product owner but have not 

assumed product owner authority.

The Chief of OTC provided an update to the OTC management model

that removes the 4 development team limitation on product owner 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 78 GAO-16-467 Immigration Benefits System

responsibilities. However, removing this restriction does not address the 

larger concern that the more development teams a product owner is 

responsible for, the less time the product owner is able to spend with 

each team. Consequently, this can impact product owner effectiveness in 

performing his or her assigned duties. The addition of a product manager 

may help to mitigate the issue, but not granting those individuals authority 

adds another layer of communication between the development team and 

product owner. This added layer of communication may, in turn, impact 

team performance, as decisions still need to be made by a product 

owner.

According to USCIS guidance, lack of inclusion and transparency with the 

development team’s decision making and processes can result in a 

disengaged product owner, or one that makes decisions without adequate 

consideration of challenges faced by the team. Moreover, as we discuss 

later, the program has faced challenges in completing work within 

committed timeframes. Product owner availability may be a contributing 

factor.

Scrum master

Scrum teams did not include an individual explicitly designated as a 

Scrum master. The Chief of OTC stated that, prior to creating the new 

position of product manager, project managers were expected to assume 

the role of Scrum master. This expectation was further corroborated in 

some contractor documentation. For example, one of the contractor 

quality management plans states that the contractor recognizes the 

benefit that the government Scrum masters provide in managing the 

team. However, according to the Chief of OTC, the program determined 

that project managers were not effective as Scrum masters because the 

development team members, who were contractors, did not take direction 

from them.

Instead, the Chief of OTC stated that an expectation was set for 

development teams to perform autonomously and provide their own 

Scrum masters, or the equivalent. However, the program did not provide 

evidence that such a role is assumed by the contractor or that such an 

expectation was conveyed to the contractor by the program office. The 

need to set clear expectations between the contractor and government in 

contracting for program services is further discussed in the contract 

management section of this report.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 79 GAO-16-467 Immigration Benefits System

According to leading practices for Agile software development,19 early on 

a program should determine who will be the users of the system and 

involve all types of users in planning the user stories. This approach helps 

ensure that the system reflects the various tasks that will be conducted by 

the users. General users of USCIS ELIS fall into three general categories: 

external customers (e.g. applicants), internal users (e.g. USCIS 

adjudicator), and external agency users (e.g. Department of State). In this 

case, the term “customer” refers to an individual, representative, or 

organization applying for immigration benefits or related services.

OTC has defined and documented internal user roles for USCIS ELIS. 

For example, the program maintains a set of user access roles that 

loosely correspond to the users of the system within USCIS, such as a 

supervisor or an adjudicator. According to one product owner, this user 

access list is added to as subject matter experts define additional roles, 

not already captured, as part of planning for a new release.

External customers are also identified in release planning documentation. 

For example, process scenarios provided by OTC’s Business Integration 

Division identify the customer or representative as a user of the system. 

Such process scenarios might reflect a future work flow for release 6.1 

(processing of deferred action for childhood arrivals applications).

The program provided evidence that both internal users and external 

customers are involved in release planning or development activities. For 

example, in preparation for release 7.1, the program included 50 

representatives from the Field Operations Directorate, 13 representatives 

from the Service Center Operations Directorate, and a host of other 

representatives from various USCIS offices. The program also provided 

summaries from focus groups held with external customers in September 

2014 and October 2015, along with more recent evidence of ongoing 

conversations.

 19Software Engineering Institute, CMMI® for Development, version 1.3, CMU/SEI-2010-

TR-033 (Nov. 2010); ISO/IEC/IEEE, Systems and Software Engineering—Developing 

User Documentation in an Agile Environment, corrected version, ISO/IEC/IEEE 

26515:2012(E) (Mar. 15, 2012); U.S. Digital Service, Playbook (version pulled Dec. 18, 

2015).

Identifying users of the system 

and involving them in release 

planning activities
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 80 GAO-16-467 Immigration Benefits System

However, the program did not provide evidence of having defined or 

considered external agency users in release planning. Moreover, there 

are no user stories in the release backlogs that clearly call out 

functionality to be built in support of external agency users. Further, the 

program did not provide evidence that such users were included in 

planning or development activities prior to release deployment.

The Chief of the Capability Delivery Division in OTC stated that 

individuals from agencies that interface with and access USCIS ELIS 

were not involved in release planning because such users only access 

information. Further, the Chief of OTC explained that external 

organizations gain access to data from USCIS ELIS via a system 

interface, the Person Centric Query System, which is supported by 

another piece of infrastructure within USCIS, the Enterprise Service Bus. 

See appendix IV for more information on these additional interfaces. 

However, by omitting these users from release planning discussions, the 

program risks overlooking functionality that USCIS ELIS could provide in 

support of the respective agency missions.

User stories should identify user roles, include estimates of complexity, 

take no longer than one sprint to complete, and describe business value. 

Based on our review of the backlogs for releases 5.0, 5.1, 6.1, 6.2, and 

7.1, as of March 2016, user stories identified a user role and contained an 

estimate at least 90 percent of the time. However, user stories did not 

define the associated business value. Further, of the approximately 1,000 

user stories tracked by the Quality Assurance Team, over 30 percent 

were not sized appropriately and took over a full sprint to develop.

Defining a single user role

According to our review of release backlogs, the program has shown 

improvement in defining the user role over time and has substantially 

addressed this practice with its most recent set of user stories. The 

backlog for release 5.0 does not differentiate between a typical user story 

and other work, such as work associated with 508, interface, and enduser testing. For example, one user story reads “Interface: Develop 

Person Search to interface with PCQS 4.0.1.” About 37 percent of user 

stories in the backlog for release 5.0 reflect either other work or user 

stories that are not assigned to user roles. Since then, the program has 

migrated to a new software tool that allows for differentiation between 

user stories, enhancements to existing functionality, and bugs. Our review 

of user stories for 5.1 through 7.1 as documented in this new tool 

indicates that less than 10% of user stories do not define a user role.

Writing well-structured user 

stories
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 81 GAO-16-467 Immigration Benefits System

Establishing a user story estimate

The program has shown improvement in including user story estimates 

over time, and has substantially addressed this practice with its most 

recent set of user stories. Most user stories did not previously include 

estimates that reflect the amount of work associated with the story 

relative to other user stories. Specifically, 231 of the 1,343 user stories 

contained in August 2015 release backlogs did not include an estimate. 

Of these, more than 80% of user stories for release 7.1 did not include an 

estimate. In addition, updated user stories that were provided in 

November 2015 for releases 6.1, 6.2, and 7.1, included no estimates. 

According to the program office, when OTC migrated from one user story 

management tool to another, data was compromised, which resulted in 

user story estimates being removed. Since then, the program has taken 

steps to reconcile the matter. In particular, backlogs for releases 6.1, 6.2, 

and 7.1, provided in March 2016, show that only 56 of 980 (6 percent) of 

user stories did not include an estimate of the amount of work associated 

with the story.

Appropriately sizing the user story

User stories are taking a full sprint or longer to be completed, indicating 

that the program may be underestimating the amount of work and time 

required to complete it for user stories. Specifically, approximately 30% of 

user stories (287 of 951) contained in the Quality Assurance Team’s 

November 2015 documentation of user stories took more than a full sprint 

(i.e. 14 days) before gaining acceptance.

The Chief of OTC speculated that user stories may take longer than a full 

sprint to be developed due to constraints outside of the control of a 

development team. For example, a user story may require testing with an 

interface in order to pass the associated acceptance criteria. However, if 

access to the interface is unavailable, completion of the story will be 

delayed. Nevertheless, the program did not provide evidence to support 

specific reasons for why so many user stories took more than a full sprint 

to complete.

In addition, while factors outside the program’s control might be one 

reason that user stories are taking longer to be developed, there may be 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 82 GAO-16-467 Immigration Benefits System

internal reasons for a delay, such as resources shifting from one 

development team to another, unavailability of the product owner, issues 

with the developer’s physical environment,20 or an inadequate estimate of 

the work needed to complete a user story. Nevertheless, development 

teams should learn over time how to better estimate the work required to 

complete individual user stories, leading to better results. Instituting 

additional controls in the development process could help to identify 

potential causes and limit the number of user stories that exceed a full 

sprint.

Defining the business value

Most user stories do not define a business value. Program officials stated 

that requirements for defining user stories that are described in USCIS 

guidance should be updated to be less prescriptive. USCIS officials 

stated that the definition for a user story does not need to include 

information about the value to the business or the actual user because 

ongoing interactions between the product owner and development team 

help address questions related to value. Officials further explained that 

development teams are familiar enough with the stories that they 

inherently understand who the story applies to and what value it serves. 

The program did not provide evidence to support that the value of a user 

story is continuously discussed with the development team; however, it is 

reasonable to assume this occurs based on the level of collaboration 

among business and development team staff. Nonetheless, defining user 

story value is not solely for the benefit of a development team, but also for 

a product owner, and affords transparency to those less integrated with 

the process, such as program management, in understanding the 

business need.

USCIS Agile Development Policy and guidance state that release 

planning should yield a prioritized backlog of work. A product owner is 

responsible for providing business direction to the team by managing and 

 20Each developer accesses a virtual machine, software that allows a single host to run 

one or more guest operating systems, on which they perform various development 

activities such as writing software code or associated tests. In order to perform these 

activities, that environment must be accessible (i.e. virtual machine physically boots up) 

and properly configured (i.e. current version of an operating system or software tools are

loaded on the machine). 

Prioritizing user stories to 

maximize the value of each 

development cycle
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 83 GAO-16-467 Immigration Benefits System

prioritizing the release backlog and making decisions about what to build 

and in what order. Guidance also recommends that the product owner 

prioritize the release and sprint backlogs based on risk, value, and 

business priorities. It suggests that failing to place the highest risk items 

of high business value first can jeopardize the ability to deliver a release 

successfully and that the product owner should make trade-off decisions 

for balancing risk and value. In addition, the Scrum Guide21 states that the 

product owner should order user stories in the backlog with more detail 

for near-term stories than long-term stories. Collectively, USCIS policy 

and guidance, along with the Scrum Guide,

22 allow the product owner 

complete discretion and flexibility in determining what constitutes value 

and the order in which user stories are prioritized.

Product owners provided an overview of the prioritization process they 

follow. According to the product owners for releases 6.1, 6.2, and 7.1, 

user story priorities are based on the need to develop specific features 

associated with an upcoming release. The product owner for release 7.1 

added that user stories are prioritized based on a release road map. 

According to the product owner, the road map lays out a sequence of 

features to develop, and then user stories are defined and placed into 

iterations/sprints. The product owner stated that user stories are 

prioritized based on where in the future workflow a particular task occurs. 

The goal is for a major release (i.e. product line) to establish a basic endto-end workflow. The product owner added that features reflected earlier 

in the workflow do not necessarily provide a greater value. Instead, these 

features are a higher priority because they must occur before a 

subsequent feature can occur on the end-to-end workflow.

The product owner for releases 6.1 and 6.2 confirmed this approach, 

stating that user stories are prioritized by feature, with the feature 

priorities reflected in release road maps. According to the product owner, 

the development teams are expected to work according to the feature 

priorities. The product owner stated that the priority order for features is 

based first on those needed for end-to-end processing of an application. 

 21Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 

22Ken Schwaber and Jeff Sutherland, The Scrum Guide:TM The Definitive Guide to Scrum: 

The Rules of the Game (Jul. 2013). 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 84 GAO-16-467 Immigration Benefits System

For example, the first feature needed might be account creation. After 

there is a product that can perform end-to-end processing and pass 

testing, the product owner may begin to prioritize additional new features. 

This product owner added that any work associated with a new interface 

is typically developed sooner as it may be riskier to complete. The 

product owner stated that the teams only prioritize features; they do not 

prioritize sub-features or the user stories that comprise those subfeatures.

The product owner for releases 6.1 and 6.2 stated that work for these 

releases has undergone a number of changes that impacts traceability 

between the road map and release backlog. According to the product 

owner, initially, development focused on a particular set of functionality 

associated with a product line. However, with changes in the external 

environment, the program office stopped development on this product line 

and shifted attention elsewhere. Following additional external events, the 

product owner, at the recommendation of the program office, shifted back 

to the original focus of development.

In addition, the product owner explained that some features were built 

across a number of sprints due to the availability of internal users. 

According to the product owner, she initially prioritized the development of 

features based on how she, in conjunction with the program office, 

determined was most appropriate. However, with increased access to 

other USCIS personnel, such as experts from service center operations, 

the product owner revisited her prioritization of user stories.

The program has provided product owners with significant flexibility in 

making development decisions and the order in which development 

occurs, and product owners demonstrated that they are taking steps to 

establish the user story priorities that they determine are appropriate and 

maximize value.

In our prior work on effective Agile software development practices,23 we 

have found that a key practice for the successful adoption of Agile 

software development is to identify measurable outcomes, not outputs, of 

 23GAO, Software Development: Effective Practices and Federal Challenges in Applying 

Agile Methods, GAO-12-681 (Washington, D.C.: Jul. 27, 2012).

Setting outcomes for Agile 

software development
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 85 GAO-16-467 Immigration Benefits System

what the program wants to achieve using the approach. An example of 

this practice is creating a vision for project outcomes (such as a decrease 

in processing time by a specific percent in a set time), rather than outputs 

(such as the amount of code produced).

The Transformation Program has not defined or set goals for the 

transition to Agile software development. Officials from the Quality 

Assurance Team stated that the goal of all metrics collected is to create a 

full picture of the strength of the USCIS ELIS process and product quality. 

These officials added that the program’s ability to deploy software 

frequently demonstrates that the Transformation Program is improving its 

ability to develop software. In addition, the Chief of OTC stated that the 

outcome or goal for the program is to deploy a product line within the 

baseline cost, schedule, and performance parameters. Furthermore, the 

USCIS CIO stated that Agile software development is a “best practice” for 

many reasons. The program’s goal is to use best practices wherever 

possible.

Collectively, all of the various responses to the question of the goals and 

associated outcomes for the transition to Agile software development 

supports the fact that the program did not establish a well understood 

goal, or set of goals. There also is a gap in expected outcomes. For 

example, the program did not provide evidence of having defined, and 

program officials did not talk about, how often they would like to deploy 

software or what level of process and product quality the program hopes 

to achieve. Without a sense for the goals and expected outcomes for 

Agile software development, while the program may be able to monitor 

progress, there is no basis against which to evaluate this progress to 

determine the success of Agile software development in the agency.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 86 GAO-16-467 Immigration Benefits System

The US Digital Services Playbook24 states that a program should develop 

metrics in order to measure performance. The playbook, along with other 

guidance,25 identifies metrics such as how well the service is meeting 

user needs at each step of the service, the quality of software code being 

developed, the progress of development from a release or program 

perspective, and the consistency of progress of each development team.

OTC and OIT are collecting a number of metrics to monitor program 

performance and are routinely discussing these metrics at weekly and 

monthly reviews. The metrics include

• Ability of service to meet user needs: Customer satisfaction surveys.

• Code quality: Production defects and incidents, automated code 

scanning results, code issue counts, and test activity timelines.

• Progress of development for a release: Cumulative flow diagram, 

burnup chart, rate of user story acceptance, and rate of ticket 

acceptance.

• Consistency of team progress: velocity.26

Moreover, in March 2015, the DHS Office of the CIO Enterprise Business 

Management Office, the DHS Office of Program Accountability and Risk 

Management, and OTC signed a memorandum of agreement regarding 

periodic program performance reporting. The purpose of this agreement 

was to set forth the terms by which these parties would work together to 

manage performance metrics reporting requirements to indicate if the 

program is adequately progressing along the approved cost, schedule, 

and performance baseline. This agreement set the expectation for 

 24In August 2014, the White House formally launched the U.S. Digital Service, a small 

team made up of digital talent that works with agencies to remove barriers to service 

delivery and help remake the digital experience that people and businesses have with 

their government. As part of this group’s efforts, it created the U.S. Digital Services

playbook. This playbook is a set of 13 key “plays” drawn from successful practices from 

the private sector and government that, if followed, should help government build effective 

electronic services.

25GAO-12-681 and Software Engineering Institute, Agile Metrics: Progress Monitoring of 

Agile Contractors, CMU/SEI-2013-TN-029 (January 2014).

26A team’s ‘velocity’ is defined as the total number of story points that the team can 

complete in a sprint. Velocity for each team can be measured and tracked over time, with 

the goal of having each team converge on a stable velocity.

Monitoring and reporting on 

program performance through 

the collection of reliable 

metrics
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 87 GAO-16-467 Immigration Benefits System

monthly performance metrics to be submitted to the DHS Office of the 

CIO and Office of Program Accountability and Risk Management by OTC.

The metrics selected and those reported to DHS do not always provide a 

full picture of performance and could be improved. In our assessment of 

program metrics, we found no issues with metrics associated with 

external customer satisfaction and the quality of USCIS ELIS, but did 

identify concerns with the reliability, consistency, or applicability of other 

metrics, such as progress and scope.

Customer satisfaction

In written responses, the Quality Assurance Team stated that the 

program measures the satisfaction of external customers. Specifically, the 

program management and evaluation division in OTC maintains a 

performance reporting dashboard. The Quality Assurance Team, in turn, 

monitors customer satisfaction metrics maintained on this dashboard. 

According to program officials, the Performance Management Division 

(now a branch under the Program and Resource Management Division) 

conducts the customer satisfaction survey once each month. The Chief of 

OTC noted that the customer satisfaction for external customers for the I90 form was 93 percent. The Chief stated that, if it goes lower, the

program would consider system changes. In addition to overall 

satisfaction, this survey also includes open-ended questions for tracking 

the aspects of the system that users like and other aspects that could be 

improved.27

However, the program does not measure internal user (e.g. adjudicator) 

satisfaction. Officials from the Quality Assurance Team stated that they 

monitor issues raised by adjudicators and adjudicator representatives 

during program reviews and retrospectives. Further, the Chief of the 

Capability Delivery Division stated that the operational test agent obtains 

internal user feedback on USCIS ELIS. However, the Chief of OTC 

explained that incident management (i.e. reporting of defects or issues by 

the field and service centers) and operational test agent reports have not 

 27We did not validate the process by which the program management and evaluation 

division collects and calculates customer satisfaction (i.e. reviews the hard data 

supporting this metric or reviews the individual surveys, question how customers are 

selected, etc.).
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 88 GAO-16-467 Immigration Benefits System

proven to be a useful tool for obtaining internal user feedback. As such, 

the Chief stated that OTC is developing a method for capturing internal 

user satisfaction. Program officials did not elaborate on the steps the 

program will take to collect internal user satisfaction or provide a time 

frame for collecting such metrics.

By not establishing metrics to obtain user feedback, the program limits its 

understanding of the value being delivered with each software release.

Code quality

In written responses, the Quality Assurance Team reported that it 

calculates production defect/incident metrics, automated code scanning 

results, and code issue counts. In addition, the team collects code 

development metrics to gauge the quality of code delivered during a 

sprint. These metrics are included as part of the monthly status report and 

used for high-level planning.

In comparing the measurements associated with these metrics with our 

findings in the area of systems integration and testing, we did not identify 

any apparent concerns associated with these metrics. The results of 

measurements associated with these metrics identify underlying 

challenges the program is facing with product quality. For example, 

production metrics show that the rate in which issues (e.g. defects, 

incidents, bugs) are found exceeds the rate the issues can be closed. 

Such metrics may indicate a quality issue somewhere in the development 

process. However, the use of the metrics allows the program to identify 

such concerns and take steps to address them.

Progress

In written responses, the Quality Assurance Team stated that it calculates 

progress and work completed. Specifically, the team calculates a burnup 

chart, contained within a cumulative flow diagram, that measures 

progress against the program’s schedule goals. The program also tracks 

release burndown rates and other progress metrics. Such metrics are 

presented during oversight meetings, such as those before the Executive 

Steering Committee.

The release burndown charts and cumulative flow diagrams do not 

present a consistent picture of program progress. In particular, there are 

discrepancies between dates the program is working towards, as reported 

in the burnup or burndown charts, and the scheduled commitment dates 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 89 GAO-16-467 Immigration Benefits System

for a release. The program is also tracking progress by user story 

completion without accounting for the relative weight (i.e. complexity) of 

each user story. As such, burndown charts and cumulative flow diagrams 

do not provide an accurate picture of the program’s progress over time. 

For example, the program reported to the Executive Steering Committee 

in October 2015 that they will complete 157 user stories to complete 

release 7.1 by February 2, 2016. However, without a perspective on the 

complexity of the user stories (i.e. story points), the number 157 lacks 

meaningful context.

The Chief of the Applied Technology Division of OIT confirmed that 

progress metrics do not account for the relative complexity of the work 

performed or of the work remaining. This official explained that it is a 

common practice to not incorporate relative complexity as, over time, 

there are enough stories of relative complexity that progress can be 

projected without accounting for complexity. OTC officials added that the 

completion date projected in progress metrics deviate from the dates 

committed to during release planning because a release plan is only an 

estimate and can change depending on business priorities. Officials 

added that the system can theoretically be deployed at any time; 

however, OTC might sometimes determine that additional functionality 

should be incorporated before the release is made available to the public.

However, even if the complexity of all user stories for release 7.1 is 

consistent, presented timelines still do not reflect realistic end dates 

based on past trends. For example, the burndown chart for release 7.1, 

presented to the Executive Steering Committee in October 2015, reported 

program completion of 2 user stories in the first 2-week period, 11 user 

stories in the next 3-week period, 7 user stories in the next 1-week period, 

and 14 user stories in the next 4-week period. At this rate, the program 

would complete 34 user stories every 10 weeks. Accordingly, the program 

should expect to complete an additional 63 user stories by February 2, 

2016, leaving 94 user stories undeveloped at deployment. However, the 

chart presented to the committee shows that all user stories will be 

completed by the scheduled goal of February 2, 2016, thus delivering all 

planned functionality and features.

Without reliable metrics by which to track progress over time, the program 

may present to stakeholders an unrealistic picture of the value they 

expect to deliver by a particular date.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 90 GAO-16-467 Immigration Benefits System

Velocity

USCIS policy, guidance, and underlying team process agreements all 

recognize velocity as a valuable metric for measuring progress and 

identifying areas for improvement. Velocity is defined as the total number 

of story points that the team can complete in an individual sprint. The goal 

is to have each development team converge on a stable velocity to allow 

a more realistic projection of future work capacity to assist commitments 

made by the team in planning for a sprint.

Our calculations of team velocity over time show no evidence of teams 

reaching a point of stable development output. Based on our calculations 

of team velocity across 29 sprints (some teams were captured across 

fewer sprints due to the nature of when they began work on USCIS ELIS), 

team velocity was erratic from sprint to sprint. For example, based on 

these metrics, the velocity for one development team was 26, 13, 2, 8, 8, 

and 26 across sprints 52 to 57, respectively. Similarly, the velocity for 

another development team was 24, 45, 16, 32, 23, and 23 across the 

same sprints.

In addition, development teams are not always using velocity measures to 

inform their planning commitments. In particular, development teams are 

consistently overcommitting to work they will complete within a given 

sprint. Specifically, based on our analysis, between sprints 35 and 63, the 

12 development teams working on USCIS ELIS collectively 

overcommitted at sprint planning 47% of the time based on the prior 

sprint velocity. The teams also accepted additional work during sprints 

(i.e. accepted additional user stories following the first day of a sprint), 

thus overcommitting 66% of the time by the end of the sprint.

Our findings are reinforced by the finding of the Quality Assurance Team. 

In consolidated release assessments, the Quality Assurance Team 

reported that, between sprints 47 and 62, all planned work items were not 

delivered by development teams in 68 percent of the sprints (47 of 69 

responses).

The Chief of OTC stated that velocity has routinely been in a state of flux 

from year to year but that such fluctuations are expected because the 

development teams are relatively new. The Chief explained that 

development teams operating under the old contract achieved a stable 

velocity but, in transitioning to the new contract, velocity again became 

unstable. Nevertheless, a team should be able to begin achieving a more 

stable velocity within a few sprints, and development teams had sufficient 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 91 GAO-16-467 Immigration Benefits System

time to achieve a stable velocity by the time the preceding measurements 

were taken.

Erratic team velocity means that the program will have difficulty 

accurately anticipating how much work development teams can complete 

in each sprint, thus limiting the program’s ability to accurately anticipate 

the amount of work that can realistically be completed in a given 

development period.

Program schedule goals

One of the metrics the program agreed to report to DHS on a monthly 

basis was an integrated master schedule to monitor program status and 

evaluate execution of the schedule.

The program has established a general programmatic goal for schedule 

that is reflected in the approved acquisition program baseline and 

underlying life cycle cost estimate. Furthermore, USCIS Agile software 

development policy requires deployment of functionality, as approved in a 

release readiness review, at least quarterly with the expectation that each 

release should not exceed six months.

Based on our review of the development period for each major (e.g. 

product line) release (the period between release planning and release 

readiness review approval), releases have exceeded the projected 

schedule defined in the life cycle cost estimate. Furthermore, there were 

no releases completed within a six-month period.28 Table 8 reflects the 

planned versus actual start and end dates for each major release along 

with the total period of development.

 28In written comments, the USCIS CIO stated that the program is meeting the goal of 

delivering software every 6 months. He stated that the program deploys minor releases, 

such as enhancements, or fixes, to existing functionality, on a weekly basis. As such, in 

his estimation, the program is meeting the schedule goal articulated in USCIS Agile 

development policy. The USCIS Agile Development Policy does not differentiate between 

a major and minor release. 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 92 GAO-16-467 Immigration Benefits System

Table 8: Planned versus Actual Release Cycles for Major Program Releases, as of April 2016

Release 

number

Planned 

release start 

date

Planned 

release end 

date

Release 

planning review 

approval date

Release readiness 

review approval 

date

Total development 

period

Schedule delay 

(approximation)

5.0 5/29/13 2/21/15 5/28/2013 2/20/2015a 21 months 0 months

5.1 2/27/13 4/25/15 11/5/13 8/27/2015 21 months 4 months

6.1 3/4/15 8/15/15 4/7/15 2/1/16 b 10 months 5 months

6.2 3/4/15 8/15/15 4/7/15 Not yet achieved 12+ months 7+ months

7.1 6/24/15 12/5/15 7/30/15 4/13/16 9 months 4 months

Source: GAO analysis of USCIS documentation. I GAO-16-467.

a

I-90 functionality was made available in November 2014 in a limited 72-hour release (release 5.0.0). 

Full deployment of this functionality did not occur until February 2015 (release 5.0.1). Lockbox 

functionality was deployed for processing the I-90 after February 2015 (release 5.0.2).

b

Deferred Action for Childhood Arrivals functionality was made available in October 2015 but was 

limited to only one Lockbox facility for the processing of applications submitted on one day. According 

to the program, the Phoenix Lockbox facility began receiving cases on a regular basis beginning in 

January 2015. All other Lockbox facilities began receiving applications in February 2015.

The program is reporting updated schedule dates as part of the monthly 

performance assessment. These dates reflect changes in the program 

schedule and the impact of schedule delays. Based on the reported 

schedule, the program appears to project only a 3-month delay from the 

initial schedule, which is within the program schedule threshold of March 

2019. However, as we have seen with releases to date, the schedule 

figures are not always presenting the full picture. For example, the 

schedule indicates an end date of November 2015 for release 6.1; 

however, the actual end date was February 2016. According to the 

program management review from February 2016 and our discussion of 

this date with the program office, Deferred Action for Childhood Arrivals 

cases were not fully being processed until February 1, 2016.

According to officials from OTC, there are a number of reasons for 

deviation from schedule. The Chief of OTC stated that their goal is to 

always deliver code but that they do not always make it available to 

internal users (i.e. go live). The Chief explained that the code base should 

always be ready for deployment. However, actual deployment may take 

longer based on the needs of the business. In addition, the deployment 

date flexes based on discussions with leadership. For example, the 

estimates for release 7.1 deployment were a best guess as of 2015. 

However, additional information regarding the release came to light 

following release planning and, consequently, the deployment date 

shifted.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 93 GAO-16-467 Immigration Benefits System

These delays increase the risk that the program will proceed with 

development or deployment in the future earlier than it might be ready in 

order to meet the committed dates. The program is also at risk of future 

schedule delays in subsequent releases, which might result in the 

program exceeding its established threshold.

Program cost goals

In addition to monitoring program schedule, the program agreed to report 

on obligations and expenditures monthly and use burn down analysis to 

monitor program costs against the life cycle cost estimate for the fiscal 

year.

The program established a cost goal that is defined in the approved 

acquisition program baseline and is based on an underlying life cycle cost 

estimate. According to this estimate, the program has established a cost 

objective of approximately $2.9 billion and a cost threshold of 

approximately $3.1 billion. The largest cost in this estimate is associated 

with release development, with a cost projection of approximately $187.6 

million. The program set an assumption that 12 development teams 

would work on each release. It then projected the cost of each 

development team against the period of development to arrive at the 

projected cost of development for that release. These projected costs 

help the USCIS CIO to determine, during release planning, whether the 

cost is justified by the capabilities to be delivered.

As noted in the previous table on schedule, the program is not meeting 

the projected start and end dates reflected in the cost estimate. For 

example, the cost estimate projected development costs for 6.0, 

subsequently broken out by the program into releases 6.1 and 6.2, 

through June 2015. However, the program did not fully deploy release 6.1 

until February 2016 and did not project to complete development of 

release 6.2 until May 2016. As a result, development of future releases 

was delayed. For example, the program just completed planning for 

release 8.0 in February 2016 although it was estimated for November 

2015. The program has also not yet finalized planning for release 9.0, 

which was scheduled to take place by February 2016.

Moreover, the program has not accounted for some additional contracts 

awarded since the cost estimate was updated. For example, the 

approved acquisition plan for the program does not include a contract for 

Agile engineering and test automation services, which carries a cost of 

over $12 million.
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 94 GAO-16-467 Immigration Benefits System

Although we found the life cycle cost estimate to be generally reliable, 

and the program is clearly reporting against this estimate, the lack of an 

update to the estimate means that performance may be tracked against a 

baseline that is no longer relevant.

Program scope

In addition to monitoring program schedule and cost, the program agreed 

to report on the percentage of sub-features completed versus the 

percentage of sub-features planned, broken down by major release.

The program reports the scope of each release in the form of subfeatures to be delivered within each release. In reporting to management, 

the program identifies the planned number of sub-features to be 

developed in each release and updates this number to reflect the actual 

number of sub-features developed. For example, in October 2015, the 

program explicitly identified the sub-features developed for release 6.1. 

Table 9 lists the most recent number of sub-features planned for nearterm releases relative to the number actually delivered.

Table 9: Transformation Program Sub-Features Delivered by Near-Term Releases, 

as reported to DHS in February 2016 

Release 

number

Planned number 

of sub-features

Actual number of 

sub-features

Cumulative number of subfeatures implemented

5.0 130 126 126 of 235a

5.1 10 10 136 of 235

6.1 13 9 145 of 235

6.2 1 Not yet achieved Not yet achieved

7.0 28 Not yet reported Not yet reported

8.0 3 Not yet achieved Not yet achieved

Source: GAO analysis of USCIS documentation. I GAO-16-467.

a

The Transformation Program will not achieve full operating capability until release 16 has been 

completed. On achieving full operating capability, the program plans to deliver a total of 235 subfeatures that, collectively, make up the full functionality of the program. This table only represents 

near-term releases and excludes sub-features planned for future releases (i.e. release 15.0). The 

program has increased the number of planned sub-features to 235, which has impacted reporting on 

the number of planned versus actually delivered sub-features.

Based on our review of the backlogs for releases 6.1, 6.2, and 7.1 (the 

program subsequently renamed release 7.0 as release 7.1), the program 

has not fully documented if it is delivering the sub-features it has intended 

to deliver in each release. The most recent backlogs provided to us in 

March 2016 include a field termed “traceability.” This field maps a user 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 95 GAO-16-467 Immigration Benefits System

story back to a supporting sub-feature and/or feature. According to this 

field:

• 6 of the 9 sub-features were not developed or were not clearly 

traceable to the backlog for release 6.1;

• The one sub-feature associated with release 6.2 either was not 

developed or was not clearly traceable to the backlog; and,

• 19 of the 28 sub-features were not developed or were not clearly 

traceable to the backlog for release 7.1.

In a written response, the Business Integration Division within OTC, which 

is responsible for defining the initial release backlog committed to in 

release planning, recognized issues in traceability of user stories to subfeatures. This division stated that the process that was used to verify the 

number of sub-features implemented against planned was based on 

verbal confirmation from the product owner. The division subsequently 

determined that this process was not effective since it relied solely on the 

review of the user stories and was not as exact and reliable as expected. 

As a result, the division stated that there could be sub-features that were 

reported as implemented by the product owner but that would not show 

any associated user stories because they were not directly mapped to the 

sub-feature in the software management tool.

To address this challenge, the Business Integration Division reported 

working in collaboration with the Capability Delivery Division of OTC to 

develop a process to manage and track requirements of USCIS ELIS at 

sub-feature level throughout the development life cycle. All requirements 

for each new release are now to be reviewed and concurred with by the 

appropriate product owner at the sub-feature level. This concurrence 

should take place after collaborating with the end users and subject 

matter experts. The requirements are then to be converted into user 

stories and entered into the new software tool, where their traceability to 

the approved set of sub-features is established by following the 

predefined hierarchy and tracked throughout the development process. 

The division noted that requirements traceability is critical to avoid scope 

creep and to demonstrate that the user stories implemented addressed 

the mission needs. However, as noted, the most recent release backlogs 

provided to us, which include this traceability between user story and subfeature, still do not reconcile the gaps in reporting.

Of additional concern is the ongoing development of sub-features in 

subsequent releases. The program road map and monthly performance 
Appendix II: Management of Agile Software 

Development for USCIS ELIS

Page 96 GAO-16-467 Immigration Benefits System

reports imply that sub-features are fully developed within each release, 

but many sub-features require additional work in subsequent releases. 

For example, sub-feature 2.4.1.1 ‘Provide Case Summary Information for 

Review’ was developed as part of release 5.0.1, the soft launch of the I90. Program projections show development work of this same sub-feature 

for all subsequent releases, 5.1 through 16. For example, the backlog for 

release 7.1 indicates that 37 user stories were developed in support of 

this already existing sub-feature.

The lack of traceability between scope metrics reported by the program 

and the release backlogs indicates a level of unreliability in reporting on 

scope. Moreover, the continual need for additional effort after delivery of a 

sub-feature raises additional concerns regarding the extent to which the 

program has effectively forecasted future work in its cost and schedule 

projections.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 97 GAO-16-467 Immigration Benefits System

This appendix describes in more detail our evaluation of the three key 

practices for systems integration and testing. It does not present new 

findings. Rather, the information detailed in this appendix is intended to 

assist USCIS in implementing the recommendations described in our 

report.

Program planning documentation1 calls for establishing an environment 

for continuous integration2 and testing3 that includes the following tools 

and practices.

• Configuration management: Identifying and documenting 

characteristics of the software code at specific points in time.

• Daily code integration: Integrating developer’s code with the main 

code base daily.

• Daily automated code builds: Daily compiling and testing of code that 

is to lead to an operational version of a system.

• Automated tests: Providing timely feedback on the development 

process through the use of small automated tests to establish a solid 

foundation for subsequent testing.

• Failing tests stop the build: Stopping the progression of code through 

the development pipeline if newly developed code is defective, as 

indicated by failing tests at any stage of the development process.

• Rapid feedback: Providing rapid feedback to the development teams 

to allow them to resolve defects prior to completing a given 

development iteration.

USCIS has established an environment for continuous integration and 

testing for USCIS ELIS. For example, the program has created a process 

 1

See, for example, USCIS ELIS Team Process Agreements, Transformation Program 

Test and Evaluation Master Plan, and Transformation Program Test Plans. 

2

Continuous integration is a practice of frequently merging an individual piece of software 

code with the main code repository, so new changes are tested continuously. 

3

Continuous testing is a practice of testing software code throughout the sprint and 

providing feedback to the developers so that they can address issues and develop 

software that meets all required quality criteria. Using this approach, code is tested in 

various environments, from the development environment to a production-like 

environment.

Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Establishing an environment 

and procedures for continuous 

integration and testing of code
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 98 GAO-16-467 Immigration Benefits System

for the automated build of software code and automated testing of that 

code during the build process. Table 10 describes in more detail the 

continuous integration and testing tools and practices in support of USCIS 

ELIS.

Table 10: Summary of Management of Continuous Integration and Testing Tools and Practices

Practice or Tool Rating Description

Configuration management ● USCIS ELIS code is placed under configuration management.

Daily code integration ● USCIS established conditions that allow for daily code integration.

Automated code builds ● Continuous integration server and other tools allow for automated code builds.

Automated tests

●

USCIS established conditions for automated tests to be written and executed by 

developers.

Code build stops with test failure ● USCIS implemented a process that can halt progress of defective code.

Rapid feedback 

●

Feedback from the automated tools can be provided to developers and program 

management instantaneously.

Source: GAO analysis of USCIS documentation I GAO-16-467.

Key: ● yes ◐ partial ○ no

The build process starts with a separate work branch in which code is 

written and tested by a developer on a local machine. When ready, the 

developer submits a request to merge and integrate the code with other 

code produced by the development team. This branch is referred to as 

the team continuous integration branch. If the merge is successful, this 

code is then compiled and must pass through a series of three gates: 

build, staging, and integration. As part of each build, a suite of automated 

tests are performed. If the build is successful, the software governing the 

build process will merge the code into the master code base. According to 

the senior continuous integration technical lead, the build process 

typically takes about an hour. Figure 9 provides a high-level overview of 

the continuous integration process followed as code is developed and 

submitted.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 99 GAO-16-467 Immigration Benefits System

Figure 9: Continuous Integration Build Process for USCIS ELIS

In order to effectively test code, a program should perform continuous 

testing at different stages of development. These tests are to include, 

among other things, unit and integration tests, functional acceptance 

tests, and code inspection.

Performing continuous testing 

through the use of unit and 

integration tests, functional 

acceptance tests, and code 

inspection
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 100 GAO-16-467 Immigration Benefits System

Unit and integration testing

According to the program’s Test and Evaluation Master Plan,4 two tests 

are to occur on new code written by developers: unit and integration tests. 

Unit tests are small, automated tests used to verify that each individual 

unit of code written by the developer works as intended. Integration tests 

are intended to verify that individual modules or components function 

together as expected when combined. USCIS policy states that both unit 

and integration tests are to cover all developed code; that is, reach 100 

percent code coverage. See figure 10 for a high-level overview of how 

unit and integration tests fit in a developer’s process for developing and 

building code.

 4

USCIS Transformation Program, Test and Evaluation Master Plan, Electronic 

Immigration System, version 1.0 (Mar. 10, 2015). The Test and Evaluation Master Plan 

outlines the overall test and evaluation strategy for the USCIS Electronic Immigration 

System. This document was updated and approved as part of the program re-baseline in 

2015.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 101 GAO-16-467 Immigration Benefits System

Figure 10: Unit and Integration Testing Process

The program’s team process agreements for releases 6.1 and 6.2 state 

that unit and integration testing is also a requirement for accepting 

individual pieces of software as complete. In addition, according to our 

standards for internal control,5 managers are to

• promptly evaluate findings from audits and other reviews, including 

those showing deficiencies and recommendations reported by 

auditors and others who evaluate agencies’ operations,

• determine proper actions in response to findings and 

recommendations from audits and reviews, and

 5

GAO, Standards for Internal Control in the Federal Government, GAO/AIMD-00-21.3.1

(Washington, D.C.: Nov. 1999). 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 102 GAO-16-467 Immigration Benefits System

• complete, within established time frames, all actions that correct or 

otherwise resolve the matters brought to management’s attention.

The program has exhibited improvement in unit code coverage since 

2015, but none of the internal applications that collectively make up 

USCIS ELIS have achieved the stated goal of 100 percent code 

coverage. For example, based on system-generated output of unit code 

coverage provided by the program, the largest application that makes up 

USCIS ELIS, ELIS2Services, achieved 49.8 percent unit code coverage 

as of August 2015. In March 2016, unit test code coverage of the same 

application improved to 64.2 percent. However, while 64.2 percent is an 

improvement, it still does not meet the original goal of the program. The 

Quality Assurance Team continued to report on this issue in April, May, 

June, and November 2015, citing 17 new tickets opened for having no or 

poor unit test coverage.

In addition, the program did not provide evidence of integration code 

coverage. The Quality Assurance Team reported in May 2015 that 

integration testing, among other tests, is not occurring with any degree of 

regularity. However, the team added that the testers do not publish their 

test results, so it is not possible to know whether testing is taking place or 

whether anything is being done with the results to improve the process.

As integration tests are not documented, the program was unable to 

provide evidence of the extent of the code covered by integration tests.

Unit and integration testing does not meet USCIS standards due to a lack 

of internal controls over ensuring standards are met. The product owner 

for releases 6.1 and 6.2 stated that they assume that unit and integration 

testing are complete and that tests meet USCIS standards prior to 

demonstrating a user story for acceptance. The product owner requires a 

test summary report for each story that demonstrates acceptance criteria 

are properly met, but does not require validation that other testing 

required by the developer took place. Moreover, the ELIS Test Plan, 

updated in July 2015, refers to integration tests as useful, but does not 

explicitly require them to be written. This is in contrast to the prior version 

of the plan covering development period from October 15, 2014, to April 

15, 2015, which stated that integration tests should be written before the 

code is written. The test plan also conflicts with the commitment made 

with DHS for integration testing as articulated in the Test and Evaluation 

Master Plan.

To improve on the coverage of unit tests, the Senior Continuous 

Integration Technical Lead stated that any new build must meet the 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 103 GAO-16-467 Immigration Benefits System

existing threshold for code coverage. For example, if the current unit test 

code coverage for the ELIS2Services application is 63.7 percent, then 

any new code submitted for this application must contain at least 63.7 

percent unit test coverage. This official explained that using the latest 

code coverage value of a passing build was always a rule for establishing 

threshold value for unit test coverage and helps to ensure a consistent 

increase in overall coverage over time.

In the documentation received from the program on March 17, 2016, the 

historical progression of code coverage indicates that the coverage trend 

is not consistently increasing as suggested by the technical lead. For 

example, coverage for InternalApp was 53.5 percent on January 13, 

2016, but it decreased to 52.3 percent on March 15, 2016. Similarly, 

InternalAppDomain decreased by 6.7 percent in the same period. In total, 

agency reports show that internal application code coverage ranged from 

31.2 percent to 96.9 percent. According to the USCIS CIO, the goal of 

100 percent code coverage is unrealistic. We agree with this opinion; 

however, this is the goal stated and committed to in the program Test and 

Evaluation Master Plan.

The Quality Assurance Team has cited inadequate testing as the primary 

cause for issues encountered in production. More concerning is that 

issues in testing and the quality of software code were issues previously 

raised in development of legacy USCIS ELIS.

Functional acceptance testing

The Transformation Program uses functional acceptance tests to verify 

that all user story acceptance criteria have been met. The Test and 

Evaluation Master Plan for the program states that functional acceptance 

test scenarios should be executed as capabilities are added or updated. 

The program also documented procedures for performing these tests.

The program did not provide evidence that functional acceptance tests 

were being performed for all user stories as required. In November 2015, 

we requested functional acceptance test scripts and results for a random 

sample of 80 user stories associated with USCIS ELIS releases 6.1 and 

6.2. Of these 80 user stories, only 74 were considered complete and 

therefore would have undergone functional acceptance testing. In 

response, the program provided evidence indicating that functional 

acceptance tests had occurred, and passed, for 11 of these 74 user 

stories. Based on the descriptions of the remaining 63 user stories, we 

expect that a subset of these would also require functional acceptance 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 104 GAO-16-467 Immigration Benefits System

tests. However, the program did not provide evidence that the testing had 

been performed.

Program officials attributed the lack of evidence for functional acceptance 

testing to its transition to a new software tool. Nevertheless, our review of 

testing documentation shows that this transition does not fully account for 

the missing test documentation. Specifically, according to officials in the 

program office, the program transitioned to a new software tool for user 

story management in September 2015. However, not all the information 

was properly migrated from the old system, including some attachments 

containing information on user stories that was evidence to support 

whether tests had been performed. Based on our observations, this would 

have affected 64 of the 80 user stories that we examined. Of the 16 that 

were not migrated, and instead were defined using the new software tool, 

6 did not include evidence of acceptance tests having been performed. 

The program provided subsequent evidence that 2 of these 6 user stories 

were ‘resolved’ into 8 other user stories for which the program provided 

no additional testing information. For another user story, the program 

provided an extract that indicated existence of a testing document but the 

program did not provide the actual test document. The program office 

was unable to explain the lack of acceptance tests for the other user 

stories.

Further, monthly status reports from the Quality Assurance Team support 

our findings and raise questions about the extent to which functional 

acceptance testing is occurring. For example, in June 2015, the Quality 

Assurance Team noted that functional testing is not occurring with any 

degree of regularity. Quality assurance team reports have not provided 

updates on functional acceptance testing since June 2015.

Without assurance that acceptance tests are being performed for each 

user story prior to acceptance by the product owner, USCIS cannot 

ensure that the user stories meet the needs of USCIS as defined in 

associated acceptance criteria.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 105 GAO-16-467 Immigration Benefits System

Code Inspection

Code inspection is the process of inspecting code automatically and 

manually6 and comparing it to internal coding guidelines. The team 

process agreements for releases 6.1 and 6.2 calls for all software code to 

undergo inspection before it is accepted as complete. According to 

industry leading practices,7 automatic and/or manual inspection should be 

performed to provide additional assurance of software quality.

In addition, the program has established a mechanism for automated 

code inspection and, in January 2016, the program provided 

documentation indicating its current results. Moreover, the tool that the 

program uses to conduct automated code inspection can notify 

developers when code does not meet coding standards.

The program has also implemented two types of manual code review. 

According to program officials, the Quality Assurance Team performs 

focused reviews of high risk code in order to focus on critical aspects of 

the software. Officials explained that the team is not large enough to 

review the entire code base and must instead focus their efforts on the 

areas that will be of the greatest impact. In addition to a review by the 

Quality Assurance Team, in April 2015, the program instituted a peer 

review process for manual review of all code.

The program provided evidence that Quality Assurance Team reviews are 

taking place. In March 2016, the program provided data on a review of 

code submitted in merge requests, along with the number of tickets 

opened based on that review, for the period from September 2015 

through March 2016. Based on this spreadsheet, 577 tickets were 

opened to address issues found during code review by the Quality 

Assurance Team. However, the program did not provide evidence that 

the peer review process is being performed and documented consistently. 

According to internal guidance available to developers, a request to 

merge code with other developers is submitted as part of continuous 

 6

Continuous testing is generally considered to be automated testing. However, this report 

addresses code inspection under the broader heading of continuous testing, even though 

code inspection includes both automated and manual inspection. 

7

IEEE, IEEE Standard 29119-1. 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 106 GAO-16-467 Immigration Benefits System

integration. Before this request is approved, a technical assignee should 

perform the peer code review and document acceptance in the software 

tool. Based on a walk-through of the software tool in which the peer 

review process is documented, we found that peer review took place for 

40 out of 53 of the 80 user stories we reviewed for which it was 

applicable, and did not take place for the remaining 13 of these user 

stories.

Some stories were excluded from these calculations because:

• Development and merging of the code for those user stories took 

place prior to June 2015, when the peer review process was initiated.

• A merge request was made but was closed rather than accepting the 

merge (we did not pursue the rationale for closure).

• The user story did not result in any code being developed and pushed 

to master (i.e. more an administrative task than actual functionality 

being built).

• The user story was still “defined,” meaning a team had not yet started 

developing it, and not “accepted,” meaning the product owner did not 

approve the story upon completion of development.

In addition, of the 13 user stories for which we could not confirm 

documented acceptance, some user stories may have been submitted as 

part of another merge request (i.e. user story 1234 was coupled with user 

story 1235 and submitted as user story 1235 merge request). However, 

we could not determine in which instances this was a cause for a gap in 

peer review.

Until the program establishes a mechanism to validate that the peer 

review process takes place, the program cannot be assured that user 

stories are adhering to program coding standards, which may result in 

poor code quality.

Interoperability testing

Interoperability testing is an independent test in a production-like 

environment that is intended to verify the capability of system 

components and external work flows, communication, and information 

Developing complete test plans 

and cases for interoperability 

and end user testing and 

documenting those results
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 107 GAO-16-467 Immigration Benefits System

sharing in a controlled condition. The program’s Test and Evaluation 

Master Plan requires execution of interoperability tests following any 

change to a system that interfaces with USCIS ELIS. In addition, leading 

practices8 detail the activities required for documenting test planning, 

execution, and results reporting. We evaluated USCIS ELIS 

interoperability and end user tests against these practices.

USCIS demonstrated that it has developed test plans, cases, and 

documented results for compatibility testing. The lead tester from the 

Interoperability Test Team stated that his team may test the parameters 

of a public facing system such as end-user parameters of different 

browsers and operating systems using a non-DHS network connection. 

This official added that the Interoperability Test Team is concerned with 

whether the system meets certain technical parameters related to

operating systems and different types of browsers interacting with USCIS 

ELIS.

However, compatibility tests do not meet the intent of interoperability 

testing as defined by the Test and Evaluation Master Plan. The test plans 

and associated cases and results provided by the program document the 

ability of USCIS ELIS to execute using specific browsers and operating 

systems that mirror the USCIS environment. This type of testing ensures 

that a system is compatible with existing software and will work within a

physical operating environment. This type of testing does not test the 

ability of USCIS ELIS to communicate and share information with other 

systems, either within or outside USCIS. Instead, program officials noted 

that interoperability, as defined in the program’s Test and Evaluation 

Master Plan, is to be tested through live interface testing.

USCIS has not developed tests plans or cases for live interface testing. 

The USCIS Test and Evaluation Master Plan states that live interface 

testing is to be performed with all internal USCIS and DHS systems 

before new functionality is promoted to the production environment. 

Program officials stated that there is no live interface test plan or 

associated test cases. Instead, the program documents test charters, 

which include a mission statement and conditions for successful 

operation of all product lines. However, the test charters provided by the 

 8

IEEE, IEEE Standard 29119–3. 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 108 GAO-16-467 Immigration Benefits System

program do not meet the requirements for a test plan or test cases. In 

particular, they do not identify the test items, scope, risks, or other 

elements of a test plan.

The program has documented some results for live interface testing. 

USCIS officials stated that test results are documented in associated 

tester notes rather than in test completion summary reports. Although 

testing documentation can be tailored on Agile projects, live interface 

session testing results are notes from the testing session and are not 

projectable to the entire scope of releases 6.1 and 6.2. In particular, those 

notes cover eight sessions from October 13 to December 28, 2015, and 

these notes do not meet the requirements for documenting test results for 

interoperability testing, such as details on what was tested and whether 

the test met the specified completion criteria.

According to the Chief of the Delivery Assurance Branch, development 

teams do not produce a detailed test plan and report because unit tests 

should be documented in source code and most of the functional tests 

should be documented in scripts. According to this official, results from 

these tests should be documented in the continuous integration build tool 

used by the program and all tests should tie to a user story. Accordingly, 

a program manager for the Independent Test and Integration division 

stated that documentation associated with a test plan, case, or results is 

addressed in development of individual user stories. For example, if a 

user story requires interfacing with the Lockbox, then acceptance criteria 

for that story would define the parameters or preconditions for testing the 

user story with the Lockbox. However, independent testing of user stories 

does not address the need for a plan, cases, or results that reflect the 

ability of USCIS ELIS to interface with other systems internal and external 

to USCIS.

In not documenting that testing occurs and the results of this testing, the 

program may be producing a system that cannot effectively exchange 

important information and use the information being exchanged in 

processing immigration applications. This is particularly important due to 

the number of systems with which USCIS communicates. See appendix 

IV for a list of systems with which USCIS ELIS must communicate.

Without a test plan and documented cases and results demonstrating 

execution of that plan for interoperability testing, USCIS cannot ensure 

that the performance between USCIS ELIS and other systems on which it 

depends is consistent and successful.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 109 GAO-16-467 Immigration Benefits System

End user testing

End user testing is conducted to verify that business value is delivered 

and user story acceptance criteria are met. The USCIS ELIS End User 

Test Strategic Plan describes end user testing as involving USCIS ELIS 

users to test functionalities and usability using real-world scenarios prior 

to implementing a release into production. Leading practices9 require that 

for tests such as end user tests, the program should establish test plans, 

test cases, and test completion reports that address certain practices. 

Specifically,

• Test plans should be uniquely identified, document test items, include 

the scope, strategy, activities, and schedule for testing, and discuss 

risks surrounding the test.

• Test cases should also be uniquely identified, trace back to 

associated requirements, describe the conditions around which the 

test occurs (i.e. environment), and define test inputs and expected 

results.

• Test completion reports should be uniquely identified, summarize the 

testing performed and any deviation from planned testing, evaluate 

the extent to which test criteria were met or not, identify factors that 

blocked progress, present any collected measures, residual risks, and 

lessons learned, and document any test deliverables produced in 

support of the testing effort.

The program did not provide evidence of having developed test plans, 

cases, or test completion reports between March 2015 and June 2015.

After this period of development, the program provided an end user test 

plan that met most of the leading practices. In June 2015, the program 

transitioned responsibility for end user testing to a new contractor. The 

plan for end user testing provided by this new contractor defined the test 

items, scope, schedule, and the general type of test performed. The only 

element that the plan did not address is identifying test completion criteria 

as part of the larger test strategy. Table 11 provides additional details 

about the extent to which the USCIS ELIS End User Test Strategic Plan

meets leading practices.

 9

IEEE, IEEE Standard 29119–3. 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 110 GAO-16-467 Immigration Benefits System

Table 11: USCIS ELIS End User Test Plan Evaluation

Criteria element Rating Description

The plan identifies project or test type for which the plan is being 

written.

Yes The plan identifies the project and test type.

The plan includes a unique identifier Yes The document can be uniquely identified.

Test items identify the objects of testing, for example a system or 

a software item.

Yes The plan identifies the object of testing.

Test scope summarizes the features of the test items to be tested 

and identifies any features that are excluded from the test and the 

reason for such exclusion.

Yes The document indicates that end user testing sessions 

will be focused on new product functionality.

Risk register identifies test-related project and product risk as well 

as risk mitigation approaches.

Yes The document includes test-related risks.

Test strategy:

Identifies the particular test to be performed (such as 

integration, interoperability)

Yes The document identifies the tests to be performed.

Identifies documentation to be produced at the conclusion of 

the test

Yes The document identifies documentation to be produced 

at the conclusion of the test. 

Specifies which test design techniques are to be applied Yes The document indicates that a scenario testing approach 

is used as a test design technique

Identifies test completion criteria No The plan does not identify the test completion criteria. 

Identifies what metrics should be collected during the test 

activities

Yes The document identified metrics that should be collected.

Specifies all relevant test data requirements N/A In the written responses, the program indicated that data 

for test scenarios is based on acceptance criteria. Since 

data requirements may vary between user stories, the 

test plan does not specify the relevant data 

requirements. 

Specifies the necessary and desired properties of the test 

environment

Yes The plan specified that end user testing occurs in end 

user testing environment and the plan includes a 

description of the test environment.

Testing activities are based on the test process to be used and 

include an activity iteration strategy for test re-execution as well 

as any dependencies

Yes Testing activities are documented in the plan. Test 

activities show the dependencies and indicate the 

session in which the End User Test Team will 

troubleshoot tester problems. 

Testing schedule identifies test milestones defined in the project 

schedule and summarizes the overall schedule of the testing 

activities.

Yes The plan states that end user testing sessions will occur 

on a weekly basis.

Source: GAO analysis of USCIS ELIS High Level End User Test Plan I GAO-16-467.

The program also provided test cases that addressed most of the leading 

practices. Specifically, we reviewed all test cases used in sessions from 

June 2015 to September 2015. These end user test cases specified 

expected test results and associated inputs. However, some of the cases 

did not describe preconditions for the test. Table 12 provides additional 
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 111 GAO-16-467 Immigration Benefits System

details about the extent to which end user test cases met leading 

practices.

Table 12: USCIS ELIS End User Test Case Evaluation

Criteria Element Rating Description

Unique Identifier to distinguish the test case from all other test 

cases. The unique identifier is not to be changed during the 

lifetime of the test case item, because it is used for traceability 

purposes.

Yes The 19 test cases were uniquely identified. 

Traceability links the test case to the test coverage item or 

provides reference to the associated requirements.

Yes The program documented traceability between user 

stories and end user test sessions.

Preconditions describe the required state of the testing 

environment and any special conditions related to the execution of 

the test case.

Partial Three test cases do not include preconditions.

Inputs specify each action required to bring the test item into a 

state where the expected result can be compared to the actual 

results. The detail of the description should be tailored to fit the 

knowledge of the testers. All required relationships between input 

events must be described.

Yes All test cases specified actions required to bring the test 

item into a state where expected results can be 

compared with the actual results.

Expected results specify the expected output and behavior 

required of the test item in response to the inputs that are given to 

the test item when it is in its precondition state. 

Yes With the exception of exploratory cases, test cases 

specified expected output and behavior of the system 

under test.

Source: GAO analysis of USCIS ELIS end user test cases I GAO-16-467.

The program also provided test summary results from June 2015 to 

January 2016 that met some of the leading practices. For example, test 

completion reports summarized the testing performed, test measures, 

and highlighted some lessons learned. However, these reports did not 

identify residual risks, report all of the factors that blocked progress, or 

define deliverables produced as a result of testing. Table 13 provides 

additional details about the extent to which end user test completion 

reports met leading practices.

Table 13: USCIS ELIS End User Test Summary Results Evaluation

Criteria element Rating Description

Documents are uniquely identified. Yes All documents are uniquely identified. 

Summary of testing performed provides details on what was 

tested and any testing constraints.

Yes All test completion reports contain summary of testing 

performed.
Appendix III: Management of Systems 

Integration and Testing for USCIS ELIS

Page 112 GAO-16-467 Immigration Benefits System

Criteria element Rating Description

Description of deviations from the planned testing, if any. Partial Test completion reports do not include information on 

deviations from the planned testing or indicate if such 

deviations existed. However, even though the test 

completion report does not include information on the 

deviation of the planned testing, a vehicle exists to 

capture and address such deviations. This is a tailored 

solution which is consistent with the Agile approach and 

uses existing Agile project management tools. 

Test completion evaluation includes information to what extent 

testing met the specified test completion criteria and explains 

why the criteria were not met where appropriate.

No None of the test completion reports include evaluations 

of test completion.

Identifies factors that blocked progress and corresponding 

solutions used to remove them.

Partial Test completion reports did not include information on 

factors that blocked progress. However, USCIS provided 

a record of six impediments between June 22, 2015, and 

August 18, 2015, and six tickets created between June 

23, 2015, and September 13, 2015, that documented 

issues encountered during testing.

Test measures present the collected test measures that could 

include test cases, defects, incidents, test coverage.

Yes All test completion reports included test measures.

Residual risks are the risks that are untreated at the end of the 

testing. This may be risks that have not been fully treated by the 

test or new risks.

No Test completion reports do not discuss risks. 

Test deliverables are produced as a result of the testing effort

and their location.

No Test completion reports do not include information on 

test deliverables and their location. 

Lessons learned Partial Three out of 14 test completion reports did not contain 

lessons learned. 

Source: GAO analysis of USCIS ELIS End User Test Summary Reports I GAO-16-467.
Appendix IV: USCIS ELIS Interfaces

Page 113 GAO-16-467 Immigration Benefits System

The goal of the Transformation Program is to link developed systems, 

enterprise services, and existing systems and capabilities to enable endto-end processing. The central project in this portfolio is the USCIS 

Electronic Immigration System (USCIS ELIS), which is to enable 

electronic filing and adjudicative case management as a component of a 

larger architecture consisting of multiple systems, services, and 

interfaces. In particular, according to officials, USCIS ELIS is to interface 

with existing systems, some of which are to be decommissioned as the 

program is fully deployed. Table 14 describes the features of systems and 

services with which USCIS ELIS is planned to interface.

Table 14: Interfaces Planned for USCIS ELIS 

Agency name System name System description

U.S. Customs and 

Border Protection

TECS (not an acronym) Automated enforcement and inspection lookout system maintained by U.S. Customs 

and Border Protection that combines information from multiple agencies, databases, 

and system interfaces to compile data relating to national security risks, public 

safety issues, current or past targets of investigations, and other law enforcement 

concerns.

U.S. Customs and 

Border Protection

Arrival and Departure 

Information System

System for storage and use of biographic, biometric indicators, and encounter data 

on non-U.S. citizens and nationals who have applied for entry, entered, or departed 

the United States. It facilitates the investigation of subjects of interest who may have 

violated their immigration status by remaining in the United States beyond the date 

of their authorized stay.

Department of 

Homeland Security

Automated Biometric 

Identification System

Originally developed in 1994 as a law enforcement system for collecting and 

processing biometrics, this system has evolved over the years into the central DHSwide system for the storage and processing of biometric data. It stores and 

processes digital fingerprints, photographs, iris scans, and facial images and links 

the biometrics with biographic information to establish and verify identity.

U.S. Immigration 

and Customs 

Enforcement

Enforcement Integrated 

Database

Database shared among several DHS law enforcement entities and with other DHS 

applications. It captures and maintains information related to the investigation, 

arrest, booking, detention, and removal of persons encountered during immigration 

and criminal law enforcement investigations and operations conducted by 

Immigration and Customs Enforcement and Customs and Border Protection.

U.S. Immigration 

and Customs 

Enforcement

Student Exchange Visitor 

Information System

System maintains real-time information on nonimmigrant students, exchange 

visitors, and their dependents.

Department of Labor Immigration Certification 

Portal

System was developed for the Department of Labor, Employment and Training 

Administration, Office of Foreign Labor Certification to provide employers and their 

representatives (i.e., attorneys/agents) a secure and reliable means to access case 

information across all of the supported visa-type programs (H-1B) and to file 

prevailing wage requests.

Department of 

Justice

Executive Office 

Immigration Review

System contains case-related information pertaining to foreign nationals and alleged 

foreign nationals brought into the immigration hearing process, including certain 

foreign nationals previously or subsequently admitted for lawful permanent 

residence.

Appendix IV: USCIS ELIS Interfaces
Appendix IV: USCIS ELIS Interfaces

Page 114 GAO-16-467 Immigration Benefits System

Agency name System name System description

Federal Bureau of 

Investigation

Name Check Service provides a name-based search of the FBI’s Central Records System and 

Universal Index. USCIS sends applicant information (name, date of birth, country of 

birth, race, and gender) to the FBI in order to conduct the name check.

Federal Bureau of 

Investigation

Next Generation 

Identification

System offers biometric identification services as a repository of data on persons of 

special interest such as wanted persons and known or suspected terrorists.

Department of State Consular Consolidated 

Database

Database holds current and archived data used by the Consular Affairs Bureau’s 

domestic offices and overseas posts, such as the names, addresses, birthdates, 

biometric data, race, identification number (e.g. Social Security or alien registration 

number), and country of origin for U.S. persons (U.S. citizens and lawful permanent 

residents), as well as foreign nationals such as immigrant and nonimmigrant visa 

applicants.

Department of the 

Treasury

Pay.gov Portal sends electronic payments and receipt of payment clearance verification. 

Services involve collection, billing, forms, and reporting.

Department of the 

Treasury

Collections Information 

Repository

Repository for supplying the latest information on deposits and details of collections 

transactions to federal agencies. System allows financial transaction information 

from all collections systems and settlement mechanisms to be exchanged in a 

single system.

USCIS Alien Change of Address 

Card

System tracks address changes submitted to USCIS on the paper form (AR-11) and 

online through the Customer Relationship Interface System’s change of address 

component.

USCIS Central Index System Database contains information on the status of applicants/petitioners seeking 

immigration benefits; supports legacy records management process that tracks the 

location of paper case files; interfaces with more than 20 internal DHS data systems 

and a small number of external governmental entities.

USCIS Computer-Linked 

Adjudication Information 

Management System 3

Information in this and associated systems includes information provided by the 

applicant for an immigration benefit, and varies depending on the benefit. Also 

collects information to indicate which steps of the adjudication process have been 

completed, such as an appointment to submit biometrics for a background check, 

other pending benefits, and whether the applicant is suspected of fraudulent activity.

USCIS Computer-Linked 

Application Information 

Management System 4

System tracks and processes applications for naturalization.

USCIS Customer Profile 

Management System

A repository of all biometric data in USCIS.

USCIS Customer Relationship 

Interface System

Web-based system that allows applicants with pending applications, petitions, or 

requests to check case status and estimated processing time; allows applicants to 

report a change of address. Can be used to record reported issues with pending 

cases such as typographical errors or non-receipt of a document and issue 

resolution.

USCIS Enterprise 

Correspondence 

Handling On-line

Centralized data system with web-based user interfaces that replaced existing 

general support systems previously used at the various data centers. Used by 

Service Center personnel to generate various types of correspondence in 

processing forms.

USCIS Enterprise Citizenship 

and Immigration Services 

Centralized Operational 

Repository

Repository for immigration and naturalization information, developed and 

implemented in an effort to streamline access to information by consolidating 

information from several other legacy systems. It will replicate and load read-only 

records from the Computer-Linked Adjudication Information Management Systems 3 

and 4, Central Index System, and Refugee, Asylum, and Parole systems.
Appendix IV: USCIS ELIS Interfaces

Page 115 GAO-16-467 Immigration Benefits System

Agency name System name System description

USCIS Enterprise Print 

Management 

Service/Notice 

Generation Service

Service allows for notices to be printed and sent to customers, including notices of 

action and proofs of benefit. Examples include receipt notices, request for evidence, 

approval/denial notices, travel documents, permanent resident cards, and 

employment authorization documents.

USCIS Fraud Detection and 

National Security Data 

System

Case management system used to record, track, and manage immigration inquiries, 

investigative referrals, law enforcement requests, and case determinations involving 

benefit fraud, criminal activity, public safety, and national security concerns.

USCIS Identity Credential 

Access Management

System provides user authentication capability.

USCIS Integrated Card 

Production System

System produces applicable documents when an immigration benefit is granted.

USCIS InfoSphere Identity 

Insight

Tool that aggregates biographic data from USCIS ELIS, legacy USCIS data 

systems, and outside data sets. Capable of detecting duplicate accounts in USCIS 

ELIS, performing identity matches across legacy data, and revealing “obvious” and 

“non-obvious” relationships between identities.

USCIS Enterprise Service Bus 

Lockbox Intake Service

Service acts as the intake for data received from the Lockbox service provider and 

delivers the data to the target USCIS system.

USCIS Marriage Fraud 

Amendment System

System supports and maintains casework resulting from the Immigration Marriage 

Fraud Amendments of 1986 by providing aggregate and statistical information on 

casework operations, allowing employees to process and control applications and 

petitions to grant or terminate the resident status of aliens.

USCIS National Appointment 

Scheduling System

An enterprisewide appointment scheduling system.

USCIS Person Centric Query 

Service

Service provides the ability to submit a single query for all transactions involving an 

immigrant across a number of connected systems.

USCIS Refugees, Asylum, and 

Parole System

Case management tool that enables USCIS to handle and process applications for 

asylum and applications for suspension of deportation or special rule cancellation of 

removal.

USCIS Verification Information 

System

Composite information system verifies immigration status data from various DHS 

databases for benefits determination and employment authorization.

Source: GAO analysis of DHS and USCIS documentation. | GAO-16-467.

Note: This table does not include every system and service with which USCIS ELIS is planned to 

interface.
Appendix V: Comments from the Department 

of Homeland Security

Page 116 GAO-16-467 Immigration Benefits System

Appendix V: Comments from the Department 

of Homeland Security
Appendix V: Comments from the Department 

of Homeland Security

Page 117 GAO-16-467 Immigration Benefits System
Appendix V: Comments from the Department 

of Homeland Security

Page 118 GAO-16-467 Immigration Benefits System
Appendix V: Comments from the Department 

of Homeland Security

Page 119 GAO-16-467 Immigration Benefits System
Appendix V: Comments from the Department 

of Homeland Security

Page 120 GAO-16-467 Immigration Benefits System
Appendix V: Comments from the Department 

of Homeland Security

Page 121 GAO-16-467 Immigration Benefits System
Appendix VI: GAO Contact and Staff 

Acknowledgments

Page 122 GAO-16-467 Immigration Benefits System

Carol C. Harris at (202) 512-4456 or chac@gao.gov

In addition to the contact named above, Michael Holland (Assistant 

Director), Mathew Bader (Analyst in Charge), Carl Barden, Naba 

Barkakati, Jennifer Beddor, Christopher Businsky, Juana Collymore, 

Philip Curtin, Nancy Glover, Julianne Gorse, Franklin Jackson, Richard 

Sayoc and Martin Skorczynski made contributions to this report.

Appendix VI: GAO Contact and Staff 

Acknowledgments

GAO Contact

Staff 

Acknowledgments

(100111)
The Government Accountability Office, the audit, evaluation, and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and 

policies; and provides analyses, recommendations, and other assistance 

to help Congress make informed oversight, policy, and funding decisions. 

GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

The fastest and easiest way to obtain copies of GAO documents at no 

cost is through GAO’s website (http://www.gao.gov). Each weekday 

afternoon, GAO posts on its website newly released reports, testimony, 

and correspondence. To have GAO e-mail you a list of newly posted 

products, go to http://www.gao.gov and select “E-mail Updates.”

The price of each GAO publication reflects GAO’s actual cost of 

production and distribution and depends on the number of pages in the 

publication and whether the publication is printed in color or black and 

white. Pricing and ordering information is posted on GAO’s website, 

http://www.gao.gov/ordering.htm. 

Place orders by calling (202) 512-6000, toll free (866) 801-7077, or 

TDD (202) 512-2537.

Orders may be paid for using American Express, Discover Card, 

MasterCard, Visa, check, or money order. Call for additional information.

Connect with GAO on Facebook, Flickr, Twitter, and YouTube.

Subscribe to our RSS Feeds or E-mail Updates. 

Listen to our Podcasts and read The Watchblog.

Visit GAO on the web at www.gao.gov.

Contact:

Website: http://www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Katherine Siggerud, Managing Director, siggerudk@gao.gov, (202) 512-

4400, U.S. Government Accountability Office, 441 G Street NW, Room 

7125, Washington, DC 20548

Chuck Young, Managing Director, youngc1@gao.gov, (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, DC 20548

GAO’s Mission

Obtaining Copies of 

GAO Reports and 

Testimony

Order by Phone

Connect with GAO

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Congressional 

Relations

Public Affairs

Please Print on Recycled Paper.