1

USCIS FY 2024

ANNUAL EVALUATION 

PLAN
2

THIS PAGE INTENTIONALLY LEFT BLANK
3

CONTENTS

Overview ........................................................................................... 5

USCIS FY 2024 Significant Evaluations.............................................. 6

Evaluation #1 .............................................................................. 6

Evaluation #2 .............................................................................. 8

Evaluation #3 ............................................................................ 10

Evaluation #4............................................................................ 12

Appendix A. Glossary ...................................................................... 15
4

THIS PAGE INTENTIONALLY LEFT BLANK
5

OVERVIEW

U.S. Citizenship and Immigration Services (USCIS) is responsible for administering the nation’s lawful 

immigration system. USCIS has the responsibility to deliver decisions about immigration benefit requests 

to individuals while ensuring the security of our nation. The work of USCIS employees makes the 

possibility of the American dream a reality for immigrants, receiving communities, the economies they 

join, and the nation as a whole. 

In accordance with the Foundations for Evidence‐Based Policymaking Act of 2018 (Evidence Act) and U.S. 

Department of Homeland Security (DHS) policy, USCIS is issuing its USCIS FY 2024 Annual Evaluation 

Plan (AEP). Along with the USCIS FYs 2023-2026 Learning Agenda, the USCIS FY 2024 AEP is part of USCIS’ 

broader efforts to expand and mature capabilities to build and use evidence in shaping strategy and 

operations. USCIS generates and uses rigorous evidence from evaluations to inform decisions about 

programs, policies, regulations, and organizations, better enabling the agency to achieve its mission. 

The AEP describes a subset of USCIS’s evaluation work for FY 2024. These evaluations, designated as 

‘significant,’ are shared with the American public and receive additional resources to ensure successful 

completion. Each year, USCIS works with stakeholders to identify new evaluations. Many evaluations are 

designed to address priority questions identified in the USCIS FYs 2023-2026 Learning Agenda. As such, 

they contribute to building a body of evidence to support decision makers while fostering organizational 

learning. USCIS evaluations are conducted consistent with relevant legal authorities and privacy, civil 

rights, and civil liberties protections.

USCIS defines significant evaluations as evaluations that:

Ø Support the USCIS FYs 2023-2026 Learning Agenda

Ø Align with USCIS leadership and Administration priorities

Ø Respond to a mandate such as GAO or OIG recommendations

Ø Have potential for agency-wide impact or engagement

Ø Have potential for high financial impact

Ø Have potential for high stakeholder impact

USCIS evaluations may include a range of evaluation types and methods to best answer the questions 

proposed. To ensure credibility and quality of evidence for learning and decision making, USCIS 

evaluations follow the principles of relevance and utility, rigor, independence and objectivity, ethics, and 

transparency. These principles align with published Federal evaluation standards.1

1 Phase 4 Implementation of the Foundations for Evidence‐Based Policymaking Act of 2018: Program Evaluation 

Standards and Practices, M‐20‐12 (OMB, 2020)
6

USCIS FY 2024 SIGNIFICANT EVALUATIONS

Evaluation #1

Name of Evaluation Naturalization Outreach Evaluation

Type of Evaluation Impact evaluation

Strategic Plan Goal 

and Objective

Goal 1: Strengthen the U.S. Legal Immigration System 

Objective 1.3: Naturalization Promotion and Process Improvement 

Learning Agenda 

Priority Question

Section C: Naturalization Promotion and Process Improvement

Priority Question: What strategies work to encourage naturalization?

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Background: USCIS estimates that there are 9.1 million people who may be 

eligible to naturalize that have not taken steps to become U.S. citizens. USCIS 

has not historically conducted targeted outreach with individuals that may be 

eligible to naturalize. USCIS plans to conduct targeted outreach to the population 

that may be eligible to naturalize and rigorously test the impact of the targeted 

outreach on individuals filing for naturalization. With the targeted outreach, 

USCIS hopes to increase the likelihood of lawful permanent residents (LPR) 

applying for naturalization and naturalizing.

Purpose and Use: The evaluation will assess the effects of a targeted outreach 

or promotion effort not regularly part of USCIS services/processes on positive 

lawful permanent resident behavior, specifically naturalization application 

filing, and the long-term outcome of naturalization. The evaluation results will 

inform process improvements, including coordination and communication with 

stakeholders.

The study will demonstrate whether the targeted outreach efforts increase 

naturalization filings. If proven, the findings will enable USCIS to incorporate such 

outreach into its regular services/processes. Testing the impact of such targeted 

outreach on behavior change will be valuable research to other federal agencies, 

universities, and research organizations, as well as state and local governments.

Key Evaluation Questions:

1. Does the receipt of a mailed notification of potential eligibility impact the 

likelihood of a LPR to apply for naturalization (submit Form N-400)?

2. Does the receipt of a mailed notification of potential eligibility impact the 

likelihood of a LPR to submit a Form N-400 online?

3. Does the receipt of a mailed notification of potential eligibility impact the 

likelihood of naturalization?
7

Evaluation #1

Data / Information 

Needed

Secondary data sources: USCIS administrative data such as gathered from USCIS 

Electronic Immigration System, USCIS Computer Linked Application Information 

Management System, and Enterprise Citizenship, Immigration Services 

Centralized Operational Repository (eCISCOR), and Form AR-11 (Alien’s Change of 

Address Card). 

No new data collection is anticipated.

Methods USCIS plans to conduct a randomized control trial (RCT) (behaviorally informed 

intervention) to identify the effect of direct communication towards Lawful 

Permanent Residents (LPRs) on: (1) their likelihood of applying for naturalization 

(i.e., submitting the N-400 form) and (2) submitting an online N-400 submission. 

As a secondary and longer-term outcome, we may examine whether the direct 

communication increases the likelihood of successful naturalization. The primary 

approach will be to mail a letter to LPRs who are likely eligible for naturalization. 

This letter addresses various behavioral barriers by including: a fresh start 

motivator (framing moments in time as new beginnings), the benefits of U.S. 

citizenship, a checklist of next steps and a social motivator (information on peer 

behavior).

Challenges and 

Limitations

The study may have implementation challenges associated with accurate address 

information and tracking successfully delivered mail. This could affect the 

interpretability of the results.

Dissemination Public disclosure is anticipated.
8

Evaluation #2

Name of Evaluation Citizenship and Integration Grant Program (CIGP): Citizenship Instruction and 

Naturalization Application Services (CINAS) Grant and Community and Regional 

Integration Network Grant (CARING) Program Evaluation

Type of Evaluation Process/implementation evaluation

Strategic Plan Goal 

and Objective

Goal 1: Strengthen the U.S. Legal Immigration System 

Objective 1.3: Naturalization Promotion and Process Improvement 

Learning Agenda 

Priority Question

Section C: Naturalization Promotion and Process Improvement

Priority Question: To what extent and in what ways do USCIS’ grant programs 

successfully prepare eligible populations for Naturalization?

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Background: The goal of the Citizenship and Integration Grant Program (CIGP) 

is to expand the availability of high-quality citizenship preparation services for 

immigrants across the nation and to provide opportunities for immigrants to 

gain the knowledge and skills necessary to integrate into the fabric of American 

society. The Citizenship Instruction and Naturalization Application Services 

(CINAS) grant is for public or nonprofit organizations that prepare immigrants 

for citizenship by offering both citizenship instruction and naturalization 

application services. The Community and Regional Integration Network Grant 

(CARING) (formerly known as the Refugee and Asylee Integration Services 

Program or RAIS) is for organizations that provide extended integration services 

to vulnerable immigrant populations who entered the United States through 

USCIS’ humanitarian programs or benefitted from those programs while already 

in the United States. These groups often experience unique challenges with civic, 

linguistic, economic, cultural, and institutional integration when resettling in the 

United States, which may impact their progress toward full civic integration.

Purpose and Use: The study is a process/implementation evaluation to assess 

whether the CINAS and CARING programs are being implemented as intended, 

including identification of deviations from the grant recipient implementation 

plan and the root causes of those deviations, with the intent of identifying 

challenges with implementation. This evaluation will inform USCIS leadership and 

program staff, Office of the Citizenship and Immigration Services Ombudsman, 

CINAS and CARING program grant recipients, Congress, OMB, and the interagency 

Naturalization Working Group. 

Understanding if the CINAS and CARING programs are being implemented as 

intended and the challenges with implementation will help USCIS determine 

what, if any, changes are needed to improve program design and implementation 

to ensure efficiency, effectiveness, and equity of future iterations of the CINAS 

and CARING grant programs. In accordance with 2 C.F.R. 200.301, USCIS will use 

findings to share lessons learned, improve program outcomes, and encourage the 

adoption of promising practices by grant recipients.
9

Evaluation #2

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Key Evaluation Questions:

1. How did CINAS and CARING grant recipients implement the grant? What 

were the causes of variation among grant recipients?

2. What external factors influenced implementation?

3. Did implementation result in the intended outputs?

4. Are participants being reached as intended?

Data / Information 

Needed

Primary data sources: Primary data sources may include USCIS CINAS and 

CARING program staff and grant recipient staff. Methods of primary data 

collection will include qualitative interviews or focus groups.

Secondary data sources: Grant recipient administrative/operational data 

reported to USCIS, which include data on grant recipients’ implemented 

activities, services provided, and individual participants; CINAS and CARING 

program administrative data, including original proposals and required progress 

reporting; and USCIS Electronic Immigration System, USCIS Computer Linked 

Application Information Management System, and Central Index System data, 

mainly collected in Form I‐485 (Application to Register Permanent Residence or 

Adjust Status), Form N‐400 (Application for Naturalization), and Form G-28 (Notice 

of Entry of Appearance as Attorney or Accredited Representative).

Methods The study is a process/implementation evaluation, to assess whether the CINAS 

and CARING Program are being implemented as intended, including deviations 

and root causes of deviations from the grant recipient implementation plan, 

toward identifying challenges with implementation. Descriptive statistical 

analysis will be used for quantitative data. Qualitative analysis, such as content 

analysis and theme identification, will be used for qualitative data.

In the future, USCIS may consider conducting an outcome evaluation to assess 

the programs’ contributions to intended outcomes, such as improved English 

language proficiency, civics knowledge, and naturalization of participating 

noncitizens.

Challenges and 

Limitations

Gaps in data access or quality could be a challenge to conducting the evaluations, 

particularly surrounding the CARING program with a small volume of recipients 

and participants.

Dissemination Public disclosure is anticipated.
10

Evaluation #3

Name of Evaluation E-Verify NextGen Program Evaluation

Type of Evaluation Formative evaluation

Strategic Plan Goal 

and Objective

Goal 1: Strengthen the U.S. Legal Immigration System 

Objective 1.2: Fair, Efficient, and Humane Adjudication 

Learning Agenda 

Priority Question

Section A: Access to the Legal Immigration System

Priority Question: What strategies are effective for improving the customer 

experience?

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Background: E-Verify is a web-based system that allows enrolled employers to 

confirm the employment eligibility of newly hired employees by electronically 

matching information provided by employees on Form I-9, Employment Eligibility 

Verification, against records available to the Social Security Administration and 

the Department of Homeland Security (DHS).

Employees and employers complete Form I-9 and employers retain Form I-9 for 

every person they hire. Currently, for employers who are enrolled in E-Verify, a 

case is created by the employer based on the information taken directly from 

the form. In the proposed enhancement to E-Verify referred to as ‘NextGen,’ 

the employer initiates the case and then the employee receives an invitation 

to securely upload their personal data into the system. If the case receives a 

Tentative Non-Confirmation (mismatch), the system generates a notification 

to the employee, and the employee may choose to resolve the mismatch and 

respond through the secure system. 

These changes are intended to enhance the E-Verify process by allowing the 

employee to have control of their own information, with direct access to E-Verify, 

rather than relying on the employer to transmit that information. The intent is 

to provide easier, faster, and more accurate results requests for employment 

authorization verification and to improve employer and employee use of the 

E-Verify system. 

Purpose and Use: USCIS plans to conduct a formative evaluation of 

enhancements that will be made to E-Verify as part of the demonstration project 

pilot NextGen, to examine outcomes, such as processing times, false mismatches, 

mismatch resolution times, and customer experience with and perceptions of the 

process. 

Understanding the characteristics and numbers of employers and employees that 

may participate, and the costs and benefits to them opting into the alternative 

process created by E-Verify NextGen will enable USCIS to plan for future business 

processes, strategically prioritize investments on system enhancements and 

improvement, policy changes, and outreach. The evaluation will inform decisions 

on making NextGen a permanent and scalable option for verification.
11

Evaluation #3

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Key Evaluation Questions:

1. How do Form I-9 burden, mismatches, mismatch resolution times, and 

customer experiences vary between the enhancement and the current 

system? 

2. Do these vary by business or employee characteristics?

Data / Information 

Needed

Primary data sources: USCIS may collect data from employer and employee 

users on user experience, and satisfaction. Methods of qualitative data collection 

for user experience, and satisfaction, will include some combination of surveys, 

interviews, and focus groups.

Secondary data sources: Administrative data captured by the E-Verify 

NextGen system on the system use, user (business and prospective employee) 

characteristics, and processing times.

Methods The planned study is a formative evaluation of enhancements that will be 

made to the verification process by using E-Verify NextGen the demonstration 

project pilot, to examine outcomes, such as processing times, false mismatches, 

mismatch resolution times, and customer experience with and perceptions of the 

process. 

Descriptive statistical analysis and inferential statistical analysis, such as 

correlational analyses and trend analyses of quantitative data, will be used 

to assess primary outcomes (efficiency information) and user characteristics. 

Qualitative analysis, such as sentiment analysis and theme identification, will 

be used to assess secondary outcomes of user experience and satisfaction. 

Both sets of data will be compared to data from the current E-Verify system to 

determine if the desired outcomes are improved.

Challenges and 

Limitations

Although E-Verify NextGen system data is available and of a good quality, the 

primary data collection from employers and employees using the E-Verify 

NextGen system might represent some challenges.

Dissemination Public disclosure is anticipated.
12

Evaluation #4

Name of Evaluation Encouraging the Use of USCIS’ Online Services Evaluation

Type of Evaluation Impact evaluation

Strategic Plan Goal 

and Objective

Goal 1: Strengthen the U.S. Legal Immigration System 

Objective 1.1: Access to the Legal Immigration System 

Learning Agenda 

Priority Question

Section A: Access to the Legal Immigration Systems

Priority Question: What strategies work to encourage people to e-file?

Evaluation 

Background, 

Purpose, Use, 

and Evaluation 

Questions

Background: The USCIS online account and filing options offer several benefits 

to customers, including the ability to track case status, view personalized 

processing times (for some forms) for various case milestones and overall 

completion, send a secure message to the USCIS Contact Center, and instantly 

receive and respond to Requests for Evidence online rather than waiting for mail 

correspondence. This results in faster processing times for Requests for Evidence 

to support a case decision. The use of online services also reduces costs and 

burdens on the agency. 

A priority for USCIS is increasing the number of customers who use these online 

services. Customers may underutilize online tools for a variety of reasons, 

including a lack of awareness that they exist, a (mis)belief that such tools are not 

secure, or a perception that online tools are complicated or cumbersome. For 

example, only around 60 percent of those who file to replace or renew their green 

card do so online. 

Purpose and Use: This evaluation aims to rigorously measure the impact 

of light-touch methods, specifically, messages aimed at encouraging more 

USCIS customers to utilize online services. USCIS hopes to understand if these 

messages are effective in promoting online services and which messages are 

most effective in order to determine if and how to scale this type of outreach and 

promotion. 

Understanding which messages are more effective at promoting the use of online 

services will enable USCIS to deploy successful messages at scale, increase 

the use of online services, and ultimately decrease the use of paper filing and 

communications.

Key Evaluation Questions:

1. Are messages effective in promoting the use of online services? For example, 

does receiving a text message affect the likelihood that a lawful permanent 

resident submits a renewal application online rather than by mail? 

2. Which message(s) are most effective? For example, which, if any, text 

message alternatives increase the likelihood that a lawful permanent 

resident submits a renewal application online rather than by mail?
13

Evaluation #4

Data / Information 

Needed

Secondary data sources: USCIS administrative data, such as gathered in Form 

I-485 (Application to Register Permanent Residence or Adjust Status) and Form 

I-90, and managed in USCIS Electronic Immigration System, USCIS Computer 

Linked Application Information Management System, and Enterprise Citizenship 

and Immigration Services Centralized Operational Repository. 

No new data collection is anticipated.

Methods The planned study is an impact evaluation of a behaviorally informed 

intervention to identify the effect of different messages on the likelihood of using 

online services.

The first study involves modifications to existing text message outreach 

processes to notify lawful permanent residents of the availability of online 

filing for Form I-90 at six, four, and two months prior to their renewal deadlines. 

Approximately 16,000-17,000 lawful permanent residents in each renewal group 

(six, four, and two months) are contacted each month, for a total of 48,000-51,000 

contacted each month. This evaluation will only affect outreach conducted to 

lawful permanent residents in the two-month group. Each individual in the study 

will be randomly assigned to one of three groups: (1) control (no text message), 

(2) status quo text message, or (3) modified language text message. The analysis 

will identify: (1) the average effect of text messages on modality of renewal filing; 

(2) the average effect of the modified text message language (relative to the 

status quo language) on modality of renewal filing; and (3) any heterogeneous 

effects by relevant subgroups (e.g., country of birth). 

The second study involves modifications to existing email notifications aimed at 

increasing online account creation for those who have recently filed a paper form 

that was eligible for online filing. The email notifications encourage the filer to 

create an online account and link their recently filed case to the online account. 

Approximately 70,000 filers on average receive this email outreach each month. 

Filers will be randomly assigned to one of three conditions: (1) the status quo 

email, (2) modified language email that emphasizes operational transparency 

and benefits, or (3) modified language email that emphasizes social norms. The 

analysis will determine (1) the average effect of each email notification variation 

on customers’ likelihood of setting up an online account and linking their case; 

and (2) any heterogeneous effects by relevant subgroups (e.g., country of birth).

Descriptive and inferential statistical analysis will be used for quantitative 

administrative data. Predetermined control variables, such as country of birth, 

will be used to determine heterogenous effects by relevant subgroups.

Challenges and 

Limitations

The findings may have limited generalizability. The findings of the study cannot 

be applied or used to make decisions related to other USCIS programs, due to the 

narrow scope of the evaluation

Dissemination Public disclosure is anticipated.
14

APPENDIX A. GLOSSARY

Terms used in the evaluation plans are defined below.

Formative evaluation – Formative evaluation assesses whether a program, policy, regulation, or 

organization approach (or some aspect of these) is feasible, appropriate, and acceptable before it is 

fully implemented. It may include process and/or outcome measures. However, it focuses on learning 

and improvement and does not aim to answer questions of overall effectiveness. It can help answer the 

questions, “Is the program, policy, regulation, or organization appropriate for this context,” “Does it 

feasibly address the identified needs,” and “Can it be implemented as designed?”

Impact evaluation – Often used for summative purposes, impact evaluation assesses the causal effect 

or impact of a program on outcomes by estimating what would have happened in the absence of the 

program or aspect of the program. This estimation requires the use of experimental/randomized control 

trial (RCT) designs or quasi‐experimental designs (QED) in which another group is compared to program 

participants. Experimental/RCT designs randomly assign (e.g., lottery draw) persons to either a treatment 

group that receives the program or policy intervention or to a control group that does not. Quasi‐

experimental groups identify a program or policy intervention group and comparison group from pre‐

existing or self‐selected groups and not through random assignment. Impact evaluation can help answer 

the question, “Does the program, policy, regulation, or organization work, or did it lead to the observed 

outcomes?”

Outcome evaluation – Used for summative purposes, outcome evaluation assesses the extent to which 

a program, policy, regulation, or organization approach has achieved certain objectives, and how it 

achieved these objectives. Outcome evaluations use non‐experimental designs characterized by the 

absence of a control or comparison group. Unlike impact evaluation, outcome evaluation cannot discern 

that outcomes result from or are a causal effect of the program. It can help answer the question, “Were the 

intended outcomes achieved?”

Primary data sources – Individuals, groups, or organizations from which new data collection is expected, 

designed specifically for the evaluation.

Process/implementation evaluation – Process/implementation evaluation assesses the extent to 

which essential elements of a program, policy, regulation, or operation are in place; conform to 

requirements, program design, professional standards, or customer expectations; and are capable of 

delivering positive outcomes. It can help answer the questions, “Was the program, policy, regulation, 

or organization implemented as intended?” or “How is it operating in practice?” In the learning agenda, 

several evaluations study process‐related questions to understand underlying mechanisms of outcomes 

achievement.
15

Quantitative surveys – Surveys are predetermined set of questions, often with set response options 

administered to samples or panels of respondents to cost‐effectively compile statistical information about 

individuals, households, and organizations. USCIS uses surveys in different ways. USCIS uses surveys to 

track variables of longer‐term interest, as well as to obtain reliable information about conditions through 

shorter‐term studies. USCIS conducts low‐burden Customer Experience (CX) surveys to gather near real‐

time impressions of customers’ touchpoint(s) or transaction(s) with a government service in terms of trust, 

overall satisfaction, and experience drivers (e.g., service quality, process, and people, when applicable). 

USCIS also uses surveys of participants in program evaluations to determine their baseline conditions and 

subsequent outcomes.

Qualitative data analysis – A flexible set of approaches to examine patterns in communicated 

information. Content analysis may focus on the presence and frequency of concepts—typically words, 

phrases, or images— or show how concepts are related to each other and the context in which they exist. 

Thematic Framework Analysis identifies patterns of meaning, or themes. Themes may be determined 

deductively (themes selected from existing research or theory) or inductively (themes built from the data) 

to develop patterns. The analysis may examine explicit content of data or examine subtext or assumptions 

from the data. 

Qualitative interviews/focus groups – These qualitative data collections use primarily open‐ended 

questions to converse with an individual respondent or with a small group of respondents simultaneously 

to collect narrative information about a subject, circumstance, or event. USCIS uses this method across 

evidence‐building activities to understand the way people think, their motivation, and their attitudes 

toward the topic or experience. Although qualitative interviews/focus groups cannot be used to infer 

causality or to measure effectiveness, they are often valuable tools for theory building and developing 

awareness of factors that affect outcomes. As such they often complement other evidence building such 

as surveys, economic analysis, and different types of program evaluation.

Secondary data sources – Existing data, or data collected for purposes other than the specific evidence 

building activity.