Treasury FY 2024 Annual Evaluation Plan

 1

Treasury FY 2024 Annual Evaluation Plan

Background.

The Foundations for Evidence-Based Policymaking Act of 2018 (“Evidence Act”) requires agencies to develop a 

multi-year learning agenda that is used to identify agency priority questions. It further requires agencies to 

develop an Annual Evaluation Plan that provides a more detailed description of significant program evaluations

that the agency plans to conduct in the following fiscal year. Agencies can define “significant” for this purpose.

Approach.

This document outlines the significant1 evaluation activities that Treasury plans to conduct in FY 2024. The 

evaluations are projects to evaluate critical programs; the evidence generated by these projects will address

priority questions in the Treasury Learning Agenda. Treasury’s Interim Evaluation Policy outlines the guiding 

principles and quality standards for all evaluation activities conducted within the Department. Further, Treasury 

is working to develop a broader evidence policy that will establish ways for the Evaluation Officer to better track 

planned significant evaluations, methodologies, and dissemination of findings. The four types of evidencebuilding activities are defined as follows (subject to refinement as policies are further developed):

Evaluation Collection and analysis of data to assess effectiveness and efficiency of programs, policies, or procedures

Statistics Collection, compilation, and processing of data for describing or estimating characteristics or insights 

concerning groups

Research Modeling or other systematic use of data to explore emerging issues or potential scenarios to generate 

new knowledge

Analysis Routine and frequent use of data that produces insights for decision making and program management

Dissemination of results for Treasury Evaluation Plan Projects

The agency evaluation policy, when published, will address Treasury’s standards for the dissemination of 

evaluation results and findings. As this policy is developed, with input from the bureaus and offices that regularly 

conduct evaluations and in consultation with the Data Governance Board (including the Chief Data Officer and 

Statistical Official), the Evaluation Officer will determine the appropriate mechanism for dissemination of findings 

internally and externally. Future evaluation plans will include more specific dissemination plans for each 

evaluation. 

Contents

Overview of Planned FY 24 Evaluation Projects ............................................................2

Details associated with Planned FY 24 Evaluation Projects 

 1.1 Tax Administration and Policy ................................................................................................. 3

 1.3 Economically Resilient Communities ....................................................................................... 5

 1.4 Resilient Housing Market ......................................................................................................... 7

 5.1 Recruit and Retain a Diverse and Inclusive Workforce ........................................................... 7

 5.4 Customer Experience Practices ............................................................................................... 8

1 Treasury defines “significant evaluations” as evaluations that are undertaken to evaluate critical programs, as defined in the Treasury implementation plan 

for the Program Management Improvement Accountability Act (PMIAA), and for which the evidence generated by the evaluation is relevant to a priority 

research question in the Treasury Learning Agenda. Evaluations required by law are also considered significant. The Treasury PMIAA implementation plan 

defines critical programs as programs that are essential to Treasury’s successful execution of its mission and for which failure, interruption, or compromise of 

these programs could: 1) Undermine Treasury’s capability to achieve its objectives; or, 2) Severely impact government operations; or, 3) Result in significant 

loss of trust in the Department by key stakeholders/the public. These programs are identified in the Treasury FY 2022 – 2026 Strategic Plan. 
Treasury FY 2024 Annual Evaluation Plan

 2

Overview of Planned FY 24 Evaluation Projects

In FY 24, Treasury is planning or exploring opportunities to conduct eleven significant evaluation projects aligned 

to eight learning agenda questions. This includes projects which began in FY 23 and will continue into FY 24 

(marked with an asterisk). Treasury’s ability to execute the projects described is dependent upon resources; this 

document reflects Treasury’s plans at the time of publication, but these projects are subject to change based on 

shifting priorities and funding levels. Projects in italics are still in the exploratory phase and execution will depend 

upon various factors, such as data availability or opportunities to partner with program participants. 

Strategic Objective Learning Agenda Question(s) Project(s)

1.1 Tax 

Administration and 

Policy

How can the IRS address taxpayer needs and 

preferences to deliver a better taxpayer 

experience?

• Customer Experience (CX) 

Analytics*

• Customer Experience, 

Expectations, and Needs Survey*

1.3 Economically 

Resilient 

Communities

To what extent are American Rescue Plan 

(ARP) programs being implemented 

equitably?

What is the impact and/or outcomes of ARP 

programs on households, business, and 

governments?

• Capital Projects Fund Descriptive 

Evaluation

• State and Local Fiscal Recovery 

Program Evaluation 

What are the impacts and/or outcomes of 

the Emergency Capital Investment Program, 

Rapid Response Program, and Community 

Development Financial Institutions (CDFI) 

Fund’s Equitable Recovery Program on CDFIs 

and minority depository institutions (MDIs)?

• Emergency Capital Investment 

Program Evaluation

How does the CDFI Fund impact and boost 

investments in low-income communities in 

the long term, including Persistent Poverty

Counties?

• CDFI Fund evaluation (in 

exploratory phase) 

1.4 Resilient 

Housing Market

What strategies deployed in the recovery 

from COVID-19 best prevented evictions and 

foreclosures? 

• Emergency Rental Assistance 

Flexibilities (in exploratory phase)

• Homeowners Assistance Fund 

Outreach (in exploratory phase)

5.1 Recruit and 

Retain a Diverse 

and Inclusive 

Workforce 

To what extent has increased diversity in 

management and leadership positions 

influenced organizational performance and 

improved employees’ sense of inclusion?

• Reducing Disparities in 

Performance Evaluations (in 

exploratory phase)

5.4 Customer 

Experience 

Practices 

What are effective strategies for serving 

hard-to-reach customers, particularly those 

who lack access to reliable internet, 

banks/financial institutions, and the 

credentials required to verify their digital 

identity?

• Encouraging Uptake of Tax Credits 

(in exploratory phase)

• To be determined – Treasury 

Customer Experience Community of 

Practice and Bureau of Fiscal 

Service exploring potential projects

*Projects began in FY 23 and are carried over to FY 24
Treasury FY 2024 Annual Evaluation Plan

 3

Details Associated with Planned FY 24 Evaluation Projects

Treasury Strategic Objective 1.1 Tax Administration and Policy

How can the IRS address taxpayer needs and preferences to deliver a better taxpayer experience?

Customer Experience (CX) Analytics

Lead Organization: IRS Research, Applied Analytics, and Statistic (RAAS) Division / Data Management Division 

Supporting Organization(s): IRS Wage & Investments Division

Evaluation Questions:

1. How can Natural 

Language Processing 

techniques improve 

the taxpayer 

experience?

2. How can machine 

learning be used to 

identify opportunities 

for process 

improvements?

3. How can we be sure 

implemented 

recommendations 

have the desired effect 

on the taxpayer 

experience?

 Data Sources

• Account 

Management 

Services (AMS) 

• Individual 

Master File (IMF) 

• Notice Delivery 

System (NDS)

• Individual 

Returns 

Transaction File 

(IRTF)

• Online Services 

(OLS)

• Customer Call 

Transcripts 

• A-11 Survey Data

• Google Analytics, 

Forsee Survey

 Analytic Tools

• The entire range of 

data and tools 

available on CDW, 

specifically SAP 

HANA database, 

Sybase IQ, R 

statistical software 

package, and 

Python.

• A visualization tool 

we created (CX 

Solution) that 

leverages a variety 

of data sources and 

organizes events 

and interactions 

over time to 

present the unique 

journeys of the 

taxpayer. 

 Analytic Approaches

• Text Mining Capabilities –

Natural Language Processing 

(NLP) models and tools 

developed to analyze, quantify, 

and categorize unstructured, 

textural data. 

• Tokenization of AMS and 

National Quality Review System 

(NQRS) narratives

• Topic Modeling using LDA 

(Latent Dirichlet Allocation)

• Issue Classification using 

Support Vector Machine (SVM) 

models

• Sentiment Analysis

• Predictive Models – Analytical 

models developed to predict 

the likelihood of a customer 

entering a specific end state 

and prediction of next journey 

steps within a tax module. 

Models include Logitsistic 

Regression, Decision Trees, 

Random Forests, and Neural 

Networks

Progress made:

In FY 22, IRS RAAS began partnering with the IRS Taxpayer Experience Office (TXO) to analyze taxpayer journeys 

and improve customer experience. The IRS is analyzing unstructured data sources (i.e., AMS notes and call 

transcripts) using Natural Language Processing and using the insights to train Customer Service Representatives 

(CSRs), suggest alternative treatments, and inform communications. The IRS is also developing models to predict 

taxpayer outcomes to inform outreach and resource decisions, identify and correct bottlenecks. This project is on 

track and will continue into FY 24.

Challenges associated with project: 

• Implementation of recommendations for process improvements requires buy-in from multiple 

stakeholders and often requires changes to established processes, procedures and training

• Each machine learning technique has pros and cons; need to have them compete against each other 

using common objectives and criteria

• Aligning data at the TIN level is not always possible

• Processing large data volumes in realtime
Treasury FY 2024 Annual Evaluation Plan

 4

How can the IRS address taxpayer needs and preferences to deliver a better taxpayer experience?

Customer Experience, Expectations, and Needs (CEEN) Survey

Lead Organization: IRS Small Business/Self-Employed (SB/SE) Division

Supporting Organization(s): none

Evaluation Questions

1. How can the IRS 

effectively measure the 

needs and expectations 

of SB/SE Taxpayers? 

2. How can the IRS 

effectively measure the 

needs and expectations 

of Tax professionals? 

3. How can the IRS 

ensure that future 

actions are priorities 

supported by taxpayers 

and tax professionals? 

Data Sources

Survey data from two surveys:

• The 2022 Customer 

Expectations, Experience and 

Needs (CEEN) survey engaged

small business and selfemployed taxpayers to measure 

expectations for interactions 

with the IRS, needs for 

compliant tax administration, 

and awareness and use of IRS 

products and services.

• The 2023 Tax Professional 

Engagement Survey will be 

developed in partnership with 

other IRS functions to better 

understand the tax professional 

community and identify 

recommendations to improve 

overall tax compliance by 

leveraging the relationship 

between tax professionals and 

their clients

Analytic Tools

• SAS, SPSS, R 

statistical 

software

package

• Microsoft Excel

Analytic Approaches

• Data collection 

through surveys

• Focus groups 

• Quantitative data 

analytics (statistical 

analysis)

Progress made:

In July 2022, the IRS received the Customer Experience, Expectations, and Needs (CEEN) survey, which was 

completed by over 3,000 SM/SE taxpayers. The CEEN survey is expected to be conducted annually and the IRS has 

begun analyzing the survey data and will continue to do so to answer the evaluation questions above. The IRS also 

received funding for a survey of tax professionals and awarded this contract. This project is on track and will 

continue into FY 24.

Challenges associated with project: 

• The IRS’s ability to obtain a representative sample of SB/SE practitioners is dependent on response rates

• If a representative sample cannot be obtained, data gathered may need to be treated as directional and 

qualitative, rather than quantitative
Treasury FY 2024 Annual Evaluation Plan

 5

Treasury Strategic Objective 1.3 Economically Resilient Communities

As a complement to the Treasury Learning Agenda, the Office of Recovery Programs (ORP) developed its own 

learning agenda that explores more nuanced questions regarding the American Rescue Programs. In FY 23, 

Treasury will conduct stakeholder engagements for the ORP learning agenda and, through this process, seek 

opportunities to collaborate with external researchers and evaluators to build evidence in support of these 

questions. 

To what extent are American Rescue Plan (ARP) programs being implemented equitably? 

What is the impact and/or outcomes of ARP programs on households, business, and governments?

Capital Projects Fund (CPF) Descriptive Evaluation

Lead Organization: Departmental Offices – Office of Recovery Programs and the General Services Administration 

(GSA) Offices of Evaluation Sciences (OES), through an interagency agreement 

Supporting Organization(s): Departmental Offices – Economic Policy

Evaluation Question:

Where are funds being 

spent by CPF recipients?

Data Sources:

• CPF 

reporting 

data

Analytic Approach:

Descriptive study of CPF’s 

implementation to identify proportion 

of funds directed to internet deserts, 

high-poverty areas, and communities 

disproportionately affected by the 

pandemic.

Challenges: 

Treasury and OES are 

exploring how to best 

evaluate how funding is 

flowing to internet deserts 

and other communities 

that lack access to 

adequate high-speed 

internet service. 

State and Local Fiscal Recovery Fund (SLFRF) Evaluation

Lead Organization: Departmental Offices – Office of Recovery Programs and the General Services Administration 

(GSA) Offices of Evaluation Sciences, through an interagency agreement 

Supporting Organization(s): Departmental Offices – Economic Policy

Evaluation Question:

What is the impact of 

specific SLFRF projects 

on priority policy areas 

such as affordable 

housing, workforce, and 

public safety?

Data Sources:

• SLFRF 

project and 

expenditure 

reports

Analytic Approach:

Using data provided by recipients as 

part of the SLFRF Project and 

Expenditure Reports, Treasury and 

GSA’s Office of Evaluation Sciences are 

actively exploring an evaluation(s) 

focused on recipient governments’ 

projects in the areas of affordable 

housing, workforce, and public safety.

Challenges: 

Identifying state and local 

government partners 

willing to collaborate and 

share data 
Treasury FY 2024 Annual Evaluation Plan

 6

How does the CDFI Fund impact and boost investments in low-income communities in the long term, 

including Persistent Poverty Counties?

Note: this project is still in the exploratory phase. Treasury and OES are working to determine the feasibility 

of the evaluation approach and to identify potential collaborators.

Community Development Financial Institution (CDFI) Fund Evaluation

Lead Organization: Departmental Offices – Domestic Finance 

Supporting Organization(s): To be determined

Evaluation Question: 

To what extent do federal 

government resources 

(specifically those of the CDFI 

Fund) enable the CDFI 

industry to fulfill its mission?

Potential Evaluation Approach: 

The CDFI Fund is exploring a multi-component research project that would help 

the Fund further its mission and inform policy development and program design. 

The Fund is interested in studying both the CDFI industry’s effectiveness at 

increasing access to capital for economically disadvantaged communities and the 

role of the CDFI Fund in supporting these institutions. This project may include a 

study of the current size of the community development finance industry, 

including CDFI market share, and an analysis of the impact of CDFI Fund awards 

on the effectiveness of a CDFI, including its financial health and performance.

What are the impacts and/or outcomes of the Emergency Capital Investment Program, Rapid Response 

Program (RRP), and Equitable Recovery Program (ERP) on CDFIs and minority depository institutions 

(MDIs)?

Emergency Capital Investment Program Evaluation

Lead Organization: Departmental Offices – Domestic Finance 

Supporting Organization(s): Departmental Offices – Office of Recovery Programs, external evaluator (TBD)

Evaluation Questions:

1. What are the initial patterns of 

investment and institutional changes 

after investment through the Emergency 

Capital Investment Program (ECIP)? 

2. How effectively did ECIP award 

recipients serve priority communities 

and borrowers following receipt of an 

investment?

3. How have ECIP, RRP, and ERP 

changed the capacity of the 

participating institutions and what 

broader implications might that have for 

sector and field level approaches to 

serving low- and moderate-income 

communities and populations? 

Data Sources:

• ECIP Quarterly 

Summary Reports 

(QSRs)

• Home Mortgage 

Disclosure Act 

(HMDA) data 

• Census data

• Future 1071 data 

collection by the 

Consumer 

Financial 

Protection Bureau 

(CFPB)

• Voluntary data 

collection from 

participants 

Analytic Approach:

In FY 2023, develop a 

descriptive report on initial 

investment patterns, with 

respect to qualified and 

deep-impact lending, along 

with providing initial insights 

into how participating 

institutions are changing. 

Beginning in FY 2024, 

engage third-party evaluator 

to determine the impact of 

funding made available to 

recipient institutions and 

identify potential field or 

sector-level implications of 

these investments

Challenges: 

Identifying 

third-party 

evaluator 

with expertise 

in CDFI and 

MDI industry; 

obtaining 

sufficient data 

to conduct 

robust 

evaluation
Treasury FY 2024 Annual Evaluation Plan

 7

Treasury Strategic Objective 1.4 Resilient Housing Market

What strategies deployed in the recovery from COVID-19 best prevented evictions and foreclosures? 

Note: these projects are still in the exploratory phase. Treasury and OES are working to determine the 

feasibility of the evaluation approach and to identify potential collaborators.

Emergency Rental Assistance (ERA) Flexibilities 

Lead Organization: Departmental Offices – Office of Recovery Programs and the General Services Administration 

(GSA) Offices of Evaluation Sciences (OES), through an interagency agreement 

Evaluation Question:

How has the use of promising practices 

that Treasury encouraged grantees to 

adopt (such as self-attestation, 

categorical eligibility, and fact-specific 

proxies) affected the equitable 

distribution of ERA funds?

Potential Evaluation Approach:

Treasury is working with individual ERA grantees to explore options for a 

quasi-experimental study of different approaches to administering ERA 

using Treasury’s promising practices and the effects on program 

applications.

Homeowners Assistance Fund (HAF) Outreach 

Lead Organization: Departmental Offices – Office of Recovery Programs and OES

Evaluation Question:

How equitable is the distribution of 

HAF funds to homeowners most in 

need of assistance?

Potential Evaluation Approach:

To answer this question, Treasury is exploring evaluation options to 

understand equity throughout the lifecycle of the HAF program, from 

standup and initial outreach to the application process to distribution of 

funds and outcomes. This includes exploring partnerships with 

HAF funding recipients to examine the effectiveness of different types of 

targeted outreach at equitably engaging homeowners that are most in 

need of HAF assistance, such as socially disadvantaged homeowners. In 

the longer-term, Treasury will work with these partners and additional 

stakeholders to consider options for evaluating equity in the distribution 

of funds and program outcomes.
Treasury FY 2024 Annual Evaluation Plan

 8

Treasury Strategic Objective 5.1 Recruit and Retain a Diverse and Inclusive Workforce 

Treasury Strategic Objective 5.4 Customer Experience Practices 

To what extent has increased diversity in management and leadership positions influenced organizational 

performance and improved employees’ sense of inclusion?

Note: this project is still in the exploratory phase. Treasury and OES are working to determine the feasibility 

of the evaluation approach and to identify potential collaborators.

Reducing Disparities in Performance Evaluations 

Lead Organization: Departmental Offices and the General Services Administration (GSA) Offices of Evaluation 

Sciences (OES), through an interagency agreement 

Potential Evaluation Approach:

Treasury Management (including the office of the Deputy Assistant Secretary for Human Resources and Chief 

Human Capital Officer (DASHR-CHCO)), the Office of Diversity Equity Inclusion and Accessibility (ODEIA), and OES 

are exploring opportunities to conduct a randomized control trial that would evaluate the effectiveness of scalable 

interventions to reduce bias in decision-making in performance evaluations. This project is still in the exploratory 

phase and feasibility has not yet been determined.

What are effective strategies for serving hard-to-reach customers, particularly those who lack access to 

reliable internet, banks/financial institutions, and the credentials required to verify their digital identity?

Note: this project is still in the exploratory phase. Treasury and OES are working to determine the 

feasibility of the evaluation approach and to identify other potential collaborators.

Encouraging Uptake of Tax Credits

Lead Organization: The Internal Revenue Service (IRS) Taxpayer Experience Office and the General Services 

Administration (GSA) Offices of Evaluation Sciences (OES)

Potential Evaluation Approach:

The IRS Taxpayer Experience Office and OES are exploring opportunities to conduct a randomized control trial that 

would evaluate the effect of outreach strategies on uptake of certain tax credits among low-income and 

multilingual populations. This project is still in the exploratory phase and feasibility has not yet been determined. 

Additional projects to be determined 

Potential Evaluation Approach:

The Treasury Customer Experience Community of Practice (COP) is exploring opportunities to conduct evaluations 

that would help the Department improve outreach and customer experience for hard-to-reach customers. The 

Bureau of the Fiscal Service is also considering evaluation projects that would build evidence related to this 

learning agenda question. In FY 22, Fiscal Service developed a bureau-level learning agenda and FY 22-23 

evaluation plan which aligns to the bureau strategic plan. In FY 23, the bureau intends to publish its FY 24 Annual 

Evaluation Plan, which will include planned projects and alignment to Treasury Learning Agenda questions.