44738 Federal Register / Vol. 86, No. 154 / Friday, August 13, 2021 / Notices 

1See http://www.gao.gov/assets/680/678551.pdf. 

2https://www.federalregister.gov/d/2016-29215. 

3https://www.congress.gov/115/plaws/publ435/ 

PLAW-115publ435.pdf. See also the website, at 

https://www.congress.gov/bill/115th-congress/ 

house-bill/4174, on the bill, H.R. 4174 of the 115th 

Congress, that became the Act. 

• Assess the extent to which access to 

health care was enhanced for 

population groups and individuals 

vulnerable to behavioral health 

disparities residing in communities 

targeted by funded interventions. 

• Assess the process of adopting and 

implementing the SPF with the target 

populations. 

• Added questions to capture details 

on the intervention and the referrals to 

the record management section 

(completed by grantee staff). 

TABLE 1—ESTIMATES OF ANNUALIZED HOUR BURDEN

Type of respondent activity Number of 

respondents 

Responses 

per 

respondent 

Total 

responses 

Hours per 

response 

Total burden 

hours 

Quarterly Progress Report ................................................... 197 4 732 4 2,928 

Adult questionnaire .............................................................. 10,000 2 20,000 .20 4,000 

Youth questionnaire ............................................................. 2,500 2 5,000 .20 1000 

Total .............................................................................. 12,697 ........................ 25,732 ........................ 7,928 

Send comments to Carlos Graham, 

SAMHSA Reports Clearance Officer, 

5600 Fishers Lane, Room 15E57–A, 

Rockville, Maryland 20857, OR email a 

copy to carlos.graham@samhsa.hhs.gov. 

Written comments should be received 

by October 12, 2021. 

Carlos Graham, 

Clearance Officer. 

[FR Doc. 2021–17290 Filed 8–12–21; 8:45 am] 

BILLING CODE 4162–20–P 

DEPARTMENT OF HOUSING AND 

URBAN DEVELOPMENT 

[Docket No. FR–6278–N–01] 

HUD Program Evaluation Policy— 

Policy Statement 

AGENCY: Office of Policy Development 

and Research, HUD. 

ACTION: Notice. 

SUMMARY: This policy statement 

articulates the core principles and 

practices of the Department of Housing 

and Urban Development’s evaluation 

and research activities. This policy 

reaffirms HUD’s commitment to 

conducting rigorous, relevant 

evaluations and to using evidence from 

evaluations to inform policy and 

practice. 

DATES: August 13, 2021. 

FOR FURTHER INFORMATION CONTACT: For 

information about this notice, contact 

Todd M. Richardson, Evaluation Officer, 

Office of Policy Development and 

Research, U.S. Department of Housing 

and Urban Development, 451 7th Street 

SW, Washington, DC 20410, telephone 

(202) 402–5922. The listed telephone 

number is not a toll-free number. 

Persons with hearing- or speechimpairments may access this number 

through TTY by calling Federal Relay 

Service at 1–800–877–8339 (this is a 

toll-free number). 

SUPPLEMENTARY INFORMATION: 

I. Background 

Evaluation activity occurs in several 

offices at HUD, but the special mission 

of HUD’s Office of Policy Development 

and Research (PD&R) is to inform HUD 

policy development and 

implementation to improve life in 

American communities through 

conducting, supporting, and sharing 

research, surveys, demonstrations, 

program evaluations, and best practices. 

Within HUD, PD&R is responsible for 

most, but not all, program evaluations. 

The office provides reliable and 

objective data and analysis to help 

inform policy decisions. 

In July 2016, GAO issued a report 

entitled ‘‘Department of Housing and 

Urban Development: Actions Needed to 

Incorporate Key Practices into 

Management Functions and Program 

Oversight,’’ (GAO 16–497) in which 

GAO presented a broad assessment of 

HUD’s management of its operations 

and programs.1 In the report, GAO 

examined HUD efforts to: (1) Meet 

Federal requirements and implement 

key practices for management functions, 

including performance planning and 

reporting and human capital, financial, 

acquisition, and information technology 

(IT) management; and (2) oversee and 

evaluate programs. 

PD&R is the primary office within 

HUD responsible for data analysis, 

research, program evaluations, and 

policy studies that inform the 

development and implementation of 

programs and policies across HUD 

offices. PD&R undertakes program 

evaluations, often by using a process 

that includes convening expert panels. 

However, GAO found that PD&R had 

neither developed agency-wide, written 

policies for its program evaluations, nor 

documented the criteria used to select 

the expert panels and review the quality 

of program evaluations. 

On December 6, 2016 (81 FR 87949), 

HUD issued a policy statement in the 

Federal Register 2 responding to the 

GAO report by setting out the core 

principles and practices of PD&R’s 

evaluation and research activities. This 

statement incorporated some language 

from a policy statement by the Office of 

Policy, Research, and Evaluation of the 

Administration for Children and 

Families of the U.S. Department of 

Health and Human Services. 

On January 14, 2019, the Foundations 

for Evidence-Based Policymaking Act of 

2018 (‘‘Evidence Act’’), Public Law 115– 

435, was enacted.3 Section 101 of the 

Evidence Act created 5 U.S.C. 311–315 

and mandated that the head of each 

agency appoint an Evaluation Officer, 

including at HUD. This officer must 

establish common standards for all HUD 

evaluations, whether performed by 

PD&R or another office. This issuance 

articulates department-wide evaluation 

standards and states other new 

principles based on PD&R’s experience 

since the November 2016 publication. 

II. HUD Program Evaluation Policy 

Section 101 of the Evidence Act 

defines ‘‘evaluation’’ to mean ‘‘an 

assessment using systematic data 

collection and analysis of one or more 

programs, policies, and organizations 

intended to assess their effectiveness 

and efficiency.’’ 

HUD has identified the following core 

principles and practices as fundamental 

to ensuring high-quality and consistent 

evaluation results: Rigor, relevance, 

transparency, independence, ethics, and 

technical innovation. This policy 

applies to all HUD-sponsored 


jbell on DSKJLSW7X2PROD with NOTICES
Federal Register / Vol. 86, No. 154 / Friday, August 13, 2021 / Notices 44739 

evaluations and regulatory impact 

analyses; they also apply to the 

selection of projects, contractors, and 

HUD staff that are involved in 

evaluations. 

Rigor 

HUD is committed to using the most 

rigorous methods that are appropriate to 

the evaluation questions and feasible 

within budget and other constraints. 

The need for rigor is not restricted to 

impact evaluations; rigor is also 

necessary in implementation or process 

evaluations, descriptive studies, 

outcome evaluations, and formative 

evaluations; in both qualitative and 

quantitative approaches. Rigor requires 

ensuring that inferences about cause 

and effect are well founded (internal 

validity); requires clarity about the 

populations, settings, or circumstances 

to which results can be generalized 

(external validity); requires that 

researchers seek to understand and 

correct for implicit bias in the 

formulation of research questions and 

methods; and requires the use of 

measures that accurately capture the 

intended information (measurement 

reliability and validity). Implicit biases 

are discriminatory biases that reflect 

unidentified traces of past experience, 

including implicit attitudes that affect 

feelings, thoughts or actions, and 

implicit stereotypes that affect how 

others are characterized. Survey 

instruments are pre-tested with 

members of the population to be studied 

to increase measurement validity. When 

statistically appropriate, and 

particularly if the number of hypotheses 

being tested is large, HUD will require 

the use of commonly accepted 

adjustments to classical statistical 

testing to reduce the probability that 

random outliers are presented as 

meaningful. 

In assessing the effects of programs or 

services, HUD evaluations use methods 

that isolate to the greatest extent 

possible the impacts of the programs or 

services from other influences, such as 

trends over time, geographic variation, 

or pre-existing differences between 

participants and non-participants. 

Where feasible, research should employ 

a treatment group and a counterfactual. 

A treatment group is a population that 

has received an intervention. A 

counterfactual is a population that did 

not receive the intervention and can 

credibly represent what would have 

happened to the treatment group in the 

absence of the intervention according to 

the above standard for rigor. For such 

causal questions, experimental 

approaches are preferred. When 

experimental approaches are not 

feasible, HUD uses the most rigorous 

approach that is feasible. 

In both quantitative and qualitative 

research, rigor means having clear 

research questions and an explicit 

analytic framework; justification for 

case selection and sampling methods in 

relation to research goals; and 

transparent, verifiable methods of 

systematic data collection and analysis, 

auditable records, and attention to 

possible alternative interpretations 

during analysis and writing. 

HUD ensures that contractors and 

grantees conducting evaluations have 

appropriate expertise through 

emphasizing the requirement for rigor in 

requests for proposals and funding 

opportunity announcements, noting that 

applicants’ capacity for rigor will be 

evaluated in the selection process. In 

addition, HUD will judge research teams 

with equal capacity for rigor to be more 

qualified if the team includes 

researchers demographically similar to 

or knowledgeable about the perspectives 

and lived experiences of the 

populations studied. 

HUD employs a strategic human 

capital development plan to hire, train, 

and retain a workforce that ensures staff 

have the tools and resources to 

accomplish the mission. 

Relevance 

The HUD evaluation agenda reflects 

the legislative requirements and policy 

issues related to HUD’s mission. HUD 

solicits input from stakeholders, both 

internal and external, including 

stakeholders with lived experience, 

such as program participants, and 

grantees, on the selection of programs to 

be evaluated, initiatives, 

demonstrations, and research questions. 

For new initiatives and demonstrations 

in particular, evaluations will be more 

feasible and useful when planned in 

advance, in concert with the 

development of the initiative or 

demonstration, rather than as an 

afterthought. HUD strives to understand 

the relevance of its completed research 

through concerted stakeholder 

engagement, including with people and 

grantees affected by HUD programs, to 

continuously improve its research 

agenda. 

Expert panels include research and 

other subject matter experts and are 

diverse in ways tailored to the study, 

including racial and ethnic diversity 

and representatives of the studied 

populations. 

HUD strives to design program 

evaluations and other analyses to better 

understand structural racism and to 

reveal unequal benefits and harms 

across social groups as relevant, with 

special attention to race, national origin, 

color, familial status, religion, disability, 

age, and sex (including gender identity 

and sexual orientation). 

To support this goal, insofar as 

feasible, HUD collects and reports data 

on race, ethnicity, gender, and income, 

and other characteristics of 

underrepresented and underserved 

communities relevant for research and 

analysis efforts. 

HUD also encourages research to 

engage studied populations. 

‘‘Engagement’’ means the deliberate and 

intentional inclusion of the thoughts 

and perspectives of studied groups, 

such as program participants, grantees, 

and underrepresented and underserved 

populations. This includes collecting 

people’s thoughts and perspectives 

through standard (valid and rigorous) 

research methods such as surveys, focus 

groups, in-depth interviews, or 

ethnography to produce published 

research that conveys studied 

populations’ thoughts and perspectives. 

In another type of engagement, people 

influence the research that is about 

them. This includes a wide range of 

activities that lie on a continuum from 

simple input (which should be 

documented and published) to full cocreation of any aspect of the research, 

from topic selection to research design, 

data collection, data analysis, 

interpretation, writing, or 

dissemination, or even being on the 

research team. An example of simple 

input is to include in the final report a 

summary of comments on the research 

by members of studied groups. 

Examples of collaboration include, but 

are not limited to, co-creating the list of 

topics to be covered in a survey or 

having members of a studied group on 

the research team. 

To raise awareness of and spur 

creative approaches to engagement of 

studied populations in program 

evaluations and HUD-sponsored 

research, HUD may require contractors 

and grantees to explain how their 

research will and will not engage 

studied populations. HUD recognizes 

that engagement must be tailored to 

particular research efforts. 

HUD retains the right to determine 

research methods. 

HUD disseminates findings in ways 

that are accessible and useful to 

policymakers, practitioners, and 

members of communities affected by 

HUD programs and policies. Published 

findings will be accessible to 

individuals with disabilities pursuant to 

Section 508 of the Rehabilitation Act. 

PD&R partners with other HUD program 

offices to inform internal and external 

stakeholders through disseminating 


jbell on DSKJLSW7X2PROD with NOTICES
44740 Federal Register / Vol. 86, No. 154 / Friday, August 13, 2021 / Notices 

4https://www.huduser.gov/portal/about/pdr_

learningagenda.html. 

5https://www.huduser.gov/portal/about/PDRResearch.html. 6https://www.ecfr.gov/cgi-bin/ 

retrieveECFR?n=pt24.1.60. 

7See https://www.archives.gov/files/federalregister/executive-orders/pdf/12866.pdf for the 

original order, and see https://www.archives.gov/ 

federal-register/executive-orders/1993-clinton.html 

for citations of, and links to, other executive orders 

that amended or supplemented this order. 

evidence from HUD-sponsored 

evaluations. 

Transparency 

HUD will release methodologically 

valid evaluations without regard to the 

findings. Evaluation reports must 

describe the methods used, including 

strengths and weaknesses, and discuss 

the generalizability of the findings. 

Evaluation reports must present 

comprehensive results, including 

favorable, unfavorable, and null 

findings. HUD will publish interim 

findings, as projected in the initial 

research design. If there are indications 

that the findings of the final report may 

differ, HUD will provide appropriate 

qualifications accompanying the 

publication to guard against 

misunderstanding or misuse of the 

interim findings. If there are interim 

findings, HUD will publish those 

findings even if there are indications 

that the findings of the final report may 

differ. When findings are highly 

relevant to current policy, HUD 

evaluations carry a foreword articulating 

the policy position of the Department 

with respect to those findings. 

If the findings of a HUD evaluation 

will have broad public interest and 

includes a counterfactual, PD&R will 

publish a synopsis of the research 

design, data collection and analysis plan 

soon after it is approved and will 

require interim and final reports that 

deviate from that document to explain 

how they deviate and why. 

HUD publishes a 5-year Learning 

Agenda 4 that outlines the research and 

evaluation that it believes would be of 

greatest value to public policy. PD&R 

lists all ongoing evaluation projects at 

the HUDUSER.gov website 5 and 

updates it quarterly. PD&R will release 

evaluation results timely, usually within 

4 months of receiving the final report. 

HUD will, where possible, archive 

administrative and evaluation data for 

secondary use by interested researchers. 

HUD typically builds requirements into 

contracts to prepare data sets for 

secondary use. Access for external 

researchers may be provided directly 

through data licenses or indirectly 

through inter-agency agreements. This 

policy may not apply for data that has 

obvious commercial value, such as 

mortgage performance data. HUD staff 

may publish the results of their 

scholarship and analysis in any forum, 

so long as they do not claim to speak for 

the Department. 

HUD evaluation contracts will 

generally permit contractors to publish 

their findings within 6 months of the 

termination of the contract if HUD has 

not already published them. 

Independence 

Independence and objectivity are core 

principles of evaluation. Agency and 

program leadership, program staff, 

service providers, and others participate 

actively in setting evaluation priorities, 

identifying evaluation questions, and 

assessing the implications of findings. 

However, it is important to insulate 

evaluation functions from undue 

influence and from both the appearance 

and the reality of bias. To promote 

objectivity, HUD protects independence 

in the design, conduct, and analysis of 

evaluations. To this end: 

• HUD conducts evaluations through 

the competitive award of grants and 

contracts to external experts who are 

free from conflicts of interest. 

• HUD also conducts evaluations inhouse and supports unsolicited external 

evaluation proposals with funding, data, 

or both. 

• The Evaluation Officer will consult 

with the HUD office with lead 

responsibility on the design of 

evaluation projects and analysis plans 

and will advise that office on whether 

to publish evaluation reports. 

Ethics 

HUD-sponsored evaluations must be 

conducted in an ethical manner and 

safeguard the dignity, rights, safety, and 

privacy of participants. HUD-sponsored 

evaluations must comply with both the 

spirit and the letter of relevant 

requirements such as regulations 

governing research involving human 

subjects. In particular, PD&R protects 

the privacy of HUD-assisted households 

and HUD-insured borrowers through its 

Rule of Eleven; that is, PD&R allows no 

disclosure of information about the 

characteristics of any group of 

individuals or households numbering 

fewer than eleven by PD&R staff, 

contractors, grantees, or licensees. 

HUD is a signatory to the Federal 

Policy for the Protection of Human 

Subjects, generally known as the 

‘‘Common Rule.’’ 24 CFR part 60, which 

includes its own requirements for 

ensuring adequate provisions to protect 

the privacy of human subjects research.6

HUD does not tolerate plagiarism, or 

fabrication or deliberate 

mischaracterization of data by staff, 

contractors or grantees who are engaged 

in evaluation activity. 

Technical Innovation 

PD&R supports and employs new 

methods of data collection and analysis 

that more reliably and efficiently answer 

research questions than old methods do. 

Application of These Principles to 

Economic Analysis of Regulations 

Economic analysis of both existing 

and proposed regulations, properly 

conducted, is a critical tool in 

improving public policy. Economists at 

HUD rely on the insights, data, and 

empirical estimates from rigorous 

program evaluations when predicting 

the economic impact of an incremental 

change to the program. In any HUD 

Regulatory Impact Analysis: 

• HUD analyzes whether the issues 

addressed by the regulation stem from a 

market failure, government failure, or 

other systemic problem, and whether 

the regulation addresses the root causes 

of those problems. 

• HUD uses and as necessary 

produces the best objective estimates of 

the benefits, costs, and transfers 

resulting from the regulation, taking into 

account gaps and uncertainties in the 

available data and methodologies. 

• HUD assesses the economic 

benefits, costs, and transfers of proposed 

regulatory actions as required by 

Executive Order 12866.7 HUD provides 

additional analysis of impacts across 

groups defined by race, ethnicity, and 

other characteristics that may define 

underrepresented and underserved 

groups when such analyses are relevant 

and feasible. 

• Where clear alternatives to the 

regulatory actions exist, HUD 

objectively estimates the benefits, costs, 

and transfers of those alternatives, and 

additional analysis of impacts of those 

alternatives across underrepresented or 

underserved groups as well. 

Todd M. Richardson, 

Evaluation Officer. 

[FR Doc. 2021–17339 Filed 8–12–21; 8:45 am] 

BILLING CODE 4210–67–P 


jbell on DSKJLSW7X2PROD with NOTICES