26290 Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices 

the Act Against AIDS National Testing’’. 

The information collected from these 

data collections was used to evaluate a 

specific AAA campaign phase. We are 

requesting the same amount of time to 

continue surveying AAA target 

audiences as new phases are developed. 

Through this extension, we plan to 

reach the remaining approved 6,445 

respondents. To obtain the remaining 

respondents, we anticipate screening 

approximately 32,220 individuals. 

Depending on the target audience for 

the campaign phase, the study screener 

will vary. The study screener may 

address one or more of the following 

items: Race/ethnicity, sexual behavior, 

sexual orientation, gender identity, HIV 

testing history, HIV status, and injection 

drug use. Each survey will have a core 

set of items asked in all rounds, as well 

as a module of questions relating to 

specific AAA phases and activities. 

Respondents will be recruited through 

national opt-in email lists, the internet, 

and external partnerships with 

community-based and membership 

organizations that work with or 

represent individuals from targeted 

populations (e.g., National Urban 

League, the National Medical 

Association). Respondents will selfadminister the survey at home on 

personal computers. There is no cost to 

the respondents other than their time. 

ESTIMATED ANNUALIZED BURDEN HOURS

Type of respondents Form name Number of 

respondents 

Number of 

responses per 

respondent 

Average 

burden per 

response 

(in hours) 

Total 

burden 

(in hours) 

Individuals (male and female) aged 18 years and 

older.

Study Screener .............

Survey ..........................

10,740 

2,148 

1 

1 

2/60 

30/60 

358 

1,074 

Total ............................................................... ....................................... ........................ ........................ ........................ 1,432 

Jeffrey M. Zirger, 

Acting Chief, Information Collection Review 

Office, Office of Scientific Integrity, Office 

of the Associate Director for Science, Office 

of the Director, Centers for Disease Control 

and Prevention. 

[FR Doc. 2018–12082 Filed 6–5–18; 8:45 am] 

BILLING CODE 4163–18–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Administration for Children and 

Families 

Criteria for Evidence of Effectiveness 

To Be Applied to Projects Identified for 

Inclusion in the What Works 

Clearinghouse of Proven and 

Promising Projects To Move Welfare 

Recipients Into Work 

AGENCY: Administration for Children 

and Families (ACF), U.S. Department of 

Health and Human Services (HHS). 

ACTION: Request for public comment. 

SUMMARY: The Administration for 

Children and Families, HHS, solicits 

comments by August 5, 2018 on the 

criteria for evidence of effectiveness for 

the What Works Clearinghouse of 

Proven and Promising Projects to Move 

Welfare Recipients into Work. Final 

criteria for evidence of effectiveness will 

be used to develop the clearinghouse. 

SUPPLEMENTARY INFORMATION: 

Invitation to Comment: HHS invites 

comments regarding this notice on the 

proposed criteria for HHS’s systematic 

review of the evidence. To ensure that 

your comments are clearly stated, please 

identify the specific criterion or other 

section of this notice that your 

comments address. 

1.0 Background 

1.1 Legislative Context 

The Consolidated Appropriations Act 

of 2017 (Pub. L. 115–31 (https://

www.congress.gov/115/plaws/publ31/ 

PLAW-115publ31.pdf)) directs the U.S. 

Department of Health and Human 

Services (HHS) to create a database of 

projects that have used a proven or 

promising approach to move welfare 

recipients into work, based on 

independent, rigorous evaluations of the 

projects, and to create a What Works 

Clearinghouse of Proven and Promising 

Projects to Move Welfare Recipients into 

Work. As stated in the statute, the 

database shall additionally ‘‘include a 

separate listing of projects that used a 

developmental approach in delivering 

services and a further separate listing of 

the projects with no or negative effects.’’ 

The statute requires HHS to establish 

criteria for evidence of effectiveness. 

1.2 The Legislation’s Direction for 

Establishing the Criteria for Evidence of 

Effectiveness 

Section 413(g)(2) of Public Law 115– 

31 charges the Secretary of Health and 

Human Services with establishing the 

criteria of effectiveness. The statute 

further stipulated that the (B) process 

for establishing the criteria— 

(i) is transparent; 

(ii) is consistent across agencies; 

(iii) provides opportunity for public 

comment; and 

(iv) takes into account efforts of 

Federal agencies to identify and 

publicize effective interventions, 

including efforts at the Department of 

Health and Human Services, the 

Department of Education, and the 

Department of Justice. 

1.3 The Employment Strategies for LowIncome Adults Evidence Review 

Prior to the enactment of Public Law 

115–31, the Office of Planning, 

Research, and Evaluation (OPRE) at the 

Administration for Children and 

Families (ACF) at HHS had developed 

the Employment Strategies for LowIncome Adults Evidence Review (ESER). 

The new statute aligns with and extends 

the work of ESER. HHS proposes 

building on this existing work to 

develop the new Clearinghouse. 

The Employment Strategies for LowIncome Adults Evidence Review (ESER) 

is a systematic review of the evaluation 

research published between 1990 and 

2014 on employment and training 

programs for low-income adults. It 

culminated in a searchable, public 

database (https://

employmentstrategies.acf.hhs.gov/). The 

review was supplemented with briefs 

synthesizing the results of the review 

and highlighting strategies that 

appeared to be promising, as identified 

by the review. To identify the programs 

and strategies—or interventions— that 

appear to be most effective in helping 

low-income adults gain and retain 

employment, ESER systematically 

identified, assessed, and synthesized 

evidence from the existing evaluation 

research literature. A core component of 

ESER’s review, as with other federal 

evidence reviews, involved assessing 

the quality of the research evidence on 

different interventions. 

To assess the quality of the evidence, 

ESER reviewed each study’s methods to 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices 26291 

determine if they were rigorous enough 

to ensure that the study’s findings could 

be considered reliable. ESER assessed 

whether the study’s methods reliably 

supported the conclusion that an 

intervention’s impacts were caused by 

the intervention and not by something 

else. The standards for assessing 

studies’ methods were defined based on 

consultation with federal experts on 

evidence reviews and researchers with 

expertise in evaluation methodology. To 

differentiate among different levels of 

the strength of evidence, ESER assigned 

a High, Moderate, or Low rating to each 

study reviewed. 246 of the 314 studies 

included in the review earned a High 

rating and 1 study earned a Moderate 

rating. The remaining 67 studies 

received a Low rating. 

Through this review, ESER was able 

to identify interventions whose findings 

could be considered most reliable. 

Studies’ ratings reflect the rigor of their 

study methods, independent of whether 

the findings were positive or negative. 

As a result, a study could be rated High 

or Moderate even if the intervention 

studied did not improve the outcomes 

for low-income adults. While the vast 

majority of studies included in ESER 

achieved a High rating (and, therefore, 

are considered to provide reliable, or 

strong, evidence), the review also found 

that, overall, null impacts were more 

prevalent than statistically significant 

impacts. 

While ESER did not assess the 

effectiveness of the interventions 

reviewed, ESER conducted a number of 

preliminary steps necessary for 

assessing effectiveness. This included 

categorizing each study’s findings 

according to whether it found positive, 

negative or null impacts for the 

interventions studied. In addition, 

through a number of synthesis briefs 

(published on the website), ESER 

qualitatively and quantitatively 

summarized the direction of impacts for 

different interventions and highlighted 

interventions associated with the 

greatest number of positive impacts. 

To be included in ESER, studies had 

to— 

• Quantitatively measure the 

effectiveness of a program or strategy 

• Be published between 1990 and 

2014 

• Study an employment program or 

strategy— an intervention— that 

Æ had a primary aim of improving 

employment-related outcomes 

Æ primarily targeted low-income 

adults 

Æ took place in the United States, 

Canada, or the United Kingdom 

To identify studies eligible for review, 

ESER issued a call for papers, 

conducted literature searches, and 

consulted with experts in workforce 

development programs that serve lowincome adults. 

ESER looked at the effects of the 

interventions on four domains, or 

outcome areas: 

• Employment 

• Earnings 

• Public benefit receipt 

• Education/training 

Outcomes were examined for short 

and longer-term impacts (longer-term 

was measured as being more than 18 

months after the intervention was 

implemented). 

The ESER website (https://

employmentstrategies.acf.hhs.gov/) 

reports key results for all eligible 

studies. The website also allows users to 

search for results by program studied, 

target population, outcome(s) of interest, 

service strategies, intervention setting, 

year of study publication, and whether 

favorable impacts were found. 

While ESER’s overall population of 

interest was low-income adults, a 

majority of the studies in ESER 

examined welfare populations. Because 

studies of interventions in a welfare 

setting typically include both recipients 

and applicants, ESER does not include 

any studies that solely focused on 

welfare recipients. ESER does, however, 

include interventions targeted to lowincome populations understood to share 

important characteristics with welfare 

recipients, such as other public benefit 

recipients, and those considered hard to 

employ, including those who have been 

homeless or formerly incarcerated. 

2.0 Process for Establishing the Criteria 

of Effectiveness for the New What 

Works Clearinghouse 

In fall 2017 and early winter 2018, 

OPRE engaged in a series of systematic 

consultations with federal and nonfederal technical experts on evidence 

reviews. In addition to representation 

from the Department of Labor (DOL) and 

the Department of Education (ED) in 

these consultations, federal 

representation included the Department 

of Justice (DOJ) and a number of HHS 

agencies/offices including the Office of 

Family Assistance (OFA), the Office of 

Planning, Research, and Evaluation 

(OPRE), the Office of the Assistant 

Secretary for Planning and Evaluation 

(ASPE) and the Agency for Healthcare 

Research and Quality (AHRQ). 

The objective of these consultations 

was to help HHS: 

(1) Develop criteria for categorizing 

interventions in the new Clearinghouse 

as proven, promising, developmental, or 

ineffective, 

(2) develop these criteria through a 

process that 

a. involved consultation with the 

Department of Labor (DOL), the 

Department of Education (ED), and 

other entities with experience 

evaluating relevant effectiveness 

research, 

b. allowed HHS to better understand 

other Federal evidence reviews’ 

standards and processes and determine 

where it would make sense for the new 

Clearinghouse to be consistent with 

these standards and processes, and 

(3) learn best practices from other 

Federal evidence reviews for identifying 

and publicizing effective interventions 

2.1 Transparent 

To ensure that the Clearinghouse’s 

procedures and standards, including the 

criteria for evidence of effectiveness, are 

transparent, HHS intends to implement 

the following practices: 

• Post the procedures and standards 

and information about the process on 

the Clearinghouse website. 

• Provide the public a means of 

contacting the Clearinghouse, for 

example, by establishing a help desk to 

respond to email inquiries. 

2.2 Consistent Across Agencies 

To ensure that the Clearinghouse is as 

consistent as possible with other federal 

evidence reviews in its processes and 

standards, HHS intends to implement 

the following practices: 

• Adopt the standards and methods 

for reviewing studies from OPRE’s 

existing Employment Strategies 

Evidence Review (ESER) (https://

www.acf.hhs.gov/opre/resource/ 

employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods), which are broadly consistent 

with other federal Clearinghouses. 

ESER’s standards and methods (e.g., 

author queries; number and training of 

reviewers; choices about reporting effect 

sizes) were developed by considering 

both the choices made by other federal 

and non-federal Clearinghouses and the 

standards of research in the 

employment and training intervention 

field. Other existing federal 

Clearinghouses have followed this same 

approach (considering both the choices 

made by other clearinghouses and the 

norms of research within their fields of 

focus). 

• In any instances where the new 

Clearinghouse’s ratings of a project’s 

strength of evidence or effectiveness 

differ from another federal evidence 

review that rates projects according to 

the same outcomes (such as the 

Department of Labor’s (DOL’s) 

Clearinghouse for Labor Evaluation and 


daltland on DSKBBV9HB2PROD with NOTICES
26292 Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices 

Research (CLEAR)), annotate the 

findings to explain the reason for the 

difference. 

2.3 Provides Opportunity for Public 

Comment 

To provide an opportunity for public 

comment on the criteria for 

effectiveness, ACF is publishing this 

Federal Register Notice. 

2.4 Takes Into Account Efforts of 

Federal Agencies To Identify and 

Publicize Effective Interventions 

To ensure the Clearinghouse reflects 

the learning of other Federal agencies 

about how to identify and publicize 

effective interventions, HHS intends to 

implement the following practices: 

• Use some of the methods adopted 

by other clearinghouses to create 

multiple products tailored to different 

audiences and use graphic design and 

other user-friendly dissemination 

elements to help users digest evidence 

quickly. 

• Include information on the 

Clearinghouse website that is especially 

useful to practitioners, such as summary 

information about projects and 

approaches. 

• Develop and incorporate alternative 

media for the Clearinghouse such as 

videos that will tailor communication to 

various groups. 

• Ensure that information is 

effectively conveyed on the 

Clearinghouse website by soliciting 

feedback from various stakeholders who 

can represent key target audiences. Key 

among these would be state or county 

Temporary Assistance for Needy 

Families (TANF) and Workforce 

Development practitioners, as well as 

evaluation researchers. 

3.0 Proposed Criteria for Evidence of 

Effectiveness 

3.1 Criteria for Well-Designed, 

Rigorous Impact Research 

HHS intends to employ the criteria 

established by OPRE’s Employment 

Strategies for Low-Income Adults 

Evidence Review (ESER) to assess the 

quality of study design and to assess the 

strength of the evidence resulting from 

studies. These criteria (referred to as 

‘‘standards and methods’’) are available 

in ESER’s Standards and Methods report 

https://www.acf.hhs.gov/opre/resource/ 

employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods. 

3.2 Proposed Criteria for Evidence of 

Effectiveness for Projects Included in the 

Clearinghouse 

3.2.1 Definition of Project and 

Approach 

The legislation requires that ratings, 

or categorizations, of evidence of 

effectiveness be applied to projects and 

approaches. To standardize definitions 

for these terms, HHS intends to define 

a project and an approach as follows: 

• Define project to be a specific 

bundle of services and/or policies 

implemented in a given context. 

• Project will be the unit that receives 

an effectiveness rating (i.e. proven, 

promising, developmental, or 

ineffective). 

• Define approach to be the guiding 

framework of specific services (e.g., 

career pathways). 

• Approaches will not be rated as 

proven, promising, developmental, or 

ineffective, but the Clearinghouse will 

include narrative summaries related to 

different approaches. 

• While the legislation does not 

require HHS to define or evaluate the 

effectiveness of program components, 

there is interest in the field in 

examining program components. Thus, 

HHS intends that the Clearinghouse 

include meta-analyses of specific 

components of projects (such as ‘‘case 

management’’ or ‘‘job search 

assistance’’) whenever appropriate and 

feasible. 

3.2.2 Parameters Guiding the 

Application of Evidence of Effectiveness 

Ratings 

Before a project can be categorized as 

being proven, promising, 

developmental, or ineffective, a number 

of preliminary definitions, or 

parameters, must be established to guide 

decision making. These include the 

outcomes for which a project’s 

effectiveness will be evaluated, how a 

favorable or unfavorable effect will be 

measured, and how an effectiveness 

rating will be applied to a project. 

3.2.2.1 Outcomes 

HHS intends that the new 

Clearinghouse will review the following 

outcomes: 

Æ Employment (short and longerterm), 

Æ earnings (short and longer-term), 

Æ educational attainment, and 

Æ public benefit receipt. 

3.2.2.2 Definition of Favorable and 

Unfavorable Effects 

HHS intends that the Clearinghouse 

consider only statistically significant 

findings (p <.05) as evidence of 

favorable or unfavorable effects. 

3.2.2.3 Pre-Defining Criteria for 

Selecting Among Multiple Outcome 

Measures 

HHS intends to reduce the likelihood 

for reporting a false positive rate for 

outcomes—an issue that can occur 

when studies use multiple measures or 

multiple outcomes to assess impacts in 

the same domain (e.g., short-term 

earnings)—by relying on the decision 

rules ESER developed to address the 

potential for multiple comparisons. 

https://www.acf.hhs.gov/opre/resource/ 

employment-strategies-for-low-incomeadults-evidence-review-standards-andmethods). 

3.2.2.4 Application of Evidence of 

Effectiveness Ratings 

HHS intends that evidence of 

effectiveness ratings will be applied 

within outcome domains; each project 

will receive ratings of effectiveness on 

each outcome domain (e.g., a project 

may be found promising for short-term 

employment but ineffective for longterm employment). There will be no 

overall rating for the project. 

3.2.3 Definition of Proven 

The legislation directs HHS to 

categorize projects as Proven, 

Promising, Developmental, or 

Ineffective. 

HHS intends that for a project to be 

considered proven, the following 

conditions must be met: 

• There must be at least two separate 

studies of the same project that meet 

evidence standards and meet criteria for 

a promising rating. 

Æ Studies are considered to be 

separate studies of the same project if 

they use non-overlapping samples to 

examine distinct implementations of the 

project. 

• There must be only favorable or 

null impacts within a given outcome 

domain. Thus, no studies that meet 

evidence standards for a given outcome 

domain can show an unfavorable impact 

within that domain. 

• Projects that have both favorable 

and unfavorable impacts in a given 

domain will be categorized as mixed. 

• A project has a limited number, or 

proportion, of null findings in a given 

domain. 

HHS is soliciting comments on how to 

best determine the ceiling for the 

number, or proportion, of null to 

positive findings in a given domain. 

If subsequent studies or replications 

result in only null findings in a given 

domain, the review will establish 

procedures for revisiting a project’s 

rating of proven. 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 109 / Wednesday, June 6, 2018 / Notices 26293 

3.2.4 Definition of Promising 

HHS intends that for a project to be 

considered promising, the following 

conditions must be met: 

• One study of a project must meet 

evidence standards. 

• That study must find only favorable 

or null impacts within a given outcome 

domain. Thus no studies that meet 

evidence standards for an outcome 

domain can show an unfavorable impact 

within the domain. 

Æ If the review examines more than 

one measure to identify impacts on a 

particular domain (e.g., Unemployment 

Insurance data and participant survey 

data), as long as one measure (among 

those selected according to 3.2.2.3 

above) finds favorable impacts for that 

outcome, the intervention can receive a 

Promising rating for that outcome. 

• Projects that have both favorable 

and unfavorable impacts in a given 

domain will be categorized as mixed. 

3.2.5 Definition of Ineffective 

HHS intends that for a project to be 

considered ineffective, the following 

conditions must be met: 

• One or more studies of a project 

must meet evidence standards. 

• There must be only findings of 

unfavorable or null effects in a given 

domain. 

• For studies finding null effect in a 

given domain, the review will include a 

measure of statistical precision—so that 

small, under-powered studies do not 

drive the effectiveness rating. If an 

intervention has been evaluated using 

only small studies, a lack of detectable 

effects could reflect either 

ineffectiveness of the intervention or the 

lack of statistical power to detect effects. 

It would be misleading to characterize 

this latter scenario as an ineffective 

project. 

3.2.6 Definition of Developmental 

HHS intends that for a project to be 

considered developmental, the 

following conditions must be met: 

• There must be at least one current, 

ongoing evaluation of the project that 

uses a study design that meets evidence 

standards but has not yet produced 

impact findings. 

3.2.7 Additional Category of Mixed 

and Definition of Mixed 

HHS intends that there be an 

additional category for categorizing 

evidence of effectiveness called mixed. 

HHS proposes that for a project to be 

considered mixed, the following 

conditions must be met: 

• One or more studies of a project 

must meet evidence standards. 

• The studies find both favorable and 

unfavorable impact estimates within the 

same domain. 

3.2.8 HHS intends that narrative 

descriptions of rated projects, narrative 

descriptions of approaches, and 

information on case studies be provided 

to users of the Clearinghouse to 

facilitate a fuller understanding of the 

field of welfare-to-work interventions. 

4.0 Submission of Comments 

Comments may be submitted until 

August 5, 2018 by email to 

OPREinfocollection@acf.hhs.gov. 

Naomi Goldstein, 

Deputy Assistant Secretary for Planning, 

Research, and Evaluation. 

[FR Doc. 2018–12160 Filed 6–5–18; 8:45 am] 

BILLING CODE 4184–09–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Health Resources and Services 

Administration 

Notice of Single Source Award Based 

on Non-Statutory Earmark to the Delta 

Region Community Health Systems 

Development Program 

AGENCY: Health Resources and Services 

Administration (HRSA), Department of 

Health and Human Services. 

ACTION: Notice. 

SUMMARY: The purpose of the Delta 

Region Community Health Systems 

Development Program is to support 

collaboration with and input from the 

Delta Regional Authority to develop a 

pilot program to help underserved rural 

communities in the Delta region identify 

and better address their health care 

needs and to help small rural hospitals 

improve their financial and operational 

performance. HRSA received an 

additional $2,000,000 in FY 2018 to 

support the Delta Region Community 

Health Systems Development Program, 

increasing the total FY 2018 resources 

from $2,000,000 to $4,000,000. The 

single award recipient, the Rural Health 

Resource Center has a need for 

additional funds to support activities 

performed within the scope of this 

program. The center will use a 

multipronged approach to deliver 

phased-in technical assistance (TA) to 

all eight Delta Region communities. 

ADDRESSES: Further information on the 

Delta Region Community Health 

Systems Development Program is 

available at: https://www.hrsa.gov/ 

ruralhealth/programopportunities/ 

fundingopportunities/?id=8d869eff0bca-4703-a821-88a9f0433b73. 

FOR FURTHER INFORMATION CONTACT: 

Rachel Moscato, Program Coordinator, 

Delta Region Community Health 

Systems Development, Federal Office of 

Rural Health Policy, HRSA, RMoscato@

hrsa.gov. 

Background 

The Delta Region Community Health 

Systems Development program is 

authorized by Section 711(b) of the 

Social Security Act, (42 U.S.C. 912 (b)), 

as amended. 

HRSA established the Delta Region 

Community Health Systems 

Development Program in FY 2017, 

under announcement HRSA–17–117, 

providing up to $2,000,000 per year to 

one awardee, the Rural Health Resource 

Center for a three-year project period: 

September 30, 2017 through September 

29, 2020. The FY 2018 House Report 

115–244 and Senate Report 115–150 

Division H of the Consolidated 

Appropriations Act of 2018 (Pub. L. 

115–141) provided direction that an 

additional $2,000,000 included in the 

appropriation to be used to support the 

Delta Program. HRSA plans to increase 

the maximum funding per year for the 

Delta Region Community Health 

Systems Development Program to 

$4,000,000 for one award recipient in 

FY 2018, as well as in subsequent 

budget periods within the three-year 

project period, should funds become 

available. 

Conclusion 

HRSA will provide $2,000,000 in 

additional resources to the current 

award recipient, the Rural Health 

Resource Center in FY 2018 to support 

additional activities within the scope of 

the Delta Region Community Health 

Systems Development Program. The 

recipient will utilize its existing 

infrastructure and a multipronged 

approach to deliver intensive assistance 

to all eight Delta Region communities, 

including onsite assessments in 

financial, operational performance, and 

quality improvement in the areas of 

population health, social services, 

emergency medical services, and 

telehealth. Please direct any questions 

or concerns to RMoscato@hrsa.gov. 

Dated: May 31, 2018. 

George Sigounas, 

Administrator. 

[FR Doc. 2018–12141 Filed 6–5–18; 8:45 am] 

BILLING CODE 4165–15–P 


daltland on DSKBBV9HB2PROD with NOTICES