29122 Federal Register / Vol. 83, No. 121 / Friday, June 22, 2018 / Notices 

Dated: June 19, 2018. 

Seema Verma, 

Administrator, Centers for Medicare & 

Medicaid Services. 

[FR Doc. 2018–13503 Filed 6–20–18; 8:45 am] 

BILLING CODE 4120–01–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Administration for Children and 

Families 

Decisions Related to the Development 

of a Clearinghouse of Evidence-Based 

Practices in Accordance With the 

Family First Prevention Services Act of 

2018 

AGENCY: Administration for Children 

and Families, HHS. 

ACTION: Request for public comment. 

SUMMARY: The Administration for 

Children and Families, HHS, solicits 

comments by July 22, 2018 on initial 

criteria and potential candidate 

programs and services for review in a 

Clearinghouse of evidence-based 

practices in accordance with the Family 

First Prevention Services Act of 2018. 

The Clearinghouse will identify 

promising, supported, and wellsupported practices for mental health 

and substance abuse prevention and 

treatment programs, in-home parent 

skill-based programs, and kinship 

navigator programs appropriate for 

children who are candidates for foster 

care pregnant or parenting foster youth, 

and the parents or kin caregivers of 

those children and youth. 

SUPPLEMENTARY INFORMATION: Invitation 

to Comment: HHS invites comments 

regarding this Notice. To ensure that 

your comments have maximum effect, 

please identify clearly the section of this 

Notice that your comment addresses. 

1.0 Background and Legislative 

Context 

The Family First Prevention Services 

Act (FFPSA) was signed into law as part 

of the Bipartisan Budget Act (H.R. 1892) 

on February 9, 2018. FFPSA enables 

States to use Federal funds available 

under parts B and E of title IV of the 

Social Security Act to provide enhanced 

support to children and families and 

prevent foster care placements through 

the provision of evidence-based mental 

health and substance abuse prevention 

and treatment services, in-home parent 

skill-based programs, and kinship 

navigator services. As described in the 

statutory language, these services and 

programs are intended ‘‘for children 

who are candidates for foster care or 

who are pregnant or parenting foster 

youth and the parents or kin caregivers 

of the children’’. 

FFPSA requires an independent 

systematic review of evidence to 

designate programs and services as 

‘‘promising,’’ ‘‘supported,’’ and ‘‘wellsupported’’ practices, defined as follows 

in section 471(e)(4)(C): 

• Promising Practice: ‘‘A practice shall be 

considered to be a ‘promising practice’ if the 

practice is superior to an appropriate 

comparison practice using conventional 

standards of statistical significance (in terms 

of demonstrated meaningful improvements 

in validated measures of important child and 

parent outcomes, such as mental health, 

substance abuse, and child safety and wellbeing), as established by the results or 

outcomes of at least one study that—(1) was 

rated by an independent systematic review 

for the quality of the study design and 

execution and determined to be welldesigned and well-executed; and (2) utilized 

some form of control (such as an untreated 

group, a placebo group, or a wait list study).’’ 

• Supported Practice: ‘‘A practice shall be 

considered to be a ‘supported practice’ if (I) 

the practice is superior to an appropriate 

comparison practice using conventional 

standards of statistical significance (in terms 

of demonstrated meaningful improvements 

in validated measures of important child and 

parent outcomes, such as mental health, 

substance abuse, and child safety and wellbeing), as established by the results or 

outcomes of at least one study that—(aa) was 

rated by an independent systematic review 

for the quality of the study design and 

execution and determined to be welldesigned and well-executed; and (bb) was a 

rigorous random-controlled trial (or, if not 

available, a study using a rigorous quasiexperimental research design); and (cc) was 

carried out in a usual care or practice setting 

and (II) the study described in sub-clause (I) 

established that the practice has a sustained 

effect (when compared to a control group) for 

at least 6 months beyond the end of 

treatment.’’ 

• Well-supported Practice: ‘‘A practice 

shall be considered to be a ‘well-supported 

practice’ if (I) the practice is superior to an 

appropriate comparison practice using 

conventional standards of statistical 

significance (in terms of demonstrated 

meaningful improvements in validated 

measures of important child and parent 

outcomes, such as mental health, substance 

abuse, and child safety and well-being), as 

established by the results or outcomes of at 

least two studies that—(aa) were rated by an 

independent systematic review for the 

quality of the study design and execution and 

determined to be well-designed and wellexecuted; and (bb) were rigorous randomcontrolled trials (or, if not available, studies 

using a rigorous quasi-experimental research 

design); and (cc) were carried out in a usual 

care or practice setting and (II) at least one 

of the studies described in sub-clause (I) 

established that the practice has a sustained 

effect (when compared to a control group) for 

at least 1 year beyond the end of treatment.’’ 

In accordance with FFPSA, practices 

must also meet the following 

requirements: 

• Book or manual: The practice has a 

book, manual, or other available writings that 

specify the components of the practice 

protocol and describe how to administer the 

practice. 

• No empirical risk of harm: There is no 

empirical basis suggesting that, compared to 

its likely benefits, the practice constitutes a 

risk of harm to those receiving it. 

• Weight of evidence supports benefits: If 

multiple outcome studies have been 

conducted, the overall weight of evidence 

supports the benefits of the practice. 

• Reliable and valid outcome measures: 

Outcome measures are reliable and valid, and 

are administrated consistently and accurately 

across all those receiving the practice. 

• No case data for severe or frequent risk 

of harm: There is no case data suggesting a 

risk of harm that was probably caused by the 

treatment and that was severe or frequent 

(section 471(e)(4)(C)(ii) of the Act). 

In order to meet these requirements, 

the Administration for Children and 

Families (ACF) in the Department of 

Health and Human Services (HHS) 

intends to establish and maintain a 

public Clearinghouse of practices, 

including culturally specific, or 

location- or population-based 

adaptations of practices, identified via a 

systematic review of evidence on 

relevant programs and services. In 

accordance with FFPSA and building 

from other federal evidence reviews, 

HHS is developing initial criteria that 

will be used to designate programs and 

services as promising, supported, and 

well-supported practices. HHS will also 

identify a preliminary list of candidate 

services and programs that will be 

considered for systematic review. 

This Notice (1) identifies and requests 

comment on potential initial criteria for 

(a) identifying eligible programs and 

services for review by the 

Clearinghouse, (b) prioritizing eligible 

programs and services for review, (c) 

identifying eligible studies aligned with 

prioritized programs and services, (d) 

prioritizing eligible studies for rating, (e) 

rating studies, and (f) rating programs 

and services as promising, supported, 

and well-supported practices. This 

Notice (2) requests comment on 

potential programs and services that 

may meet the aforementioned criteria 

and that should be considered as 

candidates for systematic review. After 

comments are received, HHS will revise 

and publish the initial criteria and a 

preliminary list of candidate programs 

and services to be considered for 

review; and begin to conduct reviews. 

This Notice is one step in ensuring that 

activities associated with the 

development of a Clearinghouse are 


daltland on DSKBBV9HB2PROD with NOTICES
Federal Register / Vol. 83, No. 121 / Friday, June 22, 2018 / Notices 29123 

transparent and build from the existing 

knowledge of States, federal agencies, 

researchers, evaluators, program and 

service developers, key stakeholders 

and experts, and the general public. 

2.0 Initial Criteria 

2.1 Program or Service Eligibility 

Criteria. Programs or services may be 

eligible for inclusion in the 

Clearinghouse if they meet the following 

criteria developed in accordance with 

FFPSA statutory language [sections 

471(e)(1)(B) and 471(e)(1)(c)]: 

2.1.1 Types of Programs and Services. 

HHS intends to limit eligibility to mental 

health and substance abuse prevention and 

treatment services, in-home parent skillbased programs (including parenting skills 

training, parent education, and individual 

and family counseling), and kinship 

navigator programs. 

2.1.2 Book/Manual/Writings Available. 

HHS intends to limit eligibility to programs 

or services that have a book, manual, or other 

available documentation that specifies the 

components of the practice protocol and 

describes how to administer the practice. 

2.2 Program or Service Prioritization 

Criteria. Timing and resources may not allow 

for a detailed review of all programs and 

services determined to be eligible by the 

criteria detailed in section 2.1 Program or 

Service Eligibility Criteria. Programs or 

services may be prioritized for review on the 

basis of the following criteria: 

2.2.1 Types of Programs and Services. As 

noted in 2.1.1. Types of Programs and 

Services, HHS intends to limit eligibility to 

mental health and substance abuse 

prevention and treatment services, in-home 

parent skill-based programs (including 

parenting skills training, parent education, 

and individual and family counseling), or 

kinship navigator programs. This Notice 

requests comment on the scope of programs 

and services and topic areas of interest 

within the aforementioned categories that 

should be prioritized for inclusion. 

2.2.2 Target Population of Interest. HHS 

intends to prioritize programs or services for 

review that have been developed or used to 

target children and families involved in the 

child welfare system or populations similar 

to those involved in the child welfare system. 

This Notice requests comment on 

populations that may be considered 

‘‘similar’’ to those involved in the child 

welfare system. 

2.2.3 Target Outcomes. HHS intends to 

prioritize programs or services for review that 

aim to impact target outcomes. Target 

outcomes should be defined in accordance 

with FFPSA statutory language [section 

471(e)(4)(C)] and include those outcomes that 

‘‘. . . prevent child abuse and neglect, and 

reduce the likelihood of foster care 

placement by supporting birth families and 

kinship families and improving targeted 

supports for pregnant and parenting youth 

and their children.’’ These may include, but 

are not limited to, ‘‘. . . important child and 

parent outcomes, such as mental health, 

substance abuse, and child safety and wellbeing.’’ This Notice requests comment on 

which types of mental health, substance 

abuse, and child and family outcomes should 

be considered as ‘target outcomes’ and 

requests research evidence to support 

recommendations of ‘target outcomes’. HHS 

does not intend to include access to service, 

satisfaction with programs and services, and 

referral to programs and services as ‘target 

outcomes’. 

2.2.4 Number of Impact Studies. HHS 

intends to prioritize programs or services 

with at least two studies with nonoverlapping analytic samples and distinct 

implementations examining effectiveness/ 

impact. 

2.2.5 In Use/Active. HHS intends to 

prioritize programs or services currently in 

use in the U.S. Programs or services that are 

no longer in operation or have no 

information available about active 

implementation will not be prioritized. 

2.2.6 Implementation and Fidelity 

Support. HHS intends to prioritize programs 

or services that have implementation training 

and staff support and/or fidelity monitoring 

tools and resources available to implementers 

in the United States. 

2.2.7 Trauma-Informed. HHS may also 

prioritize services and programs that have 

been implemented using a trauma-informed 

approach. FFPSA statutory language [section 

471(e)(4)(B)] states, ‘‘The services or 

programs to be provided to or on behalf of 

a child are provided under an organizational 

structure and treatment framework that 

involves understanding, recognizing, and 

responding to the effects of all types of 

trauma and in accordance with recognized 

principles of a trauma-informed approach 

and trauma-specific interventions to address 

trauma’s consequences and facilitate 

healing.’’ This Notice requests comment on 

the feasibility of prioritizing programs and 

services based on past implementation in 

accordance with trauma-informed principles. 

2.2.8 Delivery Setting for In-Home Parent 

Skill-Based Programs and Services. HHS 

intends to prioritize in-home parent skillbased programs and services where the 

primary service delivery strategy takes place 

in the caregivers’ place of residence. 

2.3 Study Eligibility Criteria. HHS intends 

to engage in a broad literature search to 

identify studies examining prioritized 

programs and services. This search may 

include databases, websites, existing 

literature reviews, and meta-analyses. HHS 

intends to screen studies for eligibility using 

the following criteria: 

2.3.1 Impact Study. HHS intends to limit 

eligibility to studies included in government 

reports and peer-reviewed journal articles 

that assess effectiveness (i.e., impact) using 

quantitative methods. 

2.3.2 Target Outcomes. HHS intends to 

limit eligibility to studies that examine the 

impact of the service or program on at least 

one ‘target outcome’, as described in section 

2.2.3. Target Outcomes. As noted above, this 

Notice requests comment on specific 

outcomes in accordance with FFPSA 

statutory language that should be considered 

‘target outcomes’ and requests research 

evidence to support recommendations of 

‘target outcomes’. 

2.3.3 Conducted in the U.S., U.K., 

Canada, New Zealand, or Australia. HHS 

intends to limit eligibility to studies 

conducted with samples in the U.S., U.K., 

Canada, New Zealand, or Australia to ensure 

that the evidence base reflects the 

populations where programs and services 

will be implemented. 

2.3.4 Study Published in English. HHS 

intends to limit eligibility to studies 

published in English. 

2.3.5 Published or Prepared in or after 

1990. HHS intends to limit eligibility to 

studies published or prepared in or after 

1990. 

2.3.6 Usual Care or Practice Setting. HHS 

intends to limit eligibility to studies carried 

out in a usual care or practice setting in 

accordance with FFPSA [section 

471(e)(4)(C)]. This Notice requests comment 

on the definition of usual care or practice 

settings. 

2.4 Study Prioritization Criteria. Timing 

and resources may not allow for a detailed 

rating of all studies determined to be eligible 

by the criteria identified in section 2.3 Study 

Eligibility Criteria. HHS intends to conduct a 

high-level scan of eligible studies to 

determine which should be prioritized for 

rating. This Notice requests comment on 

criteria that can be used to prioritize eligible 

studies for rating. 

2.4.1 Implementation Period: FFPSA 

[section 471(e)(1)(A) and (B)] states that the 

Secretary may make a payment to a State for 

providing services or programs ‘‘for not more 

than a 12-month period’’. This Notice 

requests comment on whether studies with 

program or service implementation periods 

of longer than 12 months should be 

considered for review and if so, whether any 

other implementation period cutoff should be 

included as a study prioritization criterion. 

2.4.2 Sample of Interest. HHS intends to 

prioritize studies that include samples of 

children and families involved in the child 

welfare system or populations similar to 

those involved in the child welfare system. 

This Notice requests comment on 

populations that may be considered 

‘‘similar’’ to those involved in the child 

welfare system. 

2.5 Study Rating Criteria. HHS intends to 

rate studies on the following criteria: 

2.5.1 Favorable Effects. HHS intends to 

rate studies based on whether they 

demonstrate at least one meaningful 

favorable effect (i.e., positive significant 

effect) on a ‘target outcome’ as specified in 

section 2.3.2 Target Outcomes. A meaningful 

effect will be defined using conventional 

standards of statistical significance (i.e., twotailed hypothesis test and a specified alpha 

level of p < .05). This Notice requests 

comment on whether and how ratings should 

consider the number or magnitude of 

favorable effects. 

2.5.2 Unfavorable Effects. HHS intends to 

rate studies based on the number of 

unfavorable effects (i.e., negative significant 

effects) on either ‘target’ or non-target 

outcomes as specified in section 2.3.2 Target 

Outcomes. Effects will be defined using 

conventional standards of statistical 

significance (i.e., two-tailed hypothesis test 

and a specified alpha level of p < .05). This 


daltland on DSKBBV9HB2PROD with NOTICES
29124 Federal Register / Vol. 83, No. 121 / Friday, June 22, 2018 / Notices 

Notice requests comment on whether and 

how studies should also be rated on the 

number of null effects on ‘target outcomes’, 

and on whether and how ratings should 

consider the number or magnitude of 

unfavorable effects. 

2.5.3 Sustained Favorable Effect. HHS 

intends for studies with at least one favorable 

effect on a ‘target outcome’, as determined by 

the criteria in 2.5.1 Favorable Effects, to be 

rated on whether or not they demonstrate a 

sustained favorable effect. As noted in 

section 471(e)(4)(C), a ‘supported practice’ 

must have at least one study that 

demonstrates ‘‘a sustained effect (when 

compared to a control group) for at least 6 

months beyond the end of treatment’’ and a 

‘well-supported practice’ must have at least 

one study that demonstrates ‘‘a sustained 

effect (when compared to a control group) for 

at least 1 year beyond the end of treatment.’’ 

HHS intends to classify studies as not 

demonstrating a sustained favorable effect 

(i.e., effects are demonstrated for less than 6 

months), demonstrating a sustained favorable 

effect of 6 months or more (but less than 12 

months), or demonstrating a sustained 

favorable effect of 12 months or more. 

2.5.4 Rigorous Study Design. HHS 

intends to rate studies as either high, 

moderate, or low on the rigor and 

appropriateness of their study design. Study 

designs that receive the highest rating will be 

either Randomized Controlled Trials (RCTs) 

or rigorous quasi-experimental designs. HHS 

defines randomized controlled trials as a 

study design in which sample members are 

assigned to the program or service and 

comparison groups by chance. Randomized 

control designs are often considered the 

‘‘gold standard’’ of research design because 

personal characteristics (before the program 

or service begins) do not affect whether 

someone is assigned to the program or 

service or control group. HHS defines a 

quasi-experimental design as a study design 

in which sample members are selected for 

the program or service and comparison 

groups in a nonrandom way. Similar to 

criteria considered in other federal evidence 

clearinghouses, rigorous study designs will 

be those that are appropriately powered, 

include an appropriate control group, 

maintain original assignment to study arms, 

and are appropriate to combat threats to 

internal validity. This Notice requests 

comment on threats to internal validity that 

should be considered. This Notice requests 

comment on appropriate thresholds for 

evaluating and assigning a rating to a study 

design. 

2.5.5 Rigorous Study Analysis. HHS 

intends to rate studies as either high, 

moderate, or low on the rigor and 

appropriateness of their analysis. Study 

analyses that receive the highest rating may 

be those that tested and established baseline 

equivalence, appropriately accounted for 

overall and differential sample attrition, 

appropriately accounted for multiple 

comparisons, and when necessary accounted 

for clustering. This Notice requests comment 

on appropriate thresholds for evaluating and 

assigning a rating to a study analysis. 

2.5.6 Reliability, Validity, and Systematic 

Administration of Outcome Measures. HHS 

intends to rate studies as either high, 

moderate, or low on the extent to which 

‘target outcome’ measures are reliable (i.e., 

the extent to which a measure produces the 

same results when used repeatedly), valid 

(i.e., the extent to which a measure captures 

what it is intended to measure), and were 

administered consistently and accurately 

across all those receiving the practice in 

accordance with FFPSA statutory language 

[section 471(e)(4)(C)] or receiving the 

appropriate comparison practice. This Notice 

requests comment on appropriate thresholds 

for evaluating and assigning a rating to the 

reliability, validity, and administration of 

‘target outcome’ measures. 

2.6 Program or Service Rating Criteria. 

HHS intends for programs or services to be 

rated as promising, supported, or wellsupported practices if they meet the below 

criteria that collectively assess the strength of 

evidence for a practice and build from the 

individual study criteria described in section 

2.5 Study Rating Criteria. These criteria were 

developed in accordance with FFPSA 

statutory language [section 471(e)(4)(C)]. 

2.6.1 Promising Practice: HHS intends to 

designate a program or service as a 

‘promising practice’ if the program or service 

has at least one study that demonstrates a 

favorable effect on a target outcome as 

described by criterion 2.5.1 Favorable Effects 

and achieves, at a minimum, moderate 

ratings on criteria 2.5.4 through 2.5.6. 

2.6.2 Supported Practice: HHS intends to 

designate a program or service as a 

‘supported practice’ if the program or service 

has at least one study that demonstrates a 

favorable effect on a target outcome as 

described by 2.5.1 Favorable Effects, 

demonstrates a sustained favorable effect on 

a target outcome of at least 6 months beyond 

the end of treatment as described in Section 

2.5.3 Sustained Favorable Effect, and 

achieves the high rating on criteria 2.5.4 

through 2.5.6. 

2.6.3 Well-Supported Practice: HHS 

intends to designate a program or service as 

a ‘well-supported practice’ if the practice has 

at least two studies with non-overlapping 

analytic samples and distinct 

implementations that demonstrate favorable 

effects as described by 2.5.1 Favorable 

Effects, demonstrate sustained favorable 

effects of at least 12 months beyond the end 

of treatment as described in Section 2.5.3 

Sustained Favorable Effect, and achieve the 

high rating on criteria 2.5.4 through 2.5.6. 

HHS does not intend to rate a program 

or service as a ‘promising’, ‘supported’, 

or ‘well-supported practice’ if there is 

an empirical basis, as evidenced by 

multiple unfavorable effects on target or 

non-target outcomes across reviewed 

studies, as described in 2.5.2 

Unfavorable Effects, that suggest the 

overall weight of evidence does not 

support the benefits of the program or 

service. This Notice requests comment 

on approaches for determining that 

promising, supported, and wellsupported practices do not constitute a 

risk of harm. As described in FFPSA 

[section 471(e)(4)(C)], ‘‘There is no 

empirical basis suggesting that, 

compared to its likely benefits, the 

practice constitutes a risk of harm to 

those receiving it’’, ‘‘If multiple outcome 

studies are conducted, the overall 

weight of evidences supports the 

benefits of the practice’’, and ‘‘There is 

no case data suggesting a risk of harm 

that was probably caused by the 

treatment and that was severe or 

frequent’’. 

3.0 Recommendations of Potential 

Candidate Programs and Services for 

Review 

This Notice requests comment on 

potential candidate programs and 

services to consider for the systematic 

evidence review. Comments should 

identify how recommended programs 

and services meet the criteria described 

in section 2.1 Program or Service 

Eligibility Criteria. These criteria 

include: Types of Programs and Services 

and Book/Manual/Writings Available. 

Comments should also identify how 

recommended programs and services 

meet the criteria described in section 2.2 

Program or Service Prioritization 

Criteria. These criteria include: Types of 

Programs and Services, Target 

Population of Interest, Target Outcomes, 

Number of Impact Studies, In Use/ 

Active, Implementation and Fidelity 

Support, Trauma-Informed, and 

Delivery Setting for In-Home Parent 

Skill-Based Programs and Services. In 

order to leverage new insights from the 

field, HHS may put forth additional 

future Notices requesting 

recommendations of potential candidate 

programs and services for review. 

4.0 Submission of Comments 

Comments may be submitted until 

July 22, 2018 by email to 

ffclearinghouse@acf.hhs.gov. 

Naomi Goldstein, 

Deputy Assistant Secretary for Planning, 

Research, and Evaluation. 

[FR Doc. 2018–13420 Filed 6–21–18; 8:45 am] 

BILLING CODE 4184–25–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Food and Drug Administration 

[Docket No. FDA–2018–N–2066] 

Tobacco Products Scientific Advisory 

Committee; Notice of Meeting 

AGENCY: Food and Drug Administration, 

HHS. 

ACTION: Notice. 

SUMMARY: The Food and Drug 

Administration (FDA) announces a 


daltland on DSKBBV9HB2PROD with NOTICES