22604 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

legume, subgroup 6B at 0.5 ppm; 

vegetable, legume, subgroup 6C at 0.2 

ppm; vegetable, foliage of legume, 

subgroup 7A at 4 ppm; wheat, aspirated 

grain fraction at 2.5 ppm; wheat, bran at 

0.5 ppm; wheat, germ at 0.5 ppm; 

wheat, grain at 0.5 ppm. Syngenta Crop 

Protection, LLC, has submitted practical 

analytical methodology for detecting 

and measuring levels of thiamethoxam 

in or on raw agricultural commodities. 

This method is based on crop specific 

cleanup procedures and determination 

by liquid chromatography with either 

ultraviolet (UV) or mass spectrometry 

(MS) detections. The limit of detection 

(LOC) for each analyte of this method is 

1.25ng injected for samples analyzed by 

UV and 0.25 nanogram (ng) injected for 

samples analyzed by MS, and the limit 

quantification (LOQ) is 0.005 ppm for 

milk and juices, and 0.01 ppm for all 

other substrates. 

2. PP 4F8237. (EPA–HQ–OPP–2014– 

0156). Dow AgroSciences, LLC, 9330 

Zionsville Rd., Indianapolis, IN 46268, 

requests to establish a tolerance in 40 

CFR part 180 for residues of the 

insecticide, sulfoxaflor (N- 

[methyloxido[1-[6-(trifluoromethyl)-3- 

pyridinyl]ethyl]-g 4- 

sulfanylidene]cyanamide), in or on 

alfalfa, forage at 7 parts per million 

(ppm); alfalfa, hay at 20 ppm; alfalfa, 

seed at 30 ppm; alfalfa, silage at 9 ppm; 

animal feed, nongrass, group 18, forage 

at 15 ppm; animal feed, nongrass, group 

18, hay at 20 ppm; animal feed, 

nongrass, group 18, silage at 9 ppm; 

buckwheat, forage at 1 ppm; buckwheat, 

grain at 0.08 ppm; buckwheat, hay at 1.5 

ppm; buckwheat, straw at 2 ppm; cacao 

bean, dried bean at 0.15 ppm; clover 

forage at 15 ppm; clover hay at 20 ppm; 

clover silage at 8 ppm; corn, field, forage 

at 0.5 ppm; corn, field, grain at 0.015 

parts ppm; corn, field, stover at 0.8 

ppm; corn, pop at 0.015 ppm; corn, pop, 

stover at 0.8 ppm; corn, sweet, at 0.01 

ppm; corn, sweet, forage at 0.6 ppm; 

corn, sweet, stover at 0.7 ppm; millet, 

forage at 0.4 ppm; millet, grain at 0.3 

ppm; oat, grain at 0.4 ppm; oat, hay at 

1 ppm; oat, straw at 2 ppm; pineapple 

at 0.09 ppm; rye, forage at 1 ppm; rye, 

grain at 0.08 ppm; rye, hay at 1.5 ppm; 

rye, straw at 2 ppm; sorghum, forage at 

0.4 ppm; sorghum, grain at 0.3 ppm; 

sorghum, stover at 0.9 ppm; teff, forage 

at 1 ppm; teff, grain at 0.08 ppm; teff, 

hay at 1.5 ppm; teff, straw at 2 ppm; 

teosinte, grain at 0.015 ppm; triticale, 

forage at 1 ppm; triticale, grain at 0.08 

ppm; triticale, hay at 1.5 ppm; triticale, 

straw at 2 ppm. The residue profile of 

sulfoxaflor is adequately understood 

and an acceptable analytical method is 

available for enforcement purposes. 

Analytical method 091116, 

‘‘Enforcement Method for the 

Determination of Sulfoxaflor (XDE–208) 

and its Main Metabolites in Agricultural 

Commodities using Offline Solid-Phase 

Extraction and Liquid Chromatography 

with Tandem Mass Spectrometry 

Detection’’ was validated on a variety of 

plant matrices. The method was 

validated over the concentration range 

of 0.010–5.0 milligrams/kilograms (mg/ 

kg) with a validated limit of detection 

(LOD) of 0.003 mg/kg and limit of 

quantitation (LOQ) of 0.010 mg/kg. 

Amended Tolerance 

3. PP 3F8205. (EPA–HQ–OPP–2013– 

0758). Syngenta Crop Protection, LLC, 

P.O. Box 18300, Greensboro, NC 27419, 

requests to amend the tolerances in 40 

CFR 180.565 for residues of the 

insecticide, thiamethoxam (3-[(2-chloro5-thiazolyl)methyl]tetrahydro-5-methylN-nitro-4H-1,3,5-oxadiazin-4-imine) and 

its metabolite (N-(2-chloro-thiazol-5- 

ylmethyl)-N’-methyl-N’-nitroguanidine), by increasing the existing 

tolerances in or on alfalfa, forage from 

0.05 to 10 parts per million (ppm); 

alfalfa, hay from 0.12 to 8 ppm; barley, 

grain from 0.4 to 0.9 ppm; barley, hay 

from 0.40 to 1.5 ppm; barley, straw from 

0.40 to 3 ppm; corn, field, forage from 

0.10 to 4 ppm; corn, field, stover from 

0.05 to 4 ppm; corn, sweet, forage from 

0.10 to 5 ppm; corn, sweet, kernel plus 

cob with husks removed from 0.02 to 

0.03 ppm; corn, sweet, stover from 0.05 

to 4 ppm; wheat, forage from 0.50 to 3 

ppm; wheat, hay from 0.02 to 8 ppm; 

wheat, straw from 0.02 to 6 ppm. 

Concurrently, Syngenta Crop Protection, 

LLC, requests to amend the tolerances in 

40 CFR 180.565 by removing tolerances 

for residues of the insecticide, 

thiamethoxam (3-[(2-chloro-5- 

thiazolyl)methyl]tetrahydro-5-methyl-Nnitro-4H-1,3,5-oxadiazin-4-imine) in or 

on grain, cereal, group 15, except barley 

at 0.02 ppm; sunflower at 0.02 ppm; and 

vegetable, legume, group 6 at 0.02 ppm, 

upon approval of the tolerances listed 

under ‘‘New Tolerances’’ for PP 3F8205. 

Syngenta Crop Protection, LLC, has 

submitted practical analytical 

methodology for detecting and 

measuring levels of thiamethoxam in or 

on raw agricultural commodities. This 

method is based on crop specific 

cleanup procedures and determination 

by liquid chromatography with either 

ultraviolet (UV) or mass spectrometry 

(MS) detections. The limit of detection 

(LOC) for each analyte of this method is 

1.25 ng injected for samples analyzed by 

UV and 0.25 nanogram (ng) injected for 

samples analyzed by MS, and the limit 

quantification (LOQ) is 0.005 ppm for 

milk and juices, and 0.01 ppm for all 

other substrates. 

List of Subjects in 40 CFR Part 180 

Environmental protection, 

Agricultural commodities, Feed 

additives, Food additives, Pesticides 

and pests, Reporting and recordkeeping 

requirements. 

Dated: April 17, 2014. 

Lois Rossi, 

Director, Registration Division, Office of 

Pesticide Programs. 

[FR Doc. 2014–09257 Filed 4–22–14; 8:45 am] 

BILLING CODE 6560–50–P 

DEPARTMENT OF HEALTH AND 

HUMAN SERVICES 

Administration for Children and 

Families 

45 CFR Part 1355 

Statewide Data Indicators and National 

Standards for Child and Family 

Services Reviews 

AGENCY: Children’s Bureau (CB), 

Administration on Children, Youth and 

Families (ACYF), Administration for 

Children and Families (ACF), 

Department of Health and Human 

Services (HHS). 

ACTION: Notice of Statewide Data 

Indicators and National Standards for 

Child and Family Services Reviews. 

SUMMARY: The Children’s Bureau 

reviews a state’s substantial conformity 

with titles IV–B and IV–E of the Social 

Security Act through the Child and 

Family Services Reviews (CFSRs). 

Statewide data indicators are used to 

inform the Children’s Bureau’s 

determination of a state’s substantial 

conformity relative to certain safety and 

permanency outcomes. This document 

advises the public of the Children’s 

Bureau’s plan to replace the statewide 

data indicators and the methods for 

calculating associated national 

standards on those indicators. We invite 

the public to comment on these 

indicators and methods before their use 

in CFSRs scheduled for Federal Fiscal 

Years (FFY) 2015 through FY 2018. 

DATES: Written comments must be 

submitted to the office listed in the 

ADDRESSES section below on or before 

May 23, 2014. 

ADDRESSES: Interested persons may 

submit written comments by any of the 

following methods: 

• Federal eRulemaking Portal: http:// 

www.regulations.gov. Follow the 

instructions for submitting comments. 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22605 

1AFCARS collects case-level information from 

state and Tribal title IV–E agencies on all children 

in foster care and those who have been adopted 

with title IV–E agency involvement. Title IV–E 

agencies must submit AFCARS data to the 

Children’s Bureau twice a year. 

2NCANDS collects child-level information on 

every child who receives a response from a child 

protective services agency due to an allegation of 

abuse or neglect. States report this data to the 

Children’s Bureau voluntarily. In FFY 2013, all 50 

states, the District of Columbia, and Puerto Rico 

submitted NCANDS data. 

• Mail or Courier Delivery: Miranda 

Lynch Thomas, Children’s Bureau, 

Administration on Children, Youth and 

Families, Administration for Children 

and Families, 1250 Maryland Avenue 

SW., 8th Floor, Washington, DC 20024. 

Instructions: If you choose to use an 

express, overnight, or other special 

delivery method, ensure that delivery 

may be made at the address listed under 

the ADDRESSES section. We urge 

interested parties to submit comments 

electronically to ensure that they are 

received in a timely manner. All 

comments received will be posted 

without change to http://

www.regulations.gov. This will include 

any personal information provided. 

FOR FURTHER INFORMATION CONTACT: 

Miranda Lynch Thomas, Children’s 

Bureau, 1250 Maryland Ave. SW., 8th 

Floor, Washington, DC 20024, (202) 

205–8138. 

SUPPLEMENTARY INFORMATION: 

Background 

The Children’s Bureau implemented 

the CFSRs in 2001 in response to a 

mandate in the Social Security 

Amendments of 1994 for the 

Department of Health and Human 

Services to issue regulations for the 

review of state child and family services 

programs under titles IV–B and IV–E of 

the Social Security Act (see section 

1123A of the Social Security Act). The 

reviews are required for the Children’s 

Bureau to determine whether such 

programs are in substantial conformity 

with title IV–B and IV–E plan 

requirements, implementing 

regulations, and relevant title IV–B and 

IV–E plans. The review process, as 

regulated at 45 CFR 1355.31–37, grew 

out of extensive consultation with 

interested groups, individuals, and 

experts in the field of child welfare and 

related areas. 

The Children’s Bureau conducted the 

first round of CFSRs from 2001 through 

2004 and the second round from 2007 

through 2010. The third round is 

scheduled to begin in FFY 2015. 

Information about the initiation of this 

latest round can be found in CFSR 

Technical Bulletin #7 issued in March 

2014 (see http://www.acf.hhs.gov/

programs/cb/resource/cfsr-technicalbulletin-7). 

The CFSRs enable the Children’s 

Bureau to: (1) Ensure conformity with 

federal child welfare requirements; (2) 

determine what is actually happening to 

children and families as they are 

engaged in child welfare services; and 

(3) assist states to enhance their capacity 

to help children and families achieve 

positive outcomes. The Children’s 

Bureau conducts the reviews in 

partnership with state child welfare 

agency staff and other stakeholders 

involved in the provision of child 

welfare services. We have structured the 

reviews to help states identify strengths 

as well as areas needing improvement 

within their agencies and programs. 

The CFSR assesses state performance 

on seven outcomes and seven systemic 

factors. The seven outcomes focus on 

key items measuring safety, 

permanency, and well-being. The seven 

systemic factors focus on key state plan 

requirements of titles IV–B and IV–E 

that provide a foundation for child 

outcomes. States that the Children’s 

Bureau determines have not achieved 

substantial conformity in all the areas 

assessed in the review are required to 

develop and implement a program 

improvement plan within two years 

addressing the areas of nonconformity. 

The Children’s Bureau supports the 

states with technical assistance and 

monitors implementation of their 

program improvement plans. States that 

are unable to complete their plans 

successfully have some of their federal 

child welfare funds withheld. 

Most relevant to this document is the 

element of the reviews that provides for 

the Children’s Bureau to determine 

whether the state is in substantial 

conformity with certain child outcomes 

based on national standards we set for 

state performance on statewide data 

indicators. The regulations at 45 CFR 

1355.34(b)(4) and (5) authorize us to 

add, amend, or suspend any of the 

statewide data indicators when 

appropriate, and to adjust the national 

standards when appropriate. Statewide 

data indicators are aggregate measures 

and we calculate them using 

administrative data available from a 

state’s submissions to the Adoption and 

Foster Care Analysis and Reporting 

System (AFCARS),1 the National Child 

Abuse and Neglect Data System 

(NCANDS),2 or a Children’s Bureauapproved alternate source for safetyrelated data. If a state is determined not 

to be in substantial conformity with a 

related outcome due to its performance 

compared to the national standard for 

an indicator, the state will include that 

indicator in a program improvement 

plan. The Children’s Bureau establishes 

performance goals for each indicator 

included in a program improvement 

plan, based on the state’s prior 

performance, that the state is expected 

to reach by the end of the program 

improvement plan period. The goal to 

be achieved is relative to the state’s 

baseline performance at the beginning of 

the program improvement plan period. 

The Children’s Bureau views the 

CFSR as a dynamic process and has 

made ongoing improvements in the 

process to best meet state and federal 

needs. Most recently, we solicited 

feedback from the public (see 76 FR 

18677, published April 5, 2011) about 

how they would envision a federal 

review process that meets the statutory 

requirements in section 1123A of the 

Social Security Act and holds child 

welfare agencies accountable for 

achieving positive outcomes for 

children and families and continuously 

improving the quality of their systems 

for doing so. In addition, we hired a 

consultant that specializes in child 

welfare measurement to work with 

Children’s Bureau data specialists. In 

2013 we also tasked a contractor to the 

Children’s Bureau to convene a panel of 

child welfare administrators and data 

measurement experts to develop 

recommendations and feedback about 

specific aspects of the review process, 

including the statewide data indicators, 

national standards, and program 

improvement. The information from 

these experts along with public 

comments has shaped our plan for 

replacing the statewide data indicators 

that will be used in the CFSRs. 

Existing Statewide Data Indicators and 

Composite Measures and Planned 

Improvements To Address Feedback 

For CFSR Round 2, the Children’s 

Bureau developed six statewide data 

indicators and measures: two indicators 

related to safety and four composite 

measures related to permanency. The 

two safety-related statewide data 

indicators focused on recurrence of 

maltreatment and maltreatment of 

children in foster care and were used to 

inform an assessment of the state’s 

substantial conformity with the safety 

outcome that children are, first and 

foremost, protected from abuse and 

neglect. The four permanency-related 

data composites were used to inform the 

assessment of a state’s substantial 

conformity with the permanency 

outcome that children have permanency 

and stability in their living situations. 

The four permanency composites used 

during CFSR Round 2 were related to 

measures of timeliness and permanency 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
22606 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

3More detailed information on the CFSR Round 

2 measures can be found on the Children’s Bureau 

Web site at http://www.acf.hhs.gov/sites/default/

files/cb/data_indicators_for_the_second_round_of_

cfsrs.pdf. 

4See Testa, M., Poertner, J. Koh, E. (2008). Can 

AFCARS be rescued? Fixing the statistical yardstick 

that measures state child welfare performance. 

Urbana, IL: CFRC; and Testa, M. & Poertner, J. 

(Eds.). (2010). Fostering accountability: Using 

evidence to guide and improve child welfare policy. 

Oxford: Oxford University Press. 

5Rates are calculated per day of foster care. 

However, we will multiply the rate by 100,000 to 

produce larger and more meaningful numbers. 

of reunification; timeliness of adoptions, 

permanency for children in foster care 

for extended time periods, and 

placement stability.3 The Children’s 

Bureau established national standards 

for each of the six data indicators and 

composites. The Children’s Bureau 

thoroughly considered the alternatives 

available to us at the time and had a 

clear rationale for supporting the 

statewide data indicators, composites 

and methods chosen. However, we 

believe there is reason to make further 

improvements now given the additional 

concerns raised by stakeholders and the 

Children’s Bureau that we summarize 

below and throughout this document as 

we describe the proposed new 

indicators. 

Cohorts Used for Statewide Data 

Indicators: The Children’s Bureau is 

planning to calculate several statewide 

data indicators using entry cohorts to 

address concerns about the 

measurement techniques that relied on 

multiple exit cohorts in prior CFSR 

rounds. Some stakeholders noted that 

relying on exit cohorts and crosssectional (also referred to as point-intime) cohorts, rather than prospective 

entry cohorts, can statistically distort 

system performance outcomes because 

they represent biased sub-samples of all 

children who are served.4 For example, 

looking only at children who exit foster 

care in a given year ignores those 

children who did not leave; it is 

therefore not representative of all 

children in foster care who would be 

affected by state improvement efforts. 

Looking only at those children in foster 

care at the end of the year biases the 

sample to include more children with 

long lengths of stay in foster care. Entry 

cohorts, which include all children 

entering foster care in a given year, 

avoid these problems and provide a 

more complete assessment of overall 

system performance and recent practice 

trends. 

While entry cohorts have 

methodological advantages, they have 

limitations in terms of assessing state 

performance with regard to children 

who have been in foster care for a long 

time because of the length of time we 

measure for state improvements. For 

example, with an entry cohort approach, 

children who had already been in foster 

care for two or more years could only 

start being tracked in a third year. To 

address this limitation, the Children’s 

Bureau will still use other cohorts in 

some of its indicators. 

Composites: The Children’s Bureau 

plans to use individual indicators rather 

than composite measures for the 

purpose of establishing national 

standards in this round of review. We 

implemented the composite approach in 

CFSR Round 2 after consultation with 

states and national experts to support a 

more holistic approach to measuring 

state performance. The expansion to 

composites from the one-dimensional 

measures used in CFSR Round 1 

allowed state performance on a 

particular domain to reflect broader 

performance, accounting for both 

strengths and weaknesses within the 

domain. Criticism of the composite 

measures used during CFSR Round 2 

included the complexity of the 

composite scores for interpretation 

purposes. 

To address these concerns and clarify 

expectations with regard to national 

performance, the Children’s Bureau 

proposes to measure state performance 

with simplified statewide data 

indicators. We propose to maintain 

some of the advantages found with the 

composite approach by implementing 

companion measures during the 

program improvement plan period to 

provide an expanded and more effective 

measurement of a domain. 

New Proposed Statewide Data 

Indicators and Methods 

The Children’s Bureau plans for the 

new statewide data indicators to 

measure maltreatment in foster care and 

re-report of maltreatment as a 

component in evaluating Safety 

Outcome 1: Children are, first and 

foremost, protected from abuse and 

neglect. We also plan for statewide data 

indicators to measure the achievement 

of permanency in 12 months for 

children entering foster care, 

permanency in 12 months for children 

in foster care for 2 years or more, reentry to foster care, and placement 

stability. These four permanency 

indicators will be used as a component 

in evaluating Permanency Outcome 1: 

Children have permanency and stability 

in their living situations. Below is a 

description of each of the six proposed 

indicators including their definition, 

justification for inclusion, calculation 

and a discussion of relevant issues. 

Following the description of the 

indicators is information on the 

methods the Children’s Bureau plans to 

use for calculating the national 

standards and our approach to 

measuring a state’s program 

improvement on the indicators should 

we find that the state is not able to meet 

the national standard. We also provide 

additional information on how the 

Children’s Bureau will share data 

information with states through profiles 

and data quality issues that impact these 

indicators and methods. 

Attachment A provides a summary of 

each planned indicator including 

numerators, denominators, exclusions, 

and adjustments. Attachment B 

provides a comparison of the data 

measures used during CFSR Round 2 

with the statewide data indicators we 

propose to use during Round 3. 

Attachment C provides information on 

the AFCARS and NCANDS data 

elements that are used to calculate the 

proposed indicators and national 

standards. 

Statewide Data Indicators for CFSR 

Safety Outcome 1: Children Are, First 

and Foremost, Protected From Abuse 

and Neglect 

Proposed Safety Performance Area 1: 

Maltreatment in Foster Care 

Indicator Definition: Of all children in 

foster care during a 12-month period, 

what is the rate of victimization per day 

of foster care? 5 The indicator includes 

all cases of substantiated or indicated 

maltreatment while in foster care and all 

days for all children in foster care at any 

point during a 12-month period. The 

denominator is all child days in foster 

care over a 12-month period, and the 

numerator is the number of instances of 

substantiated or indicated maltreatment 

among children in foster care over that 

same period. The definition of 

‘‘children’’ for this indicator (and all 

indicators) includes those under the age 

of 18. This indicator includes all 

maltreatment types by any perpetrator, 

which may include foster parents, 

facility staff members, parents, or 

others. In addition, this indicator 

includes all days for all children in 

foster care at any point during a 12- 

month period. 

Some states provide incident dates in 

their NCANDS data submissions. If a 

state provides incident dates, records 

with an incident date occurring before 

the date of removal will be excluded. 

Children in foster care for less than 8 

days and any report that occurs within 

the first 7 days of removal are excluded 

from this indicator. This indicator is 

calculated using data that match 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22607 

6Almost all states report AFCARS identifiers in 

the NCANDS data. For those states that do not, a 

Children’s Bureau-approved alternate source will be 

required to assess performance on this indicator. 

7See Drake, Jonson-Reid, Way, & Chung (2003). 

Substantiation and Recidivism. Child Maltreatment. 

Vol. 8, No. 4, 248–260. 

children across AFCARS and NCANDS 

using the AFCARS identifier. 

Justification for Inclusion: This 

indicator provides a measure of whether 

the state child welfare agency is able to 

ensure that children do not experience 

abuse or neglect while in the state’s 

foster care system. The indicator holds 

states accountable for keeping children 

safe from harm while under the 

responsibility of the state, no matter 

who perpetrates the maltreatment in 

foster care. 

Relevant Issues and Discussion: 

During CFSR Round 2, the Children’s 

Bureau had a safety data indicator 

related to maltreatment in foster care. 

For that indicator, the counts of 

children not maltreated in foster care 

were derived by subtracting the 

NCANDS count of children maltreated 

by foster care providers from the total 

count of all children in foster care on 

the last day of the year, as reported in 

AFCARS. Because of improved 

reporting by states, we can now link 

AFCARS and NCANDS data using the 

child identifier and determine whether 

maltreatment occurred during a foster 

care episode, improving accuracy on 

this indicator by using entry cohorts 

instead of the retrospective method used 

in CFSR Round 2.6 This technique also 

allows us to expand the indicator to 

include maltreatment regardless of the 

perpetrator type. This measure uses the 

report date reported in NCANDS to 

determine if a child is victimized while 

in foster care, discounting the first week 

to allow for a potential lag time between 

an incidence of maltreatment and report 

of maltreatment. For those states that 

provide incident dates, an adjustment 

will be made if the data indicates that 

the incident occurred prior to the foster 

care episode. We encourage states to 

report incident dates in NCANDS, 

which will improve the accuracy of this 

indicator. The Children’s Bureau made 

this change in response to a suggestion 

from stakeholders with regard to the 

indicators used for the last round of 

reviews that we are now able to address. 

Proposed Safety Performance Area 2: 

Re-Report of Maltreatment 

Indicator Definition: Of all children 

who received a screened-in report of 

maltreatment during a 12-month period 

(regardless of disposition type), what 

percent were reported again within 12 

months from the date of initial report? 

The denominator is the number of 

children with at least one screened-in 

report of alleged maltreatment in a 12- 

month period and the numerator is the 

number of children in the denominator 

that had another screened-in report with 

a disposition within 12 months of their 

initial report. Screened-in reports that 

have a disposition reported are 

included, regardless of whether the 

disposition is that the child is a victim 

or a non-victim. This indicator is 

calculated using data from NCANDS. 

Justification for Inclusion: This 

indicator is included to provide an 

assessment of whether the agency took 

the necessary actions to prevent a future 

report of maltreatment for children 

previously the subject of a screened-in 

report to the agency. 

Relevant Issues and Discussion: 

During CFSR Round 2, the Children’s 

Bureau had a safety performance area 

related to repeat maltreatment. That 

measure was derived from calculating 

what percent of all children who were 

victims in a substantiated or indicated 

maltreatment allegation during the first 

6 months of the reporting period were 

not victims in another substantiated or 

indicated maltreatment allegation 

within a 6-month period. We are 

expanding this measure to count all 

children with screened-in reports of 

alleged maltreatment. The Children’s 

Bureau believes that multiple reports 

regardless of whether maltreatment is 

substantiated or indicated is a viable 

measure of the agency’s attempts to 

prevent maltreatment based on research 

indicating that families with screened-in 

but unsubstantiated reports are at a high 

risk of re-report, in some cases as high 

as substantiated cases.7 Because reports 

are included regardless of disposition, 

this indicator includes both victims and 

non-victims. In addition, this indicator 

expands the time period examined to 12 

months to include more children. The 

indicator also tracks such children for 

12 months, as opposed to 6 months as 

in the prior indicator. 

The Children’s Bureau made this 

change in response to a suggestion from 

stakeholders with regard to the 

indicators used for the last round of 

reviews that we are now able to address 

with the improved quality of data 

reported in NCANDS. In addition, the 

contractor’s recommendations based on 

the expert panel convened in 2013 

expressed support for a measure of 

screened-in reports to capture repeat 

maltreatment. 

Previous CFSR data measures focused 

on substantiated and indicated reports 

of maltreatment. The growing 

implementation of differential response 

in the states (sometimes referred to as 

alternative response programs) where a 

substantial percentage of cases may 

bypass formal investigation altogether, 

however, makes a comparison of 

differential-response and nondifferential-response states difficult. In 

addition, states that initiate or expand 

differential response during an 

improvement period could show 

improvement on a substantiation-based 

measure of repeat maltreatment merely 

as an artifact of adopting differential 

response. An indicator based only on 

screened-in reports is not affected by 

differential response which contributed 

to our selecting this indicator. 

CFSR Permanency Outcome 1: Children 

Have Permanency and Stability in 

Their Living Situations 

The permanency-related statewide 

data indicators exclude children 

entering foster care at age 18 and older 

or who are already 18 and older on the 

first day of the period under review. 

Although the amendments to title IV–E 

of the Social Security Act made by the 

Fostering to Success and Increasing 

Adoptions Act of 2008 (Pub. L. 110– 

351) permit states to provide foster care 

to youth who are age 18 and older, all 

states have not exercised such an 

option. Some states provide foster care 

to youth age 18 and older, however, 

there is no consistent inclusion of this 

population of youth across states and no 

consistent construct at this time for 

what achieving permanency means for 

such older youth. Therefore, the 

Children’s Bureau believes that it is 

appropriate to limit the permanency 

statewide data indicators to children 

under age 18 in this way to maintain 

consistency as we have in prior rounds. 

Proposed Permanency Performance 

Area 1: Permanency in 12 Months for 

Children Entering Foster Care 

Indicator Definition: Of all children 

who enter foster care in a 12-month 

period, what percent discharged to 

permanency within 12 months of 

entering foster care? The denominator is 

the number of children who enter foster 

care in a 12-month period and the 

numerator is the number of children in 

the denominator who discharged to 

permanency within 12 months of 

entering foster care. For the purposes of 

this indicator, permanency includes a 

child’s discharge from foster care to 

parent(s), living with relatives, 

guardianship, or adoption. Children 

who are in foster care for less than 8 

days are excluded from this indicator. 

For children with multiple episodes 

during the same 12 month period, this 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
22608 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

measure will only evaluate the first 

episode within the period. This 

indicator is calculated using data from 

AFCARS. 

Justification for Inclusion: This 

indicator provides a focus for the child 

welfare agency’s responsibility to 

reunify or place children in safe and 

permanent homes as soon as possible 

after removal. 

Relevant Issues and Discussion: 

During CFSR Round 2, the Children’s 

Bureau included a similar measure as 

part of one of the permanency 

composites. That measure assessed the 

percent of all children entering foster 

care for the first time in a 6-month 

period that left foster care to 

reunification (or living with a relative) 

within 12 months of entering foster care. 

For CFSR Round 3, the indicator will 

stand alone and not be combined into a 

composite measure. The indicator is 

also more expansive than the prior to 

include: (1) All permanency outcomes, 

including guardianship or adoption; (2) 

all children entering foster care rather 

than first removals only; and (3) 

children entering foster care over the 

course of 12 months rather than 6 

months. 

The Children’s Bureau made this 

change in response to suggestions from 

stakeholders with regard to the 

indicators used for the last round of 

reviews that we are now able to address. 

The indicator’s expanded set of 

permanency outcomes recognizes that 

all forms of permanency represent 

equally successful outcomes for 

children. Although all permanency 

outcomes are included within this one 

indicator, states will still be able to 

analyze their data to determine which 

types of permanency they are achieving 

for children. The indicator’s expanded 

population recognizes the Children’s 

Bureau’s desire to measure performance 

for all children entering foster care 

rather than first removals only. The 

expansion to 12 months, as opposed to 

6 months in the prior indicator will 

yield more stable estimates of 

performance. A 12 month period is 

important for this indicator as this 

cohort will also serve as the basis for the 

denominator in the re-entry into foster 

care indicator (discussed further below). 

Re-entry into foster care after a 

discharge from foster care is a rarer 

event that is better captured over a 

longer period to accommodate 

variability. In addition, including a full 

12 month period lessens the effect of 

potential seasonal differences between 6 

month periods. 

Please see the section on program 

improvement plans for more 

information on how this indicator may 

be used in program improvement. 

Proposed Permanency Performance 

Area 2: Permanency in 12 Months for 

Children in Foster Care for 2 Years or 

More 

Indicator Definition: Of all children in 

foster care the first day of the year who 

had been in foster care (in that episode) 

for 2 years or longer, what percent 

discharged to permanency within the 

next 12 months? The denominator is the 

number of children in foster care on the 

first day of a 12-month period who had 

been in foster care (in that episode) for 

2 or more years, and the numerator is 

the number of children in the 

denominator who discharged to 

permanency within 12 months of the 

first day. Permanency includes 

discharge to parent(s), living with 

relatives, guardianship, or adoption. 

Children in foster care for less than 8 

days are excluded from this indicator. 

This indicator is calculated using data 

from AFCARS. 

Justification for Inclusion: The 

Children’s Bureau is committed to 

maintaining focus on the key outcome 

of achieving permanency for all 

children in foster care and shortening 

the time to permanency. With a second 

indicator of permanency in 12 months 

specifically focused on the cohort of 

children who have been in foster care 

for 2 or more years, the Children’s 

Bureau can monitor the effectiveness of 

the state child welfare agency in 

ensuring that states continue to 

emphasize permanency for children 

who have been in foster care for longer 

periods of time, to ensure they achieve 

permanency during the period under 

review. 

Relevant Issues and Discussion: 

During CFSR Round 2, this same 

individual measure was evaluated as a 

part of a composite. There are 

substantial numbers of children that 

cannot achieve permanency in 12 

months, and those children may present 

different challenges. Such children may 

have a higher likelihood of achieving 

permanency goals such as adoption and 

guardianship than those achieving 

permanency in the first year. The 

Children’s Bureau continues this 

measure because of a commitment to 

monitor performance for children and 

youth who were already in foster care, 

and hold states accountable for attaining 

permanency for them, as well. 

Proposed Permanency Performance 

Area 3: Re-Entry to Foster Care 

Indicator Definition: Of all children 

who entered foster care in a 12-month 

period who were discharged within 12 

months of that entry to reunification, 

living with a relative, or guardianship, 

what percent re-entered foster care 

within 12 months of their discharge? 

The denominator is the number of 

children who entered foster care in a 12- 

month period who discharged within 12 

months to reunification, living with a 

relative, or guardianship, and the 

numerator is the number of children in 

the denominator who re-entered foster 

care within 12 months of their discharge 

from foster care. Children in foster care 

for less than 8 days are excluded from 

this indicator. If a child re-enters foster 

care multiple times within 12 months of 

when they left, only the first re-entry 

into foster care is selected. This 

indicator is calculated using data from 

AFCARS. 

Justification for Inclusion: Although 

the Children’s Bureau believes that it is 

important to reunify children with their 

families as quickly as possible, we also 

believe that children should be 

reunified when safe and appropriate 

and with sufficient supports in place to 

prevent a subsequent removal. This 

indicator enables the Children’s Bureau 

to monitor the effectiveness of programs 

and practice that support reunification 

and other permanency goals. 

Relevant Issues and Discussion: 

During CFSR Round 2, this performance 

area was evaluated using a similar 

measure as a part of a composite. That 

measure was derived by calculating 

what percent of all children discharged 

from foster care to reunification or 

living with a relative in a 12-month 

period re-entered foster care in less than 

12 months from the date of discharge. 

This indicator differs from the measure 

used for CFSR Round 2 by limiting the 

children eligible for re-entry to the entry 

cohort. The CFSR Round 2 measure 

counted all children who left foster care 

to reunify or live with a relative, 

regardless of when they entered foster 

care. The purpose of this focus on 

current practice is in keeping with the 

rationale that new interventions may 

best be monitored in an entry cohort. 

We also expanded the denominator to 

include children who leave foster care 

for guardianship in an effort to reflect a 

more comprehensive definition of 

permanency. 

The Children’s Bureau made this 

change in response to suggestions from 

stakeholders with regard to the 

indicators used for the last round of 

reviews that we are now able to address. 

The indicator attempts to capture the 

rate of ‘‘permanency’’ for children who 

leave foster care by measuring whether 

children re-enter foster care. For this 

indicator, adoption is not included as a 

permanency outcome because it is not 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22609 

8Rates are calculated per day of foster care. 

However, we will multiply the rate by 1,000 to 

produce larger and more meaningful numbers. 

always possible to identify children 

who re-enter foster care following 

adoption. 

Please see the section on program 

improvement plans for more 

information on how this indicator may 

be used in program improvement. 

Proposed Permanency Performance 

Area 4: Placement Stability 

Indicator Definition: Of all children 

who enter foster care in a 12-month 

period, what is the rate of placement 

moves per day of foster care? 8 The 

denominator is, among children who 

enter foster care in a 12-month period, 

the total number of days these children 

were in foster care as of the end of the 

12-month period. The numerator is, 

among children in the denominator, the 

total number of placement moves during 

the 12-month period. The initial 

placement in foster care is not counted, 

but all subsequent moves occurring 

within the 12-month period are 

included in the calculation. Children in 

foster care for less than 8 days and other 

settings a child may be placed in, that 

are not considered as placement settings 

for AFCARS purposes such as trial 

home visits, are not included in this 

indicator. This indicator is calculated 

using data from AFCARS. 

Justification for Inclusion: Placement 

stability is a critical component of the 

permanency and well-being of children 

in foster care. States are responsible for 

ensuring that children who are removed 

from their homes by the state experience 

stability while they are in foster care. 

Relevant Issues and Discussion: 

During CFSR Round 2, this performance 

area was evaluated using a similar 

measure as a part of a composite. That 

measure was derived from calculating 

what percent of all children served in 

foster care during the 12-month period 

had two or fewer placement settings. 

The new indicator controls for the 

length of time children spend in foster 

care so that we can examine moves per 

day of foster care, rather than children 

as the unit of analysis, as was employed 

during CFSR Round 2. The measure 

used for CFSR Round 2 was unable to 

differentiate between children who 

moved twice from children who moved 

more. The new indicator counts each 

move to accurately capture the rate of 

placement moves, rather than the 

number of children affected. 

The Children’s Bureau believes that 

placement stability is important to the 

permanency and well-being of children 

in foster care regardless of how long 

they have been in foster care. Even so, 

our analysis of AFCARS data indicates 

that most placement moves occur 

within a child’s first 12 months of foster 

care, which is why we plan to focus this 

indicator on that time period. With this 

refined focus, the Children’s Bureau and 

states can monitor the period during 

which placement moves are most likely 

to occur and the state’s most recent 

performance. In the CFSR Round 2 

measure, placement moves were 

monitored over the life of the case 

which meant that placement instability 

for a child in the early years of foster 

care placement would affected the 

assessment of the state’s CFSR 

performance in a more recent period 

under review. We also believe that by 

confining the indicator to this period of 

time, we are better able to measure a 

state’s improvement in a subsequent 12- 

month period. The Children’s Bureau 

made this change in response to 

suggestions from stakeholders with 

regard to the indicators used for the last 

round of reviews that we are now able 

to address. 

National Standards 

National standards will be established 

for all indicators. By measuring state 

performance against national standards 

on statewide data indicators, the 

Children’s Bureau can assist states in 

continuously monitoring their 

performance on child outcomes and 

help practitioners and administrators 

better understand the entirety of their 

child welfare systems. 

We propose that the national standard 

for each indicator be set at the national 

observed performance for that particular 

indicator. The national standards will 

remain constant over the entire round of 

review, as has been the case in prior 

rounds. In CFSR round 2, national 

standards were based on the 75th 

percentile (approximately) of all states’ 

performance, with an adjustment for 

sampling error. For this round, we 

believe that the national observed 

performance, which will be similar to 

the average performance across all 

states, is a more reasonable benchmark 

and would appropriately challenge 

states to improve their performance. 

The national standard set at the 

national performance level for each 

indicator is a benchmark that is easily 

communicated to and understood by 

stakeholders, and a reasonable goal 

given the reality that states still need to 

improve practice in multiple areas. 

Setting the national standard at the 

national performance for each indicator 

is rooted in strategies central to an 

effective performance management 

system focused on continuous quality 

improvement. 

Methodology: We propose that state 

performance on each statewide data 

indicator be assessed using a multi-level 

(i.e., hierarchical) model that riskadjusts for select child- and state-level 

characteristics. Multi-level modeling is 

a widely accepted statistical method 

that enables fair evaluation of relative 

performance among states with different 

case mixes. The technique calculates 

how much variance in performance is 

due to (1) children’s individual risk 

factors; (2) random measurement errors 

(due to modest sample sizes); and (3) 

the state’s long-run ability to achieve the 

desired outcome. 

Risk Adjustment: The model we 

propose to use will incorporate some 

risk adjustment. By incorporating risk 

adjustment, the multi-level model takes 

into account and controls for factors that 

differ across the states and that can 

affect outcomes regardless of the quality 

of services the state provides. The goal 

of risk adjustment is to minimize 

differences in outcomes that are due to 

factors over which states have little 

control, such as the age of children 

coming into foster care. For example, for 

the statewide data indicator of 

permanency in 12 months for children 

entering foster care, a state may 

discharge 40% of its children to 

permanency by 12 months. Forty 

percent is the state’s observed 

performance, and is simply the number 

of children discharged to permanency 

by 12 months divided by the number of 

children eligible for such an exit. But 

this state’s risk-adjusted performance 

might be 45%. That the state’s riskadjusted performance is higher than its 

observed performance means 

permanency was achieved for more 

children than expected, given the state’s 

case mix and how other states, on 

average, performed with a similar case 

mix. 

The Children’s Bureau will finalize 

risk adjustment variables after receiving 

public comments on this document. The 

contractor’s recommendations to us 

based on feedback from the expert panel 

convened in 2013 support the use of 

risk adjustment. The Children’s 

Bureau’s consideration of particular 

risk-adjustment variables will be based 

initially on the research literature, 

recommendations based on feedback 

from the expert panel and expert 

consultants, and the availability of data. 

The Children’s Bureau will test 

proposed variables and retain only those 

variables that have a statistically 

significant relationship to the outcome 

for each statewide data indicator. For 

example, the Children’s Bureau has 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
22610 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

9 Yale New Haven Health Services Corporation, & 

Center for Outcomes Research & Evaluation. (2013). 

2013 Measures Updates and Specifications Report: 

Hospital-Level 30-Day Risk-Standardized 

Readmission Measures for Acute Myocardial 

Infarction, Heart Failure, and Pneumonia (Version 

6.0). Retrieved from http://www.qualitynet.org/dcs/ 

ContentServer?c=Page&pagename=QnetPublic/

Page/QnetTier4&cid=1219069855841. 

10See Goldstein & Spiegelhalter (2007). League 

Tables and Their Limitations. 159(3), 385–443; 

Normand & Shahian (2007). Statistical and Clinical 

Aspects of Hospital Outcomes Profiling. Statistical 

Science, 22(2), 206–226. 

11This data quality analysis is specific to 

indicators proposed for CFSR Round 3. It is a 

separate analysis from the standard data quality 

checks or validation that are performed when states 

submit their AFCARS or NCANDS data. 

12For example, the date of birth is greater than 

the date of latest removal. 

tested for possible inclusion child’s age, 

sex, number of prior removals, and 

interactions among these with the 

proposed statewide data indicators. The 

Children’s Bureau will consider riskadjustment variables at both the level of 

the child (e.g., age at entry) and the state 

(e.g., foster care entry rate). 

Assessing State Performance 

To assess state performance, the 

Children’s Bureau proposes to estimate 

each state’s risk-adjusted performance 

and the corresponding 95% interval 

estimate. The Children’s Bureau can be 

95% confident that a state’s true 

performance lies somewhere between 

the lower and upper limit of this 

interval. This interval also provides a 

way to judge whether a state’s 

performance is above or below the 

national average in a statistically 

meaningful way. 

The Children’s Bureau plans to 

compare each state’s interval estimate to 

the national observed performance, and 

assign each state to one of three groups: 

• ‘‘No different than national 

performance’’ if the 95% interval 

estimate surrounding the state’s riskadjusted performance includes the 

national observed performance. 

• ‘‘Higher than national 

performance’’ if the entire 95% interval 

estimate surrounding the state’s riskadjusted performance is higher than the 

national observed performance. 

• ‘‘Lower than national performance’’ 

if the entire 95% interval estimate 

surrounding the state’s risk-adjusted 

performance is lower than the national 

observed performance. 

Whether it is desirable for a state to 

be higher or lower than the national 

performance depends on the indicator. 

For the two permanency measures, a 

higher value is more desirable; for the 

remaining measures, a lower value is 

desirable. 

The methodology described above is 

similar to that used by the Centers for 

Medicare & Medicaid Services to 

measure hospital performance as part of 

its Hospital Inpatient Quality Reporting 

program.9 The methodology is also 

consistent with the use of such models 

in education and health care to 

distinguish statistically high- and lowperforming schools and hospitals.10

Data 

Data Profiles: The Children’s Bureau 

will provide data profiles of state 

performance to each state before the 

state’s CFSR on the statewide data 

indicators and other contextual data 

available from AFCARS and NCANDS. 

This data profile will assist the state to 

develop its statewide assessment. In 

addition, the Children’s Bureau will 

provide data profiles semi-annually to 

assist states in measuring progress 

toward the goals identified in the 

program improvement plan. 

Data Quality: Setting national 

standards and measuring state 

performance on statewide data 

indicators for CFSR purposes relies 

upon the states submitting high-quality 

data to AFCARS and NCANDS. The 

Children’s Bureau has provided states 

with consultation and technical 

assistance before and throughout both 

rounds of reviews to address data 

quality issues. Additionally, the 

Children’s Bureau has provided states 

with tools for AFCARS and NCANDS 

that allow the agency to examine its 

data for accuracy and encourages states 

to incorporate these in their ongoing 

quality assurance process to review 

data. 

During the first two rounds of the 

CFSR, there have been occasions in 

which the Children’s Bureau cannot use 

a state’s data in aggregate calculations of 

the national standard. In isolated 

circumstances, these data quality issues 

have been significant enough to prevent 

us from relying on the state’s data as an 

accurate assessment of its performance 

on a statewide data indicator. The 

Children’s Bureau would like to be clear 

about the level of data quality issues 

that prevent state data from being used 

for CFSR purposes as described below. 

Data Quality: Excluding States From 

National Standards or State 

Performance 

We analyzed every data element from 

AFCARS and NCANDS that is relevant 

to each statewide data indicator (as 

listed in Attachment C) and performed 

data quality checks across files both 

over time as well as between files.11

Examples of these checks included 

looking for the presence of the same 

child identifier in the AFCARS and 

NCANDS file and reviewing for 

consistent reporting of a child in 

AFCARS from the time the child 

entered foster through discharge and 

with an associated reason for discharge. 

This analysis revealed the scope of data 

quality issues present in current 

AFCARS and NCANDS submissions. 

Based on this analysis, we developed 

thresholds to identify data quality 

concerns and either accept or exclude 

the files when calculating national 

standards and state performance. For 

those data quality issues that are 

contained to one data file submission, 

we will consider a threshold of 5%. In 

other words, any state that has more 

than 5% of data missing or invalid 12

will be excluded from the model used 

to calculate the national standard (i.e., 

the national observed performance) and 

estimate states’ risk-adjusted 

performance. For cross-file checks, we 

are setting a higher threshold for 

exclusion. For the maltreatment in 

foster care measure, a state will be 

excluded from the national standards 

calculation and performance estimate if 

more than 10% of NCANDS victims are 

missing an AFCARS identifier. For the 

permanency indicators, a state will be 

excluded if it has more than 10% of 

dropped cases across two six month file 

submissions. 

Data Quality: Case-Level Exclusions 

For those states that do not exceed the 

data quality thresholds but still have 

identified data quality problems, we 

will include the state in national 

standards calculations and measure 

state performance but we will exclude 

child-level records with missing or 

invalid data on elements needed to 

determine the child’s outcome and 

perform the risk-adjustment. For 

example, if the risk-adjustment for an 

indicator includes age at entry, a child 

whose age at entry cannot be 

determined (due to a missing date of 

birth) will not be include in the 

analysis. We believe this exclusion will 

result in more accurate estimates of 

performance for those states with minor 

data quality issues. For each indicator, 

the Children’s Bureau will provide each 

state with a list of records that were 

excluded from the analyses. 

Program Improvement Plans 

States that fall below the national 

standard on any given indicator will be 

required to include that indicator in a 

program improvement plan. Regardless 

of which indicators a state is required to 

include in its program improvement 

plan, the Children’s Bureau will provide 

each state with a data profile that 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22611 

13These guidelines are based on a theorem 

known as Chebyshev’s inequality. When all you 

have is an overall mean and standard deviation 

(which is what we have for each state’s 

performance), the theorem guarantees that a certain 

percent of values will be k standard deviations 

away from the mean. Specifically, at least 75% of 

the values will be within two standard deviations 

of the mean, at least 89% within three standard 

deviations, at least 94% within four standard 

deviations, at least 96% within five standard 

deviations, and at least 97% within six standard 

deviations (Chong et al. (2012). Improving 

Generalization Performance in Co-Evolutionary 

Learning. IEEE, vol. 16, no. 1, 70–85; Sheppard 

(2011). Environmental Study-Soil Sample Analysis 

for the Department of Ecology at Hanford). 

includes information on the state’s 

performance on all of the statewide data 

indicators. 

Companion Measures: For two of the 

statewide data indicators, permanency 

in 12 months for children entering foster 

care and re-entry to foster care, the 

Children’s Bureau proposes to consider 

performance for program improvement 

purposes in concert with the other 

indicator. This means that if a state has 

a program improvement plan that 

includes improving on the indicator 

permanency in 12 months for children 

entering foster care, the Children’s 

Bureau’s determination of whether the 

state has improved successfully will 

take into consideration its performance 

on the re-entry to foster care indicator 

as a companion measure. Specifically, 

the state must stay above a threshold for 

the companion re-entry to foster care 

indicator as well as achieve its goal on 

the permanency in 12 months for 

children entering foster care indicator, 

to successfully complete the program 

improvement plan. The reverse is also 

true. For details about threshold 

calculations, please see the section 

below. If a state must improve on the reentry to foster care indicator in its 

program improvement plan, it must also 

include the permanency in 12 months 

for children entering foster care 

indicator as a companion measure. 

Although the Children’s Bureau believes 

that it is important to reunify children 

with their families as quickly as 

possible, we also believe that children 

should be reunified when safe and 

appropriate and sufficient supports are 

in place to prevent a subsequent 

removal. The recommendations to us 

based on the expert panel convened in 

2013 also support the use of companion 

measures in program improvement. 

Methods for Setting State 

Performance Baselines, Goals, and 

Thresholds: The key components for 

setting performance goals and 

monitoring progress over the course of 

a program improvement plan involve 

calculating baselines, goals, and 

thresholds. The Children’s Bureau 

methodology for specifying state 

improvement goals is statistically 

grounded, producing goals that are 

programmatically challenging, yet 

attainable. In addition, goals and 

thresholds should reflect each state’s 

own performance history and 

demonstrated capacity for change. In 

CFSR Round 2, the percent of 

improvement required for each state 

was the same for all states, and was 

applied to state-specific baselines. 

While this standardization was easy to 

communicate how it applied to each 

state, it also meant that high-performing 

states were asked to make larger 

absolute improvements than lowerperforming states. This approach did 

not take into account the variability in 

performance that states have shown 

over time. For CFSR Round 3, the 

Children’s Bureau plans to set 

improvement goals relative to each 

state’s past performance. 

Setting Baselines: The Children’s 

Bureau plans to set the baseline for each 

statewide data indicator included in a 

program improvement plan at the state’s 

observed performance on that indicator 

for the most recent year of data available 

before the beginning of program 

improvement plan implementation. We 

also considered using the state’s threeyear average in this calculation, but that 

approach occasionally produced goals 

that fell below the state’s performance 

in the most recent year. Consistent with 

CFSR Round 2, we will use the most 

recent year as the baseline, because it 

represents a reasonable estimate of the 

state’s current performance. 

Setting Goals and Thresholds: We 

plan to set performance goals and 

thresholds will be based on the 

variability in the state’s observed 

performance in the three most recent 

years of data. There will be 

improvement factors, as in CFSR Round 

2, but these will be driven by the 

variability in performance that the 

particular state has shown in the last 

three years. We will apply the 

improvement factors to the baseline to 

produce the concrete performance goal. 

The state’s amount of improvement 

required for a program improvement 

plan will be more than what is likely, 

in a statistical sense, to occur by chance 

alone. Conversely, we plan to set 

thresholds as the inverse of goals, which 

will identify a point by which a state is 

demonstrating a performance decline for 

companion measures that is more 

statistically than what might be 

expected by chance. 

We will use a technique called 

bootstrapping to develop goals and 

thresholds. The method calls for the 

Children’s Bureau to repeatedly sample 

a state’s past three years of performance 

estimates to construct a larger sample, 

and from that the calculation of a grand 

mean and standard deviation. The grand 

mean reflects that state’s ‘‘average’’ 

performance and the standard deviation 

reflects how much normal fluctuation in 

performance the Children’s Bureau 

might expect for that state, given its past 

performance. Then the standard 

deviation is used to calculate an 

estimate that would represent a level of 

change above and beyond the typical 

fluctuation that would otherwise be 

expected. The Children’s Bureau will 

set the magnitude at four standard 

deviations from the grand mean. At that 

level we can say with confidence that— 

if we were to randomly estimate a state’s 

performance on the indicator (using 

their past performance), and did so 100 

times—we would expect to see 

performance at this level less than 6% 

of the time (or fewer than 6 times out 

of 100). Six times out of 100 is rather 

rare, which is why we can treat it as 

representing a statistically meaningful 

change has occurred in the program.13

To determine exactly how much a 

state will need to improve, we must first 

calculate an improvement factor, which 

is the percentage difference between the 

grand mean and four standard 

deviations above the grand mean. We 

then apply that to the baseline, which 

is the observed performance in the most 

recent year. To demonstrate a sample 

calculation: 

A state may have a grand mean of 50%, a 

grand mean plus four standard deviations = 

52%, and a year 3 value of 51. This will give 

us an improvement factor of 52/50 = 1.04. If 

that is applied to the baseline of 51%, the 

program improvement plan goal will be 51% 

× 1.04 = 53.04%. 

We will use a comparable technique 

to set thresholds for companion 

measures, subtracting rather than 

adding four standard deviations to the 

grand mean (when higher performance 

on an indicator is better), which can be 

used to identify a state’s decline in 

performance. To provide an example, if 

a goal was calculated to be three 

percentage points higher than the 

baseline percent, the threshold would 

be three percentage points below it. 

Thresholds are only relevant to 

companion measures. 

By design, states with less variation in 

performance from year to year have 

more modest goals, while those showing 

greater variation have more aggressive 

goals. Overall, we believe that the goals 

are reflective of each state’s own prior 

experience and performance levels, with 

goals that are achievable and 

substantively meaningful. We 

acknowledge that a few states with the 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
22612 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

lowest variation in performance for each 

indicator are assigned very modest 

goals, while a few states with the 

highest variation in performance are 

assigned very aggressive goals. 

To address these problems at the 

extreme ends, we propose to establish 

minimum and maximum improvement 

factors. Specifically, a floor will be set 

at the top of the bottom fifth, and the 

bottom of the top fifth, ordered by the 

size of the improvement factor. While 

the impact of this rule varies somewhat 

from indicator to indicator, overall we 

believe it provides a consistent basis for 

producing goals that are achievable and 

substantively meaningful. The inverse 

would be done for the thresholds. The 

contractor’s recommendations to us 

based on the feedback from the expert 

panel convened in 2013 support the 

setting of maximum and minimum 

thresholds for improvement goals at the 

level of performance of top and bottom 

quintiles. 

Process for Final Indicators 

We are interested in comments on all 

aspects of the statewide data indicators 

proposed and the methods to calculate 

national standards and program 

improvement. After considering the 

feedback to this docuemnt, we plan to 

publish a final list of indicators and 

methods that will be used in the CFSRs 

along with the actual national 

standards. 

(Authority: 42 U.S.C. 1320a–1a; 45 CFR 

1355.31–37.) 

Mark Greenberg, 

Acting Commissioner, Administration on 

Children, Youth and Families. 

Attachment A: Proposed Statewide 

Data Indicators 

Category Measure title Measure 

description Denominator Numerator Exclusions Notes 

Safety ....... Maltreatment in 

foster care.

Of all children in 

foster care during 

a 12-month period, what is the 

rate of victimization per day of 

foster care? 

(From AFCARS) 

Among children 

in foster care 

during a 12- 

month period, 

total number of 

days these children were in foster care as of the 

end of the 12- 

month period a.

(From NCANDS) 

Among children 

in the denominator, total number of substantiated or indicated reports of 

maltreatment (by 

any perpetrator) 

during the 12- 

month period b.

—If a state provides incident 

dates, records 

with an incident 

date occurring 

before the date 

of removal will be 

excluded.

—Children in foster 

care <8 days.

—Any report that 

occurs within the 

first 7 days of removal.

Cases are matched 

across AFCARS 

and NCANDS 

using AFCARS 

ID. 

Safety ....... Re-report of maltreatment.Of all children with 

a screened-in report of alleged 

maltreatment in a 

12-month period 

(regardless of 

disposition), what 

percent had another screened-in 

report within 12 

months of their 

initial report? 

(From NCANDS) 

Number of children with at least 

one screened-in 

report of alleged 

maltreatment in a 

12-month period.

(From NCANDS) 

Number of children in the denominator that 

had another 

screened-in report within 12 

months of their 

initial report.

None ...................... Reports are included regardless of the type 

of disposition, so 

this indicator includes both victims and non-victims. 

Permanency.Permanency in 12 

months for children entering 

foster care.

Of all children who 

enter foster care 

in a 12-month 

period, what percent discharged 

to permanency 

within 12 months 

of entering foster 

care? c

(From AFCARS) 

Number of children who enter 

foster care in a 

12-month period.

(From AFCARS) 

Number of children in the denominator who 

discharged to 

permanency 

within 12 months 

of entering foster 

care or by the 

time they 

reached 18.

—Children in foster 

care <8 days.

—Children who 

enter foster care 

at age 18 or 

more.

Youth who turn 18 

while in foster 

care who were 

included in the 

denominator will 

not be counted 

as having 

achieved permanency, regardless of discharge 

reason. 

Permanency.Permanency in 12 

months for children in foster 

care 2 years or 

more.

Of all children in 

foster care on 

the first day of a 

12-month period, 

who had been in 

foster care (in 

that episode) for 

2 or more years, 

what percent discharged to permanency within 

12 months of the 

first day? 

(From AFCARS) 

Number of children in foster 

care on the first 

day of a 12- 

month period 

who had been in 

foster care (in 

that episode) for 

2 or more years.

(From AFCARS) 

Number of children in the denominator who 

discharged to 

permanency 

within 12 months 

of the 1st day or 

by the time they 

reached 18.

—Children age 18 

or more on the 

first day of the 

year.

Youth who turn 18 

while in foster 

care who were 

included in the 

denominator will 

not be counted 

as having 

achieved permanency, regardless of discharge 

reason. 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22613 

Category Measure title Measure 

description Denominator Numerator Exclusions Notes 

Permanency.Re-entry to foster 

care.

Of all children who 

enter foster care 

in a 12-month 

period who discharged within 

12 months to reunification, live 

with relative, or 

guardianship, 

what percent reentered foster 

care within 12 

months of their 

discharge? a

(From AFCARS) 

Number of children who enter 

foster care in a 

12-month period 

who discharged 

within 12 months 

to reunification, 

live with relative, 

or guardianship.

(From AFCARS) 

Number of children in the denominator who 

re-enter foster 

care within 12 

months of their 

discharge.

—Children in foster 

care < 8 days.

—Children who 

enter or exit foster care at age 

18 or more.

If a child has multiple re-entries to 

foster care within 

12 months of 

their discharge, 

only his first reentry is selected. 

Permanency.Placement Stability Of all children who 

enter foster care 

in a 12-month 

period, what is 

the rate of placement moves per 

day of foster 

care? 

(From AFCARS) 

Among children 

who enter foster 

care in a 12- 

month period, 

total number of 

days these children were in foster care as of the 

end of the 12- 

month period d.

(From AFCARS) 

Among children 

in the denominator, total number of placement 

moves during the 

12-month period e.

—Children in foster 

care < 8 days.

—Children who 

enter foster care 

at age 18 or 

more.

The initial removal 

from home (and 

into foster care) 

is not counted as 

a placement 

move. 

aFor example, if during the 12-month period there were two children in foster care, one child for 10 days (1st episode), the same child for 40 

days (2nd episode), and the other child for 100 days (his only episode), the denominator would = 150 days (10+40+100). bFor example, if during the 12-month period there were two children in foster care, and one child with 3 substantiated or indicated reports and 

the other with 1 such report, the numerator would = 4 reports (3+1). c If a child has multiple entries during the year, only his or her first entry is selected. 

dFor example, if during the 12-month period two children entered foster care, one child for 10 days and the other child for 100 days, the denominator would be 110 days (10+100). eFor example, if during the 12-month period two children entered foster care, and one child had 3 moves and the other had 1 move, the numerator would = 4 moves (3+1). 

Attachment B: Comparison of Data 

Measures—CFSR Round 2 and Round 3 

Category Measure title Proposed CFSR round 3 

indicator 

Comparable CFSR 

round 2 measure How and why it’s changed 

Safety ....... Maltreatment in foster 

care.

Of all children in foster 

care during a 12- 

month period, what is 

the rate of victimization per day a of foster 

care? 

Of all children in foster 

care during the reporting period, what percent were not victims 

of substantiated or indicated maltreatment 

by a foster parent or 

facility staff member? 

In the CFSR 2 measure, counts of children not 

maltreated in foster care are derived by subtracting the NCANDS count of children maltreated by foster care providers from the total 

count of all children placed in foster care, as 

reported in AFCARS. Because of improved reporting by states, we now link AFCARS and 

NCANDS data using the child ID and determine if maltreatment occurred during a foster 

care episode, improving accuracy on the indicator. 

This also allows us to expand the measure to include all types of perpetrators (including, for 

example, parents) under the assumption that 

states should be held accountable for keeping 

children safe from harm while in the care of the 

state, no matter who the perpetrator is. 

Safety ....... Re-report of maltreatment.Of all children with a 

screened-in report of 

alleged maltreatment 

in a 12-month period, 

what percent had another screened-in report within 12 months 

of their initial report? 

Of all children who were 

victims of substantiated or indicated 

maltreatment allegation during the first 6 

months of the reporting period, what percent were not victims 

of another substantiated or indicated 

maltreatment allegation within a 6-month 

period? 

We are expanding the measure to count all children with screened-in reports of alleged maltreatment, because research suggests children 

with prior reports are at greater risk. 

In addition, by limiting only to victims, we could 

face measurement challenges as states implement Differential Response during a monitoring 

cycle, which could have an impact on numbers 

of substantiations. 

We also propose using a full 12-month period 

rather than only 6 months to capture the denominator, to create more stable estimates. 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
22614 Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 

Category Measure title Proposed CFSR round 3 

indicator 

Comparable CFSR 

round 2 measure How and why it’s changed 

Permanency.Permanency in 12 

months for children 

entering foster care.

Of all children who enter 

foster care in a 12- 

month period, what 

percent discharged to 

permanency within 12 

months of entering 

foster care? 

Composite 1.3: Of all 

children entering foster care for the first 

time in a 6-month period, what percent discharged to reunification (or live with relative) within 12 

months of entering 

foster care or by the 

time they reached 18? 

We now count all types of permanency (reunification, live with relative, adoption or guardianship) as having ‘met’ the indicator. 

We also expanded the measure to include all 

children who entered foster care that year; not 

just those on their first removal episode. 

We also expanded the window of time for the 

entry cohort to a full year instead of 6 months; 

this will yield more stable estimates. 

Permanency.Permanency in 12 

months for children in 

foster care for 2 years 

or more.

Of all children in foster 

care on the first day of 

a 12-month period 

who had been in foster care (in that episode) for 2 or more 

years, what percent 

discharged to permanency within 12 

months of the first 

day? 

Composite 3.1: Of all 

children in foster care 

on the first day of a 

12-month period who 

had been in foster 

care (in that episode) 

for 2 or more years, 

what percent discharged to permanency within 12 

months of the first day 

or by the time they 

reached 18? 

Same measure; no change. The difference is that 

it is now evaluated on its own, rather than as 

just one part of a composite measure. 

By including the Entry Cohort Permanency indicator (listed above) as well as the Legacy Cohort Permanency indicator, we hold states accountable not only for children in their first year 

of foster care, but also those children/youth 

who have been in foster care for long periods 

of time. 

Permanency.Re-entry in 12 months ... Of all children who enter 

foster care in a 12- 

month period and discharged within 12 

months to reunification, live with relative, 

or guardianship, what 

percent re-entered 

foster care within 12 

months of their date of 

discharge? 

Composite 1.4: Of all 

children discharged 

from foster care to reunification or live with 

a relative in a 12- 

month period, what 

percent re-entered 

foster care in less 

than 12 months from 

the date of discharge? 

The new indicator is limited to those children who 

entered foster care during the year, whereas 

the CFSR Round 2 measure counted all children who discharged to reunification or live 

with relative, regardless of when they entered 

foster care. The purpose of this focus is in 

keeping with the rationale that new interventions may best be monitored in an entry cohort. 

This indicator will also be used as a companion measure with permanency in 12 

months, to ensure that states working to improve permanency rates in their entry cohort 

do not see worsening performance on rates of 

re-entry to foster care. 

We also expanded the denominator to allow discharges to guardianship, in an effort to capture 

more discharges to permanency. Exits to adoption are not included because they cannot be 

tracked reliably, as some states issue new 

child identifiers if a child who was previously 

adopted enters foster care. 

Permanency.Placement stability ........ Of all children who enter 

foster care in a 12- 

month period, what is 

the rate of placement 

moves per day b of 

foster care? 

Composite 4.1: Of all 

children served in foster care during the 12- 

month period, what 

percent had two or 

fewer placement settings? 

The proposed indicator controls for length of time 

in foster care, so we are looking at moves per 

day of foster care, rather than children as the 

unit of analysis. 

The rationale for using an entry cohort rather 

than all children served is that our analysis 

shows children entering foster care tend to 

move much more than those children/youth in 

foster care for longer periods of time, whose 

placements may have stabilized. 

In CFSR Round 2 measure, moves that took 

place prior to the monitoring period were 

counted. Now we only count those moves that 

occur during the monitoring period. The initial 

placement is not counted. 

The CFSR Round 2 measure treated children 

who moved 2 times in an episode the same as 

children who moved 15 times; both were a failure to meet the measure. The new indicator 

counts each move, so it continues to hold 

states accountable for those children/youth 

who have already moved several times. 

aThe rate may be expressed per 100,000 days because it is such a rare event. Using this metric gives us numbers greater than zero, which 

are easier to communicate. bThe rate is expressed per 1,000 days to convert the rate to a metric that gives us numbers greater than zero. 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS
Federal Register / Vol. 79, No. 78 / Wednesday, April 23, 2014 / Proposed Rules 22615 

Attachment C: Data Elements Required 

for Statewide Data Indicators 

For instruction regarding AFCARS 

data elements, refer to http://

www.acf.hhs.gov/programs/cb/resource/

afcars-tb1. 

For instruction with regard to 

NCANDS data elements, refer to 

http://www.ndacan.cornell.edu/

datasets/pdfs_user_guides/178- 

NCANDS-child2012v1-User-Guide-andCodebook.pdf. 

Primary data elements required for calculation Permanency 

by 12 months 

Re-entry by 12 

months 

Placement 

stability 

Re-report of 

maltreatment 

Maltreatment 

in foster care 

AFCARS FC Element #1: a Title IV–E Agency .................... ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #4: Record Number ......................... ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #21: Date of Latest Removal ........... ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #23: Date of Placement in Current 

Foster Care Setting .......................................................... NA NA ✓ NA NA 

AFCARS FC Element #24: Number of Placement Settings 

during this Removal Episode ........................................... NA NA ✓ NA NA 

AFCARS FC Element #56: Date of Discharge from FC ..... ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #58: Reason for Discharge .............. ✓ ✓ NA NA NA 

NCANDS CF Element #4: Child ID ..................................... NA NA NA ✓ NA 

NCANDS CF Element #6: Report Date ............................... NA NA NA ✓ ✓ 

NCANDS CF Element #27: Child Maltreatment 1—Disposition Level b ................................................................. NA NA NA NA ✓ 

NCANDS CF Element #29: Child Maltreatment 2—Disposition Level ................................................................... NA NA NA NA ✓ 

NCANDS CF Element #31: Child Maltreatment 3—Disposition Level ................................................................... NA NA NA NA ✓ 

NCANDS CF Element #33: Child Maltreatment 4—Disposition Level ................................................................... NA NA NA NA ✓ 

NCANDS CF Element #34: Maltreatment death ................. NA NA NA NA ✓ 

NCANDS CF Element #145: AFCARS ID ........................... NA NA NA NA ✓ 

aThe elements are numbered by their position in the flat ASCII files submitted by states to these reporting systems. These numbering schema 

are specific to the files utilized by ACYF. Files obtained through the National Data Archive on Child Abuse and Neglect (NDACAN) may have a 

slightly different order. b Definition of ‘victim’ includes all children with a disposition level (for any of up to four maltreatments per child) of: a) Substantiated, or b) Indicated. These do not propose including differential response victims. Victims also include children who died as a result of maltreatment. 

Additional data elements required for risk-adjusted 

Analysis c

Permanency 

by 12 months 

Re-entry by 12 

months 

Placement 

stability 

Re-report of 

maltreatment 

Maltreatment 

in foster care 

AFCARS FC Element #6: Child’s Date of Birth .................. ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #7: Child Sex ................................... ✓ ✓ ✓ NA ✓ 

AFCARS FC Element #19: Total # of Removals ................ ✓ ✓ ✓ NA ✓ 

NCANDS CF Element #14: Child Age ................................ NA NA NA ✓ NA 

NCANDS CF Element #17: Child Sex ................................. NA NA NA ✓ NA 

US Census Bureau: Child Population, by State .................. ✓ ✓ NA ✓ ✓ 

c In addition to those data elements used for risk adjustment, a few more are used to make necessary adjustments to outcomes; for example, 

we use the child’s current placement setting to determine if he or she is in Trial Home Visit before Reunification and, if so, time in foster care is 

adjusted consistent with adjustments for trial home visits used in CFSR Round 2. If a state provides NCANDS CF Element #146, Incident Date, 

an adjustment will be made to the maltreatment in foster care indicator to improve accuracy. 

[FR Doc. 2014–09001 Filed 4–22–14; 8:45 am] 

BILLING CODE 4184–25–P 

DEPARTMENT OF DEFENSE 

GENERAL SERVICES 

ADMINISTRATION 

NATIONAL AERONAUTICS AND 

SPACE ADMINISTRATION 

48 CFR Parts 4, 14, 15, and 52 

[FAR Case 2014–001; Docket No. 2014– 

0001; Sequence No. 1] 

RIN 9000–AM78 

Federal Acquisition Regulation; 

Incorporating Section K in Contracts 

AGENCIES: Department of Defense (DoD), 

General Services Administration (GSA), 

and National Aeronautics and Space 

Administration (NASA). 

ACTION: Proposed rule. 

SUMMARY: DoD, GSA, and NASA are 

proposing to amend the Federal 

Acquisition Regulation (FAR) to 

standardize the incorporation by 

reference of representations and 

certifications in contracts. 

DATES: Interested parties should submit 

written comments to the Regulatory 

Secretariat at one of the addressees 

shown below on or before June 23, 2014 

to be considered in the formation of the 

final rule. 

ADDRESSES: Submit comments in 

response to FAR Case 2014–001 by any 

of the following methods: 

• Regulations.gov: http://

www.regulations.gov. Submit comments 

via the Federal eRulemaking portal by 

searching for ‘‘FAR Case 2014–001’’. 

Select the link ‘‘Comment Now’’ that 

corresponds with ‘‘FAR Case 2014– 

001.’’ Follow the instructions provided 

at the ‘‘Comment Now’’ screen. Please 

include your name, company name (if 

any), and ‘‘FAR Case 2014–001’’ on your 

attached document. 

• Fax: 202–501–4067. 

• Mail: General Services 

Administration, Regulatory Secretariat 

(MVCB), ATTN: Hada Flowers, 1800 F 

Street NW., 2nd Floor, Washington, DC 

20405. 

Instructions: Please submit comments 

only and cite FAR Case 2014–001, in all 

correspondence related to this case. All 

comments received will be posted 

without change to http://

www.regulations.gov, including any 

personal and/or business confidential 

information provided. 


wreier-aviles on DSK5TPTVN1PROD with PROPOSALS