Race to the Top-Early Learning 

Challenge: Descriptive Study of

Tiered Quality Rating and 

Improvement Systems in Nine 

Round 1 States: OMB Forms 

Clearance Package, Part A

December 18, 2014
Contract Number:

ED-IES-10-C-0077

Mathematica Reference Number:

06844.144

Submitted to:

National Center for Education Evaluation

555 New Jersey Ave., NW, Suite 500k

Washington, DC 20208

Project Officer: Tracy Rimdzius 

Submitted by:

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Susanne James-Burdumy, 

PhD

Race to the Top-Early Learning 

Challenge: Descriptive Study of

Tiered Quality Rating and 

Improvement Systems in Nine 

Round 1 States: OMB Forms 

Clearance Package, Part A

December 18, 2014
iii

CONTENTS

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 

SUBMISSION................................................................................................... 1

A. Justification................................................................................................ 2

1. Circumstances necessitating the collection of information .................. 2

2. Purposes and uses of the data.......................................................... 10

3. Use of technology to reduce burden.................................................. 11

4. Efforts to avoid duplication of effort ................................................... 12

5. Methods to minimize burden on small entities................................... 12

6. Consequences of not collecting the data........................................... 12

7. Special circumstances....................................................................... 13

8. Federal register announcement and outside consultation ................. 13

9. Payments or gifts............................................................................... 13

10. Assurances of confidentiality............................................................. 14

11. Additional justification for sensitive questions.................................... 15

12. Estimates of hours burden................................................................. 15

13. Estimates of cost burden to respondents .......................................... 16

14. Estimates of annual costs to the federal government........................ 16

15. Reasons for program changes or adjustments.................................. 16

16. Publication plans and schedule......................................................... 16

17. Approval not to display the OMB expiration date............................... 17

18. Explanation of exceptions.................................................................. 17

REFERENCES.............................................................................................................. 18

APPENDIX A: DISTRICT LETTER

APPENDIX B: STUDY INFORMATION SHEET

APPENDIX C: RECRUITING PROTOCOL WITH DISTRICT ADMINISTRATOR

APPENDIX D: DATA COLLECTION FORM FOR DISTRICT-LEVEL 

KINDERGARTEN ASSESSMENT DATA
iv

TABLES

A.1 Study Activities and Timeline ........................................................................... 9

A.2 Research Questions and Data Sources ......................................................... 10

A.3 Burden in Hours to Respondents – RTT-ELC TQRIS Study

Recruitment and Data Collection.................................................................... 15

A.4 Timetable for Project Publications.................................................................. 16
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

1

PART A. SUPPORTING STATEMENT FOR PAPERWORK 

REDUCTION ACT SUBMISSION

This supporting statement requests clearance to recruit and collect data from approximately 14 

school districts to support the Study of Race to the Top–Early Learning Challenge Tiered Quality 

Rating and Improvement Systems (RTT-ELC TQRIS). Through TQRIS, states use a set of 

standards to evaluate the quality of early learning and development programs and promote program 

improvement. The RTT-ELC application for the first round of grants required states to develop and 

adopt a TQRIS that, according to the U.S. Department of Education (ED), “is based on a statewide 

set of tiered program standards that include: (1) early learning and development standards, (2) a 

comprehensive assessment system, (3) early childhood educator qualifications, (4) family 

engagement strategies, (5) health promotion practices, and (6) effective data practices; is clear and 

has standards that are measurable, meaningfully differentiate program quality levels, and reflect high 

expectations of program excellence commensurate with nationally recognized standards that lead to 

improved learning outcomes for children; and is linked to the state licensing system for early 

learning and development programs” (2011). The study of RTT-ELC TQRIS will provide important 

information on the structure and characteristics of TQRIS and whether and how the TQRIS ratings 

relate to child outcomes. This information is intended to inform improvements to states’ TQRIS. 

The Institute of Education Sciences (IES) at ED contracted with Mathematica Policy Research to 

conduct this study, which is being carried out as a component of the Impact Evaluation of Race to 

the Top and School Improvement Grants.1

The RTT-ELC TQRIS study comprises two distinct phases. The first study phase is a 

descriptive analysis of the TQRIS in the nine Round 1 RTT-ELC states. In each state, we will 

examine the structure of the TQRIS, including the components and indicators included in states’ 

TQRIS and how they are combined to generate overall quality ratings.

2 We will also explore the rigor 

and consistency in the measurement process (both in defining the indicators and collecting the 

rating information), the influences on program entry into and movement within each system (such 

as incentives), and key elements that administrators continue to refine and why. This phase will 

include the nine round 1 RTT-ELC states and rely on data gathered from their grant applications, 

reviews of public and other existing documents, and targeted discussions with RTT-ELC grant 

coordinators (or their designees) in these nine states.3(Clearance is not sought for these state-level data 

collection activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering 

information on the structure and characteristics of TQRIS, we will also assess the feasibility of 

examining the relationship between characteristics of center-based early care and education (ECE)

 

1 The Office of Management and Budget (OMB) has already approved recruitment and data collection activities for 

the Impact Evaluation of Race to the Top and School Improvement Grants. Recruitment activities were approved under 

OMB# 1850-0884 on June 27, 2011. Data collection activities were approved under OMB# 1850-0884 on April 4, 2012.

2 TQRIS standards are commonly grouped into categories or components. Examples of components of TQRIS 

standards include health and safety, curriculum and learning, environment, staff qualifications, family engagement, and 

administration and management. In TQRIS, programs are typically assigned component-level ratings that are then 

combined to determine an overall quality rating. Each component is composed of indicators, which are the specific 

criteria used to assess a program.

3 Documents will include TQRIS websites, and documents received from the RTT-ELC states (such as program 

guides, TQRIS standards, and descriptions of TQRIS data systems). We will also review documents shared by the RTTELC program office (including state evaluation plans and annual performance reports) and request information 

pertaining to the nine states from ongoing research efforts, such as the QRIS online compendium.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

2

programs (as measured by TQRIS ratings) and children’s kindergarten readiness outcomes. If this 

proves feasible, the second study phase will examine the relationship between TQRIS ratings and 

child outcome measures in two to three of the Round 1 RTT-ELC states where the conditions exist 

to make such analysis possible. For this phase of the study, data on TQRIS ratings, component-level 

ratings, and indicator-level ratings will come from state TQRIS databases. Data on kindergarten 

entry assessments will also be sought from databases administered at the state-level. In the event that 

the kindergarten data needed for the analysis are not available from state databases, it may be 

necessary to reach out to selected districts in the RTT-ELC states to collect such data. If this step 

proves necessary, the contractor estimates (based on study power requirements and prior recruiting 

experiences) needing to reach out to up to 42 districts in order to ultimately recruit 14 districts from 

which to collect outcomes data. Because more than nine entities may need to be included in such 

recruitment and data collection activities, this supporting statement seeks clearance for the materials 

and activities that would support the study’s district recruitment efforts and district-level data 

collection if these activities are needed. This document includes drafts of the district recruitment letter 

(Appendix A), the RTT-ELC TQRIS study information sheet (Appendix B), district recruitment call 

protocol (Appendix C), and district data request form (Appendix D).

A. Justification

1. Circumstances necessitating the collection of information

a. Statement of need for a rigorous evaluation

The RTT-ELC grants program is a joint effort by the U.S. Departments of Education (ED) and 

Health and Human Services (HHS) to strengthen the quality of early learning and development 

programs, and to increase access to high quality programs for all children, particularly low-income 

and disadvantaged children. The states that were awarded RTT-ELC grants were required to commit 

to integrating resources and aligning policies across the various state agencies that administer public 

funds related to early learning and development. RTT-ELC seeks to improve early learning and 

development programs by supporting reforms in five areas: (1) integrating state early care and 

education systems; (2) increasing access to high-quality, accountable programs; (3) promoting early 

learning and development outcomes through statewide standards and comprehensive assessment 

systems; (4) developing and supporting the early childhood workforce; and (5) building early learning 

data systems to measure child outcomes and progress.

ED and HHS have awarded RTT-ELC grants through three rounds of competition. Notices for 

the first competition—which was open to all states, the District of Columbia, and Puerto Rico—

were posted in August 2011. In that first round of competition, 35 states, the District of Columbia, 

and Puerto Rico applied for RTT-ELC grants.4 Round 1 awards were made in December 2011 to 

nine states—California, Delaware, Maryland, Massachusetts, Minnesota, North Carolina, Ohio, 

Rhode Island, and Washington. Across these states, a total of $500 million dollars was awarded, with 

grants ranging from $45 million to $70 million. A second round of RTT-ELC grants was announced 

in September 2012. Because more modest funding was available, the second round invited the next 

five highest scoring finalists from Round 1—Colorado, Illinois, New Mexico, Oregon, and 

Wisconsin—to modify their 2011 applications and create plans that could be funded with up to 50 

 

4 Alabama, Alaska, Idaho, Indiana, Louisiana, Montana, New Hampshire, North Dakota, South Carolina, South 

Dakota, Tennessee, Texas, Utah, Virginia, and Wyoming did not apply for RTT-ELC grants in the first round.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

3

percent of the amounts originally requested. All five invited states applied and, in December 2012, 

were awarded RTT-ELC grants in amounts ranging from $21 million to $35 million. A third round 

of competition was announced in April 2013. Awards ranging from $36 million to $51 million were 

made in December 2013 to six new states—Georgia, Kentucky, Michigan, New Jersey, 

Pennsylvania, and Vermont. In addition, supplementary grants were provided to six states from 

rounds 1 and 2 (California, Colorado, Illinois, New Mexico, Oregon and Wisconsin) that had 

previously only received 50 percent of the funds they requested. This study focuses on the nine 

states awarded RTT-ELC grants in Round 1.

A key aspect of the RTT-ELC program is the development and implementation of TQRIS 

under the second reform area noted above (that is, increasing access to high quality, accountable 

programs). TQRIS aim to improve child care program quality and child outcomes by promulgating 

rigorous standards, rating providers based on those standards, and publicizing the ratings of 

individual providers (Zellman et al. 2011). The usefulness of TQRIS ratings thus depends on their 

ability to reliably measure the components that comprise the overall rating and construct from them 

an overall rating that is a valid indicator of program differences with regard to child outcomes (Cizek 

2007, as cited in Zellman and Feine 2012).

Despite growing interest in the use of TQRIS to identify high quality early learning programs, 

evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the 

TQRIS ratings are associated with child outcomes. To our knowledge, five studies have examined 

the relationship between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 

2009; Elicker et al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and 

all examine TQRIS developed prior to the RTT-ELC program. Only two of these five studies—

Thornburg et al. (2009) and Sabol and Pianta (2012)—find evidence of a correlation between 

TQRIS ratings and outcomes. Thornburg and colleagues (2009) found that children attending 

programs with 3-, 4- or 5-star ratings from Missouri’s TQRIS showed significantly greater gains 

during the preschool year in social skills and behavioral outcomes relative to children attending 

programs with 1- or 2-star ratings. They did not find statistically significant differences in children’s 

vocabulary, early literacy, or math skills. Sabol and Pianta (2012) found that children who attended 

programs with 3- or 4-star ratings from Virginia’s TQRIS exhibited significantly larger increases in 

literacy skills over the prekindergarten year compared with children attending programs with 2-star 

ratings. To learn about TQRIS in states that have received RTT-ELC grants and to inform ongoing

development of states’ TQRIS, ED has contracted with Mathematica Policy Research to conduct 

this study.

According to the program requirements included in the Notice Inviting Applications (NIA) (76 

FR 53564), states awarded an RTT–ELC grant are required to “(1) comply with the requirements of

any evaluation sponsored by ED or HHS of any of the State’s activities carried out with the grant; 

(2) comply with the requirements of any cross-state evaluation—as part of a consortium of States—

of any of the State’s proposed reforms, if that evaluation is coordinated or funded by ED or HHS, 

including by using common measures and data collection instruments and collecting data necessary 

to the evaluation.” Legislative authorization for the RTT-ELC TQRIS study is found in the 

Education Science Reform Act of 2002, Part D, Section 171(b)(2), which authorizes IES to 

“conduct evaluations of Federal education programs administered by the Secretary (and as time and 

resources allow, other education programs).”

b. Research questions

The study of RTT-ELC TQRIS is designed to answer the following research questions:
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

4

1. How are TQRIS ratings defined, collected, and generated in the nine Round 1 RTTELC states and how might the structure and implementation of the rating system 

influence the reliability and validity of the ratings? 

2. Is it feasible to examine the relationship between TQRIS ratings and children’s 

kindergarten readiness outcomes in selected RTT-ELC states? 

3. What is the relationship between TQRIS ratings and kindergarten readiness 

outcomes for programs that serve preschool-age children in selected RTT-ELC 

states? 

c. Study design

As noted above, the RTT-ELC TQRIS study comprises two distinct phases. The first study 

phase is a descriptive analysis of the the TQRIS in the nine Round 1 RTT-ELC states. In each state, 

we will examine the structure of the TQRIS, including components and indicators included in states’ 

TQRIS and how they are combined to generate overall quality ratings. We will also explore the rigor 

and consistency in the measurement process (both in defining the indicators and collecting the 

rating information), the influences on program entry into and movement within each system (such 

as incentives), and key elements that administrators continue to refine and why. This phase will 

include the nine Round 1 RTT-ELC states. (Clearance is not sought for these state-level data collection 

activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering information on 

the structure and characteristics of TQRIS, we will also assess the feasibility of examining the 

relationship between characteristics of center-based ECE programs (as measured by TQRIS ratings) 

and children’s kindergarten readiness outcomes. If this proves feasible, the second study phase will 

examine the relationship between TQRIS ratings and child outcome measures in two to three of the 

Round 1 RTT-ELC states where the conditions exist to make such analysis possible. For this study 

phase, the contractor will seek to collect the necessary information from databases administered at 

the state level. If the kindergarten data needed for the analysis are not available from state databases, 

it may be necessary to reach out to approximately 42 districts to ultimately recruit 14 districts from 

which to collect outcomes data. Because more than nine entities may need to be included in such 

recruitment and data collection activities, this supporting statement seeks clearance for the materials 

and activities that would support the study’s district recruitment efforts and district-level data 

collection if these activities are needed. 

To provide context for the district recruitment and data collection activities, we provide an 

overview of the study design and data collection plans for both phases in the remainder of this 

section.

 Research Question 1: How are TQRIS ratings defined, collected, and generated 

in the nine Round 1 RTT-ELC states and how might the structure and 

implementation of the rating system influence the reliability and validity of the 

ratings?

We will answer this question by drawing on information from various sources. First, we 

extracted information from the states’ RTT-ELC applications on the structure of their TQRIS, and 

their plans for refining their TQRIS under RTT-ELC. Second, we will conduct a targeted review of 

publicly available documents, TQRIS websites, and documents received from the RTT-ELC states 

(such as program guides, TQRIS standards, and descriptions of TQRIS data systems) in response to 

our request during initial study engagement telephone discussions. We will also review documents 

shared by the RTT-ELC program office (including state evaluation plans and annual performance 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

5

reports) and request information pertaining to the nine states from ongoing research efforts. Last, 

we will conduct targeted interviews with representatives from each of the nine RTT-ELC states who 

have in-depth knowledge about the TQRIS, are able to confirm and clarify the information obtained 

from documents, and can provide information we cannot get from the documents. 

 Research Question 2: Is it feasible to examine the relationship between TQRIS 

ratings and children’s kindergarten readiness outcomes in selected RTT-ELC 

states?

We will assess the feasibility of examining the relationship between characteristics of centerbased ECE programs (as measured by TQRIS ratings) and children’s kindergarten readiness 

outcomes by determining the extent of center-based ECE program participation in states’ TQRIS, 

the availability of TQRIS and kindergarten entry assessment data, and whether it is possible to link 

students’ kindergarten entry assessment data to the TQRIS ratings of the center-based ECE 

programs they attended. Information related to feasibility will come from various sources. We 

obtained information from the states’ RTT-ELC application on their plans for implementing 

kindergarten entry assessments. Through reviews of documents obtained from state websites or 

received directly from the states in response to our requests during study engagement discussions, 

we will gather additional information about the subscales and characteristics of the kindergarten 

entry assessment as well as how and when it is administered. Finally, we will conduct interviews to 

ascertain the extent of center-based ECE program participation, and where and how TQRIS and 

kindergarten entry assessment data are housed. To confirm and/or gather the information on 

TQRIS and kindergarten assessment data we expect to interview up to three representatives from

each of the Round 1 RTT-ELC states, including a representative from the agency that oversees 

kindergarten entry assessments, a data manager or staff person familiar with the kindergarten entry 

assessment data system, and a data manager or staff person familiar with the TQRIS data system. 

The questions directed to each of the up to three respondents will be specifically targeted so that no 

question will be asked of more than one respondent per state. 

 Research Question 3: What is the relationship between TQRIS ratings and 

kindergarten readiness outcomes for programs that serve preschool-age children 

in selected RTT-ELC states?

We will examine the relationship between the TQRIS ratings and measures of children’s 

kindergarten readiness—that is, scores on kindergarten entry assessments—in a subset of the Round 

1 RTT-ELC states. We anticipate selecting two to three states for this part of the study based on the 

feasibility criteria described under research question 2. 

For each RTT-ELC state that is selected, we will (1) describe TQRIS ratings for center-based 

ECE programs and the characteristics and kindergarten readiness outcomes of children who attend 

such programs, and (2) estimate the associations between overall TQRIS ratings and kindergarten 

entry assessment scores, and (if data are available) between component- or indicator-level ratings 

and kindergarten entry assessment scores. We anticipate that data on TQRIS ratings will come from 

state TQRIS databases. Data on kindergarten entry assessments and student demographic 

characteristics will come from state databases whenever possible. If we cannot obtain the necessary 

outcomes data (i.e., kindergarten entry assessment scores) from states, we will request such data 

from selected districts within the RTT-ELC states chosen for this study component. These districtlevel requests are the focus of this OMB request.

d. Recruitment plan
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

6

The recruitment process for the RTT-ELC TQRIS study will occur in two stages. First, we will 

reach out to each of the nine Round 1 RTT-ELC states to inform them of the study; participation in 

the study is a requirement of the RTT-ELC grant. The second stage, district recruitment, will be 

conducted only if needed. In this section, we first briefly describe the state recruitment process for 

context. The majority of this section discusses district recruitment, which is the focus of this 

supporting statement. 

State engagement. In September and October 2014, we will contact each of the nine Round 1 

RTT-ELC states to introduce the study, identify respondents for telephone interviews to be 

conducted in Fall 2014, and request documents pertaining to the structure of TQRIS and 

characteristics of kindergarten entry assessments in the nine Round 1 RTT-ELC states as well as the 

availability of TQRIS and kindergarten entry assessment data. We will send introductory packages 

to the RTT-ELC grant coordinators in each state, which will include:

 A state engagement letter that will explain the study’s importance, provide an 

overview of the study, and indicate that a member of the study team will call to provide 

more details. 

 An information sheet that will provide a nontechnical description of the study. This 

document includes a summary of study activities, the study’s research questions, and a 

timeline for study activities.

The recruiter assigned to the state will follow up with the state grant coordinator to confirm 

receipt of the mailing and conduct a brief (approximately half-hour) telephone discussion with the 

appropriate agency staff person. The follow-up telephone discussions will last approximately one 

hour. Recruiters will use a standardized protocol as a guide during these telephone conversations, 

which will present background information on the study, request specific documents, and gather 

contact information for use in scheduling telephone interviews in the fall of 2014.

District recruitment. If we learn through the interviews with state representatives that we

cannot obtain the necessary kindergarten entry assessment data from a centralized source at the state 

level, we will attempt to collect such data from individual school districts. If collecting data from 

districts proves necessary, we anticipate obtaining kindergarten entry assessment data from a 

targeted sample of 14 districts with 200 state-funded preschool programs.5,6 Based on recruiting rates 

from similar previous IES studies, we further anticipate that we would need to reach out to 42 

districts to ultimately recruit 14 into the study. For all targeted districts, the contractor will send 

introductory packages to the district superintendent, which will include:

 

5 According to the Common Core of Data, on average, the number of schools serving preschool-aged children in

districts serving preschool-aged children is 14, so the contractor is assuming that it will need to obtain data from 14 

districts in order to include roughly 200 state-funded preschools in the sample (14 = 200/14). Other types of centerbased ECE programs may also be included in the analysis if it is possible to link their TQRIS ratings with kindergarten 

entry assessment data. However, if the contractor needs to obtain data from districts, it will target districts based on the 

number of state-funded preschools, since it is more likely that it will be able to link TQRIS ratings to kindergarten entry 

assessment data for children who attended such programs.

6 It is possible that only some of the two to three selected states will be able to provide the needed data and that 

the contractor will obtain the data from districts in the other state(s). The sample sizes shown here assume that none of 

the selected states can provide the needed data and that the contractor would need to collect all the necessary data from 

districts. If some states can provide data, then the sample for these analyses would be larger than 200 programs.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

7

 A district recruitment letter (Appendix A) that explains the study’s importance, 

provides an overview of the study, and indicates that a member of the study team will 

call to provide more details. 

 An information sheet (Appendix B) that provides a nontechnical description of the 

study. This document includes a summary and timeline of study activities and the study’s 

research questions.

Recruiters will make follow-up calls to the district contacts to introduce the study and arrange 

for a time to further discuss the study. Recruiting communications with each district will take place 

by phone and email. Recruitment discussions will last approximately one hour and will be guided by 

a standardized recruitment protocol (Appendix C). The protocol will guide recruiters in describing 

the study design and planned data collection activities, as well as gathering information on the 

characteristics of center-based ECE programs in the district and the availability of district-level 

kindergarten entry assessment data. Based on the information gathered during these discussions with 

district officials, the study team will identify 14 districts to participate in the second study phase. We 

estimate that these recruiting efforts will take place in spring and summer 2015 (as soon as OMB 

clearance is received).

District recruiter training and tracking. To help establish a strong rapport between the study 

team and the district administrators and their support staff, each district will be assigned one primary 

recruiter. Recruiters will participate in a half-day training session covering the following topics: (1) 

communicating the study’s importance, data collection needs, and methodological issues in effective, 

nontechnical terms, (2) describing the study, (3) procedures for district recruitment, (4) addressing 

potential recruitment issues, and (5) task management (including the recruiting schedule, weekly 

meetings, and use of the tracking system). Trainers will also share key strategies based on the 

contractor’s extensive experience recruiting districts for other IES studies, including the Impact 

Evaluation of Race to the Top and School Improvement Grants.

District research applications and MOUs. We will gain the necessary approvals and abide by 

the relevant guidelines for conducting the study in the 14 selected districts. In districts with policies 

concerning external research projects, we will seek expedited research application reviews where 

possible and emphasize that the study has received prior review by its institutional review board 

(IRB).

Following district officials’ oral commitment to participate in the study, they will be sent a 

memorandum of understanding (MOU) to sign. The MOU will describe the agreed-upon roles and 

responsibilities of the study team and of participating districts. A representative of the study 

contractor will also sign the MOU.

All contacts with districts will be documented in an electronic tracking system to minimize 

redundant contacts and to provide the Contracting Officer’s Representative (COR) with up-to-date 

information on recruiting progress. All recruiters will have access to the web-based system.

e. Data collection needs

To address its research questions, the study will collect and analyze data from several sources, 

which we describe below.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

8

RTT-ELC applications. In February through April 2013, we reviewed RTT-ELC 

Applications for Initial Funding for the nine Round 1 RTT-ELC states. From the RTT-ELC 

applications, we gathered information about the structure of each state’s TQRIS, plans for 

expanding program participation in TQRIS, kindergarten entry assessments, and existing data 

systems.

Publicly available and other existing documents. From September through December

2014, we will collect and conduct a targeted review of documents pertaining to the structure of 

TQRIS and characteristics of kindergarten entry assessments in the nine Round 1 RTT-ELC states 

as well as the availability of TQRIS and kindergarten entry assessment data. Documents will come 

from three sources: (1) the RTT-ELC program office and ongoing research efforts, (2) state 

websites, and (3) state officials. The RTT-ELC program office will be our source for annual 

performance reports which describe states’ plans, activities, and progress, as well as other pertinent 

documents (such as validation plans) related to states’ TQRIS. We anticipate that information not 

already captured by the RTT-ELC program office or ongoing research efforts will be available on 

state websites. To access documents that are not publicly available but offer important details about 

the TQRIS (such as revised standards that are not yet publicly available) and kindergarten entry 

assessments, we will request copies from state officials during initial study engagement conversations 

in September and October 2014.

Interviews with state representatives. From December 2014 through February 2015, we will 

conduct telephone interviews with state representatives from the nine Round 1 RTT-ELC states to 

confirm and clarify the information obtained from documents and, as needed, gather information 

that could not be obtained from the document reviews. During initial study engagement 

conversations, we will identify the appropriate representatives to interview. We anticipate that we 

will have multiple interview respondents – the state child care or TQRIS administrator, a data 

manager or staff person familiar with TQRIS data systems, a representative from the agency that 

oversees kindergarten entry assessments, and a data manager or staff person familiar with the 

kindergarten entry assessment data system. When it is necessary to talk to multiple respondents to 

cover the four primary topics—(1) TQRIS rating structure, (2) TQRIS data, (3) kindergarten entry 

assessments, and (4) kindergarten entry assessment data—we will develop and use tailored protocols 

for each respondent (to reduce burden). It is possible that interviews with multiple administrators in

each state will be needed to fully cover each topic. However, interviews will be targeted so that no 

topic and related series of questions will be asked of more than one administrator in each of the nine 

states. For example, we will need to interview the state child care administrator to confirm or gather 

details about the quality rating components and overall rating structure. We may also need to 

interview a data manager to understand details about data coverage and accessibility that the child 

care administrator cannot address.

TQRIS data. In fall 2015, the contractor will obtain data on center-based ECE programs’ 

TQRIS ratings from state TQRIS databases and, if the data are available, component- and indicatorlevel ratings. TQRIS data from 2014–2015 will be collected from the two to three states participating 

in the second phase of the study (analysis of the relationship between TQRIS ratings and measures 

of children’s school readiness); we anticipate that these data will be available at the state level.

Kindergarten entry assessment data. In fall 2015 through early 2016, the study will aim to 

collect kindergarten assessment data from the two to three states selected for the analysis of TQRIS 

ratings and measures of children’s school readiness. We will request kindergarten entry assessment 

data from tests administered in fall 2015. If kindergarten entry assessment and TQRIS data from the 

prior year (that is, fall 2014 kindergarten entry assessments and 2013–2014 TQRIS ratings) are 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

9

available, and if the kindergarten entry assessments and TQRIS standards used in that year were the 

same as those used in the next year (fall 2015 kindergarten entry assessments and 2014–2015 TQRIS 

ratings), we will request data for both years. We would prefer to obtain kindergarten entry 

assessment data from a centralized source at the state level. This approach would allow us to obtain 

data for all center-based ECE programs that participated in the TQRIS and whose students went on 

to attend kindergarten in the state. We will also request that the data include a student- and/or 

program-level identifier7that will allow us to link program-level TQRIS data to child-level 

kindergarten entry assessment data.8In addition to test scores and student and/or program 

identifiers, we will explore the availability of data on child gender, birth year, socioeconomic status, 

race and ethnicity, English language learner status, free or reduced-price lunch status, special 

education status, maternal education, and children’s length of enrollment in their center-based ECE 

programs.

9

If kindergarten entry assessment data are not available from a centralized source at the 

state level, we will seek to collect the data from up to 14 individual school districts. The data request 

will include a form that specifies the data needed for the study (Appendix D). 

f. Study activities and timeline

This clearance request pertains to the recruitment and collection of administrative data from 

districts, which will be conducted only if the data needed for the study cannot be obtained from 

states. Table A.1 shows the schedule of study activities. Since this package is requesting clearance 

only for district recruitment and data collection activities, only district-level activities listed in Table 

A.1 apply to this request. The timeline includes other major study activities to provide context for 

the district-level activities.

Table A.1. Study Activities and Timeline 

Study Activities Sample Respondent Mode Schedule

Review RTT-ELC 

grant applications

Round 1 RTT-ELC 

states (9) Not applicable Desk research 

February–April 

2013

Engage RTT-ELC 

states

Round 1 RTT-ELC 

states (9)

State RTT-ELC 

grant coordinator or 

designee

Telephone and email 

communication

September–

October 2014

Review documents

Round 1 RTT-ELC 

states (9) Not applicable Desk research

September–

December 2014

Interview RTT-ELC 

state representatives

Round 1 RTT-ELC 

states (9)

State RTT-ELC 

grant coordinator or 

designee Telephone interview

December 2014-

February 2015

 

7 Identifiers will be replaced with a Mathematica ID to prevent disclosure of respondent data.

8 We expect that these data linkages will be more feasible for children who attended state-funded preschool 

programs, because they might be assigned a unique state or district identifier that is linked to their kindergarten data. 

However, there may be other ways to link preschool TQRIS ratings with kindergarten assessment data. For example, the 

kindergarten entry assessment data set may include a program ID for the preschool that a student attended, and it may 

be possible to use that program ID to obtain the preschool’s rating from the TQRIS data set. It may also be possible to 

obtain preschool enrollment rosters from states or districts that contain student IDs that match the student IDs used in 

the state or district kindergarten entry assessment databases, so that TQRIS data can be linked with child outcome data.

9 Personally identifiable information will be stored in separate files to ensure confidentiality.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

10

Study Activities Sample Respondent Mode Schedule

Collect TQRIS data

Selected Round 1 

RTT-ELC states (2 

to 3)

State TQRIS 

administrator or 

designee

Secure electronic data 

submission Fall 2015

Recruit districts (if 

necessary)*

Approximately 42 

districts from 

selected Round 1 

RTT-ELC states (1 

to 3)

District 

superintendent or 

designee

Telephone and email 

communication

Spring–Summer 

2015

Collect kindergarten 

entry assessment 

(KEA) data and data 

on child 

demographics and 

other characteristics*

Selected Round 1 

RTT-ELC states (2 

to 3) and 

approximately 14 

districts in these 

states 

State or district KEA 

data administrator or 

designee

Secure electronic data 

submission Fall–Winter 2015

* Activities relevant for this supporting statement.

2. Purposes and uses of the data

To address the study’s research questions, the study will collect and analyze data from several 

sources. Table A.2 lists the questions and the data sources that will be used to answer them. This 

supporting statement is requesting clearance only for the study’s district recruiting and district data 

collection, which will be used to address the study’s third research question. All other data collection 

methods are based on publicly available information or involve targeted interviews with fewer than 

10 individuals. We describe the study’s use of each data source in more detail below. Information 

will be collected by Mathematica Policy Research, under contract with ED [contract number EDIES-10-C-0077].

Table A.2. Research Questions and Data Sources

Research Question Data Sources

1. How are TQRIS ratings defined, collected, and 

generated in the nine Round 1 RTT-ELC states and 

how might the structure and implementation of the 

rating system influence the reliability and validity of 

the ratings?

Application reviews, targeted document reviews, 

and targeted interviews with staff from the 9 Round 

1 RTT-ELC states

2. Is it feasible to examine the relationship between TQRIS 

ratings and children’s kindergarten readiness outcomes in 

selected RTT-ELC states?

Application reviews, targeted document reviews, 

and targeted interviews with staff from the 9 Round 

1 RTT-ELC states

3. What is the relationship between TQRIS ratings and 

kindergarten readiness outcomes for programs that serve 

preschool-aged children in selected RTT-ELC states?

TQRIS data from state TQRIS data systems in 2 to 

3 of the Round 1 RTT-ELC states

Kindergarten entry assessment data from state 

and/or district administrative data systems in 2 to 3 

of the Round 1 RTT-ELC states

State’s RTT-ELC Applications for Initial Funding. From the RTT-ELC applications, the 

study collected information about the structure of each state’s TQRIS, plans for expanding program 

participation in TQRIS, kindergarten entry assessments, and existing data systems. We will use this 

information to address research questions 1 and 2 and to inform the analysis for research question 3. 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

11

Document reviews. Document reviews will focus on describing the structure of TQRIS, the 

rigor and consistency in the measurement process, the influences on program entry into and

movement within each system (such as incentives), and key elements that administrators continue to 

refine and why. We will also review documents describing characteristics, properties, and 

administration procedures of the state kindergarten entry assessment. We will use this information 

to address research questions 1 and 2 and to select states to be included in analysis for research 

question 3.

State interviews. Interviews with state administrators will confirm and clarify the information 

obtained from documents and, as needed, gather information that could not be obtained from the 

document review. We will use this information to address research questions 1 and 2 and to refine 

the design for research question 3.

TQRIS data. The main explanatory variable in the analysis for research question 3 will be a 

center-based ECE program’s TQRIS rating. The data pertaining to this variable will come from state 

TQRIS databases. We will also collect information on component- and indicator-level ratings if the 

data are available. These data will be used to examine the associations between the TQRIS 

component- and indicator-level ratings and child outcomes.

Kindergarten entry assessment data. The outcome data for research question 3 will come 

from student-level administrative databases maintained by states and/or districts. These data will 

allow us to assess the relationship between TQRIS ratings and measures of kindergarten readiness. 

We will request data on student characteristics, such as sex, race/ethnicity, birth year, grade, 

eligibility for free or reduced-price lunch, English language learner status, special education status,

and preschool enrollment history, in order to control for factors other than differences in centerbased ECE program quality that might correlate with child outcomes. We will also request that the 

data include a student- and/or program-level identifier that will allow us to link program-level 

TQRIS data to child-level kindergarten entry assessment data.

3. Use of technology to reduce burden

The recruitment plan for which clearance is being sought in this submission is designed to 

minimize burden on the districts being recruited. An electronic recruiting tracking database will be 

designed and used to store state and district contact history and relevant recruiting data collected in 

the course of those communications. Effective use of this management tool will minimize redundant 

contacts with states and districts and allow for specific requests for information to be consolidated 

and coordinated among study team members. A summary of recruiting information will also be 

entered into Mathematica’s cross-study recruiting database, allowing further coordination of requests 

across ongoing education studies, as well as coordination of decisions on the timing of such 

requests, to further minimize burden while meeting each study’s requirements.

The data collection plan is also designed to obtain information in an efficient way that 

minimizes respondent burden. We will gather information only from existing data sources, using the 

most efficient methods available. Existing data sources will be electronic files provided by states (or 

districts, when not available from the state) that include TQRIS ratings, test scores for schooladministered kindergarten entry assessments, and student demographic information. We will work 

with state and/or district personnel to determine the most efficient and least burdensome way to 

collect these data and will capitalize on any electronic data systems already in place. Whenever 

possible, states and/or districts will be able to upload or enter data into any electronic spreadsheet, 

or to an equivalent file, and transfer it to Mathematica through a secure FTP site. Regardless of the 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

12

form in which it is received, these data will be converted into a consistent format so that they can be 

combined with data submitted by other districts from the state and be suitable for analysis. If it is 

too burdensome or not possible for a state or district to provide data in electronic format, we will 

provide clear instructions on how to submit the information securely in hard-copy form, to be 

coded by the study team.

4. Efforts to avoid duplication of effort

Evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the 

TQRIS ratings are associated with child outcomes. Five studies have examined the relationship 

between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 2009; Elicker et 

al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and all examine 

TQRIS developed prior to the RTT-ELC program. Only two of these five studies—Thornburg et al. 

(2009) and Sabol and Pianta (2012)—find consistent evidence of a correlation between TQRIS 

ratings and outcomes. ED determined that an in-depth, cross-state study examining the structure 

and characteristics of TQRIS in RTT-ELC states and the relationship between their ratings and

measures of kindergarten readiness is needed to inform the ongoing development of RTT-ELC 

states’ and other states’ TQRIS.

RTT-ELC states are required to validate their TQRIS by assessing whether the tiers of TQRIS 

reflect different levels of program quality and whether differences in quality ratings relate to child 

outcomes. This study will differ from efforts RTT-ELC states are undertaking in the following ways: 

 If feasible, the second component of this study will focus on the relationship between 

TQRIS ratings and kindergarten readiness measures for programs that serve preschoolaged children. We will select states and collect data with the intention of maximizing 

statistical power to examine this specific question. State validation studies will not 

necessarily focus on this particular age group or outcome. 

 This study will employ consistent methodology across states to facilitate comparisons 

between states. The methods used in states’ local evaluations will vary, depending on the 

specific questions each state has chosen to focus on. 

5. Methods to minimize burden on small entities

No small businesses or entities will be involved as respondents.

6. Consequences of not collecting the data

The recruitment and data collection plan described in this submission is necessary for ED to 

examine the relationship between TQRIS ratings and kindergarten readiness measures for programs 

that serve preschool-aged children. Examining the relationship between TQRIS ratings and 

children’s kindergarten readiness is important since a key goal of RTT-ELC is helping to ensure that 

children enter kindergarten with the skills, knowledge, and dispositions toward learning that they 

need to succeed. Failing to conduct this study would mean missing a key opportunity to learn about 

the TQRIS ratings and support the decisions that states are making as they develop and/or refine 

their TQRIS. Without state and district (if the state records are not available) administrative records, 

ED will not be able to examine whether TQRIS ratings (and which TQRIS components) are related 

to children’s kindergarten readiness. The district recruitment and data collection activities that are 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

13

the focus of this supporting statement will be conducted only if the data needed for the study’s 

analyses are not available at the state level.

7. Special circumstances

There are no special circumstances involved with recruitment or data collection activities.

8. Federal register announcement and outside consultation

a. Federal register announcement

The 60-day notice to solicit public comments was published in Volume X, page X of the 

Federal Register on DATE. The 30-day notice was published in Volume X, page X of the Federal 

Register on DATE. No public comments were received.

b. Consultations outside of the agency

The study team sought input on the study design from experts in early learning and TQRIS, 

evaluation design, and analysis during a technical working group (TWG) meeting held on July 30, 

2014. We will continue to consult with the TWG throughout the study on other issues that would 

benefit from their input. Table A.3 lists the TWG members. 

Table A.3 Members of the Technical Working Group for the RTT-ELC Study

Name Title and Affiliation

Margaret Burchinal Senior Scientist

FPG Child Development Institute

University of North Carolina

James Elicker Associate Professor

Human Development & Family Studies

Purdue University

Beth Rous Department Chair and Professor

Educational Leadership Studies

University of Kentucky

Elizabeth Stuart Associate Professor

Bloomberg School of Public Health

Johns Hopkins University

Kathryn Tout Co-Director

Early Childhood Development 

Child Trends

Martha Zaslow Director

SRCD Office for Policy & Communications

Gail Zellman Senior Research Psychologist

RAND Corporation

9. Payments or gifts

The study does not plan to give gifts to states or districts for participating in the recruitment 

process or for providing data for the study.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

14

10. Assurances of confidentiality

The study team has established procedures to ensure the confidentiality and security of its data. 

This approach will be in accordance with all relevant regulations and requirements, particularly the 

Education Sciences Institute Reform Act of 2002, Title I, Subsection (c) of Section 183, which 

requires the IES director to “develop and enforce standards designed to protect the confidentiality 

of persons in the collection, reporting, and publication of data.” The study will also adhere to 

requirements of Subsection (d) of Section 183 prohibiting disclosure of individually identifiable 

information, as well as making the publishing or inappropriate communication of individually 

identifiable information by employees or staff a felony.

The study team will protect the full privacy and confidentiality of all individuals and programs 

that provide data. The study will not obtain data associated with personally identifiable information 

(PII), as study staff will be assigning random ID numbers to all data records and then stripping any 

PII from the data records. In addition to the data safeguards described here, the study team will 

ensure that no respondent names, programs, schools, or districts are identified in publicly available 

reports or findings, and if necessary, the study team will mask distinguishing characteristics. A 

statement to this effect will be included with all requests for data:

“Mathematica Policy Research follows the confidentiality and data protection requirements of IES (The 

Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Administrative records obtained through this 

data collection will be used only for research purposes. The reports prepared for the study will summarize state-level 

findings and will not associate findings with a specific district, program, or individual student. We will not provide 

information that identifies respondents to anyone outside the study team.”

Mathematica employs the following safeguards to ensure confidentiality:

 All Mathematica employees sign a pledge that emphasizes the importance of 

confidentiality and describes their obligation.

 Secure FTP services allow encrypted transfer of large data files with clients, if necessary. 

All internal networks are protected from unauthorized access using defense-in-depth 

best practices, which incorporate firewalls and intrusion detection and prevention 

systems. The networks are configured so that each user has a tailored set of rights, 

granted by the network administrator, to files approved for access and stored on the 

LAN. Access to hard-copy documents is strictly limited. Documents are stored in locked 

files and cabinets. Discarded materials are shredded.

 Computer data files are protected with passwords, and access is limited to specific users, 

who must change their passwords regularly and conform to strong password policies.

 Especially sensitive data are maintained on removable storage devices that are kept 

physically secure when not in use.

After the study concludes, the study data will be transmitted to the National Center for 

Education Statistics (NCES) for safekeeping as a restricted-use file. All other versions of the data 

will be destroyed. Prior to transmittal, the data will undergo careful screening, and modification if 

necessary, to ensure that there is no unacceptably high level of disclosure risk for protected 

respondents. Researchers wishing to access the data for secondary analysis must apply for an NCES 

license and agree to the applicable rules and procedures guiding the use of restricted-use files.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

15

11. Additional justification for sensitive questions

No questions of a sensitive nature will be included in this study.

12. Estimates of hours burden

If we identify a state as a good candidate for inclusion in the analysis of the relationship 

between TQRIS ratings and child outcomes but find that we cannot obtain kindergarten entry 

assessment data from a centralized source at the state level, we will obtain kindergarten entry 

assessment data from a targeted sample of districts within that state. We anticipate that officials and 

staff from approximately 42 school districts will participate in initial phone calls and follow-up 

communications that will occur during the district recruitment phase of the study in spring and 

summer 2015. We estimate that we will need to contact 42 districts to discuss data collection 

activities to ultimately recruit 14 districts to participate in the study based on our experiences with 

other projects, including the Impact Evaluation of Race to the Top and School Improvement 

Grants. 

We estimate that during the district recruitment process, we will speak with a main contact in all 

42 districts and a separate person for completing the MOU in the targeted 14 districts. We expect 

that initial and follow-up discussions with the primary contact in each district will take 1.5 hours of 

their time, on average, for a total of 63 hours for the upfront recruitment contact. Reviewing 

research applications and/or completing MOUs will take 6 hours, on average, per respondent for a 

total of 84 hours across 14 districts. The total number of district staff involved in recruitment 

activities will be 56, and the total number of recruitment hours will be 147, for an average of 2.6

hours per person (Table A.4). See Appendices A, B, and C for the materials and protocol to be used 

in district recruitment phone calls.

Table A.4. Burden in Hours to Respondents – RTT-ELC TQRIS Study Recruitment and Data Collection

Activity

Total 

Number of 

Respondents

Number of 

Responses 

Per 

Respondent

Number of 

Administrations

Total 

Number of 

Responses

Burden 

Hours Per 

Response

Total 

Burden 

(Hours)

District recruiting 

communications 

42 1 1 42 1.5 63

Review of research 

applications and/or 

MOUs

14 1 1 14 6 84

Student records 

request 

14 1 1 14 8 112

Total District Staff 70 70 259

If state-level data are not available, officials from 14 school districts will be asked to provide the 

necessary student-level administrative records in fall and winter 2015. We estimate that it will take 8 

hours, on average, of district-level staff time to deliver the required student records in each of the 

participating 14 districts. The total number of burden hours for records delivery, therefore, will be 

112 (Table A.4). The data collection form is included in Appendix D and will include the following 

appropriately tailored Public Burden Statement:

According to the Paperwork Reduction Act of 1995, no persons are required to 

respond to a collection of information unless such collection displays a valid OMB 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

16

control number. Public reporting burden for this collection of information is 

estimated to average 8 hours per response, including time for reviewing instructions, 

searching existing data sources, gathering and maintaining the data needed, and 

completing and reviewing the collection of information. Send comments regarding 

the burden estimate or any other aspect of this collection of information, including 

suggestions for reducing this burden, to the U.S. Department of Education, 

Washington, DC 20202-4651 and reference the OMB Control Number 1850-0884. 

If you have comments or concerns regarding the status of your individual 

submission of this form, write directly to: Institute of Education Sciences, U.S. 

Department of Education, 555 New Jersey Ave. NW, Washington, DC 20208.

13. Estimates of cost burden to respondents

There are no additional respondent costs associated with this data collection beyond the burden 

estimated in item A12.

14. Estimates of annual costs to the federal government

The estimated cost for this three-year study is $2,968,330, or approximately $989,443 per year. 

This includes development of a detailed study design, data collection instruments, this justification 

package, state (and, if needed, district) recruitment, TQRIS and outcomes data collection, and 

analysis and report preparation.

15. Reasons for program changes or adjustments

This is a new data collection effort and will count as 259 hours toward ED’s information 

collection burden. 

16. Publication plans and schedule 

Table A.5 displays the anticipated timetable for project publications, which include an

evaluation brief, and a final report. The evaluation brief’s topic will be the descriptive analyses 

aligned with the study’s first research question, which examines the components and indicators

included in the TQRIS in each of the nine Round 1 RTT-ELC states and how they are used to 

generate overall quality ratings. This brief will be released separately and prior to the release of the

final report.

Table A.5. Timetable for Project Publications 

Deliverable Date

Evaluation Brief

Draft evaluation brief 

Revised draft evaluation brief 

Final evaluation brief 

June 2015

August 2015

November 2015

Report

Draft report 

Revised draft report 

Final report 

June 2016

August 2016

February 2017
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

17

17. Approval not to display the OMB expiration date

All recruitment and data collection materials will display the OMB expiration date.

18. Explanation of exceptions

No exceptions are requested.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

18

REFERENCES

Elicker, J., C.C. Langill, K. Ruprecht, J. Lewsader, and T. Anderson. “Evaluation of ‘Paths to 

QUALITY,’ Indiana’s Child Care Quality Rating and Improvement System: Final Report 

(Technical Report #3).” Department of Human Development and Family Studies, Center for 

Families, Purdue University, 2011.

Lahti, M., T. Sabol, R. Starr, C. Langill, and K. Tout. “Validation of Quality Rating and 

Improvement Systems (QRIS): Examples from Four States.” Research-to-Policy, Research-to–

Practice Brief OPRE 2013-036. Washington, DC: Office of Planning, Research and Evaluation, 

Administration for Children and Families, U.S. Department of Health and Human Services, 

2013.

Thornburg, K.R., W.A. Mayfield, J.S. Hawks, and K.L. Fuger. “The Missouri Quality Rating System 

School Readiness Study.” Columbia, MO: University of Missouri Center for Family Policy and 

Research, 2009. Retrieved from http://mucentermissouri.edu/MOQRSreport.pdf. 

Tout, K., R. Starr, T. Isner, J. Cleveland, L. Albertson-Junkans, M. Soli, and K. Quinn. “Evaluation 

of Parent Aware: Minnesota’s Quality Rating and Improvement System Pilot.” Washington, 

DC: Child Trends, 2011.

U.S. Department of Education. “Race to the Top – Early Learning Challenge Application for Initial 

Funding.” 2011. Retrieved May 6, 2013 from http://www2.ed.gov/programs/racetothetopearlylearningchallenge/2011-412.doc. 

Zellman, G.L., R.N. Brandon, K. Boller, and J.L. Kreader. “Effective Evaluation of Quality Rating 

and Improvement Systems for Early Care and Education and School-Age Care.” Research-toPolicy, Research-to–Practice Brief OPRE 2011-11a. Washington, DC: Office of Planning, 

Research and Evaluation, Administration for Children and Families, U.S. Department of Health 

and Human Services, 2011.

Zellman, G.L., and R. Fiene. “Validation of Quality Rating and Improvement Systems for Early 

Care and Education and School-Age Care.” Research-to-Policy, Research-to–Practice Brief 

OPRE 2012-29. Washington, DC: Office of Planning, Research and Evaluation, Administration 

for Children and Families, U.S. Department of Health and Human Services, 2012.

Zellman, G.L., M. Perlman, V. Le, and C. Setodji. “Assessing the Validity of the Qualistar Early 

Learning Quality Rating and Improvement System as a Tool for Improving Child-Care 

Quality.” Pittsburgh, PA: Rand Corporation, 2008. Accessed November 26, 2012 at 

http://www.rand.org/pubcs/monographs/MG650.html.
www.mathematica-mpr.com

Improving public well-being by conducting high quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research