Race to the Top-Early Learning 

Challenge: Descriptive Study of

Tiered Quality Rating and 

Improvement Systems in Nine 

Round 1 States: OMB Forms 

Clearance Package, Part A

April 21, 2015
Contract Number:

ED-IES-10-C-0077

Mathematica Reference Number:

06844.144

Submitted to:

National Center for Education Evaluation

555 New Jersey Ave., NW, Suite 500k

Washington, DC 20208

Project Officer: Tracy Rimdzius 

Submitted by:

Mathematica Policy Research

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone: (609) 799-3535

Facsimile: (609) 799-0005

Project Director: Susanne James-Burdumy, 

PhD

Race to the Top-Early Learning 

Challenge: Descriptive Study of

Tiered Quality Rating and 

Improvement Systems in Nine 

Round 1 States: OMB Forms 

Clearance Package, Part A

April 21, 2015
iii

CONTENTS

PART A. SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 

SUBMISSION................................................................................................... 1

A. Justification................................................................................................ 2

1. Circumstances necessitating the collection of information .................. 2

2. Purposes and uses of the data.......................................................... 10

3. Use of technology to reduce burden.................................................. 11

4. Efforts to avoid duplication of effort ................................................... 12

5. Methods to minimize burden on small entities................................... 13

6. Consequences of not collecting the data........................................... 13

7. Special circumstances....................................................................... 13

8. Federal register announcement and outside consultation ................. 13

9. Payments or gifts............................................................................... 14

10. Assurances of confidentiality............................................................. 14

11. Additional justification for sensitive questions.................................... 15

12. Estimates of hours burden................................................................. 15

13. Estimates of cost burden to respondents .......................................... 16

14. Estimates of annual costs to the federal government........................ 16

15. Reasons for program changes or adjustments.................................. 17

16. Publication plans and schedule......................................................... 17

17. Approval not to display the OMB expiration date............................... 17

18. Explanation of exceptions.................................................................. 17

REFERENCES.............................................................................................................. 18

APPENDIX A: DISTRICT LETTER

APPENDIX B: STUDY INFORMATION SHEET

APPENDIX C: DISTRICT MEMORANDUM OF UNDERSTANDING

APPENDIX D: RECRUITING PROTOCOL WITH DISTRICT ADMINISTRATOR

APPENDIX E: DATA COLLECTION FORM FOR DISTRICT-LEVEL 

KINDERGARTEN ASSESSMENT DATA
iv

TABLES

A.1 Study Activities and Timeline ......................................................................... 10

A.2 Research Questions and Data Sources ......................................................... 10

A.3 Burden in Hours to Respondents – RTT-ELC TQRIS Study

Recruitment and Data Collection.................................................................... 16

A.4 Timetable for Project Publications.................................................................. 17
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

1

PART A. SUPPORTING STATEMENT FOR PAPERWORK 

REDUCTION ACT SUBMISSION

This supporting statement requests clearance to recruit and collect data from approximately 14 

school districts to support the Study of Race to the Top–Early Learning Challenge Tiered Quality 

Rating and Improvement Systems (RTT-ELC TQRIS). Through TQRIS, states use a set of 

standards to evaluate the quality of early learning and development programs and promote program 

improvement. The RTT-ELC application for the first round of grants required states to develop and 

adopt a TQRIS that, according to the U.S. Department of Education (ED), “is based on a statewide 

set of tiered program standards that include: (1) early learning and development standards, (2) a 

comprehensive assessment system, (3) early childhood educator qualifications, (4) family 

engagement strategies, (5) health promotion practices, and (6) effective data practices; is clear and 

has standards that are measurable, meaningfully differentiate program quality levels, and reflect high 

expectations of program excellence commensurate with nationally recognized standards that lead to 

improved learning outcomes for children; and is linked to the state licensing system for early 

learning and development programs” (2011). The study of RTT-ELC TQRIS will provide important 

information on the structure and characteristics of TQRIS and whether and how the TQRIS ratings 

relate to child outcomes. This information is intended to inform improvements to states’ TQRIS. 

The Institute of Education Sciences (IES) at ED contracted with Mathematica Policy Research to 

conduct this study, which is being carried out as a component of the Impact Evaluation of Race to 

the Top and School Improvement Grants.1

The RTT-ELC TQRIS study comprises two distinct phases. The first study phase is a 

descriptive analysis of the TQRIS in the nine Round 1 RTT-ELC states. In each state, we will 

examine the structure of the TQRIS, including the components and indicators included in states’ 

TQRIS and how they are combined to generate overall quality ratings.

2 We will also explore the rigor 

and consistency in the measurement process (both in defining the indicators and collecting the 

rating information), the influences on program entry into and movement within each system (such 

as incentives), and key elements that administrators continue to refine and why. This phase will 

include the nine round 1 RTT-ELC states and rely on data gathered from their grant applications, 

reviews of public and other existing documents, and targeted discussions with RTT-ELC grant 

coordinators (or their designees) in these nine states.3(Clearance is not sought for these state-level data 

collection activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering 

information on the structure and characteristics of TQRIS, we will also assess the feasibility of 

examining the relationship between characteristics of center-based early care and education (ECE)

 

1 The Office of Management and Budget (OMB) has already approved recruitment and data collection activities for 

the Impact Evaluation of Race to the Top and School Improvement Grants. Recruitment activities were approved under 

OMB# 1850-0884 on June 27, 2011. Data collection activities were approved under OMB# 1850-0884 on April 4, 2012.

2 TQRIS standards are commonly grouped into categories or components. Examples of components of TQRIS 

standards include health and safety, curriculum and learning, environment, staff qualifications, family engagement, and 

administration and management. In TQRIS, programs are typically assigned component-level ratings that are then 

combined to determine an overall quality rating. Each component is composed of indicators, which are the specific 

criteria used to assess a program.

3 Documents will include TQRIS websites, and documents received from the RTT-ELC states (such as program 

guides, TQRIS standards, and descriptions of TQRIS data systems). We will also review documents shared by the RTTELC program office (including state evaluation plans and annual performance reports) and request information 

pertaining to the nine states from ongoing research efforts, such as the QRIS online compendium.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

2

programs (as measured by TQRIS ratings) and children’s kindergarten readiness outcomes. If this 

proves feasible, the second study phase will examine the relationship between TQRIS ratings and 

child outcome measures in two to three of the Round 1 RTT-ELC states where the conditions exist 

to make such analysis possible. For this phase of the study, data on TQRIS ratings, component-level 

ratings, and indicator-level ratings will come from state TQRIS databases. Data on kindergarten 

entry assessments will also be sought from databases administered at the state-level. In the event that 

the kindergarten data needed for the analysis are not available from state databases, it may be 

necessary to reach out to selected districts in the RTT-ELC states to collect such data. If this step 

proves necessary, the contractor estimates (based on study power requirements and prior recruiting 

experiences) needing to reach out to up to 42 districts in order to ultimately recruit 14 districts from 

which to collect outcomes data. Because more than nine entities may need to be included in such 

recruitment and data collection activities, this supporting statement seeks clearance for the materials 

and activities that would support the study’s district recruitment efforts and district-level data 

collection if these activities are needed. This document includes drafts of the district recruitment letter 

(Appendix A), the RTT-ELC TQRIS study information sheet (Appendix B), district memorandum 

of understanding (Appendix C), district recruitment call protocol (Appendix D), and district data 

request form (Appendix E).

A. Justification

1. Circumstances necessitating the collection of information

a. Statement of need for a rigorous evaluation

The RTT-ELC grants program is a joint effort by the U.S. Departments of Education (ED) and 

Health and Human Services (HHS) to strengthen the quality of early learning and development 

programs, and to increase access to high quality programs for all children, particularly low-income 

and disadvantaged children. The states that were awarded RTT-ELC grants were required to commit 

to integrating resources and aligning policies across the various state agencies that administer public 

funds related to early learning and development. RTT-ELC seeks to improve early learning and 

development programs by supporting reforms in five areas: (1) integrating state early care and 

education systems; (2) increasing access to high-quality, accountable programs; (3) promoting early 

learning and development outcomes through statewide standards and comprehensive assessment 

systems; (4) developing and supporting the early childhood workforce; and (5) building early learning 

data systems to measure child outcomes and progress.

ED and HHS have awarded RTT-ELC grants through three rounds of competition. Notices for 

the first competition—which was open to all states, the District of Columbia, and Puerto Rico—

were posted in August 2011. In that first round of competition, 35 states, the District of Columbia, 

and Puerto Rico applied for RTT-ELC grants.4 Round 1 awards were made in December 2011 to 

nine states—California, Delaware, Maryland, Massachusetts, Minnesota, North Carolina, Ohio, 

Rhode Island, and Washington. Across these states, a total of $500 million dollars was awarded, with 

grants ranging from $45 million to $70 million. A second round of RTT-ELC grants was announced 

in September 2012. Because more modest funding was available, the second round invited the next 

five highest scoring finalists from Round 1—Colorado, Illinois, New Mexico, Oregon, and 

 

4 Alabama, Alaska, Idaho, Indiana, Louisiana, Montana, New Hampshire, North Dakota, South Carolina, South 

Dakota, Tennessee, Texas, Utah, Virginia, and Wyoming did not apply for RTT-ELC grants in the first round.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

3

Wisconsin—to modify their 2011 applications and create plans that could be funded with up to 50 

percent of the amounts originally requested. All five invited states applied and, in December 2012, 

were awarded RTT-ELC grants in amounts ranging from $21 million to $35 million. A third round 

of competition was announced in April 2013. Awards ranging from $36 million to $51 million were 

made in December 2013 to six new states—Georgia, Kentucky, Michigan, New Jersey, 

Pennsylvania, and Vermont. In addition, supplementary grants were provided to six states from 

rounds 1 and 2 (California, Colorado, Illinois, New Mexico, Oregon and Wisconsin) that had 

previously only received 50 percent of the funds they requested. This study focuses on the nine 

states awarded RTT-ELC grants in Round 1.

A key aspect of the RTT-ELC program is the development and implementation of TQRIS 

under the second reform area noted above (that is, increasing access to high quality, accountable 

programs). TQRIS aim to improve child care program quality and child outcomes by promulgating 

rigorous standards, rating providers based on those standards, and publicizing the ratings of 

individual providers (Zellman et al. 2011). The usefulness of TQRIS ratings thus depends on their 

ability to reliably measure the components that comprise the overall rating and construct from them 

an overall rating that is a valid indicator of program differences with regard to child outcomes (Cizek 

2007, as cited in Zellman and Feine 2012).

Despite growing interest in the use of TQRIS to identify high quality early learning programs, 

evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the 

TQRIS ratings are associated with child outcomes. To our knowledge, five studies have examined 

the relationship between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 

2009; Elicker et al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and 

all examine TQRIS developed prior to the RTT-ELC program. Only two of these five studies—

Thornburg et al. (2009) and Sabol and Pianta (2012)—find evidence of a correlation between 

TQRIS ratings and outcomes. Thornburg and colleagues (2009) found that children attending 

programs with 3-, 4- or 5-star ratings from Missouri’s TQRIS showed significantly greater gains 

during the preschool year in social skills and behavioral outcomes relative to children attending 

programs with 1- or 2-star ratings. They did not find statistically significant differences in children’s 

vocabulary, early literacy, or math skills. Sabol and Pianta (2012) found that children who attended 

programs with 3- or 4-star ratings from Virginia’s TQRIS exhibited significantly larger increases in 

literacy skills over the prekindergarten year compared with children attending programs with 2-star 

ratings. To learn about TQRIS in states that have received RTT-ELC grants and to inform ongoing

development of states’ TQRIS, ED has contracted with Mathematica Policy Research to conduct 

this study.

According to the program requirements included in the Notice Inviting Applications (NIA) (76 

FR 53564), states awarded an RTT–ELC grant are required to “(1) comply with the requirements of

any evaluation sponsored by ED or HHS of any of the State’s activities carried out with the grant; 

(2) comply with the requirements of any cross-state evaluation—as part of a consortium of States—

of any of the State’s proposed reforms, if that evaluation is coordinated or funded by ED or HHS, 

including by using common measures and data collection instruments and collecting data necessary 

to the evaluation.” Legislative authorization for the RTT-ELC TQRIS study is found in the 

Education Science Reform Act of 2002, Part D, Section 171(b)(2), which authorizes IES to 

“conduct evaluations of Federal education programs administered by the Secretary (and as time and 

resources allow, other education programs).”

b. Research questions
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

4

The study of RTT-ELC TQRIS is designed to answer the following research questions:

1. How are TQRIS ratings defined, collected, and generated in the nine Round 1 RTTELC states and how might the structure and implementation of the rating system 

influence the reliability and validity of the ratings? 

2. Is it feasible to examine the relationship between TQRIS ratings and children’s 

kindergarten readiness outcomes in selected RTT-ELC states? 

3. What is the relationship between TQRIS ratings and kindergarten readiness 

outcomes for programs that serve preschool-age children in selected RTT-ELC 

states? 

c. Study design

As noted above, the RTT-ELC TQRIS study comprises two distinct phases. The first study 

phase is a descriptive analysis of the the TQRIS in the nine Round 1 RTT-ELC states. In each state, 

we will examine the structure of the TQRIS, including components and indicators included in states’ 

TQRIS and how they are combined to generate overall quality ratings. We will also explore the rigor 

and consistency in the measurement process (both in defining the indicators and collecting the 

rating information), the influences on program entry into and movement within each system (such 

as incentives), and key elements that administrators continue to refine and why. This phase will 

include the nine Round 1 RTT-ELC states. (Clearance is not sought for these state-level data collection 

activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering information on 

the structure and characteristics of TQRIS, we will also assess the feasibility of examining the 

relationship between characteristics of center-based ECE programs (as measured by TQRIS ratings) 

and children’s kindergarten readiness outcomes. If this proves feasible, the second study phase will 

examine the relationship between TQRIS ratings and child outcome measures in two to three of the 

Round 1 RTT-ELC states where the conditions exist to make such analysis possible. For this study 

phase, the contractor will seek to collect the necessary information from databases administered at 

the state level. If the kindergarten data needed for the analysis are not available from state databases, 

it may be necessary to reach out to approximately 42 districts to ultimately recruit 14 districts from 

which to collect outcomes data. Because more than nine entities may need to be included in such 

recruitment and data collection activities, this supporting statement seeks clearance for the materials 

and activities that would support the study’s district recruitment efforts and district-level data 

collection if these activities are needed. 

To provide context for the district recruitment and data collection activities, we provide an 

overview of the study design and data collection plans for both phases in the remainder of this 

section.

 Research Question 1: How are TQRIS ratings defined, collected, and generated 

in the nine Round 1 RTT-ELC states and how might the structure and 

implementation of the rating system influence the reliability and validity of the 

ratings?

We will answer this question by drawing on information from various sources. First, we 

extracted information from the states’ RTT-ELC applications on the structure of their TQRIS, and 

their plans for refining their TQRIS under RTT-ELC. Second, we conducted a targeted review of 

publicly available documents, TQRIS websites, and documents received from the RTT-ELC states 

(such as program guides, TQRIS standards, and descriptions of TQRIS data systems) in response to 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

5

our request during initial study engagement telephone discussions. We will also review documents 

shared by the RTT-ELC program office (including state evaluation plans and annual performance 

reports) and request information pertaining to the nine states from ongoing research efforts. Last, 

we have conducted targeted interviews with representatives from each of the nine RTT-ELC states 

who have in-depth knowledge about the TQRIS, and are able to confirm and clarify the information 

obtained from documents, and can provide information we cannot get from the documents. 

 Research Question 2: Is it feasible to examine the relationship between TQRIS 

ratings and children’s kindergarten readiness outcomes in selected RTT-ELC

states?

We will assess the feasibility of examining the relationship between characteristics of centerbased ECE programs (as measured by TQRIS ratings) and children’s kindergarten readiness 

outcomes by determining the extent of center-based ECE program participation in states’ TQRIS, 

the availability of TQRIS and kindergarten entry assessment data, and whether it is possible to link 

students’ kindergarten entry assessment data to the TQRIS ratings of the center-based ECE 

programs they attended. Information related to feasibility will come from various sources. We 

obtained information from the states’ RTT-ELC application on their plans for implementing 

kindergarten entry assessments. Through reviews of documents obtained from state websites or 

received directly from the states in response to our requests during study engagement discussions, 

we gathered additional information about the subscales and characteristics of the kindergarten entry 

assessment as well as how and when it is administered. Finally, we conducted interviews to ascertain 

the extent of center-based ECE program participation, and where and how TQRIS and kindergarten 

entry assessment data are housed. To confirm and/or gather the information on TQRIS and 

kindergarten assessment data we interviewed up to four representatives from each of the Round 1 

RTT-ELC states, including a representative from the agency that oversees kindergarten entry 

assessments, a data manager or staff person familiar with the kindergarten entry assessment data 

system, and a data manager or staff person familiar with the TQRIS data system. The questions 

directed to each of the up to four respondents were specifically targeted so that no question was

asked of more than one respondent per state. No topic or related series of questions was asked of 

more than one respondent in each of the nine states. The team developed and used tailored 

protocols for each respondent, not only to reduce burden, but also to ensure that the same (or 

similarly worded) information was not collected from more than nine individuals – thereby allowing 

the information collected from State Representatives to be exempt from the Paperwork Reduction 

Act (PRA).

 Research Question 3: What is the relationship between TQRIS ratings and 

kindergarten readiness outcomes for programs that serve preschool-age children 

in selected RTT-ELC states?

We will examine the relationship between the TQRIS ratings and measures of children’s 

kindergarten readiness—that is, scores on kindergarten entry assessments—in a subset of the Round 

1 RTT-ELC states. We anticipate selecting two to three states for this part of the study based on the 

feasibility criteria described under research question 2. 

For each RTT-ELC state that is selected, we will (1) describe TQRIS ratings for center-based 

ECE programs and the characteristics and kindergarten readiness outcomes of children who attend 

such programs, and (2) estimate the associations between overall TQRIS ratings and kindergarten 

entry assessment scores, and (if data are available) between component- or indicator-level ratings 

and kindergarten entry assessment scores. We anticipate that data on TQRIS ratings will come from 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

6

state TQRIS databases. Data on kindergarten entry assessments and student demographic 

characteristics will come from state databases whenever possible. If we cannot obtain the necessary 

outcomes data (i.e., kindergarten entry assessment scores) from states, we will request such data 

from selected districts within the RTT-ELC states chosen for this study component. These districtlevel requests are the focus of this OMB request.

d. Recruitment plan

The recruitment process for the RTT-ELC TQRIS study will occur in two stages. First, we will 

reach out to each of the nine Round 1 RTT-ELC states to inform them of the study; participation in 

the study is a requirement of the RTT-ELC grant. The second stage, district recruitment, will be 

conducted only if needed. In this section, we first briefly describe the state recruitment process for 

context. The majority of this section discusses district recruitment, which is the focus of this 

supporting statement. 

State engagement. In September and October 2014, we contacted each of the nine Round 1 

RTT-ELC states to introduce the study, identify respondents for telephone interviews, and request 

documents pertaining to the structure of TQRIS and characteristics of kindergarten entry 

assessments in the nine Round 1 RTT-ELC states as well as the availability of TQRIS and 

kindergarten entry assessment data. We sent introductory packages to the RTT-ELC grant 

coordinators in each state that, which will included:

 A state engagement letter that explained the study’s importance and provided an 

overview of the study. 

 An information sheet that provided a nontechnical description of the study. This 

document included a summary of study activities, the study’s research questions, and a 

timeline for study activities.

A recruiter assigned to the state followed up with the state grant coordinator to confirm receipt 

of the mailing and to conduct a brief (approximately half-hour) telephone discussion with the 

appropriate agency staff person. Recruiters used a standardized protocol as a guide during these 

telephone conversations, which presented background information on the study, requested specific 

documents, and gathered contact information for use in scheduling telephone interviews. All nine 

state representatives agreed to participate in phase 1 of the study during these engagement calls and 

provided the appropriate contacts with whom to schedule the interviews.

District recruitment. If we learn through the interviews with state representatives that we

cannot obtain the necessary kindergarten entry assessment data from a centralized source at the state 

level, we will attempt to collect such data from individual school districts. If collecting data from 

districts proves necessary, we anticipate obtaining kindergarten entry assessment data from a 

targeted sample of 14 districts with 200 state-funded preschool programs.5,6 Based on recruiting rates 

 

5 According to the Common Core of Data, on average, the number of schools serving preschool-aged children in

districts serving preschool-aged children is 14, so the contractor is assuming that it will need to obtain data from 14 

districts in order to include roughly 200 state-funded preschools in the sample (14 = 200/14). Other types of centerbased ECE programs may also be included in the analysis if it is possible to link their TQRIS ratings with kindergarten 

entry assessment data. However, if the contractor needs to obtain data from districts, it will target districts based on the 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

7

from previous IES studies, we further anticipate that we would need to reach out to 42 districts to 

ultimately recruit 14 into the study. For all targeted districts, the contractor will send introductory 

packages to the district superintendent, which will include:

 A district recruitment letter (Appendix A) that explains the study’s importance, 

provides an overview of the study including estimated participation time and 

confidentiality information, and indicates that a member of the study team will call to 

provide more details. 

 An information sheet (Appendix B) that provides a nontechnical description of the 

study. This document includes a summary and timeline of study activities, the study’s 

research questions, time and obligation to participate, and confidentiality information.

 A district memorandum of understanding (Appendix C) that describes the agreedupon roles and responsibilities of the study team and of participating districts.

Recruiters will make follow-up calls to the district contacts to introduce the study and arrange 

for a time to further discuss the study. Recruiting communications with each district will take place 

by phone and email. Recruitment discussions will last approximately 1.5 hours and will be guided by 

a standardized recruitment protocol (Appendix D). The protocol will guide recruiters in describing 

the study design and planned data collection activities, as well as gathering information on the 

characteristics of center-based ECE programs in the district and the availability of district-level 

kindergarten entry assessment data. Based on the information gathered during these discussions with 

district officials, the study team will identify 14 districts to participate in the second study phase. We 

estimate that these recruiting efforts will take place in fall 2015 (as soon as OMB clearance is 

received).

District recruiter training and tracking. To help establish a strong rapport between the study 

team and the district administrators and their support staff, each district will be assigned one primary 

recruiter. Recruiters will participate in a half-day training session covering the following topics: (1) 

communicating the study’s importance, data collection needs, and methodological issues in effective, 

nontechnical terms, (2) describing the study, (3) procedures for district recruitment, (4) addressing 

potential recruitment issues, and (5) task management (including the recruiting schedule, weekly 

meetings, and use of the tracking system). Trainers will also share key strategies based on the 

contractor’s extensive experience recruiting districts for other IES studies, including the Impact 

Evaluation of Race to the Top and School Improvement Grants.

District research applications and MOUs. We will gain the necessary approvals and abide by 

the relevant guidelines for conducting the study in the 14 selected districts. In districts with policies 

concerning external research projects, we will seek expedited research application reviews where 

 

(continued)

number of state-funded preschools, since it is more likely that it will be able to link TQRIS ratings to kindergarten entry 

assessment data for children who attended such programs.

6 It is possible that only some of the two to three selected states will be able to provide the needed data and that 

the contractor will obtain the data from districts in the other state(s). The sample sizes shown here assume that none of 

the selected states can provide the needed data and that the contractor would need to collect all the necessary data from 

districts. If some states can provide data, then the sample for these analyses would be larger than 200 programs.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

8

possible and emphasize that the study has received prior review by its institutional review board 

(IRB).

Following district officials’ oral commitment to participate in the study, they will be sent a 

memorandum of understanding (MOU) to sign. The MOU will describe the agreed-upon roles and 

responsibilities of the study team and of participating districts (Appendix C). A representative of the 

study contractor will also sign the MOU.

All contacts with districts will be documented in an electronic tracking system to minimize 

redundant contacts and to provide the Contracting Officer’s Representative (COR) with up-to-date 

information on recruiting progress. All recruiters will have access to the web-based system.

e. Data collection needs

To address its research questions, the study will collect and analyze data from several sources, 

which we describe below. 

RTT-ELC applications. In February through April 2013, we reviewed RTT-ELC 

Applications for Initial Funding for the nine Round 1 RTT-ELC states. From the RTT-ELC 

applications, we gathered information about the structure of each state’s TQRIS, plans for 

expanding program participation in TQRIS, kindergarten entry assessments, and existing data 

systems.

Publicly available and other existing documents. From September through December

2014, we collected and conducted a targeted review of documents pertaining to the structure of 

TQRIS and characteristics of kindergarten entry assessments in the nine Round 1 RTT-ELC states 

as well as the availability of TQRIS and kindergarten entry assessment data. Documents came from 

three sources: (1) the RTT-ELC program office and ongoing research efforts, (2) state websites, and 

(3) state officials. The RTT-ELC program office will be our source for annual performance reports 

which describe states’ plans, activities, and progress, as well as other pertinent documents (such as 

validation plans) related to states’ TQRIS. Much information not already captured by the RTT-ELC 

program office or ongoing research efforts was available on state websites. To access documents 

that were not publicly available but offer important details about the TQRIS (such as revised 

standards that are not yet publicly available) and kindergarten entry assessments, we requested copies 

from state officials during initial study engagement conversations in September and October 2014

and subsequent interviews with select respondents in each state.

Interviews with state representatives. From December 2014 through March 2015, we 

conducted telephone interviews with state representatives from the nine Round 1 RTT-ELC states 

to confirm and clarify the information obtained from documents and, as needed, gathered

information that could not be obtained from the document reviews. During initial study engagement 

conversations, the team identified the appropriate representatives to interview. The team interviewed 

between two to four administrators in each state to cover four primary topics: (1) TQRIS rating 

structure, (2) TQRIS data, (3) kindergarten entry assessments, and (4) kindergarten entry assessment 

data. For example, in some states, the team interviewed the state TQRIS administrator to confirm or 

gather details about the quality rating components and overall rating structure and separately 

interviewed a researcher or data manager to understand details about data coverage and accessibility 

that the TQRIS administrator could not address. In other states, the TQRIS administrator was able 

to address topics related to both the TQRIS rating structure and the characteristics of the data. No 

topic or related series of questions was asked of more than one respondent in each of the nine 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

9

states. The team developed and used tailored protocols for each respondent, not only to reduce 

burden, but also to ensure that the same (or similarly worded) information was not collected from 

more than nine individuals – thereby allowing the information collected from State Representatives 

to be exempt from the Paperwork Reduction Act (PRA).

TQRIS data. In fall 2015, the contractor will obtain data on center-based ECE programs’ 

TQRIS ratings from state TQRIS databases and, if the data are available, component- and indicatorlevel ratings. TQRIS data from 2014–2015 will be collected from the two to three states participating 

in the second phase of the study (analysis of the relationship between TQRIS ratings and measures 

of children’s school readiness); we anticipate that these data will be available at the state level.

Kindergarten entry assessment data. In fall 2015 through early 2016, the study will aim to 

collect kindergarten assessment data from the two to three states selected for the analysis of TQRIS 

ratings and measures of children’s school readiness. We will request kindergarten entry assessment 

data from tests administered in fall 2015. If kindergarten entry assessment and TQRIS data from the 

prior year (that is, fall 2014 kindergarten entry assessments and 2013–2014 TQRIS ratings) are 

available, and if the kindergarten entry assessments and TQRIS standards used in that year were the 

same as those used in the next year (fall 2015 kindergarten entry assessments and 2014–2015 TQRIS 

ratings), we will request data for both years. We would prefer to obtain kindergarten entry 

assessment data from a centralized source at the state level. This approach would allow us to obtain 

data for all center-based ECE programs that participated in the TQRIS and whose students went on 

to attend kindergarten in the state. We will also request that the data include a student- and/or 

program-level identifier7that will allow us to link program-level TQRIS data to child-level 

kindergarten entry assessment data.8In addition to test scores and student and/or program 

identifiers, we will explore the availability of data on child gender, birth year, socioeconomic status, 

race and ethnicity, English language learner status, free or reduced-price lunch status, special 

education status, maternal education, and children’s length of enrollment in their center-based ECE 

programs.

9

If kindergarten entry assessment data are not available from a centralized source at the 

state level, we will seek to collect the data from up to 14 individual school districts. The data request 

will include a form that specifies the data needed for the study (Appendix E). 

f. Study activities and timeline

This clearance request pertains to the recruitment and collection of administrative data from 

districts, which will be conducted only if the data needed for the study cannot be obtained from 

states. Table A.1 shows the schedule of study activities. Since this package is requesting clearance 

only for district recruitment and data collection activities, only district-level activities listed in Table 

 

7 Identifiers will be replaced with a Mathematica ID to prevent disclosure of respondent data.

8 We expect that these data linkages will be more feasible for children who attended state-funded preschool 

programs, because they might be assigned a unique state or district identifier that is linked to their kindergarten data. 

However, there may be other ways to link preschool TQRIS ratings with kindergarten assessment data. For example, the 

kindergarten entry assessment data set may include a program ID for the preschool that a student attended, and it may 

be possible to use that program ID to obtain the preschool’s rating from the TQRIS data set. It may also be possible to 

obtain preschool enrollment rosters from states or districts that contain student IDs that match the student IDs used in 

the state or district kindergarten entry assessment databases, so that TQRIS data can be linked with child outcome data.

9 Personally identifiable information will be stored in separate files to ensure confidentiality.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

10

A.1 apply to this request. The timeline includes other major study activities to provide context for 

the district-level activities.

Table A.1. Study Activities and Timeline 

Study Activities Sample Respondent Mode Schedule

Review RTT-ELC 

grant applications

Round 1 RTT-ELC 

states (9) Not applicable Desk research 

February–April 

2013

Engage RTT-ELC 

states

Round 1 RTT-ELC 

states (9)

State RTT-ELC 

grant coordinator or 

designee

Telephone and email 

communication

September–

October 2014

Review documents

Round 1 RTT-ELC 

states (9) Not applicable Desk research

September–

December 2014

Interview RTT-ELC 

state representatives

Round 1 RTT-ELC 

states (9)

State RTT-ELC 

grant coordinator or 

designee Telephone interview

December 2014-

March 2015

Collect TQRIS data

Selected Round 1 

RTT-ELC states (2 

to 3)

State TQRIS 

administrator or 

designee

Secure electronic data 

submission Fall 2015

Recruit districts (if 

necessary)*

Approximately 42 

districts from 

selected Round 1 

RTT-ELC states (1 

to 3)

District 

superintendent or 

designee

Telephone and email 

communication Fall 2015

Collect kindergarten 

entry assessment 

(KEA) data and data 

on child 

demographics and 

other characteristics*

Selected Round 1 

RTT-ELC states (2 

to 3) and 

approximately 14 

districts in these 

states 

State or district KEA 

data administrator or 

designee

Secure electronic data 

submission Fall–Winter 2015

* Activities relevant for this supporting statement.

2. Purposes and uses of the data

To address the study’s research questions, the study will collect and analyze data from several 

sources. Table A.2 lists the questions and the data sources that will be used to answer them. This 

supporting statement is requesting clearance only for the study’s district recruiting and district data 

collection, which will be used to address the study’s third research question. All other data collection 

methods are based on publicly available information or involve targeted interviews with fewer than 

10 individuals. We describe the study’s use of each data source in more detail below. Information 

will be collected by Mathematica Policy Research, under contract with ED [contract number EDIES-10-C-0077].

Table A.2. Research Questions and Data Sources

Research Question Data Sources

1. How are TQRIS ratings defined, collected, and generated 

in the nine Round 1 RTT-ELC states and how might the 

structure and implementation of the rating system 

influence the reliability and validity of the ratings?

Application reviews, targeted document reviews, 

and targeted interviews with staff from the 9 Round 

1 RTT-ELC states
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

11

Research Question Data Sources

2. Is it feasible to examine the relationship between TQRIS 

ratings and children’s kindergarten readiness outcomes in 

selected RTT-ELC states?

Application reviews, targeted document reviews, 

and targeted interviews with staff from the 9 Round 

1 RTT-ELC states

3. What is the relationship between TQRIS ratings and 

kindergarten readiness outcomes for programs that serve 

preschool-aged children in selected RTT-ELC states?

TQRIS data from state TQRIS data systems in 2 to 

3 of the Round 1 RTT-ELC states

Kindergarten entry assessment data from state 

and/or district administrative data systems in 2 to 3 

of the Round 1 RTT-ELC states

State’s RTT-ELC Applications for Initial Funding. From the RTT-ELC applications, the 

study collected information about the structure of each state’s TQRIS, plans for expanding program 

participation in TQRIS, kindergarten entry assessments, and existing data systems. We will use this 

information to address research questions 1 and 2 and to inform the analysis for research question 3. 

Document reviews. Document reviews focused on describing the structure of TQRIS, the 

rigor and consistency in the measurement process, the influences on program entry into and

movement within each system (such as incentives), and key elements that administrators continue to 

refine and why. We also reviewed documents describing characteristics, properties, and 

administration procedures of the state kindergarten entry assessment. We used this information to 

address research questions 1 and 2 and will use it to select states to be included in analysis for 

research question 3.

State interviews. Interviews with state administrators confirmed and clarified the information 

obtained from documents and, as needed, gathered information that could not be obtained from the 

document review. We will use this information to address research questions 1 and 2 and to refine 

the design for research question 3.

TQRIS data. The main explanatory variable in the analysis for research question 3 will be a 

center-based ECE program’s TQRIS rating. The data pertaining to this variable will come from state 

TQRIS databases. We will also collect information on component- and indicator-level ratings if the 

data are available. These data will be used to examine the associations between the TQRIS 

component- and indicator-level ratings and child outcomes.

Kindergarten entry assessment data. The outcome data for research question 3 will come 

from student-level administrative databases maintained by states and/or districts. These data will 

allow us to assess the relationship between TQRIS ratings and measures of kindergarten readiness. 

We will request data on student characteristics, such as sex, race/ethnicity, birth year, grade, 

eligibility for free or reduced-price lunch, English language learner status, special education status,

and preschool enrollment history, in order to control for factors other than differences in centerbased ECE program quality that might correlate with child outcomes. We will also request that the 

data include a student- and/or program-level identifier that will allow us to link program-level 

TQRIS data to child-level kindergarten entry assessment data.

3. Use of technology to reduce burden

The recruitment plan for which clearance is being sought in this submission is designed to 

minimize burden on the districts being recruited. An electronic recruiting tracking database will be 

designed and used to store state and district contact history and relevant recruiting data collected in 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

12

the course of those communications. Effective use of this management tool will minimize redundant 

contacts with states and districts and allow for specific requests for information to be consolidated 

and coordinated among study team members. A summary of recruiting information will also be 

entered into Mathematica’s cross-study recruiting database, allowing further coordination of requests 

across ongoing education studies, as well as coordination of decisions on the timing of such 

requests, to further minimize burden while meeting each study’s requirements.

The data collection plan is also designed to obtain information in an efficient way that 

minimizes respondent burden. We will gather information only from existing data sources, using the 

most efficient methods available. Existing data sources will be electronic files provided by states (or 

districts, when not available from the state) that include TQRIS ratings, test scores for schooladministered kindergarten entry assessments, and student demographic information. We will work 

with state and/or district personnel to determine the most efficient and least burdensome way to 

collect these data and will capitalize on any electronic data systems already in place. Whenever 

possible, states and/or districts will be able to upload or enter data into any electronic spreadsheet, 

or to an equivalent file, and transfer it to Mathematica through a secure FTP site. Regardless of the 

form in which it is received, these data will be converted into a consistent format so that they can be 

combined with data submitted by other districts from the state and be suitable for analysis. If it is 

too burdensome or not possible for a state or district to provide data in electronic format, we will 

provide clear instructions on how to submit the information securely in hard-copy form, to be 

coded by the study team.

4. Efforts to avoid duplication of effort

Evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the

TQRIS ratings are associated with child outcomes. Five studies have examined the relationship 

between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 2009; Elicker et 

al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and all examine 

TQRIS developed prior to the RTT-ELC program. Only two of these five studies—Thornburg et al. 

(2009) and Sabol and Pianta (2012)—find consistent evidence of a correlation between TQRIS 

ratings and outcomes. ED determined that an in-depth, cross-state study examining the structure 

and characteristics of TQRIS in RTT-ELC states and the relationship between their ratings and

measures of kindergarten readiness is needed to inform the ongoing development of RTT-ELC 

states’ and other states’ TQRIS.

RTT-ELC states are required to validate their TQRIS by assessing whether the tiers of TQRIS 

reflect different levels of program quality and whether differences in quality ratings relate to child 

outcomes. This study will differ from efforts RTT-ELC states are undertaking in the following ways: 

 If feasible, the second component of this study will focus on the relationship between 

TQRIS ratings and kindergarten readiness measures for programs that serve preschoolaged children. We will select states and collect data with the intention of maximizing 

statistical power to examine this specific question. State validation studies will not 

necessarily focus on this particular age group or outcome. 

 This study will employ consistent methodology across states to facilitate comparisons 

between states. The methods used in states’ local evaluations will vary, depending on the 

specific questions each state has chosen to focus on. 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

13

5. Methods to minimize burden on small entities

No small businesses or entities will be involved as respondents.

6. Consequences of not collecting the data

The recruitment and data collection plan described in this submission is necessary for ED to 

examine the relationship between TQRIS ratings and kindergarten readiness measures for programs 

that serve preschool-aged children. Examining the relationship between TQRIS ratings and 

children’s kindergarten readiness is important since a key goal of RTT-ELC is helping to ensure that 

children enter kindergarten with the skills, knowledge, and dispositions toward learning that they 

need to succeed. Failing to conduct this study would mean missing a key opportunity to learn about 

the TQRIS ratings and support the decisions that states are making as they develop and/or refine 

their TQRIS. Without state and district (if the state records are not available) administrative records, 

ED will not be able to examine whether TQRIS ratings (and which TQRIS components) are related 

to children’s kindergarten readiness. The district recruitment and data collection activities that are 

the focus of this supporting statement will be conducted only if the data needed for the study’s 

analyses are not available at the state level.

7. Special circumstances

There are no special circumstances involved with recruitment or data collection activities.

8. Federal register announcement and outside consultation

a. Federal register announcement

The 60-day notice to solicit public comments was published in Volume 80, page 2685 of the 

Federal Register on January 20, 2015. The Department thanks Yasmina Vinci, Executive Director 

of the National Head Start Association, for her comments. The comments expressed concerns 

about potentially limited utility of the study due to the following factors: (1) correlating TQRIS 

ratings with KEA scores would not provide an accurate picture of how children’s outcomes arise as 

a result of complex interactions between experiences and environments; (2) results of the study 

would not be generalizable to states beyond the 2 or 3 included in the analysis, and; (3) Kindergarten 

Entry Assessments are not designed to validate TQRIS and using them for this purpose in this study 

may elicit high-stakes interpretations of the relationship between TQRIS and child outcomes.

We responded to this comment by highlighting strategies that the study team is implementing 

to ensure that results of the study can be useful to the field. These strategies were already in place 

prior to receipt of the public comment. As such, we did not make any changes to procedures or 

instruments based on the comments.

b. Consultations outside of the agency

The study team sought input on the study design from experts in early learning and TQRIS, 

evaluation design, and analysis during a technical working group (TWG) meeting held on July 30, 

2014. We will continue to consult with the TWG throughout the study on other issues that would 

benefit from their input. Table A.3 lists the TWG members. 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

14

Table A.3 Members of the Technical Working Group for the RTT-ELC Study

Name Title and Affiliation

Margaret Burchinal Senior Scientist

FPG Child Development Institute

University of North Carolina

James Elicker Associate Professor

Human Development & Family Studies

Purdue University

Beth Rous Department Chair and Professor

Educational Leadership Studies

University of Kentucky

Elizabeth Stuart Associate Professor

Bloomberg School of Public Health

Johns Hopkins University

Kathryn Tout Co-Director

Early Childhood Development 

Child Trends

Martha Zaslow Director

SRCD Office for Policy & Communications

Gail Zellman Senior Research Psychologist

RAND Corporation

9. Payments or gifts

The study does not plan to give gifts to states or districts for participating in the recruitment 

process or for providing data for the study.

10. Assurances of confidentiality

The study team has established procedures to ensure the confidentiality and security of its data

as permitted by law. This approach will be in accordance with all relevant regulations and 

requirements, particularly the Education Sciences Institute Reform Act of 2002, Title I, Subsection 

(c) of Section 183, which requires the IES director to “develop and enforce standards designed to 

protect the confidentiality of persons in the collection, reporting, and publication of data.” The 

study will also adhere to requirements of Subsection (d) of Section 183 prohibiting disclosure of 

individually identifiable information, as well as making the publishing or inappropriate 

communication of individually identifiable information by employees or staff a felony.

The study team will protect the full privacy and confidentiality, as permitted by law, of all 

individuals and programs that provide data. The study will not store data associated with personally 

identifiable information (PII), as study staff will assign random ID numbers to all data records and 

then strip any PII from the data records. In addition to the data safeguards described here, the study 

team will not identify respondent names, programs, schools, or districts in publicly available reports 

or findings, and if necessary, the study team will make an effort to mask distinguishing 

characteristics in study reports. A statement to this effect will be included with all requests for data:

“Mathematica Policy Research follows the confidentiality and data protection requirements of IES (The 

Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Administrative records obtained through this 

data collection will be used only for research purposes. The reports prepared for the study will summarize state-level 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

15

findings and will not associate findings with a specific district, program, or individual student. We will not provide 

information that identifies respondents to anyone outside the study team, except as permitted by law.”

Mathematica employs the following safeguards to ensure confidentiality:

 All Mathematica employees sign a pledge that emphasizes the importance of 

confidentiality and describes their obligation.

 Secure FTP services allow encrypted transfer of large data files with clients, if necessary. 

All internal networks are protected from unauthorized access using defense-in-depth 

best practices, which incorporate firewalls and intrusion detection and prevention 

systems. The networks are configured so that each user has a tailored set of rights, 

granted by the network administrator, to files approved for access and stored on the 

LAN. Access to hard-copy documents is strictly limited. Documents are stored in locked 

files and cabinets. Discarded materials are shredded.

 Computer data files are protected with passwords, and access is limited to specific users, 

who must change their passwords regularly and conform to strong password policies.

 Especially sensitive data are maintained on removable storage devices that are kept 

physically secure when not in use.

After the study concludes, the study data will be transmitted to the National Center for 

Education Statistics (NCES) for safekeeping as a restricted-use file. All other versions of the data 

will be destroyed. Prior to transmittal, the data will undergo careful screening, and modification if 

necessary, as a safeguard against unacceptably high level of disclosure risk for protected respondents. 

Researchers wishing to access the data for secondary analysis must apply for an NCES license and 

agree to the applicable rules and procedures guiding the use of restricted-use files.

11. Additional justification for sensitive questions

No questions of a sensitive nature will be included in this study.

12. Estimates of hours burden

If we identify a state as a good candidate for inclusion in the analysis of the relationship 

between TQRIS ratings and child outcomes but find that we cannot obtain kindergarten entry 

assessment data from a centralized source at the state level, we will obtain kindergarten entry 

assessment data from a targeted sample of districts within that state. We anticipate that officials and 

staff from approximately 42 school districts will participate in initial phone calls and follow-up 

communications that will occur during the district recruitment phase of the study in spring and 

summer 2015. We estimate that we will need to contact 42 districts to discuss data collection 

activities to ultimately recruit 14 districts to participate in the study based on our experiences with 

other projects, including the Impact Evaluation of Race to the Top and School Improvement 

Grants. 

We estimate that during the district recruitment process, we will speak with a main contact in all 

42 districts and a separate person for completing the MOU in the targeted 14 districts. We expect 

that initial and follow-up discussions with the primary contact in each district will take 1.5 hours of 

their time, on average, for a total of 63 hours for the upfront recruitment contact. Reviewing 

research applications and/or completing MOUs will take 6 hours, on average, per respondent for a 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

16

total of 84 hours across 14 districts. The total number of district staff involved in recruitment 

activities will be 56, and the total number of recruitment hours will be 147, for an average of 2.6

hours per person. Averaged over the 3-year clearance period, the total annual number of 

respondents and responses are 24, and the total annual burden hours are 87 (Table A.4). See 

Appendices A, B, C, and D for the materials and protocol to be used in district recruitment phone 

calls.

Table A.4. Burden in Hours to Respondents – RTT-ELC TQRIS Study Recruitment and Data Collection

Activity

Total 

Number of 

Respondents

Number of 

Responses 

Per 

Respondent

Number of 

Administrations

Total 

Number of 

Responses

Burden 

Hours Per 

Response

Total 

Burden 

(Hours)

District recruiting 

communications 

42 1 1 42 1.5 63

Review of research 

applications and/or 

MOUs

14 1 1 14 6 84

Student records 

request 

14 1 1 14 8 112

Total District Staff 70 70 259

Average Annual

Total 24 24 87

If state-level data are not available, officials from 14 school districts will be asked to provide the 

necessary student-level administrative records in fall and winter 2015. We estimate that it will take 8 

hours, on average, of district-level staff time to deliver the required student records in each of the 

participating 14 districts. The total number of burden hours for records delivery, therefore, will be 

112 (Table A.4). The data collection form is included in Appendix E and will include the following 

appropriately tailored Public Burden Statement:

According to the Paperwork Reduction Act of 1995, no persons are required to 

respond to a collection of information unless such collection displays a valid OMB 

control number. The valid OMB control number for this information collection is 

xxxx-xxxx. Public reporting burden for this collection of information is estimated to 

average 8 hours per response, including the time to review instructions, search 

existing data resources, gather the data needed, and complete and review the 

information collection. The obligation to respond to this collection is voluntary. If 

you have comments or concerns regarding the status of your individual submission 

of this form, please contact directly: Institute of Education Sciences, U.S. 

Department of Education, 555 New Jersey Ave. NW, Washington, DC 20208.

13. Estimates of cost burden to respondents

There are no additional respondent costs associated with this data collection beyond the burden 

estimated in item A12.

14. Estimates of annual costs to the federal government

The estimated cost for this three-year study is $2,968,330, or approximately $989,443 per year. 

This includes development of a detailed study design, data collection instruments, this justification 
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

17

package, state (and, if needed, district) recruitment, TQRIS and outcomes data collection, and 

analysis and report preparation.

15. Reasons for program changes or adjustments

This is a new data collection effort. 

16. Publication plans and schedule 

Table A.5 displays the anticipated timetable for project publications, which include an

evaluation brief, and a final report. The evaluation brief’s topic will be the descriptive analyses 

aligned with the study’s first research question, which examines the components and indicators

included in the TQRIS in each of the nine Round 1 RTT-ELC states and how they are used to 

generate overall quality ratings. This brief will be released separately and prior to the release of the

final report.

Table A.5. Timetable for Project Publications 

Deliverable Date

Evaluation Brief

Draft evaluation brief 

Revised draft evaluation brief 

Final evaluation brief 

June 2015

August 2015

November 2015

Report

Draft report 

Revised draft report 

Final report 

June 2016

August 2016

February 2017

17. Approval not to display the OMB expiration date

All recruitment and data collection materials will display the OMB expiration date.

18. Explanation of exceptions

No exceptions are requested.
Contract Number: ED-IES-10-C-0077 Mathematica Policy Research

18

REFERENCES

Elicker, J., C.C. Langill, K. Ruprecht, J. Lewsader, and T. Anderson. “Evaluation of ‘Paths to 

QUALITY,’ Indiana’s Child Care Quality Rating and Improvement System: Final Report 

(Technical Report #3).” Department of Human Development and Family Studies, Center for 

Families, Purdue University, 2011.

Lahti, M., T. Sabol, R. Starr, C. Langill, and K. Tout. “Validation of Quality Rating and 

Improvement Systems (QRIS): Examples from Four States.” Research-to-Policy, Research-to–

Practice Brief OPRE 2013-036. Washington, DC: Office of Planning, Research and Evaluation, 

Administration for Children and Families, U.S. Department of Health and Human Services, 

2013.

Thornburg, K.R., W.A. Mayfield, J.S. Hawks, and K.L. Fuger. “The Missouri Quality Rating System 

School Readiness Study.” Columbia, MO: University of Missouri Center for Family Policy and 

Research, 2009. Retrieved from http://mucentermissouri.edu/MOQRSreport.pdf. 

Tout, K., R. Starr, T. Isner, J. Cleveland, L. Albertson-Junkans, M. Soli, and K. Quinn. “Evaluation 

of Parent Aware: Minnesota’s Quality Rating and Improvement System Pilot.” Washington, 

DC: Child Trends, 2011.

U.S. Department of Education. “Race to the Top – Early Learning Challenge Application for Initial 

Funding.” 2011. Retrieved May 6, 2013 from http://www2.ed.gov/programs/racetothetopearlylearningchallenge/2011-412.doc. 

Zellman, G.L., R.N. Brandon, K. Boller, and J.L. Kreader. “Effective Evaluation of Quality Rating 

and Improvement Systems for Early Care and Education and School-Age Care.” Research-toPolicy, Research-to–Practice Brief OPRE 2011-11a. Washington, DC: Office of Planning, 

Research and Evaluation, Administration for Children and Families, U.S. Department of Health 

and Human Services, 2011.

Zellman, G.L., and R. Fiene. “Validation of Quality Rating and Improvement Systems for Early 

Care and Education and School-Age Care.” Research-to-Policy, Research-to–Practice Brief 

OPRE 2012-29. Washington, DC: Office of Planning, Research and Evaluation, Administration 

for Children and Families, U.S. Department of Health and Human Services, 2012.

Zellman, G.L., M. Perlman, V. Le, and C. Setodji. “Assessing the Validity of the Qualistar Early 

Learning Quality Rating and Improvement System as a Tool for Improving Child-Care 

Quality.” Pittsburgh, PA: Rand Corporation, 2008. Accessed November 26, 2012 at 

http://www.rand.org/pubcs/monographs/MG650.html.
www.mathematica-mpr.com

Improving public well-being by conducting high quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research