**Race to the Top -**  **Early Learning Challenge**  **: Descriptive Study of**  **Tiered Quality**

**Rating and Improvement Systems**  **in Nine**  **Round 1 States : OMB**  **Forms Clearance**  **Package ,**  **Part A** April 21 , 201 5Contract Number:

ED - IES - 10 - C- 0077 Mathematica Reference Number: 06844.144 Submitted to: National Center for Education Evaluation 555 New Jersey Ave., NW , Suite 500k Washington, DC 20208 Project Officer: Tracy Rimdzius Submitted by: Mathematica Policy Research P.O. Box 2393 Princeton, NJ 08543 -2393 Telephone: (609) 799- 3535 Facsimile: (609) 799- 0005 Project Director: Susanne James - Burdumy, PhD **Race to the Top -**  **Early Learning Challenge**  **: Descriptive Study of**  **Tiered Quality**

**Rating and Imp**  **rovement Systems in Nine**  **Round 1 States :**  **OMB Forms**  **Clearance Package**  **, Part A**  April 21 , 201 5iii **CONTENTS** PART A. SUPPORTING STATEMENT FOR PAPER WO RK REDUCTION ACT SUBMISSION ................................ ................................ ................................ ... 1 A. Justification ................................ ................................ ................................ 2 1. Circumstances necessitating the collection of information .................. 2 2. Purposes and uses of the data ................................ .......................... 10 3. Use of technology to reduce burden ................................ .................. 11 4. Efforts to avoid duplication of effort ................................ ................... 12 5. Methods to minimize burden on small entities ................................ ... 13 6. Consequences of not collecting the data ................................ ........... 13 7. Special circumstances ................................ ................................ ....... 13 8. Federal register announcement and outs ide consultation ................. 13 9. Payments or gifts ................................ ................................ ............... 14 10. Assurances of confidentiality ................................ ............................. 14 11. Additional justification for sensitive questions ................................ .... 15 12. Estimates of hours burden ................................ ................................ . 15 13. Estimates of cost burden to responden ts ................................ .......... 16 14. Estimates of annual costs to the federal government ........................ 16 15. Reasons for program changes or adju stments ................................ .. 17 16. Publication plans and schedule ................................ ......................... 17 17. Approval not to display the OMB expiration date ............................... 17 18. Explanation of exceptions ................................ ................................ .. 17 REFERENCES ................................ ................................ ................................ .............. 18 APPENDIX A: DISTRICT LETTER APPENDIX B: STUDY IN FORMATION SHEET APPENDIX C : DISTRICT MEMORANDUM OF UNDERSTANDING APPENDIX D : RECRUITING PROTOCOL WI TH DISTRICT ADMINI STRATOR APPENDIX E : DATA COLLECTION FORM FOR DISTRICT- LEVEL

KINDERGARTEN ASSESSM ENT DATAiv **TA BLES**  A.1 Study Activities and Timeline ................................ ................................ ......... 10 A.2 Research Questions and Data Sources ................................ ......................... 10 A.3 Burden in Hours to Respondents – RTT- ELC TQRIS Study

Recruitment and Data Collection ................................ ................................ .... 16 A.4 Timetable for Project Publications ................................ ................................ .. 17Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

1 **PA RT**  **A. SUPPORTING S**  **TA TEMENT FOR PA** **PERWO RK** 

**REDUCTION A CT SUBMIS**  **SION** This supporting statement requests clearance to recruit and collect data from approximately 14 school districts to support the Study of Race to the Top
–Early Learning Challenge Tiered Quality Rating and Improvement Systems (RTT
-ELC TQRIS). Through TQRIS, states use a set of standards to evaluate the quality of early learning and development programs and promote program improvement. The RTT
-ELC application for the first round of grants required states to develop and adopt a TQRIS that, according to the U.S. Department of Education (ED), “is based on a statewide set of tiered program standards that include: (1) early learning and development standards, (2) a comprehensive assessment system, (3) early childhood educator qualifications, (4) family engagement strategies, (5) health promotion practices, and (6) effective data practices; is clear and has standards that are measurable, meaningfully differentiate program quality levels, and reflect high expectations of program excellence commensurate with nationally recognized standards that lead to improved learning outcomes for children; and is linked to the state licensing system for early learning and development programs” (2011). The study of RTT
-ELC TQRIS will provide important information on the structure and characteristics of TQRIS and whether and how the TQRIS ratings relate to child outcomes. This information is intended to inform improvements to states’ TQRIS. The Institute of Education Sciences (IES) at ED contracted with Mathematica Policy Research to conduct this study, which is being carried out as a component of the Impact Evaluation of Race to the Top and School Improvement Grants. 1 The RTT
-ELC TQRIS study comprises two distinct phases. The first study phase is a descriptive analysis of the TQRIS in the nine Round 1 RTT
-ELC states. In each state, we will examine the structure of the TQRIS, including the components and indicators included in states’ TQRIS and how they are combined to generate overall quality ratings . 2 We will also explore the rigor and consistency in the measurement process (both in defining the indicators and collecting the rating information), the influences on program entry into and movement within each system (such as incentives), and key elements that administrators continue to refine and why. This phase will include the nine round 1 RTT
-ELC states and rely on data gathered from their grant applications, reviews of public and other existing documents, and targeted discussions with RTT
-ELC grant coordinators (or their designees) in these nine states. 3 (Clearance is not sought for these state-level data coll ection activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering information on the structure and characteristics of TQRIS, we will also assess the feasibility of examining the relationship between characteristics of center-based early care and education (ECE) 1 The Office of Management and Budget (OMB) has already approved recruitment and data collection activities for the Impact Evaluation of Race to the Top and School Improvement Grants. Recruitment activities were approved under OMB# 1850
- 0884 on June 27, 201 1. Data collection activities were approved under OMB# 1850
- 0884 on April 4, 2012 . 2 TQRIS standards are commonly grouped into categories or components . Examples of components of TQRIS standards include health and safety, curriculum and learning, environment, staff qualifications, family engagement, and administration and management. In TQRIS , programs are typically assigned component-level ratings that are then combined to determine an overall quality rating. Each component is compo sed of indicators , which are the specific criteria used to assess a program. 3 Documents will include TQRIS websites, and documents received from the RTT
-ELC state s (such as program guides, TQRIS standards, and descriptions of TQRIS data systems). We will also review documents shared by the RTT
ELC program office (including state evaluation plans and annual performance reports) and request information pertaining to the nine states from ongoing research efforts, such as the QRIS online compendium.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

2 programs (as measured by TQRIS ratings) and children’s kindergarten readiness outcomes . If this proves feasible, the second study phase will examine the relationship between TQRIS ratings and child outcome measures in two to three of the Round 1 RTT
-ELC states where the conditions exist to make such analysis possible. For this phase of the study, data on TQRIS ratings, component
-level ratings, and indicator
-level ratings will come from state TQRIS databases. D ata on kindergarten entry assessments will also be sought from databases administered at the state
-level. In the event that the kindergarten data needed for the analysis are not available from state databases, it may be necessary to reach out to selected districts in the RTT
-ELC states to collect such data. If this step proves necessary, the contractor estimates (based on study power requirements and prior recruiting experiences) needing to reach out to up to 42 districts in order to ultimately recruit 14 districts from which to collect outcomes data. Because more than nine entities may need to be included in such recruitment and data collection activities, this supporting statement seeks clearance for the materials and activities that would support the study’s district recruitment efforts and district
-level data collection if these activities are needed . This document includes drafts of the district recruitment letter (Appendix A), the RTT
-ELC TQRIS study information sheet (Appendix B), district memorandum of understanding (Appendix C), district recruitment call protocol (Appendix D), and district data request form (Appendix E). **A. Justification**  **1. Circumstances necessitating the collection of information**  **a. Statement of need for a rigorous**  **evaluation** The RTT
-ELC grants program is a joint effort by the U.S. Departments of Education (ED) and Health and Human Services (HHS) to strengthen the quality of early learning and development programs, and to increase access to high quality programs for all children, particularly low
-income and disadvantaged children. The states that were awarded RTT
-ELC grants were required to commit to integrating resources and aligning policies across the various state agencies that administer public funds related to early learning and development. RTT
-ELC seeks to improve early learning and development programs by supporting reforms in five areas: (1) integrating state early care and education systems; (2) increasing access to high
-quality, accountable programs; (3) promoting early learning and development outcomes through statewide standards and comprehensive assessment systems; (4) developing and supporting the early childhood workforce; and (5) building early learning data systems to measure child outcomes and progre ss. ED and HHS have awarded RTT
-ELC grants through three rounds of competition. Notices for the first competition
—

which was open to all states, the District of Columbia, and Puerto Rico
— were posted in August 2011. In that first round of competition, 35 states, the District of Columbia, and Puerto Rico applied for RTT
-ELC grants. 4 Round 1 awards were made in December 2011 to nine states—California, Delaware, Maryland, Massachusetts, Minnesota, North Carolina, Ohio, Rhode Island, and Washington. Across these states, a total of $500 million dollars was awarded, with grants ranging from $45 million to $70 million. A second round of RTT
-ELC grants was announced in September 2012. Because more modest funding was available, the second round invited the next five highest scoring finalists from Round 1
—Colorado, Illinois, New Mexico, Oregon, and 4 Alabama, Alaska, Idaho, Indiana, Louisiana, Montana, New Hampshire, North Dakota, South Carolina, South Dakota, Tennessee, Texas, Utah, Virginia, and Wyoming did not apply for RTT
- ELC grants in the first round.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

3 Wisconsin
—to modify their 2011 applications and create plans that could be funded with up to 50 percent of the amounts originally requested. All five invited states applied and, in December 2012, were awarded RTT
-ELC grants in amounts ranging from $21 million to $35 million. A third round of competition was announced in April 2013. Awards ranging from $36 million to $51 million were made in December 2013 to six new states—Georgia, Kentucky, Michigan, New Jersey, Pennsylvania, and Vermont. In addition, supplementary grants were provided to six states from rounds 1 and 2 (California, Colorado, Illinois, New Mexico, Oregon and Wisconsin) that had previously only received 50 percent of the funds they requested. This study focuses on the nine states awarded RTT
-ELC grants in Round 1. A key aspect of the RTT
-ELC program is the development and implementation of TQRIS under the second reform area noted above (that is, increasing access to high quality, accountable programs). TQRIS aim to improve child care program quality and child outcomes by promulgating rigorous standards, rating providers based on those standards, and publicizing the ratings of individual providers (Zellman et al. 20 11). The usefulness of TQRIS ratings thus depends on their ability to reliably measure the components that comprise the overall rating and construct from them an overall rating that is a valid indicator of program differences with regard to child outcomes (Cizek 2007, as cited in Zellman and Feine 2012). Despite growing interest in the use of TQRIS to identify high quality early learning programs, evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the TQRIS ratings are associated with child outcomes. To our knowledge, five studies have examined the relationship between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 2009; Elicker et al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and all examine TQRIS developed prior to the RTT
-ELC program. Only two of these five studies
— Thornburg et al. (2009) and Sabol and Pianta (2012)
—find evidence of a correlation between TQRIS ratings and outcomes. Thornburg and colleagues (2009) found that children attending programs with 3
- , 4
- or 5
- star ratings from Missouri’s TQRIS showed significantly greater gains during the preschool year in social skills and behavioral outcomes relative to children attending programs with 1
- or 2
- star ratings. They did not find statistically significant differences in children’s vocabulary, early literacy, or math skills. Sabol and Pianta (2012) found that children who attended programs with 3
- or 4
- star ratings from Virginia’s TQRIS exhibited significantly larger increases in literacy skills over the prekindergarten year compared with children attending programs with 2
- star ratings. To learn about TQRIS in states that have received RTT
-ELC grants and to inform ongoing development of states’ TQRIS , ED has contracted with Mathematica Policy Research to conduct this study. According to the program requirements included in the Notice Inviting Applications (NIA) (76 FR 53564), states awarded an RTT
–ELC grant are required to “(1) comply with the requirements of any evaluation sponsored by ED or HHS of any of the State’s activities carried out with the grant; (2) comply with the requirements of any cross - state evaluation
—as part of a consortium of States
— of any of the State’s proposed reforms, if that evaluation is coordinated or funded by ED or HHS, including by using common measures and data collection instruments and collecting data necessary to the evaluation.” Legislative authorization for the RTT
-ELC TQRIS study is found in the Education Science Reform Act of 2002, Part D, Section 171(b)(2), which authorizes IES to “conduct evaluations of Federal education programs administered by the Secretary (and as time and resources allow, other education programs).” **b. Research**  **q uestions** Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

4 The study of RTT
-ELC TQRIS is designed to answer the following research questions: 1. How are TQRIS ratings defined, collected, and generated in the nine Round 1 RTT
ELC states and how might the structure and implementation of the rating system influence the reliability and validity of the ratings? 2. Is it feasible to examine the relationship between TQRIS ratings and children’s kindergarten readiness outcomes in selected RTT
-ELC states? 3. What is the relationship between TQRIS ratings and kindergarten readiness outcomes for programs that serve preschool
-age children in selected RTT
-ELC states? **c. Study**  **d esign**  As noted above, the RTT
-ELC TQRIS study comprises two distinct phases. The first study phase is a descriptive analysis of the the TQRIS in the nine Round 1 RTT
-ELC states. In each state, we will examine the structure of the TQRIS, including components and indicators included in states’ TQRIS and how they are combined to generate overall quality ratings. We will also explore the rigor and consistency in the measurement process (both in defining the indicators and collecting the rating information), the influences on program entry into and movement within each system (such as incentives) , and key elements that administrators continue to refine and why. This phase will include the nine Round 1 RTT
-ELC states. (Clearance is not sought for these state-level data collection activities, as they require only targeted interactions with fewer than 10 respondents.) In gathering information on the structure and characteristics of TQRIS, we will also assess the feasibility of examining the relationship between characteristics of center-based ECE programs (as measured by TQRIS ratings) and children’s kindergarten readiness outcomes . If this proves feasible, the second study phase will examine the relationship between TQRIS ratings and child outcome measures in two to three of the Round 1 RTT
-ELC states where the conditions exist to make such analysis p ossible. For this study phase, the contractor will seek to collect the necessary information from databases administered at the state level. I f the kindergarten data needed for the analysis are not available from state databases, it may be necessary to reach out to approximately 42 districts to ultimately recruit 14 districts from which to collect outcomes data. Because more than nine entities may need to be included in such recruitment and data collection activities, this supporting statement seeks clearance for the materials and activities that would support the study’s district recruitment efforts and district
-level data collection if these activities are needed . To provide context for the district recruitment and data collection activities, we provide an overview of the study design and data collection plans for both phases in the remainder of this section.  **Research Question**  **1: How**  **are TQRIS**  **ratings defined,**  **collected, and**  **generated in**  **the nine**  **Round 1**  **RTT -** **ELC states**  **and how**  **might the**  **structure and**  **impl ementation**  **of the**  **rating system**  **influence the**  **reliability and**  **validity of**  **the ratings**  **?** We will answer this question by drawing on information from various sources. First, we extracted information from the states’ RTT
-ELC applications on the structure of their TQRIS, and their plans for refining their TQRIS under RTT
-ELC. Second, we conducted a targeted review of publicly available documents, TQRIS websites, and documents received from the RTT
-ELC states (such as program guides, TQRIS standards, and descriptions of TQRIS data systems) in response toContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

5 our request during initial study engagement telephone discussions. We will also review documents shared by the RTT
-ELC program office (including state evaluation plans and annual performance reports) and request information pertaining to the nine states from ongoing research efforts. Last, we have conducted targeted interviews with representatives from each of the nine RTT
-ELC states who have in
-depth knowledge about the TQRIS, and are able to confirm and clarify the information obtained from documents, and can provide information we cannot get from the documents.  **Research Question**  **2: Is**  **it feasible**  **to examine**  **the relationship**  **between TQRIS**  **ratings and**  **children’s kindergarten**  **readiness outcomes**  **in selected**  **RTT -** **ELC states?**  We will assess the feasibility of examining the relationship between characteristics of centerbased ECE programs (as measured by TQRIS ratings) and children’s kindergarten readiness outcomes by determining the extent of center-based ECE program participation in states’ TQRIS, the availability of TQRIS and kindergarten entry assessment data, and whether it is possible to link students’ kindergarten entry assessment data to the TQRIS ratings of the center-based ECE programs they attended. Information related to feasibility will come from various sources. We obtained information from the states’ RTT
-ELC application on their plans for implementing kindergarten entry assessments. Through reviews of documents obtained from state websites or received directly from the states in response to our requests during study engagement discussions, we gathered additional information about the subscales and characteristics of the kindergarten entry assessment as well as how and when it is administered. Finally, we conducted interviews to ascertain the extent of center
-based ECE program participation, and where and how TQRIS and kindergarten entry assessment data are housed. To confirm and/or gather the information on TQRIS and kindergarten assessment data we interviewed up to four representatives from each of the Round 1 RTT
-ELC states, including a representative from the agency that oversees kindergarten entry assessments, a data manager or staff person familiar with the kindergarten entry assessment data system, and a data manager or staff person familiar with the TQRIS data system . The questions directed to each of the up to four respondents were specifically targeted s o that no question was asked of more than one respondent per state. No topic or related series of questions was asked of more than one respondent in each of the nine states. The team developed and used tailored protocols for each respondent, not only to reduce burden, but also to ensure that the same (or similarly worded) information was not collected from more than nine individuals
– thereby allowing the information collected from State Representatives to be exempt from the Paperwork Reduction Act (PRA).  **Research Question**  **3: What**  **is the**  **relationship between**  **TQRIS ratings**  **and kindergarten**  **readiness outc** **omes for**  **programs that**  **serve preschool** **- age**  **children in selected RTT** **- ELC states?**  We will examine the relationship between the TQRIS ratings and measures of children’s kindergarten readiness
—that is, scores on kindergarten entry assessments
— in a subset of the Round 1 RTT
-ELC states. We anticipate selecting two to three states for this part of the study based on the feasibility criteria described under research question 2. For each RTT
-ELC state that is selected, we will (1) describe TQRIS ratings for center-based ECE programs and the characteristics and kindergarten readiness outcomes of children who attend such programs, and (2) estimate the associations between overall TQRIS ratings and kindergarten entry assessment scores, and (if data are available) between component- or indicator
-level ratings and kindergarten entry assessment scores. We anticipate that data on TQRIS ratings will come fromContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

6 state TQRIS databases. Data on kindergarten entry assessments and student demographic characteristics will come from state databases whenever possible. If we cannot obtain the necessary outcomes data (i.e., kindergarten entry assessment scores) from states, we will request such data from selected districts within the RTT
-ELC states chosen for this study component. These district
level requests are the focus of this OMB request. **d. Recruitment plan**  The recruitment process for the RTT
-ELC TQRIS study will occur in two stages. First, we will reach out to each of the nine Round 1 RTT
-ELC states to inform them of the study; participation in the study is a requirement of the RTT
-ELC grant. The second stage, district recruitment, will be conducted only if needed. In this section, we first briefly describe the state recruitment process for context. The majority of this section discusses district recruitment, which is the focus of this supporting statement. **State engagement** **.** In September and October 2014, we contacted each of the nine Round 1 RTT
-ELC states to introduce the study, identify respondents for telephone interviews, and request documents pertaining to the structure of TQRIS and characteristics of k indergarten entry assessments in the nine Round 1 RTT
-ELC states as well as the availability of TQRIS and kindergarten entry assessment data. We sent introductory packages to the RTT
-ELC grant coordinators in each state that, which will included:  A **state engagement**  **letter** that explained the study’s importance and provided an overview of the study.  An **information sheet**  that provided a nontechnical description of the study. This document included a summary of study activities, the study’s research questions, and a timeline for study activities. A recruiter assigned to the state followed up with the state grant coordinator to confirm receipt of the mailing and to conduct a brief (approximately half
-hour) telephone discussion with the appropriate agency staff person. Recruiters used a standardized protocol as a guide during these telephone conversations, which presented background information on the study, requested specific documents, and gathered contact information for use in scheduling telephone interviews . All nine state representatives agreed to participate in phase 1 of the study during these engagement calls and provided the appropriate contacts with whom to schedule the interviews. **District recruitment.**  If we learn through the interviews with state representatives that we cannot obtain the necessary kindergarten entry assessment data from a centralized source at the state level, we will attempt to collect such data from individual school districts. If collecting data from districts proves necessary, we anticipate obtaining kindergarten entry assessment data from a targeted sample of 14 districts with 200 state- funded preschool programs. 5 , 6 Based on recruiting rates
5 According to the Common Core of Data, on average, the number of schools ser ving preschool
- aged children in districts serving preschool
- aged children is 14, so the contractor is assuming that it will need to obtain data from 14 districts in order to include roughly 200 state- funded preschools in the sample (14 = 200/14). Other types of centerbased ECE programs may also be included in the analysis if it is possible to link their TQRIS ratings with kindergarten entry assessment data. However, if the contractor need s to obtain data from districts, it will target districts based on theContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

7 from previous IES studies, we further anticipate that we would need to reach out to 42 districts to ultimately recruit 14 into the study. For all targeted districts, the contractor will send introductory packages to the district superintendent, which will include:  A **district recruitment**  **letter** (Appendix A) that explains the study’s importance, provide s an overview of the study including estimated participation time and confidentiality information, and indicate s that a member of the study team will call to provide more details.  An **information sheet**  (Appendix B) that provide s a nontechnical description of the study. This document includes a summary and timeline of study activities , the study’s research questions , time and obligation to participate, and confidentiality information .  A **district memorandum**  **of understanding**  (Appendix C) that describes the agreed
upon roles and responsibilities of the study team and of participating districts . Recruiters will make follow
-up calls to the district contacts to introduce the study and arrange for a time to further discuss the study. Recruiting communications with each district will take place by phone and email. Recruitment discussions will last approximately 1.5 hours and will be guided by a standardized recruitment protocol (Appendix D) . The protocol will guide recruiters in describing the study design and planned data collection activities , as well as gathering information on the characteristics of center-based ECE programs in the district and the availability of district
-level kindergarten entry assessment data. Based on the information gathered during these discussions with district officials, the study team will identify 14 districts to participate in the second study phase. We estimate that these recruiting efforts will take place in fall 2015 (as soon as OMB clearance is received). **District r ecruit** **er t** **raining and t** **racking.** To help establish a strong rapport between the study team and the district administrators and their support staff, each district will be assigned one primary recruiter. Recruiters will participate in a half
-day training session covering the following topics: (1) communicating the study’s importance, data collection needs , and methodological issues in effective, nontechnical terms, (2) describing the study, (3) procedures for district recruitment, (4) addressing potential recruitment issues , and (5) task management ( including the recruiting schedule, weekly meetings, and use of the tracking system). Trainers will also share key strategies based on the contractor’s extensive experience recruiting districts for other IES studies , including the Impact Evaluation of Race to the Top and School Improvement Grants . **District r** **esearch a** **pplications and MOUs.** We will gain the necessary approvals and abide by the relevant guidelines for conducting the study in the 14 selected districts . In districts with policies concerning external research projects, we will seek expedited research application reviews where (continued) number of state-

funded preschools, since it is more likely that it will be able to link TQRIS ratings to kindergarten entry assessment data for children who attended such programs. 6 It is possible that only some of the two to three selected states will be able to provide the needed data and that the contractor will obtain the data from districts in the other state(s). The sample sizes shown here assume that none of the selected states can provide the needed data and that the contractor would need to collect all the necessary data from districts. If some states can provide data, then the sample for these analys e s would be larger than 200 programs .Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

8 possible and emphasize that the study has received prior review by its institutional review board (IRB). Following district officials’ oral commitment to participate in the study, they will be sent a memorandum of understanding (MOU) to sign. The MOU will describe the agreed
-upon roles and responsibilities of the study team and of participating districts (Appendix C) . A representative of the study contractor will also sign the MOU. All contacts with districts will be documented in an electronic tracking system to minimize redundant contacts and to provide the C ontracting O fficer’s Representative (COR ) with up -to -date information on recruiting progress. All recruiters will have access to the web
-based system. **e. Data collection needs**  To address its research questions, the study will collect and analyze data from several sources , which we describe below. **RTT -** **ELC applications.**  In February through April 2013, we reviewed RTT
-ELC Applications for Initial Funding for the nine Round 1 RTT
-ELC states. From the RTT
-ELC applications, we gathered information about the structure of each state’s TQRIS, plans for expanding program participation in TQRIS, kindergarten entry assessments, and existing data systems. **Publicly available**  **and other**  **existing documents.**  From September through December 2014, we collected and conducted a targeted review of documents pertaining to the structure of TQRIS and characteristics of kindergarten entry assessments in the nine Round 1 RTT
-ELC states as well as the availability of TQRIS and kindergarten entry assessment data. Documents came from three sources: (1) the RTT
-ELC program office and ongoing research efforts, (2) state websites , and (3) state officials. The RTT
-ELC program office will be our source for annual performance reports which describe states’ plans, activities, and progress, as well as other pertinent documents (such as validation plans) related to states’ TQRIS. Much information not already captured by the RTT
-ELC program office or ongoing research efforts was available on state websites. To access documents that were not publicly available but offer important details about the TQRIS (such as revised standards that are not yet publicly available) and kindergarten entry assessments, we requested copies from state officials during initial study engagement conversations in September and October 2014 and subsequent interviews with select respondents in each state . **Interviews with**  **state representatives.**  From December 2014 through March 2015 , we conducted telephone interviews with state representatives from the nine Round 1 RTT
-ELC states to confirm and clarify the information obtained from documents and, as needed, gathered information that could not be obtained from the document reviews. During initial study engagement conversations, the team identif ied the appropriate representatives to interview. The team interviewed between two to four administrators in each state to cover four primary topics : (1) TQRIS rating structure, (2) TQRIS data, (3) kindergarten entry assessments, and (4) kindergarten entry assessment data. For example, in some states , the team interviewed the state TQRIS administrator to confirm or gather details about the quality rating components and overall rating structure and separately interviewed a researcher or data manager to understand details about data coverage and accessibility that the TQRIS administrator could not address. In other states, the TQRIS administrator was able to address topics related to both the TQRIS rating structure and the characteristics of the data. N o topic or related series of questions was asked of more than one respondent in each of the nineContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

9 states. The team develop ed and used tailored protocols for each respondent, not only to reduce burden, but also to ensure that the same (or similarly worded) information was not collected from more than nine individuals
– thereby allowing the information collected from State Representatives to be exempt from the Paperwork Reduction Act (PRA) . **TQRIS d** **ata.** In fall 2015, the contractor will obtain data on center-based ECE programs’ TQRIS ratings from state TQRIS databases and, if the data are available, component- and indicator
level ratings. TQRIS data from 2014
–2015 will be collected from the two to three states participating in the second phase of the study (analysis of the relationship between TQRIS ratings and measures of children’s school readiness); we anticipate that these data will be available at the state level. **Kindergarten e** **ntry a** **ssessment d** **ata.** In fall 2015 through early 2016 , the study will aim to collect kindergarten assessment data from the two to three states selected for the analysis of TQRIS ratings and measures of children’s school readiness. We will request kindergarten entry assessment data from tests administered in fall 2015. If kindergarten entry assessment and TQRIS data from the prior year (that is, fall 2014 kindergarten entry assessments and 2013
–2014 TQRIS ratings) are available, and if the kindergarten entry assessments and TQRIS standards used in that year were the same as those used in the next year (fall 2015 kindergarten entry assessments and 2014
–2015 TQRIS ratings), we will request data for both years. We would prefer to obtain kindergarten entry assessment data from a centralized source at the state level. This approach would allow us to obtain data for all center-based ECE program s that participated in the TQRIS and whose students went on to attend kindergarten in the state. We will also request that the data include a student
- and/ or program-level identifier7 that will allow us to link program-level TQRIS data to child
-level kindergarten entry assessment data. 8 In addition to test scores and student and/ or program identifiers, we will explore the availability of data on child gender, birth year, socioeconomic status, race and ethnicity, English language learner status, free or reduced
-price lunch status, special education status, maternal education, and children’s length of enrollment in their center-based ECE program s. 9 If kindergarten entry assessment data are not available from a centralized source at the state level, we will seek to collect the data from up to 14 individual school districts. The data request will include a form that specifies the data needed for the study (Appendix E). **f. Study activities and timeline**  This clearance request pertains to the recruitment and collection of administrative data from districts , which will be conducted only if the data needed for the study cannot be obtained from states . Table A.1 shows the schedule of study activities . Since this package is requesting clearance only for district recruitment and data collection activities, only district
-level activities listed in Table
7 Identifiers will be replaced with a Mathematica ID to prevent disclosure o f respondent data. 8 We expect that these data linkages will be more feasible for children who attended state- funded preschool programs, because they might be assigned a unique state or district identifier that is linked to their kindergarten data. However, there may be other ways to link preschool TQRIS ratings with kindergarten assessment data. For example, the kindergarten entry assessment data set may include a program ID for the preschool that a student attended, and it may be possible to use that program ID to obtain the preschool’s rating from the TQRIS data set. It may also be possible to obtain preschool enrollment rosters from states or districts that contain student IDs that match the student IDs used in the state or district kindergarten entry as sessment databases, so that TQRIS data can be linked with child outcome data. 9 Personally identifiable information will be stored in separate files to ensure confidentiality.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

10 A.1 apply to this request. The timeline includes other major study activities to provide context for the district
-level activities . **Table A.1. Study**

**A ctivities and Timeline**  Study Activities Sample Respondent Mode Schedule Review RTT- ELC grant applications Round 1 RTT- ELC states (9) Not applicable Desk research February–April 2013 Engage RTT- ELC states Round 1 RTT- ELC states (9) State RTT- ELC grant coordinator or designee Telephone and email communication September – October 2014 Review documents Round 1 RTT- ELC states (9) Not applicable Desk research September – December 2014 Interview RTT- ELC state representatives Round 1 RTT- ELC states (9) State RTT- ELC grant coordinator or designee Telephone interview December 2014 - March 2015 Collect TQRIS data Selected Round 1 RTT- ELC states (2 to 3) State TQRIS administrator or designee Secure electronic data submission Fall 2015 Recruit districts (if necessary)* Approximately 42 districts from selected Round 1 RTT- ELC states (1 to 3) District superintendent or designee Telephone and email communication Fall 2015 Collect kindergarten entry assessment (KEA) data and data on child demographics and other characteristics* Selected Rou nd 1 RTT- ELC states (2 to 3) and approximately 14 districts in these states State or district KEA data administrator or designee Secure electronic data submission Fall – Wi nter 2015 * Activit ies relevant for this supporting statement. **2. Purposes and uses of**  **the data** To address the study’s research questions, the study will collect and analyze data from several sources. Table A.2 lists the questions and the data sources that will be used to answer them. This supporting statement is requesting clearance only for the study’s district recruiting and district data collection, which will be used to address the study’s third research question. All other data collection methods are based on publicly available information or involve targeted interviews with fewer than 10 individuals. We describe the study’s use of each data source in more detail below. Information will be collected by Mathematica Policy Research, under contract with ED [contract number ED
IES
- 10
- C
- 0077]. **Table A.2. Research Questions and Data Sources** Research Question Data Sources 1. How are TQRIS ratings defined, collected, and generated in the nine Round 1 RTT- ELC states and how might the structure and implementation of the rating system influence the reliability and validity of the ratings? Application reviews, targeted document reviews, and targeted interviews with staff from the 9 Round 1 RTT- ELC statesContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

11 Research Question Data Sources 2 . Is it feasible to examine the relationship between TQRIS ratings and children’s kindergarten readiness outcomes in selected RTT- ELC states ? Application reviews, targeted document reviews, and targeted interviews with staff from the 9 Round 1 RTT- ELC states 3. Wh at is the relationship between TQRIS ratings and kindergarten readiness outcomes for programs that serve preschool -age d children in selected RTT- ELC states? TQRIS data from state TQRIS data systems in 2 to 3 of the Round 1 RTT- ELC states Kindergarten entry assessment data from state and/or district administrative data systems in 2 to 3 of the Round 1 RTT- ELC states **State’ s**  **RTT -** **ELC Applications**  **for Initial**  **Funding .**  From the RTT
-ELC applications, the study collected information about the structure of each state’s TQRIS, plans for expanding program participation in TQRIS, kindergarten entry assessments, and existing data systems. We will use this information to address research questions 1 and 2 and to inform the analysis for research question 3. **Document reviews.**  Document reviews focus ed on describing the structure of TQRIS , the rigor and consistency in the measurement proce ss, the influences on program entry into and movement within each system (such as incentives), and key elements that administrators continue to refine and why. We also reviewed documents describing characteristics, properties, and administration procedures of the state kindergarten entry assessment. We used this information to address research questions 1 and 2 and will use it to select states to be included in analysis for research question 3. **State interviews**  **.** Interviews with state administrators confirmed and clarif ied the information obtained from documents and, as needed, gathered information that could not be obtained from the document review. We will use this information to address research questions 1 and 2 and to refine the design for research question 3. **TQRIS d** **ata.** The main explanatory variable in the analysis for research question 3 will be a center-based ECE program’s TQRIS rating. The data pertaining to this variable will come from state TQRIS databases. We will also collect information on comp onent- and indicator
-level ratings if the data are available. These data will be used to examine the associations between the TQRIS component- and indicator
-level ratings and child outcomes . **Kindergarten entry**  **assessment data**  **.** The outcome data for research question 3 will come from student
-level administrative databases maintained by states and/or districts. These data will allow us to assess the relationship between TQRIS ratings and measures of kindergarten readiness. We will request data on student characteristics , such as sex, race/ethnicity, birth year, grade, eligibility for free or reduced
-price lunch, English language learner status , special education status, and preschool enrollment history, in order to control for factors other than differences in centerbased ECE program quality that might correlate with child outcomes . We will also request that the data include a student
- and/or program-level identifier that will allow us to link program-level TQRIS data to child
-level kindergarten entry assessment data. **3. Use of technology to reduce burden**  The recruitment plan for which clearance is being sought in this submission is designed to minimize burden on the districts being recruited. An electronic recruiting tracking database will be designed and used to store state and district contact history and relevant recruiting data collected inContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

12 the course of those communications. Effective use of this management tool will minimize redundant contacts with states and districts and allow for specific requests for information to be consolidated and coordinated among study team members. A summary of recruiting information will also be entered into Mathematica’s cross
- study recruiting database, allowing further coordination of requests across ongoing education studies, as well as coordination of decisions on the t iming of such requests, to further minimize burden while meeting each study’s requirements. The data collection plan is also designed to obtain information in an efficient way that minimizes respondent burden. We will gather information only from existing data sources, using the most efficient methods available. Existing data sources will be electronic files provided by states (or districts, when not available from the state) that include TQRIS ratings, test scores for school
administered kindergarten entry assessments, and student demographic information. We will work with state and/or district personnel to determine the most efficient and least burdensome way to collect these data and will capitalize on any electronic data systems already in place. Whenever possible, states and/or districts will be able to upload or enter data into any electronic spreadsheet, or to an equivalent file, and transfer it to Mathematica through a secure FTP site. Regardless of the form in which it is received, these data will be converted into a consistent format so that they can be combined with data submitted by other districts from the state and be suitable for analysis. If it is too burdensome or not possible for a state or district to provide data in electronic format, we will provide clear instructions on how to submit the information securely in hard
- copy form, to be coded by the study team. **4. Efforts to avoid duplication of effort**  Evidence is limited as to whether TQRIS ratings distinguish quality and the extent to which the TQRIS ratings are associated with child outcomes. Five studies have examined the relationship between child outcomes and TQRIS ratings (Zellman et al. 2008; Thornburg et al. 2009; Elicker et al. 2011; Tout et al. 2011; Sabol and Pianta 2012, as cited in Lahti et al., 2013), and all examine TQRIS developed prior to the RTT
-ELC program. Only two of these five studies
— Thornburg et al. (2009) and Sabol and Pianta (2012)
—find consistent evidence of a correlation between TQRIS ratings and outcomes. ED determined that an in
-depth, cross - state study examining the structure and characteristics of TQRIS in RTT
-ELC states and the relationship between their ratings and measures of kindergarten readiness is needed to inform the ongoing development of RTT
-ELC states’ and other states’ TQRIS . RTT
-ELC states are required to validate their TQRIS by assessing whether the tiers of TQRIS reflect different levels of program quality and whether differences in quality ratings relate to child outcomes. This study will differ from efforts RTT
-ELC states are undertaking in the following ways:  If feasible, the second component of this study will focus on the relationship between TQRIS ratings and kindergarten readiness measures for programs that serve preschool
aged children. We will select states and collect data with the intention of maximizing statistical power to examine this specific question. State validation studies will not necessarily focus on this particular age group or outcome.  This study will employ consistent methodolog y across states to facilitate comparisons between states. The methods used in states’ local evaluations will vary, depending on the specific questions each state has chosen to focus on.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

13 **5. Methods to minimize burden on small entities**  No small businesses or entities will be involved as respondents. **6. Consequences of not collecting the data**  The recruitment and data collection plan described in this submission is necessary for ED to examine the relationship between TQRIS ratings and kindergarten readiness measures for programs that serve preschool
-aged children. Examining the relationship between TQRIS ratings and children’s kindergarten readiness is important since a key goal of RTT
-ELC is helping to ensure that children enter kindergarten with the skills, knowledge, and dispositions toward learning that they need to succeed. Failing to conduct this study would mean missing a key opportunity to learn about the TQRIS ratings and support the decisions that states are making as they develop and/or refine their TQRIS. Without state and district (if the state records are not available) administrative records, ED will not be able to examine whether TQRIS ratings (and which TQRIS components) are related to children’s kindergarten readiness . The district recruitment and data collection activities that are the focus of this supporting statement will be conducted only if the data needed for the study’s analyses are not available at the state level. **7. Special circumstances**  There are no special circumstances invol ved with recruitment or data collection activities. **8. Federal register announcement and outside consultation**  **a. Federal register announcement**  The 60
-day notice to solicit public comments was published in Volume 80 , page 2685 of the Federal Register on January 20, 2015 . The Department thanks Yasmina Vinci, Executive Director of the National Head Start Association, for her comments . The comments expressed concerns about potentially limited utility of the study due to the following factors: (1) correlating TQRIS ratings with KEA scores would not provide an accurate picture of how children’s outcomes arise as a result of complex interactions between experiences and environments ; (2) results of the study would not be generalizable to states beyond the 2 or 3 included in the analysis , and; (3) K indergarten Entry Assessments are not designed to validate TQRIS and using them for this purpose in this study may elicit high
- stakes interpretations of the relationship between TQRIS and child outcomes . We responded to this comment by highlight ing strategies that the study team is implementing to ensure that results of the study can be useful to the field. These strategies were already in place prior to receipt of the public comment. As such, we did not make any changes to procedures or instruments based on the comments . **b. Consultations outside of the agency**  The study team sought input on the study design from experts in early learning and TQRIS, evaluation design, and analysis during a technical working group (TWG) meeting held on July 3 0, 2014. We will continue to consult with the TWG throughout the study on other issues that would benefit from their input. Table A. 3 lists the TWG members.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

14 **Table A.3 Members of the Technical Working Group for the RTT -**  **ELC Study Name**  **Title and Affiliation** Margaret Burchinal Senior Scientist FPG Child Development Institute University of North Carolina James Elicker Associate Professor Human Development & Family Studies Purdue University Beth Rous Department Chair and Professor Educational Leadership Studies University of Kentucky Elizabeth Stuart Associate Professor Bloomberg School of Public Health Johns Hopkins University Kathryn Tout Co - Director Early Childhood Development Child Trends Martha Zaslow Director SRCD Office for Policy & Communi cations Gail Zellman Senior Research Psychologist RAND Corporation **9. Payments or gifts**  The study does not plan to give gifts to states or districts for participating in the recruitment process or for providing data for the study. **10. Assurances of**  **confidentiality** The study team has established procedures to ensure the confidentiality and security of its data as permitted by law. This approach will be in accordance with all relevant regulations and requirements, particularly the Education Sciences Institute Reform Act of 2002, Title I, Subsection (c) of Section 183, which requires the IES director to “develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data.” The study will also adhere to requirements of Subsection (d) of Section 183 prohibiting disclosure of individually identifiable information, as well as making the publishing or inappropriate communication of individually identifiable information by employees or staff a f elony. The study team will protect the full privacy and confidentiality, as permitted by law, of all individuals and programs that provide data. The study will not store data associated with personally identifiable information (PII), as study staff will as sign random ID numbers to all data records and then strip any PII from the data records. In addition to the data safeguards described here, the study team will not identify respondent names, programs, schools, or districts in publicly available reports or findings, and if necessary, the study team will make an effort to mask distinguishing characteristics in study reports . A statement to this effect will be included with all requests for data:
“Mathematica Policy Research follows the confidentiality and dat a protection requirements of IES (The Education Sciences Reform Act of 2002, Title I, Part E, Section 183). Administrative records obtained through this data collection will be used only for research purposes. The reports preparedfor the study will summar ize state-levelContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

15 findings and will not associate findings with a specific district, program, or individual student. We will not provide information that identifies respondents to anyone outside the study team , except as permitted by law

. ” Mathematica employs the following safeguards to ensure confidentiality:  All Mathematica employees sign a pledge that emphasizes the importance of confidentiality and describes their obligation.  Secure FTP services allow encrypted transfer of large data files with clients, if necessary. All internal networks are protected from unauthorized access using defense
-in
-depth best practices, which incorporate firewalls and intrusion detection and prevention systems. The networks are configured so that each user has a tailored set of rights, granted by the network administrator, to files approved for access and stored on the LAN. Access to hard
- copy documents is strictly limited. Documents are stored in locked files and cabinets. Discarded materials are shredded.  Computer data files are protected with passwords, and access is limited to specific users, who must change their passwords regularly and conform to strong password policies.  Especially sensitive data are maintained on removable storage devices that are kept physically secure when not in use. After the study concludes, the study data will be transmitted to the National Center for Education Statistics (NCES) for safekeeping as a restricted
-use file. All other versions of the data will be destroyed. Prior to transmittal, the data will undergo careful screening, and modification if necessary, as a safeguard against unacceptably high level of disclosure risk for protected respondents. Researchers wishing to access the data for secondary analysis must apply for an NCES license and agree to the applicable rules and procedures guiding the use of restricted
-use files. **11. Additional justification for sensitive questions**  No questions of a sensitive nature will be included in this study. **12. Estimates of hours burden**  If we identify a state as a good candidate for inclusion in the analysis of the relationship between TQRIS ratings and child outcomes but find that we cannot obtain kindergarten entry assessment data from a centralized source at the state level, we will obtain kindergarten entry assessment data from a targeted sample of districts within that state. We anticipate that officials and staff from approximately 42 school districts will participate in initial phone calls and follow
-up communications that will occur during the district recruitment phase of the study in spring and summer 2015 . We estimate that we will need to contact 42 districts to discuss data collection activities to ultimately recruit 14 districts to participate in the study based on our experiences with other projects, including the Impact Evaluation of Race to the Top and School Improvement Grants. We estimate that during the district recruitment process, we will speak with a main contact in all 42 districts and a separate person for completing the MOU in the targeted 14 districts. We expect that initial and follow
-up discussions with the primary contact in each district will take 1.5 hours of their time, on average, for a total of 63 hours for the upfront recruitment contact. Reviewing research applications and/or completing MOUs will take 6 hours, on average, per respondent for aContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

16 total of 84 hours across 14 districts. The total number of district staff involved in recruitment activities will be 56, and the total number of recruitment hours will be 147 , for an average of 2.6 hours per person. Averaged over the 3
-year clearance period, the total annual number of respondents and responses are 24, and the total annual burden hours are 87 (Table A. 4 ). See Appendices A, B, C, and D for the materials and protocol to be used in district recruitment phone calls. **Table A. 4**  **. Burden in Hours to Respondents**  **– RTT**  **- ELC TQRIS**  **Stud y**  **R ecruitment and Data Collection**  Activity Total Number of Respondents Number of

Responses Per Respondent Number of

Administrations Total

Number of Responses Burden

Hours Per Response Total

Burden (Hours) District recruiting communications 42 1 1 42 1.5 63 Review of research applications and/or MOUs 14 1 1 14 6 84 Student records request 14 1 1 14 8 112 **Total District Staff 70**  **70 259**  **Average Annual Total**  **24 24**  **87** If state
-level data are not available, officials from 14 school districts will be asked to provide the necessary student
-level administrative records in fall and winter 201 5 . We estimate that it will take 8 hours, on average, of district
-level staff time to deliver the required student records in each of the participating 14 districts. The total number of burden hours for records delivery, therefore, will be 112 (Table A. 4

) . The data collection form is included in Appendix E and will include the following appropriately tailored Public Burden Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this information collection is xxxx-xxxx . Public reporting burden for this collection of information is estimated to average 8 hours per response, including the time to review instructions, search existing data re sources, gather the data needed, and complete and review the information collection. The obligation to respond to this collection is voluntary. If you have comments or concerns regarding the status of your individual submission of this form, please contact directly: Institute of Education Sciences, U.S. Department of Education, 555 New Jersey Ave. NW, Washington, DC 2020 8. **13. Estimates of cost burden to respondents**  There are no additional respondent costs associated with this data collection beyond the burden estimated in item A12. **14. Estimates of annual costs to the federal government**  The estimated cost for this three
-year study is $2,968,330, or approximately $989,443 per year. This includes development of a detailed study design, data collection instruments , this justificationContract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

17 package, state (and, if needed, district) recruitment, TQRIS and outcomes data collection, and analysis and report preparation. **15. Reasons for program changes or adjustments**  This is a new data collection effort. **16. Publication plans and schedule**  Table A. 5 displays the anticipated timetable for project publications, which include an evaluation brief, and a final report. The evaluation brief’s topic will be the descriptive analyses aligned with the study’s first research question, which examines the components and indicators included in the TQRIS in each of the nine Round 1 RTT
-ELC states and how they are used to generate overall quality ratings . This brief will be released separately and prior to the release of the final report. **Table A. 5**  **. Timetable for Project Publications** Deliverable Date **Evaluation Brief** Draft evaluation brief Revised draft evaluation brief Final evaluation brief June 2015 August 2015 November 2015 **Report** Draft report Revised draft report Final report June 2016 August 2016 February 2017 **17. Approval not to display the OMB expiration date**  All recruitment and data collection materials will display the OMB expiration date. **18. Explanation of exceptions**  No exceptions are requested.Contract Number: ED
- IES
- 10
-C
-0077 M athematica Policy Research

18 **REFERENCES** Elicker, J., C.C. Langill, K. Ruprecht, J. Lewsader, and T. Anderson. “Evaluation of ‘Paths to QUALITY, ’ Indiana’s Child Care Quality Rating and Improvement System: Final Report (Technical Report #3). ” Department of Human Development and Family Studies, Center for Families, Purdue University, 2011. Lahti, M., T. Sabol, R. Starr, C. Langill, and K. Tout. “Validation of Quality Rating and Improvement Systems (QRIS): Examples from Four States.” Research
-to -Policy, Research
-to– Practice Brief OPRE 2013
- 036. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2013. Thornburg, K.R., W.A. Mayfield, J.S. Hawks, and K.L. Fuger. “The Missouri Quality Rating System School Readiness Study.” Columbia, MO: University of Missouri Center for Family Policy and Research, 2009. Retrieved from http://mucentermissouri.edu/MOQRSreport.pdf . Tout, K., R. Starr, T. Isner, J. Cleveland, L. Albertson
- Junkans, M. Soli, and K. Quinn. “Evaluation of Parent Aware: Minnesota’s Quality Rat ing and Improvement System Pilot.” Washington, DC: Child Trends, 2011. U.S. Department of Education. “Race to the Top
– Early Learning Challenge Application for Initial Funding.” 2011. Retrieved May 6, 2013 from http://www2.ed.gov/programs/racetothetop
earlylearningchallenge/2011
-412.doc . Zellman, G.L., R.N. Brandon, K. Boller, and J.L. Kreader. “Effective Evaluation of Quality Rating and Improvement Systems for Early C are and Education and S chool
-Age Care . ” Research
-to Policy, Research
-to–Practice Brief OPRE 2011
- 11a. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services, 2011. Zellman, G.L. , and R. Fiene. “Validation of Quality Rating and Improvement Systems for Early Care and Education and School
-Age Care .” Research
-to -Policy, Research
-to–Practice Brief OPRE 2012
-29. Washington, DC: Office of Planning, Research and Evaluation, Administration for Children and Families, U.S. Department of Health and Human Services , 2012 . Zellman, G.L., M. Perlman, V. Le, and C. Setodji. “Assessing the Validity of the Qualistar Early Learning Quality Rating and Improvement System as a Tool for Improving Child
- Care Quality.” Pittsburgh, PA: Rand Corporation, 2008. Accessed November 26, 2012 at http://www.rand.org/pubcs/monographs/MG650.html.**www.mathematica -** **mpr.com**

Improving public well - being by conducting high quality, objective research and surveys Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ W ashington, DC

Mathematica® is a registered trademark of Mathematica Policy Research