American Institutes for Research 1000 Thomas Jefferson Street NW Washington, DC 20007 - 3835 202.403.5000 | TTY 877.334.3499 **www.air.org** Copy right © 2015 American Institutes for Research. All rights reserved. 1695_03/15 

# Task Order 18: Study on Sustaining the Positive Effects of Preschool Draft 2 : OMB Clearance Request Prepared for U.S. Department of Education Office of Planning, Evaluation and Policy Development Policy and Program Studies Service March 2015

JANUARY 2015

## Contents **Page**

Introduction ................................ ................................ ................................ ................................ ...... 1 Supporting Statement for Paperwork Reduction Act Submission . **Error! Bookmark not defined.** Justification (Part A) ................................ ................................ .. **Error! Bookmark not defined.** A1. Circumstances Making Collection of Information Necessary **Error! Bookmark not defined.**  A2. Use of Information ................................ .......................... **Error! Bookmark not defined.** A3. Use of Improved Technology to Reduce Burden ............ **Error! Bookmark not defined.** A4. Efforts to Avoid Duplicat ion of Effort ............................ **Error! Bookmark not defined.** A5. Efforts to Minimize Burden on Small Businesses and Other Small Entities **Error! Bookmark not defined.** A6. Consequences of Not Collecting the Data ...................... **Error! Bookmark not defined.** A7. Special Circumstances Causing Particular Anomalies in Data Collection**Error! Bookmark not defined.** A8. Federal Register Announcement and Consultation ......... **Error! Bookmark not defined.** A9. Pay ment or Gift to Respondents ................................ ..... **Error! Bookmark not defined.** A 10. Assurance of Confidentiality ................................ ........ **Error! Book mark not defined.**  A11. Sensitive Questions ................................ ....................... **Error! Bookmark not defined.** A12. Es timated Response Burden ................................ .......... **Error! Bookmark not defined.** A13. Estimate of Annualized Cost for Data Collection Activities **Error! Bookmark not defined.** A14. Estimate of Annualized Cost to Federal Government .. **Error! Bookmark not defined.** A15. Reasons for Changes in Estimated Burden ................... **Error! Bookmark not defined.** A16. Plans for Tabulation and Publication ............................ **Error! Bookmark not defined.** Case Study Analytic Approach ................................ ................... **Error! Bookmark not defined.** A17. Display

of Expiration Date for OMB Approval ............ **Error! Bookmark not defined.** A18. Exceptions to Certification for Paperwork Reduction Act Submissions **Error! Bookmark not defined.**  Supporting Statement for Paperwork Reduction Act Submission . **Error! Bookmark not defined.** Description of Statistical Methods (Part B) ............................... **Error! Bookmark not defined.** B1. Sampling Design ................................ ............................. **Error! Bookmark not defined.** B2. Procedures for Data Collection ................................ ....... **Error! Bookmark not defined.** B3. Methods to Maximize Response Rate ............................. **Error! Bookmark not defined.** B4. Expert Review and Piloting Procedures .......................... **Error! Bookmark not defined.** B5. Individuals and Organizations Involved in the Project ... **Error! Bookmark not defined.** References ................................ ................................ ...................... **Error! Bookmark not defined.**Appendix A. Recruitment Materials ................................ ......... A

–**Error! Bookmark not defined.** 

Appendix B. Consent Form ................................ ...................... B

–**Error! Bookmark not defined.** Appendix C. Pre - Interview Survey for Districts and Principals C

–**Error! Bookmark not defined.** Appendix D. District Interview Protocol ................................ .. D

–**Error! Bookmark not defined.** Appendix E. Principal Interview Protocol ................................ E

–**Error! Bookmark not defined.** Appendix F. K

– 3 Teacher/Program Staff Focus Group Protocol F

–**Error! Bookmark not defined.** Appendix G. Pre school Teacher/Program Staff Interview/Focus Group Protocol G

–**Error! Bookmark not defined.** Appendix H. Funder Protocol ................................ ................... H

–**Error! Bookmark not defined.** Appendix I . Evaluator Protocol ................................ ................. I

–**Error! Bookmark not defined.**Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 1



## Introduc **tion** The Policy

and Program Studies Service (PPSS), within the U.S. Department of Education’s Office of Planning, Evaluation and Policy Development, requests Office of Management and Budget (OMB) clearance for the case study component of the Study on Sustaining the Positive Effects of Preschool . This study

aims to accomplish two main goals. The first is to produce a literature review summarizing what is known about policies, programs, and practices that can help students in kindergarten through Grade 3 (K

– 3) bu ild on the positive effects of preschool or make cognitive, social - emotional, and academic gains. The second goal is to provide detailed case study descriptions of five programs that help disadvantaged students in K

– 3 build on the positive effects of presc hool or lead to positive cognitive, social - emotional, and academic outcomes by using policies, programs, and practices related to two key topic areas: (1) preschool and K

– 3 alignment and (2) differentiated instruction.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

—2



## Supporting Statement for Paperwork Re **duction Act Submission**  **Justification (Part A) A1. Circumstances Making Collection of Information Necessary**  Study Overview Preschool can improve academic, behavioral, social - emotional, and cognitive outcomes for students of varying backgrounds, including d isadvantaged students, at least in the short term. Research shows that participation in high - qualit y preschool can improve young children’s readiness skills for elementary school (e.g., Andrews, Jargowsky , & Kuhne, 2012). However, without additional and co ntinuous supports, as these children continue through the elementary grades, participation in preschool cannot overcome the multiple challenges faced b y disadvantaged children. It is important to identify way s to sustain earl y cognitive, social emotional, and academic outcomes in the years immediately following preschool in order to give all students opportunities to thrive academicall y and sustain gains that might have been made with the benefit of preschool. The study involves two key

components: (1) a l iterature review and (2) case studies. The goals of each component are as follows: 1. From the extant literature, summarize what is known about policies, programs, and practices that have the potential to aid practitioners and policy - makers in sustaining the positive effects of preschool. The review will focus on two specific areas: • Preschool and K

– 3 alignment • Diff erentiated instruction 2. Provide detailed case study descriptions of five programs that help disadvantaged students in K

– 3 have positive cognitive, s ocial - emotional, and/or academic outcomes and may build on the positive effects of preschool by using policies, programs, and practices from the two topic areas above. Building on promising practices and potential sites uncovered during the literature revi ew, the case stud y component of this study will examine the design and implementation of policies, programs, or strategies related to the study ’s two topics of interest at five sites that implement programs aimed at cognitive, social - emotional, or academic outcomes for disadvantaged students in Grades K

– 3 . Of particular interest are economically disadvantaged children, children who are learning English as their second language, and children who come from homeless, ne glected, or migrant populations. Data collection activities for the case studies will include interviews with key staff, observations of program activities or classrooms, and a review of key program documents. Anal ys es of the data collected within and acro ss the sites will provide practitioners and policy

makers with much - needed information about the implementation of preschool to Grade 3 alignment and differentiated instruction.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 3

Conceptual Framework For preschool - age children at risk of falling behind in sc hool, attending a high - qualit y earl y learning and care program, including Head Start, has been found to help improve their readiness for school and school success through higher test scores, better attendance, reduced placement in special education, and re duced grade - level retention (Andrews, Jargowsky , & Kuhne, 2012; Barnett, 2008; Karol y & Bigelow, 2005; Rey nolds, 1994; Rey

nolds et al., 2007). Other lasting benefits can include higher rates of high school completion, a greater likelihood of attending coll ege, and increased lifetime earnings (Karol y, Kilburn, & Cannon, 2005; Gormley & Phillips, 2005 ; Rey

nolds & Ou, 2011 ). Accordingl y, the federal government has supported preschool education through programs such as the U.S. Department of Health and Human S ervices’ (HHS’s) Head Start program and the U.S. Department of Education’s (ED’s) preschool services for children with disabilities (Individuals with Disabilities Education Act, Part B) as strateg ies for improving the education and life trajectories of at - risk children. Because of the promise of preschool to improve outcomes for disadvantaged students, states are increasingl y implementing universal prekindergarten programs. Furthermore, in his 2014 State of the Union address, President Barack Obama called f or high - quality

preschool for all children and requested additional federal investments in the 2015 fiscal year budget for Child Care and Development Block Grants, Head Start and Earl y Head Start, and Preschool Development Grants to states. The president a lso has proposed expan ded Earl y Head Start - child care partnerships, expanded home visiting programs, and new partnerships with states to provide high - quality preschool to four - ye ar - olds in families earning less than 200 percent of the federal poverty line. R esearch shows , however, that not all students who experience preschool achieve positive, long term outcomes (Barnett, 2008; L ee & Loeb, 1995). Some preschool program evaluations document that strong initial benefits fade, the “ preschool fade - out effect, ” sometimes as soon as earl y elementary school (Manship, Madsen, Mezzanote, & Fain, 2013; Ramey et al., 2000 ; U.S. Department of Health and Human Services, 2010 ). Preschool effects may fade for many

reasons, including poor elementary school quality , lack o f parental supports, lack of continuous follow - up with participating students, or insufficient intensity or duration of the program (e.g., Brooks Gunn , 2003; Lee & Loeb, 1995). Therefore, the early elementary

context can help or hinder children’s continued academic and social progress. As the country prepares to expand preschool access for all four - ye ar - old children and to align preschool programs with kindergarten - through 12th - grade elementary

and secondary

education systems, the need for quality informati on about how to sustain the benefits of preschool has heightened. The study is guided by

a conceptual framework that considers factors associated with preschool (e.g., dosage, quality) and postpreschool education (e.g., preschool and K

– 3 alignment, differ entiated instruction) and context (e.g., school environment) as contributors to students’ later outcomes. Achieving desired outcomes from preschool programs is not a mere matter of children’s participation. We know that the quality of the preschool program matters (e.g., Peisner - Feinberg et al., 2001), and that the context in which children live influences their outcomes. As Lee and Loeb (1995) have noted, the effects of preschool often fade in earl y elementary school if theAmeri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

—4

quality of schools children att end after preschool is poor. Two postpreschool program approaches, described further below, may be ways to sustain preschool’s positive effects. **Preschool and K**

**– 3 Alignment.**  Preschool and K

– 3 alignment or PK

– 3 1 alignment reflects coordination among standards, curricula, teacher instructional practices, student assessment, and teacher professional development between the preschool years and the early elementary

school ye ars. The effects of preschool may be more s ustainable if curricula and instructional strategies from preschool through Grade 3 are well - aligned (Brooks - Gunn, 2003). When implemented as intended, PK

– 3 alignment policy

or practices should provide a coherent educational experience for a student startin g in preschool. It may be a way to capitalize and sustain the investment of earl y education (Bogard & Takanishi, 2005). **Differentiated Instruction** . The premise of differentiated instruction is that teaching practices and curricula should vary to meet the diverse needs and skills of the individual student and to optimize students’ learning experiences (Tomlinson, 2000; 2001). One explanation for the preschool fade - out effect, described previously , is that children who make early gains in preschool may

not h ave the opportunity to maintain their growth rate or learning trajectory because early

elementary instruction is not differentiated to meet their current skill level. Case Study Research Questions The case studies that are the focus of this Office of Mana gement and Budget (OMB) clearance request will provide practitioners and policy makers with in - depth information about the design and implementation of programs that may sustain and build on the effects of preschool as children progress through the earl y el ementary grades, addressing the very real problem of preschool fade - out effects. Specificall y, the case studies will highlight characteristics (e.g., resources, personnel, staff, training, setting, population served) of PK

– 3 or differentiated instruction p rograms that aim to increase cognitive, social - emotional, or academic student outcomes . They

also will explore implementation challenges and supports within urban and rural settings, which will, in turn, provide information on implementation and sustainabi lity issues that may be unique to each setting. The case studies aim to answer the following research questions: 1. What are the characteristics (e.g., resources, personnel, staff characteristics, training, setting, population served) of PK

– 3 or differentiat ed instruction programs that aim to increase cognitive, social - emotional, or academic outcomes of students? 2. On what research, theory, and/or experiences did the designers of these programs base the program structure and content? 3. What are the challenges o f implementing these programs and how have staff and leaders tried to overcome these challenges? 4. How does the organization implementing the program ensure its sustainability?

1 PK refers to prekindergarten.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 5

**A2. Use of Information** The case studies will be of immediate interest and significance for practitioners and policy makers because they

will offer detailed information about supports and interventions that school sy stems can use to sustain the benefits of preschool. Building on the study ’s literature review, the case studies will provide an in - depth look at how practices related to preschool and K

– 3 alignment and differentiated instruction can be successfully developed and implemented in real - world contexts. The study team will use the interview, observation, and document review data collected during program site visits to develop individual case summaries for each of the five sites in the case study sample. In addition, the study team will use the data to produce a final publicly available report of cross - case findings, which wil l include detailed descriptions of each site and the site’s program or policy . In designing this report, the study team will focus on developing a useful product addressing multiple audiences, including program directors, principals, teachers, federal and state policy

makers planning for earl y childhood investments and supports , and researchers conducting studies on this topic. T he information will be useful for policymakers as they look at adopting and implementing potential new initiatives . D istrict admini strators and principals will be able to take the specific information included in the case studies to implement similar interventions in their own districts and schools. **A3. Use of Improved Technology to Reduce Burden** The recruitment and data collection plans for the case study component of this study

reflect sensitivity

to issues of efficiency

and respondent burden. Beginning with site selection, the study team will use Internet searches related to each topic and will enter relevant information into a database that will track each program/site, its characteristics, and its progression through the site selection process. Once potential sites are identified, the study team will use online materials available to determine the extent to which each program has undergone internal or external evaluation and corresponding evidence for effectiveness. It will conduct screening interviews by email and telephone to reduce respondent burden and facilitate convenience for participant s. During the data collection process, the study team will continue to use technology to reduce burden whenever possible. For example: 1. Interviews and focus groups will be audiotaped and then transcribed at a later date to reduce the amount of time that participants will engage in interview activities. 2. A phone number and e - mail address will be provided to study participants, allowing them to contact research staff directly with any questions they

may have. **A4. Efforts to Avoid Duplication of Effort** The stu dy team will avoid duplication of effort by using preexisting data (e.g., program information available on school or district websites, published program evaluations) whenever possible to guide site selection and data collection. For example, the study team will determine whether an y of the proposed data collection elements for the case studies can be addressed through preexisting policy or evaluation documents. This step will reduce the number of questions asked in the case study interviews and focus group s, thus limiting respondent burdenAmeri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 6

and minimizing duplication of previous data collection efforts and information. In addition, the study team will avoid selecting sites for which case studies have already been conducted, except in cases where the extant c ase stud y does not go into the detail needed to address the study’s research questions or was completed so long ago that it would be useful to learn updated information about the site’s work. **A5. Efforts to Minimize Burden on Small Businesses and Other Sma ll Entities**  Some school districts likely to be involved in this study have fewer than 50,000 students, and are thus considered small entities . Because we have minimized the burden on these (and all) districts and are offering a small stipend for participat ing, we do not believe study activities will have a significant economic impact on these small entities. **A6. Consequences of Not Collecting the Data** The data to be collected through the case studies are needed to inform state and local efforts to develop and implement programs that can successfully sustain the benefits of preschool as children advance through the elementary grades. Failure to collect the data proposed through these case studies will limit the information available to the Department to guide federal policy development and technical assistance related to programs to sustain the effects of preschool. In addition, it would prevent the distribution of in - depth information to policymakers and practitioners across the nation about the use of such p rograms. The absence of this case study report could therefore hinder state, district, and school stakeholders’ abilit y to make careful and informed decisions about promising policies, programs, and practices to sustain the benefits of preschool. **A7. Speci al Circumstances Causing Particular Anomalies in Data Collection**  None of the special circumstances listed applies to this data collection.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 7

**A8. Federal Register Announcement and Consultation** 1. **Federal Register Announcement .**  A 60 - day notice to solicit public c omments was published in the Federal Register on DATE (Volume XX, Number XX). 2. **Consultations Outside the Agency .**  A technical working group (TWG) w as consulted as part of this study

and will continue to provide conceptual and methodological considerations fo r the collection, anal ys is, and reporting of the case study data. The TWG members provided comments on the study design and they nominated sites for inclusion in the case study sample in 2014. The TWG will review the protocols and proposed case study sites in late spring 2015 . The study ’s TWG members, listed in Exhibit 4, bring together expertise in research on the effects of preschool among disadvantaged children and the sustainability of those effects, the needs of special student populations, approaches for aligning prekindergarten through third grade, and methods for conducting rigorous research reviews and case studies. **Exhibit 4. Technical Working Group Members Name**  **Affiliation Lindy Buch**  Retired director of Early Childhood Education and Family Services in the Office of Great Start at the Michigan Department of Education **Margaret Burchinal** Senior scientist at the Frank Porter Graham Child Development Institute at the University of North Carolina, Chapel Hill, and an adjunct professor in the Department of Education at the University of California, Irvine **Linda Espinosa** Retired professor, Early Childhood Education, University of Missouri; served as codirector of the National Institute for Early Edu cation Research **Kristie Kauerz** Research assistant professor of P

–3 Policy and Leadership at the University of Washington **Ellen Kisker** Principal investigator, What Works Clearinghouse Early Childhood Education for Children With Disabilities Review Team **A9. Payment or Gift to Respondents** Each participating school will receive an honorarium ($100 for purchasing classroom materials or school supplies) as recognition of their time and effort to participate in the study . In most case study sites, two schools will be included in the study . **A10. Assurance of Confidentiality** As researchers, the study team is vitally concerned with maintaining the anony mity and security of its records. The contractors’ project staff has extensive experience collecting information and maintaining the confidentiality , securit y, and integrity of interview, focus group, and observation data. All members of the study team have obtained their certification on the use of human subjects in research as well as federal security clearances. T his training addresses the importance of the confidentialit y assurances given to respondents and the sensitive nature handling data. The team also has worked with the Institutional Review Board (IRB) at American Institutes for Research (AIR) to seek and re ceive approval of this study , thereby ensuring that the dataAmeri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 8

collection complies with professional standards and government regulations designed to safeguard research participants. The following data protection procedures will be in place: The study team will protect the identity of individuals from whom we collect data for the study to the extent possible (given the small number and size of some districts included in the study) and will use it for research purposes only . Respondents’ names will be used fo r data collection purposes only

and will be disassociated from the data prior to analy sis. As information is gathered from respondents or sites, each respondent will be assigned a unique identification number, which will be used in anal ys is files as well a s printout listings on which data are displayed. Respondents’ unique identification number also will be used for data linkage across sources (e.g., interview and observation data). Any identifiable information will be kept in secured locations and identifi ers will be destro yed as soon as they are no longer required. In addition, the study

team will shred all interview protocols, observation rubrics, forms, and other hard - cop y documents containing identifiable data as soon as the need for the hard copies no longer exists. Prior to beginning interviews, focus groups, or observations, a member of the research team will explain to participants what will be discussed, how the data will be used and stored, and how anony mity will be maintained. Participants will b e instructed that they can stop participating at any time. The study ’s goals, data collection activities, participation risks and benefits, and uses for the data will be detailed in a consent form that all participants will read and sign prior to beginning an y data collection activities. Participants will be informed that sites will be named in case stud y reports but that individuals will not be named specificall y. Signed consent forms will be collected from site visitors and stored in secure file cabinets at the contractors’ offices. All electronic data will be protected using several methods. The contractors’ internal networks are protected from unauthorized access b y defense - in - depth best practices, which incorporate firewalls and intrusion detection and prevention systems. Access to computer s ys tems is password protected, and network passwords must be changed on regular basis and conform to the contractors’ strong password policies. The networks also are configured so that each user has a tailored set of rights, granted b y the network administrator, to files approved for access and stored on the local area network (LAN). Access to all electronic data files and workbooks associated with this study will be limited to researchers on the case study data colle ction and anal ys is team. Any

files that are saved outside these secure folders (e.g., to transmit data files between stud y team members at AIR and case study sites ) will be encry

pted and requi re a strong password to access. **A11. Sensitive Questions** This study

will not include the collection of sensitive information. The only data to be collected directly from case study

participants will focus on district and school policies and practices rather than on individual people. District and school policies and practices are within the public domain (e.g., schools communicate their policies and programs to students and parents in a variet y of way s). In this sense, the data are not sensitive in nature.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 9

**A12. Estimated Response Burden** It is estimated that the total hour burden for the case study data collection is 50 hours. This translates to an estimated cost of $1,576.10 based on the average hourl y wage of participants. Exhibit 5 summarizes the estimates of respondent burden for the various study activities across all five case study

sites. The estimated burden associated with the data collection at each individual case study site is 10 hours. This estimate assumes the following data collection activities at each of the five sites:  One - and - one - half

- hour interviews w ith one or two district program leaders  One - hour interviews with principals at one or two schools within each district  One - hour interviews or focus groups with five program teachers , preschool teachers, and/ or other program staff at each of two schools within each district  O ne - hour interview with the program funder (e.g., from a foundation or private funding agency) , if applicable  One - hour interview with the program evaluator, if applicable The data collection for each site also will include two observations of program activities, but these observations have been excluded from estimates of response burden because the observed program activities will be part of respondents’ customary and usual business practices. **Exhibit 5. Estimated Total Hou r and Monetary Cost Burden of Case Study Data Collection** 

**Task Total** 

**Sample Size**  **Estimated Response**  **Rate Number of** 

**Respondents Time** 

**Estimate (in**  **H ours)**  **Total**

**Hour Burden**  **Hourly**

**Rate 2**  **Estimated**

**Monetary Cost of**  **Burden** District Superintendent/ Program Staff Interview 10 100% 10 1 .5 1 5 $ 43.88 $ 65 8. 2 0 District Superintendent/ Program Staff Pre Survey 10 100% 10 0,5 5 $43.88 $219.40 Principal Interview 10 100% 10 1 10 $42.19 $421.90 Principal Pre - Survey 10 100% 10 0,5 5 $42.19 $210.95 Preschool Teacher Interview/Focus Group 10 100% 10 1 10 $28.21 $2 82.10 Elementary Teacher or other Program Staff Interview/Focus Group 2 5 100% 2 5 1 2 5 $ 28.21 $ 705 . 2 5 Program Funder 4 100% 4 1 4 $29.99 $119.96

2 Wage informat ion was collected from the Bureau of Labor Statistics (BLS). Funder/evaluator wages were based on the BLS “Community Services Manager” salary . Salaries were converted to hourly wages using **http://www.convertunits.com/salary/ .** Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 10

Evaluator Interview 4 100% 4 1 4 $ 29.99 $119.96 Total for Case Study Data Collection 8 3 **—** 8 3 **—** 78 **—** $2,737.72 Annualized Burden 28 **—** 2 7 hours, 40 minutes **—** 26 **—** $ 912.57 **A13. Estimate of Annualized Cost for Data Collection Activities** There are no additional annualized costs for data collection activities associated with this data collection beyond the total hour burden estimated in item A12. **A14. Estimate of Annualized Cost to Federal Government** The estimated cost to the federal government for the Task Order 18 case studies, including development of the case study research plan and data collection instruments as well as data collection, data analy sis, and report preparation, is $321,888 for the tw o years of the stud y, or approximately

$160,944 per y ear. **A15. Reasons for Changes in Estimated Burden** This is a new data collection . **A16. Plans for Tabulation and Publication** Case Study Analytic Approach The study team will establish and adhere to a set of qualitative anal yt ic procedures and standards to limit bias and ensure reliable findings. For the case studies to be of value, they must be grounded in rigorous methods of qualitative research. Experts in qualitative research methods make clear that wel l - planned, sy stematic, and transparent qualitative data collection and anal ys is techniques yi eld reliable, transferable findings (Anfara, Brown, & Mangione, 2002; Creswell, 1998). Qualitative site visit data will be anal yz ed through a carefull y structured five - step anal yt ic process guided b y the study ’s research questions and conceptual framework. The process is designed to build reliability and validit y into the case study process by both creating a chain of evidence and using triangulation to identify the mes (Yin, 2003). The anal ysis process will incorporate standards of evidence, triangulation of data, and procedures for measuring and ensuring interrater agreement. **Exhibit 6**  **. Overview of Qualitative Analysis Process**

To anal yz e site visit data, the research team will rely on a set of codes based on constructs underlying the interview and focus group protocols, classroom observation protocols, and Develop codebook Interview,

document, and observation note coding Within - site

analysis (question set) Case

summaries Cross - case

analysisAmeri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 11

documents collected from sites. Coding interview and observation data is central to qualitative anal ys is. Through a s ys tematic coding process, analy sts will identify information associated with specific constructs that will anchor within - case and cross - case analy ses.  First, a preliminary

code list will be drafted and codes wil l be piloted with a subset of data (for example, selected interview transcript portions from several sites) to determine whether the set of codes covers the topics reflected in the interview, whether they are appropriatel y specific, and whether the definit ions in the codebook are clear.  Once the codebook is finalized, AIR will conduct training to ensure that analy sts agree on the application of each code.  Analy sts will use a qualitative software program (e.g., NVivo or Dedoose), to facilitate the coding p rocess. Such a program allows anal ys ts to assign relevant codes to data and then compare coded data across sources.  Three anal ys ts will code case study data; this team will meet weekl y to discuss any questions about how to apply the codes and discuss any d isputable data source. Any disagreements in codes will be resolved b y discussion and consensus among coders.  To measure interrater reliability , more than one analy st will code 20 percent of data to ensure at least 80 percent agreement. If 80 percent agree ment is not reached after coding the first interview or document together, coding staff will be retrained and another interview will be coded together. This process will be repeated until 80 percent agreement is reached consistently on three interviews cod ed together. For any text on which 80 percent agreement is not reached on codes, the senior analy st’s codes will serve as the master data. Codes for anal yz ing case study data will be structured so that anal ys ts can apply more than one code to the same int erview passage or observation note, as applicable. Case Summaries After all raw data have been coded, anal ys ts will identify the patterns, themes, and categories that are most relevant to the research questions of the study . This process of data reduction involves noting the prevalence of each response, group differences, and associations among data sources. Specificall y, analy sts will use the software program to query coded data in order to summarize findings for each case, producing both individual site and cross - site case summaries. Creating case summaries will be a sy stematic process that relies on coded data and not any one researcher’s or respondent’s perspective on the program or policy . To this end, anal ys ts will review the frequency

of coded respon ses to characterize which themes are common and which are outliers, identifying the most common themes. Analy sts also will examine differences in responses among respondent ty pes , such as by

role (e.g., principal, elementary school teacher, preschool teach er, funder). AIR will establish clear standards of evidence in order to draw conclusions. For most topics, anal ys ts will seek convergence of perspectives to draw conclusions (i.e., at least two people agree, with no contradictory evidence). However, given the limited number of respondents and cases in this study , there might be topics for which there will be onl y one knowledgeable respondent (e.g., a district leader may be the onl y person knowledgeable about the developmentAmeri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 12

of the program). AIR will adjust standards of evidence to account for this. Before the question set is completed, project leadership and PPSS will establish final decision rules regarding the specific number of respondents required to count as evidence. Cross - Case Analysis In the final st age of anal ysis, the study team will use case summaries for each site to compare themes across sites . Through this review process, themes, policies, or practices common to more than one site can be identified and included in a final case study report . AIR will conduct cross case anal ys is to the degree possible because it allows for more general conclusions to be drawn about how effective policies, programs, or practices can be sustained. If the programs or policies examined through case studies are very dif ferent, cross - case analy sis may be less appropriate, and AIR will include more emphasis on examining themes within each case Reporting In summary, d ata collected for each case/site will be analy zed and included in five internal case summary reports as well as a publicly released case study

report. The case study

report will begin with an introductory

section that (1) features an audience - appropriate overview of the study and (2) outlines common themes that emerged from the data analysis. The report will feature cross - case findings as well as detailed descriptions of each site and the site’s program or policy . In designing this report, AIR will focus on developing a useful product addressing multiple audiences, including program directors, principals, teachers, federal and state policy makers planning for early

childhood investments, and researchers conducting studies on this topic. The proposed timeline for data collection and reporting activities is described in detail below and shown in Exhibit 7 . **Exhibit 7**  **. Timeline for Data Collection Activities and Reporting Activity**  **Time Frame** Work with PPSS/OEL/HHS to identify potential case study sites December 2014

– March 2015 Draft interview, focus group, observation, and document review protocols December 2014

–January 2015 Revise site visit protocols March 2015 TWG meeting to review potential case study sites and protocols April 2015 Submit OMB package and revisions March

–August 2015 Conduct site visitor training September 2015 Conduct site visits September - October 2015 Code site visit data November 2015

–January 2016 Submit draft site - level reports (case summaries) March 2016 Conduct cross - case analysis March

–April 2016 Submit final site - level reports May 2016 Submit draft case study report April 2016Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

— 13

**Activity Time Frame**  Submit final case study report October 2016 Note: PPSS=Policy and Program Studies Service; OEL=Office of Early Learning; HHS=U.S. Department of Health and Human Services; TWG=technical working group; OMB=Office of Management and Budget **A17. Di splay of Expiration Date for OMB Approval**  All data collection instruments will display the OMB approval expiration date. **A18. Exceptions to Certification for Paperwork Reduction Act Submissions** No exceptions to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83 - I are requested.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

—21



## References Andrews, R. J., Jargowsky, P., & Kuhne, K. (2012). The effects of Texas’s pre - kindergarten program on academic performance (CALDER Working Paper No. 84). Washington, DC: National Center for Analy sis of Longitudinal Data on Educational Research. Anfara, V. A., Brown, K. M., & Mangione, T. L. (2002). Qualitative anal ys is on stage: Making the research process more public. Educational Researcher, 31 (7), 28

– 38. Barnett, W. S. (2008). Preschool education and its lasting effects: Research and policy implications . Boulder, CO, & Tempe, AZ: Education and the Public Interest Center & Education Policy

Research Unit. Retrieved from http://nepc.colorado.edu/files/PB Barnett - EARLY - ED_FINAL.pdf Bo gard, K., & Takanishi, R. (2005). PK

– 3: An aligned and coordinated approach to education for children 3 to 8 ye ars old. Social Policy Report, 19 (3), 3

–22. Brooks - Gunn, J. (2003). Do you believe in magic? What we can expect from earl y childhood intervention programs. Social Policy Report, 17(1), 1; 3

– 14. Creswell, J. W. (1998). Qualitative inquiry and research design: Choosing among five designs. Thousand Oaks, CA: Sage. Gormley , W. T., & Phillips, D. (2005). The effects of universal pre -K in Oklahoma: Rese arch highlights and policy implications . Washington, DC: Georgetown University. Retrieved from http://explore.georgetown.edu/publications/index.cfm?Action=Vie w&DocumentID=1438 7 Karol y, L. A., & Bigelow, J. H. (2005). The economics of investing in universal preschool education in California . Santa Monica, CA: RAND Corporation. Karol y, L. A., Kilburn, M. R., & Cannon, J. S. (2005). Early childhood interventions: Proven results, future promise . Santa Monica, CA: RAND Corporation. Retrieved from http://www.rand.org/pubs/monographs/MG341 Lee, V. E., & Loeb, S. (1995). Where do Head Start attendees end up? One reason wh y preschool effects fade out. Educational Evaluation and Policy Analysis, 17(1), 62

– 82. Manship, K., Madsen, S., Mezzanotte, J., & Fain, G. (2013). Evaluation of the S tretch to K indergarten program: 2012 findings. San Mateo, CA: American Institute s for Research. Peisne r - Feinberg, E. S., Burchinal, M. R., Clifford, R. M., Culkin, M. L ., Howes, C., Kagan, S. L., & Yazejian, N. (2001). The relation of preschool chil d - care quality to children's cognitive and social developmental trajectories through second grade. Child D evelopment, 72 (5), 1534

– 1553.Ameri can Institutes for Research Task Order 18: Study on Su staining the Positive Effects of Preschool

—22

Ramey , S. L., Ramey , C. T., Phillips, M. M., L anzi, R. G., Brezausek, C., Katholi, C. R., & Snyder, S. (2000). Head Start children’s entry into public school: A report on the National Head Start/Public School Early Childhood Transition Demonstration Study. Birmingham, AL: Civitan International Research Center, Universit y of Alabama at Birmingham. Rey nolds, A. J. (1994). Effec ts of a preschool plus follow - on intervention for children at risk . Developmental Psychology, 30 (6), 787

– 804. Rey nolds, A. J., & Ou, S. (2011). Paths of effects from preschool to adult well - being: A confirmatory

anal ys is of the Child - Parent Center Program. Child Development, 82 (2), 555

– 582. Rey nolds, A. J., Temple, J. A., Ou, S., Robertson, D. L ., Mersky , J. P., Topitzes, J. W., & Niles, M. D. (2007). Effects of a school - based early childhood intervention on adult health and well - being: A 19 - ye ar follow - up of low - income families. Archives of Pediatric and Adolescent Medicine, 161 (8), 730

– 739. Tomlinson, C. A. (2000). Differentiation of instruction in the elementary grades. ERIC Digest . Sy racuse, NY: ERIC Clearinghouse on Elementary and Earl y Childhood Educat ion. Tomlinson, C. A. (2001). How to differentiate instruction in mixed - ability classrooms (2nd ed.) Alexandria, VA: ASCD. U.S. Department of Health and Human Services. (2010). Head S tart I mpact S tudy: Final report . Washington, DC: U.S. Department of Health and Human Services, Administration for Children and Families, Office of Planning, Research, and Evaluation. Yin, R. K. (2003). Case study research, design and methods (3rd ed.). Newbury Park, CA: Sage.