American Institutes for Research 1000 Thomas Jefferson Street NW Washington, DC 20007 - 3835 202.403.5000 | TTY 877.334.3499 **www.air.org** Copy right © 2015 American Institutes for Research. All rights reserved. 1695_03/15 

# Task Order 18: Study on Sustaining the Positive Effects of Preschool Draft 2 : OMB Clearance Request Prepared for U.S. Department of Education Office of Planning, Evaluation and Policy Development Policy and Program Studies Service March 2015

JANUARY 2015

## Contents **Page**

Supporting Statement for Paperwork Reduction Act Submission . **Error! Bookmark not defined.** Justification (Part A) ................................ ................................ .. **Error! Bookmark not defined.** A1. Circumstances Making Collection of Information Necessary **Error! Bookmark not defined.** A2. Use of Information **Error! Bookmark not defined.**  A3. Use of Improved Technology to Reduce Burden **Error! Bookmark not defined.** A4. Efforts to Avoid Duplication of Effort **Error! Bookmark not defined.** A5. Efforts to Mi nimize Burden on Small Businesses and Other Small Entities **Error! Bookmark not defined.** A6. Consequences of Not Collecting the Data **Error! Bookmark not defined.** A7. Special Circumstances Causing Particular Anomalies in Data Collection**Error! Bookmark not defined.** A8. Federal Register Announcement and Consultation **Error! Bookmark not defined.** A9. Pay ment or Gift to Respondents **Error! Bookmark not defined.** A 10. Assurance of Confidentiality **Error! Bookmark not defined.** A11. Sensitive Questions **Error! Book mark not defined.**  A12. Es timated Response Burden **Error! Bookmark not defined.** A13. Estimate of Annualized Cost for Data Collection Activities **Error! Bookmark not defined.** A14. Estimate of Annualized Cost to Federal Government **Error! Bookmark not defined.** A15. Reasons for Changes in Estimated Burden **Error! Bookmark not defined.** A16. Plans for Tabulation and Publication **Error! Bookmark not defined.** Case Study Analytic Approach ................................ ................... **Error! Bookmark not defined.** A17. Display

of Expiration Date for OMB Approval **Error! Bookmark not defined.** A18. Exceptions to Certification for Paperwork Reduction Act Submissions **Error! Bookmark not defined.** Supporting Statement for Paperwork Reduction Act Submission ................................ ................. 14 Description of Statistical Methods (Part B) ................................ ................................ ............... 14 B1. Sampling Design 14 B2. Procedures for Data Collection 15 B3. Methods to Maximize Response Rate 19 B4. Expert Review and Piloting Procedures 19 B5. Individuals and Organizations Involved in the Project 19 References ................................ ................................ ...................... **E rror! Bookmark not defined.**  Appendix A. Recruitment Materials ................................ ......... A

–**Error! Bookmark not defined.**Appendix B. Consent Form ................................ ...................... B

–**Error! Bookmark not defined.**

Appendix C. Pre - Interview Survey for Districts and Principals C

–**Error! Bookmark not defined.** Appendix D. District Interview Protocol ................................ .. D

–**Error! Bookmark not defined.** Appendix E. Principal Interview Protocol ................................ E

–**Error! Bookmark not defined.** Appendix F. K

– 3 Teacher/Program Staff Focus Group Protocol F

–**Error! Bookmark not defined.** Appendix G. Preschool Teacher/Program Staff Interview/Focus Group Protocol G

–**Error! Bookmark not defined.** Appendix H. Funder Protocol ................................ ................... H

–**Error! Bookmark not defined.** Appendix I . Evaluator Protocol ................................ ................. I

–**Error ! Bookmark not defined.** Ameri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 14



## Supporting Statement for Paperwork Reduction Act **Submission Description of Statistical Methods (Part B)**  **B1. Sampling Design**  The overall purpose of the case studies is t o provide detailed information regarding how sites have succeeded in implementing their successful programs. The study team will therefore select a purposive sample of five sites that show evidence of successful practice fo r the selected topic(s). Sites wi ll be districts or sy stems so that it is possible to learn from practices that are implemented more widely than in just one school or program. A site must demonstrate success among its students b y showing positive outcomes in cognitive, social - emotional, o r academic domains. **Site Selection Procedures.** To select sites for case studies, the study team considered sites or programs that emerge from the following sources:  Descriptions and findings from the research literature  Recommendations from earl y childh ood experts , including • American Institutes for Research ( AIR ) staff working on projects related to preschool and earl y elementary

school programs • Technical working group (TWG) members • Program and Policy

Studies Service ( PPSS ) , Office of Earl y Learning ( OEL ) , and Administration for Children and Families ( ACF ) partners • State o ffices of e arl y l earning • National - level associations , including the National Association of Elementary School Principals, the National Association for the Education of Young Children, an d the National Institute for Out - of - School Time To cull a list of potential sites from these sources, AI R first acquired any publicly available documents about the program (e.g., information from Web searches, public reports, and brochures) and summarized information about the programs (e.g., key activities, target grade levels, reported outcomes). Any site that did not meet initial criteria

— including topic focus, age group targeted, or length of time program has been in place

— was eliminated from considerat ion. Meanwhile, AIR and the U.S. Department of Education ( ED ) prioritized a list of potential sites by targeting programs that ( a) focused on a larger range of grades, not just p re kindergarten and kindergarten ; (b) were districts implementing policies or p rograms across multiple schools or were schools that served as example s of a larger state policy ; and ( c) had some evidence that the policies or practices were intentionally focused on sustaining the effects of preschool. The study team looked for performance data and any program implementation information about the prioritized sites from extant sources . When unavailable, the team requested them from theAmeri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 15

principal or district administrator . 1 Using this information, AI R conduct ed additional screening to eliminate sites that have not been able to continue the program or policy , shown grade - to grade decreases in achievement (based on available district or school assessment [e.g., progress monitoring or screening data]), or have shown obvious marked inconsistency in program implementation. From this list of finalists, AI R will select five final case study sites that will best answer the stud y’s research questions, working closely with the TWG , PPSS, OEL , and ACF. A back - up list of sit es will be developed in case an y first - choice sites do not wish to participate in the study . Across the sites, the study team will aim to include variation in approach, geographic location, urbanicit y, and other factors that w ill be informed by the topic(s ) of selected focus. Although this sample will be purposive and in no way representative, diversifying the cases as much as possible can provide insights into implementation issues that arise within and across particular contexts. In order to recruit all f

ive sites, AIR has developed recruitment materials , including an overview of the study

that describes activities requested of participants, an introductory letter to district superintendents from ED, and an introductory letter to principals from AI R. These materials will be sent to site s when they are invited to participate. AIR staff will then follow up by phone. Copies of the study ’s recruitment materials are included in Appendix A. The study team will complete district research requirements, as needed, b efore collect ing data . **B2. Procedures for Data Collection**  Discussions of the data collection procedures of each of the main components of the case stud y ( document reviews and interviews) follow . Data Collection Activities Exhibit 8 presents an overview of the data collections planned for this study, with details following . **E xhibit**  **8 . Overview of Data Collection Activities**  **Document Review**  **Interviews Program Activity**  **Observations** Program characteristics   Context   Participants   Program’s philosophy and model   Staffing and expenditures   Program activities and engagement  

1 In keeping with Office of Management and Budget requirements, we limited requests for additional information to fewer than 10 respondents.Ameri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 16

Classroom characteristics  Implementation challenges and solutions  Staff training  Program sustainability  Student outcomes  Document Review The study team will collect and review documents in stages as follows:  During the process of identifying and recruiting potential sites, as described earlier, we will have acquired some publicly available documents describing the program and will have requested from sites some nonpublic documents, including more recent reports or activity descriptions.  During site visits , site visitors will request other relevant documents available on - site, including training materials, student work samples, organizational charts, and any other information about the program’s goals and results. Review of all of these documents will be directed by a structured protocol in which anal ys ts will respond to guided questions about information related to the program’s goals and activities, focusing on strategies and successful implementation for the site in sustaining positive outcomes. Semistructured Interviews At each site, site visitors will interview up to five program staff, five pro gram teachers, and the program funder (e.g., from a foundation or private funding agency) and program evaluator, if applicable. AIR has designed interview protocols to be semistructured, guiding respondents to comment on questions of interest, but also mai ntaining a conversational tone to capture breadth of information. All interviews will be audiorecorded and later transcribed to facilitate coding. Interview protocols (included in Appendi ces C – I ) include questions about key constructs of interest, includin g characteristics of the program; the resources, personnel, staff characteristics, and/or training that facilitate implementation of the program; and program sustainability . AIR has developed a separate set of interview protocols for each respondent type. There is a pre interview survey

for district staff and principals (Appendix C), as well as interview protocols to be used with ( Appendix D ) district staff, ( Appendix E ) principals, ( Appendix F ) elementary teachers or other program staff, ( Appendix G ) presc hool teachers, ( Appendix H ) funders, and ( Appendix I ) program evaluators. Teacher interviews may be conducted as individual interviews or focus groups , as determined by the schedules and roles of the selected interviewees. Timing will be determined in cons ultation with each school’s principal based on the teachers’ schedules. Each protocol will include questions that apply across all both topics of focus

— differentiated instruction and PK

– 3 alignment

— in addition to questions that focus specifically on each o f the topics to be asked as appropriate depending on the program.Ameri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 17

Program Activity Observations Case study

site visits may

include observations of classroom instruction, teacher meetings, and other activities relevant to the program being implemented. Th ese observations will be guided by structured observation rubrics, but these are not included in the OMB package because they do not create additional burden for participants. Observation forms will be customized to each of the five programs and depend on the age - range and setting of the program. Institutional Review Board (IRB) Approval All data collection protocols will be submitted to AI R’s I RB for approval. AI R’s I RB, whic h is registered with the Office of Human Research Protection and operates under a federalwide assurance comparable to many universities, is responsible for reviewing all research conducted by the organization and its subcontractors. The IRB ensures that pr ojects involving human subjects comply

with professional standards and government regulations designed to safeguard participants and that research team members, including subcontractor staff, are adequatel y trained. The criteria that a study must meet for AIR’s I RB approval include assurances that risks to participants are minimized and balanced b y benefits, subject selection is equitable, participants are informed about risks and give un coerced consent, privacy of the subjects is adequatel y protected, and the rights and welfare of populations that might be vulnerable to coercion or undue influence are protected. Conducting Site Visits Data collection will take place over approximately a one - month period in the fall of the 2015

– 16 school y ear. Before data co llection, AIR will train site visitors and carefully plan with each site the schedule and logistics of the visit. Training Before site visits are conducted, all members of the case study team will participate in a one - day in - person or webinar - based trainin g to ensure that they thoroughl y understand the content of the protocols and site visit procedures. Staff members will review best practices for interviews , in addition to conducting practice interviews. Staff will discuss strategies for (1) avoiding leadi ng questions, (2) ensuring consistency , and (3) conducting interviews in a way that is conversational, yet still directed toward collecting the intended information systematically. Important procedural issues to be addressed include guidelines for ensuring respondent privacy . For observations, we will thoroughl y introduce and practice the protocol to be used, making sure that observers understand the goal of each item and section. Observers will use video clips to practice taking high - quality observation no tes and making summaries or ratings. Lo gistics Each site visit will likely

take two day s. Two researchers will visit each site. This approach will allow two interviews to be conducted simultaneously , as needed, to minimize scheduling burden on respondents. The two site visitors will both be present for an y focus groups, with one research er serving as the lead interviewer and the other researcher serving as the secondary interviewer, asking probes or encouraging additional participants to respond . All interviews andAmeri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 18

focus groups will be recorded for later transcription. The two researchers will co - observe in order to compare and debrief about activity observations. Site visit schedules will be developed in concert with the appropriate staff at the selected sites. The same pair of researchers will be responsible for scheduling and conducting visits, thus developing a rapport with staff at each site. A sample site visit schedule, reflecting a visit to one school, is presented in Exhibit 9 . **Exhibit 9**  **. Sample Site Visit Schedule**  **Day/ Time**  **Site Visitor 1 Site Visitor 2**  Before site visit, by phone Conduct interview with superintendent Conduct interview with district curriculum director Day 1, 8 :30

– 9 :30 Conduct interview with principal at School 1 Support principal interview; ask follow - up questions as needed , take brief notes Day 1, 9:30 - 10:30 Conduct i nterview with program evaluator Support program evaluator interview; ask follow - up questions as needed, take brief

notes Day 1, 10:30 - 12:00 Observe Classroom Activities Day 1, 12: 00

– 1:00 Lunch break Day 1, 1:00

–2:00 Conduct focus group with preschool

teachers at/affiliated with School 1 Conduct interview with preschool center director Day 1, 2:50

– 3:50 Conduct focus group with elementary teacher s at School 1 Support teacher focus group; ask follow - up questions as needed , take brief notes Day 2, 9:00

– 10:00 Conduct interview with principal at School 2 Support principal interview; ask follow - up questions as needed, take brief notes Day 2, 10:00

– 11:00 Conduct interview with elementary teacher 1 Conduct interview with elementary teacher 2 Day 2, 11:00

– 12:00 Conduct interview with elementary teacher 3 Conduct interview with other key program staff (e.g., coach) Day 2, 12:00

– 1:00 Lunch break Day 2, 1:00

–2:00 Conduct focus group with preschool

teachers at or affiliated with School 2 Conduct interview with preschool center director Day 2, 2:15 - 3:15 Observe Teacher Meeting After site visit, b y phone Conduct i nterview with funder Support funder interview; ask follow - up questions as neededAmeri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

— 19

**B3. Methods to Maximize Response Rate**  Data collection is a complicated process that requires careful planning. The research team has developed interview and observation protocols that are appropriatel y tailored to the respondent group and are designed to place as little burden on respondents as possible. The team will also pilot core data collection instruments with similar respondents in schools local to AIR’s offices to ensure that they

are user - friendl y and easily understandable, all of which increases participants’ willingness to participate in the data collection activities and thus increases response rates. In addition to careful instrument design, a high response rate among the case study s ites may

be ensured through careful recruitment materials. T hese recruitment materials emphasize the social incentive to respondents by stressing the importance of the data collections as part of a study of innovative practices that will provide much - neede d information to districts and schools. AIR’s experience in past evaluations has demonstrated the importance and value of building a consensus of support with participating districts. This level of consensus leads to districts and schools that have the capacity , willingness, and commitment to cooperate fully with the research and data collection responsibilities. Investing in site development at the front end reduces problems at the back end, helping to ensure smooth implementation of the study . **B4. Expert Review and Piloting Procedures**  Following publication of this Office of Management and Budget submission for 60 - day public comment, the study

team will pilot test all case study protocols by

asking staff in si milar roles in non - case - study

programs to participate in a mock interview or classroom observation. We plan to conduct two pilot tests for each protocol, requesting participation from sites or personnel with whom we have worked previously . We will ask for feedback on questions that are confusing and incorporate changes in a final version. We also will note how long each interview takes. For classroom observation protocols, we will review ratings and notes across multiple observers for one given activit y to ensure the form generates reliable data (i.e., at least 80 percent agreement) in each section. These pilot tests will determine whether the protocols are an appropriate length, which items need to be revised for clarit y, and whether additional items are n eeded to develop a full picture of the program. **B5. Individuals and Organizations Involved in**  **the Project**  AIR is the contractor for Task Order 18: Sustaining the Positive Effects of Preschool . The project director is Dr. Kathry n Drummond, who is supported by an experienced team of researchers that will guide and conduct the case study data collection and analy sis (see Exhibit 10 for a list of key staff involved in the project , their role s , and c ontact information ) .Ameri can Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool

—20

**Exhibit 10**  **. Key Staff**  **Involved in the**  **Project**

**R ole**  **Organization Contact Name**  **Telephone Number**  Project Director AI R Dr. Kathryn Drummond 541 - 521 - 8517 Case Studies Task Leader AI R Karen Manship 650 - 843 - 8198 Site Visitor AI R Jennifer Anthony 650 - 843 - 8101 Site Visitor AI R Connie Chandra 650 - 843 - 8170 Quality Assurance Monitor AI R Dr. Eboni Howard 312 - 588 - 7339 Quality Assurance Monitor AI R Dr. Kerstin LeFloch 202 - 403 - 5649