American Institutes for Research

1000 Thomas Jefferson Street NW

Washington, DC 20007-3835

202.403.5000 | TTY 877.334.3499

www.air.org

Copyright © 2015 American Institutes for Research. All rights reserved.

1695_03/15

Task Order 18: Study on 

Sustaining the Positive Effects 

of Preschool

Draft 2: OMB Clearance Request

Prepared for

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

March 2015

JANUARY 2015

Contents

Page

Supporting Statement for Paperwork Reduction Act Submission.Error! Bookmark not defined.

Justification (Part A)..................................................................Error! Bookmark not defined.

A1. Circumstances Making Collection of Information NecessaryError! Bookmark not defined.

A2. Use of Information Error! Bookmark not defined.

A3. Use of Improved Technology to Reduce Burden Error! Bookmark not defined.

A4. Efforts to Avoid Duplication of Effort Error! Bookmark not defined.

A5. Efforts to Minimize Burden on Small Businesses and Other Small EntitiesError! Bookmark not defin

A6. Consequences of Not Collecting the Data Error! Bookmark not defined.

A7. Special Circumstances Causing Particular Anomalies in Data CollectionError! Bookmark not defined

A8. Federal Register Announcement and Consultation Error! Bookmark not defined.

A9. Payment or Gift to Respondents Error! Bookmark not defined.

A10. Assurance of Confidentiality Error! Bookmark not defined.

A11. Sensitive Questions Error! Bookmark not defined.

A12. Estimated Response Burden Error! Bookmark not defined.

A13. Estimate of Annualized Cost for Data Collection ActivitiesError! Bookmark not defined.

A14. Estimate of Annualized Cost to Federal Government Error! Bookmark not defined.

A15. Reasons for Changes in Estimated Burden Error! Bookmark not defined.

A16. Plans for Tabulation and Publication Error! Bookmark not defined.

Case Study Analytic Approach...................................................Error! Bookmark not defined.

A17. Display of Expiration Date for OMB Approval Error! Bookmark not defined.

A18. Exceptions to Certification for Paperwork Reduction Act SubmissionsError! Bookmark not defined

Supporting Statement for Paperwork Reduction Act Submission.................................................14

Description of Statistical Methods (Part B)...............................................................................14

B1. Sampling Design 14

B2. Procedures for Data Collection 15

B3. Methods to Maximize Response Rate 19

B4. Expert Review and Piloting Procedures 19

B5. Individuals and Organizations Involved in the Project 19

References......................................................................................Error! Bookmark not defined.

Appendix A. Recruitment Materials.........................................A–Error! Bookmark not defined.
Appendix B. Consent Form ......................................................B–Error! Bookmark not defined.

Appendix C. Pre-Interview Survey for Districts and PrincipalsC–Error! Bookmark not defined.

Appendix D. District Interview Protocol..................................D–Error! Bookmark not defined.

Appendix E. Principal Interview Protocol................................ E–Error! Bookmark not defined.

Appendix F. K–3 Teacher/Program Staff Focus Group ProtocolF–Error! Bookmark not defined.

Appendix G. Preschool Teacher/Program Staff Interview/Focus Group ProtocolG–Error! Bookmark not define

Appendix H. Funder Protocol...................................................H–Error! Bookmark not defined.

Appendix I. Evaluator Protocol ................................................. I–Error! Bookmark not defined.
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—14

Supporting Statement for Paperwork Reduction Act 

Submission

Description of Statistical Methods (Part B)

B1. Sampling Design

The overall purpose of the case studies is to provide detailed information regarding how sites 

have succeeded in implementing their successful programs. The study team will therefore select 

a purposive sample of five sites that show evidence of successful practice for the selected 

topic(s). Sites will be districts or systems so that it is possible to learn from practices that are 

implemented more widely than in just one school or program. A site must demonstrate success 

among its students by showing positive outcomes in cognitive, social-emotional, or academic 

domains. 

Site Selection Procedures. To select sites for case studies, the study team considered sites or 

programs that emerge from the following sources: 

 Descriptions and findings from the research literature 

 Recommendations from early childhood experts, including

• American Institutes for Research (AIR) staff working on projects related to preschool 

and early elementary school programs

• Technical working group (TWG) members

• Program and Policy Studies Service (PPSS), Office of Early Learning (OEL), and 

Administration for Children and Families (ACF) partners

• State offices of early learning

• National-level associations, including the National Association of Elementary School 

Principals, the National Association for the Education of Young Children, and the 

National Institute for Out-of-School Time

To cull a list of potential sites from these sources, AIR first acquired any publicly available 

documents about the program (e.g., information from Web searches, public reports, and 

brochures) and summarized information about the programs (e.g., key activities, target grade 

levels, reported outcomes). Any site that did not meet initial criteria—including topic focus, age 

group targeted, or length of time program has been in place—was eliminated from consideration. 

Meanwhile, AIR and the U.S. Department of Education (ED) prioritized a list of potential sites 

by targeting programs that (a) focused on a larger range of grades, not just prekindergarten and 

kindergarten; (b) were districts implementing policies or programs across multiple schools or 

were schools that served as examples of a larger state policy; and (c) had some evidence that the 

policies or practices were intentionally focused on sustaining the effects of preschool.

The study team looked for performance data and any program implementation information about 

the prioritized sites from extant sources. When unavailable, the team requested them from the 
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—15

principal or district administrator.

1 Using this information, AIR conducted additional screening 

to eliminate sites that have not been able to continue the program or policy, shown grade-tograde decreases in achievement (based on available district or school assessment [e.g., progress 

monitoring or screening data]), or have shown obvious marked inconsistency in program 

implementation. From this list of finalists, AIR will select five final case study sites that will best 

answer the study’s research questions, working closely with the TWG, PPSS, OEL, and ACF. A 

back-up list of sites will be developed in case any first-choice sites do not wish to participate in 

the study. Across the sites, the study team will aim to include variation in approach, geographic 

location, urbanicity, and other factors that will be informed by the topic(s) of selected focus. 

Although this sample will be purposive and in no way representative, diversifying the cases as 

much as possible can provide insights into implementation issues that arise within and across 

particular contexts.

In order to recruit all five sites, AIR has developed recruitment materials, including an overview 

of the study that describes activities requested of participants, an introductory letter to district 

superintendents from ED, and an introductory letter to principals from AIR. These materials will 

be sent to sites when they are invited to participate. AIR staff will then follow up by phone.

Copies of the study’s recruitment materials are included in Appendix A. The study team will 

complete district research requirements, as needed, before collecting data. 

B2. Procedures for Data Collection

Discussions of the data collection procedures of each of the main components of the case study 

(document reviews and interviews) follow.

Data Collection Activities

Exhibit 8 presents an overview of the data collections planned for this study, with details 

following.

Exhibit 8. Overview of Data Collection Activities

Document 

Review Interviews

Program Activity 

Observations

Program characteristics 



Context  

Participants 



Program’s philosophy and model 



Staffing and expenditures 



Program activities and 

engagement 



 

1

In keeping with Office of Management and Budget requirements, we limited requests for additional information to 

fewer than 10 respondents.
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—16

Classroom characteristics 

Implementation challenges and 

solutions 

Staff training 

Program sustainability 

Student outcomes 

Document Review 

The study team will collect and review documents in stages as follows:

 During the process of identifying and recruiting potential sites, as described earlier, we 

will have acquired some publicly available documents describing the program and will 

have requested from sites some nonpublic documents, including more recent reports or 

activity descriptions.

 During site visits, site visitors will request other relevant documents available on-site, 

including training materials, student work samples, organizational charts, and any other 

information about the program’s goals and results. 

Review of all of these documents will be directed by a structured protocol in which analysts will 

respond to guided questions about information related to the program’s goals and activities, 

focusing on strategies and successful implementation for the site in sustaining positive outcomes. 

Semistructured Interviews 

At each site, site visitors will interview up to five program staff, five program teachers, and the 

program funder (e.g., from a foundation or private funding agency) and program evaluator, if 

applicable. AIR has designed interview protocols to be semistructured, guiding respondents to 

comment on questions of interest, but also maintaining a conversational tone to capture breadth 

of information. All interviews will be audiorecorded and later transcribed to facilitate coding.

Interview protocols (included in Appendices C–I) include questions about key constructs of 

interest, including characteristics of the program; the resources, personnel, staff characteristics, 

and/or training that facilitate implementation of the program; and program sustainability. AIR 

has developed a separate set of interview protocols for each respondent type. There is a preinterview survey for district staff and principals (Appendix C), as well as interview protocols to 

be used with (Appendix D) district staff, (Appendix E) principals, (Appendix F) elementary

teachers or other program staff, (Appendix G) preschool teachers, (Appendix H) funders, and 

(Appendix I) program evaluators. Teacher interviews may be conducted as individual interviews 

or focus groups, as determined by the schedules and roles of the selected interviewees. Timing 

will be determined in consultation with each school’s principal based on the teachers’ schedules. 

Each protocol will include questions that apply across all both topics of focus—differentiated 

instruction and PK–3 alignment—in addition to questions that focus specifically on each of the 

topics to be asked as appropriate depending on the program. 
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—17

Program Activity Observations

Case study site visits may include observations of classroom instruction, teacher meetings, and 

other activities relevant to the program being implemented. These observations will be guided 

by structured observation rubrics, but these are not included in the OMB package because they

do not create additional burden for participants. Observation forms will be customized to each of 

the five programs and depend on the age-range and setting of the program.

Institutional Review Board (IRB) Approval 

All data collection protocols will be submitted to AIR’s IRB for approval. AIR’s IRB, which is 

registered with the Office of Human Research Protection and operates under a federalwide 

assurance comparable to many universities, is responsible for reviewing all research conducted 

by the organization and its subcontractors. The IRB ensures that projects involving human 

subjects comply with professional standards and government regulations designed to safeguard 

participants and that research team members, including subcontractor staff, are adequately 

trained. The criteria that a study must meet for AIR’s IRB approval include assurances that risks 

to participants are minimized and balanced by benefits, subject selection is equitable, participants 

are informed about risks and give uncoerced consent, privacy of the subjects is adequately 

protected, and the rights and welfare of populations that might be vulnerable to coercion or 

undue influence are protected.

Conducting Site Visits

Data collection will take place over approximately a one-month period in the fall of the 2015–16 

school year. Before data collection, AIR will train site visitors and carefully plan with each site 

the schedule and logistics of the visit.

Training

Before site visits are conducted, all members of the case study team will participate in a one-day

in-person or webinar-based training to ensure that they thoroughly understand the content of the 

protocols and site visit procedures. Staff members will review best practices for interviews, in 

addition to conducting practice interviews. Staff will discuss strategies for (1) avoiding leading 

questions, (2) ensuring consistency, and (3) conducting interviews in a way that is 

conversational, yet still directed toward collecting the intended information systematically. 

Important procedural issues to be addressed include guidelines for ensuring respondent privacy. 

For observations, we will thoroughly introduce and practice the protocol to be used, making sure 

that observers understand the goal of each item and section. Observers will use video clips to 

practice taking high-quality observation notes and making summaries or ratings. 

Logistics

Each site visit will likely take two days. Two researchers will visit each site. This approach will 

allow two interviews to be conducted simultaneously, as needed, to minimize scheduling burden 

on respondents. The two site visitors will both be present for any focus groups, with one 

researcher serving as the lead interviewer and the other researcher serving as the secondary

interviewer, asking probes or encouraging additional participants to respond. All interviews and 
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—18

focus groups will be recorded for later transcription. The two researchers will co-observe in 

order to compare and debrief about activity observations. Site visit schedules will be developed 

in concert with the appropriate staff at the selected sites. The same pair of researchers will be 

responsible for scheduling and conducting visits, thus developing a rapport with staff at each site. 

A sample site visit schedule, reflecting a visit to one school, is presented in Exhibit 9.

Exhibit 9. Sample Site Visit Schedule

Day/Time Site Visitor 1 Site Visitor 2

Before site

visit, by 

phone

Conduct interview with 

superintendent

Conduct interview with 

district curriculum director

Day 1, 8:30–

9:30

Conduct interview with principal at 

School 1

Support principal interview; ask follow-up 

questions as needed, take brief notes

Day 1, 9:30-

10:30

Conduct interview with program 

evaluator

Support program evaluator interview; ask 

follow-up questions as needed, take brief 

notes

Day 1, 

10:30- 12:00 Observe Classroom Activities

Day 1, 

12:00–1:00 Lunch break

Day 1, 

1:00–2:00

Conduct focus group with preschool 

teachers at/affiliated with School 1

Conduct interview with 

preschool center director

Day 1,

2:50–3:50

Conduct focus group with elementary

teachers at School 1

Support teacher focus group; ask follow-up 

questions as needed, take brief notes

Day 2, 

9:00–10:00

Conduct interview with 

principal at School 2

Support principal interview; ask follow-up 

questions as needed, take brief notes

Day 2, 

10:00–11:00

Conduct interview with

elementary teacher 1

Conduct interview with 

elementary teacher 2

Day 2, 

11:00–12:00

Conduct interview with 

elementary teacher 3

Conduct interview with

other key program staff (e.g., coach)

Day 2, 

12:00–1:00 Lunch break

Day 2, 

1:00–2:00

Conduct focus group with preschool 

teachers at or affiliated with School 2

Conduct interview with 

preschool center director

Day 2, 

2:15-3:15 Observe Teacher Meeting

After site 

visit, by 

phone

Conduct interview with funder Support funder interview; ask follow-up 

questions as needed
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—19

B3. Methods to Maximize Response Rate

Data collection is a complicated process that requires careful planning. The research team has 

developed interview and observation protocols that are appropriately tailored to the respondent 

group and are designed to place as little burden on respondents as possible. The team will also 

pilot core data collection instruments with similar respondents in schools local to AIR’s offices 

to ensure that they are user-friendly and easily understandable, all of which increases 

participants’ willingness to participate in the data collection activities and thus increases 

response rates. 

In addition to careful instrument design, a high response rate among the case study sites may be 

ensured through careful recruitment materials. These recruitment materials emphasize the social 

incentive to respondents by stressing the importance of the data collections as part of a study of 

innovative practices that will provide much-needed information to districts and schools. AIR’s 

experience in past evaluations has demonstrated the importance and value of building a 

consensus of support with participating districts. This level of consensus leads to districts and 

schools that have the capacity, willingness, and commitment to cooperate fully with the research 

and data collection responsibilities. Investing in site development at the front end reduces 

problems at the back end, helping to ensure smooth implementation of the study.

B4. Expert Review and Piloting Procedures

Following publication of this Office of Management and Budget submission for 60-day public 

comment, the study team will pilot test all case study protocols by asking staff in similar roles in 

non-case- study programs to participate in a mock interview or classroom observation. We plan 

to conduct two pilot tests for each protocol, requesting participation from sites or personnel with 

whom we have worked previously. We will ask for feedback on questions that are confusing and 

incorporate changes in a final version. We also will note how long each interview takes. For 

classroom observation protocols, we will review ratings and notes across multiple observers for 

one given activity to ensure the form generates reliable data (i.e., at least 80 percent agreement) 

in each section. These pilot tests will determine whether the protocols are an appropriate length, 

which items need to be revised for clarity, and whether additional items are needed to develop a 

full picture of the program.

B5. Individuals and Organizations Involved in the Project

AIR is the contractor for Task Order 18: Sustaining the Positive Effects of Preschool. The 

project director is Dr. Kathryn Drummond, who is supported by an experienced team of 

researchers that will guide and conduct the case study data collection and analysis (see 

Exhibit 10 for a list of key staff involved in the project, their roles, and contact information).
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—20

Exhibit 10. Key Staff Involved in the Project

Role Organization Contact Name Telephone 

Number

Project Director AIR Dr. Kathryn Drummond 541-521-8517

Case Studies Task Leader AIR Karen Manship 650-843-8198

Site Visitor AIR Jennifer Anthony 650-843-8101

Site Visitor AIR Connie Chandra 650-843-8170

Quality Assurance Monitor AIR Dr. Eboni Howard 312-588-7339

Quality Assurance Monitor AIR Dr. Kerstin LeFloch 202-403-5649