American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—I-1

1695_03/15

Appendix I. Evaluator Protocol
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—I-2

Study on Sustaining the

Positive Effects of Preschool

Evaluator Protocol

Interviewer_____________________________________ Interviewee ID #_________________

Site __________________________________________ Date/Time_____________________

Introduction

My name is ______, and I work for American Institutes for Research (AIR). We’ve been 

contracted by the U.S. Department of Education to conduct case studies on sites that are 

implementing particular programs to support early elementary students. 

As you may know, the purpose of the study is to document characteristics of programs and 

policies in early elementary school that support student growth and development, such as PK–3 

alignment, differentiated instruction, and other programs designed to sustain the advantages of 

preschool. We are interested in learning more about the theoretical or practical background of 

these programs, how they are implemented, how they are sustained and funded, and their 

outcomes. 

I want to reassure you that your responses to this interview will be used only for research 

purposes. Though your site’s name will appear named in the final report, we will not provide 

your name in any reporting or to anyone outside the study team, except as required by law. 

Because we are producing case studies of only a small number of named sites, it may be possible 

for a reader to identify you based on the data that we report, even though your name will not be 

used. If there is information that you do not want shared directly in any reporting, please let me 

know. I will now give you a consent form that provides more details about the study and your 

rights as a participant.

Do you have any questions about the study?

I plan to audio record this session, solely for our note-taking purposes. Only research staff will 

use the audio recording, and we’ll destroy it when the project is done. Is that okay with you? 

Role and Perspective

1. I’d like to start by asking you to tell me a bit about yourself. I understand that you are the 

evaluator of [PROGRAM]. Briefly, can you tell me what your role has been in relation to 

[PROGRAM]? 

a. How did you come to be involved with [PROGRAM]? 

b. What related experiences did you have before you took on this role?

2. Can you tell me more about the organization that you work for and about the kind of work 

they typically engage in?
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—I-3

3. Why did [ORGANIZATION] become interested in partnering with [or evaluating] 

[PROGRAM]?

a. Can you describe the sequence of [PROGRAM’s] initiation and the 

[ORGANIZATION’s] involvement?

Evaluation Questions

4. When did your work evaluating [PROGRAM/STRATEGY] begin? 

5. Can you tell me about the basic study design of the evaluation?

6. What types of information have you collected over the course of the evaluation work? 

a. Why did you select this information to collect?

7. How does the evaluation measure students’ progress? 

8. Has the evaluation documented gains in student outcomes in kindergarten through third 

grade?

If interviewee reports effect: 

a. For which outcomes have you seen gains?

b. How are these gains measured or documented?

c. Have these gains been consistent over time? 

d. Can you share or point us to the data or report illustrating these patterns?

If no effect: 

a. What do you think are the barriers to seeing more student gains?

9. Has the evaluation documented changes in teacher practices? Teacher turnover? 

If yes:

a. Have you seen any patterns?

b. Can you share any data with us?

If no: 

a. What do you think are the barriers to seeing changes in teacher practice?

10. What else have you learned from the evaluation so far? 

11. How have you shared the results of your evaluation with district and school staff? 

12. In what ways and for how long will you continue evaluating the program?

13. Do you expect your work to help the district improve the program? If so, how?
American Institutes for Research Task Order 18: Study on Sustaining the Positive Effects of Preschool—I-4

Success, Challenges, and Closing

14. What challenges have you encountered working with a district on this type of initiative? How 

have you overcome those challenges?

15. Overall, what is working well with [PROGRAM]? How do you know?

16. What aspect of [PROGRAM] needs strengthening? Why? 

17. What challenges have you noticed in the district(s) and schools in implementing 

[PROGRAM], and what steps may be needed to address them?

18. [We already have (XX DOCUMENT)] OR [We do not have any written documents on 

(PROGRAM)]. Are there [other] written resources that document the program and its 

evaluation? 

a. Could we get a copy of those?

19. Those are all the questions I have. Are there any questions that I should ask about 

[PROGRAM] that I didn’t? Is there anything else you’d like to share with me today in regard 

to [PROGRAM]?