# Teacher and Leader Evaluation Systems Study:

**District Interview Protocol**

**Paperwork Burden Statement** According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB number. The valid OMB control number of this information collection is 1850
- 0890 . The time required to comple te this information colle ction is estimated to average 90 minutes per interview. If you have any comments concerning the accuracy of the time estimates(s) or suggestion s for improving this form, please write to: U.S. Department of Education, Wa shington, 20 202
- 4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Me lanie Ali, U.S. Department of Education, Institute of Education Sciences, Room 502B, 555 New Jersey Avenue, N W, Wa shington, DC 20208
- 5500.D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

2 [ Date ]

[ Recipient Title ] [ Recipient Street Address ] [ Recipient City , ST Zip ] Dear [Recipient], Thank you for your district’s continued participation in the Teacher and Leader Evaluation Sy stems Study . I am writing regarding the upcoming district interview. I have attached a cop y of the interview questions for your review. I encourage you to review the questions beforehand in case you need to consult a colleague regarding a question prior to the interview. In preparation for the interview, I’d like to acquire some documentation regarding yo ur s ys tems for evaluating teachers and principals that are i n effect in 2012

- 13. In particular, I am requesting that you send me blank copies of the forms y ou use for rating teacher and principal performance. If you have a separate form for classroom observations, please send that as well. In addition, I am request ing an y w ritten guidance that you provide to those who complete the ratings. Guidance may

include, but is not limited to, evaluator handbooks, manuals, or memos regarding how to rate instruction. Gathering these documents is an important part of our data collection process and will help us ensure that we are full y prepared for our i nterview with you.
I appreciate your assistance in helping us collect this information. Please send us this information at your earliest convenience. Electronic versions of these forms can be mailed to away ne@air.org , o r sent via standard mail to my address, below.
Sincerel y ,
Andrew Wayne, Ph.D. Project Director, TLES Study Am erican Institutes for Research 1000 Thomas Jefferson Street, NW Washington, D.C. 20007D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

3 **DISTRICT INTERVIEW PROTOCOL**

**I .**  **Respondent Backgrou**  **nd Information**  The questions in this section will be asked of all respondents to verify that each respondent has the information necessary to answer the interview questions. In setting up the interview, the interviewer will work with each district to identify respondent(s) with the necessary knowledge to respond to our questions and allocate questions to respondent ( s ) accordingly. First , we would like to discuss your specific responsibilities related to the topics we are going to ask a bout. **A. Information related to district role in teacher and principal**  **evaluation**

1. Please describe your position in the district and your role in the following areas, if an y:
a. T eacher evaluation b. T eacher transfer and staffing
c. Teacher professional development d. P rincipal evaluation and professional development
e. Implementation of activities associated with TLES

**II. Questions Related to Teachers and Teacher Evaluation** The questions in this section will be asked of respondents who are knowledgeable about policies and practices related to teachers and teacher evaluation. Now , I’d like to focus on current policies and practices related to teachers

. I will be asking specificall y about policies that are in effect for the 2012

– 13 school y ear. **B. Teacher evaluation policies and practices** 

1. R atings of teacher performance

a. According to your district’s human resources policies, are all teachers supposed to receive an annual performance rating? Does the frequency differ depending on the t ype of teacher (e.g. , novice teachers, teachers who received a satisfactory rating in the past)? I f such ratings are not given annually , how often do teachers receive an overall performance rating?D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

4 b. Is the overall rating derived by completing one of the forms that yo u provided? [Determine which form collected ahead of the interview is the correct form ]
c. What sources of information are used to determine the overall performance rating? Are the ratings based in part on:

i. Observations of classroom practice? [If yes, ask Q uestion A.2 below ] ii. Analy ses of the achievement test scores of the teacher’s students? [If

yes, ask Q uestion A.3 below ] iii. Administrator’s judgments about other aspects of performance (e.g., communication with parents). [If yes, obtain a brief description ] iv. Other sources (e.g., portfolios). [If yes, obtain a brief description

] d. Are there rules for how information from these sources should be combined to create the overall performance rating? 2. Details on classroom observations as a source of information for teacher performance ratings
a. To help determine performance ratings for teachers , how often are teachers required to be observed in their classroom? Does the answer differ depending on the ty pe of teacher (e.g., novice teachers, te achers who received a satisfactory rating in the past)? i. Are there rules about how long observations ty picall y need to last? ii. Are there rules about whether observations should be announced in advance? b. Who conducts classroom observations of teachers (e.g. , principals, assistant principals)?
c. Are the criteria for classroom observations those that appear on the overall evaluation form we discussed earlier, or is there a separate form used for classroom observations? [If separate, confirm understanding of which of the forms collected ahead of the interview is the form for classroom observations] d. Are those who observe given flexibility to use a different form or different set of criteria? If so, please explain.D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

5 e. Does the district require that observers be trained in how to conduct classroom observations and/or how to provide feedback based on an observation?

i. Who is responsible for providing the training to classroom observers? ii. Approximately

how many

hours of training are spent on how to rate teacher practice throu gh observation ? iii. Approximately

how many

hours of training are spent on how to provide feedback? iv. What, if any , assessment do observers need to pass before they conduct classroom observ ations for purposes of rating teachers ? 3. [If applicable

] Details on student achievement as a source of information for teacher ratings
a. In taking student achievement into account to determine the annual performance ratings for teacher s , are there requirements about which ty pes of student assessment data may be u sed? [If yes

] Which of the following may be used [Note yes or no for each ] : i. State mandated tests ii. District mandated tests iii. F inal examination grades iv. Other (please specify) b. [If not already addressed] For teachers in untested grades or subjects, is there a different policy regarding how student achievement is to be taken into account? If so, please briefl y describe the policy

. c. In anal yz ing the student achievement information, is it your policy to focus on the level of student achievement at the end o f the ye ar or the growth in achievement over the course of a y ear ? If growth is considered, how is growth determined? 4. Performance feedback for teachers

a. How often are teachers supposed to receive feedback on their performance? For example, are teachers give n feedback after each observation of their teaching? Are there other occasions when teachers are supposed to receive feedback?D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

6 b. For these feedback occasions, I’d like to know whether teachers are supposed to receive (i) oral feedback, (ii) ratings, and/or (iii) a written narrative of some kind.
c. After an observation, is there a time frame within which feedback must be given? 5. Use of teacher performance information

a. Can teachers earn additional pay based on th eir performance? b. Are teachers required to set professional growth goals? Inwhat way , if an y, is teacher performance taken into account in goal setting?
c. Are certain teachers required to participate in professional development based on their performance? d. In what way , if any , is teacher performance taken into account in determining whether teachers earn tenure in your district?
e. Are some teachers on contracts that have to be renewed? I f so, in what way , if any , is teacher performance taken into account in d ecisions regarding contract renewal. f. Were an y teachers identified for layoffs in the last 12 months? If so, in what way , if an y, was teacher performance taken into account in decisions regarding layoffs. g. Were an y teachers terminated for performance reasons in the last 12 months? h. In addition to the uses of individual teacher performance information, which were just mentioned, does the district use teacher performance information that has been aggregated (e.g., school

- level summary , district

- level summary)? For what purposes is such information used (e.g., for planning district

- wide professional development)? **C. Teacher mobility**  **policies and practices** Next, Iwould like to ask y ou about policies related to teacher mobility in your district. We would like to understand the process b y which teachers change schools.

1. First, I have some questions about teachers who seek to transfer from one school to another within the district

—that is, voluntary

transfers.
a. Which teachers are eligible to voluntarily transfer? How many ye ars of experience (in the district) do teachers need before they can voluntaril y transfer?D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

7 b. Do certain t ypes of voluntary transfers receive priority over others in filling vacancies (e.g., because of seniorit y)? If so, which teachers receive priority?
c. Can voluntary

transfers fill positions that are currently held b y novice teachers or other ty pes of teachers? d. Do principals have an y discretion regarding voluntary transfers? 2. I also have some questions about teachers who are involuntarily transferred between schools. Under what circumstances c an a teacher be invo luntaril y transferred?
a. When school enrollments change, in what way s do principals have discretion to select the teachers who will be involuntarily transferred? b. Are ther e other circumstances under which a teacher can be transferred involuntaril y? What role does the principal play in identifying who will be transferred under those circumstances ? 3. Were an y of the policies you just described new in 2012

– 13? [I f so:] Could yo u say a little about the timeline for the implementation of that policy? I ’m trying to get a sense of whether it was in full effect throughout the 2012

– 13 school y ear. **D. Implementation of TLES c**  **omponents for t**  **eacher evaluation**  Now, I’d like to turn to the evaluation s ys tem components related to teacher evaluation that have been implemented in the pilot schools only

—the schools that were randomly assigned to receive services using the stud y’ s resources. My first questions are abo ut the classroom observation sy stem, and then I have some questions about the student growth s ys tem.

1. Approach to implementing TLES components

a. After the initial summer training for those who conduct classroom observations, we had the principals and DSOs co nvene for additional support meetings approximately once per month. Have yo u do ne an ything outside these meetings to help p rincipals and DSOs successfull y conduct classroom observations and feedback sessions?  b. Have you encounter ed any challenges with the performance of principals in observing teachers and providing feedback ? What about DSOs? How did you address those challenges?

 Th is item and some additional items on this protocol are shaded to indicate that the burden for the items was not counted in the burden estimate. The respondent for the s e item s will be the individual in the district who is paid by the study to assist with study coordination activities.D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

8 c. We provided some parameters for scheduling the observations. Have you encountered any challenges in ensuring that the observations were conducte d in the specified time windows. What strategies were used to overcome an y challenges? [ A sk with respect to (i) principals and (ii) DSOs] d. Have you personall y use d the [ CLASS/FFT ] reporting s ys tem , or directed others to use it on your behalf ? i. P lease indicate if you used the reporting sy stem for management of the implementation of the classroom observations. Did you use it for tracking purposes, to ensure that observat ions were talking place? Did you use it to monitor the quality with which observation reports were being prepared b y observers? ii. Please indicate if you used the reporting s ys tem to access performance information. Did you use it to access individuals’ performance information? Did yo u use it to access perfo rmance information for particular groups of teachers (e.g., all teachers in a particular school). iii. Were there an y challenges that limited yo ur use of the [CLASS/FFT] reporting s ys tem? Were there challenges in getting access to the reporting s ys tem, or were there challenges in understanding the reports that the s ys tem provides? How did you overcome these challenges? iv. Can you identify

any suggested improvements to the [CLASS/FFT] reporting s ys tem?
e. Have you personall y use d the student growth reporting s ys tem , or directed another person to use it on y our behalf ? i. Please indicate how you used the sy stem
. Did you use it to access individuals’ performance information? Did you use it to access performance information for partic ular groups of teachers (e.g., all reading teachers)? Did you use it to access student growth information for individual schools? ii. Were there an y challenges that limited yo ur use of the student growth reporting s ys tem? Were there challenges in getting access to the reporting s ys tem, or were there challenges in understanding theD IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

9 reports that the s ys tem provides? How did you overcome these challenges? iii. Can you identify

any suggested improvements to the student growth reporting s ys tem? f. Was the webinar pro vided shortly after the release of the student growth results sufficient, or did you perceive a need for additional support? 2. Approach to i ntegration of TLES and current evaluation sy stem practices

a. In your implementation plan, you indicated that the distric t [WOULD/WOULD NOT] use TL ES observations in place of the observations required under the district’s current policy

. Has this decision give n rise to an y challenges for the district, and how were those challenges overcome? b. In your implementation plan, you i ndicated that TLES performance information [SHOULD/COULD/SHOULD NOT] be taken into account in giving performance ratings to teachers. Has this decision give n rise to any challenges for the district, and how were those challenges overcome?
c. To clarify , could performance information from TLES be used in (i) tenure review , (ii) contract renewal , or (iii) termination proceedings? 3. Implementation of additional, local uses of TLES sy stem In reviewing your implementation plan, I see that you have planned several activities to make use of the TLES s ys tem for teacher evaluation

. For study purposes, we are very interested in knowing which of these activities were implemented and when. We’d also like to know about the challenges you have encountered and the solution s that you have developed to overcome those challenges. [The interviewer will refer to the implementation plan as updated after the spring 2013 implementation planning visit. Because there is no spring 2014 implementation planning visit, the interviewer for the spring 2014 district interview will need to begin by asking about the status of each activity (i.e., implemented vs. not implemented) before proceeding with the questions. ]
a. [

For each activit

y that has reportedly already been implemented] I see you have already

implemented [ACTIVI TY]. i. Could you briefl y describe the timeline for implementing this activity? I ’m trying to get a sense of whether it was in full effect throughout the 2012

– 13 school y ear. [

D etermine when the activity wasD IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

10 initiated and the extent of implementation achieved since initiation. For example, indicate how many pilot schools implemented the activity and what proportion of teachers in those schools were likely affected

. ] ii. In implementing this activity , did you enc ounter any challenges, and how did yo u overcome those challenges? b. [For each activity that has reportedly NOT yet been implemented] I see you have not yet implemented [ACTIVI TY].

i. Could you say

briefly why this activity

has not y et been initiated? Have you encountered an y specific challenges?D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

11 **I I**  **I. Questions Related to Principals and Principal Evaluation** These questions will be asked of respondents who are knowledgeable about policies and practices related to principals and principal evaluation. Now , I’d like to focus on current policies and practices related to principals. I will be asking specificall y about policies that are in effect for the 2012

– 13 school y ear. **E. Principal evaluation policies and practices** 

1. R atings of principal performance

a. According to your district’s human resources policies, are all principals supposed to receive an annual performance rating? Does the frequency

differ depending on the t ype of principal (e.g. , novices, those who received a satisfactory rating in the past)? If such ratings are not given annually , how often do principals receive an overall performance rating? b. Is the overall rating derived by completing one of the forms that yo u provided? [Determ ine which form collected ahead of the interview is the correct form ]
c. What sources of information are used to determine an overall performance rating? For each source, please indicate how often it is required to be used. i. [If not already discussed] Are principals required to complete a self

- assessment as part of the principal evaluation process? ii. [If not already discussed] Inwhat way, if an y, is input from teachers supposed to be taken into account to determine performance ratings for principals ? 2. [If app licable

] Details on student achievement as a source of information for principal performance ratings
a. In taking student achievement into account to determine performance ratings for principals , are there requirements about which ty pes of student assessment data may be used? [If yes

] Which of the following may be used [Note yes or no for each ] : i. State mandated tests ii. District mandated tests iii. F inal examination gradesD IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

12 iv. Other (please specify) b. In anal yz i ng the student achievement information, is it your policy to focus on the level of student achievement at the end of the ye ar or the growth in achievement over the course of a y ear ? If growth is considered, how is growth determined? 3. Performance feedback fo r principals

a. How often are principals supposed to receive feedback on their performance from their supervisor ? For each occasion on which feedback is expected , I’d like to know whether the feedback is supposed to be based on the form identified earlier, or an y other form. I’d also like to know whether the feedback is supposed to include (i) oral feedback, (ii) ratings, and/or (iii) a written narrative of some kind. b. Were an y principals terminated for performance reasons in the last 12 months?
c. In addition to uses of individual principal performance information, does the district use principal performance information that has been aggregated (e.g., district

- level summary)? For what purposes is such information used (e.g., for planning district

- wide coaching of principals)? **F. Implementation of TLES components for principal evaluation**  Now, I’d like to turn to the evaluation s ys tem components related to principal evaluation that have been implemented in the pilot schools only , using the study ’s resources.

1. Approach to implementing TLES components

a. First I have some questions about y our experiences with administering the VAL

- ED survey

i. In administering the survey , did you encounter any concerns about the survey from the intended respondents? Please describe an y concerns you encountered, and the t yp e of respondent who expressed the concern (e.g., elementary school teachers). If you tried to address the concern, how did you do so, and were you successful? ii. Did you have an y challenges in getting the intended respondents to r espond and to submit their responses within the intended timeline? How did yo u address those challenges?D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

13 iii. Did you think the respondent training was adequate? Did you do any thing to supplement the respondent training? b. My next questions are about feedback giv en to principals after each VAL

- ED administration.

i. Did you encounter any challenges in the feedback sessions with principals? If so, what were the challenges, and how did you overcome those challenges? ii. Did you think the supervisor training was adequate? Di d you do an ything to supplement the supervisor training?
c. I also have some questions about the VAL

- ED reporting s ys tem. Did you personall y use the reporting s ys tem , or direct others to use it on your behalf

? i. Please indicate if the reporting s ys tem was used to access performance information. Was it used to access individuals’ performance information? Was it used to access performance information for all principals collectivel y? ii. Were there an y challenges that limited yo ur use of the VAL

- ED reporting sy st em? Were there challenges in getting access to the reporting s ys tem, or were there challenges in understanding the reports that the s ys tem provides? How did you overcome these challenges? iii. Can you identify

any suggested improvements to the VAL

- ED reporting sy stem? 2. Approach to integration of TLES and current evaluation sy stem practices

a. In your implementation p lan, you indicated that TLES performance information [SHOULD/COULD/SHOULD NOT] be taken into account in giving performance ratings to principals. Has th is decision give n rise to any challenges for the district? I f so, how were those challenges overcome? b. To clarify , could performance information from TLES be used in way s that had consequences for principals (e.g., promotion, termination, etc.)? Please explain. 3. Implementation of additional, local uses of TLES sy stemD IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

14 In reviewing your implementation plan, I see that you have planned several activities to make use of the TLES s ys tem for principal evaluation

. For purposes of the stud y, we are very intereste d in knowing which of these activities were implemented and when. We’d also like to know about the challenges you have encountered, and solutions that you have developed to overcome those challenges. [The interviewer will refer to the implementation plan a s updated after the spring 2013 implementation planning visit. Because there is no spring 2014 implementation planning visit, the interviewer for the spring 2014 district interview will need to begin by asking about the status of each activity (i.e., imple mented vs. not implemented) before proceeding with the questions. ]
a. [For each activity that has reportedly already been implemented.]

I see you have alread y i mplemented [ACTIVI TY ].

i. Could you briefl y describe the timeline for implementing this activity? I ’m trying to get a sense of whether it was in full effect throughout the 2012

– 13 school y ear. [

D etermine when the activity was initiated and the extent of implementation achieved since initiation. For example, indicate how many pilot schools implemented the activity and what proportion of teachers in those schools are likely affected] ii. In implementing this activity , have you encountered any challenges, and how have y ou overcome those challenges? b. [For each activity that has reportedly NOT yet been implemented] I see you have not yet implemented [ACTIVI TY].

i. Could you say

briefly why this activity

has not y et been initiated? Have you encountered an y specific challenges?
**IV .**  **Use of T**  **ime** These questions will be asked of respondents who are knowledgeable about tim e spent by district staff in support of the student growth component of the study. **G. Use of time to support student growth component of study**  One purpose of our interviews is to determine the total amount of time spent by district staff in working with the American Institutes for Research ( AI R) to implement the student growth component of TLES. Please answer the following questions with respect to the most recentl y completed round of student growth reports.D IS TR IC T I NTERVIEW P ROTOCOL S PRI NG 2013

15 1. After the district agreed on the covariates to be included in the value

- added model, h ow many

central office staff members were involved in wo rking with AIR to implement the student growth component of TLES? If applicable, please divide the staff into those considered “regular” and “ senior.” How many regular staff were involved, and how many senior staff were involved ? 2. I’m interested in determining the amount of time that regular and senior staff spent after the district agreed on the covariates to be included in the value

added model

. For each of the following, p lease provide an estimate of the total amount of staff time spent by regular s taff and the total time spent by senior staff.
a. Discussing data requests with AIR b. Preparing and submitting data to AIR
c. Troubleshooting and resubmitti ng information as needed