# S UPPORTING	ST AT EMENT B : REQUEST	FOR	CLEARANCE	OF	INFORMA TION COLLECTION	FORMS	FOR

“ IMP AC TS	OF	A	DET

AILED	CHECKLIST	ON FORMA TIVE	FEEDBA CK TO TEA CHER S”
October 2014

**Submitted to: Submitted by**  : U.S. Department of Education SEDL Institute of Education Sciences 4700 Mueller Blvd. 555 New Jersey

Ave. NW, Rm. 308 Austin , TX 78723 Washington, DC 20208 Phone: ( 800 ) 476

- 6861

4700 Mueller Blvd. Austin, TX 78723 800

- 476

- 6861 www.relsouthwest.org This publication was prepared for the Institute of Education Sciences (IES) under contract ED

- IES

- 12

- C

- 00012 by Regional Educational Laboratory Southwest , administered by SEDL . The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

### Contents

Study Background

................................ ................................ ................................ ........................... 4 Impact Research Questions

................................ ................................ ................................ ......... 6 Implementation Fidelity

Research Questions

................................ ................................ .............. 6 Exploratory

Research Questions

................................ ................................ ................................ . 6 1. Respondent Universe and Sampling Methods

................................ ................................ ............ 8 Degree of Accuracy

Needed

................................ ................................ ................................ ........ 9 2. D escription of Procedures for the Collection of Information

................................ ..................... 9 Data Collection for Principal and Teacher Recruitment

................................ ........................... 10 Data Sources

................................ ................................ ................................ .............................. 11 Principal Baseline and Follow

-up Survey

................................ ................................ .............. 12 Teacher Baseline and Follow

-up Survey

................................ ................................ ............... 13 3. Description of Procedures for Maximizing Response Rates

................................ .................... 17 4. Description of Tests, Procedures, and Methods

................................ ................................ ........ 18 5. Individuals Consulted on Statistical Aspects of the Design and Key Staff

.............................. 19 References

................................ ................................ ................................ ................................ ..... 20 APPENDIX A

– Teacher and Principal Survey Instruments
................................ ........................ 21 Spring 2015 School Leader Survey

................................ ............ **Error! Bookmark not defined.** Spring 2016 School Leader Survey

................................ ............ **Error! Bookmark not defined.** Spring 2015 Teacher Survey

................................ ...................... **Error! Bookmark not defined.** Spring 2016 Teacher Survey

................................ ...................... **Error! Bookmark not defined.** APPENDIX B

– Principal and Teacher Recruitment Materials

................................ ................... 87 Wave 1 Principal Recruitment Script

– SPRI NG 2015

................................ .............................. 87 Wave 1 Teacher Recruitment Script

– – SPRI NG 2015

............. **Error! Bookmark not defined.** Wave 2 Principal Recruitment Script

– – SPRI NG 2016

............ **Error! Bookmark not defined.** Wave 2 Teacher Recruitment Script

– – SPRI NG 2015

............. **Error! Bookmark not defined.** Wave 1 Principal Recruitment Script Follow

-up ....................... **Error! Bookmark not defined.** Wave 1 Teacher Recruitment Script Follow

-up ......................... **Error! Bookmark not defined.** APPENDIX C

– Secondary Data Elements

................................ ................................ ................ 103Teacher and School Administrative Data Variables List

................................ ........................ 103

NM TEACH Reflect Data Variables List

................................ ... **Error! Bookmark not defined.** Principal and Teacher Contact Information

................................ **Error! Bookmark not defined.**4



### Study	Background The U.S. Department of Education (ED) requests clearance for the recruitment materials a nd data collection protocols under the OMB generic clearance agreement (OMB Number [IES to complete]) for activities related to the Regional Educational Laboratory Program (REL). ED, in consultation with SEDL , is planning a clustered randomized evaluation in New Mexico to test the effectiveness of materials intended to improve the feedback that principals provide in one

- onone conferences to their teachers about their classroom instruction. The study includes an impact anal ys is and an implementation analys is

. New Mexico Public Education Department (NM PED) staff have identified the topic of principal feedback to teachers as an area where New Mexico needs assistance. In particular, NM PED staff have received comments that principals do not feel adequately prepared to provide teachers with feedback about their instructional practices. The NM PED has committed to ensure that every student has equitable access to an effective principal and teacher every day they are in school. Although NM PED recognizes that the conferences are likel y an area in the overall evaluation system in need of improvement, it has limited resources and time to focus on the post

observation conference step in the evaluation cycle. Therefore, any potential interventions to address this i ssue must be eas y to implement and require few resources. This impact study will examine whether an enhanced feedback guide (Conversation Protocol), relative to business

- as -usual guidance to principals and teachers, achieves the following: 1. improves the structure and content of the principal

- teacher feedback conversation, 2. improves perceptions that the feedback delivered to teachers is useful 3. better targets guidance to teachers’ regarding professional development that is tied to their formal classroom observation scores 4. improves quality

of teacher instruction as measured by subsequent formal observation ratings on the state’s tool, called NM TEACH, and 5. increases student achievement and state standardized tests. To examine these outcomes, ED’s contractor will invite all public school principals of charter and non

- charter schools in the state to participate in the study . Blocking by school district, a randomly selected half of principals who agree to participate (along with teachers in their school who also agree to participate in the study) will be assigned to the treatment group . The remaining half of principals (along with teachers in their school who also agree to participate in the study) will be assigned to the control group. In summary, the study

wi ll compare principals and teachers in the treatment and control5

conditions, which are detailed in Table 1. **Table 1**  **. Treatment and Control Conditions**  **Treatment g**  **roup Control** 

**g roup**  **Principals** NM PED

- sponsored professional development for principals offered during the summer, of which 1.5

- 2 hours were devoted to feedback to teachers. Training conducted in regional in

-person professional development sessions led by the state’s contractor, Southern Regional Education Board (SREB). All principals in the state are required to participate as a part of their job requirement.  

List of documents to bring to each feedback conversation  Receipt of Conversation Protocol, which includes 24

- item checklist to use during each feedback conversation.

-  Link to a 5

-minute video in which principal testifies about her experience using the checklist.  Business

- as -usual guide to remind principals about “5 stages of feedback” described in NM PED

- sponsored principal professional development to use for feedback conversations

- 

**Teachers** Business

- as -usual guide: A document the includes a reminder to teachers about their right to feedback within 1 0 calendar days of formal observation under the NM TEACH Educator Effectiveness System, and a link to the NM PED provided FAQ on teacher evaluation (http://ped.state.nm.us/ped/NMTeach_FAQ.html) 

List of documents to bring to each feedback conversation  24

- item checklist to use during each feedback conversation

-  5

-minute video in which teacher testifies about her experience using the checklist 
ED’s contractor will request that the treatment group principals and control group principals do6

three things: (1) disseminate the relevant guide (and a link to the video, where relevant) to all school leaders and to all teachers in his/her building, (2) use the guide and view the video, where relevant (along with all other schools leaders in the buil ding who observe teachers) with 100% of the teachers that they

formally observe during school year 2015

- 2016, and (3) complete a baseline survey

in fall 2015 and a follow

-up survey in spring 2016. ED’s contractor will request treatment group teachers and c ontrol group teachers to do two things: (1) use the guide they received from ED’s contractor (and view the video, where relevant) in the feedback conversations they

have with their school leader(s), and (2) complete a baseline survey in fall 2015 and a fol low

-up survey in spring 2016. 

### Impact	Research	Questions The evaluation will examine the impact of providing a post

- classroom observation feedback checklist and a video to principals and teachers on principal , teacher, and student outcomes. To do so, this evaluation will use a cluster RCT design with random assignment at the school level to address the following impact and implementation questions. Does provision of the feedback checklist and video \ relative to the business as usual guidance: 1) improve the content and structure of post

- observation conference according to principals and teachers? 2) improve principals’ and teachers’ perceptions of the utility of post

- observation conference? 3) increase the amount of time it takes to complete the post

- observation conference? 4) cause principals to recommend and teachers to take professional development that is aligned to needs identified in the formal observations? 5) improve the quality

of teachers’ subsequent instructional practices as measured b y principals’ formal c lassroom observation scores? 

### Implementation	Fidelity	Research	Questions ED’s contractor will also examine the implementation of the Feedback Checklist relative to the business

- as -usual guidance by analyzing re sponses to survey questions regarding the ty pe of guidance used . The implementation portion of the study will address the following research questions: 6) How extensively

do principals and teachers in the treatment and control groups report using the form of guidance they were assigned? 7) How extensively

do principals and teachers in the treatment and control groups report using the form of guidance they were not assigned (i.e. cross

- overs)? 

### Exploratory	Research	Questions ED’s contractor will also perform exploratory analyses examining the association of the7

enhanced feedback protocol on subgroups of teachers and principals. In addition, we will also examine whether provision of the treatment guide improves subsequent student performance on state standardized tests. Although the design of the study allows for causal anal ys is of these questions, for reasons of limited sample size within New Mexico and limited time for the treatment or control condition to influence behavior and student achievement, we will likely have insufficient statistical power to detect differences. Nevertheless, these questions have high policy relevance for NM PED, so we include them to provide insights for follow

- on studies and for NM PED’s improvement of subsequent iterations of NM TEACH observations and feedback. The exploratory

research questions are : 8) Does the provision of the checklist to principals and teachers increase student achievement relative to business

- as -usual guidance regarding feedback? 9) Is there sugge stive evidence that the effects of providing the checklist varies according to the following subgroups of principals and to the following subgroups of teachers: a) extensive users of the checklist

– i.e., teachers who have received three versus two formal observations and principals who have used the checklist in schools that have a greater than average number of teachers within the given district b) personal beliefs and attitudes about NMTEACH evaluation system c) accountability

pressure (as measured by prior ye ar’s school accountability grade) d) those principals who have received training on the NMTEACH Observation Rubric e) those who work in schools that teachers rate as collaborative and supportive f) those with greater qualifications, including more years of experience and certification ty pe g) those who teach core subjects versus non

- core subjects h) those who work in elementary versus middle or high schools i) teachers in the top and bottom quartile of the NM TEACH summative effectiveness score distribution j) principals in the top and bottom quartile of the NM TEACH School Leader effectiveness score distribution Be yond subgroup anal ys es, ED’s contractor will also gather information about the perceptions users have of the treatment checklist and the control guide. 10) What are principals’ and teachers’ perceptions of the checklist and videos versus business

- as usual feedback guidance? For example, do they report that it is easy

to use, burdensome, formulaic, lengthy, useful for creation of professional development plans, and helpful in causing teachers to commit to instructional changes?8

11) What are the main reasons why principals report not using the form of guidance they received? checklist or the business

- as -usual guidance? OMB approval is being requested for a multimode data colle ction and anal ysis of principal, teacher, and student outcomes in public schools in New Mexico . The project consists of data collection from NM PED and participating principals and teachers who work in New Mexico public schools . Specifically, in this OMB c learance package, ED is requesting clearance for the following data collection approaches:
 Recruitment materials for participating principals and teachers
 Extant administrative records data collections from NM PED
 Two waves of web

-based surveys of principals and teachers in treatment and control schools

. ED believes that the data collections for which clearance is being requested represent the bare minimum necessary

to assess the impact and implementation of the Conversation Protocol. The remainder of Part B address es the following: respondent universe and sampling methods; description of procedures for maximizing response rates; description of tests, procedures and methods; and contact information for statistical consultants and key staff. 

### 1.	Respondent **Universe	and	Sampling	Methods** This section describes the respondent universe and sampling methods for recruitment. To understand the impact of providing guidance to principals and teachers regarding feedback sessions with teachers following formal classroom observations, ED’s contractor will implement a randomized controlled trial (RCT) evaluation. The recruitment strategy will focus on both New Mexico public school principals and New Mexico public school teachers

. ED’s contractor will conduct recruitment in a top
- down approach, with principal recruitment preceding teacher recruitment. First, ED’s contractor will send out an introductory email, and then invite via email all principals in public schools governed b y the New Mexico Public Education Department (NM PED) to participate in a research study . As of the school year 2012

–2013, th ere were 855 school principals across the elementary, middle, and high school levels

. 1 There are no additional inclusion criteria for principals recruited into the study

. We anticipate that after three rounds of recruitment follow

-up to nonresponding principals, about 510 principals will consent to participate in the study.

1 While the precise number of schools may shift minimally by the time the principal recruitment takes place in August 2015, this change is not expected to impact our statistical power calculations.9

Next, in schools where principals consent to participate in the study, ED’s contractor will randomly select up to 10 teachers to participate in the study

. In schools with less than 10 teachers , all teachers will be invited to participate. In schools where more than 10 teachers work, ED’s contractor will sample 10 using simple random sampling to invite them to participate. Across New Mexico, there are approximately 21,500 FTE public school teachers according to Common Core of Data. We anticipate that after three rounds of recruitment to nonresponding teachers, about 5,130 teachers will consent to participate in the study

. 

### Degree	of	Accuracy	Needed The power anal ys es to estimate the minimum detectable effect size (MDES) were conducted using the Optimal Design software package, version 3.01 (Spy brook, Bloom, Congdon, Hill, Martínez , and Raudenbush, 2011), with interpolation of results as necessary to account for the non- constant number of schools per district. ED’s contractor drew from prior studies using student achievement and principal and teacher survey data to predict participation rates and the anticipated correlations of the covariates with the outcomes of interest. The study will examine continuous principal and teacher outcomes, binary teacher and principal outcomes , and continuous student outcomes. We anticipate that our principal sample will be sufficient to detect minimum effect sizes between 0.1 2 and 0. 21 standard deviations for continuous outcomes with 80% power and a 5% Typ e I error rate, under reasonable assumptions for participation, attrition, and baseline correlation. In addition, the sample of students in these schools with participating principals will be sufficient to detect minimum effect sizes of between 0.09 and 0.14 s tandard deviations for individual grade level student outcomes, again with 80% power and a 5% T yp e I error rate, under reasonable assumptions for participation, attrition, and baseline correlation. The sample of teachers and principals will be sufficient to detect minimum effect sizes of between 0.11 and 0.18 standard deviations for continuous outcomes, and 0. 15 and 0. 2 4 standard deviations for binary outcomes with 80% power and a 5% T ype I error rate, under reasonable assumptions for participation, attrition, and baseline correlation

. 

### 2.	Description	of **Procedures	for the**  **Collection of	Information**  ED’s contractor, REL Southwest, will manage data collection and ensure quality and timeliness. ED ’s contractor propose s to collect both primary and secondary

data . The primary

data will consist of a Web

-based baseline and follow

-up survey for principal s and teachers (for four total survey instruments). The draft recruitment texts for study participation are included in Appendix B , and the survey

instruments (along with the consent language) for data collection are included in Appendix A

. The secondary data consists of formal observation scores by school leaders of10

teacher practice as well as principal, teacher and school characteristics and student achievement scores collected b y NM PED. The data request of extant data from the NM PED is included in Appendix C. The content of Appendix A, B, and C are also included in Statement A of this OMB package. 

### Data	Collection	for	Principal	an **d	Teacher	Recruitment** ED’s contractor will contact all principals in public schools governed b y the New Mexico Public Education Department (NM PED) to invite them to participate in the study. Recruitment will be facilitated b y clear materials and communication about the nature of the study and the requirements for participation, including complying with random assignment. The following sections describe more full y each step in the recruitment process: **(1) Obtain principal and teacher email addresses from NM PED.**  Starting in January

2015, NM PED will provide to ED’s contractor their existing file of principal and teacher email addresses for all those who worked in school year 2014

- 2015. This list will be cleaned in February

and March 2015 in advance of recr uitment in April 2015. ED’s contractor will also obtain from NM PED the official start and end day of each NM school district’s school ye ar. **(2) Recruit principals and administer baseline principal**  **survey.** NM PED will send an introductory email to all principals in the first week of April 2015 ( or after OMB approval if this occurs after April 2015 ), describing REL SW, explaining the study, and endorsing the study

and encouraging principals to participate. REL SW will then send a follow

up email to princ ipals (up to three times to non

-respondents)

– first week of May, third week of May and first week of June

—seeking their participation in the study and issuing a link and password to the baseline principal survey

. We will also mail the recruitment letter to the principal’s school up to two times. If a principal has not responded to any of the three emails or two letters , they will be considered to be out of the study

. The principal recruitment email is included in Appendix A. **(3) Recruit teachers and admi nister baseline**  **teacher survey (two weeks after step 2).**  During April

- May 2015 while the teacher and principal rosters are being cleaned provided b y NM PED, ED’s contractor will randomly select up to 10 teachers who are teaching in the 2014

- 15 school year in a given school to be invited to participate in the study using simple random sampling . At schools with fewer than 10 teachers, all teachers will be selected for recruitment. After the principal in the schools has consented to participate in the study, the introductory study recruitment email will be sent to teachers in the school in the third week of April, asking them to11

participate in the study. Up to three emails will be issued until the close of the school year, and up to two letters will be sent to the schools, at which point non -respondents will be excluded from the study . The recruitment email will include a link and password to the baseline teacher survey . The teacher recruitment emails is included in Appendix B

. **(4) Conduct random assignment of principals.** Once the window has closed for study recruitment in earl y summer 2015, ED’s contractor will randomly assign 50% of participating principals to the treatment group according to the following blocked random selection procedures. Using data from NM PED about 2014

- 2015 school characteristics, ED’s contractor will randomize participating schools within each district, stratifying schools according to school level (elementary, middle and high school). Prior to randomiz ation, ED’s contractor will also examine the distributions of school size and racial/ethnic composition (high and low percent Hispanic and Native American) among participating schools to determine if any further advantage may be gained by

stratifying on these characteristics as well. 2 Incases where onl y a single school from a district is participating, ED’s contractor will combine multiple districts and randomize within matched districts. Randomization will occur in the first week of August 2015. By definition, teachers and students whose principals who are in the treatment group will belong to the treatment group. 3 

### Data	Sources ED’s contractor will collect both primary and secondary

data to understand the impact of providing the Conversation Checklist to principals and teachers relative to the business

- as -usual control guide

. The primary data will consist of two baseline surveys in spring 2015 (one to teachers and one to principals) and follow

-up surveys in spring 2016. The secondary data consists of (a) f

ormal observation scores by school leaders of teachers’ classrooms throughout the 2014

- 15 and 2015

- 2016 school year; (b) principal, teacher and school characteristics

. Table 2 provides a summary

table of the data collection, including the data sources, data elements, and timeline. The following section describes the data ty pes in more detail. **Table 2** 

**– Data Sources, Data Elements and Timeline for Data Collection**  **Data Source Type of**  **Data Timeline for Data Collection** 

2 Given that all non

- charter principals in the state will be recruited for the study, stratification is not necessary to produce a representative sample. Randomizing within district optimizes power. Among small districts, stratifying by school level guards against treatment and control schools being confined to mutually exclusive school levels within a district. Given the homogeneity of within

- district school composition, ED’s Contractor does not expect further stratification by size or race/ethnicity to be advantageous, but will allow for that option if it reduces the potential for im balances in treatment assignment on these school characteristics across the participating schools. 3 If a principal moves away from the school, we will assign the new principal to the same group as the previous principal, and send them the relevant guide. The regression analyses will control for teacher and principal mobility.12

**Data Source Type of**  **Data Timeline for Data Collection**  Principals Baseline survey April 1 5 , 2015

– June 15, 2015

Follow

-up survey April 15, 2016

– June 15, 2016 Teachers Baseline survey April 30 , 2015

– June 1 , 2015

Follow

-up survey April 15, 2016

– June 1, 2016 NM PED Principal and Teacher

Contact Information April 1, 2015, July 1, 2015, April 1, 2016 Student

- level achievement data June 2015, June 2016 School Report Card Grades July 2015, July

2016 School administrative data January 2015, December 2015, December 2016 NM TEACH observation rubric scores from REFLECT for 2014

- 15 June 2015

NM TEACH observation rubric scores from REFLECT for 2015

- 16 November 2015, February 2016, June 2016, December 2016

ED’s contractor will collect two waves of survey data from both principals and teachers. To increase the precision of the treatment effect estimates, ED’s contractor has designed the baseline (wave 1) and post

- treatment (wave 2) survey to contain identical items within respondent ty pe. This is to allow us to include baseline responses as a predictor within our re gression models for each correlative outcome of interest. The onl y exception to this rule is in wave 2 surveys where ED’s contractor added a battery of questions for treatment group respondents and for control group respondents to understand their use (or not) and perceptions of the Conversation Checklist. **Principal	Baseline	and	Follow -**  **up	Survey** The baseline and follow

- up principal survey instruments contains eight sections that cover the following topics (see Attachment B for the full survey instrument):
 Your Position and Job Responsibilities : ty pe of school leader, years of experience as school leader
 Your Perceptions of the Quality Effectiveness of Your School’s Teaching Staff This Year : subjective rating of the quality of teachers
 Your Practices When Observing Teachers This Year (SY 2014

- 2015) for Evaluation Purposes : number of teachers observed formall y, length of post

- observation conference, frequency of informal observations, practices and perceptions of the post

- observation13

conference, (the follow

-up survey includes questions about whether the treatment and control guides were received, used, frequency of use, and perceptions about the guide)
 Professional Development Offered to Teachers This Year (SY 2014

- 2015) : ty pe of professional development offered, quality of professional development, alignment of professional development options with NM TEACH Observation Rubric
 Training You Received About NM TEACH to Date : ty pes of training received, including observation rubric, software, interpreting ratings, providing feedback, utility of training
 Your Personal Views of NM TEACH Educator Effectiveness System : subjective rating of evaluation s ys tem
 Teacher Retention and Improvement : number of teachers who have left the school, reason for dismissal
 Optional Comments from You : comments on the observation rubric, professional development available, and training on teacher evaluation system.
**Teacher Baseline**  **and	Follow -**  **up	Survey** The baseline and follow

- up teacher survey instruments contains seven sections that cover the following topics (see Attachment B for the survey instrument) :  Your Position and Job Responsibilities : ty pe of teacher, teaching arrangement, grades taught, ye ars of experience
 Your Experiences This Year (SY 2014

- 2015) of Formal Observations for Evaluation Purposes : number of formal observations, length of post

- observation conference , practices and perceptions of the post

- observation conference, ratings on two of the four domains in the observation rubric, self

-rating on two of four domains in the observation rubric (the follow

-up survey include s questions about whether the treatment and control guides were received, used , frequency of use, and perceptions about the guide)
 Professional Development You Received This Year (SY 2014

- 2015) : type of professional development taken, quality of profess ional development, availability of and support for professional development
 Your Perceptions of the Quality Effectiveness of Your School’s Leadership Last Year : subjective rating of the quality of school leadership
 Your Personal Vi ews of NM TEACH Educator Effectiveness System : subjective rating of evaluation s ys tem
 Your NM TEACH Summative Evaluation Score Rating : overall rating, feedback received from principal after overall rating
 Optional Comments from You : comments on the observation rubric, professional development available, and evaluation system.14

All student

- level information will be collected through extant data from NM PED

. To reduce burden on survey

respondents, all respondent and school characteristics will be collected from NM PED’s secondary

data sources. ED’s contractor will provide a list of requested data fields to NM PED prior to the first data collection request. NM PED will also provide school report card grades for schools years 2014

- 2015 (baseline) and 2015

- 2016 (outcome). 4 In addition, NM PED will provide ED’s contractor with classroom observation

- level scores from the NM TEACH Observation rubric. Appendix C contains a list of the data elements that will be obtained from NM PED. The impact and exploratory anal ys is measures will be constructed from survey questions and administrative data sources, as described in Table 3 . Th e following items are outcome measures derived from principal and teacher surveys:  **Good Practices Index** : this index is a measure of whether the practices that are recommended in the Conversation Checklist are implemented in the feedback conference. There will be one index for principals, and one for teachers of good practices. We anticipate that in each index, the items will include whether the conversation included at least one practice that the teacher does well, identified at least one challenge, prioritized next steps, ended on a positive note, and mutually developed next steps.
 **Feedback Conversation Perceptions Index** : We anticipate that this index will be derived from the extent to which principals agree with statements regarding the feedback conference, , the principal gave specific feedback, the principal gave actionable feedback, the teacher had an equal chance to speak during the conversation, the principal or teacher feels positive about the feedback provided, the principal or teacher experienced a high level of conflict during the feedback conference, and the principal found it difficult to provide criticism.
 **PD is tailored to need**  **s identified in formal classroom observation:** this measure will be based on a survey

question that asks the teacher whether they were recommended to take PD as a result of a rating in each of the five Domain 2 and five Domain 3 items of the NM TEACH Observ ation Rubric.
 **Fraction of targeted professional development that was recommended by the principal and taken by the teacher:**  This outcome measure will be based on teacher survey responses about taking professional development that was recommended. Of all of the recommended PD, we anticipate estimating the fraction that was taken by the teacher in the 2015

- 2016 school year (including PD taken in person and online).

4 School grades are computed on a continuous scale from 1 to 100, and are publicly reported annually

.15

**Table 3 .**  **Impact and Exploratory Analysis Outcome Variable, Research Question**  **, and Data Source**  **Mapping Outcome**  **Research**

**Question Data source**  **Impact outcomes** Good Practices Index (measure of post

- observation feedback conference content and structure) RQ1 & RQ9 principal survey

- Wave 1 survey : Q12b, Q12c, Q12d, Q12e, Q12f, Q12g, Q12h, Q12j, Q12m, Q12n, Q12o, Q12u

- Wave 2 survey : Q12b, Q12c, Q12d, Q12e, Q12f, Q12g, Q12h, Q12j, Q12m, Q12n, Q12o, Q12u teacher survey

- Wave 1 survey : Q11b, Q11c, Q11d, Q11e, Q11f, Q11g, Q11h, Q11j Q11p, Q11q, Q11s, Q11w

- Wave 2 survey : Q11b, Q11c, Q11d, Q11e, Q11f, Q11g, Q11h, Q11j Q11p, Q11q, Q11s, Q11w Feedback Conversation Perceptions Index (measure of the perceived quality

of feedback) RQ2 & RQ9 principal survey

- Wave 1 survey : Q12a, Q12k, Q12l, Q12p, Q12q, Q12r, Q12s, Q12t

- Wave 2 Survey : Q12a, Q12k, Q12l, Q12p, Q12q, Q12r, Q12s, Q12t teacher survey

- Wave 1 survey : Q11a, Q11i, Q11k, Q11l, Q11m, Q11n, Q11o, Q11r, Q11t, Q1 1u

- Wave 2 survey : Q11a, Q11i, Q11k, Q11l, Q11m, Q11n, Q11o, Q11r, Q11t, Q11u Average number of minutes it takes to complete the post

observation conference RQ3 & RQ9 Principal survey

- Wave 1 survey : Q10

- Wave 2 survey : Q10 Teacher Survey

- Wave 1 survey : Q9

- Wave 2 survey : Q9 PD is tailored to needs identified in formal classroom observation RQ4 & RQ9 teacher survey

- Wave 1 survey : Q12, Q14

- Wave 2 survey : Q15 Fraction of targeted Professional development that was recommended by the RQ4 &

RQ9 teacher survey

- Wave 1 survey : Q13

- Wave 2 survey : Q1516 **Outcome Research** 

**Question Data source**  princip al and taken b y the teacher Domain level Observation Rubric score RQ5 & RQ9 REFLECT Observation proficiency

level RQ5 &

RQ9 REFLECT **Exploratory questions** School

- level student achievement scores RQ8 NM PED Administrative Data Effects b y subgroup RQ9 Teacher Survey

- wave 1: Q1

– Q7; Q16

– Q22

- wave 2: Q1

– Q7; Q17

– Q24 principal wave 1 & 2: Q1

- Q6; Q15

- Q22 NM PED Administrative Data Note: REFLECT is the system designed by Teachscape for NM PED that stores observation rubric scores and observation narratives For cases where multiple survey items are the source of the outcome measure, we will conduct exploratory

factor anal ys es to generate an index of the given construct. For each construct, we anticipate testing whether the survey items measure the latent variable described in Table 3 . The precise number of items retained in each scale will depend on each item’s factor loading. As an initial step, we will a scree plot to visually display the Eigenvalues, and we anticipate retaining factors for which the minimum Eigenvalue is greater than or equal to one. However, we will not apply a strict test, and may, for substantive reasons, retain factors with Eigenvalues of less than one or drop factors with Eigenvalues of greater than one. We will then use oblique rotation and varimax rotation to determine which items load most highly onto which of the retained factors. We intend to retain only factors for which we have an underl ying theory or hy pothesis for the apparent latent variable measured by a factor, and we wil l derive our theories from the theory

of action about the Conversation Checklist. Namely, we will test if the feedback checklist, as intended, makes conversations seem more mutual (rather than from principals to teachers), make feedback more specific, and lead to actionable next steps. Operationall y, we will generate the scales b y averaging the respondent

- level sums of item

- level responses derived from the relevant survey items. A coefficient Alpha of 0.70 or greater is generall y considered an acceptable l evel of internal consistency within a given factor, with lower values indicating a potential lack of adequate reliability

. Meanwhle, the following are the outcome measures that are collected from NM PED from the REFLECT teacher observation rubric s ys tem or from administrative data:17

 **NMTEACH Observation rubric scores** : These are item level measures from the REFLECT system on each of the four domains of the NM TEACH Observation tool, and proficiency

levels associated with the item level measures. We will examine changes to the Domain

- level scores and proficiency ratings from the prior year as well as changes over the course of the ye ar when the Checklist is available.
 **Student achievement on tests** . We will estimate whether the treatment raises average school

-wide achievement on standardized tests. We will do this by modelling student

level achievement data hierarchicall y to account for the clustering of students in schools and to estimate the effect of the guide on average student achievement according to whether students are in treatment or control group schools. The school

- level student achievement will include separately modeled math, ELA, and science PARCC scores. We will examine the difference in student achievement between students who have teachers that are using and not using the Conversation Checklist in the feedback conference, controlling for prior achievement scores and relevant student and school covariates. We will examine differences overall at the school level, with sub

- anal ys es b y grade.
 **School report card data.**  Additionally, the academic component of the school report card measure will be used as a complimentary impact measure for student achievement observed at the school level. NM PED will provide the continuous scale academic composite measure currently in use in the state comprising 90 percent of the final school report card grade. 

### 3.	Description	of	Procedures	for **Maximiz ing**  **Response	Rates** ED is committed to obtaining complete data for this evaluation. A large share of the impact anal ys is question for the evaluation relies heavily on administrative data

. ED’s contractor anticipate s a 95

-percent response rate from NM PED on student, teacher and principal measures in the administrative data , as well as teacher observation scores . A key to achi eving complete administrative data is tracking the data components from NM PED with e

- mail and telephone contact to the appropriate parties to resolve issues of missing or delayed data files

. ED’s contractor has a strong working relationship with representative s from NM PED. All administrative data files will be reviewed for consistency and completeness

. I f a data file has too many missing values or if an instrument in the implementation study has too few items completed to be counted as a response , ED’s contractor will seek to obtain more complete responses by e -mail or phone. Based on its prior experience with administering surveys to principals and teachers in a variety18

of schools, districts, and states, ED’s contractor expect s the response rate for the baseline survey to be at 85 percent for those principals and teachers who consent to participate in the study

. We will contact non

-responding principals and teachers up to four times to encourage participation. Three f ollow

-up email reminders will be sent to individual respondents in the event that responses are not obtained for W eb

-based surveys . Finally, as a fourth and fifth contact attempt we will send a reminder letter to their school (twice) , after which point we will deem the person a non-re spondent and exclude them from the study

. In addition, a number of steps will be taken to maximize response rates. For example, sampled respondents will receive advance communications explaining the study, introducing the REL Southwest, provide an assuranc e of confidentiality, and encourage them to participate as a way to help NM PED refine and improve NM TEACH . Respondents also will be given a contact number to reach ED’s contractor with questions

. Finally, respondents will receive an incentive for participating in the study :

 $ 25 gift card for principal for completing the baseline survey, and $ 25 gift card for completing the follow

-up survey
 $ 25 gift card for teachers for completing the baseline survey, and $ 25 gi ft card for completing the follow

-up survey Finally, ED’s contractor will consider the most recent statistical literature and work with IES to determine the most appropriate method for handling missing data. They will use appropriate anal ytic methods to account for missing data, and consider options such as complete case anal ys es with regression adjustment, maximum likelihood methods, non

-response weights, or fitting the models specified above in a Bayesian hierarchical framework. Upon obtaining the data, ED’s contractor will examine the extent of missing data overall and b y treatment group. 

### 4.	Description	of **Tests ,**  **Procedures ,**  **and	Methods** To measure the impact of the Conversation Checklist on the structure and content of the feedback conference , the quality of the feedback provided by principals and received by teachers, the professional development that is recommended by principals and completed by teachers, ED’s contractor will administer principal and teacher surveys. The survey items have been reviewed b y RAND colleagues who were formerly employed as teachers and colleagues with content expertise

. ED’s contractor has also conducted cognitive interviews of two retired New Mexico teachers and five retired principals by asking the participants to complete the survey and comment on the clarity of the questions, whether the questions assess the intended constructs, whether the tone of the questions is appropriate for the audience, and whether the length of the survey

is suitable. During 1.5

- 2 hour phone calls, ED’s contractor solicited detailed19

f eedback on the clarity, tone, and wording of these documents. With each participant, we mailed them hard cop y materials several days in advance of our scheduled call, asking them to complete the survey

and read the materials prior to the call. After the c all, ED’s contractor asked participants to return the hard copies with comments on them in a pre

-paid return envelope. Based on the feedback from the pilot test, ED’s contractor has refined and clarified the survey questions and made clarifications and ref inements to the feedback protocol. For example, ED’s contractor learned that tenure is a term not widel y understood and agreed upon in New Mexico, and that it set by

the state rather than districts. ED’s contractor has therefore dropped some of the tenure questions on the survey . ED’s contractor also changed some words that sounded negative or punitive in tone about teachers to retired principals, and ensured that the list of documents participants in the treatment checklist are accurate and described in waysthat principals and teachers would understand.


### 5. **Individuals	Consulted	on	Statistical	Aspects	of	the	Design	and	Key	Staff** ED’s contractor consulted Lou Mariano, Senior Statistician and director of the Statistics Unit at RAND Corporation concerning the methodology, study de sign, and data collection approach/burden

. The following individuals were consulted on the statistical, data collection, and analytic aspects of the principal feedback evaluation study through REL Southwest’s Technical Working Group ( TWG) : **Dan Goldhaber, Ph.D .**  Director, CALDER (National Center for Anal ys is of Longitudinal Data in Education Research) Vice President, American Institutes for Research (AIR) Director, Center for Education Data & Research (CEDR), University of Washington Bothell Co

- Editor, Education Finance and Policy 3876 Bridge Way

N,Suite 201 Seattle, WA 98103 Ph: 206

- 547

- 1562 Fax: 206

- 547

- 1641 E

-mail: dgoldhab@uw.edu **Geoffrey Borman, Ph.D.** Professor of Education, University of Wisconsin

—Madison Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research20

Training Program Senior Researcher, Consortium for Policy Research in Education. 348 Ed ucation Building 1000 Bascom Mall Madison, WI53706

- 1326 Ph: 608

- 263

- 3688 Fax: 608

- 265

- 3135 E

-mail: gborman@education.wisc.edu **Johannes M.(Hans) Bos , Ph.D.**  Vice President and P ro gram Director, International Development, Evaluation, and Research (IDER) Program American Institutes for Research 2800 Campus Drive, Suite 200 San Mateo, CA 94403 Ph: 650

- 843

- 8100 Fax: 650

- 843

- 8200 E

-mail: jbos@air.org **W. Steven Barnett, Ph.D.** Board of Governors Professor and Director of the National Institute for Early Education Research Rutgers University 73 Easton Avenue New Brunswick, NJ 08901 Ph: 848

- 932

- 4350 x23132 Fax: 732

- 932

- 4360 E

-mail: sbarnett@nieer.org


### References Raudenbush, S.W., Spy

brook, J., Congdon, R., Liu, X , & Martinez, A., Bloom, H., & Hill, C. (2011). Optimal Design Software Plus Empirical Evidence (Version 3.0) [Software]. www.wtgrantfoundation.org.21



### A **PPENDIX A**  **– Teacher**  **and	Principal Survey	Instrument** **s Spring	2015	School	Leader	Survey**  [[ 

### Version : On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2015 survey for school leaders about the feedback you give to teachers’ re garding their instruction as a part of the NM TEACH Educator Effectiveness System
. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {mo nth, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . Survey results will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “subm it” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have acc ess to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants.** At the same time that we are emailing you, REL Southwest is also inviting all New Mexico public school school

- based principals to participate in this study. Up to 10 teachers per school are also being surveyed

.22

**Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teache r evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will provide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are min imal. **Voluntary Participation.** Your participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a p articipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that y ou have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the informa tion and choose to participate. **Thank you very much for your help in this important study!**

--------------------- ------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button

— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions

— or23

if you accidentally get disconnected from the survey

— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning u ntil you click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous p age”). **Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

----------------------------------------**[NEW SURVEY PAGE]** 

----------------------------------------------- **Your	Position	and	Job	Responsibilities** 

### [q1] Which of the following best describes your position as a school leader this school year (2014

- 2015)?
 If you are not a school leader, you may have received thi s survey in error. Please reply to the invitation email or call {XXX}. 

##  principal 

##  vice principal/assistant principal 

##  director 

##  dean 

##  other: _______________________ 

### [q2x] Including this school year (2014

- 2015), how many years have you been a school administrator? (Fill in each box.)
 “School administrator” includes principal, assistant principal, and other equivalent titles. 

### [q2xa] Years as a school administrator in your current school 

### [q2xb] Years as a school administrator in your current district (including current school) 

### [q2xc] Years as a school administrator, total, in any location (including other districts and current district)24



### [q3] In addition to your administrative responsibilities, do you currently (2014

- 2015) have official teaching responsibilities (that is, formally instruct K

- 12 students as the teacher of record in one or more classes)?



##  Yes 

##  No 

### [q4] Do you play a role in filling teaching positions at your school (for example, recruiting, interviewing, and/or hiring)? 

##  Yes 

##  No

----- -----------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Perceptions	of	the	Quality of	Your	School’s	Teaching	Staff	This	Year**  

### [q 5 ] Indic ate your level of agreement with the following statement: “ **Overall** , I was satisfied with the quality of the whole teaching staff in my school this year (2014

- 2015).” [insert slider graphic here. tick marks for disagree strongly, disagree somewhat, agree somewhat, agree strongly] 

### [q 6 x] Use the slider to answer each it em below. During this school year (2014

- 2015), how many teachers in your school

... None or almost none A few About half Most All or nearly all 

### [q 6 a] Are highly effective instructional leaders. [insert slider graphic here] 

### [q 6b ] Are struggling and/or ineffective instructional leaders. [insert slider graphic here] 

### [q 6c ] Had a good grasp of the subject matter they teach [insert slider graphic here] 

### [q 6d ] Engaged in collaborative conversations about teaching and learning on a weekly or biweekly basis [insert slider graphic here]25

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Practices	When	Observing	Teachers	This	Year	(SY 2014**  **- 2015)	for	Evaluation**  **Purposes**
[q7] Thinking about the entire school year 2014

- 2015, please estimate the number of teachers you will formally evaluate yourself using the NM TEACH Observation Rubric {link}

. [enter number here that ranges from 0

– 50] [q8] Thinking about the entire school year 2014

- 2015, how many other school leaders besides yourself will formally evaluate teachers using the NM TEACH Observation Rubric {link} ? [enter number here that range s from 0

– 5] 

### [q 9 ] Thinking about the entire school year 2014

- 2015, how often did you personally **formally observe**  teachers in your school building using the NM TEACH Observation Rubric {link}? 

### [q 8 ] Thinking about the entire school year 2014

- 2015, how o ften did you personally **formally observe**  teachers in your school building using the New Mexico Observation Rubric {link}? 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teachers **3 or more times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teacher **2 times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) **some teachers 2 times and other teachers 3 times or more** 

. 

##  Other: ________________________________ 

### [q 10 ] Use the slider to estimate h ow long post

- observation conferences typically t ook you during school year 2014

- 2015
. Think only of conferences after formal observations; do not include i nformal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes to 60 minutes with 10 minute intervals] 

### [q 11 ] Thinking about the entire school year 2014

- 2015, how frequently did you provide **informal feedback** about tea ching and learning to individual teachers

— e.g. walk

- throughs, wows and wonders.26



##  At least once a week. 

##  At least once a month. 

##  At least once per semester. 

##  Other: ___________________ 

##  Optional comment here: 

### [q1 2 x] Thinking about **all** your formal observations from school year 2014

- 2015, use the slider to indicate h ow much you agree or disagree with the following statements about your post
formal observation conferences with teachers. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 2 a] The **feedback session itself**, separate from professional development I may have recommended , helped teachers **improve instruction** 
. [insert slider graphic here] 

### [q1 2b ] In each conference, the **teacher brought documents**  to the conference , documents such as the lesson plan, PDP. [insert slider graphic here] 

### [q1 2c ] In each conference, **I brought**  **documents** to conference such as the teacher’s report card and the most recent PDP. [insert slider graphic here] 

### [q1 2d ] In each conference, I identified **at least one positive practice**  that the teacher does **well**

. [insert slider graphic here] 

### [q1 2e ] In each conference, I identified **at least one**  **challenge** facing the teacher. [insert slider graphic here] 

### [ q12f] In ea ch conference, **I used my scores**  o n the NM TEACH Observation Rubric for that teacher to determine which instructional **practices** to praise or c ritique with the teacher. [insert slider graphic here] 

### [ q12g ] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine wh at **professional development**  to recommend to the teacher. [insert slider graphic here] 

### [q1 2h ] In each conference, I **ended** the [insert slider graphic here]27

conference on a **positive note**
. 

### [q1 2i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 2j ] In each conference, the teacher and I **mutually developed next steps** for their instruction. [insert slider graphic here] 

### [q1 2k ] In each conference provided **specific** feed back to teachers about their performance. [insert slider graphic here] 

### [q1 2l ] In each conference I provided **actionable** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2m ] In each conference, **teachers committed to**  **specific set of next steps**  to improve their instruction. [insert slider graphic here] 

### [q1 2n ] In each conference, I **spoke for almost all of the time**  during the conference. [insert slider graphic here] 

### [q1 2o ] There was a **high level of collaboration**  in feedback conferences. [insert slider graphic here] 

### [q1 2p ] There was a **high level of conflict** in feedback conferences. [insert slider graphic here] 

### [q1 2q ] It was **hard to provide negative**  **feedback** to teachers about their performance. [insert slider graphic here] 

### [q1 2r ] The large majority of teachers seemed to **trust and accept** my feedback. [insert slider graphic here] 

### [q1 2s ] **I feel positive** about the feedback I gave teachers in conferences. [insert slider graphic here] 

### [q1 2t ] **I enjoyed**  most of the post

- observation feedback conferences. [insert slider graphic here] 

### [q1 2u ] I provided all teachers I reviewed with a **written or online summary** of the observation with my comments on it

. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

----------------------------------28

**Professional	Development Offered	to**  **Teachers	This	Year	(2014 -**  **2015)** The questions in this section ask about professional development for teachers.
 Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, institutes, professional conferences, online courses, self

- directed learning, and so forth. 

### [q1 3 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the professional development experiences of teachers at your school this year (2014

- 2015, including summer 2014).
 Consider professional development offered by the district and other external sources as well as professional development offered by the school. The professional development options for teachers this year....
Don’t know Disagree

Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 3 a] Were mostly

developed by school leaders or teachers at my school. 

##  [insert slider graphic here] 

### [q1 3b ] Were mostly

job

- embedded rather than one

- time onl y sessions. 

##  [insert slider graphic here] 

### [q1 3c ] Were varied and numerous enough to allow me to customize my PD recommendations to teachers based on their performance on specific domains of the NM TEACH Observation Rubric.{link} 

##  [insert slider graphic here]


### [q1 3d ] Helped teachers improve on elements measured by the NM TEACH Observation Rubric. 

##  [insert slider graphic here] 

### [q1 3e ] Helped teachers improve other elements of the NM TEACH evaluation besides the NM TEACH Observation Rubric (e.g., student achievement or “other” multiple measure). 

##  [insert slider graphic here]


### [q 13f] Were well aligned with the Common Core State Standards and/or 

##  [insert slider graphic here]29

curriculum based on these standards. 

### [q1 3g ] Have been convenient enough to promote teachers’ participation. 

##  [insert slider graphic here] 

### [q1 3h ] Were sufficiently resource d (for example, substitute coverage, funding to cover expenses, stipends) to allow teachers to participate in the professional development they need to do their jobs successfully. 

##  [insert slider graphic here]


### [q1 3i ] District

- provided professional development offerings (workshops, inservices, online courses, etc.) have enough slots and availability to accommodate all teachers who wish or need to participate. 

##  [insert slider graphic here]


### [q1 3j ] Teachers have had easy access to a catalog of professional development opportunities aligned with the NM TEACH Observation Rubric. 

##  [insert slider graphic here]



### [q1 4x ] Thinking about the professional development that teachers at your school have participated in this year (20 14

- 2015, including summer 2014), use the slider to indicate how useful do think each of the following has been for improving teacher effectiveness

. Not Applicable Not at all

useful Slightly

useful Moderately

useful Very useful 

### [q1 4 a] Workshops or inservices for teachers at your school only (typically on

- site) 

##  [insert slider graphic here] 

### [q1 4 b] Workshops, inservices, institutes, or conferences organized by your district for teachers from multiple schools 

##  [insert slider graphic here] 

### [q1 4 c] Online professional development offered by or through your district 

##  [insert slider graphic here] 

### [q1 4 d] District

- or state
- sponsored induction program, mentoring, or 

##  [insert slider graphic here]30

other professional development activities for new or beginning teachers 

### [q1 4 e] Instructional coaching (provided by school

- based coaches or district coaches) 

##  [insert slider graphic here] 

### [q1 4f] Professional development led by me or a school leader such as study groups or s chool

- wide s taff meetings 

##  [insert slider graphic here] 

### [q1 4g ] School

- based teacher collaboration (grade

- level or subject

- area teams, professional learning communities, study groups, etc.) 

##  [insert slider graphic here] 

### [q1 4h ] Videos of sample lessons 

##  [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Training	You	Received	A bout	NM	TEACH	to	Date**  

### [q1 5 x] What types of training have you received to date regarding the NM TEACH Educator Effectiveness sys tem ? (Check all that apply.) 

##  [q1 5 a] [exclusive option]No professional development received 

##  [q1 5 b] Information on the NM TEACH Educator Effectiveness system (overall purpose, specific components, consequences for teachers, relationship to professional developm ent, and so forth) 

##  [q1 5 c] How to use the NM TEACH Observation Rubric 

##  [q1 5 d] How to provide feedback to individual teachers after formal classroom observations 

##  [q1 5 e] How to enter scores from NM TEACH Observation Rubric into Reflect , the Teachscape software 

##  [q1 5 f] How to use Reflect, the Teachscape software 

##  [q1 5 g] How to interpret a teachers’ NM TEACH Summative Evaluation Score 

##  [q1 5 h] How to give NM TEACH Observation Rubric ratings consistent with those given by other raters31



##  [q1 5 i] How to understand and make use of NM TEACH Summative Evaluation Score to help improve teaching at my school 

##  [q1 5j] How to direct teachers to professional development based on their NM TEACH Summative Evaluation Score 

##  [q1 5 k] Other: 

### [q1 6 ] [if 15b

- k ==yes] Who sponsored the training you attended about NM TEACH Educator Effectiveness system ? Mark as many as apply. 

##  State Public Education Department 

##  My school district 

##  Other: __________________ 

##  Don’t know 

### [q1 7 ] [if 15b

- k ==yes] In total, how many minutes of t raining did you receive this year (2014

- 2015, including summer 2014) about how to provide feedback to teachers

. 

##  0 minutes 

##  15 minutes 

##  30 minutes 

##  60 minutes 

##  120 minutes 

##  240 minutes 

##  Other: 

### [q1 8 ] [if 15d==yes] Thinking about the training on how to conduct post
- observation conferences with teachers , indicate how useful you think the training was. 

##  Not at all useful 

##  Slightly useful 

##  Moderately useful 

##  Very useful 

### [q1 9 ] I understand how the overall teacher ratings work in the NM teacher evaluation system (NM TEACH). 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree Strongly32

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q 20 x] Indicate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness System from the school year 20 14

- 2015. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 20 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider graphic here] 

### [q 20 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the students they teach. [insert slider graphic here] 

### [q 20 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance

. [insert slider graphic here] 

### [q 20 d] The NM TEACH Educator Effectiveness System requires so much of my time that it interferes with other important activities. [insert slider graphic here] 

### [q 20 e] I have a clear idea of what the NM TEACH Educator Effectiveness System views as “good instruction.” [insert slider graphic here] 

### [q 20f] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my teachers’ performance. [insert slider graphic here] 

### [q 20g ] In the NM TEACH Educator Effectiveness System , teachers who do not score “Effective” or better often leave teaching voluntarily. [insert slider graphic here]


### [q 21 x] Indicate how much you agree or disagree with each of the following statements about the NM TEACH Observation Rubric.33 Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 21 a] The 6 elements of Domain 1 (Planning and Preparation) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 b] The 5 elements of Domain 2 (Creating an Environment for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 c] The 5 elements of Domain 3 (Teaching for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ i nstructional practices. [insert slider graphic here] 

### [q 21 d] The 4 elements of Domain 4 (Professionalism) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 e] Using the NM TEACH Observation Rubric {link} helps me to provide constructive, specific feedback to teachers to improve their instruction. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Teacher Retention	and	Improvement** 

. 

### [q 22 x] Over the past year (since approximately May 1, 2014), how many teachers at your school... None (0 1 teacher 2 teachers 3 teachers More than Don’t34 teachers) 3 teachers know 

### [q22 a] Moved from effective to highly effective on the NM TEACH Summative Evaluation

. 

##       [q22b ] Were put on an **professional growth plan** or entered probationary

status 

##       [q22c ] Were **dismissed** (that is, had their district employment as a teacher terminated) due to receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      
[q22d] **Left teaching voluntarily** after receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. 

### [q2 3 ] Do you have any comments about the NM TEACH Observation Rubric?


### [q24] Do you have any comments about professional development available to teachers in your district?


### [q2 5 ] Do you have any comments about training either your district or NM PED have offered about the NM TEACH teacher evaluation system?35



### [q2 6 ] Do you have any feedback about this survey?

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

--------------------------- ------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thanks	Again** You have completed the survey and may now close it.36



### Spring	2016	School	Leader	Survey [[ 

### Version : On

- line administration. Wave 2.]] **Understanding	Your	Participation	i n	the	S**  **tudy** Welcome to the spring 2016 survey for school leaders about the feedback you give to teachers’ regarding their instruction as a part of the NM TEACH Educator Effectiveness System . This is a follow

- on to our fall 2015 survey to learn about the feedback you have provided to teachers this school year. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teache rs under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school prin cipals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers

. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers about their performance. Survey results will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only u se them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey re sponses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants.** REL Southwest originally invited all New Mexico public school school

- based principals to participa te in this study. Only those principals who completed surveys in spring 2015 are being sent this survey now in spring 2016. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher eval uation system. You will be providing valuable information37

that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** You r participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions

. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a parti cipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you h ave read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate. **Thank you very much for your help in this important study!**

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button

— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions

— or if you acciden tally get disconnected from the survey

— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until yo u click on the “submit” button on the final page of the survey.38

**Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”). **Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

----------------------------------------**[NEW SURVEY PAGE]**

-- --------------------------------------------- **Your	Position	and	Job	Responsibilities** We would like to update our information about your position. 

### [q1] Which of the following best describes your position as a school leader this school year (2015

- 2016)?
 I f you are not a school leader, you may have received this survey in error. Please reply to the invitation email or call {XXX}. 

##  principal 

##  vice principal/assistant principal 

##  director 

##  dean 

##  other: _______________________ 

### [q2x] Including this school year (2015

- 2016), how many years have you been a school administrator? (Fill in each box.)
 “School administrator” includes principal, assistant principal, and other equivalent titles. 

### [q2xa] Years as a school administrator in your current school 

### [q2xb] Years a s a school administrator in your current district (including current school) 

### [q2xc] Years as a school administrator, total, in any location (including other districts and current district) 

### [q3] In addition to your administrative responsibilities, do you currently (2015

- 2016) have official teaching responsibilities (that is, formally instruct K

- 12 students as the teacher of record in one or more classes)?39 

##  Yes 

##  No 

### [q4] Do you play a role in filling teaching positions at your school (for example, recruiting, interviewing, and/or hiring)? 

##  Yes 

##  No

----------------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Perceptions	of	the	Quality of	Your	School’s	Teaching	Staff	This	Year**  

### [q 5 ] Use the slider to indic ate your level of agreement with the following statement: “Overall, I was satisfied with the quality of the whole teaching staff in my school this year (2015

- 2016).” [insert slider graphic here. Tick marks at disagree strongly, disagree somewhat, agree so mewhat, agree strongly.] 

### [q 6 x] Use the slider to answer the following items. D uring this school year (2015

- 2016), how many teachers in your school

... None or almost none A few About half Most All or nearly all 

### [q 6 a] Are highly effective instructional leaders. [insert slider graphic here] 

### [q 6b ] Are struggling and/or ineffective instructional leaders. [insert slider graphic here] 

### [q 6c ] Had a good grasp of the subject matter they teach [insert slider graphic here] 

### [q 6d ] Engaged in collaborative conversations about teaching and learning on a weekly or biweekly basis [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

----------------------------------40

**Your	Practices	When	Observing	Teachers	This	Yea r	(SY	2015**  **- 2016)	for	Evaluation**  **Purposes** 

### [q 7 ] Thinking about the entire school year 2014

- 2015, please estimate the number of teachers you will formally evaluate yourself using the NM TEACH Observation Rubric {link}

. [enter number here that ranges from 0

– 50] 

### [q 8 ] Thinking about the entire school year 2014

- 2015, how many other school leaders besides yourself will formally evaluate teachers using the NM TEACH Observation Rubric {link} ? [enter number here that range s from 0

– 5] 

### [q 9 ] Thinking about the entire school year 2015

- 2016, how often did you personally **formally observe**  teachers in your school building using the NM TEACH Observation Rubric {link}? 

##  I or another school leader formally observed (for 15 + minute s using the NM TEACH Observation Rubric) all teachers **3 or more times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teacher **2 times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) **some teachers 2 times and other teachers 3 times or more**

. 

##  Other: ________________________________ 

### [q 10 ] Estimate h ow long post

- observation conferences typically t ook you during school year 2014

- 2015
. Think only of conferences after **formal observations** ; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes to 60 minutes with 10 minute intervals 

### [q 11 ] Thinking about the entire school year 2014

- 2015, how frequently did you provide **informal feedback** about teaching and learning to individual teachers

— e.g. walk

- throughs, wows and wonders. 

##  At least once a week. 

##  At least once a month. 

##  At least once per semester. 

##  Other: ___________________ 

##  Opti onal comment here:41



### [q1 2 x] Thinking about **all** your formal observations from school year 201 5

- 201 6 using the NM TEACH Observation Rubric , use the slider to indicate h ow much you agree or disagree with the following statements about your post
- formal observation conferences with teachers. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 2 a] The **feedback session itself**, separate from professional development I may have recommended , helped teachers **improve instruction** 
. [insert slider graphic here] 

### [q1 2b ] In each conference, the **teacher brought documents**  to the conference , documents such as the lesson plan, PDP. [insert slider graphic here] 

### [q1 2c ] In each conference, **I brought**  **documents** to conference such as the teacher’s report card and the most recent PDP. [insert slider graphic here] 

### [q1 2d ] In each conference, I identified at least one positive practice that the teacher does **well**

. [insert slider graphic here] 

### [q1 2e ] In each conference, I identified **at least one**  **challenge** facing the teacher. [insert slider graphic here] 

### [ q1 2f] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine which instructional **practices** to praise or c riticize with the teacher. [insert slider graphic here] 

### [ q1 2g ] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine wh at **professional development**  to recommend to the teacher. [insert slider graphic here] 

### [q1 2h ] In each conference, I **ended** the conference on a **positive note**
. [insert slider graphic here] 

### [q1 2i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 2j ] In each conference, the teacher and I **mutually developed next steps** for t heir instruction. [insert slider graphic here]42



### [q1 2k ] In each conference I provided **specific** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2l ] In each conference I provided **actionable** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2m ] In each conference, **teachers committed to**  **specific set of next steps** to improve their instruction. [insert slider graphic here] 

### [q1 2n ] In each conference, I **spoke for almost all of the time**  during the conference **.** [insert slider graphic here] 

### [q1 2o ] There was a **high level of collaboration**  in feedback conferences. [insert slider graphic here] 

### [q1 2p ] There was a **high level of conflict** in feedback conferences. [insert slider graphic here] 

### [q1 2q ] It was **hard to provide negative**  **feedback** to teachers about their performance. [insert slider graphic here] 

### [q1 2r ] The large majority of teachers seemed to **trust and accept** my feedback. [insert slider graphic here] 

### [q1 2s ] **I feel positive**  about the feedback I gave teachers in conferences. [insert slider graphic here] 

### [q1 2t ] **I enjoyed**  most of the post

- observation feedback conferences. [insert slider graphic here] 

### [q1 2u ] I provided all teachers I reviewed with a **written or online summary**  of the observation with my comments on it

. [insert slider graphic here]



### [q 13a ] Have you ever received the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” , a guide to conducting formal feedback sessions? The first page of the Conversation Checklist looks like this:43



##  Yes 

##  No 

### [q13b] Have you ever used the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” when conducting the formal feedback conference in the 2015

- 2016 school year ?



##  Yes 

##  No 

### [q 13c ] [If q 11b = yes)] With how many of your teachers, if any, have you used the
“ Conversation Checklist ”? 

##  None 

##  A few 

##  About half 

##  Most 

##  All or nearly all 

### [q13d ] [If treatment=1| (if treatment = 0 & q 13a = yes)] Did you view the testimonial video that was distributed along with the Conversation Checklist ? 

##  Yes44



##  No 

##  I don’t know 

### [ q13e ] Have you ever received the “ Guidance for New Mexico Principals’ About Provision of Feedback to Teachers ” document? It looks like this:



##  Yes 

##  No 

### [ q13f] If q 13e = yes)] With how many of your teachers, if any, have you used the advice included in the “Recommended Practices ” document ? 

##  None 

##  A few 

##  About half 

##  Most 

##  All or nearly all 

##  [q 13g ][If q 13c = none] Please select all of the reasons you have not used the Conversation Checklist

. 

##  [q 13g a] I have never seen the Conversation Checklist

. 

##  [q 13g b] I haven’t yet used the Conversation Checklist , but I plan to.45



##  [q 13g c] I got the Conversation Checklist too late for me to use in teacher conferences. 

##  [q 13g d] The Conversation Checklist looks burdensome

— e.g., too long, too hard. 

##  [q 13g e] I don’t need it; I already have a way of holding feedback conversations with teachers that works well. 

##  [q 13g f] I’ve read the Conversation Checklist and am using some concepts, but I’m not following the checklist. 

##  [q 13g g] Other: ______________________________________________________ 

### [q 13h ][If q13c ~= all & q13c ~= none] Which type of teachers did you use the
“ Conversation Checkl ist ” with? Select all that apply. 

##  [ q1 3 ha ] Only highly effective teachers 

##  [ q1 3 h b ] Only satisfactory teachers 

##  [ q1 3 h c ] Only struggling teachers 

##  [ q1 3 h d ] Only new teachers 

##  [ q1 3 h e ] Only experienced teachers 

##  [ q1 3 h h ] Other: _________________________________________________________


### [ q14a] [if 13C ~= none ] Use the slider to estimate h ow long post

- observation conferences typically t ook you during school year 2015

- 2016 when you used the “ Conversation Checklist. ” Think only of conferences after formal observations; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes, 15 min, 30 min., 45 min., 60 min, 75 min, 90 min] 

### [q 14b ] [if 

### q 13f~ = none ] Use th e slider to estimate h ow long post

- observation conferences typically t ook you during school year 2015

- 2016 after you reviewed the “ Guidance for New Mexico Principals’ About Provision of Feedback to Teachers ”

. Think only of conferences after formal observat ions; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes, 15 min, 30 min., 45 min., 60 min, 75 min, 90 min] 

### [q15x ] If 13C ~= none ] Use the slider to indicate h ow much you agree or disagree with the following statements about your opinions about the Conversation Checklist this year (school year 2015

- 2016).46 Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 5a ] The Conversation Checklist is **easy to use**  . [insert slider graphic here] 

### [q1 5b ] The Conversation Checklist **takes too**  **much time** to use. [insert slider graphic here] 

### [q1 5c ] The Conversation Checklist makes the feedback conversation feel **formulaic**

. [insert slider graphic here] 

### [q1 5d ] The Conversation Checklist provided me and teachers a **helpful structure** for feedback. [insert slider graphic here] 

### [q1 5e ] I provided **more critical feedback** to teachers than I had in prior conferences where I didn’t use the Conversation Checklist

. [insert slider graphic here] 

### [q1 5f] The Conversation Checklist helps teachers **commit to specific set of next steps**  to improve their instruction. [insert slider graphic here] 

### [q1 5g ] I use the Conversation Checklist as the basis to create teachers’ **professional development plans**  (PDPs). [insert slider graphic here] 

### [q1 5h ] I use the Conversation Checklist as the basis to create **professional growth plans** (PGPs) for struggling teachers. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]** 

---------------------------------- **Professional	Development Offered	to**  **Teachers	This	Year	(2015 -**  **2016)** The questions in this section ask about professional development for teachers.
 Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, institutes, professional conferences, online courses, self

- directed learning, and so forth. 

### [q1 6 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the professional development experiences of teachers at your school this year (2015

- 2016, including summer 2015).
 Consider professional development offered by the district and other external sources as well as p rofessional development offered by the school.47

The professional development options for teachers this year....
Don’t know Disagree

Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 6 a] Were mostly

developed by school leaders or teachers at my school. 

##  [insert slider graphic here] 

### [q1 6b ] Were mostly

job

- embedded rather than one

- time onl y sessions. 

##  [insert slider graphic here] 

### [q1 6c ] Were varied and numerous enough to allow me to customize my PD recommendations to teachers based on their performance on specific domains of the NM TEACH Observation Rubric.{link} 

##  [insert slider graphic here]


### [q1 6d ] Helped teachers improve on elements measured by the NM TEACH Observation Rubric. 

##  [insert slider graphic here] 

### [q1 6e ] Helped teachers improve other elements of the NM TEACH evaluation besides the NM TEACH Observation Rubric (e.g., student achievement or “other” multiple measure). 

##  [insert slider graphic here]


### [q1 6f] Were well aligned with the Common Core State Standards and/or curriculum based on these standards. 

##  [insert slider graphic here] 

### [q1 6g ] Have been convenient enough to promote teachers’ participation. 

##  [insert slider graphic here] 

### [q1 6h ] Were sufficiently resourced (for example, substitute coverage, funding to cover expenses, stipends) to allow teachers to participate in the professional development they need to do their jobs successfully. 

##  [insert slider graphic here]


### [q1 6i ] District

- provided professional development offerings (workshops, inservices, online courses, etc.) have enough slots and availability to 

##  [insert slider graphic here]48

accommodate all teachers who wish or need to participate. 

### [q1 6j ] Teachers have had easy access to a catalog of professional development opportunities aligned with the NM TEACH Observation Rubric. 

##  [insert slider graphic here]



### [q1 7x ] Thinking about the professional development that teachers at your school have participated in this year (2015

- 2016, including summer 2015), use the slider to indicate h ow useful you think each of the following has been for improving teacher effectiveness

. Not Applicable Not at all

useful Slightly

useful Moderately

useful Very useful 

### [q1 7 a] Workshops or inservices for teachers at your school only (typically on

- site) 

##  [insert slider graphic here] 

### [q1 7 b] Workshops, inservices, institutes, or conferences organized by your district for teachers from multiple schools 

##  [insert slider graphic here] 

### [q1 7 c] Online professional development offered by or through your district 

##  [insert slider graphic here] 

### [q1 7 d] District

- or state
- sponsored induction program, me ntoring, or other professional development activities for new or beginning teachers 

##  [insert slider graphic here] 

### [q1 7 e] Instructional coaching (provided by school

- based coaches or district coaches) 

##  [insert slider graphic here] 

### [q1 7f] Professional development led by me or a school leader such as study groups or s chool

- wide s taff meetings 

##  [insert slider graphic here] 

### [q1 7g ] School

- based teacher collaboration (grade

- level or subject

- area teams, professional 

##  [insert slider graphic here]49

learning communities, study groups, etc.) 

### [q1 7h ] Videos of sample lessons 

##  [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Training	You	Received	A bout	NM	TEACH	to	Date**  

### [q1 8 x] What types of professional development have you received to date regarding the teacher evaluation system, NM TEACH? (Check all that apply.) 

##  [q1 8 a] [exclusive option]No professional development received 

##  [q1 8 b] Information on the NM TEACH Educator Effectivenes s system (overall purpose, specific components, consequences for teachers, relationship to professional development, and so forth) 

##  [q1 8 c] How to use the NM TEACH Observation Rubric 

##  [q1 8 d] How to provide feedback to individual teachers after formal classroom o bservations 

##  [q1 8 e] How to enter scores from NM TEACH Observation Rubric into Reflect , the Teachscape software 

##  [q1 8 f] How to use Reflect, the Teachscape software 

##  [q1 8 g] How to interpret a teachers’ NM TEACH Summative Evaluation Score 

##  [q1 8 h] How to give NM TEACH Observation Rubric ratings consistent with those given by other raters 

##  [q1 8 i] How to understand and make use of NM TEACH Summative Evaluation Score to help improve teaching at my school 

##  [q1 8j] How to direct teachers to professional development based on their NM TEACH Summative Evaluation Score 

##  [q1 8 k] Other: 

### [q1 9 ] [if 18b

- k ==yes] Who sponsored the training you attended about NM TEACH Educator Effectiveness system ? Mark as many as apply. 

##  State Public Education Department 

##  My school district 

##  Other:

__________________ 

##  Don’t know 

### [q20 ] [if 18b

- k ==yes] In total, how many minutes of training did you receive this year (2014

-50

2015, including summer 2014) about how to provide feedback to teachers

. 

##  0 minutes 

##  15 minutes 

##  30 minutes 

##  60 minutes 

##  120 minutes 

##  2 40 minutes 

##  Other: 

### [q 21 ] [if 18d==yes] Thinking about the training on how to conduct post

- observation conferences with teachers , use the slider to indicate how useful you think the training was. Not at all useful Slightly useful Moderately useful Very useful [insert slider graphic here] 

### [q 22 x] Use the slider to indicate how much you agree or disagree with each of the following statements about your understanding of NM TEACH. Disagree strongly Disagree

somewhat Agree

somewhat Agree

strongly 

### [q 22 a] I understand how the overall teacher ratings work in the NM teacher evaluation system (NM TEACH). [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal Views	of	NM	TEACH**  

### [q 23 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the teacher effectiveness ratings that became available in school year 2015

- 2016. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 23 a] The NM TEACH Educator Effectiveness System does a good job distinguishing [insert slider graphic here]51

effective from ineffective teachers. 

### [q 23 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the students they teach. [insert slider graphic here] 

### [q 23 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance. [insert slider graphic here] 

### [q 23 d] The NM TEACH Educat or Effectiveness System requires so much of my time that it interferes with other important activities. [insert slider graphic here] 

### [q 23 e] I have a clear idea of what the NM TEACH Educator Effectiveness System views as “good instruction.” [insert slider graphic here] 

### [q 23 g] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my teachers’ performance. [insert slider graphic here] 

### [q 23 h] In the current NM TEACH Educator Effectiveness System , teachers who do poorly on their evaluation often leave teaching voluntarily. [insert slider graphic here]


### [q 24 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Observation Rubric. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q2 4 a] The 6 elements of Domain 1 (Planning and Preparation) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 b] The 5 elements of Domain 2 (Creating an Environment for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 c] The 5 elements of Domain 3 (Teaching [insert slider graphic here]52

for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. 

### [q2 4 d] The 4 elements of Domain 4 (Professionalism) of the NM TEACH Observation Rubric {link}are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 e] Using the NM TEACH Observation Rubric {link} helps me to provide constructive, specific feedback to tea chers to improve their instruction. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Teacher Retention	and	Improvement**  

### [q2 5 x] Over the past year (since approximately May 1, 2015), how many teachers at your school...
None (0 teachers) 1

teacher 2

teachers 3

teachers More

than 3 teachers Don’t

know 

### [q25 a] Moved from effective to highl y effective on the NM TEACH Summative Evaluation

. 

##       [q2 5b ] Were put on an professional growth plan or entered probationary

status 

##       [q2 5c ] Were **dismissed** (that is, had their district employment as a teacher terminated) due to receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      
[q2 5d] Left teaching voluntarily after receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      53

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Optional Comments	from	You**  The following four questions are optional if you have any feedback you wish to share. 

### [q2 6 ] [if treatment = 1] Do you have any suggestions to help us improve the Principal

Teacher Post

- Observation Conversation Checklist ? 

### [q27] Do you have any comments about professional development available to teachers in your district? 

### [q2 8 ] Do you have any comments about the NM TEACH Observation Rubric? 

### [q2 9 ] Do you have any comments about training either your district or NM PED have offer ed about the NM TEACH teacher evaluation system?

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

---------------------------------------**[NEW SURVEY PAGE]**

---------------------------------------------54

**Thanks	Again** You have completed the survey and may now close it.55



### Spring	2015	Teacher	Survey [[ 

### Version : Teachers. On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2015 survey for school leaders about the feedback you give to teachers’ regarding their ins truction as a part of the NM TEACH Educator Effectiveness System

. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System , and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {mo nth, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness System training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researcher s will link your survey responses to basic employment information, such as race, ethnicity and years of experience that REL Southwest researchers will obtain from NM PED. Survey results will **not** be used to evaluate you, other teachers, or school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey li nk provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you o r your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants** Ineach o f the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study Up to 10 teachers in your school were selected at random to be surveyed

.56

**Benefits.** Your completion of this survey gives you the opportuni ty to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will provide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a p articipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that y ou have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate.. **Thank you very much for your help in this important study!**

--------------------- ------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” b utton at the **bottom** of the page. Do **not** use your browser’s back button

— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions

— or if you accidentally get disconnected from the survey

— you can return t o your previous57

responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until you click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”). **Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

----------------------------------------**[INTERNAL CODES]**

----------------------------------------------- [[ Treatment code (treatment): REL Southwest will pre

- identify via an internal code that will not be listed on the survey whether teacher respondent works at a school that belongs to the treatment group or th e control group. treatment = 1 indicates teacher is in the treatment group. treatment = 0 = control group. Unless otherwise noted, survey questions will appear for both groups of teachers.]]

----------------------------------------**[NEW SURVEY PAGE]**

------ ----------------------------------------- **Your	Position	and	Job	Responsibilities** 

### [required] [q1] Which of the following best describes your current position as a teacher of K

12 students? (Check one.)
 If you are not a teacher of K

- 12 students, or do not h ave teaching responsibilities in 2014

- 2015, you have received this survey in error. Please let us know by replying to the invitation email or by calling {number}. 

##  Regular education teacher 

##  Special education teacher 

##  Other kind of teacher: _________________

__________________ 

### [q2x] What subject area(s) are you currently teaching ? (Check all that apply.) 

##  [q2a] English/language arts/reading/writing 

##  [q2b] Mathematics 

##  [q2d] Science 

##  [q2d] Social Studies58



##  [q2e] Foreign language(s) 

##  [q2f] Visual or performing arts (art, music, etc.) 

##  [q2g] Physical education/health education 

##  [q2h] Career/technical education 

##  [q2i] Other subject area(s): [include fill

- in box] 

### [q3] Which one of the following three groups of teachers do you belong to? These gr oups are defined by New Mexico’s teacher evaluation system, NM TEACH. 

##  Group A : tested subject, tested grade 

##  Group B : grade 3
– 12 subject not linkable to NM’s standard based assessments 

##  Group C : grade K

- 2 teacher 

### [q4] Which of the following best describes your current teaching arrangement? (Check one.) 

##  I teach a single group of students all or most of the day in multiple subject areas. (Traditional elementary arrangement; sometimes called “self

- contained.”) 

##  I teach several classes of different st udents during the day in a particular subject (for example, math) or perhaps two subjects (for example, you teach some math classes and some science classes). (Traditional secondary arrangement; sometimes called “subject

specific” or “departmentalized”; at elementary level, sometimes called “subject matter specialist.” Also typical arrangement for physical education, art, music, etc.) 

##  I mainly teach selected students released from (or in) their regular classes in specific skills or to address specific needs (for example, special education, reading, English as a second language, gifted and talented.) (Sometimes called “pull out”, “resource,” or “push in” instruction.) 

##  I am one of two or more teachers who are jointly responsible for teaching the same subject(s ) to a group of students (for example, in the same classroom), all or most of the time and/or in a majority of classes. (Sometimes called “co

- teaching” or “job share.”) 

##  Other (please describe): _______________________________________ 

### [q5] Including this s chool year (2014

- 2015), how many years have you been teaching, total, regardless of location? Total years teaching: [fill in number: must be whole number less than 60] 

### [q6] Including this school year (2014

- 2015), how many years have you been teaching in your current school ?59

Years in current school: [fill in number: must be whole number less than 60] 

### [q7] Which of the following best describes your current position as a teacher of K

- 12 students? (Check all that apply ) 

##  Spanish bilingual teacher 

##  Native American bilingual teacher 

##  Other bilingual teacher: ___________________________________ 

##  Not a bilingual teacher

----------------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Experiences	This	Year	(SY	2014 -**  **2015)	of	Formal	Observations	for	Evaluation Purposes**  

### [ * required question] [q8] Thinking back over this whole school year 2014

- 2015, how many times were you **formally observed** by a school leader? School leader is defined as a principal, assistant principal, dean, or coach charged with authority to conduct formal observations of 20+ minutes using the NM TEACH Observation Rubric {link to rubric}. 

##  3 or more t imes 

##  2 times 

##  1 time 

##  0 times 

##  Other: ________________________________ 

### [q9] [If q8 = 1, 2, or 3+ times ] Use the slider to estimate h ow long the post

- observa tion conferences typically took. Think only of conferences after formal observations; do not include informal feedback you might have received from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 min, 15 min, 30 min, 45 min, 60 min, 75 min, 90 min.] 

### [q10] How fr equently did you receive **informal feedback** this year from a school leader

— for example, “ wow or wonder” statements after brief walk

- throughs. 

##  At least once a week60



##  At least once a month 

##  At least once per semester or trimester 

##  Never 

##  Other:

___________________ 

### [q11x] [If q8 = 1, 2, or 3+ times] Use the slider to indicate the extent to which you agree or disagree with the following statements about all of the conferences you had with a school leader following his or her **formal observation(s)** of your class this year (school year 2014

- 2015)

.
Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q11a] The **feedback session itself**, **separate** from professional development I may have been recommended to take , helped **me improve**  my instruction. 

### [q1 1b ] In each conference, I **brought documents**  to the conference , documents such as the lesson plan, and PDP. [ insert slider here] 

### [q1 1c ] In each conference, **the school leader brought documents**  to conference , such as my summative score and the most recent PDP. [ insert slider here]


### [q11 d ] In each conference the school leader identified **at least one positive practice** that I do well. [ insert slider here] 

### [q11 e ] In e ach conference the school leader identified **at least one**  **challenge** facing me. [ insert slider here] 

### [q11 f] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine which **instructional practices**  to praise or c ritique
. [ insert slider here] 

### [q11 g ] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine what **professional development** to recommend to the teacher. [ insert slider here]61



### [q1 1h ] In each conference, the school leader **ended** the conference on a **positive note** 
. [ insert slider here] 

### [q1 1i ] Each conference followed a **predictable format** 

. [ insert slider here] 

### [q11j ] In each conference , y school leader and I **mutually developed next steps** for my instruction. [ insert slider here]


### [q11 k ] I walked away with a **clear understanding**  of my school leader’s feedback. [ insert slider here] 

### [q11 l ] My school leader **listened** to what I had to say during the conferences. [insert slider here] 

### [q11 m ] The observations were done **to me,**  **rather than for**  **me** . [insert slider here] 

### [q11 n ] I received **specific** feedback about my performance. [insert slider here] 

### [q11 o ] I received **actionable** feedback about my performance. [insert slider here] 

### [q11 p ] During the conferences, I **committed to**  **a specific set of next steps**  to improve my instruction. [insert slider here] 

### [q11 q ] ] In each conference, the school leader **spoke for almost all of the time** 

. [insert slider here] 

### [q11 r ] There was a **high level of conflict** in the conference. [insert slider here] 

### [q11 s ] I n each conference I obtained recommendations for **professional development**  **tailored** to my specific needs. [insert slider here] 

### [q11 t ] I **trusted and accepted** the feedback I received. [insert slider here] 

### [q11 u ] **I feel positive** about the feedback I got during the conferences. [insert slider here] 

### [q1 1v ] **I enjoyed**  most of the post

observation feedback conferences. 

### [q1 1w ] I was always provided with a **written or online summary**  of the observation and my [insert slider graphic here]62

school leader’s comments on it. [ 

### q12x ] Please indicate whether you were ever rated this school year b y your school leader as needing to improve upon or else excelling in the following observation items from NM TEACH, whether you received a recommendation to take up professional development (PD), and how you rate yourself on the item. Mark all that apply. NM TEACH Observation Item I was rated at

least once as excelling on this item I was rated at

least once as needing to improve on this item I was

recommended to take PD because of this item I don’t

know how I was rated How would

yo u rate yo urself this school ye ar on this item? 2A: Creating an environment of respect and rapport [Ineffective, Minimally Effective, Effective, Highl y Effective, Exemplary I don’t know] 2B: Organizing Physical Space 2C: Establishing a culture for learning 2D: Managing classroom procedures 2E: Managing Student Behavior 3A: Communicating with Students 3B: Using questioning and discussion techniques 3C: Engaging students in learning 3D: Assessment in63

Instruction 3E: Demonstrating flexibility and responsiveness

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development	You	Received	This	Year	(SY	2014 -**  **2015)** The questions on this page ask about all of the professional development you have participated in during the 2014

- 2015 school year, including summer 2014. 

##  Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, courses, institutes, professional conferences, self

- directed learning, and so forth. 

##  Consider mandatory/required professional development as well as voluntary professional development. 

### [q13x] Please indicate all of the topics below that one or more school leader recommend you take. In addition, please also include which topics were covered in professional development you actual ly participated in this year (2014

- 2015, including summer 2014) , and topics that you would have liked to have participated in professional development, but it was not available to you . Select as many topics as apply.
A	school leader recommen ded	I	take this	PD I	took	the

PD	online I	received

the	PD	in person have	liked

to	take	the


### [q13a] Classroom management 

##    [q13b] Professionalism and communication in the classroom 

##    [q13c] Special education and inclusion 

##    [q13d] Standards

- based lesson plans 

##    [q13e] Common Core Standards 

##    [q13f] Assessment (includes continuous assessment, portfolio assessment, Common Core aligned assessments) 

##    [q13g] Data

- driven instruction 

##    [q13h] Early childhood 

##   64



### [q13i] Instructional strategies or content in the content area(s) you teach

— e.g., Mathematics; or Literacy PD 

##    [q13j] Response to Intervention 

##    [q13k] Teachscape Reflect training 

##    [q13l] Other NM TEACH training 

##    [q13m ] Sheltering Instruction for Language development 

##    [q13n ] Language Acquisition Design 

##    [q13o ] Scaffolding instruction 

##    [q13 p ] Other: _______________________________ 

##    [q13 q ] Other:

_______________________________ 

##    [q13 r ] Other: _______________________________ 

##    [q1 4 x] Thinking about the professional development you have participated in this year (2014

- 2015, including summer 2014), use the slider to indicate how much you agree or disagree with each of the following statements

. My professional development experiences this year... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4 a] Were mostly

developed by school leaders or teachers at my

school. [insert slider here] 

### [q1 4b ] Were mostly

job

- embedded rather than one

- time only sessions. [insert slider here] 

### [q1 4 a] Were varied and numerous enough to allow the school leader to customize PD recommendations based on my performance on specific domains of the NM TEACH Observation Rubric.{link} [insert slider here]



### [q1 4 b] Have been useful for helping me improve the achievement of my students [insert slider here] 

### [q1 4 c] Have been useful for improving my instruction [insert slider here] 

### [q1 4 d] Have taken more time than they were worth [insert slider here] 

### [q1 4 e] Have been well aligned with the [insert slider here]65

Common Core State Standards and/or curriculum based on these standards 

### [q1 5 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the support you have received related to your professional development experiences this year (SY 2014

- 2015, including summer 2014). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 5 a] I had easy access to a list of the possible professional development courses or offerings I could take. [insert slider graphic here]


### [q1 5c ] My school leader(s) has encouraged and supported my participation in professional development. [insert slider here] 

### [q1 5d ] Sufficient resources (for example, substitute coverage, funding to cover expenses, stipends) have been available to allow me to participate in the professional development I need to teach effectively. [insert slider here]



### [q1 5e ] I have had sufficient flexibility in my schedule to pursue the professional development opportunities recommended for me. [insert slider here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

------------- --------------------- **Your	Perceptions	of	the	Quality of	Your	School’s	Leadership	Last	Year**  

### [q1 6 ] Rate your level of agreement with the following statement: “Overall, I was satisfied with the quality of my school’s leaders (principals, assistant principal s, deans, department chairs) this year (school year 2014

- 2015).” 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree strongly66



### [q1 7 x] Thinking about this year (2014

- 2015), use the slider to indicate how much you agree or disagree with the following statements. “The school leaders who conduct formal observations of my classroom ... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 7 a] Understand the subject matter content that I teach. [insert slider here] 

### [q1 7b ] Understand what skills are needed for teachers to foster meaningful student learning. [insert slider here] 

### [q1 7c ] Foster an atmosphere of trust and mutual respect in my school. [insert slider here] 

### [q1 7d ] Understand the 3 rating components of the NM TEACH Summative Evaluation Report (student achievement, observations, multiple measures). [insert slider here]



### [q1 7e ] Help to improve my teaching. [insert slider here] 

### [q1 7f] Are instructional leaders. [insert slider here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q1 8 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness S ys tem from school year 2014

- 2015. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 8 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider here]



### [q1 8 b] The NM TEACH Educator Effectiveness System is fair to all teachers, [insert slider here]67

regardless of their personal characteristics or those of the students they teach. 

### [q 18 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance. [insert slider here] 

### [q1 8d ] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my performance. [insert sl ider here]



### [q18e ] I understand how the overall teacher ratings work in the NM TEACH Educator Effectiveness System
. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your NM	TEACH	Summative	Evaluation	Score**  

### [q19 ] Do you know what your level of Educator Effectiveness (Ineffective, Minimally Effective, Effective, Highly Effective, Exemplary) was for the school year 201 4

- 201 5 ? 

##  Yes 

##  No 

### [q20] [if q19=yes] How did you learn your level of Educator Effectiveness for the school year 201 4

- 201 5 ? Select all that apply. 

##  I received it via email from New Mexico Public Education Department 

##  I logged into my teacher portal online 

##  A school leader told me 

##  I received it via my mailbox at school 

##  Other: ______________________ 

##  Other: ______________________ 

##  I don’t recall 

### [q21] [if q19=yes] Did a school leader ever discuss with you how to improve your Educator Effectiveness score ?68



##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  I don’t recall 

### [q22] [if q19=yes] Did a school leader ever discuss with you how to interpret your student achievement growth score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  Not Applicable 

##  I don’t recall

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. If you do not, proceed to the next and f inal page. 

### [q2 3 ] Do you have any comments about the NM TEACH Observation Rubric? 

### [q24] Do you have any comments about professional development available to teachers in your district? 

### [q2 5 ] Do you have any comments about the NM TEACH Educator Effectiveness System ? 

### [q2 6 ] Do you have any feedback about this survey?

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!**69

Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thanks	Again** You have completed the survey and may now close it.70



### Spring	2016	Teacher	Survey [[ 

### Version : Teachers. On

- line administration. Wave 2.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2016 survey for teachers about feedback you receive about your inst ruction as a part of the NM TEACH Educator Effectiveness System . This is a follow

- on to our spring 2015 survey to learn about the feedback you received from your school leaders last year. **Study Purpose.** With funding from the federal Department of Educatio n as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Ed ucation Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers about their performance. REL Southwest researchers will link your survey responses to basic employment information, such as race, ethnicity and years of experience that REL Southwest researchers will obtain from NM PED. Survey r esults will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, on ce you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other informatio n that could potentially be used to identify individuals or schools. **Participants.** Ineach of the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study . Up to 10 teachers in your school were selected at random to be surveyed. Only those teachers who completed surveys in spring 2015 are being sent this survey

now in spring 2016.71

**Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teache r evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$25 online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** You r participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a parti cipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you h ave read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate. **Thank you very much for your help in this important study!**

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button

— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions

— or if you acciden tally get disconnected from the survey

— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until yo u click on72

the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”). **Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

----------------------------------------**[INTERNAL CODES]**

--- -------------------------------------------- Treatment code (treatment): REL Southwest will pre

- identify via an internal code that will not be listed on the survey whether teacher respondent works at a school that belongs to the treatment group or the con trol group. treatment = 1 indicates teacher is in the treatment group. treatment = 0 = control group. Unless otherwise noted, survey questions will appear for both groups of teachers. Prior respondent code: REL Southwest will pre

- identify those respondent s who, from the same email address, completed the corresponding item in wave 1 spring 2015 survey. For these respondents, prior = 1.

----------------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Position	an d	Job	Responsibilities**  We would like to update our information about your position. 

### [q1] Which of the following best describes your current position as a teacher of K

- 12 students ? (Check one.)
 If you are not a teacher of K

- 12 students, or do not have teaching responsibilities in 2015

- 2016, you have received this survey in error. Please let us know by replying to the invitation email or by calling {number}. 

##  Regular education teacher 

##  Special education teacher 

##  Other kind of teacher:

___________________________________ 

### [q2x] What subject area(s) are you currently teaching? (Check all that apply.) 

##  [q2a] English/language arts/reading/writing73



##  [q2b] Mathematics 

##  [q2d] Science 

##  [q2d] Social Studies 

##  [q2e] Foreign language(s) 

##  [q2f] Visual or p erforming arts (art, music, etc.) 

##  [q2g] Physical education/health education 

##  [q2h] Career/technical education 

##  [q2i] Other subject area(s): [include fill

- in box] 

### [q3] Which one of the following three groups of teachers do you belong to? These groups are def ined by New Mexico’s teacher evaluation system, NM TEACH. 

##  Group A : tested subject, tested grade 

##  Group B : grade 3
– 12 subject not linkable to NM’s standard based assessments 

##  Group C : grade K

- 2 teacher 

### [q4] Which of the following best describes your curren t teaching arrangement? (Check one.) 

##  I teach a single group of students all or most of the day in multiple subject areas. (Traditional elementary arrangement; sometimes called “self

- contained.”) 

##  I teach several classes of different students during the day in a particular subject (for example, math) or perhaps two subjects (for example, you teach some math classes and some science classes). (Traditional secondary arrangement; sometimes called “subject

specific” or “departmentalized”; at elementary level, som etimes called “subject matter specialist.” Also typical arrangement for physical education, art, music, etc.) 

##  I mainly teach selected students released from (or in) their regular classes in specific skills or to address specific needs (for example, special education, reading, English as a second language, gifted and talented.) (Sometimes called “pull out”, “resource,” or “push in” instruction.) 

##  I am one of two or more teachers who are jointly responsible for teaching the same subject(s) to a group of studen ts (for example, in the same classroom), all or most of the time and/or in a majority of classes. (Sometimes called “co

- teaching” or “job share.”) 

##  Other (please describe): _______________________________________ 

### [q5] [if prior q5 blank] Including this sch ool year (2015

- 2016), how many years have you been teaching, total, regardless of location? Total years teaching: [fill in number: must be whole number less than 60]74



### [q6] [if prior q6 blank] Including this school year (2015

- 2016), how many years have you been teaching in your current school ?

Years in current school: [fill in number: must be whole number less than 60] 

### [q7] Which of the following best describes your current position as a teacher of K

- 12 students? (Check all that apply ) 

##  Spanish bilingual teacher 

##  Native American bilingual teacher 

##  Other bilingual teacher: ___________________________________ 

##  Not a bilingual teacher

----------------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Experiences	This	Year	(SY	2015 -**  **2016)	of	Formal	Observations	for	Evaluation Purposes**  

### [ * required question ] [q8] Thinking back over this whole school year 2015

- 2016, how many times were you **formally observed** by a school leader? School leader is defined as a principal, assistant principal, dean, or coach charged with authority to conduct formal observations of 20+ minutes using the NM TEACH Observation Rubric {link to rubric}. 

##  3 or more times 

##  2 times 

##  1 time 

##  0 times 

##  [q9] [If q8 = 1, 2, or 3+ times ] Use the slider to estimate h ow long the formal post
observ ation conferences typically took ? Think only of conferences after formal observations; do not include informal feedback you might hav e received from classroom walk

- throughs. [insert slider graphic here. tick marks for 0 min, 15 min, 30 min, 45 min, 60 min, 75 min, 90 min]75



### [q10] How frequently did you receive **informal feedback** this year from a school leader

— for example, “ wow or wonder” statements after brief walk

- throughs. 

##  At least once a week 

##  At least once a month 

##  At least once per semester or trimester 

##  Never 

##  Other: ___________________ 

### [q1 1 x] [If q8 = 1, 2, or 3+ times] Use the slider to indicate h ow much you agree or disagree with the following statements about all of the conferences you had with a school leader following his or her **formal observation(s)** of your class this year (school year 2015

- 2016). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 1 a] The **feedback session itself**, **separate** from professional development I may have been recommended to take , **helped me**  **improve** my instruction. [insert slider graphic here] 

### [q1 1b ] In each conference, I **brought documents**  to the conference , documents such as the lesson plan, and PDP. [ insert slider here] 

### [q1 1c ] In each conference, **the school leader brought documents**  to conference , such as my summative score and the most recent PDP. [ insert slider here]


### [q1 1d ] In each conference the school leader identified **at least one positive practice** that I do well. [insert slider graphic here] 

### [q1 1e ] In e ach conference the school leader identified **at least one challenge**  facing me. [insert slider graphic here] 

### [q11 f] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine which **instructional practices**  to praise or c ritique
. [insert slider graphic here] 

### [q1 1g ] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine what [insert slider graphic here]76

**professional development** to recommend to the teacher. 

### [q1 1h ] In each conference, the school leader **ended** the conference on a **positive note**
. [insert slider graphic here] 

### [q1 1i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 1j ] In each conference, m y school leader and I **mutually developed next steps** for my instruction. [insert slider graphic here]

. 

### [q1 1k ] I walked away with a **clear understanding**  of my school leader’s feedback. [insert slider graphic here] 

### [q1 1l ] My school leader **listened** to what I had to say during the conferences. [insert slider graphic here] 

### [q1 1m ] The observations were done **to me,**  **rather than for**  **me** . [insert slider graphic here] 

### [q1 1n ] I received **specific** feedback about my performance. [insert slider graphic here] 

### [q1 1o ] I received **actionable** feedback about my performance. [insert slider graphic here] 

### [q1 1p ] During the conferences, I **committed to**  **a specific set of next steps**  to improve my instruction. [insert slider graphic here] 

### [q11 q ] ] In each conference, the school leader **spoke for almost all of the time** 

. [insert slider here] 

### [q1 1r ] There was a **high level of conflict** in the conference. [insert slider graphic here] 

### [q1 1s I n each conference I obtained recommendations for **professional development**  **tailored** to my specific needs. [insert slider graphic here] 

### [q1 1t ] I **trusted and accepted** the feedback I received. [insert slider graphic here] 

### [q1 1u ] **I feel positive** about the feedback I got during the conferences. [insert slider graphic here]77



### [q1 1v ] **I enjoyed**  most of the post

observation feedback conferences. [insert slider graphic here] 

### [q1 1w ] I was always provided with a **written or online summary**  of the observation and my school leader’s comments on it. [insert slider graphic here]

[ 

### q12x ] Please indicate whether you were ever rated this school year b y your school leader as needing to improve upon or else excelling in the following observation items from NM TEACH, whether you received a recommendation to take up professional development (PD), and how you rate yourself on the item. Mark all that apply. NM TEACH Observation Item I was rated at

least once as excelling on this item I was rated at

least once as needing to improve on this item I was

recommended to take PD because of this item I don’t

know how I was rated How would

yo u rate yo urself this school ye ar on this item? 2A: Creating an environment of respect and rapport [Ineffective, Minimally Effective, Effective, Highl y Effective, Exemplary I don’t know] 2B: Organizing Physical Space 2C: Establishing a culture for learning 2D: Managing classroom procedures 2E: Managing Student Behavior 3A: Communicating with Students 3B: Using questioning78

and discussion techniques 3C: Engaging students in learning 3D: Assessment in Instruction 3E: Demonstrating flexibility and responsiveness 

### [q 13a ][if treatment=1 & q8 > 0] Have you ever received the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” , a guide to conducting formal feedback sessions? The first page of the Conversation Checklist looks like this:



##  Yes 

##  No 

### [q13b][if q8 > 0] Has your school leader ever used the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” when conducting the formal feedback conference in the 2015

- 2016 school year ? 

##  Yes 

##  No79



### [q 13c ] [If q 13b = yes)] For how many of your feedback sessions did your school leader(s) use the “ Conversation Checklist ”? 

##  3 or more observations 

##  2 observations 

##  1 observation 

##  None of the observations 

##  I don’t know 

### [q13d [If 13a = yes)] Did you view the testimonial video that was distributed along with the Conversation Checklis t ? 

##  Yes 

##  No 

##  I don’t know 

### [q13e][ ] During the formal feedback sessions, h as a school leader ever used the “ Guidance for New Mexico Principals’ About Provision of Feedback to Teachers ” to guide any of your formal feedback sessions during the 2015

- 2016 school year? The first page of the Conversation Checklist looks like this:



##  Yes 

##  No80



### [q14x ] If 13b=yes ] Use the slider to indicate h ow much you agree or disagree with the following statements about your opinions about the Conversation Checklist this year (school year 2015

- 2016). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4a ] The Conversation Checklist was **easy to use** 
. [insert slider graphic here] 

### [q1 4b ] The Conversation Checklist **took too**  **much time** to use. [insert slider graphic here] 

### [q1 4c ] The Conversation Checklist made the feedback conversation feel **formulaic**

. [insert slider graphic here] 

### [q1 4d ] The Conversation Checklist provided me and my school leader(s) a **helpful structure**  for feedback. [insert slider graphic here] 

### [q1 4e ] I received **more critical feedback** when my school leader used the Conversation Checklist than in prior conferences when he/she didn’t use the Protocol. [insert slider graphic here] 

### [q1 4f] The Conversation Checklist helped me **commit to specific set of next steps**  to improve my instruction. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development	You	Received	This	Year	(SY	2015 -**  **2016)** The questions on this page ask about all of the professional development (PD) you have participated in during the 2015

- 2016 school year

. **Don’t include any professional development you may have participated in prior to your first formal observation this school year.**  

##  Professional development includes (but is not limited to) workshops, inservi ces, coaching/mentoring, collaborative planning/study, courses, institutes, professional conferences, self

- directed learning, and so forth. 

##  Consider mandatory/required professional development as well as voluntary professional development. 

### [q1 5 x] Please i ndicate all of the topics below that one or more school leader recommend you81

take. In addition, please also include which topics were covered in professional development you actually participated in this year (2015

- 2016). Select as many topics as apply.
A	school leader recommen ded	I	take this	PD I	took	the

PD	online I	received

the	PD	in person have	liked

to	take	the


### [q1 5 a] Classroom management 

##    [q1 5 b] Professionalism and communication in the classroom 

##    [q1 5 c] Special education and inclusion 

##    [q1 5 d] Standards

- based lesson plans 

##    [q1 5 e] Common Core Standards 

##    [q1 5 f] Assessment (includes continuous assessment, portfolio assessment, Common Core aligned assessments) 

##    [q1 5 g] Data

- driven instruction 

##    [q1 5 h] Early childhood 

##    [q1 5 i] Instructional strategies or content in the content area(s) you teach

— e.g., Mathematics; or Literacy PD 

##    [q1 5j] Response to Intervention 

##    [q1 5 k] Teachscape Reflect training 

##    [q1 5 l] Other NM TEACH training 

##    [q15m ] Sheltering Instruction for Language development 

##    [q15n ] Language Acquisition Design 

##    [q15o ] Scaffolding instruction 

##    [q1 5 m] Other: _______________________________ 

##    [q1 5 n] Other: _______________________________ 

##    [q1 5 o] Other: _______________________________ 

##   
[q1 6 x] Thinking about the professional development you have participated in this year (2015

- 2016, including summer 2015), use the slider to indicate how much you agree or disagree with each of the following statements

.82

My professional development experiences this year... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4 a] Were mostly

developed by school leaders or teachers at my

school. [insert slider graphic here] 

### [q1 4b ] Were mostly

job

- embedded rather than one

- time only sessions. [insert slider graphic here] 

### [q1 6 a] Were varied and numerous enough to allow the school leader to customize PD recommendations based on my performance on specific domains of the NM TEACH Observation Rubric.{link} [insert slider graphic here] 

### [q1 6 b] Have been useful for helping me improve the achievement of my students [insert slider graphic here] 

### [q1 6 c] Have been useful for improving my instruction [insert slider graphic here] 

### [q1 6 d] Have taken more time than they were worth [insert slider graphic here] 

### [q1 6 e] Have been well aligned with the Common Core State Standards and/or curriculum based on these standards [insert slider graphic here] 

### [q1 7 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the **support** you have received related to your professional development experiences this year (SY 2015

- 2016, including summer 2015). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 7 a] I had easy access to a list of the possible professional development courses or offerings I could take. [insert slider graphic here] 

### [q1 7 b] My school leader(s) has encouraged and supported my participation in professional development. [insert slider graphic here] 

### [q1 7 c] Sufficient resources (for example, substitute coverage, funding to cover expenses, stipends) have been available to allow me to participate in the professional [insert slider graphic here]83

development I need to teach effectively. 

### [q1 7 d] I have had sufficie nt flexibility in my schedule to pursue the professional development opportunities recommended for me. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Perceptions of	the	Quality**  **of	Your	School’s	Leadership** 

### [q1 8 ] Rate your level of agreement with the following statement: “Overall, I was satisfied with the quality of my school’s leaders (principals, assistant principals, deans, department chairs) this year (school ye ar 2015

- 2016).” 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree strongly 

### [q1 9 x] Thinking about this year (2015

- 2016), use the slider to indicate how much you agree or disagree with the following statements. “The school leaders who conduct formal observations of my classroom ... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 9 a] Understand the subject matter content that I teach. [insert slider graphic here] 

### [q1 9b ] Understand what skills are needed for teachers to foster meaningful student learning. [insert slider graphic here] 

### [q1 9c ] Foster an atmosphere of trust and mutual respect in my school. [insert slider graphic here] 

### [q1 9d ] Understand the 3 rating components of the NM Teach Summative Evaluation Report (student achievement, observations, multiple measures). [insert slider graphic here] 

### [q1 9e ] Help to improve my [insert slider graphic here]84

teaching. 

### [q1 9f] Are instructional leaders. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q 20 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness S ys tem from school year 2015

- 2016. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 20 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider graphic here] 

### [q 20 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the st udents they teach. [insert slider graphic here] 

### [q 20c ] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my performance. [insert slider graphic here] 

### [q20d ] I understand how the overall teacher ratings work in the NM TEACH Educator Effectiveness System
. [insert slider graphic here]

-----------------------------------------------------**[NEW SURVEY PAGE]**

---------------------------------- **Your NM	TEACH	Rating**  **Summative	Evaluation** 

### [q21 ] Do you know what your level of Educator Effectiveness (Ineffective, Minimally Effective, Effective, Highly Effective, Exemplary) was for the school year 201 5

- 201 6 ? 

##  Yes 

##  No 

### [q22] [if q21=yes] How did you learn your level of Educator Effectiveness for the school year85

201 5

- 201 6 ? Select all that apply. 

##  I received it via email from New Mexico Public Education Department 

##  I logged into my teacher portal online 

##  A school leader told me 

##  I received it via my mailbox at school 

##  Other:

______________________ 

##  Other: ______________________ 

##  I don’t recall 

### [q23] [if q21=yes] Did a school leader ever discuss with you how to improve your Educator Effectiveness score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  I don’t recall 

### [q24] [if q21=yes] Did a school leader ever discuss with you how to interpret your student achievement growth score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  Not Applicable 

##  I don’t recall

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. If you do not, proceed to the next and final page.86



### [q2 5 ] [if 13a=yes ] Do you have any suggestions to help us improve the Checklist for New Mexico Principals’ Provision of Feedback to Teachers ? 

### [q26] Do you have any comments about professional development available to teachers in your district? 

### [q2 7 ] Do you have any comment s about the NM TEACH Observation Rubric? 

### [q2 8 ] Do you have any comments about the NM TEACH teacher evaluation system?

---------------------------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the surve y invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

---------------------------------------**[NEW SURVEY PAGE]**

------------- -------------------------------- **Thanks	Again** You have completed the survey and may now close it.87



### A **PPENDIX B**  **– Principal	and	Teacher	Recruitment	Materials**  **W ave	1	Principal	Recruitment	Script** 
**– SPRING	2015**  Subject: Invitation to take $25 survey to provide feedback on the NM TEACH teacher evaluation process (and receive $25) title first_name_ppal last_name_ppal School_name full_district_name Dear title last_name_ppal, The Regional Education Lab Southwest (RELSouthwest) is studying principals’ fe edback to teachers in an effort to improve the teacher evaluation process. **Will you please be part of the study?**  Participation involves three things: 1. Complete a 30

-minute online survey now using the link below (and receive $25). 2. Receive a short guide about providing effective feedback to teachers for yo ur use throughout the 2015

- 2016 school year. You and your teachers will be assigned at random to one of two ty

pes of guidance. 3. Complete a second 30

-minute online survey in spring 2016 (and receive an additional $25). All New Mexico public school principals are invited to participate in the study. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Onl y researchers from the REL Southwest will view your answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.** https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional q uestions about the survey, please feel free to contact me at [MMIC survey coordinator email address88

here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Ques tions**  **How was I selected for this study?** We are asking all New Mexico public school principals to complete the survey.
**What do I get for completing the surveys?** You will receive a $25 gift card for completing each survey. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.c om/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH trainin g for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the study results?** Only Southwest REL researchers will see your answers to survey questions. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyo ne or alter it.89

**Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.90



### W**ave	1	Teacher	Recruitment	Script**
**– –**  **SPRING	2015** Subject: Invitation to takesurvey to provide feedback about the NM TEACH teacher evaluation process (and receive $25) title first_name_teacher last_name_teacher School_n ame full_district_name Dear title last_name_teacher, The Regional Education Lab Southwest (RELSouthwest) is studying the quality of the feedback you receive after formal observations that are a part of NM TEACH. **Will you please be part of the study?**  Your participation involves three things: 1. Complete a 30

-minute online survey using the link below (and receive $25). 2. Receive a short document about effective feedback sessions between you and your school leader for your use throughout the 2015

- 2016 school ye ar. You and your principal will be assigned at random to one of two types of guidance. 3. Complete a second 30

-minute online survey in spring 2016 (and receive an additional $25). RELSouthwest has first invited all New Mexico public school principals to p articipate in the study. Then in all schools where principals agree to participate, REL Southwest is inviting up to 10 teachers to participate. You are one of only 10 teachers selected by

REL Southwest to participate. Since you were randoml y selected, we c an’t replace you with any other teacher. Thus, your participation is very important. Principals who agree to participate are assigned at random to receive one of two types of guidance about effective feedback to teachers. Teachers who participate in the st udy receive the same guidance as their principals

. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.**91

https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions How was I selected for this survey?**  We are asking up to 10 teachers to participate in each school where a principal has agreed to participate in the study.
**What do I get for participating in the study?** You will receive a $25 gift card for completing each of two surveys. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgi ftcard.com/gift

- cards/

. **What is the purpose of the study and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organizat ion. **Who will get the survey results?** Only Southwest REL researchers will see your answers to the surveys. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents.92

**Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encoura ge you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([surv ey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll fre e at {XX}, or c/o IRB, {address}.93



### W**ave	2	Principal	Recruitment	Script**
**– –**  **SPRING	2016** Subject: Invitation to take final $25 survey to improve the NM TEACH teacher evaluation process title first_name_ppal last_name_ppal School_name full_district_name D ear title last_name_ppal, Thank you for participating in the Regional Education Lab Southwest (REL Southwest) study of New Mexico principals’ feedback to teachers! You completed our spring 2015 survey

. See your responses here. **We are now inviting you to complete the second and final principal survey.**  As a reminder, all New Mexico public school principals were invited to participate in the study, which is supported by b oth the New Mexico Public Education Department and your school district. (To see the letter of support from NM PED and districts, visit {REL Southwest website} .) **Click this link to read your rights as a research subject and complete the spring 2016 survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view your answers, and all answers will be aggregated before being included in the public report. The survey results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC surv ey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher94

**Frequently Asked Questions How was I selected for this survey?**  We are asking all public sch ool principals who completed the spring 2015 survey (for $25) to also complete the 2016 survey (for another $25).
**What do I get for completing the survey?** You will receive a $25 gift card for completing the survey. You’ll be able to choose your gift car d from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest R egional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the stu dy, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the survey?** Southwest REL researchers are conducti ng this survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see individual results. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey onl ine, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for t he protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.95



### W**ave	2	Teacher	Recruitment	Script**
**– –**  **SPRING	2015** Subject: Invitation to take final $25 survey to improve the NMTEACH teacher evaluation process title first_name _teacher last_name_teacher School_name full_district_name Dear title last_name_teacher, Thank you for participating in the Regional Education Lab Southwest (REL Southwest) study of New Mexico principals’ feedback to teachers! You completed our spring 201 5 survey

. See your responses here. **We are now inviting you to complete the second and final teacher survey.** As a reminder, all New Mexico public school principals were invited to participate in the study, which is supported by b oth the New Mexico Public E ducation Department and your school district. (To see the letter of support from NM PED and districts, visit {REL Southwest website} .) **Click this link to read your rights as a research subject and complete the spring 2016 survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions**96

**How was I selected for this survey?**  We are asking all teachers who completed the spring 2015 survey (for $25) to take the spring 2016 survey (for another $25). We invited up to 10 teachers to take the 2015 survey in each school where a principal has agreed to partic ipate in the study.
**What do I get for completing the survey?** You will receive a $25 gift card for completing the survey. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give t o teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the survey?** Southwest REL researchers are conducting this survey. The researchers are from the RAND Corporation, which is an independent, nonpro fit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see individual results. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents. **Am I required to complete the survey?** No, the survey is voluntary. Ho wever, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link sh own above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project partici pants): {email}, toll free at {XX}, or c/o IRB, {address}.97



### W**ave	1	Principal	Recruitment	Script**
**– SPRING	2015**  Subject: Reminder: Invitation to take survey to provide feedback on the NM TEACH teacher evaluation process (and receive $25) title first_name_ppal last_name_ppal School_name full_district_name Dear title last_name_ppal, Recently you received an email asking you to be a part of a study under The Regional Education Lab Southwest (RELSouthwest) on principals’ feedback to teachers. **W i** **ll you please be part of the study?**  Participation involves three things: 1. Receive a short guide about providing effective feedback to teachers for yo ur use throughout the 2015

- 2016 school year. You and your teachers will be assigned at random to one of two ty

pes of guidance. 2. Complete a 30

-minute online survey now using the link below (and receive $25). 3. Complete a second 30

-minute online survey in spring 2016 (and receive an additional $25). All New Mexico public school principals are invited to participat e in the study. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Onl y researchers from the REL Southwest will view your answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y,98

[MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions How was I selected for this study?**  We are asking all New Mexico public school principals to complete the survey.
**What do I get for completing the surveys?** You will receive a $25 gift card for completing each survey. You’ll be able to choose your gift card from more than 200 merchants. T o see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conduc ting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}

. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the surve y. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the study results?** Only Southwest REL researchers will see your answers to survey questions. The survey results will be s ummarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that coul d identify respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback proces s. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you ha ve general questions, you can99

contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.100



### W**ave	1	Teacher	Recruitment	Script**
**– –**  **SPRING	2015** Subject: Reminder: Invitation to take survey to provide feedback about the NM TEACH teacher evaluation process (and receive $25) title first_name_teacher last_name_teacher School_name full_district_name Dear title last_name_teacher, Recently you received an email asking you to be a part of a study under The Regional Education Lab Southwest (RELSouthwest) on the quality of the feedback you receive after formal observations that are a part of NM TEACH

. **W ill you please be part of the st** **udy?** Your participation involves three things: 1. Complete a 30

-minute online survey using the link below (and receive $25). 2. Receive a short document about effective feedback sessions between you and your school leader for your use throughout the 2015

- 2016 school year. You and your principal will be assigned at random to one of two types of guidance. 3. Complete a second 30

-minute online survey in spring 2016 (and receive an additional $25). RELSouthwest has first invited all New Mexico public school principals to participate in the study. Then in all schools where principals agree to participate, REL Southwest is inviting up to 10 teachers to participate. You are one of only 10 teachers selected by

REL Southwest to participate. Since you were randoml y selecte d, we can’t replace you with any other teacher. Thus, your participation is very important. Principals who agree to participate are assigned at random to receive one of two types of guidance about effective feedback to teachers. Teachers who participate in the study receive the same guidance as their principals

. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.** https://survey .rand.org/research/hosting/ced/101

**Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please fee l free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions How was I selected for this**  **survey?** We are asking up to 10 teachers to participate in each school where a principal has agreed to participate in the study.
**What do I get for participating in the study?** You will receive a $25 gift card for completing each of two surveys. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the study and how will the resul ts be used?**  T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For mor e information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest RE L researchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see y our answers to the surveys. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents.102

**Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TE ACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?**  If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have c oncerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.103



### A **PPENDIX C**  **– Secondary	Data	Elements**  **Student, teacher and principal administrative data variables** Below we list the names of the files and variables we will be requesting from NM PED on students, teachers and principals as a part of the analysis. The information is based on the Student Teacher Accountability Reporting System (STARS) Volume 1 User Guide, 2014

- 15. ST ARS is a comprehensive student, staff, and course information system that provides a standard data set for each student served by Ne w Mexico’s 3Y/4Y

– Grade 12 public education system. STARS uses the eScholar data warehouse which integrates detail student, staff, and course information via a standard set of templates that provide a consistent format for loading data from various student information systems used in public school districts or charter schools. For each data requested we list the Domain name, Template Name, and variable name and a variable description. **Staff Domain, Staff Assignment Template**

**Field Name Description**  District Code PED defined three character district co de. Staff ID Masked Staff ID fixed across years and locations Assignment Code PED defined assignment or position codes. School Year Date School Year Location Code PED define d three character location code of school Assignment Date Report Date Percentage of Time Assigned The portion of the staff member’s time that is allocated to the assignment. **Staff Domain, Staff Development Fact Template**

**Field Name Description**  District Code PED defined three character district code. Staff ID Masked Staff ID fixed across years and locations Activity Code Professional Development Activity Code Activity Start Date Professional Development Activity Start Date **Staff Domain, Staff Template**

**Field Name Description**  District Code PED defined three character district code. Staff ID Masked Staff ID fixed across years and locations Gender Code Gender of Staff Member Ethnic Code Ethnicity Code of Staff Member Teaching of Principal Years of Experience The number of years that the staff member has been em ployed by any school district in any state as a TEACHER or PRINCIPAL (includes current district). Teaching of Principal Years of Experience in District The number of years that the staff member has been em ployed by the current school district as a TEACHER or PRINCIPAL. Highest Degree Earned Highest degree earned by the staff member. Staff Qualification Status Staff Qualification Status such as Certified or Substitute teacher104

Hispanic Indicator Indicator of whether the staff memb er is of Hispanic ethnicity Tribal Affiliation The primary New Mexico Tribe or Pueblo with which the staff member is affiliated or enrolled
**Student Domain,**  **Student Template** 

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Alternate Student ID District's local student identifier. Location Code PED define d three character location code of school School Year Date School Year Current Grade Level Student current grade level Birthdate Student date of birth. Gender Code Gender code of Student Ethnic Code Ethnicity Code of Student Hispanic Indicator Indicator of whether the student is of Hispanic ethnicity Economic Status Student participation in free or reduced lunch program. Economic Disadvantaged Status Code Identifies the type of Direct Cert the student is receiving , e.g. SNAP, TANF, Homeless. Special Education Student is IEP Y/N Level of Integration Amount of special education service. Repeating Last Year If student is in grades K

- 12, Y/N for retained or failed to be promoted following the completion of the past school year Gifted Participation Code Indication of whether the student is Gifted Migrant Status Indicator of migratory status English Proficiency Code Current or former ELL student.
**Assessment Domain,**  **Assessment Fact Template** 

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Test Description Description of type of test Assessment School Year Date School Year Item Description Code Item description code Test Date Date of Assessment Location Code PED defined three character location code. Score Score of administered test. Scoring Model The Measurement used to determine the outcomes the respective assessment105

**Attendance Domain,**  **Student Daily Attendance Template**

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Location Code PED defined three character location code. Attendance Date Report date of student unexcused absence. Attendance Code Provide the Attendance Code that indicates the unexcused absence status for the date.
**Attendance Domain,**  **Student Summary Attendance Template**

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Location Code PED defined three character location code. School Year Date School Year Period Start Date Attendance period start date. Period End Date Attendance period end date. Days Present Provide the total number of days within the Reporting Period that the student was marked as Present at the location. Days Enrolled Provide the total number of days within the Reporting Period that the student was counted as Enrolled at the location.
**Course and Grades Domain,**  **Course Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Course Name Course Name Semester Indicator of reporting period Course Code PED

- defined 8 character course code offered by the school.
**Course and Grades Domain,**  **Course Instructor Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Primary Instructor ID State issued student identification number. Course Name Course Name Semester Indicator of reporting period Course Code PED

- defined 8 character course code offered by the106 school. Section Code Class section number assigned by school district to identify distinct classes. Snapshot Date Field indicating the report date. See valid values. Special Course Indicator Indicator for honors, gifted, special education, etc. **Course and Grades Domain,**  **Student Course Enrollment Template**

**Field Na me**  **Description** District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Enrollment Period Reporting date Student ID State issued student identification number. Section Code Class section number assigned by school district to identify district classes. Course Code PED

- defined 8 character course code offered by the school.
**Enrollment Domain,**  **School Enrollment Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Student ID State issued student identification number. Enrollment Date T he actual date the mobility transaction occurred. Enrollment Code Code identifying the type of mobility transaction. Enroll Change Code Provide a code that indicates the student enrollment was a result of SCHOOL CHOICE.107

**NM TEACH Reflect Data** Below we list the variables we will be requesting from NM PED from the Teachscape operated REFLECT teacher evaluation data s ys tem. Staff ID that is consistent across school years and data sources Teacher Evaluation Group (A, B, or C) Graduated Consideration Tag District ID School year School NCES ID Masked Evaluator ID Observation number Observation date NM TEACH Summative evaluation component

- Observation Element name NM TEACH Summative evaluation component

- Observation Element Score (Level of Performance) NM TEACH Summative evaluation component

- Student Achievement measure name NM TEACH Summative evaluation component

- Student achievement measure score NM TEACH Summative evaluation component

- Multiple measure name NM TEACH Summative evaluation component

- Multiple measure score NM TEACH Summative evaluation overall score NM TEACH Summative evaluation effectiveness rating108

**Principal and Teacher contact information File 1** 

**– Principal contact information**  Principal Last Name Principal First Name Principal Middle Name or initial Principal Maiden Name Principal primary

email address Principal secondary

email address School Name School year School NCES ID

**File 2**

**– Teacher contact information**  Teacher Last Name Teacher First Name Teacher Middle Name or initial Teacher Maiden name Teacher primary

email address Teacher secondary

email address School Name School year School NCES ID