# S UPPORTING	ST AT EMENT A : REQUEST	FOR	CLEARANCE	OF	INFORMA TION COLLECTION	FORMS	FOR

“ IMP AC TS	OF	A	DET

AILED	CHECKLIST	ON FORMA TIVE	FEEDBA CK TO TEA CHER S” August 2014

**Submitted to: Submitted by**  : U.S. Department of Education SEDL Institute of Education Sciences 4700 Mueller Blvd. 555 New Jersey

Ave. NW, Rm. 308 Austin , TX 78723 Washington, DC 20208 Phone: ( 800 ) 476

- 6861

4700 Mueller Blvd. Austin, TX 78723 800

- 476

- 6861 www.relsouthwest.org This publication was prepared for the Institute of Education Sciences (IES) under contract ED

- IES

- 12

- C

- 00012 by Regional Educational Laboratory Southwest , administered by SEDL . The content of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted.

### Contents

OVERVIEW

................................ ................................ ................................ ................................ ... 5 Impact Research Questions

................................ ................................ ................................ ......... 7 Implementation Fidelity

Research Questions

................................ ................................ .............. 7 Exploratory

Research Questions

................................ ................................ ................................ . 7 A. JUSTIFICAT ION

................................ ................................ ................................ ...................... 9 1. Circumstances Necessitating the Data Collection

................................ ................................ .. 9 2. How, by

Whom, and for What Purpose the Information I s to Be Used

............................... 17 3. Use of Information Technology

................................ ................................ ........................... 18 4. Efforts to Avoid Duplication of Effort

................................ ................................ ................. 19 5. Sensitivity

to Burden on Small Entities

................................ ................................ ................ 19 6. Consequences of Less Frequent Collection and Obstacles to Burden Reduction

................ 19 7. Circumstances Requiring Special Information Collection

................................ ................... 20 8. Federal Register Announcement and Consultation Outside the Agency

............................. 20 a. Federal Register Announcement
................................ ................................ ........................ 20 b. Consultations Outside the Agency

................................ ................................ ..................... 20 c. Unresolved Issues

................................ ................................ ................................ ............... 21 9. Pay ments or Gifts to Respondents

................................ ................................ ........................ 22 10. Assurances of Confidentiality

................................ ................................ ............................ 22 11. Justification for Sensitive Questions

................................ ................................ .................. 24 12. Estimates of Hour Burden

................................ ................................ ................................ .. 24 Burden for Study

Recruitment

................................ ................................ ............................... 26 Burden for Extant Data Collection

................................ ................................ ......................... 27 Burden for Survey

Data Collection

................................ ................................ ........................ 27 13. Estimated Total Annual Cost Burden to Respondents or Recordkeepers

.......................... 27 Total Cost to Respondents

................................ ................................ ................................ ...... 27 14. Estimated Annualized Cost to the Federal Government

................................ .................... 28 15. Program Changes or Adjustments

................................ ................................ ...................... 28 16. Plans for Tabulation, Publication, and Schedule for Project

................................ .............. 28a. Tabulation Plans

................................ ................................ ................................ ................. 28

b. Publication Plans

................................ ................................ ................................ ................ 29 c. Project Time Schedule

................................ ................................ ................................ ........ 30 17. Display

of Expiration Date

................................ ................................ ................................ . 31 18. Exceptions to the Certification Requirement

................................ ................................ ...... 31 REFERENCES

................................ ................................ ................................ ............................. 32 APPENDIX A

– Teacher and Principal Survey Instruments
................................ ........................ 34 Spring 2015 School Leader Survey

................................ ............ **Error! Bookmark not defined.** Spring 2016 School Leader Survey

................................ ............ **Error! Bookmark not defi ned.**  Spring 2015 Teacher Survey

................................ ...................... **Error! Bookmark not defined.** Spring 2016 Teacher Survey

................................ ...................... **Error! Bookmark not defined.** APPENDIX B

– Principal and Teacher Recruitment Materials

................................ ................... 95 Wave 1 Principal Recruitment Script

– SPRI NG 2015

................................ .............................. 95 Wave 1 Teacher Recruitment Script

– – SPRI NG 2015

............. **Error! Bookmark not defined.** Wave 2 Principal Recruitment Script

– – SPRI NG 2016

............ **Error! Bookmark not defined.** Wave 2 Teacher Recruitment Script

– – SPRI NG 2015

............. **Error! Bookmark not defined.** Wave 1 Principal Recruitment Script Follow

-up ....................... **Error! Bookmark not defined.** Wave 1 Teacher Recruitment Script Follow

-up ......................... **Error! Bookmark not defined.** APPENDIX C

– Secondary Data Elements

................................ ................................ ................ 111 Teacher and School Administrative Data Variables List

................................ ........................ 111 NM TEACH Reflect Data Variables List

................................ ... **Error! Bookmark not defined.** Principal and Teacher Contact Information

................................ **Error! Bookmark no t defined.**  APPENDIX D

– Technical Working Group Suggestions

................................ .......................... 117 APPENDIX E

– Educational Sciences Reform Act

................................ ................................ ... 119 APPENDIX F

– IRB Approval

................................ ................................ ................................ ... 120 APPENDIX G

– Study

Participants’ Consent

................................ ................................ ............. 124 Principal Wave 1 and Wave 2 Consent

................................ ...... **Error! Bookmark not defined.** Teacher Wave 1 and Wave 2 Consent

................................ ................................ ..................... 128 APPENDIX H

- Confidentiality Form and Affidavits

................................ ................................ 1325 

### OVERVIEW The U.S. Department of Education (ED) requests clearance for the recruitment materials and data collection protocols under the OMB clearance agreement (OMB Number [IES to complete]) for activities related to the Regional Educational Laboratory Program (REL ). ED, in consultation with SEDL , is planning a clustered randomized evaluation in New Mexico to test the effectiveness of materials intended to impro ve the feedback that principals provide in one

- on - one conferences to their teachers about their classroom instruction. The study includes an impact anal ys is and an implementation analysis

. New Mexico Public Education Department (NM PED) staff have identified the topic of principal feedback to teachers as an area where New Mexico needs assistance. In particular, NM PED staff have received comments from principals that they do not feel adequately prepared to provide teachers with feedback about their instructional practices. The NM PED has committed to ensure that every

student has equitable access to an effective principal and teacher every day they are in school. Although NM PED recognizes that the conferences are likely an area in the overall evaluation system in need of improvement, it has limited resources and time to focus on the post

- observation conference step in the evaluation cycle. Therefore, any potential interventions to address this issue must be easy to implement and re quire few resources. This impact study

will examine whether an enhanced feedback guide (Conversation Protocol) , relative to business

- as -usual guidance to principals and teachers , achieves the following : 1. improves the structure and content of the principal

- teacher feedback conversation, 2. improves perceptions that the feedback delivered to teachers is useful 3. better targets guidance to teachers’ regarding professional development that is tied to their formal classroom observation scores 4. improves quality

of teacher instruction as measured by subsequent formal observation ratings on the state’s tool, called NM TEACH, and 5. increases student achievement and state standardized tests

. To examine these outcomes, ED contractors will invite all public school principals of charter and non - charter schools in the state to participate in the study . Blocking by school district, a randomly selected half of principals who agree to participate (along with teachers in their school who also agree to partic ipate in the study) will be assigned to the treatment group . The remaining half of principals (along with teachers in their school who also agree to participate in the study) will be assigned to the control group. In summary, the study

will compare princ ipals and teachers in the treatment and control conditions, which are detailed in Table 1.6

**Table 1**  **. Treatment and Control Conditions**  **Treatment g**  **roup Control** 

**g roup**  **Principals** NM PED

- sponsored professional development for principals offered during the summer, of which 1.5

- 2 hours were devoted to feedback to teachers. Training conducted in regional in

-person professional development sessions led by the state’s contractor, Southern Regional Education Board (SREB). All principals in the state are required to participate as a part of their job requirement s .  

List of documents to bring to each feedback conversation  24

- item checklist to use during each feedback conversation.  5

- minute video in which principal testifies about her experience using the checklist.  Business

- as -usual guide to remind principals about “5 stages of feedback” described in NM PED

- sponsored principal professional development to use for feedback conversations 
**Teachers** Business

- as -usual guide: A document the includes a reminder to teachers about their right to feedback within 10 calendar days of formal observation under the NM TEACH Educator Effectiveness System, and a link to the NM PED provi ded FAQ on teacher evaluation (http://ped.state.nm.us/p ed/NMTeach_FAQ.html ) 

List of documents to bring to each feedback conversation  24

- item checklist to use during each feedback conversation  5

- minute video in which teacher testifies about her experience using the checklist 

ED contractors will request that the treatment group principals and control group principals do three things: (1) disseminate the relevant guide (and a link to the video, where relevant) to all school leaders and to all teachers in his/her building, (2) use the guide and view the video, where relevant (along with all other schools leaders in the building who observe teachers) with 100% of the teachers that they

formally observe during school year 20 15

- 20 16, and (3) complete a baseline survey

in fall 2015 and a follow

-up survey in spring 2016. ED contractors will request treatment group teachers and control group teachers to do two things: (1) use the guide they received from ED contractors (and view the video, where relevant) in the feedback conversations7

they have with th eir school leader(s), and (2) complete a baseline survey in fall 2015 and a follow

-up survey

in spring 2016. 

### Impact	Research	Questions The evaluation will examine the impact of providing a post

- classroom observation feedback checklist and a video to principals and teachers on principal , teacher, and student outcomes. To do so, this evaluation will use a cluster RCT design with random assignment at the school level to address the following impact and implementation questions. Does provision of the feedback checklist and video relative to the business as usual guidance: 1) improve the content and structure of post

- observation conference according to principals and teachers? 2) improve principals’ and teachers’ perceptions of the uti lity of post

- observation conference? 3) increase the amount of time it takes to complete the post

- observation conference? 4) cause principals to recommend and teachers to take professional development that is aligned to needs identified in the formal observations? 5) improve the quality

of teachers’ subsequent instructional practices as measured by principals’ formal c lassroom observation scores? 

### Implementation	Fidelity	Research	Questions ED contractors will also examine the implementation of the Feedback Checklis t relative to the business

- as -usual guidance by analyzing responses to survey questions regarding the ty pe of guidance used . The implementation portion of the study will address the following research questions: 6) How extensively

do principals and teachers in the treatment and control groups report using the form of guidance they were assigned? 7) How extensively

do principals and teachers in the treatment and control groups report using the form of guidance they were not assigned (i.e. cross

- overs)? 

### Explorator**y	Research	Questions** ED contractors will also perform exploratory analyses examining the association of the enhanced feedback protocol on subgroups of teachers and principals. Inaddition, we will also examine whether provision of the treatment guide improves subsequent student performance on state standardized tests. Although the design of the study allows for causal analysis of these questions, for rea sons of limited sample size within New Mexico and limited time for the treatment or control condition to influence behavior and student achievement , we will likely have insufficient statistical power to detect differences. Nevertheless, these questions have high policy relevance for NM PED, so we include them to provide insights for follow

- on studies and for NM PED’s improvement of subsequent iterations of NM TEACH observations and feedback. The exploratory

research questions are :8

8) Does the provision of the checklist to principals and teachers increase student achievement relative to business

- as -usual guidance regarding feedback? 9) Is there suggestive evidence that the effects of providing the checklist varies according to the following subgroups of principals and to the following subgroups of teachers: a) extensive users of the checklist

– i.e., teachers who have received three versus two formal observations and principals who have used the checklist in schools that have a greater than average number of teachers within the given district b) personal beliefs and attitudes about NMTEACH evaluation system c) accountability

pressure (as measured by prior year’s school accountability grade) d) those principals who have received training on the NMTEACH Observation Rubric e) those who work in schools that teachers rate as collaborative and supportive f) those with greater qualifications, including more years of experience and certification ty pe g) those who teach core subjects versus non

- core subjects h) those who work in elementary versus middle or high schools i) teachers in the top and bottom quartile of the NM TEACH summative effectiveness score distribution j) principals in the top and bottom quartile of the NM TEACH School Leader effectiveness score distribution Be yond subgroup anal ys es, ED contractors will also gather information about the perceptions users have of the treatment checklist and the control guide. 10) What are principals’ and teachers’ perceptions of the checklist and videos versus business

- as usual feedback guidance? For example, do they report that it is easy

to use, burdensome, formulaic, lengthy, useful for creation of professional development plans, and helpful in causing teachers to commit to instructional changes? 11) What are the main reasons why princ ipals report not using the form of guidance they received? checklist or the business

- as -usual guidance? The project consists of data collection from NM PED and participating principals and teachers who work in New Mexico public schools . Specifically, in this OMB clearance package, ED is requesting clearance for the following data collection approaches:
 Recruitment materials for participating principals and teachers
 Extant administrative records data collections from NM PED
 Two waves of web

-based surveys of principals and teachers in treatment and control schools

. ED believes that the data collections for which clearance is being requested represent the bare minimum necessary

to assess the impact and implementation of the Conversation Protocol.9



### A. **JUSTIFICATION 1.		Circumstances	Necessitating	the	Data	Collection**  New Mexico is committed to ensure that every student has equitable access to an effective principal and teacher. As a part of this mission, the NM PED has committed to “implementing a rigorous, uniform [classroom] observation protocol, providing immediate constructive feedback [to teachers] , using meaningful student data, and other multiple measures [that] will provi de valuable information to aid the personal development and growth of each teacher and principal.” NM PED undertook the task of reforming their teacher evaluation s ys tem starting in 2011. In 2012

- 2013, New Mexico piloted a new teacher evaluation system called NM TEACH that is mandated in all public schools starting in the 2013

- 2014 school year. Under NM TEACH, teachers are evaluated each school year based on three primary components: 1. evidence of student achievement improvement (such as value added scores on state and end of course exams), 2. average scores on the state

- developed teacher observation rubric (NM TEACH), and 3. locally

selected measures from a menu of options approved by the state (e.g., teacher attendance or student surveys). These components are combined into a summative evaluation score, with 50% of the summative evaluation score based on improved student achievement, 25% based on the observation of teacher practice, and the final 25% based on the locally selected measures

. Regarding the second c omponent of teacher evaluations, NM PED requires that a principal or a designated school leader observe his or her teachers formall y two or three times per ye ar (with 20 minute observations), and informally throughout the school year (with several short “w alk

- throughs” lasting 3

- 5 minutes each). For formal observations, the school leader must rate the teacher using the NM TEACH Observation Rubric

—a single rubric that applies to all grade levels and subjects taught. The combined scores from the 2

- 3 required formal observations comprise the second e lement of teachers’ ratings. As the teacher evaluation cycle is conceived, a critical step to improving instruction through the revised teacher evaluation system is the required conference between the teacher and the school leader that takes places within 10 days after each of the formal classroom observations. In this conference the school leader reviews his or her classroom observation rubric scoring with the teacher and makes recommendations for improvement and professional development. Note that this study will onl y recruit one type of school leader

— principals

—and we therefore onl y refer to principals when describing the conduct of formal observations and feedback below. 1

1 The recruitment does not include all types of school leaders such as assistant principals or deans because it is a school

-based decision which school leaders, if any, besides principals formally observe and rate teachers using the NM TEACH Observation Rub ric. Therefore, among school leaders, ED contractors will recruit only principals into the study, and then ask participating principals to disseminate the guide they receive (whether the treatment or control guide) to all relevant school leaders in the bui lding who also conduct formal classroom observations.10

In the research literature, much attention has been paid to the development of the classroom formal observation rubrics and the extent to which teacher observations align with other measures of student learning. 2 However, NM PED, like the broader field, has paid much less attention to the next step in the annual cycle of the teacher evaluation: the feedback conversation in which the principal conveys to the teacher the results of the scored observation. There is a dearth of scientificall y rigorous research about effective feedback, and there are few resources dedicated to teaching pri ncipals how to convey the results of the classroom observations in a manner that helps teachers improve their instruction and leads to improved student outcomes. The teacher feedback process is perhaps the least developed link in new teacher evaluation systems . The majority of academic attention has been devoted to the mechanics of the value added scores and the relative weight of multiple measures (Mihal y et al. 2013; Bill and Melinda Gates Foundation, 2012; McCaffrey, Lockwood, Koretz, Louis, and Hamilton, 2004), as well as the development of observation rubrics and their alignment with other measures of student learning (Bill and Melinda Gates Foundation, 2012; Pianta and Hamre, 2009; Hill, Kapitula and Umland, 2010). Yet the quality and clarity of the feedback is the primary means b y which teacher evaluation systems would allow teachers to recognize the aspect of their practice that needs improvement and could encourage teachers to seek out appropriat e professional development and to improve teachers’ instruction. It is therefore a critical point in the teacher evaluation cycle to study. The research literature provides some limited evidence that teachers improve their instruction when they

receive professional learning opportunities that are ongoing and closely connected to teachers’ curriculum and instruction (Supovitz and Turner, 2000; Correnti, 2007; Correnti and Rowan, 2007), as well as those that are aligned with what school districts value (Penuel et al., 2007; Garet et al., 2001). Given that teacher

-principal conferences are meant to occur throughout the year and directly

reflect what teachers are doing in their classrooms, those conferences could thus be an important mechanism for encouraging b oth teacher growth and directing teachers to appropriate professional development. Most recently, Hill (2009) provides evidence that professional development remains largel y ineffectual for improving practice, often coming in the form of short

- term, low

- quality training that teachers find difficult to apply in their classrooms. If tightly linked to teachers’ observed classroom practices, recommended professional development could become increasingly relevant to teachers’ instructional practices. (Little, 19 93; Ball and Cohen, 1999; Wilson and Berne, 1999). To date, only

two studies have examined the feedback given to teachers by principals, and both are based on the evaluation system in Chicago Public Schools. The University of Chicago

2 See, for example, Mihaly et al., 2013; Bill and Melinda Gates Foundation, 2012; Pianta and Hamre, 2009; and Hill, Kapitula and Umland, 2010.11

Consortium on Chicag o School Research (CCSR) found that Chicago Public Schools’ expectations for principal

- teacher conferences pre

- and post

- observation did not align with principal and teachers’ actual practices (Sartain et al., 2011). In that CCSR study, onl y 10% of the que stions principals asked of teachers reflected high expectations and required deep reflection about instructional practice and 65% of questions required limited response and focused on affirmation of the principal’s views. Additionally, principals tended to talk for 75% of the conference, and teachers onl y 25%. Overall, Chicago Public School principals reported a need for guidance on how to provide actionable feedback, which is precisely what this study aims to do for New Mexico principals. A more recent study by CCSR (Sporte et al., 2013) of the same teacher evaluation system found that administrators list the provision of useful feedback to teachers as an area where principals need professional development. The theories emplo yed in our intervention comport with the main lessons from a broad literature providing guidance on the most effective methods and features for delivering job performance feedback and for conducting performance appraisals. Inthe field of clinical supervision, Heckman

- Stone (2003) pro vides some guidance for the proper structure of performance evaluation and feedback based on observations from supervision of students in the clinical practices. In providing feedback, she recommends that feedback standards should be agreed upon and understood by

both parties and be provided regularly to build a strong relationship and avoid misunderstanding. Additionally, positive feedback should be provided prior to negative evaluations, and that any feedback should be properly contextualized into speci fic skill areas so that a supervisee is clear on what skills are being assessed and how they are being assessed. The standard professional development that the New Mexico PED offers in the summer to all principals about NM TEACH covers some, but not all, o f these principles. The business

- as -usual guidance to principals does not spell out ways to offer positive feedback before negative feedback , does not contextualize feedback so that the supervise is clear on what skills are being assessed (i.e., no referen ce to the NM Observation Rubric), and does not include a discussion of how teachers are being assessed. In both health and education research, several simple, low

- cost interventions such as the one studied here have proven effective. Most relevant to this study, the distribution of a one

-page checklist of tasks before surgery with common

- sense items on it such as nurses and surgeons should introduce themselves before surgery and correctl y time the use of antiseptics had the extremely large impact of reducing surgical complications by 36% and deaths by

47% (Hay

nes et al., 2009 in Yeager & Walton, 2011). The author of the surgery checklist, Atul Gawande, claims that the sur

prisingly large effects of the checklist is due to health care professionals simply (and erroneousl y) assuming that surgeons and nurses were doing all the items on the checklist without verifying that fact. New Mexico PED staff with whom the SW REL researchers have spoken profess a similar assumption that principals and teachers will carry out common - sense steps in feedback conversations (e.g., review in detail the ratings on the12

observations on NM TEACH Observation Rubric, offer both positive and negative feedback, mutually develop a set of next steps for the teacher where the professional development explicitly ties to the scores on the Observation Rubric) but admit that they have no way

to verify that assumption. The limited research on feedback within these new teacher evaluation systems like NM TEACH suggest that feedback conversations do not adhere to the seemingl y common sense guidelines provided by NM PED

- sponsored professional development or by the Carnegie checklist. Yeager and Walton (2011) revi ew six education studies that have had substantial, lasting impacts on student achievement even though the interventions are short, provide information

- only, do not teach academic content but instead seek to influence students’ beliefs and attitudes about school. The authors deduce two lessons from the interventions in the six studies: (1) the interventions use students’ perspectives to influence student behavior, and (2) they use “stealth tactics” like requiring fewer hours of study participants time to receive the session, not stigmatizing students, and not being didactic. Our study builds off of this work on psychological interventions by examining a low

- cost, brief intervention tailored to New Mexico that could theoreticall y improve the quality

of feedb ack for teachers. Given the critical role of principal feedback in teacher evaluation s ys tems and the limited research about it, the study proposed here is a cluster randomized controlled trial evaluation to estimate the impact of providing principals and their teachers with a detailed checklist and video suggesting how to structure the formative observation feedback conversation

—as compared to a reminder to principals about the 5 steps for feedback outlined in the required state

- sponsored professional dev elopment session

—on the content and structure of the post

- observation conference, the perceived quality of the feedback itself, the quality and take

-up of professional development for teachers, teacher practice as measured by subsequent formal classroom observation scores, and, by extension, student achievement. The combination of the Conversation checklist and the video are intended to be a low

- cost intervention

—an intervention that primarily

consists of information

—that capacity

- constrained state departm ents of education could theoreticall y implement with success. As stated in the overview, the study proposed here is a cluster randomized controlled trial evaluation to estimate the impact of providing principals and their teachers with about a checklist to follow during the feedback conversations regarding teachers’ instruction relative to the minimal guidance offered currentl y to principals about effective feedback. Blocking b y school district, a randomly

selected half of principals who agree to participate in the study (along with teachers in their school who also agree to participate in the study) will receive the Conversation Protocol, which contains an introduction about the importance of feedback, a feedback conversation checklist to be used during the post

- observation conference , and a link to a video testimonial in which a principal or a teacher describe their experience using the checklist. The13

checklist is a customized version of the Teacher Feedback Conv ersation Protocol provided in “Strategies for Enhancing the Impact of Post

- Observation Feedback for Teachers,” developed by coauthors at the Carnegie Foundation for the Enhancement of Teaching (Myung and Martinez, 2013). While this version of the Conversation protocol has not been used in New Mexico, The Vision Network, located in Delaware, has used the Carnegie version of the checklist with principals in the 2013

- 2014 school year and will again use it in the 2014

- 2015 school year. The designer of the checklist, My ung, also reports that other schools have piloted the use of the checklist, but we are not aware of its widespread usage as an official tool adopted by districts or state education agencies

. To conduct a placebo randomized controlled trial , ED contractors will also provide a business

as -usual guide to control group principals and control group teachers participating in the study

. The control guide for principals will list the five steps for feedback discussed in the summer 2014 professional development attended b y NM principals . This will mimic “business

- as -usual” for principals in New Mexico, whi ch is professional development onl y provided to principals and not to teachers about feedback. The control group principals and teachers will not receive a link to the video nor will they receive the checklist. Meanwhile, to mimic “business

- as -usual” for teachers, control group teachers will receive a short PDF with content from the state NM PED website reminding them of their right to a post

- observation feedback conversation within 10 calendar days of the observation. The treatment guide ( called the Conversation Protocol) will go beyond business as usual by providing a brief introduction explaining the importance of a feedback , providing a video testimonial, and, most importantly, providing a detailed checklist for the feedback session

. Checklists have been shown to have large impacts on practice in other settings (Hay n es et al., 2009 )

. ED contractors have applied the following four lessons from other effective simple, low

cost, information

- only

interventions in designing this study : 1. For the treatment group, systematize the information provided to bridge both the

temporal gap between in

- person professional development and feedback conversations and the conceptual gap between theory and practice by

offering an eas y- to
- follow checklist for both principals and teachers to break down the feedback conversation into discrete s teps for use before and during the feedback session. 2. For the treatment group, present a video (one video for principals; and second video

for teachers) about the checklist from a principal’s (teacher’s) perspective to minimize their resistance to the message, rather than have professional development consultants or state or district staff persons tell principals and teachers what to do. 3. Avoid stigmatizing either the treatment or control group b y framing the study as trying out two alternatives for improving feedback to teachers. 4. Keep the information brief to respect principals’ and teachers’ bus y schedules, and

because interventions requiring fewer hours of time have proven more effective than14 longer lasting interventions in several relevant education studies (Yeager & Walton, 2011 )

. The Conversation Protocol c onforms to the schedule of formal observations and the required elements of NM TEACH Observation Rubric and is structured based on the Carnegie Protocol with the following designated sections: A. Op en warmly

and get down to business. ( Thanks for meeting with me. Could you remind me of the lesson goal? Let’s clarify what we want to accomplish. ) B. Focus on what’s going well. ( What worked? In addition, here’s what I saw .) C. Id entify

challenges facing the te acher. ( What were some things you feel could have done better? It sounds like the challenges are ....) D. Generate ideas for addressing the challenges. ( Let’s approach these one by one ...) E. Prioritize next steps. F. End positively

( Was this conversation helpful? Than ks for your time and insights .) The theory

of change , presented in Figure 1 , emphasizes the links in the chain that might be expected to connect principals’ feedback with changes to teacher instruction and, subsequentl y, to student learning. We envision that the process starts with a principal’s formal observation of a teacher’s classroom lesson. Any expected impacts of that observation on teacher instruction and student learning would then depend upon whether the teacher and principal used the guide both to reflect on the instructional strengths and challenges in the lesson and discuss those strengths and challenges in a way

that led to actionable and useful feedback to the teacher during the teacher

-principal conference. Then, subsequent links in the chain depend upon the teachers’ positive perceptions of the teacher

-principal conference and a desire to seek out the recommended professional development (PD ) , the quality of the PD itself and the extent to which the teacher is able to utilize the PD to improve his/her instruction. **Figure 1**

**– Theory of Change** 15

ED contractors propose to collect both primary and secondary

data to understand the impact of providing the Conversation Protocol to principals and teachers on principals and teachers’ perceptions about the quality of feedback, teachers’ professional development, teachers’ instructional quality

as measured b y classroom observation scores and student achievement. The primary data will consist of a principal and a teacher W eb

-based baselin e and follow

-up survey (for four total survey

instruments). The secondary data consists of formal observation scores by school leaders of teacher practice as well as principal, teacher and school characteristics and student achievement scores collected by NM PED.16

Table 2 shows the timeline for all three data collection activities described in this OMB package. **Table 2**  **. Data Collection Timeline**
**Data Collection Purpose**  **Requesting OMB**  **Clearance? May**  **/**

**June 2015**  **May/**

**June 2016**  Extant data for recruitment from New Mexico Public Education Department  Email addresses and first and last

name for principals and teachers to recruit them into study via email (by April 1, 2015) Yes X

Extant administrative school and student data from New Mexico Public Education Department  Teachers’ NM TEACH

Observation Rubric scores (RQ5)  Student achievement data from spring state standardized test (RQ 9)  Description of study schools, principal qualifications, teacher qualifications, and student population (RQ10) Yes X X

Principal surveys  Perceptions of post

- observation

conferences (RQ1, 2)  Behaviors in post

- observation conferences with teachers (RQ3, 4, 7, 8, 10)  Use of checklist and guidance (RQ 11, 12) Yes X X

Teacher surveys  Perceptions of post

- observation

conferences (RQ1, 2)  Behaviors in post

- observation conferences with teachers (RQ3, 4, 7, 8, 10)  Use of checklist and guidance (RQ 11, 12) Yes X X

See Appendix A for the teacher and principal survey instruments, and Appendix B for the Recruitment Materials. Appendix C contains the secondary data elements to be provided b y NM PED to ED Contractors

. Appendix D summarizes the suggestions received ED’s contractors for this study and the responses to those suggestions. Th e data collection involved in this study is17

authorized by

the Educational Sciences Reform Act (ESRA) of 2002. Please see Appendix E for the ESRA. Appendix F provides the IRB approval by Human Subjects Protection Committee at the research organization carrying out the study

. Appendix G provides the written consent forms for study participants. 

### 2. **How,	by	Whom,	and	for	What	Purpose	the	Information	I s	to	Be	Used** 

ED’s contractor for RELSouthwest will analy ze the data to be collected through this study using statistical models and procedures that are preapproved by the Institute of Education Sciences (IES). The contractor will then summarize the findings in a report that will undergo review for quality and relevance b y the National Center for Education Evaluation and Regional Assistance’s (NCEE’s) external review contractor. After the report has undergone IES review, findings will be presented to members of the NM PED (the primary audience) and published through IES for educators and education researchers (secondary audience). The primary

audience f

or this study

is NM PED, and other state and local agencies considering ways to improve feedback fro m principals to teachers. The results from all of the research questions will be used by New Mexico to decide whether to adopt the checklist and video as a mandatory or recommended element of teacher evaluation. The study will also offer detailed informati on to NM PED about teachers’ use of professional development (PD), their perceptions of it, the PD topics they

pursue, and PD recommended to them. NM PED currentl y has no way to gather this information, because PD is arranged for and determined by local school districts in the state. In addition, the study will provide NM PED with detailed feedback from principals and teachers about the state’s teacher evaluation system, supports for it (or lack thereof), and suggestions for improvement. Answers to the expl oratory and implementation research questions could help NM PED fill gaps in their knowledge regarding district and school practices in professional development and teacher feedback. More generall y, the results of the study will provide concrete evidenc e to state policy makers regarding the utility

of low

- cost, eas y to implement conversation checklists , scripts, and testimonial videos if they can encourage teachers both to seek out appropriate professional development opportunities and improve their instr uction

. Should this New Mexico Guide improve teachers or principals’ description of the structure and content of the feedback conversation and perceptions of the feedback process or subsequent classroom observation scores, it suggests that other localities may benefit from developing customized checklists built around their evaluation system but incorporating the core theories of effective performance feedback. If the checklist and video do not influence then it suggests that the checklist and video do not add value to the pre

- existing professional development about classroom observation , and an examination of more intensive interventions is warranted.18

To provide this information to the primary audience and the secondary audience, ED’s contractor must col lect the four forms of data described in Table 2. Namel y, the contractor must collect **extant data** from NM PED that includes:
 teacher and principal contact information for ED’s contractor to recruit participants into the study ;  principals’ NM TEACH Observation Rubric ratings of teachers ’ classroom to determine if classroom ratings improve following distribution of the treatment and control guide (research question 5 ) ;  student

- level achievement data to determine if student achievement improves in tre atment and control schools following distribution of the guide (research question 9 ) ; and
 school

- level, teacher

- level, and principal

- level characteristics to determine if those characteristics moderate take

-up and perceptions of the treatment and the control group guide (research question 10 ). We estimate that NM PED will devote a total of 20 hours to provide ED’s contractors with contact information and then another 105 hours from NM PED for school

- , teacher

- , and principal

- level data. The contractor must also administer a **principal survey** and a **teacher survey** at two time points to determine if there were changes in study participants’:
 Perceptions of the post

- observation conferences following receipt of the treatment and control guide (research questions 1 and 2).
 Behaviors in the post

- observation conferences following receipt of the treatment and control guide (research question 3 and 4 ).
 Perceptions and use of the guidance they received (research questions 6, 7, 8, 11, and 12 ). The surveys will take up to 30 minutes to complete, and they will be administered to all study participants in spring 2015 and again in spring 2016. For the purposes of this OMB package, it is assumed that 600 principals and 3,000 teachers will complete th e spring 2015 and the spring 2016 surveys . 

### 3.		Use	of	Information	Technology The data collection will be conducted in a manner that utilizes technology to reduce respondent burden and improve accuracy . Where feasible, RELSW researchers will collect al l possible data from administrative sources rather than through primary data collection . For example, electronic school records on teachers, principals and schools will be gathered from NM PED and transferred directl y to ED contractors using secure file transfer procedures , and thus reduce the number of questions posed to teachers and principals in their surveys . Clear instructions will be19

provided on the method of data transfer from NM PED to ED Contractors

. For data that can onl y be obtained directl y from principals and teachers, ED contractors will employ MMICTM (Multim ode Interviewing Capability ), a comprehensive information system developed b y RAND , to administer the surveys. 3 MMIC is used to manage the whole data collection process from questionnaire design, sample management, and fieldwork monitoring. Using MMIC, ED contractors will email to study participants a link to online surveys. T o reduce the burden on respondents, the MMIC software is flexible, and allow s survey respondents to participate using a multitude of devices like computers, PDAs, and Smart phones , and to switch between devices while completing the survey . When requested, questionnaires will be transmitted to and from the respondent b y fax. A telephone number to a staffe d MMIC help desk and electronic mail address are included in the questionnaire if anyone has questions. These procedures are designed to minimize the survey burden on respondents. 

### 4.		Efforts	to **Avoid Duplication**  **of	Effort** In an effort to avoid duplication of effort, this study will use extant administrative records where possible to understand the impact of the Conversation Protocol. As described in Table 2, ED ’s c ontractors will collect principals’ rating scores of teachers on NM TEACH Observation Rubrics from secondary

data sources collected by NM PED, rather than ask principals to provide this information in surveys. As a part of this same secondary data request, ED’s contractors will also collect school

- level characteristics such as size, level (elementary, middle, high), accountability status, teacher

- level characteristics such as degree earned, race, gender, and job title, principal

level characteristics such as degree earned, race, gender, and job title, and student

- level characteristics like student achievement to minimize the length of surveys administered directl y to principals and teachers. The only data collected that will be unique to this study are the w eb

based principal and teacher survey data , which are also described in Table 2. 

### 5. **Sensitivity	to	Burden	on	Small	Entities** The use of administrative records will reduce the burden on school educators by ensuring that only the minimum amount of original data is requested from schools in order to meet the objectives of this study

. Aside from surveys emailed directl y to participating principals and teachers, ED’ s contractors will not contact schools to request additional data. 

### 6.		Consequences	of	Less	Frequent	Collection	and	Obstacles	to	Burden **Reduction** The Education Science Reform Act of 2002 states that the central mission and primary function

3 For more information, see http://www.rand.org/labor/mmic. html20

of the regional education laboratories is to support applied research and provide technical assistance to state and local education agencies within their region (ESRA, Part D, section 174[f]). If the proposed data were not collected, REL Southwest would not be fulfilling its central mission to serve the states in the region and provide support for evidence

-based research. The research questions and the intervention contemplated in this study respond to the concerns raised b y NM PED, which is a cons tituent member of the REL Southwest. If the proposed data were not collected , NM PED and, by extension, other SEAs would not know whether a low

- cost informational intervention such as this one has an impact on the quality of principal feedback to teachers, and b y extension academic outcomes. This is a one

- time study

(i.e., not recurring) and therefore periodicity is not addressed. 

### 7.		Circumstances	Requiring	Special	Information	Collection There are no special circumstances. 

### 8. **Federal	Register	Announcemen t	and	C**  **onsultation	Outside	the	Agency a.	Federal	Register	Announcement**  A 6 0

- day Federal Register Notice was published on [DATE TO BE COMPLETED BY IES]. A 30

- day notice was published on [DATE TO BE COMPLETED BY IES ]

. **b.	Consultations	Outside	the	Agency** The evaluation team has consulted the Technical Working Group (TWG) of researchers with expertise in large

- scale randomization studies and impact evaluation that was formed to review studies under the Southwest REL . We have consulted the TWG on the overal l study design, data collection and survey

instruments. Major recommendations from the TWG are included in Appendix D

. Members of the TWG include: **Dan Goldhaber , Ph.D**  **.** Director, CALDER (National Center for Anal ys is of Longitudinal Data in Education Re search) Vice President, American Institutes for Research (AIR) Director, Center for Education Data & Research (CEDR), University of Washington Bothell Co

- Editor, Education Finance and Policy 3876 Bridge Way

N, Suite 201 Seattle, WA 98103 Ph: 206

- 547

- 1562 Fax: 206

- 547

- 164121

E

- mail: dgoldhab@uw.edu **Geoffrey Borman, Ph.D.**  Professor of Education, University of Wisconsin

—Madison Deputy Director of the University of Wisconsin's Predoctoral Interdisciplinary Research Training Program Senior Researcher, Consortium for Policy Research in Education. 348 Education Building 1000 Bascom Mall Madison, WI53706

- 1326 Ph: 608

- 263

- 3688 Fax: 608

- 265

- 3135 E

- mail: gborman@education.wisc.edu **Johannes M. (Hans) Bos**  **, Ph.D.** Vice P resident and P rogram Director, International Development, Evaluation, and Research (IDER) P rogram American Institutes for Rese arch 2800 Campus Drive, Suite 200 San Mateo, CA 94403 Ph: 650

- 843

- 8100 Fax: 650

- 843

- 8200 E

- mail: jbos@air.org **W. Steve**  **n Barnett** **, Ph.D.** Board of Governors Professor and Director of the National Institute for Early Education Research Rutgers University 73 Easton Avenue New Brunswick, NJ 08901 Ph: 848

- 932

- 4350 x23132 Fax: 732

- 932

- 4360 E

- mail: sbarnett@nieer.org
**c.	Unresolved	Issues** There are no unre solved issues.22



### 9.		Payments	or	Gifts	to	Respondents ED contractors propose to provide teachers and principals a $ 25 gi ft certificate upon completion of each of two 30

- minute surveys (baseline survey before the intervention, and follow

-up survey after the intervention), for a total of $ 50 per person. These amounts were set based on the hourl y wage rate of principals ($44.23/hour) and teachers ($32.52/hour) and the estimated length of the survey . Incentives will be distributed electronically (i.e., a link to a gift card) after respondents complete the surveys . The only other data for the study

will be obtained from NM PED, which has already agreed in concept to provide such data and will not be compensated for the time required to draw together the data files for transmission to ED’s contractor . As a constituent member of the REL Southwest, NM PED identified this topic as of interest of them, and thus the study is being done at their request. 

### 10.		Assurances	of	Confidentiality The data collection efforts that are the focus of this clearance package will be conducted in accordance with all relevant federal regulations and requirements. The Southwest REL will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I , Part E, Section 183 requires “ All collection, maintenance, use, and wide dissemination of data by

the Institute ” to “ conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h). ” These citations refer to the Privacy

Act, the Famil y Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. All survey

responses will be kept strictly confidential, and will only b e use d for the purpose of the study . No one at the school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. The project has been approved b y RAND ’s Human Subjects Protection Committee, which serves as RAND’s Institutional Review Board ( IRB00000051 ) to review research involving human subjects. RAND is registered with the Office of Human Research Protection (OHRP) as a research institution ( IORG0000034 )

. RAND's Federal wide Assurance for the Protection of Human Subjects (FWA00003425, effective until July 1, 2018) serves as our assurance of compliance with federal regulations. In addition, for student information, the data collection efforts will ensure that all individually identifiable information about students, their academic achievements, their families and information with respect to individual schools, shall remain confidential in accordance with section 552a of Title 5, United States Code, the confidentiality standards of subsection (c) of this23

section, and sections 444 and 445 of the General Education Provision Act. The study will also adhere to requirements of subsection (d) of section 183 prohibiting disclosure of individually identifiable information as well as making the publishing or inappropriate communication of individually

identifiable information by employees or staff a felony

. ED contractors will protect the confidentiality of all information collected for the study and will use it for research purposes only . No information that identifies any study participant will be released publicly . Information from participating institutions and respondents will be presented at aggregate levels in reports. Information on respondents will be linked to their institution ty pe (e.g., elementary

schools vs. middle schools) but not to any individually

identifiable information. No individually

identifiable information will be maintained b y the study team upon study completion

. Respondents will be assured that all information identifying them and the s chool will be kept confidential. All members of the study team have obtained their certification on the use of human subjects in research

. The following safeguards are routinel y employed at RAND , the contractor that will execute this study, to carry out confidentiality

assurances:

 All employees at ED’s contractor working on this project will sign a confidentiality pledge emphasizing its importance and describing their obligations under it (please see Appendix H for the confidentiality

pledge).  All research projects that have access to identifiable private or proprietary data need to have a Data Safeguarding Plan reviewed and approved by RAND’s Human Subjects Protection Committee. The Data Safeguarding Plan includes information on who is responsible for data safeguarding, the ty

pes of sensitive information to be transferred and stored, the mode of data transfer, client and respondent agreements, disclosure risks, audit and monitoring plans, and the procedures to be employed for data safeguarding.
 Any electronic transmission and sharing of individually identifiable data will be encry pted. This procedure will prevent anyone without permission to access and enter the data s ys tem
 Access to the data shall be limited to the minimum number of individuals necessary to achieve the approved purpose and to those individuals on a need

- to
- know basis only

.  Id entifiable data wil l be stored in a locked container when not in use

. We will store original and derivative data files only on disks (e.g. servers, local hard disks) that are not routinel y backed up. We will keep all hardcop y materials containing sensitive data in a locked file cabinet when not in use.
 When no longer needed, we will discard sensitive output in a shredder or sensitive

- waste container. We will destroy all individual linkages to data after a respondent ceases participation in the project. Also, the RELstudy team has submitted to the NCEE security officer a list of the names of all people who will have access to respondents and data. The contractor, on behalf of ED, will track24

new staff and staff who have left the study and ensure that signatures will be ob tained or clearances revoked, as necessary

. ED’s contractor will make certain that all data are held in strict confidentiality, as just described, and that in no instance will responses or data be made available except in in aggregate statistical form. The following statement will appear on all letters to respondents on data collection: Per the policies and procedures required b y the Education Sciences Reform Act of 2002, Title I , Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district , school , or individual. The contractor will not provide information that identifies you or your di strict to anyone outside the study

team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony

. 

### 11.		Justification	for	Sensitive	Questions Teachers completing the survey will be asked about the frequency, structure and content of the post
- observation feedback conference, the type of professional development courses they were recommended to take and completed, the quality of the professional devel opment courses offered by the school district, their opinions about the effectiveness of the school leadership and the teacher evaluation system. Teachers will also be asked more sensitive questions, such as to rate their own performance and to rate the q uality of the school leadership. These questions are sensitive because teachers would risk embarrassment or damage to their reputation or employability were their answers to be revealed. As noted above, all responses will be kept confidential and the data safeguarded. The questions regarding teachers’ own ratings of performance are important to anchor the principal as a strict or lenient rater, and the questions regarding the quality

of school leadership are an important control variables to establish the school environment. Principals completing the survey will be asked about the structure and content of the post

- observation feedback conference, the type and quality of professional development offered b y the school district, the training they received on the observation rubric, and their opinions about the teacher evaluation system
. Principals will also be asked to answer similar sensitive questions such as to rate the quality of his/her teachers. While all responses will be kept confidential, these questions are important to establish the environment of the school. In addition, participants will be informed that their responses are voluntary, and they may decline to answer any question. 

### 12. **Estimates	of	Hour Burden**  There are three components for which ED’s contractor has calculated hours of burden for this clearance package: recruitment activities , extant data provided b y the state to ED’s contractor,25

and survey

data collected from study participants by

ED’s contractor. Table 3 shows the hourl y burden o verall and for all three component s . The total burden associated with this study, across three study years, is 5, 3 5 2 .75 hours , with an annualized burden of 1, 784

. 25 hours over three ye ars . The recruitment burden is 1, 10 7

. 7 5 hours , the extant data collection burden is 105 hours, and the survey

data collection burden is 4,140 hours. The annualized number of responses is 10, 800 (for a total of 31,743 across all three ye ars). **Table 3**  **. Total Estimated Hourly Burden**

**Instrument Person** 

**Incurring Burden**  **Number**

**of responden**  **ts Response** 

**s per Responde**  **nt Total**  **Respons es**  **Hours**

**per Respons**  **e Total**  **Burden Hours**  **Recruitment** Principal and Teacher Contact Information (name, email) State Data Manager 1 3 3 10 30 F irst Principal contact (e

- mail ) Principal 855 1 855 0.05 42.75 Round 1 Follow

-up for nonresponding Principal Principal 500 1 500 0. 05 25 Round 2 Follow

-up for nonresponding Principal Principal 400 1 400 0. 05 20 Round 3 Follow

-up for nonresponding Principal Principal 300 1 300 0. 05 15 First Teacher contact (e

- mail ) Teacher 6 , 000 1 6 , 000 0.05 300 Round 1 Follow

-up for nonresponding Teacher Teacher 5 , 000 1 5 , 000 0.05 250 Round 2 Follow

-up for nonresponding Teacher Teacher 4 , 500 1 4 , 500 0.05 225 Round 3 Follow

-up for nonresponding Teacher Teacher 4,000 1 4,000 0.05 200

**Subtotal ---**  **21, 556**  **--- 21,**  **558 10**  **.4 (across**  **respons es)**  **1, 10**  **7 .75** 
**Extant Data Collection** 26

**Instrument Person** 

**Incurring Burden**  **Number**

**of responden**  **ts Response** 

**s per Responde**  **nt Total**  **Respons es**  **Hours**

**per Respons**  **e Total**  **Burden Hours**  Teacher and School Administrative Data State Data Manager 1 3 3 15 45 Teacher Observation Score Data State Data Manager 1 3 3 20 60
**Subtotal ---**  **1 ---**  **6 35**  **(across respons**  **es) 105**  **Survey Data Collection**  Principal Consent Principal 600 1 600 0.15 90 Teacher Consent Teacher 3 , 000 1 3 , 000 0.15 450 Wave 1 Principal Survey Principal 600 1 600 0.5 300 Wave 2 Principal Survey Principal 600 1 600 0.5 300 Wave 1 Teacher Survey Teacher 3,000 1 3,000 0.5 1 , 500 Wave 2 Teacher Survey Teacher 3,000 1 3,000 0.5 1,500 **Subtotal ---**  **10, 800**  **--- 10**  **, 800**  **2.3 4,140** 
**Totals**
--- **32,357 ---**  **32,36 4**  **47 .7**  **(across respons**  **es) 5**  **, 3**  **5 2**  **.75**

**Burden	for Study	Recruitment**  Our recruitment strategy

will focus on both New Mexico public school principals and New Mexico public school teachers . ED’s contractor will conduct recruitment in a top

- down approach, with principal recruitment preceding teacher recruitment. Our burden table is a conservative estimate of the highest potential burden with this recruitment approach, but ED’s contractor anticipates that fewer districts and schools will actually be involved in recruitment activities

. First, ED’s contractor will collect contact information for principals and teachers from the state data manager at NM PED , with a second collection 2 months later to ensure that the information is up to date, at a total burden of 20 hours for NM PED

. P rincipals will be contacted b y email, with an estimated burden of 3 minutes to read and respond to the email (0.05 hours) . We will re

contact non
- responding principals up to three times via email and then twice by letter mailed to their school

. The target sample size for initial contact with principals is 855 principals with an27

estimated response rate of 7 0 percent ( 600 principals ) . Once the principals have been enrolled in the study, we will then contact up to 10 teachers in the school by email, with an estimate d burden of three minutes per teacher per email to read and respond to the email. We will re

- contact non
responding teachers up to three times via email and then twi ce by letter mailed to their school

. The target sample size for initial contact with teachers is at most 6 , 000 teachers, with an estimated sample of at most 3 , 000 teachers consenting to participate in the study

. **Burden	for	Extant	Data	Colle ction**  The total estimated burden for NM PED to compile and transmit secondary data to ED’s contractors is 105 hours. This calculation assumes one data manager works a collective total of 2.625 weeks on compiling the data request. **Burden	for	Survey	Data	Collection** The total estimated burden for survey data co llection for the impact and implementation study is 4,140 hours . Principals and teachers will each be consented into the study, and will be surveyed two times, once at baseline, and a second time one year later

. 

### 13.		Estimated	Total	Annual	Cost	Burden **to	Respondents	or	Recordkeepers Total	Cost	to	Respondents**  The total cost to respondents for the two components of this study

—recruitment activities and survey data collection

—is provided in Table 4

. The total respondent cost associated with this study is approximately

$ 184,590 . The annualized cost for each ye ar of the three

- ye ar study is $ 61,530 . The recruitment cost is $ 37,152 ; the respondent cost for the data collection is $ 1 47,438

.
**Table 4**  **. Total**  **Cost to Respondents**
**Instrument Person**  **Incurring Burden**  **Total**

**Burden Hours**  **Hourly**

**Wage Rate**  **Total**

**Respondent Cost**  **Recruitment** Principal and Teacher Contact In formation (name, email) State Data Manager 20 $45 $900 F irst Principal contact (e

- mail ) Principal 42.75 $44.23 $1,891 Round 1 Follow

-up for nonresponding Principal Principal 25 $44.23 $1,106 Round 2 Follow

-up for nonresponding Principal Principal 20 $44.23 $885 Round 3 Follow

-up for nonresponding Principal Principal 15 $44.23 $663 First Teacher contact (e

- mail ) Teacher 300 $32.52 $9,756 Round 1 Follow

-up for nonresponding Teacher 250 $32.52 $8,13028 **Instrument Person**  **Incurring Burden**  **Total**

**Burden Hours**  **Hourly**

**Wage Rate**  **Total**

**Respondent Cost**  Teacher Round 2 Follow

-up for nonresponding Teacher Teacher 225 $32.52 $7,317 Round 3 Follow

-up for nonresponding Teacher Teacher 200 $32.52 $6,504 **Subtotal ---**  **1,0 9**  **7.75 ---**  **$37,152 Survey**  **Data Collection** Principal Consent Principal 90 $44.23 $3,981 Teacher Consent Teacher 450 $32.52 $14,634 Wave 1 Principal Survey Principal 300 $44.23 $13,269 Wave 2 Principal Survey Principal 300 $44.23 $ 13,269 Wave 1 Teacher Survey Teacher 1 , 500 $32.52 $48,780 Wave 2 Teacher Survey Teacher 1,500 $32.52 $ 48,780 Teacher and School Administrative Data State Data Manager 45 $45 $2,025 Teacher Observation Score Data State Data Manager 60 $45 $2,700 **Subtotal ---**  **4,245 $**  **147,438 Totals** 
--- **5,342.75 $184,590**  Note: The hourly wage rates used to calculate total respondent cost for principals are based on the salary schedule of Albuquerque High school principals with more than 10 years of experience (http://www.aps.edu/human

resources/salary

- schedules/salaries/spe

- salary

- schedule ). For teachers, the salary is based on the salary schedule of Albuquerque Level III teachers with a doctorate (http://www.aps.edu/human

-resources/salary

- schedules/salaries/at3

- salary

- schedule ). For the data manager hourly wage rates are based on ED’s contractor’s knowledge of New Mexico wage rates. 

### 14.		Estimated **Annualized Cost	to	the	Federal	Government**  The total cost to the federal government for work conducted over all four years is $1,133,000 and the estimated annualized cost to the federal government for each ye ar of the study is $ 283,250

. 

### 15.		Program	Changes	or	Adjustments This is a new study

. 

### 16.		Plans	for	Tabulation,	Publication **,	and	Schedule	for	Project**
**a .	Tabulation	Plans**  All results for RELrigorous studies will be made available to the public through peer

- reviewed evaluation reports that are published by IES. The datasets from these rigorous studies will be29

turned over to the REL’s IES project officer. After the study report is finalized, ED’s contractor will prepare restricted

-use data files in accordance with NCES standards. These files will contain all the primary survey data collected for the study with all personal identifiers removed. Thorough documentation will be provided for each data file, including a detailed codebook and explanations of the unit of observation, weights, and methods for handling missing data. These data will become IES restricted

-use data sets requiring a user’s license that is applied for through the same process as NCES restri cted

-use data sets. Even the ED contractor would be required to obtain a restricted

-use license to conduct any work with the data beyond the original evaluation. **b.	Publication Plans**  All results for RELstudies are made available to the public through peer

- reviewed reports that are published b y IES. The data sets from these studies will be turned over to the REL ’s IES project officer. These data may become IES restricted

-use data sets requiring a user’s license that is applied for through the same process as National Center for Education Statistics restricted

-use data sets (see http://nces.ed.gov/pubs96/96860rev.pdf for procedures related to obtaining and using restricted

-use data sets)

. The REL contractor also would be required to obtain a restricted

use license to conduct any work with the data beyond the original report
. The answer to RQ1

– RQ4 will indicate whether the treatment guide relative to the control guide improved or otherwise changed principals’ and teachers’ perceptions and execution of post

observation conferences. To anal yz e the impact of the feedback protocol on teacher and principal outcomes, we will examine differences in outcomes between teachers or principals who were in schools that were randomly assigned to receive the treatment guide with their counterparts in the control group who were assigned to receive the control guide. The data for the evaluation is hierarchical, with students and teachers nested within school (or principals), nested within districts. Because units within a group are not statistically independent, ED’s contractors will use hierarchical linear modeling to account for the statistical dependence of the error terms. Randomizati on will occur at the principal/school level within district, and ED’s contractors will stratify the random assignment by school level. ED’s contractors will estimate the intent

- to
- treat effect in a two

- level model for principals and a three

- level model for teachers. Using similar modeling techniques, the answer to RQ5 will indicate whether the treatment guide, relative to the control guide, improved principals’ subsequent ratings of teachers’ classroom instructional techniques. For each outcome in the imp act anal ys is (RQ1

- 5), ED’s contractor will first construct the outcome measure b y using exploratory factor analysis, taking the measure directly from the survey

or using the measure from Observation Rubric administrative data. ED’s contractor will then es timate six regressions, one for each of the following outcomes:30

 Good Practices Index (measure of post

- observation feedback conference content and structure)
 Feedback Conversation Perceptions Index (measure of the perceived quality of feedback)
 Average numb er of minutes it takes to complete the post

- observation conference
 PD is tailored to needs identified in formal classroom observation
 Fra ction of targeted Professional D evelopment that was recommended b y the principal and taken b y the teacher
 Domain leve l Observation Rubric score
 Observation proficiency level Analysis for RQ6 and RQ7 will compare all principals and all teachers in the treatment group to their counterparts in the control group. These questions will assess the fidelity of implementation, the enacted contrast in service between the control and treatment group, and will help NM PED understand the likely proportion of users who would take up a guide where NM PED to issue one to school staff. Analysis for RQ8 through RQ9 will go beyond the hy pothesized first

- order impacts of guidance on post
- observations conferences and classroom instruction to understand what factors, if any , moderate the use of the guidance (RQ 9) and whether the feedback guidance impacts student achievement, which is the ultimate outcome of interest. However, RQ 8 and 9 are framed as exploratory

because ED’s contractors anticipate that sample size limitations and a single school ye ar of implementation may be insufficient to detect statistically significant effects at the principal

- and teacher

- subgroup level and on student achievement. These research questions will employ the same statistical model specifications for the impact research questions, with the introduction of interaction terms to assess differences in responses o f principals and teachers according to their membership in the subgroup of interest. Finally, descriptive anal ys e s for RQ10 and 11 will provide NM PED feedback about how the two ty pes of guidance might be improved in future use and barriers to its use. This will allow NM PED, and potentially

other capacity- constrained State Education Agencies, to adopt a revised version of feedback guidance and professional development for users of the guidance that will further enhance feedback and subsequent classroom in struction. No responses or data will be reported for individual staff members, students, or schools. Reported data will contain no fewer than four cases per reported table cell to protect confidentiality

and mask individually

identifiable data. **c.	Project Time	Schedule**  The timeline for the activities in this project, including data collection, analyses and reporting are31

in Table 5

. **Table 5** 

**– Project Time Schedule**  Documentation of Institutional Review B oard approval May 1, 2014 Draft Office of Management and Budget (OMB) package August 15 , 2014 Submit 60 day

FRN TBD Submit 30 day

FRN TBD Draft proposal approved by ED March 12, 2014 Final proposal approved by ED August 15 , 2014 Expected OMB Clearance Date April 1, 2015 Start Teacher Recruitment April 15, 2015 Start Principal Recruitment April 15, 2015 Complete Teacher Recruitment and Collect Wave 1 Survey August 7, 2015 Complete Principal Recruitment and Collect Wave 1 Survey August 7, 2015 Complete random Assignment August 15, 2015 Distribute Conversation Protocol to Principals at Treatment Schools and Principal Control Guide to Principals at Control Schools August 15, 2015 Distribute Conversation Protocol to Teachers at Treatment Schools and Teacher Control Guide to Teachers at Control Schools 1 st day of School, Fall 2015 Collect First round of NM TEACH Observation rubric scores December 1, 2015 Collect Second round of NM TEACH Observation rubric scores February 1, 2016 Collect Wave 2 Teacher Survey May 1, 2016 Collect Wave 2 Principal Survey May 1, 2016 Collect spring 2016 student achievement data from NM PED June 15, 2016 Complete Data Anal ys es December 31, 2016 Draft Making Impact Report March 1,2017 Revised Making Impact Report May 1,2017 Draft Stated Briefl y Report July 1, 2017 Final Making Impact Report approved by ED September 1,2017 Revised Stated Briefly

Report September 1,2017 Restricted Use Files October 15, 2017 Final Stated Briefl y Report approved by ED November 30 , 2017


### 17. **Display	of Expiration	Date**  Approval not to display

the expiration date for OMB approval is not requested. 

### 18.		Exceptions	to	the	Certification	Requirement No exceptions to the certificate statement are being sought.32 

### R **EFERENCES** Ball, D.L. and Cohen, D.K. (1999). Developing practice, developing practitioners: Toward a practice

-based theory

of professional education. In L .D. Hammond and G. Sy kes (Eds.), Teaching as the learning profession: Handbook of policy and practice, pp. 3

- 32. San Francisco: Jossey

- Bass. Bill and Melinda Gates Foundation. (2012). Gathering feedback for teaching: Combining high

quality observations with student surveys and achievement gains . Seattle, WA: Author. Available: http://www.metproject.org/downloads/MET_Gathering_Feedback_Practioner_Brie f.pdf Correnti, R. (2007). An empirical investigation of professional development effects on literacy instruction using daily

logs. Educational Evaluation and Policy Analysis , **29** (4): p. 262

- 295. Correnti, R. and Rowan, B. (2007). Opening up the black box: Literacy instruction in schools participating in three comprehensive school reform programs. American Educational Research Journal, **44** (2), p. 298

- 338. Garet, M.S., Porter, A.C., Desimone, L., Birman, B.F. and Yoon, K.S. (2001). What makes professional deve lopment effective? Results from a national sample of teachers. American Journal of Educational Research , **38** (4): p. 915

- 945. Hay nes, A. B., Weiser, T. G., Berry, W. R., Lipsitz, S. R., Breizat, A. H. S., Dellinger, E. P., ... & Gawande, A. A. (2009). A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine , 360 (5), 491

- 499. Heckman

- Stone, C. (2003). Trainee Preferences for Feedback and Evaluation in Clinical Supervision. Clinical Supervisor, 22(1) , 21

- 34. Hill, H.C. (2009). Fixing teacher professional development. The Phi Delta Kappan , **90** (7): 470

- 476. Hill, H.C., L . Kapitula, and K. Umland. (2011). A validity argument approach to evaluating teacher value

- added scores . American Educational Research Journal, **48** (3): p. 794

- 831. Little, J.W. (1993). Teachers’ professional development in a climate of educational reform. Educational Evaluation and Policy Analysis , **15** (2): 129

- 151 McCaffrey, D.F., Lockwood, J.R., Koretz, D., Louis, T.A. and Hamilton, L. (20 04). Models for value

- added modeling of teacher effects. Journal of Educational and Behavioral Statistics , **29** (1): 67

- 101. Mihaly, K., McCaffrey, D. F., Staiger, D. O., and Lo ckwood, J. R. (2013). A composite estimator of effective teaching. Seattle, WA: Bill and Melinda Gates Foundation. My ung, J. and Martinez, K. (2013). Strategies for enhancing the impact of post

- observation feedback for teachers. Stanford, CA: Carnegie F oundation for the Advancement of Teaching. Pianta, R.C. and Hamre, B.K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational33

Researcher, **38** (2): pp. 109

- 119. Penuel, W.R., Fishman, B.J., Yamaguchi, R. and Gallagher, L.P. (2007). What makes professional development effective? Strategies that foster curriculum implementation. American Journal of Educational Research , **44** (4): p. 921

- 958. Sartain, L ., Stoelinga, S. R. , and Brown, E.R. (2011). Rethinking teacher evaluation in Chicago: Lessons learned from classroom observations, principal

- teacher conferences, and district implementation . Chicago: Consortium on Chicago School Research. Sporte, S.E., Stevens, W.D., Heal ey, K., Jiang, J. and H. Hart. 2013. Teacher evaluation in practice: Implementing Ch icago’s REACH Students

. Chicago: Consortium on Chicago School Research. Supovitz, J.A. and Turner, H.M. (2000). The effects of professional development on science teaching practices and classroom culture. Journal of Research in Science Teaching, **37** (2), p. 963

- 980. Wilson, S.M. , and Berne, J. (1999). Teacher learning and the acquisition of professional knowledge: An examination of research on contemporary professional development. Review of Research in Education , 24: 173

- 209. Yeager, D. S., & Walton, G. M. (2011). Social

-psychological interventions in education They ’re not magic. Review of Educational Research , 81 (2), 267

- 301.34



### A **PPENDIX A**  **– Teacher	and	Principal**  **Survey Instrument** **s Spring	2015	School	Leader	Survey**  [[ 

### Version : On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2015 survey for school leaders about the feedback you give to teachers’ regarding their instruct ion as a part of the NM TEACH Educator Effectiveness System
. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to t eachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public schoo l principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and te achers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . Survey results will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provide d in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No on e at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants.** At the same time that we are emailing you, REL Southwest is also inviting all New Mexico public school school

- based principals to participate in this study. Up to 10 teachers per school are also being surveyed

. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educ ator Effectiveness System
.35

**Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will provide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your participation in this survey is completely voluntary. You can decide not to participate or to discontin ue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at { XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you have read and understood the information provided to you about your participation in this surve y. 

##  I have read and understood the informa tion and choose to participate. **Thank you very much for your help in this important study!**

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button
— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions
— or if you accidentally get disconnected from the survey
— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until you click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when yo u click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”).36

**Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the q uestions on the page. This is completely optional.

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Position	and	Job	Responsibilities** 

### [q1] Which of the following best describes your position as a school leader this school year (2014

- 2015)?
 If you are not a school leader, you may have received this survey in error. Please reply to the invitation email or call {XXX}. 

##  principal 

##  vice principal/assistant principal 

##  directo r 

##  dean 

##  other: _______________________ 

### [q2x] Including this school year (2014

- 2015), how many years have you been a school administrator? (Fill in each box.)
 “School administrator” includes principal, assistant principal, and other equivalent titles. 

### [q2xa] Years as a school administrator in your current school 

### [q2xb] Years as a school administrator in your current district (including current school) 

### [q2xc] Years as a school administrator, total, in any location (including other districts and current district) 

### [q3] In addition to your administrative responsibilities, do you currently (2014

- 2015) have official teaching responsibilities (that is, formally instruct K

- 12 students as the teacher of record in one or more classes)?



##  Yes 

##  No 

### [q4] Do you play a role in filling teaching positions at your school (for example, recruiting, interviewing, and/or hiring)? 

##  Yes 

##  No37

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Perceptions	of	the	Quality of	Your	School’s	Teaching	Staff	This	Year**  

### [q 5 ] Indic ate your level of agreement with the following statement: “ **Overall** , I was satisfied with the quality of the whole teaching staff in my school this year (2014

- 2015).” [in sert slider graphic here. tick marks for disagree strongly, disagree somewhat, agree somewhat, agree strongly] 

### [q 6 x] Use the slider to answer each item below. During this school year (2014

- 2015), how many teachers in your school

... None or almost none A few About half Most All or nearly all 

### [q 6 a] Are highly effective instructional leaders. [insert slider graphic here] 

### [q 6b ] Are struggling and/or ineffective instructional leaders. [insert slider graphic here] 

### [q 6c ] Had a good grasp of the subject matter they teach [insert slider graphic here] 

### [q 6d ] Engaged in collaborative conversations about teaching and learning on a weekly or biweekly basis [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Practices	When	Observing	Teachers	This	Year	(SY	2014 -**  **2015)	for	Evaluation Purposes** 
[q7] Thinking about the entire school year 2014

- 2015, please estimate the number of teachers you will formally evaluate yourself using the NM TEACH Observation Rubric {link}

. [enter number here that ranges from 0
– 50] [q8] Thinking about the entire school year 2014

- 2015, how many other school leaders bes ides yourself will formally evaluate teachers using the NM TEACH Observation Rubric {link} ?38

[enter number here that ranges from 0
– 5] 

### [q 9 ] Thinking about the entire school year 2014

- 2015, how often did you personally **formally observe**  teachers in your school building using the NM TEACH Observation Rubric {link}? 

### [q 8 ] Thinking about the entire school year 2014

- 2015, how often did you personally **formally observe**  teachers in your school building using the New Mexico Observation Rubric {link}? 

##  I or anoth er school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teachers **3 or more times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teacher **2 times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) **some teachers 2 times and other teachers 3 times or more**

. 

##  Other: ________________________________ 

### [q 10 ] Use the slider to estimate h ow long post

- observati on conferences typically t ook you during school year 2014

- 2015
. Think only of conferences after formal observations; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes to 60 minutes with 10 minute intervals] 

### [q 11 ] Thinking about the entire school year 2014

- 2015, how frequently did you provide **informal feedback** about teaching and learning to individual teachers
— e.g. walk

- throughs, wows and wonders. 

##  At least once a week. 

##  At least once a month. 

##  At least once per semester. 

##  Other: ___________________ 

##  Optional comment here: 

### [q1 2 x] Thinking about **all** your formal observations from school year 2014

- 2015, use the slider to indicate h ow much you agree or disagree with the following statements about your post
formal observation conferences with teachers. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 2 a] The **feedback session itself**, separate from professional development I may have recommended , helped teachers **improve instruction** 
. [insert slider graphic here] 

### [q1 2b ] In each conference, the **teacher** [insert slider graphic here]39

**brought documents** to the conference , documents such as the lesson plan, PDP. 

### [q1 2c ] In each conference, **I brought**  **documents** to conference such as the teacher’s report card and the most recent PDP. [insert slider graphic here] 

### [q1 2d ] In each conference, I identified **at least one positive practice**  that the teacher does **well**

. [insert slider graphic here] 

### [q1 2e ] In each conference, I identified **at least one**  **challenge** facing the teacher. [insert slider graphic here] 

### [ q12f] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine which instructional **practices** to praise or c ritique with the teacher. [insert slider graphic here] 

### [ q12g ] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine wh at **professional development**  to recommend to the teacher. [insert slider graphic here] 

### [q1 2h ] In each conference, I **ended** the conference on a **positive note**
. [insert slider graphic here] 

### [q1 2i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 2j ] In each conference, the teacher and I **mutually developed next steps**  for their instruction. [insert slider graphic here] 

### [q1 2k ] In each conference provided **specific** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2l ] In each conference I provided **actionable** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2m ] In each conference, **teachers committed to**  **specific set of next steps** to improve their instruction. [insert slider graphic here] 

### [q1 2n ] In each conference, I **spoke for almost all of the time**  during the conference. [insert slider graphic here] 

### [q1 2o ] There was a **high level of collaboration**  in feedback conferences. [insert slider graphic here] 

### [q1 2p ] There was a **high level of conflict** in [insert slider graphic here]40

feedback conferences. 

### [q1 2q ] It was **hard to provide negative**  **feedback** to teachers about their performance. [insert slider graphic here] 

### [q1 2r ] The large majority of teachers seemed to **trust and accept** my feedback. [insert slider graphic here] 

### [q1 2s ] **I feel positive**  about the feedback I gave teachers in conferences. [insert slider graphic here] 

### [q1 2t ] **I enjoyed**  most of the post

- observation feedback conferences. [insert slider graphic here] 

### [q1 2u ] I provided all teachers I reviewed with a **written or online summary**  of the observation with my comments on it

. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development Offered	to**  **Teachers	This	Year	(2014 -**  **2015)** The questions in this section ask about professional development for teachers.
 Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, institutes, p rofessional conferences, online courses, self

- directed learning, and so forth. 

### [q1 3 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the professional development experiences of teachers at your school this year (2014

- 2015, including summer 2014).
 Consider professional development offered by the district and other external sources as well as professional development offered by the school. The professional development options for teachers this year....
Don’t know Disagree

Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 3 a] Were mostly

developed by school leaders or teachers at my school. 

##  [insert slider graphic here] 

### [q1 3b ] Were mostly

job

- embedded rather than one

- time onl y sessions. 

##  [insert slider graphic here] 

### [q1 3c ] Were varied and numerous enough to allow me to customize my 

##  [insert slider graphic here]41

PD recommendations to teachers based on their performance on specific domains of the NM TEACH Observation Rubric.{link} 

### [q1 3d ] Helped teachers improve on elements measured by the NM TEACH Observation Rubric. 

##  [insert slider graphic here] 

### [q1 3e ] Helped teachers improve other elements of the NM TEACH evaluation besides the NM TEACH Observation Rubric (e.g., student achievement or “other” multiple measure). 

##  [insert slider graphic here]


### [q 13f] Were well aligned with the Common Core State Standards and/or curriculum based on these standards. 

##  [insert slider graphic here] 

### [q1 3g ] Have been convenient enough to promote teachers’ participation. 

##  [insert slider graphic here] 

### [q1 3h ] Were sufficiently resourced (for example, substitute coverage, funding to cover expenses, stipends) to allow teachers to participate in the professional development they need to do their jobs successfull y. 

##  [insert slider graphic here]


### [q1 3i ] District

- provided professional development offerings (workshops, inservices, online courses, etc.) have enough slots and availability to accommodate all teachers who wish or need to participate. 

##  [insert slider graphic here]


### [q1 3j ] Teachers have had easy access to a catalog of professional development opportunities aligned with the NM TEACH Observation Rubric. 

##  [insert slider graphic here]


### [q1 4x ] Thinking about the professional development that teachers at you r school have participated in this year (2014

- 2015, including summer 2014), use the slider to indicate how useful do think each of the following has been for improving teacher effectiveness

. Not Applicable Not at all

useful Slightly

useful Moderately

useful Very useful42



### [q1 4 a] Workshops or inservices for teachers at your school only (typically on

- site) 

##  [insert slider graphic here] 

### [q1 4 b] Workshops, inservices, institutes, or conferences organized by your district for teachers from multiple schools 

##  [insert slider graphic here] 

### [q1 4 c] Online professional development offered by or through your district 

##  [insert slider graphic here] 

### [q1 4 d] District

- or state
- sponsored induction program, mentoring, or other professional development activities for new or beginning teachers 

##  [insert slider graphic here] 

### [q1 4 e] Instructional coaching (provided by school

- based coaches or district coaches) 

##  [insert slider graphic here] 

### [q1 4f] Professional development led by me or a school leader such as study groups or s chool

- wide s taff meetings 

##  [insert slider graphic here] 

### [q1 4g ] School

- based teacher collaboration (grade

- level or subject

- area teams, professional learning communities, study groups, etc.) 

##  [insert slider graphic here] 

### [q1 4h ] Videos of sample lessons 

##  [ insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Training	You	Received	A bout	NM	TEACH	to	Date**  

### [q1 5 x] What types of training have you received to date regarding the NM TEACH Educator Effectiveness system ? (Check all that apply.) 

##  [q1 5 a] [exclusive option]No professional development received 

##  [q1 5 b] Information on the NM TEACH Educator Effectiveness system (overall purpose, specific components, consequences for teachers, r elationship to professional development, and so forth) 

##  [q1 5 c] How to use the NM TEACH Observation Rubric43



##  [q1 5 d] How to provide feedback to individual teachers after formal classroom observations 

##  [q1 5 e] How to enter scores from NM TEACH Observation Rubric into Reflect , the Teachscape software 

##  [q1 5 f] How to use Reflect, the Teachscape software 

##  [q1 5 g] How to interpret a teachers’ NM TEACH Summative Evaluation Score 

##  [q1 5 h] How to give NM TEACH Observation Rubric ratings consistent with those given by other raters 

##  [q1 5 i] How to understand and make use of NM TEACH Summative Evaluation Score to help improve teaching at my school 

##  [q1 5j] How to direct teachers to professional development based on their NM TEACH Summative Evalua tion Score 

##  [q1 5 k] Other: 

### [q1 6 ] [if 15b

- k ==yes] Who sponsored the training you attended about NM TEACH Educator Effectiveness system ? Mark as many as apply. 

##  State Public Education Department 

##  My school district 

##  Other: __________________ 

##  Don’t know 

### [q1 7 ] [if 15b

- k ==yes] In total, how many minutes of training did you receive this year (2014

- 2015, including summer 2014) about how to provide feedback to teachers

. 

##  0 minutes 

##  15 minutes 

##  30 minutes 

##  60 minutes 

##  120 minutes 

##  240 minutes 

##  Other: 

### [q1 8 ] [if 15d==yes] Thinking about the training on how to conduct post

- observation conferences with teachers , indicate how useful you think the training was. 

##  Not at all useful 

##  Slightly useful 

##  Moderately useful 

##  Very useful 

### [q1 9 ] I understand how the overall teacher ratings work in the NM teacher evaluation system44

(NM TEACH). 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree Strongly

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q 20 x] Indicate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness System from the school year 20 14
- 2015. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 20 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider graphic here] 

### [q 20 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the students they teach. [insert slider graphic here] 

### [q 20 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance

. [insert slider graphic here] 

### [q 20 d] The NM TEACH Educator Effectiveness System requires so much of my time that it interferes with other important activities. [insert slider graphic here] 

### [q 20 e] I have a clear idea of what the NM TEACH Educator Effectiveness System views as “good instruction.” [insert slider graphic here] 

### [q 20f] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my teachers’ performance. [insert slider graphic here] 

### [q 20g ] In the NM TEACH Educator Effectiveness System , teachers who do not score “Effective” or better often leave teaching voluntarily. [insert slider graphic here]45



### [q 21 x] Indicate how much you agree or disagree with each of the following statements about the NM TEACH Observation Rubric. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 21 a] The 6 elements of Domain 1 (Planning and Preparation) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 b] The 5 elements of Domain 2 (Creating an Environment for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 c] The 5 elements of Domain 3 (Teaching for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ i nstructional practices. [insert slider graphic here] 

### [q 21 d] The 4 elements of Domain 4 (Professionalism) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q 21 e] Using the NM TEACH Observation Rubric {link} helps me to provide constructive, specific feedback to teachers to improve their instruction. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Teacher Retention	and	Improvement** 

. 

### [q 22 x] Over the past year (since approximately May 1, 2014), how many teachers at your school... None (0 teachers) 1 teacher 2 teachers 3 teachers More than 3 teachers Don’t

know46



### [q22 a] Moved from effective to highly effective o n the NM TEACH Summative Evaluation

. 

##       [q22b ] Were put on an **professional growth plan** or entered probationary

status 

##       [q22c ] Were **dismissed** (that is, had their district employment as a teacher terminated) due to receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      
[q22d] **Left teaching voluntarily** after receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. 

### [q2 3 ] Do you have any comments about the NM TEACH Observation Rubric?


### [q24] Do you have any comments about professional development available to teachers in your district? 

### [q2 5 ] Do you have any comments about training either your district or NM PED have offered about the NM TEACH teacher evaluation system?



### [q2 6 ] Do you have any feedback about this survey?47

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

--------------------------- ------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Thanks	Again** You have completed the survey and may now close it.48



### Spring	2016	School	Leader	Survey [[ 

### Version : On

- line administration. Wave 2.]] **Understanding	Your	Participation	i n	the	S**  **tudy** Welcome to the spring 2016 survey for school leaders about the feedback you give to teachers’ regarding their instruction as a part of the NM TEACH Educator Effectiveness System . This is a follow

- on to our fall 2015 survey to learn about the feedback you have provided to teachers this school year. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teache rs under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school prin cipals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers

. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers about their performance. Survey results will **not** be used to evaluate you, other teachers, or school leade rs. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, on ce you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other informatio n that could potentially be used to identify individuals or schools. **Participants.** REL Southwest originally invited all New Mexico public school school

- based principals to participate in this study. Only those principals who completed surveys in spring 20 15 are being sent this survey now in spring 2016. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help Ne w Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
.49

**Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions

. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the surv ey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o I RB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and c hoose to participate. **Thank you very much for your help in this important study!**

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button
— it will not work. **Returning to the survey itself.** If you would like to complet e the survey in multiple sessions
— or if you accidentally get disconnected from the survey
— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full p age you completed). You can keep returning until you click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save an d go to next page” or “go back to previous page”). **Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is50

completely optional.

-------- --------------------------------**[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Position	and	Job	Responsibilities** We would like to update our information about your position. 

### [q1] Which of the following best describes your position as a school leader this school year (2015

- 2016)?
 If you are not a school leader, you may have received this survey in error. Please reply to the invitation email or call {XXX}. 

##  principal 

##  vice principal/assistant principal 

##  director 

##  dean 

##  other:

_______________________ 

### [q2x] Including this school year (2015

- 2016), how many years have you been a school administrator? (Fill in each box.)
 “School administrator” includes principal, assistant principal, and other equivalent titles. 

### [q2xa] Years as a school administrator in your current school 

### [q2xb] Years as a school administrator in your current district (including current school) 

### [q2xc] Years as a school administrator, total, in any location (including other districts and current district) 

### [q3] In addition to your administrative responsibilities, do you currently (2015

- 2016) have official teaching responsibilities (that is, formally instruct K

- 12 students as the teacher of record in one or more classes)?



##  Yes 

##  No 

### [q4] Do you play a role in filling teaching positions at your school (for example, recruiting, interviewing, and/or hiring)? 

##  Yes 

##  No51

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Perceptions	of	the	Quality of	Your	School’s**  **Teaching	Staff	This	Year** 

### [q 5 ] Use the slider to indic ate your level of agreement with the following statement: “Overall, I was satisfied with the quality of the whole teaching staff in my school this year (2015

- 2016).” [insert slider graphic here. Tick marks at disagree strongly, disagree somewhat, agree somewhat, agree strongly.] 

### [q 6 x] Use the slider to answer the following items. D uring this school year (2015

- 2016), how many teachers in your school

... None or almost none A few About half Most All or nearly all 

### [q 6 a] Are highly effective instructional leaders. [insert slider graphic here] 

### [q 6b ] Are struggling and/or ineffective instructional leaders. [insert slider graphic here] 

### [q 6c ] Had a good grasp of the subject matter they teach [insert slider graphic here] 

### [q 6d ] Engaged in collaborative conversations about teaching and learning on a weekly or biweekly basis [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

----------------------------- ----- **Your	Practices	When	Observing	Teachers	This	Year	(SY	2015 -**  **2016)	for	Evaluation Purposes**  

### [q 7 ] Thinking about the entire school year 2014

- 2015, please estimate the number of teachers you will formally evaluate yourself using the NM TEACH Observation Rubric {link}

. [enter number here that ranges from 0
– 50] 

### [q 8 ] Thinking about the entire school year 2014

- 2015, how many other school leaders bes ides yourself will formally evaluate teachers using the NM TEACH Observation Rubric {link} ? [enter number here that ranges from 0
– 5] 

### [q 9 ] Thinking about the entire school year 2015

- 2016, how often did you personally **formally observe**  teachers in your school building using the NM TEACH Observation Rubric {link}?52



##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teachers **3 or more times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) all teacher **2 times**
. 

##  I or another school leader formally observed (for 15 + minutes using the NM TEACH Observation Rubric) **some teachers 2 times and other teachers 3 times or more**

. 

##  Other:

________________________________ 

### [q 10 ] Estimate h ow long post

- observation conferences typically t ook you during school year 2014

- 2015
. Think only of conferences after **formal observations** ; do not include informal feedback from classroom walk

- throughs. [i nsert slider graphic here. Tick marks for 0 minutes to 60 minutes with 10 minute intervals 

### [q 11 ] Thinking about the entire school year 2014

- 2015, how frequently did you provide **informal feedback** about teaching and learning to individual teachers
— e.g. walk

- throughs, wows and wonders. 

##  At least once a week. 

##  At least once a month. 

##  At least once per semester. 

##  Other: ___________________ 

##  Optional comment here: 

### [q1 2 x] Thinking about **all** your formal observations from school year 201 5
- 201 6 using the NM TEACH Observation Rubric , use the slider to indicate h ow much you agree or disagree with the following statements about your post
- formal observation conferences with teachers. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 2 a] The **feedback session itself**, separate from professional development I may have recommended , helped teachers **improve instruction** 
. [insert slider graphic here] 

### [q1 2b ] In each conference, the **teacher brought documents**  to the conference , documents such as the lesson plan, PDP. [insert slider graphic here] 

### [q1 2c ] In each conference, **I brought**  **documents** to conference such as the teacher’s report card and the most recent PDP. [insert slider graphic here] 

### [q1 2d ] In each conference, I identified at [insert slider graphic here]53

least one positive practice that the teacher does **well**

. 

### [q1 2e ] In each conference, I identified **at least one**  **challenge** facing the teacher. [insert slider graphic here] 

### [ q1 2f] In ea ch conference, **I used my scores**  o n the NM TEACH Observation Rubric for that teacher to determine which instructional **practices** to praise or c riticize with the teacher. [insert slider graphic here] 

### [ q1 2g ] In ea ch conference, **I used my scores** o n the NM TEACH Observation Rubric for that teacher to determine wh at **professional development**  to recommend to the teacher. [insert slider graphic here] 

### [q1 2h ] In each conference, I **ended** the conference on a **positive note**
. [insert slider graphic here] 

### [q1 2i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 2j ] In each conference, the teacher and I **mutually developed next steps** for their instruction. [insert slider graphic here] 

### [q1 2k ] In each conference I provided **specific** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2l ] In each conference I provided **actionable** feedback to teachers about their performance. [insert slider graphic here] 

### [q1 2m ] In each conference, **teachers committed to**  **specific set of next steps** to improve their instruction. [insert slider graphic here] 

### [q1 2n ] In each conference, I **spoke for almost all of the time**  during the conference **.** [insert slider graphic here] 

### [q1 2o ] There was a **high level of collaboration**  in feedback conferences. [insert slider graphic here] 

### [q1 2p ] There was a **high level of conflict** in feedback conferences. [insert slider graphic here] 

### [q1 2q ] It was **hard to provide negative**  **feedback** to teachers about their performance. [insert slider graphic here] 

### [q1 2r ] The large majority of teachers seemed to **trust and accept** my feedback. [insert slider graphic here] 

### [q1 2s ] **I feel positive** about the feedback I [insert slider graphic here]54

gave teachers in conferences. 

### [q1 2t ] **I enjoyed**  most of the post

- observation feedback conferences. [insert slider graphic here] 

### [q1 2u ] I provided all teachers I reviewed with a **written or online summary** of the observation with my comments on it

. [insert slider graphic here]



### [q 13a ] Have you ever received the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” , a guide to conducting formal feedback sessions? The first page of the Conversation Checklist looks like this:



##  Yes 

##  No 

### [q13b] Have you ever used the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” when conducting the formal feedback conference in the 2015

- 2016 school year ?



##  Yes 

##  No 

### [q 13c ] [If q 11b = yes)] With how many of your teachers, if any, have you used the
“ Conversation Checklist ”?55



##  None 

##  A few 

##  About half 

##  Most 

##  All or nearly all 

### [q13d ] [If treatment=1| (if treatment = 0 & q 13a = yes)] Did you view the testimonial video that was distributed along with the Conversation Checklist ? 

##  Yes 

##  No 

##  I don’t know 

### [ q13e ] Have you ever received the “ Guidance for New Mexico Principals’ About Provision of Feedback to Teachers ” document? It looks like this:



##  Yes 

##  No 

### [ q13f] If q 13e = yes)] With how many of your teachers, if any, have you used the advice included in the “Recommended Practices ” document ? 

##  None 

##  A few 

##  About half 

##  Most 

##  All or nearly all56



##  [q 13g ][If q 13c = none] Please select all of the reasons you have not used the Conversation Checklist

. 

##  [q 13g a] I have never seen the Conversation Checklist

. 

##  [q 13g b] I haven’t yet used the Conversation Checklist , but I plan to. 

##  [q 13g c] I got the Conversation Checklist too late for me to use in teacher conferences. 

##  [q 13g d] The Conversation Checklist looks burdensome
— e.g., too long, too hard. 

##  [q 13g e] I don’t need it; I already have a way of holding feedback conversations with teachers that works well. 

##  [q 13g f] I’ve read the Conversation Checklist and am using some concepts, but I’m not following the checklist. 

##  [q 13g g] Other: ______________________________________________________ 

### [q 13h ][If q13c ~= all & q13c ~= none] Which type of teachers did you use the
“ Conversation Checklist ” with? Select all that apply. 

##  [ q1 3 ha ] Only highly effective teachers 

##  [ q1 3 h b ] Only satisfactory teachers 

##  [ q1 3 h c ] Only struggling teachers 

##  [ q1 3 h d ] Only new teachers 

##  [ q1 3 h e ] Only experienced teachers 

##  [ q1 3 h h ] Other: _________________________________________________________


### [ q14a] [if 13C ~= none ] Use the slider to estimate h ow long post

- observation conferences typically t ook you during school year 2015

- 2016 when you used the “ Conversation Checklist. ” Think only of conferences after formal observations; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes, 15 min, 3 0 min., 45 min., 60 min, 75 min, 90 min] 

### [q 14b ] [if 

### q 13f~ = none ] Use the slider to estimate h ow long post

- observation conferences typically t ook you during school year 2015

- 2016 after you reviewed the “ Guidance for New Mexico Principals’ About Provision o f Feedback to Teachers ”

. Think only of conferences after formal observations; do not include informal feedback from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 minutes, 15 min, 30 min., 45 min., 60 min, 75 min,57

90 min] 

### [q15x ] If 13C ~= none ] Use the slider to indicate h ow much you agree or disagree with the following statements about your opinions about the Conversation Checklist this year (school year 2015

- 2016). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 5a ] The Conversation Checklist is **easy to use**  . [insert slider graphic here] 

### [q1 5b ] The Conversation Checklist **takes too**  **much time** to use. [insert slider graphic here] 

### [q1 5c ] The Conversation Checklist makes the feedback conversation feel **formulaic**

. [insert slider graphic here] 

### [q1 5d ] The Conversation Checklist provided me and teachers a **helpful structure** for feedback. [insert slider graphic here] 

### [q1 5e ] I provided **more critical feedback** to teachers than I had in prior conferences where I didn’t use the Conversation Checklist

. [insert slider graphic here] 

### [q1 5f] The Conversation Checklist helps teachers **commit to specific set of next steps**  to improve their instruction. [insert slider gra phic here] 

### [q1 5g ] I use the Conversation Checklist as the basis to create teachers’ **professional development plans**  (PDPs). [insert slider graphic here] 

### [q1 5h ] I use the Conversation Checklist as the basis to create **professional growth plans** (PGPs) for struggling teachers. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development Offered	to**  **Teachers	This	Year	(2015 -**  **2016)** The questions in this section ask about professional development for teachers.
 Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, institutes, professional conferences, online courses, self

- dire cted learning, and so forth. 

### [q1 6 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the professional development experiences of teachers at your school this year (2015

- 2016, including summer 2015).58

 Consider professional development offered by the district and other external sources as well as professional development offered by the school. The professional development options for teachers this year....
Don’t know Disagree

Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 6 a] Were mostly

developed by school leaders or teachers at my school. 

##  [insert slider graphic here] 

### [q1 6b ] Were mostly

job

- embedded rather than one

- time onl y sessions. 

##  [insert slider graphic here] 

### [q1 6c ] Were varied and numerous enough to allow me to customize my PD recommendations to teachers based on their performance on specific domains of the NM TEACH Observation Rubric.{link} 

##  [insert slider graphic here]


### [q1 6d ] Helped teachers improve on elements measured by the NM TEACH Observation Rubric. 

##  [insert slider graphic here] 

### [q1 6e ] Helped teachers improve other elements of the NM TEACH evaluation besides the NM TEACH Observation Rubric (e.g., student achievement or “other” multiple measure). 

##  [insert slider graphic here]


### [q1 6f] Were well aligned with the Common Core State Standards and/or curriculum based on these standards. 

##  [insert slider graphic here] 

### [q1 6g ] Have been convenient enough to promote teachers’ participation. 

##  [insert slider graphic here] 

### [q1 6h ] Were sufficiently resourced (for example, substitute coverage, funding to cover expenses, stipends) to allow teachers to participate in the professional development they need to do their jobs successfully. 

##  [insert slider graphic here]


### [q1 6i ] District

- provided professional development offerings (workshops, inservices, online courses, etc.) have enough slots and availability to accommodate all teachers who wish 

##  [insert slider graphic here]59

or need to participate. 

### [q1 6j ] Teachers have had easy access to a catalog of professional development opportunities aligned with the NM TEACH Observation Rubric. 

##  [insert slider graphic here]


### [q1 7x ] Thinking about the professional development that teachers at your school have participated in this year (2015

- 2016, including summer 2015), use the slider to indicate h ow useful you think each of the following has been for improving teacher effectiveness

. Not Applicable Not at all

useful Slightly

useful Moderately

useful Very useful 

### [q1 7 a] Workshops or inservices for teachers at your school only (typically on

- site) 

##  [insert slider graphic here] 

### [q1 7 b] Workshops, inservices, institutes, or conferences organized by your district for teachers from multiple schools 

##  [insert slider graphic here] 

### [q1 7 c] Online professional development offered by or through your district 

##  [insert slider graphic here] 

### [q1 7 d] District

- or state
- sponsored induction program, me ntoring, or other professional development activities for new or beginning teachers 

##  [insert slider graphic here] 

### [q1 7 e] Instructional coaching (provided by school

- based coaches or district coaches) 

##  [insert slider graphic here] 

### [q1 7f] Professional development led by me or a school leader such as study groups or s chool

- wide s taff meetings 

##  [insert slider graphic here] 

### [q1 7g ] School

- based teacher collaboration (grade

- level or subject

- area teams, professional learning communities, study groups, etc.) 

##  [insert slider graphic here] 

### [q1 7h ] Videos of sample lessons 

##  [insert slider graphic here]60

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Training	You	Received	A bout	NM	TEACH	to	Date**  

### [q1 8 x] What types of professional development have you received to date regarding the teacher evaluation system, NM TEACH? (Check all that apply.) 

##  [q1 8 a] [exclusive option]No professional development received 

##  [q1 8 b] Information on the NM TEACH Educator Effectivenes s system (overall purpose, specific components, consequences for teachers, relationship to professional development, and so forth) 

##  [q1 8 c] How to use the NM TEACH Observation Rubric 

##  [q1 8 d] How to provide feedback to individual teachers after formal classroom o bservations 

##  [q1 8 e] How to enter scores from NM TEACH Observation Rubric into Reflect , the Teachscape software 

##  [q1 8 f] How to use Reflect, the Teachscape software 

##  [q1 8 g] How to interpret a teachers’ NM TEACH Summative Evaluation Score 

##  [q1 8 h] How to give NM TEACH Observation Rubric ratings consistent with those given by other raters 

##  [q1 8 i] How to understand and make use of NM TEACH Summative Evaluation Score to help improve teaching at my school 

##  [q1 8j] How to direct teachers to professional development based on their NM TEACH Summative Evaluation Score 

##  [q1 8 k] Other: 

### [q1 9 ] [if 18b

- k ==yes] Who sponsored the training you attended about NM TEACH Educator Effectiveness system ? Mark as many as apply. 

##  State Public Education Department 

##  My school district 

##  Other:

__________________ 

##  Don’t know 

### [q20 ] [if 18b

- k ==yes] In total, how many minutes of training did you receive this year (2014

- 2015, including summer 2014) about how to provide feedback to teachers

. 

##  0 minutes 

##  15 minutes 

##  30 minutes 

##  60 minutes 

##  120 minutes 

##  2 40 minutes61



##  Other: 

### [q 21 ] [if 18d==yes] Thinking about the training on how to conduct post

- observation conferences with teachers , use the slider to indicate how useful you think the training was. Not at all useful Slightly useful Moderately useful Very useful [insert slider graphic here] 

### [q 22 x] Use the slider to indicate how much you agree or disagree with each of the following statements about your understanding of NM TEACH. Disagree strongly Disagree

somewhat Agree

somewhat Agree

strongly 

### [q 22 a] I understand how the overall teacher ratings work in the NM teacher evaluation system (NM TEACH). [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal Views	of	NM	TEACH**  

### [q 23 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the teacher effectiveness ratings that became available in school year 2015

- 2016. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 23 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider graphic here] 

### [q 23 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the students they teach. [insert slider graphic here] 

### [q 23 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance. [insert slider graphic here] 

### [q 23 d] The NM TEACH Educat or Effectiveness System requires so much of my time that it [insert slider graphic here]62

interferes with other important activities. 

### [q 23 e] I have a clear idea of what the NM TEACH Educator Effectiveness System views as “good instruction.” [insert slider graphic here] 

### [q 23 g] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my teachers’ performance. [insert slider graphic here] 

### [q 23 h] In the current NM TEACH Educator Effectiveness System , teachers who do poorly on their evaluation often leave teaching voluntarily. [insert slider graphic here]


### [q 24 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Observation Rubric. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q2 4 a] The 6 elements of Domain 1 (Planning and Preparation) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 b] The 5 elements of Domain 2 (Creating an Environment for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 c] The 5 elements of Domain 3 (Teaching for Learning) of the NM TEACH Observation Rubric {link} are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 d] The 4 elements of Domain 4 (Professionalism) of the NM TEACH Observation Rubric {link}are a useful way to measure the quality of teachers’ instructional practices. [insert slider graphic here] 

### [q2 4 e] Using the NM TEACH Observation Rubric {link} helps me to provide constructive, specific feedback to tea chers to improve their instruction. [insert slider graphic here]63

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Teacher Retention	and	Improvement**  

### [q2 5 x] Over the past year (since approximately May 1, 2015), how many teachers at your school...
None (0 teachers) 1

teacher 2

teachers 3

teachers More

than 3 teachers Don’t

know 

### [q25 a] Moved from effective to highl y effective o n the NM TEACH Summative Evaluation

. 

##       [q2 5b ] Were put on an professional growth plan or entered probationary

status 

##       [q2 5c ] Were **dismissed** (that is, had their district employment as a teacher terminated) due to receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      
[q2 5d] Left teaching voluntarily after receiving a low effectiveness rating on the NM TEACH Summative Evaluation 

##      

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Optional Comments	from	You**  The following four questions are optional if you have any feedback you wish to share. 

### [q2 6 ] [if treatment = 1] Do you have any suggestions to help us improve the Principal

Teacher Post

- Observation Conversation Checklist ? 

### [q27] Do you have any comments about professional development available to teachers in your district? 

### [q2 8 ] Do you have any comments about the NM TEACH Observation Rubric? 

### [q2 9 ] Do you have any comments about training either your district or NM PED have offer ed about the NM TEACH teacher evaluation system?64

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thanks	Again** You have completed the survey and may now close it.65



### Spring	2015	Teacher	Survey [[ 

### Version : Teachers. On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2015 survey for school leaders about the feedback you give to teachers’ regarding their ins truction as a part of the NM TEACH Educator Effectiveness System

. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System , and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {mo nth, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness System training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researcher s will link your survey responses to basic employment information, such as race, ethnicity and years of experience that REL Southwest researchers will obtain from NM PED. Survey results will **not** be used to evaluate you, other teachers, or school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey li nk provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you o r your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants** Ineach o f the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study Up to 10 teachers in your school were selected at random to be surveyed

. **Benefits.** Your completion of this survey gives you the opportuni ty to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator66

Effectiveness System
. **Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will provide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a p articipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that y ou have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate.. **Thank you very much for your help in this important study!**

--------------------- ------------------**[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” b utton at the **bottom** of the page. Do **not** use your browser’s back button
— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions
— or if you accidentally get disconnected from the survey
— you can return t o your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until you click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”).67

**Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

---------------------------------------- **[INTERNAL CODES]**

----------------------------------------------- [[ Treatment code (treatment): REL Southwest will pre

- identify via an internal code that will not be listed on the survey whether teacher respondent works at a school that belongs to the treatment group or th e control group. treatment = 1 indicates teacher is in the treatment group. treatment = 0 = control group. Unless otherwise noted, survey questions will appear for both groups of teachers.]]

---------------------------------------- **[NEW SURVEY PAGE]**

------ ----------------------------------------- **Your	Position	and	Job	Responsibilities** 

### [required] [q1] Which of the following best describes your current position as a teacher of K

12 students? (Check one.)
 If you are not a teacher of K

- 12 students, or do not h ave teaching responsibilities in 2014

- 2015, you have received this survey in error. Please let us know by replying to the invitation email or by calling {number}. 

##  Regular education teacher 

##  Special education teacher 

##  Other kind of teacher: _________________

__________________ 

### [q2x] What subject area(s) are you currently teaching ? (Check all that apply.) 

##  [q2a] English/language arts/reading/writing 

##  [q2b] Mathematics 

##  [q2d] Science 

##  [q2d] Social Studies 

##  [q2e] Foreign language(s) 

##  [q2f] Visual or performing arts (art, music, etc.) 

##  [q2g] Physical education/health education 

##  [q2h] Career/technical education 

##  [q2i] Other subject area(s): [include fill

- in box] 

### [q3] Which one of the following three groups of teachers do you belong to? These gr oups are defined by New Mexico’s teacher evaluation system, NM TEACH.68



##  Group A : tested subject, tested grade 

##  Group B : grade 3
– 12 subject not linkable to NM’s standard based assessments 

##  Group C : grade K

- 2 teacher 

### [q4] Which of the following best describes your current teaching arrangement? (Check one.) 

##  I teach a single group of students all or most of the day in multiple subject areas. (Traditional elementary arrangement; sometimes called “self

- contained.”) 

##  I teach several classes of different st udents during the day in a particular subject (for example, math) or perhaps two subjects (for example, you teach some math classes and some science classes). (Traditional secondary arrangement; sometimes called “subject

specific” or “departmentalized”; at elementary level, sometimes called “subject matter specialist.” Also typical arrangement for physical education, art, music, etc.) 

##  I mainly teach selected students released from (or in) their regular classes in specific skills or to address specific needs (for example, special education, reading, English as a second language, gifted and talented.) (Sometimes called “pull out”, “resource,” or “push in” instruction.) 

##  I am one of two or more teachers who are jointly responsible for teaching the same subject(s ) to a group of students (for example, in the same classroom), all or most of the time and/or in a majority of classes. (Sometimes called “co

- teaching” or “job share.”) 

##  Other (please describe): _______________________________________ 

### [q5] Including this s chool year (2014

- 2015), how many years have you been teaching, total, regardless of location? Total years teaching: [fill in number: must be whole number less than 60] 

### [q6] Including this school year (2014

- 2015), how many years have you been teaching in your current school ?

Years in current school: [fill in number: must be whole number less than 60] 

### [q7] Which of the following best describes your current position as a teacher of K

- 12 students? (Check all that apply ) 

##  Spanish bilingual teacher 

##  Native American bilingual teacher 

##  Other bilingual teacher: ___________________________________ 

##  Not a bilingual teacher69

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Experiences	This	Year	(SY	2014 -**  **2015)	of	Formal	Observations	for	Evaluation Purposes**  

### [ * required question] [q8] Thinking back over this whole school year 2014

- 2015, how many times were you **formally observed** by a school leader? School leader is defined as a principal, assistant principal, dean, or coach charged with authority to conduct formal observations of 20+ minutes using the NM TEACH Observation Rubric {link to rubric}. 

##  3 or more t imes 

##  2 times 

##  1 time 

##  0 times 

##  Other: ________________________________ 

### [q9] [If q8 = 1, 2, or 3+ times ] Use the slider to estimate h ow long the post

- observa tion conferences typically took. Think only of conferences after formal observations; do not include informal feedback you might have received from classroom walk

- throughs. [insert slider graphic here. Tick marks for 0 min, 15 min, 30 min, 45 min, 60 min, 75 min, 90 min.] 

### [q10] How fr equently did you receive **informal feedback** this year from a school leader
— for example, “ wow or wonder” statements after brief walk

- throughs. 

##  At least once a week 

##  At least once a month 

##  At least once per semester or trimester 

##  Never 

##  Other:

___________________ 

### [q11x] [If q8 = 1, 2, or 3+ times] Use the slider to indicate the extent to which you agree or disagree with the following statements about all of the conferences you had with a school leader following his or her **formal observation(s)** of your class this year (school year 2014

- 2015)

.
Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q11a] The **feedback session itself**, **separate** from professional development I may have70

been recommended to take , helped **me improve**  my instruction. 

### [q1 1b ] In each conference, I **brought documents**  to the conference , documents such as the lesson plan, and PDP. [ insert slider here] 

### [q1 1c ] In each conference, **the school leader brought documents**  to conference , such as my summative score and the most recent PDP. [ insert slider here]


### [q11 d ] In each conference the school leader identified **at least one positive practice** that I do well. [ insert slider here] 

### [q11 e ] In e ach conference the school leader identified **at least one**  **challenge** facing me. [ insert slider here] 

### [q11 f] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine which **instructional practices**  to praise or c ritique
. [ insert slider here] 

### [q11 g ] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine what **professional development** to recommend to the teacher. [ insert slider here] 

### [q1 1h ] In each conference, the school leader **ended** the conference on a **positive note** 
. [ insert slider here] 

### [q1 1i ] Each conference followed a **predictable format** 

. [ insert slider here] 

### [q11j ] In each conference , y school leader and I **mutually developed next steps** for my instruction. [ insert slider here]


### [q11 k ] I walked away with a **clear understanding**  of my school leader’s feedback. [ insert slider here] 

### [q11 l ] My school leader **listened** to what I had to say during the conferences. [insert slider here] 

### [q11 m ] The observations were done **to me,**  **rather than for**  **me** . [insert slider here] 

### [q11 n ] I received **specific** feedback about my performance. [insert slider here]71



### [q11 o ] I received **actionable** feedback about my performance. [insert slider here] 

### [q11 p ] During the conferences, I **committed to**  **a specific set of next steps**  to improve my instruction. [insert slider here] 

### [q11 q ] ] In each conference, the school leader **spoke for almost all of the time** 

. [insert slider here] 

### [q11 r ] There was a **high level of conflict** in the conference. [insert slider here] 

### [q11 s ] I n each conference I obtained recommendations for **professional development**  **tailored** to my specific needs. [insert slider here] 

### [q11 t ] I **trusted and accepted** the feedback I received. [insert slider here] 

### [q11 u ] **I feel positive** about the feedback I got during the conferences. [insert slider here] 

### [q1 1v ] **I enjoyed**  most of the post

observation feedback conferences. 

### [q1 1w ] I was always provided with a **written or online summary**  of the observation and my school leader’s comments on it. [insert slider graphic here]
[ 

### q12x ] Please indicate whether you were ever rated this school year b y your school leader as needing to improve upon or else excelling in the following observation items from NM TEACH, whether you received a recommendation to take up professional development (PD), and how you rate yourself on the item. Mark all that appl y. NM TEACH Observation Item I was rated at

least once as excelling on this item I was rated at

le ast once as needing to improve on this item I was

recommended to take PD because of this item I don’t

know how I was rated How would

yo u rate yo urself this school ye ar on this item? 2A: Creating an environment of respect and rapport [Ineffective, Minimally Effective, Effective, Highl y Effective, Exemplary I don’t72 know] 2B: Organizing Physical Space 2C: Establishing a culture for learning 2D: Managing classroom procedures 2E: Managing Student Behavior 3A: Communicating with Students 3B: Using questioning and discussion techniques 3C: Engaging students in learning 3D: Assessment in Instruction 3E: Demonstrating flexibility and responsiveness

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development	You	Received	This	Year	(SY	2014 -**  **2015)** The questions on this page ask about all of the professional development you have participated in during the 2014

- 2015 school year, includin g summer 2014. 

##  Professional development includes (but is not limited to) workshops, inservices, coaching/mentoring, collaborative planning/study, courses, institutes, professional conferences, self

- directed learning, and so forth. 

##  Consider mandatory/required professional development as well as voluntary professional development. 

### [q13x] Please indicate all of the topics below that one or more school leader recommend you take. In addition, please also include which topics were covered in professional development you actually participated in this year (2014

- 2015, including summer 2014) , and topics that you would have liked to have participated in professional development, but it was not available to you . Select as many topics as apply.
A	school leader I	took	the

PD	online I	received

the	PD	in have	liked73 recommen ded	I	take this	PD person to	take	the


### [q13a] Classroom management 

##    [q13b] Professionalism and communication in the classroom 

##    [q13c] Special education and inclusion 

##    [q13d] Standards

- based lesson plans 

##    [q13e] Common Core Standards 

##    [q13f] Assessment (includes continuous assessment, portfolio assessment, Common Core aligned assessments) 

##    [q13g] Data
- driven instruction 

##    [q13h] Early childhood 

##    [q13i] Instructional strategies or content in the content area(s) you teach
— e.g., Mathematics; or Literacy PD 

##    [q13j] Response to Intervention 

##    [q13k] Teachscape Reflect training 

##    [q13l] Other NM TEACH training 

##    [q13m ] Sheltering Instruction for Language development 

##    [q13n ] Language Acquisition Design 

##    [q13o ] Scaffolding instruction 

##    [q13 p ] Other: _______________________________ 

##    [q13 q ] Other: _______________________________ 

##    [q13 r ] Other: _______________________________ 

##    [q1 4 x] Thinking about the professional development you have participated in this year (2014

- 2015, including summer 2014), use the slider to indicate how much you agree or disagree with each of the following statements

. My professional development experiences this year... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4 a] Were mostly

developed by school leaders or teachers at my

school. [insert slider here] 

### [q1 4b ] Were mostly

job

- embedded rather than one

- time only sessions. [insert slider here]74



### [q1 4 a] Were varied and numerous enough to allow the school leader to customize PD recommendations based on my performance on specific domains of the NM TEACH Observation Rubric.{link} [insert slider here]



### [q1 4 b] Have been useful for helping me improve the achievement of my students [insert slider here] 

### [q1 4 c] Have been useful for improving my instruction [insert slider here] 

### [q1 4 d] Have taken more time than they were worth [insert slider here] 

### [q1 4 e] Have been well aligned with the Common Core State Standards and/or curriculum based on these standards [insert slider here]



### [q1 5 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the support you have received related to your professional development experiences this year (SY 2014

- 2015, including summer 2014). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 5 a] I had easy access to a list of the possible professional development courses or offerings I could take. [insert slider graphic here]


### [q1 5c ] My school leader(s) has encouraged and supported my participation in professional development. [insert slider here] 

### [q1 5d ] Sufficient resources (for example, substitute coverage, funding to cover expenses, stipends) have been available to allow me to participate in the professional development I need to teach effectively. [insert slider here]



### [q1 5e ] I have had sufficient flexibility in my schedule to pursue the professional development opportunities recommended for me. [insert slider here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Perceptions	of	the	Quality of	Your	School’s	Leadership	Last	Year** 75



### [q1 6 ] Rate your level of agreement with the following statement: “Overall, I was satisfied with the quality of my school’s leaders (p rincipals, assistant principals, deans, department chairs) this year (school year 2014

- 2015).” 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree strongly 

### [q1 7 x] Thinking about this year (2014

- 2015), use the slider to indicate how much you agree or disagree with the following statements. “The school leaders who conduct formal observations of my classroom ... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 7 a] Understand the subject matter content that I teach. [insert slider here] 

### [q1 7b ] Understand what skills are needed for teachers to foster meaningful student learning. [insert slider here] 

### [q1 7c ] Foster an atmosphere of trust and mutual respect in my school. [insert slider here] 

### [q1 7d ] Understand the 3 rating components of the NM TEACH Summative Evaluation Report (student achievement, observations, multiple measures). [insert slider here]



### [q1 7e ] Help to improve my teaching. [insert slider here] 

### [q1 7f] Are instructional leaders. [insert slider here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q1 8 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness S ys tem from school year 2014

- 2015. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 8 a] The NM TEACH Educator [insert slider here]76

Effectiveness System does a good job distinguishing effective from ineffective teachers. 

### [q1 8 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the students they teach. [insert slider here]


### [q 18 c] The NM TEACH Educator Effectiveness System ignores important aspects of teacher performance. [insert slider here] 

### [q1 8d ] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my performance. [insert sl ider here]



### [q18e ] I understand how the overall teacher ratings work in the NM TEACH Educator Effectiveness System
. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your NM	TEACH	Summative	Evaluation	Score**  

### [q19 ] Do you know what your level of Educator Effectiveness (Ineffective, Minimally Effective, Effective, Highly Effective, Exemplary) was for the school year 201 4
- 201 5 ? 

##  Yes 

##  No 

### [q20] [if q19=yes] How did you learn your level of Educator Effectiveness for the school year 201 4
- 201 5 ? Select all that apply. 

##  I received it via email from New Mexico Public Education Department 

##  I logged into my teacher portal online 

##  A school leader told me 

##  I received it via my mailbox at school 

##  Other: ______________________ 

##  Other: ______________________ 

##  I don’t recall 

### [q21] [if q19=yes] Did a school leader ever discuss with you how to improve your Educator77

Effectiveness score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  I don’t recall 

### [q22] [if q19=yes] Did a school leader ever discuss with you how to interpret your student achievement growth score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  Not Applicable 

##  I don’t recall

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. If you do not, proceed to the next and f inal page. 

### [q2 3 ] Do you have any comments about the NM TEACH Observation Rubric? 

### [q24] Do you have any comments about professional development available to teachers in your district? 

### [q2 5 ] Do you have any comments about the NM TEACH Educator Effectiveness System ? 

### [q2 6 ] Do you have any feedback about this survey?

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey.78

Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the survey invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thanks	Again** You have completed the survey and may now close it.79



### Spring	2016	Teacher	Survey [[ 

### Version : Teachers. On

- line administration. Wave 2.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2016 survey for teachers about feedback you receive about your inst ruction as a part of the NM TEACH Educator Effectiveness System . This is a follow

- on to our spring 2015 survey to learn about the feedback you received from your school leaders last year. **Study Purpose.** With funding from the federal Department of Educatio n as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Ed ucation Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers about their performance. REL Southwest researchers will link your survey responses to basic employment information, such as race, ethnicity and years of experience that REL Southwest researchers will obtain from NM PED. Survey r esults will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, on ce you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other informatio n that could potentially be used to identify individuals or schools. **Participants.** Ineach of the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study . Up to 10 teachers in your school were selected at random to be surveyed. Only those teachers who completed surveys in spring 2015 are being sent this survey

now in spring 2016. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teache r evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator80

Effectiveness System
. **Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$25 online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** You r participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a parti cipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you h ave read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate. **Thank you very much for your help in this important study!**

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Using	This	Survey** Here are some tips that may help you navigate and complete this survey. **Returning to previous pages.** Be sure to use the survey “go back to previous page” button at the **bottom** of the page. Do **not** use your browser’s back button
— it will not work. **Returning to the survey itself.** If you would like to complete the survey in multiple sessions
— or if you acciden tally get disconnected from the survey
— you can return to your previous responses by clicking on the link that was in your invitation email. This will take you to where you left off (through the last full page you completed). You can keep returning until yo u click on the “submit” button on the final page of the survey. **Saving your responses.** Your responses on any given survey page are saved when you click on either button at the bottom of the page (“save and go to next page” or “go back to previous page”).81

**Adding optional comments.** Toward the bottom of each survey page is a space where you can write in any comments you may have pertaining to the questions on the page. This is completely optional.

---------------------------------------- **[INTERNAL CODES]**

--- -------------------------------------------- Treatment code (treatment): REL Southwest will pre

- identify via an internal code that will not be listed on the survey whether teacher respondent works at a school that belongs to the treatment group or the con trol group. treatment = 1 indicates teacher is in the treatment group. treatment = 0 = control group. Unless otherwise noted, survey questions will appear for both groups of teachers. Prior respondent code: REL Southwest will pre

- identify those respondent s who, from the same email address, completed the corresponding item in wave 1 spring 2015 survey. For these respondents, prior = 1.

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Position	an d	Job	Responsibilities**  We would like to update our information about your position. 

### [q1] Which of the following best describes your current position as a teacher of K

- 12 students ? (Check one.)
 If you are not a teacher of K

- 12 students, or do not have teaching responsibilities in 2015

- 2016, you have received this survey in error. Please let us know by replying to the invitation email or by calling {number}. 

##  Regular education teacher 

##  Special education teacher 

##  Other kind of teacher:

___________________________________ 

### [q2x] What subject area(s) are you currently teaching? (Check all that apply.) 

##  [q2a] English/language arts/reading/writing 

##  [q2b] Mathematics 

##  [q2d] Science 

##  [q2d] Social Studies 

##  [q2e] Foreign language(s) 

##  [q2f] Visual or p erforming arts (art, music, etc.) 

##  [q2g] Physical education/health education 

##  [q2h] Career/technical education 

##  [q2i] Other subject area(s): [include fill

- in box]82



### [q3] Which one of the following three groups of teachers do you belong to? These groups are def ined by New Mexico’s teacher evaluation system, NM TEACH. 

##  Group A : tested subject, tested grade 

##  Group B : grade 3
– 12 subject not linkable to NM’s standard based assessments 

##  Group C : grade K

- 2 teacher 

### [q4] Which of the following best describes your curren t teaching arrangement? (Check one.) 

##  I teach a single group of students all or most of the day in multiple subject areas. (Traditional elementary arrangement; sometimes called “self

- contained.”) 

##  I teach several classes of different students during the day in a particular subject (for example, math) or perhaps two subjects (for example, you teach some math classes and some science classes). (Traditional secondary arrangement; sometimes called “subject

specific” or “departmentalized”; at elementary level, som etimes called “subject matter specialist.” Also typical arrangement for physical education, art, music, etc.) 

##  I mainly teach selected students released from (or in) their regular classes in specific skills or to address specific needs (for example, special education, reading, English as a second language, gifted and talented.) (Sometimes called “pull out”, “resource,” or “push in” instruction.) 

##  I am one of two or more teachers who are jointly responsible for teaching the same subject(s) to a group of studen ts (for example, in the same classroom), all or most of the time and/or in a majority of classes. (Sometimes called “co

- teaching” or “job share.”) 

##  Other (please describe): _______________________________________ 

### [q5] [if prior q5 blank] Including this sch ool year (2015

- 2016), how many years have you been teaching, total, regardless of location? Total years teaching: [fill in number: must be whole number less than 60] 

### [q6] [if prior q6 blank] Including this school year (2015

- 2016), how many years have you been teaching in your current school ?

Years in current school: [fill in number: must be whole number less than 60] 

### [q7] Which of the following best describes your current position as a teacher of K

- 12 students? (Check all that apply ) 

##  Spanish bilingual teacher 

##  Native American bilingual teacher 

##  Other bilingual teacher: ___________________________________83



##  Not a bilingual teacher

---------------------------------------- **[NEW SURVEY PAGE]**

----------------------------------------------- **Your	Experiences	This	Year	(SY	2015 -**  **2016)	of	Formal	Observations	for	Evaluation Purposes**  

### [ * required question ] [q8] Thinking back over this whole school year 2015

- 2016, how many times were you **formally observed** by a school leader? School leader is defined as a principal, assistant principal, dean, or coach charged with authority to conduct formal observations of 20+ minutes using the NM TEACH Observation Rubric {link to rubric}. 

##  3 or more times 

##  2 times 

##  1 time 

##  0 times 

##  [q9] [If q8 = 1, 2, or 3+ times ] Use the slider to estimate h ow long the formal post
observ ation conferences typically took ? Think only of conferences after formal observations; do not include informal feedback you might hav e received from classroom walk

- throughs. [insert slider graphic here. tick marks for 0 min, 15 min, 30 min, 45 min, 60 min, 75 min, 90 min] 

### [q10] How frequently did you receive **informal feedback** this year from a school leader
— for example, “ wow or wonder” statements after brief walk

- throughs. 

##  At least once a week 

##  At least once a month 

##  At least once per semester or trimester 

##  Never 

##  Other: ___________________ 

### [q1 1 x] [If q8 = 1, 2, or 3+ times] Use the slider to indicate h ow much you agree or disagree with the following statements about all of the conferences you had with a school leader following his or her **formal observation(s)** of your class this year (school year 2015

- 2016). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 1 a] The **feedback session itself**, **separate** [insert slider graphic here]84

from professional development I may have been recommended to take , **helped me**  **improve** my instruction. 

### [q1 1b ] In each conference, I **brought documents**  to the conference , documents such as the lesson plan, and PDP. [ insert slider here] 

### [q1 1c ] In each conference, **the school leader brought documents**  to conference , such as my summative score and the most recent PDP. [ insert slider here]


### [q1 1d ] In each conference the school leader identified **at least one positive practice** that I do well. [insert slider graphic here] 

### [q1 1e ] In e ach conference the school leader identified **at least one challenge**  facing me. [insert slider graphic here] 

### [q11 f] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine which **instructional practices**  to praise or c ritique
. [insert slider graphic here] 

### [q1 1g ] In ea ch conference, **the school leader used my**  **scores** o n the NM TEACH Observation Rubric to determine what **professional development** to recommend to the teacher. [insert slider graphic here] 

### [q1 1h ] In each conference, the school leader **ended** the conference on a **positive note**
. [insert slider graphic here] 

### [q1 1i ] Each conference followed a **predictable format** 

. [insert slider graphic here] 

### [q1 1j ] In each conference, m y school leader and I **mutually developed next steps** for my instruction. [insert slider graphic here]

. 

### [q1 1k ] I walked away with a **clear understanding**  of my school leader’s feedback. [insert slider graphic here] 

### [q1 1l ] My school leader **listened** to what I had to say during the conferences. [insert slider graphic here] 

### [q1 1m ] The observations were done **to me,**  **rather than for**  **me** . [insert slider graphic here] 

### [q1 1n ] I received **specific** feedback about my [insert slider graphic here]85

performance. 

### [q1 1o ] I received **actionable** feedback about my performance. [insert slider graphic here] 

### [q1 1p ] During the conferences, I **committed to**  **a specific set of next steps**  to improve my instruction. [insert slider graphic here] 

### [q11 q ] ] In each conference, the school leader **spoke for almost all of the time** 

. [insert slider here] 

### [q1 1r ] There was a **high level of conflict** in the conference. [insert slider graphic here] 

### [q1 1s I n each conference I obtained recommendations for **professional development**  **tailored** to my specific needs. [insert slider graphic here] 

### [q1 1t ] I **trusted and accepted** the feedback I received. [insert slider graphic here] 

### [q1 1u ] **I feel positive** about the feedback I got during the conferences. [insert slider graphic here] 

### [q1 1v ] **I enjoyed**  most of the post

observation feedback conferences. [insert slider graphic here] 

### [q1 1w ] I was always provided with a **written or online summary**  of the observation and my school leader’s comments on it. [insert slider graphic here]

[ 

### q12x ] Please indicate whether you were ever rated this school year b y your school leader as needing to improve upon or else excelling in the following observation items from NM TEACH, whether you received a recommendation to take up professional development (PD), and how you rate yourself on the item. Mark all that appl y. NM TEACH Observation Item I was rated at

least once as excelling on this item I was rated at

least once as needing to improve on this item I was

recommended to take PD because of this item I don’t

know how I was rated How would

yo u rate yo urself this school ye ar on this item? 2A: Creating an environment of respect and rapport [Ineffective, Minimally Effective, Effective, Highl y86 Effective, Exemplary I don’t know] 2B: Organizing Physical Space 2C: Establishing a culture for learning 2D: Managing classroom procedures 2E: Managing Student Behavior 3A: Communicating with Students 3B: Using questioning and discussion techniques 3C: Engaging students in learning 3D: Assessment in Instruction 3E: Demonstrating flexibility and responsiveness 

### [q 13a ][if treatment=1 & q8 > 0] Have you ever received the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” , a guide to conducting formal feedback sessions? The first page of the Conversation Checklist looks like this:87



##  Yes 

##  No 

### [q13b][if q8 > 0] Has your school leader ever used the “ Checklist for New Mexico Principals’ Provision of Feedback to Teachers ” when conducting the formal feedback conference in the 2015

- 2016 school year ? 

##  Yes 

##  No



### [q 13c ] [If q 13b = yes)] For how many of your feedback sessions did your school leader(s) use the “ Conversation Checklist ”? 

##  3 or more observations 

##  2 observations 

##  1 observation 

##  None of the observations 

##  I don’t know 

### [q13d [If 13a = yes)] Did you view the testimonial video that was distributed along with the Conversation Checklis t ? 

##  Yes 

##  No 

##  I don’t know88



### [q13e][ ] During the formal feedback sessions, h as a school leader ever used the “ Guidance for New Mexico Principals’ About Provision of Feedback to Teachers ” to guide any of your formal feedback sessions during the 2015

- 2016 school year? The first page of the Conversation Checklist looks like this:



##  Yes 

##  No


### [q14x ] If 13b=yes ] Use the slider to indicate h ow much you agree or disagree with the following statements about your opinions about the Conversation Checklist this year (school year 2015

- 2016). Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4a ] The Conversation Checklist was **easy to use** 
. [insert slider graphic here] 

### [q1 4b ] The Conversation Checklist **took too**  **much time** to use. [insert slider graphic here] 

### [q1 4c ] The Conversation Checklist made the feedback conversation feel **formulaic**

. [insert slider graphic here] 

### [q1 4d ] The Conversation Checklist provided me and my school leader(s) a **helpful structure**  for feedback. [insert slider graphic here] 

### [q1 4e ] I received **more critical feedback** when my school leader used the Conversation Checklist than in prior conferences when he/she didn’t use the Protocol. [insert slider graphic here]89



### [q1 4f] The Conversation Checklist helped me **commit to specific set of next steps**  to improve my instruction. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Professional	Development	You	Received	This	Year	(SY	2015 -**  **2016)** The questions on this page ask about all of the professional development (PD) you have participated in during the 2015

- 2016 school year

. **Don’t include any professional development you may have participated in prior to your first formal observation this school year.**  

##  Professional development includes (but is not limited to) workshops, inservi ces, coaching/mentoring, collaborative planning/study, courses, institutes, professional conferences, self

- directed learning, and so forth. 

##  Consider mandatory/required professional development as well as voluntary professional development. 

### [q1 5 x] Please i ndicate all of the topics below that one or more school leader recommend you take. In addition, please also include which topics were covered in professional development you actually participated in this year (2015

- 2016). Select as many topics as apply.
A	school leader recommen ded	I	take this	PD I	took	the

PD	online I	received

the	PD	in person have	liked

to	take	the


### [q1 5 a] Classroom management 

##    [q1 5 b] Professionalism and communication in the classroom 

##    [q1 5 c] Special education and inclusion 

##    [q1 5 d] Standards

- based lesson plans 

##    [q1 5 e] Common Core Standards 

##    [q1 5 f] Assessment (includes continuous assessment, portfolio assessment, Common Core aligned assessments) 

##    [q1 5 g] Data
- driven instruction 

##    [q1 5 h] Early childhood 

##    [q1 5 i] Instructional strategies or content in the content area(s) you teach
— e.g., Mathematics; or Literacy PD 

##    [q1 5j] Response to Intervention 

##    [q1 5 k] Teachscape Reflect training 

##   90



### [q1 5 l] Other NM TEACH training 

##    [q15m ] Sheltering Instruction for Language development 

##    [q15n ] Language Acquisition Design 

##    [q15o ] Scaffolding instruction 

##    [q1 5 m] Other: _______________________________ 

##    [q1 5 n] Other: _______________________________ 

##    [q1 5 o] Other: _______________________________ 

##   
[q1 6 x] Thinking about the professional development you have participated in this year (2015

- 2016, including summer 2015), use the slider to indicate how much you agree or disagree with each of the following statements

. My professional development experiences this year... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 4 a] Were mostly

developed by school leaders or teachers at my

school. [insert slider graphic here] 

### [q1 4b ] Were mostly

job

- embedded rather than one

- time only sessions. [insert slider graphic here] 

### [q1 6 a] Were varied and numerous enough to allow the school leader to customize PD recommendations based on my performance on specific domains of the NM TEACH Observation Rubric.{link} [insert slider graphic here] 

### [q1 6 b] Have been useful for helping me improve the achievement of my students [insert slider graphic here] 

### [q1 6 c] Have been useful for improving my instruction [insert slider graphic here] 

### [q1 6 d] Have taken more time than they were worth [insert slider graphic here] 

### [q1 6 e] Have been well aligned with the Common Core State Standards and/or curriculum based on these standards [insert slider graphic here] 

### [q1 7 x] Use the slider to indicate how much you agree or disagree with each of the following statements about the **support** you have received related to your professional development experiences this year (SY 2015

- 2016, including summer 2015).91 Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 7 a] I had easy access to a list of the possible professional development courses or offerings I could take. [insert slider graphic here] 

### [q1 7 b] My school leader(s) has encouraged and supported my participation in professional development. [insert slider graphic here] 

### [q1 7 c] Sufficient resources (for example, substitute coverage, funding to cover expenses, stipends) have been available to allow me to participate in the professional development I need to teach effectively. [insert slider graphic here] 

### [q1 7 d] I have had sufficie nt flexibility in my schedule to pursue the professional development opportunities recommended for me. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Perceptions of	the	Quality**  **of	Your	School’s	Leadership** 

### [q1 8 ] Rate your level of agreement with the following statement: “Overall, I was satisfied with the quality of my school’s leaders (principals, assistant principals, deans, department chairs) this year (school ye ar 2015

- 2016).” 

##  Disagree strongly 

##  Disagree somewhat 

##  Agree somewhat 

##  Agree strongly 

### [q1 9 x] Thinking about this year (2015

- 2016), use the slider to indicate how much you agree or disagree with the following statements. “The school leaders who conduct formal observations of my classroom ... Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q1 9 a] Understand the subject matter content that I teach. [insert slider graphic here] 

### [q1 9b ] Understand what skills are needed for teachers to foster meaningful student learning. [insert slider graphic here] 

### [q1 9c ] Foster an atmosphere of [insert slider graphic here]92

trust and mutual respect in my school. 

### [q1 9d ] Understand the 3 rating components of the NM Teach Summative Evaluation Report (student achievement, observations, multiple measures). [insert slider graphic here] 

### [q1 9e ] Help to improve my teaching. [insert slider graphic here] 

### [q1 9f] Are instructional leaders. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your	Personal	Views	of	NM	TEACH Educator	Effectiveness	System**  

### [q 20 x] Use the slider to r ate how much you agree or disagree with each of the following statements about the NM TEACH Educator Effectiveness S ys tem from school year 2015

- 2016. Disagree Strongly Disagree

Somewhat Agree

Somewhat Agree

Strongly 

### [q 20 a] The NM TEACH Educator Effectiveness System does a good job distinguishing effective from ineffective teachers. [insert slider graphic here] 

### [q 20 b] The NM TEACH Educator Effectiveness System is fair to all teachers, regardless of their personal characteristics or those of the st udents they teach. [insert slider graphic here] 

### [q 20c ] The NM TEACH Educator Effectiveness System has helped me pinpoint specific things I can do to help improve my performance. [insert slider graphic here] 

### [q20d ] I understand how the overall teacher ratings work in the NM TEACH Educator Effectiveness System
. [insert slider graphic here]

----------------------------------------------------- **[NEW SURVEY PAGE]**

---------------------------------- **Your NM	TEACH	Rating**  **Summative	Evaluation** 

### [q21 ] Do you know what your level of Educator Effectiveness (Ineffective, Minimally Effective, Effective, Highly Effective, Exemplary) was for the school year 201 5
- 201 6 ?93



##  Yes 

##  No 

### [q22] [if q21=yes] How did you learn your level of Educator Effectiveness for the school year 201 5
- 201 6 ? Select all that apply. 

##  I received it via email from New Mexico Public Education Department 

##  I logged into my teacher portal online 

##  A school leader told me 

##  I received it via my mailbox at school 

##  Other:

______________________ 

##  Other: ______________________ 

##  I don’t recall 

### [q23] [if q21=yes] Did a school leader ever discuss with you how to improve your Educator Effectiveness score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  I don’t recall 

### [q24] [if q21=yes] Did a school leader ever discuss with you how to interpret your student achievement growth score ? 

##  Yes, we had a conference 

##  Yes, in passing 

##  No 

##  Not Applicable 

##  I don’t recall

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Optional	Comments	from	You** The following four questions are optional if you have any feedback you wish to share. If you do94

not, proceed to the next and final page. 

### [q2 5 ] [if 13a=yes ] Do you have any suggestions to help us improve the Checklist for New Mexico Principals’ Provision of Feedback to Teachers ? 

### [q26] Do you have any comments about professional development available to teachers in your district? 

### [q2 7 ] Do you have any comment s about the NM TEACH Observation Rubric? 

### [q2 8 ] Do you have any comments about the NM TEACH teacher evaluation system?

--------------------------------------- **[NEW SURVEY PAGE]**

--------------------------------------------- **Thank	You!** Thank you so much for your time to complete this survey. Within the next 2 weeks, you will receive an email message with instructions about how to claim your gift card. This email message will come from {iCard ( services@icardgiftcard.com )} and will be sent to the same email address at which you received the survey invitation, unless you indicate otherwise here: 

##  Yes, please send my gift card information to the same email address to which you sent the surve y invitation. 

##  Please send my gift card information to a different email address, which I am providing here: _______________ 

##  No, I do not wish to receive a gift card email from {iCard}.

--------------------------------------- **[NEW SURVEY PAGE]**

------------- -------------------------------- **Thanks	Again** You have completed the survey and may now close it.95



### A **PPENDIX B**  **– Principal	and	Teacher**  **Recruitment	Materials W** **ave	1	Principal	Recruitment	Script**
**– SPRING	2015**  Subject: Invitation to take $25 survey to provide feedback on the NM TEACH teacher evaluation process (and receive $25) title first_name_ppal last_name_ppal School_name full_district_name Dear title last_name_ppal, The Regional Education Lab Southwest (RELSouthwest) is studying principals’ feedback to teachers in an effort to improve the teacher evaluation process. **Will you please be part of the study?**  Participation involves three things: 1. Complete a 30

- minute online survey now using the link below (and receive $25). 2. Receive a short guide about providing effective feedback to teachers for yo ur use throughout the 2015

- 2016 school year. You and your teachers will be assigned at random to one of two ty

pes of guidance. 3. Complete a second 30

- minute online survey in spring 2016 (and receive an additional $25). All New Mexico public school principals are invited to participate in the study. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Onl y researchers from the REL Southwest will view your answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the result s will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.** https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here].96

Below are some answers to frequentl y aske d questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southw est RELresearcher **Frequently Asked Questions How was I selected for this study?**  We are asking all New Mexico public school principals to complete the survey.
**What do I get for completing the surveys?** You will receive a $25 gift card for completing each survey. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.c om/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH traini ng for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the study results?** Only Southwest REL researchers will see your answers to survey questions. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process.97

**How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyo ne or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.98



### W**ave	1	Teacher	Recruitment	Script**
**– –**  **SPRING	2015** Subject: Invitation to takesurvey to provide feedback about the NM TEACH teacher evaluation process (and receive $25) title first_name_teacher last_name_teacher School_name full_district_name Dear title last_name_teacher, The Regional Education Lab Southwest (RELSouthwest) is studying the quality of the feedback you receive after formal observations that are a part of NM TEACH. **Will you please be part of the study?**  Your participation involves three things: 1. Complete a 30

- minute online survey using the link below (and receive $25). 2. Receive a short document about effective feedback sessions between you and your school leader for your use throughout the 2015

- 2016 school year. You and your principal will be assigned at random to one of two types of guidance. 3. Complete a second 30

- minute online survey in spring 2016 (and receive an additional $25). RELSouthwest has first invited all New Mexico public school principals to participate in the study. Then in all schools where principals agree to participate, REL Southwest is inviting up to 10 teachers to participate. You are one of only 10 teachers selected by

REL Southwest to participate . Since you were randoml y selected, we can’t replace you with any other teacher. Thus, your participation is very important. Principals who agree to participate are assigned at random to receive one of two types of guidance about effective feedback to teac hers. Teachers who participate in the study receive the same guidance as their principals

. **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopef ully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.**99

https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Question s**  **How was I selected for this survey?** We are asking up to 10 teachers to participate in each school where a principal has agreed to participate in the study.
**What do I get for participating in the study?** You will receive a $25 gift card for completing each of two surveys. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgi ftcard.com/gift

- cards/

. **What is the purpose of the study and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organizat ion. **Who will get the survey results?** Only Southwest REL researchers will see your answers to the surveys. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents.10 0

**Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encoura ge you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([surv ey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll fre e at {XX}, or c/o IRB, {address}.101



### W**ave	2	Principal	Recruitment	Script**
**– –**  **SPRING	2016** Subject: Invitation to take final $25 survey to improve the NM TEACH teacher evaluation process title first_name_ppal last_name_ppal School_name full_district_name D ear title last_name_ppal, Thank you for participating in the Regional Education Lab Southwest (REL Southwest) study of New Mexico principals’ feedback to teachers! You completed our spring 2015 survey

. See your responses here. **We are now inviting you to complete the second and final principal survey.**  As a reminder, all New Mexico public school principals were invited to participate in the study, which is supported by b oth the New Mexico Public Education Department and your school district. (To see the letter of support from NM PED and districts, visit {REL Southwest website} .) **Click this link to read your rights as a research subject and complete the spring 2016 survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey results will help NM PED refine NM TEACH, and hopefull y help other states improve their teacher evaluation systems. If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional question s about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher102

**Frequently Asked Questions How was I selected for this survey?**  We are asking all public school principals who completed the spring 2015 survey (for $25) to also complete the 2016 survey (for another $25).
**What do I get for completing the survey?** You will receive a $25 gift card for completing the survey. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?**  T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the survey?**  Southwest REL researchers are conducting this survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see individual results. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportu nity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyo ne or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.103



### W**ave	2	Teac her	Recruitment	Script** 
**– –**  **SPRING	2015** Subject: Invitation to take final $25 survey to improve the NMTEACH teacher evaluation process title first_name_teacher last_name_teacher School_name full_district_name Dear title last_name_teacher, Thank you for p articipating in the Regional Education Lab Southwest (REL Southwest) study of New Mexico principals’ feedback to teachers! You completed our spring 2015 survey

. See your responses here. **We are now inviting you to complete the second and final teacher surv ey.**  As a reminder, all New Mexico public school principals were invited to participate in the study, which is supported by b oth the New Mexico Public Education Department and your school district. (To see the letter of support from NM PED and districts, vi sit {REL Southwest website} .) **Click this link to read your rights as a research subject and complete the spring 2016 survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] **REL Southwest researchers will keep your survey answers confidential.** The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being include d in the public report. The survey results will help NM PED refine NM TEACH, and hopefull y help other states improve their teacher evaluation systems. If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions**104

**How was I selected for this survey?** We are asking all teachers who completed the spring 2015 survey (for $25) to take the spring 2016 survey (for another $25). We invited up to 10 teachers to take the 2015 survey in each school where a principal has agreed to participate in the study.

**Wh at do I get for completing the survey?**  You will receive a $25 gift card for completing the survey. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH , and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback rega rding how to improve NM TEACH training for principals and teachers. **Who is conducting the survey?** Southwest REL researchers are conducting this survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see individual results. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encour age you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your o wn individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([sur vey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll fr ee at {XX}, or c/o IRB, {address}.105



### W**ave	1	Principal	Recruitment	Script**
**– SPRING	2015**  Subject: Reminder: Invitation to take survey to provide feedback on the NM TEACH teacher evaluation process (and receive $25) title first_name_ppal last_name_ppal School_name full_district_name Dear title last_name_ppal, Recently you received an email asking you to be a part of a study under The Regional Education Lab Southwest (RELSouthwest) on principals’ feedback to teachers. **W ill you please be part of**  **the st udy?**  Participation involves three things: 1. Receive a short guide about providing effective feedback to teachers for yo ur use throughout the 2015

- 2016 school year. You and your teachers will be assigned at random to one of two ty

pes of guidance. 2. Complete a 30

- minute online survey now using the link below (and receive $25). 3. Complete a second 30

- minute online survey in spring 2016 (and receive an additional $25). All New Mexico public school principals are invited to participate in the study. **REL Southwest r esearchers will keep your survey answers confidential.**  The survey will not be used to evaluate you or your school. Onl y researchers from the REL Southwest will view your answers, and all answers will be aggregated before being included in the public report . The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a r esearch subject and to take the survey.**  https://survey .rand.org/research/hosting/ced/ **Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here], tel. [MMIC survey coordinator # here]. Sincerel y,106

[MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions How was I selected for this study?**  We are asking all New Mexico public school principals to complete the survey.
**What do I get for completing the surveys?** You will receive a $25 gift card for completing each survey. You’ll be able to choose your gift card from more than 200 merchants. T o see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the survey and how will the results be used?** T he Southwest Regional Educational Lab is conduc ting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more information about the study, visit {REL Southwest website}

. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest REL researchers are conducting this study, including the surve y. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the study results?** Only Southwest REL researchers will see your answers to survey questions. The survey results will be s ummarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that coul d identify respondents. **Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback proces s. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contact if I have more questions or concerns?** If you ha ve general questions, you can107

contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.108



### W**ave	1	Teacher	Recruitment	Script**
**– –**  **SPRING	2015** Subject: Reminder: Invitation to take survey to provide feedback about the NM TEACH teacher evaluation process (and receive $25) title first_name_teacher last_name_teacher School_name full_district_name Dear title last_name_teacher, Recently you received an email asking you to be a part of a study under The Regional Education Lab Southwest (RELSouthwest) on the quality of the feedback you receive after formal observations that are a part of NM TEACH

. **W ill you please be part of the study?**  Your participation involves three things: 1. Complete a 30

- minute online survey using the link below (and receive $25). 2. Receive a short document about effective feedback sessions between you and your school leader for your use throughout the 2015

- 2016 school year. You and your principal will be assigned at random to on e of two types of guidance. 3. Complete a second 30

- minute online survey in spring 2016 (and receive an additional $25). RELSouthwest has first invited all New Mexico public school principals to participate in the study. Then in all schools where principals agree to participate, REL Southwest is inviting up to 10 teachers to participate. You are one of only 10 teachers selected by

REL Southwest to participate. Since you were randoml y selected, we can’t replace you with any other teacher. Thus, your participation is very important. Principals who agree to participate are assigned at random to receive one of two types of guidance about effective feedback to teachers. Teachers who participate in the study receive the same guidance as their principals

. **REL Sout hwest researchers will keep your survey answers confidential.**  The survey will not be used to evaluate you or your school. Only researchers from the REL Southwest will view yo ur answers, and all answers will be aggregated before being included in the public report. The survey will give you an opportunity to provide feedback on the evaluation process, and the results will help NM PED refine NM TEACH, and hopefully help other states improve their teacher evaluation systems. **Click this link to read your rights as a research subject and to take the survey.**  https://survey .rand.org/research/hosting/ced/109

**Login Code:** [MMIC code here] If you prefer that we use a different email address for you, please send your request to the survey coordinator, [MMIC staff person here]. Below are some answers to frequentl y asked questions. If you have any additional questions about the survey, please feel free to contact me at [MMIC survey coordinator email address here] , tel. [MMIC survey coordinator # here]. Sincerel y, [MMIC survey

coordinator name] Survey Coordinator Southwest RELresearcher **Frequently Asked Questions How was I selected for this survey?**  We are asking up to 10 teachers to participate in each school where a principal has agreed to participate in the study.
**What do I get for participating in the study?** You will receive a $25 gift card for completing each of two surveys. You’ll be able to choose your gift card from more than 200 merchants. To see the full list of merchants, go to http://www.icardgiftcard.com/gift

- cards/

. **What is the purpose of the study and how will the results be u sed?**  T he Southwest Regional Educational Lab is conducting this study to learn about the feedback principals give to teachers under NM TEACH, and teachers’ perceptions of that feedback. The study is funded by the U.S. Department of Education. For more infor mation about the study, visit {REL Southwest website}. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve NM TEACH training for principals and teachers. **Who is conducting the study?** Southwest REL resea rchers are conducting this study, including the survey. The researchers are from the RAND Corporation, which is an independent, nonprofit, nonpartisan research organization. **Who will get the survey results?** Only Southwest REL researchers will see your ans wers to the surveys. The survey results will be summarized across all survey respondents, and Southwest REL will present summaries of results to NM PED and to the federal Department of Education. We adhere to strict codes of confidentiality and will NEVER release individual responses in any way that could identify

respondents.110

**Am I required to complete the survey?** No, the survey is voluntary. However, we strongly encourage you to participate. The survey is an opportunity to share your views of NM TEACH and to describe the post

- observation feedback process. **How do I complete the survey?** You complete the survey online, accessing it from a link shown above. This is your own individual link, so please do not share it with anyone or alter it. **Who can I contac t if I have more questions or concerns?**  If you have general questions, you can contact [first and last name of survey coordinator]. [He/She] is available by email ([survey email account]@rand.org) or by phone (1

- 866

- MMIC

- – toll

- free). If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}.111 

### A **PPENDIX C**  **– Secondary	Data	Elements**  **Student, teacher and principal administrative data variables**  Below we list the names of the files and variables we will be requesting from NM PED on students, teachers and principals as a part of the analysis. The information is based on the Student Teacher Accountability Reporting System (STARS) Volume 1 User Guide, 2014

- 15. ST ARS is a comprehensive student, staff, and course information system that provides a standard data set for each student served by Ne w Mexico’s 3Y/4Y

– Grade 12 public education system. STAR S uses the eScholar data warehouse which integrates detail student, staff, and course information via a standard set of templates that provide a consistent format for loading data from various student information systems used in public school districts or charter schools. For each data requested we list the Domain name, Template Name, and variable name and a variable description. **Staff Domain, Staff Assignment Template**

**Field Name Description**  District Code PED defined three character district code. Staff ID Masked Staff ID fixed across years and locations Assignment Code PED defined assignment or position codes. School Year Date School Year Location Code PED define d three character location code of school Assignment Date Report Date Percentage of Time Assigned The portion of the staff member’s time that is allocated to the assignment. **Staff Domain, Staff Development Fact Template**

**Field Name Description**  District Code PED defined three character district code. Staff ID Masked Staff ID fixed across years and locations Activity Code Professional Development Activity Code Activity Start Date Professional Development Activity Start Date **Staff Domain, Staff Template**

**Field Name Description**  District Code PED defined three character district code. Staff ID Masked Staff ID fixed across years and locations Gender Code Gender of Staff Member Ethnic Code Ethnicity Code of Staff Member Teaching of Principal Years of Experience The number of years that the staff member has been em ployed by any school district in any state as a TEACHER or PRINCIPAL (includes current district). Teaching of Principal Years of Experience in District The number of years that the staff member has been em ployed by the current school district as a TEACHER or PRINCIPAL. Highest Degree Earned Highest degree earned by the staff member. Staff Qualification Status Staff Qualification Status such as Certified or Substitute teacher112

Hispanic Indicator Indicator of whether the staff memb er is of Hispanic ethnicity Tribal Affiliation The primary New Mexico Tribe or Pueblo with which the staff member is affiliated or enrolled
**Student Domain,**  **Student Template** 

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Alternate Student ID District's local student identifier. Location Code PED define d three character location code of school School Year Date School Year Current Grade Level Student current grade level Birthdate Student date of birth. Gender Code Gender code of Student Ethnic Code Ethnicity Code of Student Hispanic Indicator Indicator of whether the student is of Hispanic ethnicity Economic Status Student participation in free or reduced lunch program. Economic Disadvantaged Status Code Identifies the type of Direct Cert the student is receiving , e.g. SNAP, TANF, Homeless. Special Education Student is IEP Y/N Level of Integration Amount of special education service. Repeating Last Year If student is in grades K

- 12, Y/N for retained or failed to be promoted following the completion of the past school year Gifted Participation Code Indication of whether the student is Gifted Migrant Status Indicator of migratory status English Proficiency Code Current or former ELL student.
**Assessment Domain,**  **Assessment Fact Template** 

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Test Description Description of type of test Assessment School Year Date School Year Item Description Code Item description code Test Date Date of Assessment Location Code PED defined three character location code. Score Score of administered test. Scoring Model The Measurement used to determine the outcomes the respective assessment113

**Attendance Domain,**  **Student Daily Attendance Template**

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Location Code PED defined three character location code. Attendance Date Report date of student unexcused absence. Attendance Code Provide the Attendance Code that indicates the unexcused absence s tatus for the date.
**Attendance Domain,**  **Student Summary Attendance Template**

**Field Name Description**  District Code PED defined three character district code. Student ID State issued student identification number. Location Code PED defined three character location code. School Year Date School Year Period Start Date Attendance period start date. Period End Date Attendance period end date. Days Present Provide the total number of days within the Reporting Period that the student was marked as Present at the location. Days Enrolled Provide the total number of days within the Reporting Period that the student was counted as Enrolled at the location.
**Course and Grades Domain,**  **Course Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Course Name Course Name Semester Indicator of reporting period Course Code PED

- defined 8 character course code offered by the school.
**Course and Grades Domain,**  **Course Instructor Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Primary Instructor ID State issued student identification number. Course Name Course Name Semester Indicator of reporting period Course Code PED

- defined 8 character course code offered by the114 school. Section Code Class section number assigned by school district to identify distinct classes. Snapshot Date Field indicating the report date. See valid values. Special Course Indicator Indicator for honors, gifted, special education, etc. **Course and Grades Domain,**  **Student Course Enrollment Template**

**Field Na me**  **Description** District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Enrollment Period Reporting date Student ID State issued student identification number. Section Code Class section number assigned by school district to identify district classes. Course Code PED

- defined 8 character course code offered by the school.
**Enrollment Domain,**  **School Enrollment Template**

**Field Name Description**  District Code PED defined three character district code. Location Code PED defined three character location code. School Year Date School Year Student ID State issued student identification number. Enrollment Date T he actual date the mobility transaction occurred. Enrollment Code Code identifying the type of mobility transaction. Enroll Change Code Provide a code that indicates the student enrollment was a result of SCHOOL CHOICE.115

**NM TEACH Reflect Data** Below we list the variables we will be requesting from NM PED from the Teachscape operated REFLECT teacher evaluation data s ys tem. Staff ID that is consistent across school years and data sources Teacher Evaluation Group (A, B, or C) Graduated Consideration Tag District ID School year School NCES ID Masked Evaluator ID Observation number Observation date NM TEACH Summative evaluation component

- Observation Element name NM TEACH Summative evaluation component

- Observation Element Score (Level of Performance) NM TEACH Summative evaluation component

- Student Achievement measure name NM TEACH Summative evaluation component

- Student achievement measure score NM TEACH Summative evaluation component

- Multiple measure name NM TEACH Summative evaluation component

- Multiple measure score NM TEACH Summative evaluation overall score NM TEACH Summative evaluation effectiveness rating116

**Principal and Teacher contact information File 1** 

**– Principal contact information**  Principal Last Name Principal First Name Principal Middle Name or initial Principal Maiden Name Principal primary

email address Principal secondary

email address School Name School year School NCES ID

**File 2**

**– Teacher contact information**  Teacher Last Name Teacher First Name Teacher Middle Name or initial Teacher Maiden name Teacher primary

email address Teacher secondary

email address School Name School year School NCES ID117



### APPENDIX	D **– Technical	Working	Group	Suggestions**  In January

2014, Southwest RELresearchers obtained comments from two TWG reviewers: Dan Goldhaber and Geoff Borman. This Appendix summarizes in bullet form the conceptual suggestions that Southwest RELresearchers received. Line edits are omitted from this summary . The researchers response are noted under each bullet prefaced b y “RESPONSE.”

 **T he power of the intervention to produce changes in practice may be compromised**  **if there are limited quality PD opportunities for principals to recommend in order**  **for teachers to improve their instruction.**  **This could lead the investigators to incorrectly conclude that the guide was not useful for changing practice and,**  **eventually, achie vement.** 

o RESPONSE: While ED’s contractor agree s that the quality of the PD modules could be an important moderator of the impact of the Guide, the decentralized acquisition or creation of PD by districts or by

schools within districts makes it impractical to know ahead of time what the PD will be on offer during the school ye ar 2015

- 2016. However, ED’s contractor believe s that this lack of a standardized menu of pre

- approved/pre

- screened PD sessions is representative of the norm and thus a good test of the efficacy of a feedback guide in other states besides New Mexico.
 **Unless it is vital for treatment group**  **teachers to receive the treatment**  **guide, it**  **seems inadvisable because the threat of cross**  **- over seems quite strong.**  o RESPONSE: ED’s contractor decided to distribute the guide to teachers as well as to principals for the following two reasons: (a) To minimize Hawthorne effects, ED’s contractor intend s to trump principals’ survey responses regarding usage of the guide with teachers’ responses about having seen or ever used the feedback checklist. (b) ED’s contractor intend s to disseminate the checklist to teachers because the point of the treatment guide is to structure a two

- way, mutual conversation instead of principals talking at teachers (as is currentl y the common practice). ED’s contractor agree s this increases the risk that the treatment guide will be shared with control group school staff. ED’s contractor think the best idea to discourage contamination and reduce John Henry effects is to disseminate a “Guide” to all principals, blinding them to their status in treatment or control group, which is addressed in the next point

.  **T he provision of**  **[on ly]**  **the treatment**  **guide and the reliance on self -**  **reported survey data (and principal** **- reported classroom observation scores) could cause Hawthorne** 118

**effects due to both principals and teachers expecting improvement solely because they have been offered a new**  **treatment. Providing**  **a "guide" t o all participating**  **principals could, potentially, minimize the risk of cross**  **- over (because everyone gets**  **something), and would certainly address t he issue of a Hawthorne effect.**  o RESPONSE: ED’s contractor agree s and has ado pted this suggestion.
 **N one of the**  **outcome measur es are well** **- validated outcomes**  **, and it is not clear**  **whether the authors intend to construct composite measures from the surveys or whether outcomes will be based on single items.**  o RESPONSE: ED’s contractor agree d and has included student achievement as an outcome (a well

- validated outcome) and developed composite indexes derived from survey

questions.
 **T he estimated recruitment rates of 65% for principals and 60% of tea**  **chers seem relatively high.**  o RESPONSE: ED’ s contractor has powered the study for a range of participation rates, between 50 and 65 percent for principals and between 50 and 60 percent for teachers, and demonstrated that for the research design the drop

- off is very small from the upper to the lower ends of these ranges (approximately 0.02 to 0.03 in MDES). ED’s contractor based the upper end of our anticipated participation rates on prior experience with a similar efforts in New York City Public Schools, where there was also a direct connection to regular school responsibilities (although somewhat weaker than in the case of the current study) and participation was well over 60% (McCombs, Kirby , and Mariano, 2009)

.  **T he teacher inclusion cri** **teria for the study could bias the results of the study.**  o RESPONSE: ED’s contractor agree d , and has dropped eligibility screeners for either principals or teachers. The onl y remaining eligibility criteria is that the study participant work in NM public schools at the time of the survey

.  **The study would be considerably better if it had value added components**  **.** o RESPONSE: ED’s contractor agree d that value

- added would be a highl y useful, validated outcome. Therefore , ED’s contractor added student achievement a s an outcome measure.
 **It would be beneficial to include charter schools.**  o RESPONSE: This suggestion has been enacted.119 

### APPENDIX	E **– Educational	Sciences	Reform	Act** 120 

### APPENDIX	F **– IRB	Approval** 

NOTIFICATION OF INITIAL APPROVAL

From: Caroly

n Tschopik To: Kata Mihal y - PI Cc: Heather Schwartz

- Primary Contact Lou Mariano

- Study

Team Heather Schwartz

- Study Team
ID: 2014

- 0239 Title: New Mexico teacher feedback Description: The above study has been approved and all requirements have been met.

Approval of the study is for the period of 4/22/2014 to 4/21/2015. Action Required: No action is required of you at this time. CURRENT STUDY REVIEW STATUS Project Number: There are no items to display Funding Source: Department of Education Prime Recipient: Other, Specify :

SEDL121

RAND Unit(s): Education Human Subjects: Yes Ty pe of Review: Expedited Component Approvals: NM students Procedure Approved Motion Category secondary data Tue Apr 22

00:00:00 PDT 2014 Does not involve

human subjects

principals & teachers who work in any NM public school Procedure Approved Motion Category

secondary data Tue Apr 22

00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:**

 7

- Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)
principals & teachers whom we will recruit for 2 focus groups to pilot our surveys Procedure Approved Motion Category

focus group to pilot our surveys Tue Apr

22 00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:**

 7

- Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus122 groups, etc.)

principals whom we will recruit to participate in study Procedure Approved Motion Category

Randomized Controlled Trial intervention Tue Apr

22 00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:**

 7

- Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

Survey Tue Apr

22 00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:**

 7

- Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.)

teachers whom we will recruit to participate in the study Procedure Approved Motion Category

Randomized Controlled Trial intervention Tue Apr

22 00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:**

 7

- Individual or group characteristics or behavior (Minimal risk surveys,123 interviews, focus groups, etc.)

Survey Tue Apr

22 00:00:00 PDT 2014 Expedited

- Approved **Expedited Category:** 

 7

- Individual or group characteristics or behavior (Minimal risk surveys, interviews, focus groups, etc.) Contingencies: Description Date Due Date Completed

There are no items to display Assurance number: FWA00003425 IRB number: IRB00000051 Administrator: Caroly

n Tschopik

The HSPC is RAND's Institutional Review Board to review research involving human subjects, as required b y federal regulations. RAND's "Federalwide Assurance for the Protection of Human Subjects" (FWA000034 25, effective through July 1, 2018, at http://intranet.rand.org/groups/hspc/fwa.pdf ) serves as our assurance of compliance with the regulations of 16 federal departments and agencies. According to this assurance, the Committee is responsible for review regardless of source of funding.124 

### APPENDIX	G **– Study	Participants’	Consent**  As shown in Appendix B, ED’s contractor will first email all New Mexico principals inviting them to participate in the study . Those principals who agree will click on a link enclosed in the email invitation, which will direct principals to the following consent form. Principals must click on the button indicating that they have read and understand the terms of participation and agree to participate. Upon clicking that button, they are directed to the spring 2015 survey . Teachers have a parallel consent proc ess. The consent statements shown here a duplicate of the first section of the surveys included in Appendix A.


### Spring	2015	School	Leader	Survey [[ 

### Version : On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the sp ring 2015 survey for school leaders about the feedback you give to teachers’ regarding their instruction as a part of the NM TEACH Educator Effectiveness System
. **Study Purpose.** With funding from the federal Department of Education as a part of the Southw est Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedba ck regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . Survey results will **not** be used to evaluate you, other teachers, or sc hool leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, on ce you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will125

never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey responses that include respondents’ names, school names, or other informati on that could potentially be used to identify individuals or schools. **Participants.** At the same time that we are emailing you, REL Southwest is also inviting all New Mexico public school school

- based principals to participate in this study. Up to 10 teachers per school are also being surveyed

. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Me xico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will pro vide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your participation in this survey is completely vo luntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating tha t you have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the informa tion and choose to participate. **Thank you very much for your help in this important study!** 

### Spring	2016	School **Leader	Survey** [[ 

### Version : On

- line administration. Wave 2.]]126

**Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2016 survey for school leaders about the feedback you give to teachers’ regarding their instruction as a part of the NM TEACH Educator Effectiveness System . This is a follow

- on to our fall 2015 survey to learn about the feedback you have provided to teachers this school year. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback. For more information about the study, please see { REL Sou thwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback re garding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers abo ut their performance. Survey results will **not** be used to evaluate you, other teachers, or school leaders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation emai l, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, distric t, or the state will have access to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants.** REL Southwest originally invited all New Mexico publi c school school

- based principals to participate in this study. Only those principals who completed surveys in spring 2015 are being sent this survey now in spring 2016. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
.127

**Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** You r participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions

. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a parti cipant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you h ave read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate. **Thank you very much for your help in this important study!**128



### Spring	2015	Teacher **Survey** [[ 

### Version : Teachers. On

- line administration. Wave 1.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2015 survey for school leaders about the feedback you give to teachers’ regarding their instruction as a part of the NM TEACH Educator Effectiveness System

. **Study Purpose.** With funding from the federal Department of Education as a part of the Southwest Regional Educational Lab , this study is intended to learn about the feedback principals give to teachers under the NM TEACH Ed ucator Effectiveness System , and teachers’ perceptions of that feedback. For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate i n a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness System training for principals and teachers. **Survey Purpose.** Th is survey is intended to gather information on the provision of feedback to teachers after formal observations by school leaders. REL Southwest researchers will link your survey responses to basic employment information, such as race, ethnicity and years o f experience that REL Southwest researchers will obtain from NM PED. Survey results will **not** be used to evaluate you, other teachers, or school leaders. REL Southwest researchers will ask study participants to fill out an on

- line, 30

- minute survey once in spring 2015 and again in spring 2016 . **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; res ponses will be saved as you enter them. However, once you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpos e of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have access to survey responses that include r espondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants** Ineach of the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study Up to 10 teachers in your school were selected at random to be surveyed

.129

**Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System
. **Compensation.** Within two weeks of completion of th is online survey, you will receive via email a link to your **$ 25**  **online gift card** for completing the survey. We will provide another $25 online gift card upon completion of the spring 2016 survey to each participant. **Risks.** The risks associated with your participation in this study are minimal. **Voluntary Participation.** Your partic ipation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have an y questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate.. **Thank you very much for your help in this important study!** 

### Spring	2016	Teacher	Survey [[ 

### Version : Teachers. On

- line administration. Wave 2.]] **Understanding	Your	Participation	in	the	S tudy**  Welcome to the spring 2016 survey for teachers about feedback you receive about your instruction as a part of the NM TEACH Educator Effectiveness System . Th is is a follow

- on to our spring 2015 survey to learn about the feedback you received from your school leaders last year. **Study Purpose.** With funding from the federal Department of Education as a part of the130

Southwest Regional Educational Lab , this study i s intended to learn about the feedback principals give to teachers under the NM TEACH Educator Effectiveness System, and teachers’ perceptions of that feedback For more information about the study, please see { REL Southwest website}. In {month, year} REL Southwest researchers asked all New Mexico’s public school principals to participate in a study about evaluation feedback to teachers. Results will be used to provide New Mexico Public Education Department feedback regarding how to improve the NM TEACH Educator Effectiveness S ys tem training for principals and teachers. **Survey Purpose.** This survey is intended to gather information on the provision of feedback after formal observations by school leaders to teachers about their performance. REL Southwest res earchers will link your survey responses to basic employment information, such as race, ethnicity and years of experience that REL Southwest researchers will obtain from NM PED. Survey results will **not** be used to evaluate you, other teachers, or school le aders. **Procedures.** Online completion of the survey should take approximately 30 minutes. By using your personal survey link provided in the invitation email, you will be able to complete the survey in multiple sessions; responses will be saved as you enter them. However, on ce you click “submit” at the very end of the survey, you will no longer be able to return. **Confidentiality.** REL Southwest will keep all survey responses **confidential** and will only use them for the purpose of the study. Our reports will contain summaries, and responses will never be presented in any way that would permit readers to identify you or your school. No one at your school, district, or the state will have acc ess to survey responses that include respondents’ names, school names, or other information that could potentially be used to identify individuals or schools. **Participants.** Ineach of the participating New Mexico public schools, REL Southwest has invited the principal to participate in this study . Up to 10 teachers in your school were selected at random to be surveyed. Only those teachers who completed surveys in spring 2015 are being sent this survey

now in spring 2016. **Benefits.** Your completion of this survey gives you the opportunity to reflect on and share your perspective about the teacher evaluation system. You will be providing valuable information that may help New Mexico Public Education Department improve the NM TEACH Educator Effectiveness System . **Compensation.** Within two weeks of completion of the online survey, you will receive via email a link to your **$25 online gift card** for completing the survey. **Risks.** The risks associated with your participation in this study are minimal.131

**Voluntary Participation.**  Your participation in this survey is completely voluntary. You can decide not to participate or to discontinue your participation at any time. A few survey questions, marked with a red asterisk, are required for routing to later questions. **Questions?** If you have any questions about this survey or the study as a whole, please feel free to call us toll

- free at {XXX}. You can also contact {XX}, the survey coordinator, at {email} or by phone at {XX}. If you have concerns or questions about your rights as a participant, contact the REL Southwest at SEDL (which is responsible for the protection of project participants): {email}, toll free at {XX}, or c/o IRB, {address}. **[Required question] Informed Consent.** By clicking the box below, you are indicating that you have read and understood the information provided to you about your participation in this survey. 

##  I have read and understood the information and choose to participate. **Thank you very much for your help in this important study!**132 

### APPENDIX	H **- Confidentiality	Form	and	Affidavits** 

**CONFIDENTIALITY AGREEMENT** Impacts of a Detailed Checklist on Feedback to Teachers (RAND under Contract No. ED

- IES

- 12

- C

- 0012 ) Safeguards for Individuals Against Invasion of Privacy : In accordance with the Privacy Act of 1974 (5 United States Code 552a), the Education Sciences Reform Act of 2002 (Public Law 107

- 279), the Federal Statistical Confidentiality Order of 1997, the E

- Government Act of 2002 (Public Law 107

- 347), and the Computer Security Act of 198 7, American Institutes for Research (RAND ) and all its subcontractors are required to comply with the applicable provisions of the legislation, regulations, and guidelines and to undertake all necessary safeguards for individuals against invasions of privacy . To provide this assurance and these safeguards in performance of work on this project, all staff , consultants, and agents of RAND , and its subcontractors who have any access to study

data, shall be bound by

the following assurance. **Assurance of Confi dentiality**  1. In accordance with all applicable legislation, regulations, and guidelines, RAND assures all respondents that their responses may be used onl y for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (ESRA 2002), 20 U.S. Code, § 9573]. 2. The following safeguards will be implemented to assure that confidentiality is protected as allowable b y law (20 U.S.C. § 9573) by all employees, consulta nts, ag ents, and representatives of RAND and all subcontractors and that physical security of the records is provided: a. All staff with access to data will take an oath of nondisclosure and sign an affidavit to that effect. b. At each site where these items are processed or maintained, all confidential records that will permit identification of individuals shall be kept in a safe, locked room when not in use or personally

attended by

project staff. c. When confidential records are not locked, admittance to the room or area in which they reside shall be restricted to staff sworn to confidentiality on this project. d. All electronic data shall be maintained in secure and protected data files, and personall y identifying information shall be maintained on separate files from statistical data collected under this contract. e. All data files on network or multi

-user systems shall be under strict control of a database manager with access restricted to project staff sworn to confidentiality , and then only on a need

- to
- know basis.133

f. All data files on single

-user computers shall be password protected and all such machines will be locked and maintained in a locked room when not attended by project staff sworn to confidentiality

. g. External electronicall y stored data files (e.g., tapes on diskettes) shall be maintained in a locked storage device in a locked room when not attended b y project staff sworn to confidentiality

. h. Any data released to the general public shall be appropriatel y masked such that linkages to individually

identifying inf

ormation are protected to avoid individual identification in disclosed data. i. Data or copies of data may not leave the authorized site for any reason. 3. Staff, consultants, agents, or RAND and all its subcontractors will take all necessary steps to ensure that the letter and intent of all applicable legislation, regulations, and guidelines are enforced at all times through appropriate qualifications standards for all personnel working on this project and through adequate training and periodic follow

-up procedures. By my signature affixed below, I hereb y swear and affirm that I have carefully read this statement and full y understand the statement as well as legislative and regulatory assurances that pertain to the confidential nature of all records to be handle d in regard to this project, and will adhere to all safeguards that have been developed to provide such confidentiality . As an employee, consultant, agent, or representative of RAND or one of its subcontractors, consultants, agents, or representatives, Iunderstand that I am prohibited by law from disclosing any

such confidential information to anyone other than staff, consultant, agents, or representatives of RAND , its subcontractors, or agents, and Institutes of Education Science. I understand that any wi llful and knowing individual disclosure or allowance of disclosure in violation of the applicable legislation, regulations, and guidelines is punishable by law and would subject the violator to possible fine or imprisonment. (Signature) (Date)134 **AFFIDAVI T OF NONDISCLOSURE**  Impacts of a Detailed Checklist on Feedback to Teachers (RAND under Contract No. ED

- IES

- 12

- C

- 0012 ) [insert name] [insert p osition] Date of Assignment to Impacts of a Detailed Checklist on Feedback to Teachers: November 2013 RAND 1776 Main Street Santa Monica , CA 90401

- 3208 I, [insert name] , do solemnly swear (or affirm) that when given access to any Impacts of a Detailed Checklist on Feedback to Teachers databases or files containing individually identifiable information, I will not: i. use or reveal any individually i dentifiable information furnished, acquired, retrieved or assembled by

m e or others, under the provisions of Section 183 of the Education Sciences Reform Act of 2002 (PL1 07

- 279) and Title V, subtitle A of the E

- Government Act of 2002 (PL 107

- 347) for any purpose other than statistical purposes specified in the NCES survey, project or contract; ii. make any disclosure or publication whereb y a sample unit or survey respondent could be identified or the data furnished by o r related to any

p articular person under this section could be identified; or iii. permit anyone other than the individuals authorized by the Commissioner of the National Center for Education Statistics to examine the individual reports.
(Signature) (The penalty

f or unlawful disclosure is a fine of not more than $250,000 [under 18 U.S.C. 3571] or imprisonment for not more than 5 years [under 18 U.S.C. 3559], or both. The word "swear" should be stricken out wherever it appears when a person elects to affirm the affi davit rather than to swear to it.) State of _____________________________ County of _______________________________ Subscribed and sworn/affirmed before me, ______________________, a Notary Public in and135

for ________________County, State of ______________ __________, on this date,

______________________.

___________________________________________ Notary Public My commission expires: _____________________________.