## EVALUATI**NG THE RETIRED MENTORS FOR NEW TEACHERS**  **PROGRAM**
**PAPERWORK REDUCTION ACT CLEARANCE REQUEST**  **SECTION A** 

Prepared For: Institute of Education Science s United States Department of Education Contract No. ED IES
- 12C
- 0007

Prepared By: REL Central Regional Educational Laboratory

September 16, 20131



## Table	of	Contents **Section A. Justification ................................**  **................................ ................................**  **....................... 2**  A1. Circumstances Necessitating Collection of Information
................................ ................................ ..... 2 Statement of Need
................................ ................................ ................................ ................................ 2 Overview of the Study Design
................................ ................................ ................................ ............... 3 Overview of the Specific Data Collection Plan
................................ ................................ ...................... 7 Table 1. Timeline for the Data Collection
................................ ................................ ............................. 8 A2. How, by Whom, and for What Purpose Information is to Be Used
................................ ................... 8 A3. Use of Automated, Electronic, Mechanic al or Other Technological Collection Techniques
............ 10 A4. Efforts to Identify Duplication.
................................ ................................ ................................ ......... 10 A5. Sensitivity to Burden on Small Entities
................................ ................................ ............................. 10 A6. Consequence to Federal Program or Policy Activities if the Collection is Not Conducted or is Conducted Less Frequently
................................ ................................ ................................ ..................... 10 A7. Special Circumstances
................................ ................................ ................................ ...................... 11 A8. Federal Register Announcement and Consultation
................................ ................................ .......... 11 Federal Register Announcement
................................ ................................ ................................ ......... 11 Consultations Outside the Agency
................................ ................................ ................................ ...... 11 A9. Payment or Gift to Respondents
................................ ................................ ................................ ...... 12 A10.Confidentiality of the Data
................................ ................................ ................................ .............. 12 A11. Additional Justification for Sensitive Questions
................................ ................................ ............. 13 A12. Estimates of Hour Burden
................................ ................................ ................................ ............... 13 A13. Estimate for the Total Annual Cost Burden to Respondents or Record Keepers
........................... 15 A14.Estimates of Annualized Costs to the Federal Government
................................ ............................ 16 A15. Reasons for Program Changes or Adjustments
................................ ................................ .............. 16 A16. Plans for Tabulation and Publication.
................................ ................................ ............................. 16 Impact Analysis
................................ ................................ ................................ ................................ ... 17 Implementation Analysis
................................ ................................ ................................ ..................... 21 Publication Plans
................................ ................................ ................................ ................................ . 22 A17. Approval to Not Display the Expiration Date for OMB Approval
................................ ................... 23 A18. Exception to the Certification Statement
................................ ................................ ....................... 23 **References ................................**  **................................ ................................**  **................................ .......**  **24**2 

## Section	A.	Justification The U.S. Department of Education (ED) requests OMB clearance for data collection related to the Regional Educational Laboratory (REL) program. ED, in consultation with REL Central under contract ED IES
- 12C
-0007 , has planned a study of a program that uses retired master educators to provide mentoring support to new teachers in high need elementary schools in Colorado’s Aurora Public School District
. This program is referred to as the “Retired Mentors for New Teachers Program.” OMB approval is being requested for REL Central ’s data collection for this project, including a REL Central teacher survey and a total of ten focus groups conducted by REL Central with mentee teachers and mentors in the program . The study will also draw upon several types of data that the school district collect s . These include teacher turnover rates in high

- need elementary schools, data from school administrator evalua tions of teacher performance, mentor records of support, and student assessment scores for students served by teachers in participating elementary schools. OMB approval is not being sought for this data since it is collected by the district and not REL Central. **A1. C**  **ircumstances Necessitating	Collection	of	I**  **nformation** This data collection is authorized by the Educational Sciences Reform Act (ESRA) of 2002 ( see Appendix A) . Part D, Section 174(f)(2 ) of ESRA states that as part of their central mission and primary function, each regional educational laboratory “shall support applied research by
. . . developing and widely d isseminating, including through Inter net- based means, scientifically valid research, information, reports, and publications that are usable for improving academic achievement, closing achievement gaps, and encouraging and sustaining school improvement, to
— schools, districts, institutions of h igher education, educators (including early childhood educators and librarians), parents, policymakers, and other constituencies, as appropriate, within the region in which the regional educational laboratory is located.” **Statement of Need** The importance of teacher effectiveness has been well

- supported by studies demonstrating that teachers vary in their ability to produce student achievement gains; all else being equal, students taught by some teachers experience greater achievement gains than those taught by other teachers (Konstantopoulos & Chung, 2010; Aaronson , Barrow, & Sander, 2007; Nye, Konstantopoulos, & Hedges, 2004; Wright, Horn, & Sanders, 1997). These findings have encouraged wide interest in identifying the most effective programs and practices to enhance teacher effectiveness. Studies of induction and mentoring for beginning teachers also show positive impacts on student achievement, teacher retention, and teacher instructional practices (Ingersoll & Strong, 2011; Strong, 2006) , including positive impacts for new or probationary teachers ( Smith & Ingersoll, 2004 )
. However, few studies have used experimental designs, and not one has examined the impact of using retired master teachers to deliver mentoring support.3

Understanding t he impacts of such a program on students and teachers comes at a particularly critical time for many states and districts in the Central Region
. Conversations with chief state school o fficers in the R egion and research alliance members suggest strong receptivity to expa nding teacher mentoring in high

- need schools. Student achievement continues to lag in high

- poverty schools, large numbers of teachers are expected to retire in the near future, and state and district budget shortfalls continue to preclude man y districts from undertaking more expensive approaches to supporting their teachers. While retirements may exacerbate existing teacher turnover issues in high

- need schools, retirees also represent a growing, untapped, potentially lower

- cost resource of talent to guide and support the next generation of teachers needed for schools and districts to succeed. In fact, one report indicates that “the nation’s teaching population is older than it has ever been. Almost half of the K

- 12 public sc hool teaching force is over age 50, steadily approaching the average teacher retirement age of 59. During the next decade, at least “ 1.8 million K

- 12 teachers and school leaders will be eligible to retire” (Foster, 2010, p. 2 ). The report also indicates th at surveys show many of these retirees want supplemental careers where they can continue to contribute their expert ise and skills to the field of education after retiring. It is for these reasons that the results of this study will be of interest not only to large, urban districts, such as A urora P ublic Schools, but to suburban and rural districts as well who also grapple with an aging teacher workforce and teacher turnover challenges. **Overview of the Study Design** The purpose of the study is to evaluate the impacts of the Retired Mentors for New Teachers Program on teacher effectiveness and teacher retention. The program is based on the following theory of change: **By providing probationary teachers in high need schools with regular coaching and support from a**  **recently retired, master educator, the Retired Mentors for New Teachers program will enhance the probationary teacher’s instructional practice leading to an increase in student achievement on district**  **assessments and improved teacher evaluation performan ce and retention.**  This theory of change is supported by the logic model shown in Figure 1. This model outlines the current situation and needs in the target district, the major components of the intervention provided through the Retired Mentors for New Teachers Program , and the targeted outcomes.4

**Figure 1. MENTORING INTERVENTION LOGIC MODEL** 

The study will randomly assign teachers to either receive added mentoring support from a retired master educator along with the district’s typical mentoring support, or to receive only the district’s typical mentoring support . The typical mentoring approach currently operated in Aurora can best be described as a “buddy” mentoring model that assigns each new teacher to work with a more experienced “buddy” teacher for one year. The district expects the “buddy” mentor to spend 15 contact hours with their mentee over the course of the year, but the degree of mentoring provided can vary significantly based on the willingness and availability of the mentor to advise the mentee
. Only district t eachers in their first year in th e profession receive the typical mentoring model. Teachers with more than one year of teaching experience do not receive district mentoring. In contrast to the typical mentoring approach, the intervention includes the following components :

 The provision of highly

- experienced mentors to work directly with teachers in their first three years with the district regardless of prior experience . Mentors will be assigned by school and will be selected by a team of district leaders, including instructi onal coordinators and the head of professional development, to ensure their quality. **Major Intervention Components**

 Pair probationary teachers in high need elementary schools with recently retired, master educat ors that have: o History of success in the district. o Years of experience with district expectations and student challenges
. o Flexibility to meet with probationary teachers before, during, or after school. o No input into teacher evaluations.

 Mentoring provided over two school years.
 Two half days of summer professional development for probationary teachers.
 Individualized , one - on - one mentoring and classroom support.
 M entoring support with other probationary teachers
.  Mentor meetings with principals to ensure co mmon understanding of school priorities. **Improved Outcomes**  Students : Improved instruction delivered by probationary teachers i ncreas es their students’ achievement as measured by reading and math assessments. (Confirmatory outcome measure). Teachers : Improvement in administrator evaluations of probationary teachers. (Exploratory outcome measure). Reduced probationary teacher turnover. (Exploratory outcome measure). **Current Situation and**  **Need Diagnosis** 

- Children in high need elementary schools continue to underperform in reading and math.

- Probationary teachers require added support to deliver effective reading and math instruction.

- Retaining teachers in high need schools remains an ongoing challenge.5

 Criteria for mentor selection will include substantial teaching or school leadership experience in APS (a minimum of five years), experience working in high

- need school s, and a record of excellence in the district as reflected by student performance, performance reviews, and reputation as an effective educator. Selected mentors will meet quarterly to discuss and continually improve their practice.

 Treatment group partic ipants will receive i ndividualized mentoring and classroom support
. Treatment teachers will receive tailored mentoring throughout the course of each year. This tailored m entoring will be flexible , using the mentor’s experience to identify the key needs of each teacher that they work with and to create a mentoring plan that matches each teacher’s individual strengths and weaknesses. It is anticipated that: o The first year of mentoring will focus on creating and maintaining an effective classroom environment for student learning. In this regard, mentors will draw upon content from California’s Continuum of Teaching Practice (BTSA, 2012). More details on this Continuum and the creation of an effective classroom environment for student learning are provided below. o Establishment of an effective classroom environment for student learning provides the foundation for establishing “conditions of learning” in the classroom, which will also be a focus of mentoring support. At APS, “conditions of learning” refers to the model of learning espoused by Brian Cambour ne. This model contains seven key learning elements: 1) immersion; 2) demonstration; 3) expectation; 4) responsibility; 5) use; 6) approximation; and 7) response. 1 o Subsequent support will delve more deeply into:
 Mastery of academic content,
 Use of additional assessment techniques to guide instruction,
 Use of data to inform instruction,
 Development of capacity for reflective teaching and continuous improvement, and
 Other improvement opportunities based on teacher, school, or district priorities.

o Mentoring will draw on a number of strategies based on the needs of each teacher. The district does not believe that if a mentor chose not to use a specific strategy, that it would weaken the program. Thi s is because the decision regarding the best mix of approaches to use with each mentee relies on the judgment of each mentor. This is an especially important component, because every mentee teacher comes to the district with a unique set of needs, skills, strengths, and weaknesses. The types of strategies

1 For more information see: http://instruction.aurorak12.org/files/2011/06/Conditions

- ofLearning

- Detailed.pdf6 used by mentors may include observation, co

-teaching, lesson modeling, and visits to other classrooms or schools to observe model instruction.

o Mentoring will include both individual and group support th at involves groups of probationary treatment teachers within schools. Group work will be designed to build a community of learners among probationary teachers within the school that can be sustained past the life of the mentoring program.

 Treatment group participants will be involved in group support . Mentors and the probationary teachers in each school receiving the treatment will meet 1

- 2 times per month as a group. While treatment teachers may teach different grade levels, the purpose of these treatment group meetings will be to: share common challenges, share strategies for addressing common challenges, identify any areas where additional mentoring focus is needed or desired, and b uild a network within the school that can support new teachers when the mentor is not present.

 Retired m entors will meet with school principals . Each mentor will meet with the school principal to discuss a variety of topics designed to improve the mentor’ s understanding of the current school culture and setting and to allow the principal to communicate key priorities and expectations for all new teachers at that school. Mentor meetings are not to be used to share information regarding individual mentee performance
. A n initial meeting with the principal will establish the expectations around the mentoring program, including that mentors do not serve in an evaluative capacity but are designed to serve merely as a support and critical friend to mentees. Subs equent meetings with the principal over the course of the year will be designed for the mentor to: o Gain input from the principal on potential areas of desired focus for new teacher support based on school

- or district
- level priorities and schoolwide benchm ark testing data ; and o Discuss appropriate alignment of mentoring support with other district coaching or induction initiatives
. The project will include two analyses : 1) a n implementation analysis , designed to document the inte rvention’s implementation and to provide information regarding the contrast between the mentoring received by teachers in the treatment and control groups; and 2) an impact analysis , with confirmatory impact questions focused on student achievement in read ing and math, and exploratory impact questions focused on teacher retention and on principal evaluations of teachers. The primary purpose of this study is to provide an unbiased estimate of the mentoring program’s impact on student achievement in reading a nd math. The secondary purpose of the study is to provide an unbiased estimate of the program’s impact on teacher retention and teachers’ instructional quality. The impact analysis will be supplemented with the implementation analysis, which is designed to document the inte rvention’s implementation and to provide detailed information on the contrast between the7

mentoring received by teachers in the treatment and control groups. The implementation analysis will provide useful information needed to interpret the findings regarding the impact of the program. **Overview of the Specific Data Collection Plan** The impact analysis will use student test scores in reading and math on district

- administered assessments, teacher turnover data in study schools, and teacher evaluation ratings . Assessments will be given in the Fall and Spring of each school year and will provide a pre

- and post
- source of data with which to compare the performance of students served by teachers in the control group with the performance of stude nts served by teachers in the treatment group. The district will also provide its data regarding teacher turnover (teachers leaving the district) across control and treatment groups, and scoring data from the district’s teacher evaluation process, which wi ll allow comparison of evaluation ratings across control and treatment groups. The implementation analysis requires collecting data using three researcher

- developed instruments: (1) a teacher survey questionnaire ; (2) a focus group protocol for teacher men tees; and ( 3) a focus group protocol for mentors
. The implementation analysis also will make use of existing data , including detailed records of mentor support. Mentors will make preliminary notes for each mentee on an ongoing basis and submit completed records monthly
. These records will document the dosage and type of mentoring provided to each teacher
. REL Central will administer the teacher survey electronically to all probationary teachers (control and treatm ent ) at participating school sites once in the Spring of each school year during both years of the intervention
. The surveys will be used to gather data on teacher background and prior experiences as well as to gather data on the dosage of mentoring received (both typical mentoring and added mentoring provided through the intervention), the types of mentoring support received, and to gather any lessons learned regarding the program. The survey instrument is included in Attachment A. REL Central will conduct four 60
- minute focus groups in the spring of each school year with probationary teachers that are receiving the treatment in participating schools, using a semi

- structured Tea cher Focus Group Protocol developed by REL Central. The purpose of these focus g roups will be to gather input from the teachers regarding the quality, frequency, and intensity of mentoring support received and to validate data gathered through the mentor records of support regarding content of mentoring received. The focus group proto col is included as Attachment B. Focus groups will also be conducted once each spring with mentors. Focus groups will use a Mentor Focus Group Protocol developed by REL Central. The focus groups will address a variety of topics, including implementation fidelity and adherence to the program design, lessons l earned in implementation, and key areas of mentoring focus
. Focus groups will be limited to 60 minutes, with questions prioritized to aid focus group facilitators in the event of time constraints. The focus group protocol is included as Attachment C. Records of m entor support will be filled out monthly by each mentor for work con ducted with each mentee teacher to document topics of mentoring each month, and methods used to deliver support8

(such as in

- cl ass support, organizing site visits, modeling instruction, etc.). The completion of the record of s upport form is part of each mentor’s job as required by the district
. This expectation is communicated to mentors by the district head of professional develo pment prior to the start of the school year, including the expectation that record of s upport forms are to be completed and turned in monthly to the district
. The district head of professional development will meet with mentors at least quarterly and will monitor any issues associated with the regular and timely completion of the Record of Support forms. Consistent completion of the forms will contribute to the head of professional development’s evaluation of mentor performance. Mentors will be compensated directly by the district for their time in filling out the records of support. The records will be provided to the district head of professional development who will share them with REL Central. **Table 1. Timeline for the Data Collection**

**Timeframe Data**  **Collection** Spring 2014, 2015 Administer teacher s urvey, conduct mentee

and mentor focus groups July 2014, 2015 District provides data on teacher turnover as

well as teacher evaluation data May 2014, October 2014 District provides data from fall student

assessment measure in reading and math May 2014, 2015 District provides data from spring student

assessment measure in reading and math The implementation analysis data collection schedules are based on the assumption that OMB approval will be received for these data collections. If OMB approval is not received , only data will be analyzed using data that the district collects
. **A2.	How,	by	Whom,	and	for	What	P urpose**  **I nforma** **tion	is	to	Be	U sed**  Findings from this study will be used by state and district lea ders to inform decisions regarding the types of support to provide new and probationary teachers. This study will result in a report intended for district and state leaders who are responsible for developing and implementing teacher mentoring and support systems and overseeing support for teachers’ professional growth and effectiveness. The report will provide the findings in an accessible format, describing possible interpretations and implications of the results. These findings will be helpful for states and districts in the Central R egion who are exploring innovative ways to provide support to new teachers in their schools. Researchers will also facilitate discussions of the results with state and district leaders to examine and int erpret the results from this study. For the study’s impact analysis, student achievement data from the district’s MAP R eading and M ath Assessments will be used to compare the achievement of children served by a teacher that received added mentoring support through the intervention to the achievement of children served by teachers in the control group. The MAP assessments are standardized tests given by the district that are aligned to standards across the nation . The district has purchased and used these assessments and will deliver9

them and collect the data in study schools
. The district’s d ata on teacher turnover will be used to compare turnover rates (the proportion of teachers leaving the school district) between treatment and control groups of teachers. D istrict
- collected d ata on teacher evaluations will be used to compare teacher performance across treatment and control groups as measured by school administrative leader scoring on a district
- created evaluation rubric. For the study’s implementation an alysis, d ata collected from teacher surveys and focus groups as well as mentor focus groups (both conducted by REL Central) will be used to assess the experiences of probationary teachers with mentoring and to compare these experiences across the treatment and control groups. These data sources will inform researchers about : whether the intervention is being implemented as planned; the types of support provided by mentors; and lessons learned through the mentoring process. M entor records of support, which a re already collected by the district, will be used to triangulate the survey and focus group input. The table below provides an overview of the data collection methods that will be used to answer implementation questions. **Table 2. Connecting Data**  **Collection Methods to Implementation Research Questions**  

### Data **Collection Methods Implementation Research Question**  Teacher survey Records of

Mentor Support Mentee

focus groups Mentor

focus groups Are the components of the mentoring program being implemented as planned? 

#    

Is the program reaching the intended target population with the appropriate services at the planned rate and “dosage”? 

#    

What are the background characteristics and experience of mentors and mentees? 

#  

How does the mentoring experience and dosage differ between the 

# 10

treatment and control group teachers?
What are key lessons learned that might improve the program over time? 

#  

**A3.** 

### Use of Automated, Electronic, Mechanical or Other Technological **Collection Techniques**  The teacher survey will be administered online using survey software such as SurveyMonkey in order to reduce the burden on respondents. REL Central’s conversations with the district indicate that teachers are familiar and comfortable with SurveyMonkey and electronic surveys. Respondents will be sent a link via email that will lead them to the online survey. A progress bar will indicate the percentage of the survey instrument completed so teachers can see the progress they will make in responding to questions. Reminder emails will be sent electronically to facilitate higher response rates. Mentee and mentor focus group discussions will be captured electronically through note taking on laptop computers. The focus groups will be recorded t o ensure accuracy . Records of Mentor Support will be collected by the district electronically and coded by the mentor . **A4.	Efforts	to	Identify	D uplication.**  The impact of the mentoring program will be examined using several sources of existing district data
. The existing data includes teacher evaluation data , teacher turnover data , and data from student reading and math assessments
. Mentor records of support will assist in tracking program implementation. However, implementation will also be examined using several new sources of data, including information gathered from surveys of treatment and control group teachers and focus groups of mentors and treatment group teachers . The district does not currently collect such implementation data on the program , and this type of information is not available from any other data sources. **A5. Sensitivity	to	Burden	on	Small	Entities**  The data collection does not involve schools or other small entities. **A6. C**  **onsequence to	Federal	Program	or	Policy	Activities	if	the	Collecti**  **on	is	Not	Conducted	or is	Conducted	Less	F**  **requently** If the proposed data were not collected, the goals of the IES Regional Educ ation Laboratory program may not be met. For instance , REL Central would not be able to contribute to key IES goals, including go als to “conduct and support high quality stu dies on key regional priorities” that “ incorporate data and research into everyday decision
- making .” If the data were not collected , decision makers in the region11

would not have access to the research needed to help meet these key goals. 2 Furthermore, the district would like to use the research

- based evaluation results to inform policy decisions about their mentoring program in the future. Because the district’s program is designed specifically as a two

--year int ervention, REL Central will assess ongoing implementation over two school years. **A7. Special	Circumstances**  There are no special circumstances. **A8. Federal	Register	Announcement	and	Consultation**  **Federal Register Announcement** We will publish Federal Register Notices to allow both a 60

- day and 30
- day public comment period. We will summarize and address public comments received. The 60

- day Federal Register Notice was published on [DATE] ( Attachment X). The 30

- day Federal Register Notice was published on [DATE] ( Attachment X). **Consultations Outside the Agency** We have consulted with Aurora Public School District leaders o n the availability of data, the clarity of the instructions, and the data gathering instrument s . Additionally, we have consulted with content and methods experts to develop the study including technical working group members Dr. Bruce Randel, President of Century Analytics ; Dr. Linda Damon, retired Director of Professional Learning; and Dr. Michelle Reininger, Assistant Professor at Stanford Uni versity and Executive Director of the Stanford Center for Education Policy Analysis
. As appropriate, we will consult with representatives of those from whom information is to be obtained , such as research alliance members, to address any public comments re ceived on burden. **T able 3. I**  **ndividuals consulted on the statistical, data collection, and analytic aspects of this study Name**  **Title Organization**  **Contact Information**
Michelle Reininger Executive Director Center for Education Policy Analysis at Stanford University reininger@stanford.edu ;

(650) 736

- 1258
Bruce Randel President Century Analytics bruce.randel@centuryanalytics. com ; (720) 488

- 5503 Linda Damon Retired (Former Director

of Professional Learning) (Formerly) Aurora Public Schools damon.l@att.net

2 IES goals for Regional Educational Laboratories can be found on the following website: http://ies.ed.gov/ncee/edlabs/about/
.12

**A9. Payment	or	Gift	to	R** **espondents** The school district is covering all payments to teachers and mentors who participate in surveys or focus groups . This compensation is not paid by REL funds
. Teachers will be offered an incentive of $ 25 each to participate in a survey or focus group
. Focus groups are expected to take approximately an hour, while surveys are expected to take no more than 2 0 minutes. Because the district is paying all incentives, the incentive amounts are determined based on district requirements and standard operating p ractice. The district rate for compensating teachers for work outside of regular contract hours or for participating in research activities is $25 per hour. The results of the Reading First Impact Study ( RFIS ) incentives study were considered, which found that incentives significantly increase response rates in surveys (Gamse, Bloom, Kemple, & Jacob, 2008). Teachers will receive the incentive upon completion of the survey. Maintaining use of current district stipend levels during the study has the added be nefit of allowing the district to maintain consistency in how it compensates teachers if the district wish es to continue its own similar data gathering through surveys and focus groups past the life of this study. **A10. Confidentiality	of	the	Data**  REL Centra l will be following the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 , which states : “All collection, maintenance, use, and wide dissemination of data by the Institute ” are required to “ conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h). ” These citations refer to the Pr ivacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. Subsection (c) of section 183 referenced above requires the Director of IES to “ develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data
. ” Subsection (d) of section 183 prohibits the disclosure of individually identifiable information , and makes any publishing or communicating of individually identifiable information by employee s or staff a felony. REL Central will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating i nstitutions and respondents will be presented at aggregate levels in reports. Unique identification numbers will be assigned to each participating teacher and used to identify all survey responses. The ID number/name association files will be kept secure i n a confidential file separate from the data analysis file. No information identifying respondents will be included in the study data files or reports. Data from the online survey software system will be downloaded and deleted from the online system after the survey window closes. These data files will then be stripped of any identifying information. Information will be reported in aggregate so that individual responses are not identifiable. All identification lists will be destroyed at the end of the proje ct. The research team is trained to follow strict guidelines for soliciting consent, administering data collection instruments, and preservi ng respondent confidentiality. All members of the research team have successfully completed13

the Collaborative Instit utional Training Initiative (CITI) course in the Protection of Human Research Subjects courses through Liberty IRB. REL Central will collect consent forms from study participants which will also be stored in a locked file. C op ies of the teacher and mentor consent form s are provided in Attachments D and E , respectively
. A copy of the affidavit of non

- disclosure is provided in Attachment s I and J for each researcher who will have access to the data. All study materials will include the following language: Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific individual. We will not provide information that identifies you to anyone outside the study team, except as required by law. Any willful disclosure of such information for non

-statistical purposes, without the informed consent of the respondent, is a class E felony. **A11. Additional	J** **ustif**

**ication	for	S ensitive**  **Questions** No questions of a highly sensitive nature are included in the survey or interview. **A12.	Estimates	of	Hour	B urden**  The total burden associated with the two -year data collection is 274 hours (see Table 4 ). Approximately 10 0 respondents will be invited to take the teacher survey . The target response rate is 80%
. Of the 100 teachers, 50 will be randomly assigned to rece ive the added mentoring treatment from a retired, master educator and 50 will be randomly assigned to the control group and will receive the district’s business as usual mentoring support. While both treatment and control group teachers will be asked to ta ke the survey, the survey will be considerably shorter for control group teachers (about half as many questions) because they will not be asked to answer a set of questions on their experience with a retired mentor. The approximate time for completing the survey for control group teachers is 1 0 minutes. The approximate time for completing the survey for treatment group teachers is 2 0 minutes. There will also be up to three email reminders to individuals who do not complete the survey on the first notificat ion. Each follow

- up email is estimated to take 2.5 minutes (or 0.04 hours) for each teacher to read
. Approximately 40 teachers will participate in one - hour focus groups once per school year (in the Spring) , and a pproximately 8 mentors will participate in a one - hour focus group once per school year (in the Spring)
. An in
- person meeting to discuss the research project with 100 teachers will take approximately 30 minutes and will allow teachers to sign consent forms and will includ e information about survey and focus group participation
.14

It is anticipated that district personnel will spend no more than 40 hours per school year providing required data for the study. It should be noted that the district already collects and compiles th e data required for the study . T herefore, only a small amount of additional time is needed to provide this data to REL Central
. With regard to use of the MAP assessment for the study’s student outcome measure, the assessment will be administered by the di strict and not REL Central. Collection of assessment data will also be conducted by the district and not REL Central. The district has agreed to share this assessment data with REL Central
. T he following table presents each data collection task, the numb er of individuals participating in each data collection task for each of the two years, the average time per participant for each task, and the total time to complete each task
. **Table 4**  **: Participation**  **t imes** 

**Data collection task Number of**  **Participants Average Time** 

**(hours) Total Burden** 

**(hours)**
Approximately 100 teachers will be invited to attend an in

- person meeting in which

they will learn about the study and will sign consent forms. In
- person meeting **Year 1** 100 0.50 50 Of the 100 teachers, there will be 50 in the treatment group. With an 80% response rate in the first year, 40 are expected to complete the survey, with an estimated burden of 2 0 minutes. There will also be 50 in the control group. With an 80% response rate in the first year, 40 are expected to complete the survey, with an estimated burden of 1 0 minutes. We expect a 20% attrition rate in survey completion from year 1 to 2, leaving us with 32 participants in each group in year 2. Online survey for treatment group teachers **Year 1** 40 0. 33 13. 2 Online survey for control group teachers **Year 1** 40 0. 17 6. 8 Online survey for treatment group teachers **Year 2** 32 0. 33 10. 5 6 Online survey for control group teachers **Year 2** 32 0. 17 5. 44 Approximately half of the 100 teachers will require reminders to take the online survey each year. Each of the 3 email reminders is estimated to take about 2.5 minutes (0.04 hours). Read survey reminders **Year 1** 50 0.12 (0.04 x 3) 6 Read survey reminders **Year 2** 50 0.12 (0.04 x 3) 6 Forty teachers (4 focus groups with a maximum of 10 teachers) will participate in a focus gro up each year. Eight mentors (1 focus group with a maximum of 8 mentors)

will participate in a focus group each year. Teacher focus groups **Year 1** 40 1 40 Mentor focus groups **Year 1** 8 1 815 **Data collection task Number of**  **Participants Average Time** 

**(hours) Total Burden** 

**(hours)**
Teacher focus groups **Year 2** 40 1 40 Mentor focus groups **Year 2** 8 1 8 District staff will spend a total of 40 hours per year providing required data to support the project Provision of data **Year 1** 1 1 40 Provision of data **Year 2** 1 1 40 **Year 1 Total 164**  **Year 2 Total 110**  **Total across two school years (2013** **– 14, 2014** **– 15)**  **274**

T he total burden in hours over the two years of the study is 274 hours. **A13. Estimate	for	the	Total	Annual	Cost	Burden	to	Respondents	or	R** **ecord K** **eepers** There are no start- up costs for this collection. As s hown in Table 5 , the annual cost is $ 3, 3 60 in the first school year and $ 2 , 960 in the second school year, which the district has agreed to pay. The total cost over two years is therefore $ 6,320
. The district w ill compensate mentees and mentors for any time associated with participating in focus groups or surveys. To reiterate, this compensation will not be paid by REL funds. The district will compensate teachers $25 for participation in each survey or focus gro up, based on current district policy. The district compensates its mentors based on flat daily rates which amount to an average cost of $45 per hour. The following table presents the number of participants, the time to complete each data collection task an d the associated monetary burden on the district to compensate participants (at the rate of $25 per focus group or completed survey for teachers and $45 per hour for mentors )
. **Table 5**  **: Participation**  **t imes**  **and estimated monetary burden**

**Data collection task Number of**  **p articipants**  **Participation**

**c ost** * **Estimated**

**monetary burden**  **district will pay**  Of the 100 teachers, there will be 50 in the treatment group. With an 80% response rate, 40 are expected to complete the survey. There will also be 50 in the control group. With an 80% response rate, 40 are expected to complete the survey. Online survey for treatment group teachers **Year 1** 40 $25 $1,00016 **Data collection task Number of**  **p articipants**  **Participation**

**c ost** * **Estimated**

**monetary burden**  **district will pay**  Online survey for control group teachers **Year 1** 40 $25 $1,000 Online survey for treatment group teachers **Year 2** 32 $25 $800 Online survey for control group teachers **Year 2** 32 $25 $800 Forty teachers will participate in focus group s each year. Eight mentors will participate in a focus group each year. Teacher focus groups **Year 1**  40 $25 $1000 Mentor focus groups **Year 1**  8 $ 4 5 $360 Teacher focus groups **Year 2**  40 $25 $1000 Mentor focus groups **Year 2**  8 $ 4 5 $360 **Year 1 Total $3,**  **360 Year 2 Total**  **$ 2,960**  **Total across two school years (2013** **– 14, 2014** **– 15)**  **-- $**  **6 ,320** 

**A14.Estimates	of	Annualized	C osts	to	the	Federal**  **G overnment**  The total estimated cost for this study is $ 1 , 025, 000 over five years. The average yearly cost is $ 2 0 5 ,000. The costs include developing, administering, and analyzing the survey; conducting and analyzing the focus group resul ts; gathering and analyzing district data (teacher turnover, teacher evaluations, student reading and math performance ) and report writing and development
. The costs also include personnel costs of several federal employees involved in project oversight an d analysis that amount to an annual cost of $5,000 for fed eral labor. The total annual cost for the evaluation is therefore the sum of the annual contracted evaluation cost ($200,000 ) and the annual federal labor cost ($5,000) or a total of $205,000 per ye ar. **A15. Reasons	for	Program	Changes	or	A** **djustments** This is a new study. **A16. P** **lans	for Tabulation	and	P** **ublication.** The district will administer the mentoring intervention over two school years to the group of teachers receiving the treatment. Teachers will be randomly assigned to treatment and control groups and the17

school district will collect data on the characteristics of the sample of schools such as : school size, student
-teacher ratio, principal turnover rate, and teacher turnover rate. **I mpact**  **Analysis** The impact analysis will address the following confirmatory questions: 1. What is the impact of the Retired Mentors for New Teachers program on elementary students’ reading assessments? 2. What is the impact of the Retired Mentors for New Teachers program on elementary students’ assessments in math? The impact analysis will address the following exploratory questions: 1. Does being a member of the treatment group in the Retired Mentors for New Teacher program impact teacher turnover? 2. Does being a member of the treatment group impact principal evaluations of teachers? Analyses will be conducted to provide descriptive statistics of the sample of schools. Descriptive statistics will be calculated and presented for the following school

- level variables : S chool size , student
teacher ratio, p rincipal turnover rate , teacher turnover rate
. A second set of descriptive analyses will describe the characteristics of teachers assigned to the treatment and control groups as a result of the baseline characteristics obtained from the teacher survey. Statistics will be presented for : Gender, e ducation level , n umber of years of total teaching experience, n umber of years teaching in Aurora , and g rade level assignment at year- 1 baseline
. A third set of d escriptive analyses will compare the students in the treatment and control groups using : Grade level , gender, race/ethnicity, and e ligibility for free or reduced price lunch
. **Data Collection Procedures** The table below summarizes the instruments used in the study as w ell as the corresponding proposed implementation dates
. **Table 6**  **. Data collection instruments**  **for impact analyses**  Data Collection Instrument for Confirmatory Impact Research Questions **Data Already Collected by the District?** Student performance on grades 1–5 MAP Reading Assessments Yes. The district currently has a contract with NWEA to use the MAP assessment. Student performance on grades 1–5 MAP Math Assessments Yes. The district currently has a contract with NWEA to use the MAP assessment
. **Data Collection Instrument for Exploratory**  **Impact Research Questions: Data Already Collected by the District?** 18 Data provided by the district of principal evaluation ratings of probationary and control teachers in participating schools. Yes. These ratings are currently generated by principals in all schools and are collected electronically by the district. Teacher turnover data collected from the district human resource s department Yes. Teacher turnover data is routinely collected by the district.
For data on student characteristics and performance, the data collection is focused at the district level. For the two research questions addressing student reading and math achievement, REL Central will use the results of MAP Assessments (developed by Northwe st Evaluation Association

- NWEA). MAP is a standardized test, given under standardized conditions. It is an adaptive, electronic assessment which provides RIT scores aligned on a common vertical scale. The adaptive nature of MAP allows the assessment to a djust the difficulty of each question based on how well a student answers previous questions. As the student answers correctly, questions become more difficult. If the student answers incorrect ly, the questions become easier . The assessment then produces final RIT scores in both reading and math ( Northwest Evaluation Association , 2009). NWEA conducts RIT scale norming studies every three years. The latest study was conducted in 2011 and included data drawn from over 5 million students taking the test in all 50 states (Northwest Evaluation Association , 2011)
. For the research question on teacher turnover, REL Central will use data provided and electronically collected by the district’s human resource s office. For this study, teacher turnover is defined as those study sample teachers who leave the Aurora Public School district before the start of the 201 5– 16 school year (that is, as of the July 2015 data collection point). We use this definition because one of the goals of the intervention is to reduce teach er turnover from the district. The impact on turnover will be estimated at the teacher level, which is the level of random assignment. For the research question on principal evaluation of teachers, REL Central will use data gathered by the district using a teacher evaluation rubric. This rubric outlines the knowledge and skills required of an effective teacher within five major standards: pedagogical expertise ( u p to 24 points ; establish ing a safe and respectful learning environment ( u p to 24 points) ; plan ning and deliver ing effective instruction ( u p to 32 poi nts); reflect ing on practice ( u p to 12 points) ; and demonstrating leadership ( u p to 16 points). Teachers will receive a rating for several elements under each of the five quality standards from their principal or other administrative supervisor. The ratings are rigorously scored based on a check list on a five point scale from “not evident (0 points)” to “exemplary (4 points).” All administrators in Aurora Public Schools that directly supervise teachers will be trained by the district to reliably administer the rubric for all teachers during the S pring of 2013. Baseline Equivalency We plan to test the baseline equivalency for several reasons. First, it is teachers who are randomly assigned to the experi mental groups, not the students. Second, student mobility within the target district is relatively high, such that attrition rates may exceed those considered acceptable by the What19

Works Clearinghouse ( U.S Department of Education , n.d. ). Establishment of baseline equivalency of the analytic samples will be critical to supporting a causal inference. These analyses will compare the treatment and control students on the baseline data for the confirmatory outcomes only for those students included in the impact analysis sample. These analyses will account for the cluster of students within classroom. We plan to include baseline measures of student achievement in all of our impact analyses. The study can meet What Works Clearinghouse standards if the difference in baseline achievement is no greater than 0 .25 standard deviation s . Should a baseline difference be grea ter than 0 .25 standard deviation we still plan to estimate the program’s impact because Aurora Public Schools is interested in the impact of the program despite any baseline differences. We will inform the district that findings from a study sample where the baseline equivalence difference exceeds the What Works Clearinghouse standard should be interpreted with caution. Impact A nalysis Models Program impacts will be estimated for the confirmatory outcomes and each of the exploratory outcomes. Analytic models described here pertain directly to the confirmatory student achievement outcomes and will serve as the benchmark impact model. The p urpose of the benchmark confirmatory impact model will be to provide an unbiased estimate of the mentoring program’s impact on the two confirmatory outcomes : student reading achievement and student math achievement. The impact will be estimated at the teac her/classroom level, which is the level of random assignment. The purpose of the first exploratory research question will be to estimate the impact of the intervention on probationary teacher turnover. For this outcome, turnover is defined as those teachers who leave the district any time during the study period up to July 2015. The impact on turnover will be estimated at the teacher level, the level of random assignment. The purpose of the second exploratory research question will be to estimate the impact of the intervention on principal evaluations of probationary teachers. The district will use an evaluation rubric that is based on five standards of teaching quality. Though each of the standards are evaluated on a five

point ordinal scale, the combination of the results across the five

- point scales generates a value for each evaluated teacher that closely resembles an interval scale. For the purposes of our analysis, we will therefore treat the teach er evaluation results data as an interval scale. The data that results from these evaluations will be used as the dependent variable in the exploratory analysis of the impact of mentoring on teacher evaluations. Table 7 is an example shell of a table that we will use to present the results of our impact analysis.20

**Table 7. Example t** **able s**  **hell**

**Regression adjusted posttest means (standard deviation)** 

**Measure Intervention group**  **Control group Estimated**  **difference a**  **(standard error) 95 percent** 

**confidence interval**  **p -** **value Effect size**  **b** Outcome A () () () Outcome B () () () Outcome C () () ()

a. Estimated difference may not equal difference between means because of rounding. b. Calculated as the estimated difference divided by control group standard deviation. Note: Numbers in parentheses are standard errors. All results are based on analysis using a mixed

- model approach to account for the sources of variability in the data that resulted from the nested structure of the school environment. Sensitivity Analyses Sensi tivity analyses will be conducted for the confirmatory impact analyses of the student achievement outcomes only. Sensitivity analyses will test the robustness of the benchmark impact estimates to : inclusion of covariates; treatment of missing pretest data ; inclusion of the students’ year 1 teacher status variables; inclusion of the teacher transfer variables ; possible moderation of the treatment effect by teacher grade level ; and potential moderation of the year 2 treatment effect by student exposure levels to the treatment in year 1
. Addressing Multiple Comparisons The program’s impact will be estimated for two confirmatory student achievement outcomes: English language arts scores and mathematics scores. Further, impacts will be estimated for these two out comes using the year 1 student sample and also using the year 2 student sample. The What Works Clearinghouse applies a correction for multiple hypotheses testing to studies that test a given outcome measure with multiple comparison groups ( U.S Department of Education, n.d. ). The What Works Clearinghouse does not apply a correction to outcomes from different domains, only to outcomes within21

the same domain. This study’s two confirmatory outcomes fall into two domains as defined by the What Works Clearinghous e. Student English language arts scores come under the general reading achievement domain ( U.S Department of Education, n.d. ). Student math scores come under the elementary school mathematics achievement domain ( U.S Department of Education, n.d. ). The year 1 comparison group and the year 2 comparison group are different comparison groups. Given the above, we plan to apply the Benjamini

- Hochberg correction separately to the results from two comparison groups used for the reading outcome and to the results from the two comparison groups used for the mathematics outcome. **Implementation A** **nalysis** The primary purpose of th e implementation analysis is to capture the extent to which the program was implemented as intended and to better understand any vari ations in implementation across school sites. A secondary purpose is to contribute to a greater understanding of the context and process of implementation, which will be of great interest to Aurora Public Schools and any other districts who might be intere sted in implementing this intervention in the future
. Key implementation research questions include:
 Are the components of the mentoring program being implemented as planned?  Is the program reaching the intended target population (core subject elementary school teachers in their first three years at the district) with the planned “dosage”?  What are the background characteristics and experience of mentors and mentees?  How does the mentoring experience and dosage differ between the treatment and control group teachers? Using data collected through surveys, mentor records of support, and focus groups, we will describe the following for the treatment stream:
 Teacher and mentor background and prior experience  The extent to which mentoring met expected dosage levels  The extent to which mentoring met program fidelity expectations with regard to receiving a mix of mentoring support through individual and group sessions  The methods of mentoring support provided (for example, whether mentors used observation, co -te aching, or modeling of instruction)  Key topics addressed in mentoring  Whether there are any special circumstances and constraints surro unding the mentoring experience. Qualitative findings will be summarized as needed using traditional qualitative data analysis methods. The approach used will follow methods explicated by Miles and Huberman (1994). This approach emphasizes well

- defined study variables to ensure the compa rability of cross

- site data and reduction of data using data displays and matrices so that common themes can be identified. We will analyze all data for key themes that characterize the implementation of the program, including the methods of support received by the intervention group. Such themes include a comparison of the average amount and intensity of mentoring activities received by the treatment and control group22

teachers. To ensure the validity of these comparisons, we will use uniform survey data co llected annually from teachers in both research groups to conduct these analyses. **Publication Plans** All results for REL Central’s rigorous studies will be made available to the public through peer

- reviewed evaluation reports that are published by IES. The datasets from these rigorous studies will be turned over to the REL’s IES project officer. REL Central will follow IES guidelines in creating restricted
- use datasets and supporting documentation which will be provided at the end of the project, . Even the REL contractor would be required to obtain a restricted

- use license to conduct any work with the data beyond the original evaluation. In preparation for the production of a draft report in 2015 , we expect to convene a number of meetings in the S p ring and Fall of 2015 to review the report outlines and report. These meetings will include TWG members and other relevant experts. The interim and final reports will target a district

- level policymaking audience and will be disseminated via the official I ES channels, including the national REL website. The study’s timetable for data collection and analysis is presented below : **Table 8**  **: Schedule of a**  **ctivities Activity**  **Schedule**

**Implementation Analysis** Hold in

- person consent meeting Upon OMB approval Administer teacher surveys Spring 2014, Spring 2015 Collect records of mentor support Spring 2014
–Spring 20 15 Conduct teacher focus groups Spring 2014, Spring 2015 Conduct mentor focus groups Spring 2014, Spring 2015 Analyze and report June– November 2015 **Impact Analysis** District provides descriptive data on schools, teachers, and students Spring 2014 District provides student reading and math performance data Spring 2014, Fall 2014, Spring 2015 Collect teacher evaluation data Ju ly 2014, Ju ly 2015 Collect teacher turnover data July 2014, July 2015 Analyze and report June– November 201523

**A17. Approval	to	Not	Display	the	Expiration	D**  **ate	for OMB	Approval**  We are not requesting approval to not display the expiration data for OMB approval. **A18. Exception	to	the	Certification	S** **tatement** No exceptions to the certification statement are being sought.24 

## References Aaronson, D., Barrow, L., & Sander, W. (2007). Teachers and student achievement in the Chicago public schools. Journal of Labor Economics , 25 (1), 95
– 135. Enders, C. K., & Tofighi, D. (2007). Centering predictor variables in cross

- sectional multilevel models: A new look at an old issue. Psychological Methods, 12 (2), 121

- 138. Foster, E. (2010). How boomers can contribute to student success: E merg ing encore career opportunities in K
- 12 e ducation
. Washington, D.C. : National Commission on Teaching and America’s Future
. Gamse, B. C, Bloom, H.S., Kemple, J.J., Jacob, R.T., ( 2008). Reading First Impact Study: Interim Report (NCEE 200 8–4016). Washin gton, D
. C
. : National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Glazerman, S., Isenberg, E., Dolfin, S., Bleeker, M., Johnson, A., Grider, M. &Jacobus, M . (2010). Impacts of Comprehensive Teacher Induction: Final Results From a Randomized Controlled Study (NCEE 2010
-4027). Washington, DC: National Center for Education
. Hedges, L. V. (2007). Effect sizes in cluster

- randomized designs
. Journal of Educational and Behavioral Statis tics, 32(4), 341
–370. Hedges, L. V. (2009).“Effect sizes in nested designs.” In H. Cooper, L. V. Hedges & J. C. Valentine (eds.), Handbook of research synthesis and meta

-analysis, 2nd Edition. New York: Russell Sage Foundation. Hedges, L. V. & Olkin, I. (1985). Statistical methods for meta

-analysis . San Diego, CA: Academic Press. Ingersoll, R. & Strong, M. (2011) “The Impact of Induction and Mentoring Programs for Beginning Teachers: A Critical Review of the Research,” Review of Educational Research. Retrieved from http://rer.sagepub.com/content/81/2/201. Konstantopoulos, S., & Chung, V. (2010). The persistence of teacher effects in elementary grades. American Educational Research Journal 48 (2), 361
–386. Miles, M. D. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook. Newbury Park, CA: Sage. Northwest Evaluation Association. (2005). RIT scale norms for use with achievement level tests and measures of academic progress. Lake Oswego, OR: Author. Northwest Evaluation Association. (2 009). Technical manual for Measures of Academic ProgressTM and Measures of Academic Progress for primary gradesTM. Lake Oswego, OR: Author.25

Northwest Evaluation Association . (2011). 2011 RIT Scale Norms, Frequently Asked Questions
. Lake Oswego, OR: Author. Nye, B., Konstantopoulos, S., & Hedges, L. V. (2004). How large are teacher effects? Educational Evaluation and Policy Analysis , 26(3) , 237–257. Smith, T. & Ingersoll, R. (2004). What are the effects of induction and mentoring on beginning teacher turnover ? American Educational Research Journal, 41 (3), 68 1–7 14
. Strong, M. (2006). Does new teacher support affect student achievement? [Research brief]. Santa Cruz, CA: The New Teacher Center. Wright, S. P., Horn, S. P., & Sanders, W. L. (1997). Teacher and classroom context effects on student achievement: implications for teacher evaluation. Journal of Personnel Evaluation in Education 11 : 57–67.