APPENDICES

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 136 

Appendix D: Summary of Performance Evaluations Released 

During FY 2016–17 and Expected During FY 2017–18 

For a complete list of program evaluations and studies from the Office of Planning, Evaluation, 

and Policy Development, please visit 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html. For a complete list of evaluation 

studies of the National Center for Education Evaluation and Regional Assistance, please visit 

http://ies.ed.gov/ncee/projects/evaluation/index.asp. 

Evaluation Reports From FY 2016–17

National Center for Education Evaluation and Regional Assistance

Literacy 

Summary of Research Generated by Striving Readers on the Effectiveness of 

Interventions for Struggling Adolescent Readers 

1. Study Purpose: Striving Readers was a discretionary grant program focused on raising 

reading achievement of middle school and high school students through intensive 

interventions for struggling readers and enhancing the quality of literacy instruction across 

the curriculum. The 2006 and 2009 cohorts of grantees were required to conduct rigorous, 

experimental evaluations. The Institute of Education Sciences (IES) provided technical 

assistance to support grantees in conducting evaluations that would meet What Works 

Clearinghouse (WWC) standards. This study also conducted a systematic review of the 

17 evaluations of 10 different interventions funded by Striving Readers across the 16 grants. 

2. Key Question Addressed: 

 What do the evaluations funded by the Striving Readers program tell us about effective 

interventions for improving the achievement of struggling adolescent readers? 

3. Design: The study is descriptive; it provided technical assistance to the local evaluations of 

Striving Readers grantees and also reviewed the completed evaluations against the WWC 

evidence standards. The report summarizes the results of this systematic review of the 

evaluations. 

4. Estimated or Actual Completion Date: The final report was released in October 2015. 

5. Key Findings: 

 Fifteen of the 17 evaluations of the interventions funded by the Striving Readers grant 

program met WWC evidence standards with or without reservations. 

 Based on findings from the 15 evaluations that met WWC evidence standards with or 

without reservations, 4 of the 10 interventions funded by the Striving Readers grant 

program had positive, potentially positive, or mixed effects on reading achievement. 

 The Striving Readers studies not only expand the evidence base on effective reading 

interventions for adolescents, but also increase the number of high-quality effectiveness 

studies reviewed by the WWC on this topic. Three of the four interventions that had 

positive, potentially positive, or mixed effects on reading achievement had not previously 

been reviewed by the WWC. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 137 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20164001/ 

School Choice 

Evaluation of the DC Opportunity Scholarship Program: Descriptive Reports Based on 

Parent and Principal Surveys 

1. Study Purpose: The April 2011 Scholarships and Opportunities for Results Act (SOAR Act) 

provided for a five-year continuation of a school choice program for low-income residents of 

Washington, DC. The program, still titled the Opportunity Scholarship Program (OSP), now 

provides annual scholarships of $8,000 (for grades K–8) or $12,000 (for grades 9–12) to 

enable low-income students to attend private schools in DC in lieu of the public schools 

already available to them. The new law also mandated another independent, rigorous 

evaluation of the effectiveness of the program. 

2. Key Questions Addressed:

 What is the impact of the OSP on student academic achievement and other measures of 

student success, overall and for subgroups of students identified in the statute as high 

priority? 

 What effect does the program have on student and parent perceptions of school safety 

and satisfaction, and on parents’ involvement in education of their children? 

 Does the program change students’ instructional environments and opportunities? 

3. Design: The evaluation will primarily compare outcomes of approximately 1,800 student 

applicants randomly assigned by lottery to either receive a scholarship or not receive a 

scholarship. Lotteries of program applicants were conducted in spring 2012 (cohort 1), 

spring 2013 (cohort 2), and spring 2014 (cohort 3). Data will be collected for three follow-up 

years for each of the cohorts and, for students in both the scholarship and nonscholarship 

groups. The contractor will administer academic assessments, and conduct student, parent, 

and principal surveys each spring (spring 2013–17). Prior to the first impact report, 

descriptive reports based on application forms parents filled out when they applied to the 

OSP, principal surveys, and extant data will be released. 

4. Estimated or Actual Completion Date: The second and third descriptive reports—focused on 

select findings from parent and principal surveys—were released in 2016. The first impact 

report is scheduled for completion in early 2017. 

5. Key Findings from the Second Report: 

 The majority of parents seeking a private school voucher for their child were satisfied 

with the school their child was attending at the time of application. Fifty-seven percent of 

applicants’ parents gave their child’s current school at time of application a grade of ‘A’

or ‘B’, while 14 percent gave their child’s school a ‘D’ or ‘F.’ 

 Most applicants’ parents (49 percent) chose academic quality as their top priority when 

selecting a new school. The next most commonly selected top priority was school safety 

(selected by 18 percent of applicants’ parents). 

 Among the 49 percent of parents that said academic quality was their top priority for a 

new school, over 85 percent were satisfied with the academic quality of the school their 

child was attending when they applied for a scholarship. This suggests that families do 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 138 

not always apply for private school vouchers because they are dissatisfied with key 

elements of the schools their children are already attending. 

Key Findings from the Third Report: Spring 2014 survey responses from principals at DC 

public schools—both traditional and charter—and private schools that participated in the 

OSP indicate that private schools participating in the OSP: 

 Were more likely than public schools to report that student behavior, student motivation, 

parental support for learning, and the instructional skills of teachers were excellent or 

very good; 

 Were less likely than public schools to suspend students, use metal detectors, and have 

on-site security personnel; 

 Taught reading and math for fewer minutes a week across grade levels than public 

schools; and 

 Were less likely than public school to have instructional programs for non-English 

speakers and students with learning disabilities. 

6. Links to Additional Information: https://ies.ed.gov/ncee/pubs/20164003/, 

https://ies.ed.gov/ncee/pubs/20164007/

Students with Disabilities 

Evaluation of Response to Intervention Practices for Elementary School Reading 

1. Study Purpose: The focus of this study was on the implementation and effects of Response 

to Intervention (RtI) practices for elementary school reading. RtI is a multistep approach to 

providing early and more intensive intervention and monitoring within the general education 

setting. In principle, RtI begins with research-based instruction provided to students in the 

general education classroom, followed by screening of all students to identify those who 

may need systematic progress monitoring, intervention, or support. Students who are not 

responding to the general education curriculum and instruction are provided with 

increasingly intense interventions through a “tiered” system, and they are frequently 

monitored to assess their progress and inform the choice of future interventions. This 

evaluation occurred under the National Assessment of the Individuals with Disabilities 

Education Act (IDEA), which permits districts to use RtI to inform decisions regarding a 

child’s eligibility for special education.

2. Key Questions Addressed:

 How do RtI practices for early grade reading vary across schools? 

 How do schools experienced with RtI vary the intensity of reading instruction for children 

at different reading levels? 

 What are the effects on grade 1–3 reading achievement of providing reading 

interventions to children who fell just below school-determined standards for each grade 

on screening tests? 

3. Design: The evaluation relied on a combination of descriptive data collection from school 

staff and regression discontinuity methods to address the research questions, and focused 

on practices in place during the 2011–12 school year in a sample of 146 elementary schools 

from 13 states. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 139 

4. Actual Completion Date: The study report was released in November 2015. 

5. Key Findings:

 Full implementation of the RtI framework in grade 1–3 reading was reported by 

86 percent of the experienced elementary schools sampled for in-depth study, compared 

with 56 percent of a random sample of elementary schools from the same 13 states. 

 The schools sampled for in-depth study of their RtI practices provided more intense 

small-group instruction and more reading intervention services for groups of students 

reading below grade level than for groups of students reading at or above grade level. 

However, these schools’ implementation of RtI differed in some ways from descriptions 

in the prior literature: 

 In grade 1, 45 percent of schools provided intervention services to some groups of 

students at all reading levels, rather than only for reading groups below grade level. 

 In grade 1, 67 percent of schools provided at least some reading intervention during 

the core reading block, rather than only in addition to the core. 

 Grade 1 students who scored just below school-determined benchmarks on fall 

screening tests, and who were assigned to reading interventions, had lower spring 

reading scores than similar students in the same schools who were not assigned to 

those interventions. Unexplored, but plausible, factors that may be related to these 

negative effects include incorrect identification of students for intervention and a 

mismatch between the interventions provided and these students’ instructional needs. 

No statistically significant effects of assignment to reading interventions were found for 

students in Grades 2 or 3. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20164000/

Teacher Effectiveness 

Evaluation of the Teacher Incentive Fund: Implementation and Impacts of Pay-forPerformance After Three Years 

1. Study Purpose: The Teacher Incentive Fund (TIF) was renamed the Teacher and School 

Leader Incentive Program with reauthorization of the Elementary and Secondary Education 

Act (ESEA) in the Every Student Succeeds Act (ESSA). The purpose of the TIF program is 

to develop and implement performance-based compensation systems (PBCSs) for teachers, 

principals, and other personnel in high-need schools. Research indicates that high-quality 

teachers are critical to raising student achievement in low-performing schools, but schools 

most in need often have difficulty in attracting and retaining high-quality teachers. 

Performance pay is a policy promoted by the TIF program to improve the quality of teachers 

in high-need schools. This evaluation studies performance pay that provides substantial and 

differentiated bonus pay to high-performing teachers in low-performing schools with highneed students. 

2. Key Questions Addressed:

 What are the characteristics of all TIF districts and their performance-based 

compensation systems? What implementation experiences and challenges did TIF 

districts encounter? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 140 

 How do teachers and principals in schools that did or did not offer pay-for-performance 

bonuses compare on key dimensions, including their understanding of TIF program 

features, exposure to TIF activities, allocation of time, and attitudes toward teaching and 

the TIF program? 

 What is the impact of pay-for-performance bonuses on students’ achievement on state 

assessments of math and reading? 

 How do pay-for-performance bonuses affect educator mobility, including whether 

mobility differs by educator effectiveness? 

3. Design: Study schools were randomly assigned within a grant to either implement all 

components of the PBCS or the PBCS with a 1 percent across-the-board bonus in place of 

the differentiated effectiveness incentive component of the PBCS. Data collection will 

include a grantee survey, a survey of teachers and principals, teacher and principal school 

assignment records, student record information (such as student demographics and student 

test scores), and grantee interviews to document implementation information, as well as to 

conduct impact analyses. 

4. Estimated or Actual Completion Date: A report on the implementation and early impacts 

after one year was released in September 2014. The second report on the ongoing 

implementation and impacts after two years was released in September 2015. The third 

report on implementation and impacts after three years was released in August 2016. A 

fourth and final report is scheduled for release in the fall of 2017. 

5. Key Findings: The main findings among all TIF districts with 2010 awards are: 

 Similar to the previous two years, most districts (88 percent) implemented at least three 

of the four required program components for teachers. 

 By the third year, reported implementation challenges decreased, with no more than 

one-fifth of TIF districts reporting any major challenges. 

For the subset of 10 districts that agreed to participate in a random assignment study, 

key findings on the effect of pay-for-performance on educators include the following: 

 After three years of TIF implementation, average student achievement remained 1 to 

2 percentile points higher in schools that offered pay-for-performance bonuses than 

in schools that did not. This difference was equivalent to a gain of about four 

additional weeks of learning. 

 At least half of the evaluation districts each year met the grant guidance for awarding 

differentiated performance bonuses for teachers. However, in each year, no more 

than 2 of the 10 districts awarded bonuses for teachers that were substantial or 

challenging to earn. 

 Teachers’ understanding of performance measures continued to improve, but only about 

60 percent of teachers correctly reported being eligible for a performance bonus. In 

addition, teachers believed that the maximum bonus they could earn was no more than 

two-fifths the size of the actual maximum bonus that districts awarded, a finding similar 

to previous years. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20164004/ 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 141 

Focusing on Mathematical Knowledge: The Impact of Content-Intensive Teacher 

Professional Development 

1. Study Purpose: Title II, Part A of the current ESEA provides monetary resources to improve 

teacher quality through professional development. Currently there is relatively little rigorous 

empirical evidence on the components of an effective professional development program. At 

the same time, there has been renewed policy interest in fostering a globally competitive 

STEM workforce, which will require teachers to effectively impart mathematics to their 

students. This study will present implementation and impact findings from a random 

assignment evaluation of an intensive math professional development intervention focused 

on enhancing teacher content knowledge and integrating this knowledge into the classroom. 

2. Key Questions Addressed:

 What is the average impact on (a) teachers’ content knowledge, (b) teachers’ classroom 

practices, and (c) student achievement of offering a specialized professional 

development intervention (in this case, Intel Math, Mathematics Learning Communities, 

and Video Feedback Cycles) relative to “business-as-usual” professional development? 

 How is the professional development intervention implemented? 

3. Design: Approximately 200 volunteer fourth-grade math teachers from six districts around 

the country participated in this study during the 2013–14 school year. The intervention has 

three components: i) Intel Math (80 hours largely focused on developing teachers’

understanding of K–8 mathematics), ii) Mathematics Learning Communities (which offers 

teachers opportunities to connect their learning to the classroom through a professional 

learning community aligned to Intel Math that focuses on student thinking), iii) Video 

Feedback Cycles (which provide teachers with individualized feedback three different times 

based on videotaped lessons of their classroom teaching reviewed by certified analysts 

using the Mathematical Quality of Instruction [MQI] rubric). Within each participating school, 

fourth-grade teachers were randomly assigned to receive either the professional 

development intervention or business-as-usual professional development. Data collection 

includes measures of teacher knowledge and practice, and student achievement obtained 

from administrative records. 

4. Estimated or Actual Completion Date: A report on implementation and impacts was released 

in September 2016. 

5. Key Findings: 

 The professional development had a positive impact on teacher knowledge: Average 

scores on a study-administered math test were 21 percentile points higher for teachers 

who received the study professional development than for those who did not. 

 The professional development had a positive impact on some aspects of instruction: 

Average ratings of teachers’ use and quality of math explanations in the classroom were 

23 percentile points higher for teachers who received the study professional 

development than for those who did not. 

 The professional development did not have a positive impact on student achievement: 

Students of teachers who received the professional development scored, on average, 

2 percentile points lower than control teachers’ students on both a study-administered 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 142 

math test and state assessment. In general, this difference was not statistically 

significant. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20164010/

Other 

Case Studies of Schools Receiving School Improvement Grants: Final Report 

1. Study Purpose: School Improvement Grants (SIGs) are authorized by Title I, Section 

1003(g) of the ESEA. The purpose of the grants—awarded based on the Title I funding 

formula to states, which then competitively distribute the funds to districts applying on behalf 

of their eligible schools—is to support the turnaround of the nation’s persistently lowestachieving schools. The first cohort of SIG, which is the focus of this study, received funding 

beginning in the 2010–11 school year. To qualify for the three-year grant in the first cohort, 

schools must (among other requirements) have been willing to implement one of four 

prescribed intervention models: turnaround, restart, closure, or transformation. About 

$546 million was allocated in FY 2009 for SIG with a supplement of $3 billion from the 

Recovery Act. With the possibility of rollover funds, this amounts to a $3.5 billion injection 

into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This study 

will provide descriptively rich, primarily qualitative information for a small set of schools 

receiving SIG in the first cohort to implement an intervention model beginning in the 

2010–11 school year. 

2. Key Questions Addressed:

 What is the background and context of these persistently lowest-achieving schools? 

How do the leadership and staff in these schools define the performance problem, and 

to what do they attribute their problems? 

 What leadership styles do the principals of these persistently lowest-achieving schools 

exhibit? What actions do these schools engage in to try to improve their history of low 

performance? 

 What is the change process in these persistently lowest-achieving schools, particularly in 

terms of school climate and staff capacity? 

 What improvements do school stakeholders perceive during the three-year course of 

SIG, and do these improvements appear to be sustainable? 

3. Design: This study employs a school-level case study design. A core sample of 25 SIG 

schools was purposively selected from six states to represent a range of geographic 

regions, urbanicities, school sizes, racial/ethnic compositions, socioeconomic statuses, SIG 

intervention models, and SIG funding levels, among other factors. Data collection took place 

over three school years, beginning in spring 2011 and concluding in spring 2013, and 

included interviews with each state’s SIG leaders, a teacher survey, and site visits to the 

case study schools, which included analysis of fiscal records, as well as interviews and 

focus groups with district officials, principals, teachers, parents, union officials, external 

support providers, and students. 

4. Estimated or Actual Completion Date: A final report was released in April 2016. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 143 

5. Key Findings: 

 Most case study schools (22 of 25) replaced their principal at least once in the year 

before SIG or the first two years of SIG: Two of the four SIG intervention models 

required the principal to be replaced. About half of the new principals were described by 

school staff as an improvement over their predecessor. 

 Twelve of the 25 case study schools replaced at least half of their teachers by the 

second year of SIG: Respondents in more than half of the 12 schools characterized the 

change as positive. All but one of the 25 schools created new nonteaching positions in 

the first two years of SIG, with the most frequent positions being instructional, 

technology, or data coaches and additional school administrators. 

 In 15 of the 25 case study schools, most of the teacher professional development was 

job-embedded: According to teacher survey responses, professional development more 

often focused on math, literacy, or data use than classroom management or improving 

instruction for English learners and special education students. In most schools, 

teachers reported changing their practice after participating in professional learning on 

math, literacy, or data use. 

 Sustaining improvements may be challenging: In more than half of the 12 schools the 

study followed for all three years of SIG, teachers felt their school had changed in 

primarily positive ways. However, just two of these schools show strong prospects for 

sustaining improvement, while six show mixed prospects, and four show weak 

prospects. The schools that had higher organizational capacity by the third year of SIG 

had higher sustainability prospects. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20164002/

Policy and Program Studies Service (PPSS) 

Implementing Accountability and Supports Under ESEA Flexibility 

1. Study Purpose: This study examines the early implementation of the ESEA Flexibility 

initiative at the state and local levels. The study includes reviews of state and local 

documents; interviews with education officials at the state, district, and school level in early 

implementing states; and analysis of baseline student outcome and other data obtained 

through extant data sources. 

2. Key Questions Addressed:

 What are the primary components of state accountability and support systems under 

ESEA Flexibility? 

 What modes of communication did states and districts use to inform and build local-level 

understanding of state accountability and support systems under ESEA Flexibility? How 

did local-level officials perceive state accountability systems under ESEA Flexibility, 

including the criteria being used to identify low-performing schools? 

 What approaches did states take to identify and intervene in low-performing districts and 

schools? What improvement strategies did principals report implementing? 

 What supports were states and districts providing to low-performing schools? How did 

officials perceive their capacity to implement the ESEA Flexibility provisions for state 

support and accountability systems, and what challenges did they report? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 144 

3. Design: The primary data sources for this study included state ESEA Flexibility applications, 

state and district policy documents, and interviews with officials identified by state and 

district leaders as holding primary responsibility for accountability and support systems 

under ESEA Flexibility. 

4. Actual Completion Date: The report was released in February 2016. 

5. Key Findings: 

 In developing updated accountability systems, five of the 12 states in the study used a 

combined subgroup as part of their system, and eight of the 12 devised their own goals 

for measuring student achievement. Eight of the 12 created a designation category for 

low-performing schools that did not meet the priority and focus designation criteria. 

 State officials reported providing support to priority and focus schools through regional 

technical assistance organizations, external support providers, and early warning 

systems. Ten of the 13 priority and focus school principals described receiving more 

frequent or more intensive support than in years prior to ESEA Flexibility. 

 State officials described building capacity through cross-agency collaborations, 

developing staff expertise, partnering with districts, implementing organizational 

changes, and consolidating funding streams. 

 Nearly all state officials (11 of 12) reported holding periodic committees, advisory 

councils, and commissions to share ESEA Flexibility update to districts. 

6. Link to Additional Information: http://www2.ed.gov/rschstat/eval/title-i/implementingaccountability-esea-flexibility/report.pdf

Evaluation of the Teacher Incentive Fund: Final Report 

1. Study Purpose: This study examines program implementation in the first two cohorts of 

grantees for the TIF program (2006 and 2007), which included 33 grantees. All 33 grantees 

implemented performance pay systems for principals and other school administrators; 31 

grantees also included teachers in their performance pay systems. The final report 

complements a previous 2012 report from this study on the early implementation of the 

program, and analyzes award payouts and educator perspectives on a variety of issues 

related to implementation. 

2. Key Questions Addressed: 

 How did the size and distribution of incentive awards vary across educators and 

grantees? 

 To what extent were incentive payments based on student achievement vs. other factors 

such as supports for improving practice, performance evaluations, and teaching in hardto-staff schools or subjects? 

 How did participating educators perceive the fairness and effectiveness of the 

performance pay system? 

3. Design: Data collection included surveys of a representative sample of teachers and 

principals in schools served by TIF projects; financial data for awards paid to educators; 

interviews with TIF project directors, teachers, principals, and other stakeholders; and 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 145 

reviews of extant documents. The surveys were conducted in spring 2011, and the incentive 

award payout data are primarily for the 2010–11 school year. 

4. Actual Completion Date: The report was released in June 2016. 

5. Key Findings: 

 Across all educators who received an incentive award, the average award was $3,651 

for teachers and $5,508 for administrators. These average awards represented 

approximately 8 percent of average teacher salaries and 6 percent of average principal 

salaries. 

 Within grantees, the average teacher award ranged from $1,170 for the grantee with the 

smallest average award to $8,772 for the grantee with the largest average award 

(2–24 percent of average salaries), and the average administrator award ranged from 

$814 to $10,711 (1–13 percent of average salaries). 

 Awards for student achievement gains comprised the largest share of incentive payouts 

to teachers and administrators; 64 percent of teacher award payouts and 63 percent of 

administrator award payouts were based on student achievement. 

 Smaller proportions of incentive payouts were based on performance evaluations 

(13–14 percent), participation in supports for improving practice (12–14 percent), and 

working in hard-to-staff schools and subjects (6–12 percent). 

 Teachers were less likely than principals to agree that their performance pay system 

was fair (46 percent vs. 64 percent). However, 55 percent of both teachers and 

principals agreed that the possible award size was large enough to motivate them to 

earn it. 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#tq 

Case Studies of the Implementation and Use of Kindergarten Entry Assessments (KEAs) 

1. Study Purpose: The purpose of the KEAs implementation case studies is to examine the 

processes, accomplishments, challenges, and solutions of four states implementing KEAs 

and to share what they have learned with federal and state policymakers and the field. Of 

particular interest is to identify what is working well in states that are early adopters of KEAs. 

This information is needed to support the technical assistance efforts of the Department and 

to inform KEA efforts across the nation. 

2. Key Questions Addressed:

 How did the four case study states develop or adopt KEAs? 

 How did the four states train teachers to administer KEAs and to what extent did 

teachers feel prepared to do so? 

 What were the KEA implementation experiences of the 12 case study districts? 

 To what extent did the states, districts, and schools in the study use KEA results to 

inform policy and practice? 

 What challenges did the case study sites experience with KEAs, and what strategies did 

sites use or suggest using to address these challenges? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 146 

3. Design: The study consisted of document reviews, telephone interviews with state agency 

respondents and local preschool directors, and in-person interviews with district 

administrators, principals, kindergarten teachers, and other KEA assessors. Data were 

collected in 12 districts and 23 schools in Maryland, Oregon, Pennsylvania, and Washington 

between January and June 2015 and include information from a total of 201 interviews. 

4. Actual Completion Date: The report was released in August 2016. 

5. Key Findings: 

 State officials and stakeholders considered multiple criteria when developing or adopting 

KEA measures: reliability and validity, appropriateness for all students, usefulness for 

informing classroom instruction, usefulness for informing early learning policies and 

program improvement, feasibility of administration by teachers, and cost. 

 The four states trained teachers on KEA administration through self-paced webinars, 

in-person presentations, and train-the-trainer models. A majority of the interviewed 

teachers said the training prepared them to administer the KEA to students, though 

many teachers reported that they had difficulty in determining what were appropriate 

accommodations for English learner students and students with disabilities and indicated 

that they needed further assistance. 

 District officials reported working to reduce the burden associated with KEA data 

collection and entry by purchasing new technology, providing staffing assistance to 

teachers with KEA administration, and omitting or delaying other assessments. 

 Although the majority of interviewed teachers reported that they had not yet used formal 

KEA reports to inform their instructional practices, a few teachers said that the 

impressions they gained while administering the KEA helped them to understand their 

students’ strengths and needs and to assign students to instructional groups. 

 District administrators and teachers identified challenges with administering KEAs with 

English learner students and students with disabilities, using KEA results to inform 

instruction, and sharing KEA data with parents; they suggested that state officials could 

address these challenges by providing explicit training on these topics, on-site coaching, 

and tailored reports to help educators use and share the data. 

6. Link to Additional Information: http://www2.ed.gov/rschstat/eval/disadv/kindergarten-entryassessments/report.pdf

A Study of Practices to Improve Learning in Kindergarten Through Third Grade 

1. Study Purpose: This study summarized how (a) aligning preschool through third grade 

education and (b) differentiating instruction for children in kindergarten and first grade may 

build on the positive effects of preschool and help students in kindergarten through third 

grade (K–3) make cognitive, social-emotional, and academic gains. In addition, the study 

conducted case studies of five programs that may build on the positive effects of preschool 

by using policies, programs, and practices for alignment and differentiation. 

2. Key Questions Addressed in the Literature Review: 

 What approaches does the research and theoretical literature suggest for aligning 

preschool through third grade (P–3) education, and what is the quality of the research 

studies? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 147 

 What are the findings from studies of differentiated instruction on children in kindergarten 

and first grade, and what is the quality of these studies? 

Key Questions Addressed in the Case Studies: 

 What approaches did the five programs use to implement P–3 alignment? 

 In programs that implemented differentiated instruction, what approaches did staff use? 

3. Design: The literature review covered two topics: (1) preschool and K–3 alignment, and 

(2) differentiated instruction in kindergarten and first grade. The case studies examined five 

sites that implemented P–3 alignment or differentiating instruction in kindergarten and first 

grade. The research team interviewed principals, teachers, evaluators, and funders to 

understand programs’ characteristics, challenges and solutions, and the sustainability of the 

programs. 

4. Actual Completion Date: The literature review was released in August 2016, and the case 

study report was released in December 2016. 

5. Key Findings from the Literature Review: 

 Nearly all qualitative studies and policy and theory articles on P–3 alignment suggest 

aligning standards, curriculum, instruction, assessments, and environments across 

preschool and grades K–3.

 P–3 alignment could be supported by establishing similar teacher education and training 

requirements and developing longitudinal data systems that integrate preschool and 

K–12 data. 

 Of the 17 quantitative studies of differentiated instruction, one RCT of the Individualized 

Student Instruction With Assessment to Instruction intervention demonstrated positive 

results on reading outcomes and had the potential to meet the criteria for strong causal 

evidence. 

 Qualitative studies of differentiated instruction indicate that opportunities for peer 

collaboration and guidance by mentors may be helpful to improve teacher practice 

related to differentiation. 

Key Findings from the Case Studies:

 All five programs aligned instruction across grades by coordinating standards, curricula, 

instructional practices and professional development. 

 Common elements of these programs included the use of professional learning 

communities, coaches, parent engagement, and play-based or student-initiated learning. 

 All five programs reported using strategies to accommodate students’ different skill 

levels, including modifying assignments, adapting learning materials, providing different 

levels of support, or using small-group instruction. 

6. Links to Additional Information: 

Literature review: http://www2.ed.gov/rschstat/eval/disadv/p-3-alignment-differentiatedinstruction/report.pdf 

Case studies: http://www2.ed.gov/rschstat/eval/implementing-early-strategies/report.pdf
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 148 

Study of Emerging Teacher Evaluation Systems 

1. Study Purpose: This study will provide descriptive information on the design and early 

implementation of teacher evaluation systems in eight school districts. The findings are 

intended to help other districts and states learn from the experiences of eight districts 

featured in the study, and apply the design and implementation lessons to their own work as 

it relates to teacher evaluation and support. 

2. Key Questions Addressed: 

 What key priorities and measures informed the design of the new teacher evaluation 

systems? 

 What steps did the districts take prior to full implementation to test the system and 

prepare teachers and staff to implement it? 

 How did the districts structure and conduct the classroom observation component of 

their teacher evaluation systems? 

 How did the districts analyze student performance and other data to evaluate teacher 

performance? 

 How did the districts use, or plan to use, teacher evaluation results to make personnel 

decisions? To what extent were professional development and career advancement 

decisions tied to evaluation results? 

 What administrative structures did districts use to support their new teacher evaluation 

systems? 

 What are the perceived early effects of the teacher evaluation systems on the 

professional practices of teachers, principals and district administrators? 

3. Design: This descriptive study relied on interviews with key district administrators, principals, 

teachers, and representatives of community stakeholder groups, from eight districts, who 

were involved in the development and early implementation of the respective districts’ 

teacher evaluation system. Given the limited sample, the findings cannot be generalized to 

other districts. 

4. Actual Completion Date: The final report was released in November 2016. 

5. Key Findings: 

 Teachers and central office staff generally agreed that the foremost goal of the teacher 

evaluation system was to improve instruction.

 Teacher and principal input during the design and/or pilot test phase strongly influenced 

decisions regarding system modifications in six districts, according to district 

administrators. 

 Classroom observations varied in frequency, duration, and degree of formality in all eight 

districts. In addition, principals reported challenges in finding time to conduct teacher 

observations. 

 Six districts used multiple approaches for measuring teacher impact on student 

performance, including individual and/or school-level value-added models. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 149 

 Districts used teacher evaluation results for a range of purposes, including targeted 

professional development and support, career ladders and performance pay, and in 

some instances, redeployment or release of teachers identified as ineffective. 

 The majority of districts created relatively simple, streamlined structures to administer 

their teacher evaluation systems. 

 Teachers reported that they believed that the classroom observations and feedback 

helped them become better teachers. 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#tq

State Efforts to Promote Equitable Access to Effective Teachers 

1. Study Purpose: This report provides a broad overview of state efforts, as of the 2011–12 

school year, to monitor equitable access to qualified and effective teachers among schools; 

develop and adopt multiple measures of teacher performance to rate teachers among at 

least three performance levels; and implement targeted strategies for promoting equitable 

access to qualified and effective teachers in schools serving high proportions of poor and/or 

minority students. The report examines the use of measures of teacher qualifications and 

teacher performance in the implementation of these strategies. 

2. Key Questions Addressed: 

 What measures did states use to monitor equitable access to qualified and effective 

teachers among schools? 

 To what extent were states developing or using multiple measures of teacher 

performance to rate teachers among at least three performance levels? 

 What strategies did states use to promote equitable access to qualified and effective 

teachers in schools serving high proportions of poor and/or minority students? 

3. Design: This report is based on telephone interviews with officials in SEAs in all states, the 

District of Columbia, and Puerto Rico. Study staff also reviewed extant sources including 

equity plans, Consolidated State Performance Reports, reports from federal monitoring 

visits, and other information on state websites. The interviews were conducted between 

August 2011 and January 2012, and the study reflects state efforts that were underway at 

that time, and predates implementation of the fall 2011 ESEA Flexibility initiative as well as 

the recent state equity plans submitted under the Excellent Educators for All initiative. 

4. Actual Completion Date: The final report was released in January 2017. 

5. Key Findings: 

 In 2011–12, states most commonly monitored equitable access to qualified and effective 

teachers among schools using measures of teacher qualifications. 

 Four states reported using measures of teacher performance—student achievement 

growth and/or measures of teacher practice—to monitor equitable access to qualified 

and effective teachers among schools, and nine states used teacher performance 

measures to monitor the quality of the teacher workforce overall. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 150 

 In two of the four states that reported using teacher performance measures to monitor 

equitable access among schools, officials reported seeing larger inequities than were 

previously detected using measures of teacher qualifications alone. 

 Six states reported that they had adopted multiple measures of teacher performance and 

were using them to rate teachers among at least three performance levels in 2011–12, 

and 38 states indicated that there were in the process of developing such measures. 

 Offering monetary incentives was the most common strategy that states reported using 

in disadvantaged schools to promote equitable access to qualified and effective teachers 

among schools (24 states). 

 Other state-reported strategies that were directed specifically at disadvantaged schools 

for promoting equitable access were specialized professional development (14 states) 

and teacher recruitment and preparation programs (14 states). 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#tq 

Feasibility Study on Improving the Quality of School-Level Expenditure Data 

1. Study Purpose: This study explores the feasibility of improving the collection of school-level 

expenditure data by examining the nature and quality of school-level fiscal data collection in 

five states and four school districts that had developed their own systems for collecting and 

reporting school-level expenditures: Florida, Hawaii, Ohio, Rhode Island, Texas, Baltimore 

City, Hillsborough County, Houston, and Los Angeles 

2. Key Questions Addressed: 

 In states and districts that have developed systems to report expenditures at the school 

level, what types of personnel and nonpersonnel expenditures are included in the 

school-level data? 

 To what extent do the sites track actual expenditures to individual schools versus 

allocating or prorating expenditures to schools using formulas? 

 How consistent are school-level expenditure data obtained from these systems with 

similar data from other sources? How do the funding amounts attributed to individual 

schools based on formula allocations compare to those based on tracking actual 

expenditures? 

 What lessons can other states and districts learn from these sites if they wish to 

implement systems for reporting accurate and reliable data on school-level 

expenditures? 

3. Design: The study included (1) surveys and interviews of officials to understand the process 

of collecting school-level expenditure data and (2) collection and analysis of school-level 

spending data to examine data quality issues. The study examined three aspects of data 

quality: the comprehensiveness of school-level spending data, consistency with other data 

sources, and the relative accuracy of allocating expenditures to schools by formula (rather 

than tracking actual expenditures for each school). 

4. Actual Completion Date: The final report was released in January 2017. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 151 

5. Key Findings: 

 Study sites reported that they attributed most categories of spending to the school level, 

including salaries for teachers, administrators, and other support staff as well as 

nonpersonnel items, such as textbooks, instructional materials, furniture and equipment, 

and computers and software. 

 The school districts and states in this study attributed an average of three-quarters of 

operational expenditures to individual schools, demonstrating that it is feasible to link a 

significant share of spending to the school level. 

 Most of the expenditures that the study sites attributed to schools were directly tracked 

to schools (85 percent) rather than simply being allocated by formula (8 percent). 

 Comparisons between the site-reported school-level expenditures and other data 

sources showed a relatively high degree of consistency for salary expenditures, but 

nonpersonnel expenditure data were much less consistent. 

 Allocating expenditures to schools by formula (e.g., based on total salaries or staff) 

appeared relatively accurate for health benefits and less accurate for pension benefits, 

pupil support staff, and instructional support staff. 

 Instituting a system for collecting school-level expenditure data typically required new 

hardware and software (eight sites), changes to charts of accounts (six sites), and staff 

training (eight sites). 

 Advice that interviewees offered for others aiming to implement school-level expenditure 

data systems was to get stakeholders involved, communicate clearly and frequently, and 

think long-term about future data needs. 

6. Link to Additional Information: 

https://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#title 

Study of Experiences and Needs of Rural Education Achievement Program (REAP) 

Grantees and Subgrantees 

1. Study Purpose: This descriptive study examined how grantees and subgrantees use REAP 

funds provided through the Small, Rural School Achievement (SRSA) and Rural and LowIncome Schools (RLIS) programs—on their own or in combination with other federal funds—

as well as to explore technical assistance needs related to REAP. 

2. Key Questions Addressed:

 What role do states play in supporting the Department’s SRSA and RLIS eligibility and 

award determination process? 

 How do districts use their SRSA or RLIS funds? 

 To what extent do SRSA-eligible districts use REAP Flex and for what purposes? 

 What recommendations do states and districts have for improving the operation of the 

SRSA and RLIS programs? 

3. Design: The study consisted of: 1) a survey of a sample of approximately 1,000 SRSA 

grantees and RLIS subgrantee districts; 2) telephone interviews with a sample of 30 SRSA 

grantees and RLIS subgrantees; and 3) telephone interviews with REAP coordinators in all 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 152 

states receiving REAP funds. Data collection began in winter 2015 and was completed by 

spring 2015. At the state level, the study included interviews with all state REAP 

coordinators about state goals and priorities, the planning process for use of RLIS funds, the 

eligibility process for districts, management and distribution of SRSA and RLIS funds, and 

recommendations for the program. At the school district level, the study included an online 

survey of a nationally representative sample of REAP coordinators about the REAP 

eligibility determination process, administration of REAP, and challenges and technical 

assistance needs related to REAP, as well as telephone interviews with a subsample of 

30 districts. The interviews included questions about program administration, technical 

assistance needs, and recommendations for changing the REAP program to better meet the 

needs of rural districts. 

4. Actual Completion Date: The report was completed in December 2016. 

5. Key Findings: 

 States supported the Department in determining REAP eligibility by providing districtlevel data and reviewing the accuracy of Department-provided data. 

 All 43 states with RLIS-eligible districts chose to make subgrants to districts on the basis 

of a funding formula rather than on a competitive basis, and 28 of these states based the 

subgrant amount entirely on average daily attendance (ADA). 

 Districts most frequently used SRSA and RLIS funds to improve or expand access to 

technology (71 percent of SRSA districts and 71 percent of RLIS districts) and to provide 

educator professional development (45 percent of SRSA districts and 58 percent of RLIS 

districts). 

 Forty-six percent of SRSA district coordinators reported exercising REAP-Flex; of these, 

82 percent reported that they used funds eligible for REAP-Flex to maintain a stable 

level of funding for ongoing activities. 

 The majority of both district and state REAP coordinators were highly satisfied with 

REAP as a whole. However, they provided recommendations for improvement to REAP 

in three categories: (1) improved timelines for eligibility and award determination, 

(2) more information on allowable uses of funds and REAP-Flex, and (3) revised 

eligibility criteria. 

6. Link to Additional Information: 

https://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#rural
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 153 

Summary of Performance Evaluations Expected During FY 2017 and 

FY 2018*

National Center for Education Evaluation and Regional Assistance

Pathways to Career or College 

Enhancing Advising to Improve College Fit in Upward Bound 

1. Study Purpose: Growing concern over college enrollment and completion rates has 

heightened interest in cost-effective strategies to improve the outcomes of low-income 

students, including those in high-profile college access programs. One aspect of this 

concern is that many low-income high school students fail to enroll in colleges that are well 

aligned with their qualifications, talents, and needs. This report provides the early impacts of 

a low-cost enhancement to current college advising approaches in Upward Bound that is 

designed to improve college fit and persistence. The approach includes professional 

development for Upward Bound advisors and user-friendly packets of information for 

students that demonstrate the value of considering a range of institutional indicators in their 

college application, search, and acceptance process.

2. Key Questions Addressed: 

 Can an enhanced college advising approach improve upon what Upward Bound 

grantees are already doing—i.e., does it have positive effects on the number of colleges 

to which Upward Bound participants apply, the quality/selectivity of the colleges in which 

they enroll, and their persistence? 

 In what types of grantees is this approach most effective and with what types of 

students? 

3. Design: About 200 Upward Bound grantees that volunteered were randomly assigned in 

spring 2015 so that half received the professional development and packets to begin using 

with their rising 2015–16 seniors (the treatment group) and half did not receive the 

enhancements for use with rising 2015–16 seniors (the control group); however, the control 

group did receive training and packets a year later to use with rising 2016–17 seniors. A 

survey administered in spring 2016 to seniors in both sets of projects collected information 

about their college planning, including the number of applications. The study will also 

examine those seniors’ later enrollment and persistence in college using administrative 

records. 

4. Estimated or Actual Completion Date: The first report on early impacts (number of college 

applications, FAFSA completion by March 15) is expected in fall 2017. A second report 

examining impacts on college enrollment and fit is expected in early 2018. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/pathways_upward.asp 

 

*

 This section was last updated in April 2017. Key findings are included for reports that were released during the first half of FY 2017 

(October 2016 to March 2017). All other reports that are expected but not yet released in the second half of FY 2017 or in FY 2018 

have estimated completion dates only. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 154 

Is Access to Data Enough? The Effects of Providing Information on Student FAFSA 

Completion Directly to School Districts 

1. Study Purpose: Completing a Free Application for Federal Student Aid (FAFSA) is a key but 

challenging task for many students applying to college, particularly for low-income students. 

The Department has responded to concerns about the complexity of the FAFSA, including 

through the FAFSA Completion Project which started with a 20-district pilot in 2010 and 

expanded in 2012 to almost 100 more. One component of the project was consistent urging 

by federal officials and some technical assistance for participating districts to help them 

conduct FAFSA promotion and outreach activities districtwide. But the Department also 

provided districts with ongoing and direct access to federal data on which seniors had not 

submitted a FAFSA so that schools could provide targeted support to individual students. 

The evaluation tested whether that data access had an impact on FAFSA completion rates 

above and beyond the other FAFSA activities the districts might have implemented. The 

study was conducted by IES staff and designed to be low-burden, and low-cost, depending 

only on the Department’s data for measuring effectiveness.

2. Key Question Addressed: 

 Does making the student-level data available improve FAFSA completion rates? 

3. Design: Just over 60 districts that had more than one high school participated in the 

evaluation, for a total of 567 schools. IES randomly assigned the schools within each district 

to either an “early start” (treatment) group, which would have access to the FAFSA 

completion status for 2013 seniors, or a “later start” (control) group, which would have 

access for 2014 seniors. Because many districts had relatively small numbers of schools, 

pairs of schools were matched prior to random assignment to help ensure that important 

characteristics of schools were balanced between the treatment and control groups. At the 

beginning of the evaluation, districts provided the lists of all seniors in each school and 

these were matched against the Department’s federal student aid database so that FAFSA 

completion rates in March 2013 and September 2013 could be compared between the two 

groups of schools. 

4. Estimated or Actual Completion Date: The report is expected in fall 2017. 

5. Link to Additional Information:

https://ies.ed.gov/ncee/projects/evaluation/pathways_fafsa.asp

The Effectiveness of Text Messaging to Support College Transition of GEAR Up Students 

1. Study Purpose: GEAR Up is one of the U.S. Department of Education’s college access 

programs, funding states or local partnerships of districts and postsecondary institutions to 

serve students in high need schools beginning in 7th grade. The 2008 amendments to the 

Higher Education Act (HEA) allowed GEAR Up grantees to serve participating students 

beyond high school graduation and into a first year of college, when they might be 

dispersing to a variety of college campuses. This demonstration evaluates a low-cost way to 

provide these services, building on emerging evidence that customized text messages can 

help students overcome logistical and behavioral challenges that might otherwise derail their 

college matriculation and persistence into sophomore year. The messages include 

reminders and information relating to college registration, course selection, financial aid 

award and renewal, meeting with college advisors and faculty, and tuition payments, and 

provide a way to access real-time support from GEAR Up counselors. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 155 

2. Key Questions Addressed: 

 Do the text-based reminder strategies improve GEAR Up students’ rates of college 

enrollment and persistence? 

 For which types of students is the strategy more or less beneficial? 

3. Design: The first grantees eligible to use funds to support a year of post-high school 

services and who have seniors entering that phase have been targeted for the 

demonstration. Across about 80 schools, approximately 6,000 seniors in school years 

2015–16 or 2016–17 will be randomly assigned to receive either the college transition 

services grantees originally proposed in their applications or those regular transition 

services plus the customized reminders and support through text messages. Reminders and 

support will begin at the end of students’ high school senior year and continue into the 

spring of their expected first year of college. The study team will administer a survey before 

the reminder messaging begins, to collect information on students’ experiences with college 

advising and their intended college (so that the messages can be tailored to individual 

schools’ deadlines and requirements). College enrollment and persistence, as well as 

FAFSA renewal, will be measured using administrative records. 

4. Estimated or Actual Completion Date: The first report examining impacts on initial college 

enrollment is expected in fall 2018. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/pathways_gearup.asp

Upward Bound at 50: Reporting on Implementation Practices Today 

1. Study Purpose: Upward Bound is the oldest and largest of the federal college access 

programs targeted to low-income students and those who would represent the firstgeneration of college completers in their families. First established in 1965, the program 

currently serves more than 60,000 high school students at a cost of about $4,300 per youth 

with a wide array of academic and college transition support services. While much about the 

structure of Upward Bound and the services to be offered are prescribed in legislation, little 

is currently known about the intensity, duration, and mix of services provided by projects or 

about how they are delivered. Because of the importance of its mission, and the 

comprehensiveness and costs of its services, Upward Bound has long been of interest to 

policymakers. This report describes the approaches to providing program services as 

reported by Upward Bound project directors. 

2. Key Questions Addressed: 

 Within the core service areas of the program, where do projects focus their efforts? 

 How are services delivered to Upward Bound participants? 

 In what ways does the focus or delivery of services vary across different Upward Bound 

projects? 

3. Design: This descriptive report relies primarily on findings from a summer 2013 survey of all 

regular Upward Bound project directors. 

4. Estimated or Actual Completion Date: The report was released in November 2016. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 156 

5. Key Findings: 

 In four of the seven core service areas—coursework, tutoring, college exposure, and 

college application assistance—there was a dominant approach (used by at least 

50 percent of projects) to how projects focused their activities. There was no dominant 

approach to how projects focused their efforts when it came to academic advising, 

ACT/SAT prep, and financial aid prep services. 

 When, where, and how services were delivered differed across service areas. There was 

no dominant approach to when projects offered services except for tutoring, which was 

typically available after school. The dominant location for services (where) was at the 

projects’ host institution for coursework, college entrance exam prep, and college and 

financial aid application assistance, but other services were more likely to be provided at 

students’ high schools. Finally, tutoring and college entrance exam preparation services 

were most commonly delivered (how) in groups, while academic advising, college 

application assistance, and financial aid assistance were typically provided one-on-one.

 Variation in the focus and delivery of services appears related to the urbanicity and type 

of institution (four-year, two-year, and nonhigher education) that hosts the project but not 

to other project characteristics examined. There were few substantive differences (at 

least 10 percentage points) in the percentage of projects reporting each potential 

approach by project size (number of students served), per-student funding, and whether 

the host institution was a Minority-Serving Institution. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20174005/ 

School Choice 

Evaluation of the DC Opportunity Scholarship Program: Impacts After One and Two 

Years 

1. Study Purpose: The April 2011 SOAR Act provided for a five-year continuation of a school 

choice program for low-income residents of Washington, DC. The program, still titled the 

Opportunity Scholarship Program (OSP), now provides annual scholarships of $8,000 (for 

grades K–8) or $12,000 (for grades 9–12) to enable low-income students to attend private 

schools in DC in lieu of the public schools already available to them. The new law also 

mandated another independent, rigorous evaluation of the effectiveness of the program. 

2. Key Questions Addressed:

 What is the impact of the OSP on student academic achievement and other measures of 

student success, overall and for subgroups of students identified in the statute as high 

priority? 

 What effect does the program have on student and parent perceptions of school safety 

and satisfaction, and on parents’ involvement in education of their children? 

 Does the program change students’ instructional environments and opportunities? 

3. Design: The evaluation will primarily compare outcomes of approximately 1,800 student 

applicants randomly assigned by lottery to either receive a scholarship or not receive a 

scholarship. Lotteries of program applicants were conducted in spring 2012 (cohort 1), 

spring 2013 (cohort 2), and spring 2014 (cohort 3). Data will be collected for three follow-up 

years for each of the cohorts and, for students in both the scholarship and nonscholarship 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 157 

groups. The contractor will administer academic assessments, and conduct student, parent, 

and principal surveys each spring (spring 2013–16). Prior to the first impact report, 

descriptive reports based on application forms parents filled out when they applied to the 

OSP, principal surveys, and extant data were released. 

4. Estimated or Actual Completion Date: The first impact report is expected in spring 2017. The 

second impact report is expected by early 2018. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/choice_soar.asp

Parent Information and School Choice Evaluation 

1. Study Purpose: School choice is among the most visible K-12 education policy trends to 

emerge over the last two decades. For school choice to be effective, it seems critical that 

parents are able to navigate school choice systems and process large amounts of complex 

information about schools and application procedures to make informed choices. However, 

few studies have rigorously examined school choice information strategies or attempted to 

identify effective methods of information presentation. This evaluation is designed to 

address this significant gap in the literature. 

2. Key Questions Addressed:

 What is the optimal way to present school choice information? (E.g., which presentation 

formats make school performance/achievement data easiest to understand?) 

 What is the right amount of information to provide to parents whom might be making a 

school choice? (E.g., is there a limit to the number of choice options and amount of 

information that parents can reasonably consider?) 

 How is school choice information best organized? (E.g., does the order in which 

information is presented shape how it is understood and used? Is there a way to make 

the information parents most often report caring about salient?) 

3. Design: The evaluation findings will be based on low cost quick turn-around “lab” based 

experiments. These studies will take advantage of opportunistic samples of low income 

parents in order to test a large number of potentially promising information strategies and to 

obtain findings within a relatively short period of time. 

4. Estimated or Actual Completion Date: A short user-friendly guide, based on the evaluation’s 

findings, that will help school districts and other providers of school choice information 

design or refine their own parent information materials is expected in October 2017. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/choice_parent.asp

Students with Disabilities 

Impact Evaluation of Training in Multi-Tiered Systems of Support for Behavior (MTSS-B)

1. Study Purpose: Training school staff in supporting student behavior is becoming 

increasingly attractive to districts and schools as a vehicle for school improvement. 

Implementation of MTSS-B is an approach to improving school and classroom climate as 

well as student outcomes. MTSS-B is a multi-tiered, systematic framework for teaching and 

reinforcing behavior for all students as well as for providing additional support to those who 

need it. Over a third of U.S. districts report implementing MTSS-B at the elementary school 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 158 

level. Recent studies have shown the promise of MTSS-B, and a large-scale study of the 

effectiveness of MTSS-B is needed. 

2. Key Questions Addressed:

 What MTSS-B training and support activities were conducted? What MTSS-B activities 

occurred in the schools receiving MTSS-B Training? How do these MTSS-B activities 

differ from those in schools that do not receive the training? 

 What is the impact on school staff practices, school climate and student outcomes of 

providing training in the MTSS-B framework plus universal (Tier I) positive behavior 

supports and a targeted (Tier II) intervention? 

 What are the impacts for relevant subgroups (e.g., at-risk students)? 

3. Design: This is a randomized trial of the impact of training in MTSS-B on school climate, 

school staff practice, and student outcomes. The contractor, with assistance and input from 

the Department and in consultation with a panel of experts, selected a MTSS-B training 

provider, the Center for Social Behavior Support (CSBS), which is a collaboration between 

the Illinois-Midwest PBIS Network at the School Association for Special Education in 

DuPage, Illinois (SASED) and the PBIS Regional Training and Technical Assistance Center 

at Sheppard Pratt, in Maryland. Approximately 90 elementary schools will be randomly 

assigned to either training in MTSS-B including universal supports (Tier I) plus targeted 

interventions for at-risk students (Tier II) or a business-as-usual control group. Treatment 

schools will receive training in MTSS-B prior to and across two school-years, 2015–2016 

(Tier I) and 2016–2017 (Tiers I and II), and implement MTSS-B across these two years. 

Data collection will include staff surveys, teacher ratings of student behavior, classroom 

observations, site visits, and student records data. 

4. Estimated or Actual Completion Date: The report is expected in 2018. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/disabilities_MTSSB.asp

National Evaluation of the IDEA Technical Assistance and Dissemination Program: Final 

Report 

1. Study Purpose: IDEA, which was most recently reauthorized in 2004, provides funds to 

assist states and local educational agencies in making available a free, appropriate public 

education for children with disabilities. Funded at $12.6 billion in FY 2010, IDEA supports 

early intervention services for infants and toddlers, special education and related services 

for children ages 3 through 21, and early intervening services for students not in special 

education but in need of academic or behavioral support. IES is conducting studies under 

Section 664 of IDEA to assess the implementation and effectiveness of key programs and 

services supported under the law. 

As specified in IDEA Part D, the Technical Assistance and Dissemination (TA&D) program is 

to provide technical assistance, support model demonstration projects, disseminate useful 

information, and implement activities that are supported by scientifically based research to 

meet the needs of children with disabilities. The National Evaluation of the IDEA TA&D 

program is designed to describe the products and services provided by the TA&D program 

grantees, state and local needs for technical assistance, and the role that the TA&D 

program plays in meeting these needs and supporting implementation of IDEA. The State 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 159 

Deaf-Blind Projects are part of the TA&D Program and are the focus of the evaluation’s final 

report. 

2. Key Questions Addressed:

 What technical assistance and dissemination activities do State Deaf-Blind Projects 

provide and how does this vary across the states? 

 How do State Deaf-Blind Projects collaborate with other organizations in their state, with 

other technical assistance providers, and across the network of State Deaf-Blind 

Projects? 

 What are the needs for technical assistance among direct service providers who work 

with children and youth with deaf-blindness? 

 How satisfied are direct service providers with services received from the State DeafBlind Projects? 

3. Design: Data collection for the interim report included administering surveys to the national 

and regional level TA&D program grantees, all state IDEA Part B and Part C administrators, 

and a sample of state-level special education program staff. Data were collected between 

November 2012 and March 2013. State-level administrators and staff reported on their 

receipt of technical assistance from TA&D program grantees during the 2010–11 school 

year. Grantees reported on their goals and activities from the beginning of their current grant 

through the interview date. The funding period for the centers included in the interim report 

varied, with the earliest end date in 2012 and the latest in 2014. Data collection for the final 

report included administering surveys to the 48 state level deaf-blind projects awarded 

grants in 2013 and to a sample of local level special and general education providers 

currently working with deaf-blind children. 

4. Estimated or Actual Completion Date: The final report is expected in summer 2017. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/disabilities_idea2004.asp

Preparing for Life After High School 

1. Study Purpose: Phase I of the National Longitudinal Transition Study 2012 (NLTS 2012), 

also referred to as the Study of Transition Outcomes for Youth with Disabilities Phase I, is 

the third in a series examining the characteristics and school experiences of a nationally 

representative sample of youth with disabilities. NLTS 2012 focuses on youth ages 13 to 21 

(in December 2011), but also includes a small sample of students without disabilities to 

enable direct comparisons of students with and without individualized education programs 

(IEPs). It is part of the congressionally mandated National Assessment of the IDEA and is 

supported with funds authorized under Section 664 of IDEA. 

2. Key Questions Addressed:

 What are the personal, family, and school characteristics of youth with disabilities in 

public schools across the country? 

 What regular education, special education, transition planning, and other relevant 

services and accommodations do youth with disabilities receive? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 160 

 How do the services and accommodations differ from those of youth not served under 

IDEA, including those identified for services under Section 504 of the Rehabilitation Act? 

 How do the services and accommodations for youth with disabilities vary with the 

characteristics of youth? 

 How much have the services and accommodations of youth with disabilities changed 

over time? 

3. Design: Phase I of the study collected survey data (spring 2012–summer 2013) on a 

nationwide sample of 12,000 youth in school, of which 10,000 are students with IEPs across 

the federal disability categories. Surveys of both youth and their parents/guardians were 

administered. 

4. Estimated or Actual Completion Date: A report assessing existing research on the 

effectiveness of transition strategies for youth with disabilities was released August 2013. 

Two volumes of a report describing the survey results were released in March 2017. The 

third and final volume of this report is expected later in 2017. 

5. Key Findings from the first two volumes of the March 2017 report include: 

 Youth with an IEP, particularly those with intellectual disability and emotional 

disturbance, are more likely than their peers to be socioeconomically disadvantaged. 

Youth with an IEP are 12 percentage points more likely to live in low-income households 

and are less likely to have parents who are employed or have a college education. 

Among disability groups, youth with intellectual disability and youth with emotional 

disturbance are more socioeconomically disadvantaged and more likely to attend a 

lower-performing school than youth with an IEP overall. In contrast, youth with autism 

and youth with speech or language impairments are less socioeconomically 

disadvantaged and less likely to attend a lower-performing school than youth with an IEP 

overall. 

 The vast majority of youth with and without an IEP feel positive about school, but those 

with an IEP experience bullying and are suspended at higher rates. Like their peers, 

more than 80 percent of youth in special education report that they are happy with 

school and with school staff. However, not only do youth with an IEP more commonly 

experience some types of bullying (e.g., being teased or called names) but, according to 

parent reports, they are more than twice as likely to be suspended or expelled from 

school. Among the disability groups, youth with emotional disturbance are most likely to 

report being teased and are suspended, expelled, and arrested at more than twice the 

rates of youth with an IEP on average. 

 Youth with an IEP are more likely than other youth to struggle academically, yet less 

likely to receive some forms of school-based support. Half of all youth with an IEP report 

they have trouble with their classes, about 15 percentage points more than reported by 

their peers. However, they are less likely to report receiving school-based academic help 

before or after regular hours, although their parents more commonly help with homework 

and attend a parent-teacher conference. Among youth with an IEP, those with autism, 

intellectual disability, and multiple disabilities are least likely to receive school-provided 

instruction outside of school hours though most likely to receive modified tests and 

assignments. 

 Youth with an IEP lag their peers in planning and taking steps to obtain postsecondary 

education and jobs. Substantially fewer youth with an IEP expect to enroll in 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 161 

postsecondary education or training, compared to youth without an IEP. Reflecting these 

gaps, youth in special education are almost half as likely as their peers to report taking 

college entrance and placement tests. Forty percent report having recent paid work 

experience while in high school, compared with 50 percent of youth without an IEP. 

Among youth with an IEP, the three groups least likely to receive academic supports 

before or after school—youth with autism, intellectual disability, and multiple 

disabilities—are also least likely to take these steps to prepare for college and 

employment. 

 Youth with autism, deaf-blindness, intellectual disability, multiple disabilities, and 

orthopedic impairments are most at-risk for not transitioning successfully beyond high 

school. Youth in these groups are less likely than all youth with an IEP to have key 

characteristics and experiences linked to success after high school, such as performing 

typical daily living tasks, engaging with friends and in school activities, or preparing for 

college, careers, and independent living. 

6. Link to Additional Information: 

https://ies.ed.gov/ncee/projects/evaluation/disabilities_nlts2012.asp 

Preschool Special Education Programs and Practices Supporting Children with 

Disabilities 

1. Study Purpose: Most recently authorized in 2004, IDEA provides funding to states to support 

special education and related services for children and youth with disabilities, including 

young children ages 3–5 years old. Currently there is limited information available on the 

curricula and interventions being used across states to support young children with 

disabilities. Phase I of the Evaluation of Preschool Special Education Practices is being 

conducted to address the primary objective of assessing the feasibility of conducting a largescale impact study of curricula or interventions promoting the literacy, language, and/or 

social-emotional skills of preschool-age children with disabilities. A secondary objective of 

the Phase I study is to provide nationally representative descriptive information about 

preschool special education programs and the specific curricula or interventions being 

delivered to preschool children with disabilities. 

2. Key Questions Addressed: 

 Which curricula and interventions are used nationally for preschool children with 

disabilities to promote learning of language, literacy, and social emotional skills? 

 How are states and school districts structuring programs to serve children ages 3 

through 5 with disabilities? 

 What staff resources are available to support the instruction of children ages 3 through 5 

with disabilities? 

3. Design: The study collected survey data from state and district grantees of IDEA funds to 

obtain nationally representative information on the programs, services, curricula and 

interventions available to children ages 3–5 identified for special education services. The 

survey sample included a nationally representative sample of 1,200 school district preschool 

special education coordinators and state Section 619 coordinators in all 50 states and DC. 

Additionally, an evidence review will be conducted to identify preschool curricula and 

interventions focused on improving literacy, language, and social emotional skills of 

preschool-aged children with disabilities for the design of impact study options. If an impact 

study is feasible to conduct, preparations for the impact study will occur under Phase I. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 162 

4. Estimated or Actual Completion Date: A descriptive report on the survey findings is 

expected in 2018. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/disabilities_preschool.asp 

Study of Early Intervention and Special Education Services and Personnel 

1. Study Purpose: IDEA was most recently authorized in 2004 to promote a free appropriate 

public education for children with disabilities. Funded at $12.5 billion in FY 2015, IDEA 

supports early intervention services for infants and toddlers (IDEA Part C), special education 

and related services for children ages 3 through 21 (IDEA Part B), and early intervening 

services for students not in special education but in need of academic or behavioral support. 

Under Section 664 of IDEA 2004, IES is conducting studies to assess the implementation 

and effectiveness of key programs and services supported under the law. This study is 

supporting the analysis of extant data to examine early intervention and special education 

service delivery and the personnel providing services. 

2. Key Questions Addressed:

 How has the delivery of early intervention services changed over time, and how do these 

services differ for subgroups defined by age and state? 

 How have special education services changed over time, and how do these services 

differ for subgroups defined by age, disability category, and state? 

 How has the distribution of personnel providing special education services changed over 

time? 

3. Design: This study includes new descriptive analysis of extant data available in public use or 

restricted formats. Among the data sources that are being used are cross-sectional Section 

618 data submitted by states to the Department and the NCES Schools and Staffing Survey. 

4. Estimated or Actual Completion Date: The report is expected in summer 2017. 

5. Link to Additional Information: 

http://ies.ed.gov/ncee/projects/evaluation/disabilities_persserv.asp 

Teacher Effectiveness 

Does Content-Focused Teacher Professional Development Work? Findings from Three 

Institute of Education Sciences Studies 

1. Study Purpose: Federal and local governments continue to invest billions of dollars each 

year in professional development for teachers. Until recently, there has been little rigorous 

evidence to inform the design and delivery of these professional development programs. 

Nevertheless, there has been growing consensus that deepening teachers’ content 

knowledge is an essential component of effective professional development in both reading 

and mathematics. Over the past decade, IES conducted three large-scale random 

assignment studies of teacher professional development in different grades in reading and 

math. These studies, which are the focus of this brief, reveal a common pattern of findings 

on the impact of intensive, content-focused professional development on teaching and 

learning. The findings also highlight unresolved issues that future research might explore to 

advance our understanding and inform professional development policy and practice. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 163 

2. Key Questions Addressed: This brief synthesizes findings from three large-scale random 

assignment studies of professional development that were conducted by the National Center 

for Education Evaluation and Regional Assistance in IES. Each study examined the impact 

of teacher professional development on teacher knowledge, practice, and student 

achievement. 

3. Design: Although the professional development programs in each study were different, they 

all emphasized building teachers’ content knowledge or knowledge about content-specific 

pedagogy. The programs combined summer institutes with periodic teacher meetings and 

coaching during the school year. These programs were compared to the substantially less 

intensive professional development that teachers typically received in study districts. The 

three studies included 270 second-grade reading teachers, 165 fourth-grade math teachers, 

and 195 seventh-grade math teachers. 

4. Estimated or Actual Completion Date: The brief was released in November 2016. 

5. Key Findings: 

 The professional development improved teachers’ knowledge and some aspects of their 

practice; 

 Improving teachers’ knowledge and practice did not lead to positive impacts on student 

achievement; 

 Most of the measured aspects of teachers’ knowledge and practice were not correlated 

with student achievement; and 

 The consistent pattern of findings suggests that future studies might seek to better 

understand on what aspects of teacher knowledge and practice professional 

development should focus, and how professional development can achieve a larger 

impact on knowledge and practice that also impacts student achievement. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20174010/

Impact Evaluation of Professional Development for Teachers on Data-Driven Instruction 

1. Study Purpose: Title II, Part A of ESEA funds a broad array of activities to improve teacher 

quality, including professional development. One possible focus of professional 

development is how to use ongoing data from student performance and assessments. 

These data provide information about students’ academic needs and are already available 

to principals and teachers but may not be used effectively. This study evaluates the 

effectiveness of professional development for teachers to use such data to guide instruction. 

Existing evidence on data-driven instruction strategies is limited; however, it suggests that 

professional development that helps teachers focus on individual students’ academic needs 

is promising. 

2. Key Questions Addressed:

 What are the effects on student achievement of providing professional development to 

principals and teachers that focuses on using individual student academic information 

already available to school staff to guide instruction? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 164 

 What are the effects of providing this professional development on teachers’ and 

principals’ use of individual student academic information to inform instructional support, 

planning, and practice? 

3. Design: The study is a randomized controlled trial with a sample of 104 schools in 

12 districts. In each district, schools were randomly assigned to treatment (professional 

development) or to control (business as usual) status. Data collection will consist of a 

teacher survey, a principal survey, and student administrative records, including student 

state standardized achievement test scores. 

4. Estimated or Actual Completion Date: The report is expected in summer 2018. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_datadriven.asp 

Impact Evaluation of Teacher and Leader Evaluation Systems 

1. Study Purpose: The study is designed to examine the implementation of a package of 

performance evaluation system components and the impact of their use for formative 

purposes. These are components that states and districts might elect to include in their 

evaluation systems, with support from Title II Part A funds under the ESSA. The 

components include measures of student achievement growth, classroom practices, and 

principal leadership. 

2. Key Questions Addressed:

 What is the impact of the performance evaluation system on student achievement? 

 What is the impact on teachers’ classroom practices? 

 What are districts’ and educators’ experiences with implementation? 

3. Design: Within each of eight districts, approximately 15 elementary and middle schools were 

randomly assigned to receive the study’s measures of student achievement growth, 

classroom practices, and principal leadership during 2012–13 and 2013–14 (treatment 

group) or to participate only in the district’s usual performance evaluation system during the 

same time period. In treatment schools, each year, teachers received four rounds of 

classroom observations and feedback sessions and information about their students’

achievement growth in math and/or reading using a statistical model known as value-added. 

Principals in treatment schools received feedback on their leadership practices twice per 

year. Study data collection included surveys of teachers and principals, observations of 

teachers’ classroom practices, and collection of student records data. The study also looks 

at the extent to which the measures were implemented as intended and whether the ratings 

from the measures reliably distinguish educator performance. 

4. Estimated or Actual Completion Date: A report on first year implementation findings was 

released in November 2016. A final report on impacts on educator practices and student 

achievement as well as implementation during the study’s second year is expected in fall 

2017. 

5. Key Findings: The main findings from the study’s first report are: 

 The study’s performance measures were implemented generally as planned. Teachers 

and principals received multiple rounds of ratings and feedback on their practices. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 165 

However, fewer principals and teachers accessed their student growth reports than the 

study intended. 

 Both classroom observation and student growth measures differentiated teacher 

performance, although observation scores were mostly at the upper end of the scale. 

Overall, observation scores varied across teachers, and both value-added scores and 

average classroom observation scores over the year had sufficient reliability to capture 

performance differences among some teachers. 

 The principal leadership measure differentiated performance, but principal self-ratings, 

teachers’ ratings of the principal, and the principal’s supervisor’s ratings of the principal 

often differed. 

 Both teachers and principals in treatment schools reported receiving more feedback on 

their performance than did their counterparts. For example, teachers and principals in 

treatment schools reported spending more total time in performance feedback sessions 

across the year than teachers and principals in the control schools. 

6. Links to Additional Information:

https://ies.ed.gov/ncee/projects/evaluation/tq_performance.asp

Impact Evaluation of the Teacher Incentive Fund 

1. Study Purpose: The TIF was renamed the Teacher and School Leader Incentive Program 

with reauthorization of the ESEA in the ESSA. The purpose of the TIF program is to develop 

and implement performance-based compensation systems (PBCSs) for teachers, principals, 

and other personnel in high-need schools. Research indicates that high-quality teachers are 

critical to raising student achievement in low-performing schools, but schools most in need 

often have difficulty in attracting and retaining high-quality teachers. Performance pay is a 

policy promoted by the TIF program to improve the quality of teachers in high-need schools. 

This evaluation studies performance pay that provides substantial and differentiated bonus 

pay to high-performing teachers in low-performing schools with high-need students. 

2. Key Questions Addressed:

 What are the characteristics of all TIF grantee districts and their performance-based 

compensation systems? What implementation experiences and challenges did TIF 

districts encounter? 

 How do teachers and principals in schools that did or did not offer pay-for-performance 

bonuses compare on key dimensions, including their understanding of TIF program 

features, exposure to TIF activities, allocation of time, and attitudes toward teaching and 

the TIF program? 

 What is the impact of pay-for-performance bonuses on students’ achievement on state 

assessments of math and reading? 

 How do pay-for-performance bonuses affect educator mobility, including whether 

mobility differs by educator effectiveness? 

3. Design: Study schools were randomly assigned within a grant to either implement all 

components of the PBCS or the PBCS with a 1 percent across-the-board bonus in place of 

the differentiated effectiveness incentive component of the PBCS. Data collection will 

include a grantee survey, a survey of teachers and principals, teacher and principal school 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 166 

assignment records, student record information (such as student demographics and student 

test scores), and grantee interviews to document implementation information, as well as to 

conduct impact analyses. 

4. Estimated or Actual Completion Date: The first report was released in September 2014. The 

second report was released in September 2015. The third report was released in August 

2016. The fourth and final report, which will include updated impacts, is expected in fall 

2017.

5. Key Findings from the Third Report: The main findings among all TIF districts with 2010 

awards are: 

 Similar to the previous two years, most districts (88 percent) implemented at least three 

of the four required program components for teachers. 

 By the third year, reported implementation challenges decreased with no more than 

one-fifth of TIF districts reporting any major challenges. 

 For the subset of 10 districts that agreed to participate in a random assignment study, 

key findings on the effect of pay-for-performance on educators include the following: 

 After three years of TIF implementation, average student achievement remained 1 to 

2 percentile points higher in schools that offered pay-for-performance bonuses than in 

schools that did not. This difference was equivalent to a gain of about four additional 

weeks of learning. 

 At least half of the evaluation districts each year met the grant guidance for awarding 

differentiated performance bonuses for teachers. However, in each year, no more than 

2 of the 10 districts awarded bonuses for teachers that were substantial or challenging to 

earn. 

 Teachers’ understanding of performance measures continued to improve, but only about 

60 percent of teachers correctly reported being eligible for a performance bonus. In 

addition, teachers believed that the maximum bonus they could earn was no more than 

two-fifths the size of the actual maximum bonus that districts awarded, a finding similar 

to previous years. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_incentive.asp

Impact Evaluation of Support for Principals 

1. Study Purpose: Title II, Part A, the Improving Teacher State Formula Grants program, is the 

primary federal funding under ESEA to support high-quality educators. The program targets 

high-poverty districts and funds a broad array of allowable activities for principals and 

teachers, such as support for certification, teacher mentoring and induction, intensive 

professional development, recruitment, retention, and merit-based teacher and principal pay 

strategies as well as class size reduction. Principals, through a collective focus on 

instructional and organizational leadership and human capital management, have the 

potential to greatly influence the quality of instruction. However, there is limited evidence 

about the effectiveness of principal professional development programs and their ability to 

improve principals’ leadership skills and school quality. This evaluation studies professional 

development for principals and thus provides an important source of information for this 

program. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 167 

2. Key Questions Addressed:

 What are the professional development experiences of principals? 

 What are the initial impacts on school climate and educator behaviors of providing 

principals structured and intensive professional development? 

 What are the impacts on teacher retention, the effectiveness of instructional staff, and 

student achievement of providing principals with structured and intensive professional 

development? 

3. Design: The study team will randomly assign within 10 districts a total of 100 elementary 

schools to a treatment or control group. Treatment group principals will be offered intensive 

professional development provided by the University of Washington’s Center for Educational 

Leadership during the 2015–16 and 2016–17 school years. The Center for Educational 

Leadership was competitively selected to provide the professional development focused on 

in this study. The professional development includes a heavy emphasis on instructional 

leadership activities, including support in conducting school walkthroughs and classroom 

observations with constructive feedback to facilitate teacher growth focused on improving 

student achievement. Control group principals will receive supports normally offered by the 

district. Data collection will include: information about the professional development 

delivered and experienced by the participating principals; teacher and principal surveys and 

periodic logs of principal daily activities to document intermediate outcomes, such as 

principal behaviors and school climate; and administrative records to document student 

outcomes (e.g., achievement, behavior, attendance) and teacher outcomes (e.g., retention 

of effective teachers, quality of newly hired teachers). 

4. Estimated or Actual Completion Date: The first report, which will focus on implementation 

and intermediate outcomes, is expected in spring 2018. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_principals.asp

Study of Teacher Preparation Experiences and Early Teacher Effectiveness 

1. Study Purpose: A primary focus of the ESSA’s Title II Part A is on the improvement of 

teacher quality. Little research exists, however, to inform how best to prepare teachers for 

the classroom. This study provides descriptive information on the preparation experiences of 

a large sample of novice teachers. It also will examine whether the instructional skills that 

teachers learn about and have opportunities to practice in their preparation programs are 

associated with teachers’ effectiveness once they are in the classroom. 

2. Key Questions Addressed:

 To what extent do teachers report participating in various preparation experiences? How 

much variation is there within and between programs? 

 Among novice teachers, what are the relationships between the teachers’ experience in 

their preparation programs and the achievement of students in their classroom? 

3. Design: Approximately 3,200 novice language arts and/or math teachers from grades 4 

through 6 participated in the study. In spring 2015, the teachers responded to a survey 

focused on their preparation experiences related to instructional skills that have been shown 

to be associated with teacher value-added scores. For each instructional skill, the teachers 

were asked about the nature of their learning experiences, including how they learned (e.g., 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 168 

through coursework or through practice in K-12 classroom) and the extent to which they 

found the experience useful. The study will also compute value-added scores for teachers, 

based on students’ state math and English language arts tests, and examine the 

relationships between teacher preparation experiences and teacher value-added scores. 

4. Estimated or Actual Completion Date: A report describing teachers’ preparation experiences 

and the relationship between experiences and teacher value-added scores is expected by 

fall 2017. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/tq_teacherprep_early.asp. 

Study of the Distribution of Effective Teaching 

1. Study Purpose: There is a persistent achievement gap in the United States where students 

from high-income families outperform those from low-income families on achievement tests. 

There is also substantial variation in the effectiveness of teachers. A key question for 

policymakers is whether policy initiatives focused on providing low-income students with 

equal access to effective teachers can address the achievement gap. This study provided 

information about the extent to which disadvantaged students received less-effective 

teaching than other students. The study also examined teacher mobility in participating 

districts and how patterns of mobility might contribute to unequal access. 

2. Key Questions Addressed:

 Are low-income students taught by less effective teachers than high-income students? If 

so, to what extent would providing equal access to effective teachers reduce the student 

achievement gap? 

 Are there differences between high- and low-poverty schools in teacher hiring, transfer, 

and attrition? If so, are they consistent with inequitable access to effective teachers for 

low-income students? 

3. Design: The study documented low-income students’ access to effective teachers, as 

measured by value added across the 2008–09 through 2012–13 school years. The study 

also described district polices designed to address inequitable distribution of effective 

teaching implemented during those years. Lastly, the study examined teacher mobility 

patterns within participating districts. Annual data collection included district administrative 

records, such as student achievement to conduct value added analyses, as well as annual 

semi-structured interviews with district leadership to provide information on district policies. 

District personnel data were also collected to examine teacher mobility within participating 

districts. The study was conducted in 29 geographically dispersed school districts. 

4. Estimated or Actual Completion Date: The final report on school years 2008–09 through 

2012–13 was released in October 2016. 

5. Key Findings from the Final Report: The final report focused on low-income students’ access 

to effective teachers and teacher mobility patterns in 26 districts and found that: 

 There are small differences in the effectiveness of teachers of high- and low-income 

students, on average. The average teacher of a low-income student is just below the 

50th percentile of effectiveness based on value-added, while the average teacher of a 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 169 

high-income student is at the 51st percentile. Providing low-income students with equally 

effective teachers would not substantively reduce the achievement gap. 

 In a subset of the study districts, there is meaningful inequity in teacher effectiveness in 

math. In 3 of the 26 study districts, providing low-income students with teachers whose 

effectiveness is equal to that of high-income students over a five-year period would 

reduce the math achievement gap by at least a tenth of a standard deviation of student 

achievement, the equivalent of about 4 percentile points. 

 Teacher hiring patterns are consistent with small inequities in access to effective 

teachers. High-poverty schools have more newly hired teachers than low-poverty 

schools, but this difference is likely to have a small influence on equity because 

(1) relatively few teachers are new hires (11 percent of teachers in high-poverty schools 

and 5 percent in low-poverty schools), and (2) performance of newly hired teachers 

improves quickly. On average, newly hired teachers become as effective as the average 

teacher after one year. 

 Teacher transfer patterns are also consistent with small inequities in access to effective 

teachers. Teachers who transfer to schools in a higher poverty category are less 

effective (43rd percentile) than the average district teacher. Teachers who transfer to 

schools in a lower poverty category are nearly as effective (48th percentile) as the 

average district teacher. These patterns likely have a small influence on equity since just 

under 4 percent of all teachers transfer across poverty categories each year. 

 Teacher attrition patterns do not contribute to inequity. Teachers who leave a district are 

less effective (44th percentile) than the average teacher, and more teachers leave highpoverty schools than low-poverty schools (10 percent versus 7 percent, respectively). 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_distribution.asp

Other 

Evaluation of Investing in Innovation 

1. Study Purpose: Investing in Innovation (i3) is a competitive grant program that funds entities 

with a record of improving student achievement in order to expand the implementation of, 

and investment in, evidence-based interventions to significantly improve student education 

outcomes. The i3 program supports three types of grants that are differentiated by level of 

prior evidence supporting the intervention, the scope of the proposed implementation, and 

the amount of funding to be provided. Scale-up and Validation grants require prior evidence 

of effectiveness, while Development grants support innovative strategies. 

Grantees receiving funds under this program are required to conduct an independent 

evaluation of their project and must agree, along with their independent evaluator, to 

cooperate with evaluation technical assistance provided by the Department and its 

contractor. The purpose of this technical assistance is to maximize the strength of the 

impact studies and the quality of their implementation data and performance feedback. 

2. Key Questions Addressed:

 What types of supports were provided to i3 local evaluators and how were these 

supports implemented? 

 Were the i3-funded evaluations well-designed and well-implemented? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 170 

 What did the i3-funded evaluations add to the evidence base on educational 

interventions? 

3. Design: This study is descriptive and includes technical assistance to support the i3 local 

evaluators. The study also includes a systematic review of the completed evaluations 

against the WWC evidence standards as well as other criteria for assessing the strength of 

the impact and implementation aspects of the evaluations. 

4. Estimated or Actual Completion Date: The first report, which will focus primarily on findings 

for the first cohort (FY 2010), is expected in fall 2017. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/assistance_ita.asp 

Evaluation of NAEP Achievement Levels 

1. Study Purpose: Under the provisions of P.L. 107-279, the Secretary is required to provide 

for continuing review of the National Assessment of Educational Progress (NAEP) 

assessment. The legislation identifies the issues to be addressed in the reviews, one of 

which includes the requirement to evaluate whether the NAEP achievement levels, 

established by the National Assessment Governing Board, are “reasonable, valid, reliable 

and informative to the public.” Section 303(e)(2)(C) of the Education Sciences Reform Act of 

2002 (P.L. 107-279) states that NAEP achievement levels shall be used on a trial basis until 

the Commissioner of the National Center for Education Statistics (NCES) determines, as a 

result of an evaluation under Sec. 303(f), that such levels are “reasonable, valid, and 

informative to the public.” This independent evaluation was intended to provide IES and the 

Commissioner of NCES with information necessary to inform the decision about whether the 

current trial status of the NAEP achievement levels can be removed or whether they should 

remain in trial status. 

2. Key Objectives:

 Determine how “reasonable, valid, reliable and informative to the public” will be 

operationalized in this study. 

 Identify the kinds of objective data and research findings that will be examined. 

 Review and analyze extant information related to the study’s purpose. 

 Gather other objective information from relevant experts and stakeholders, without 

creating burden for the public through new, large-scale data collection. 

 Organize, summarize, and present the findings from the evaluation in a written report, 

including a summary that is accessible for nontechnical audiences, discussing the 

strengths/weaknesses and gaps in knowledge in relation to the evaluation criteria. 

 Provide, prior to release of the study report, for an independent external review of that 

report for comprehensiveness, objectivity, and freedom from bias. 

3. Design: This study focused on the achievement levels used in reporting NAEP results for 

the reading and mathematics assessments in grades 4, 8, and 12. Specifically, the study 

reviewed developments over the past decade in the ways achievement levels for NAEP are 

set and used and will evaluate whether the resulting achievement levels are “reasonable, 

valid, reliable, and informative to the public.” The study relied on an independent committee 

of experts with a broad range of expertise related to assessment, statistics, social science, 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 171 

and education policy. The project is receiving oversight from the Board on Testing and 

Assessment and the Committee on National Statistics of the National Research Council. 

4. Actual Completion Date: The prepublication version of the report from this study was 

released in November 2016 and may be downloaded at 

https://www.nap.edu/catalog/23409/evaluation-of-the-achievement-levels-for-mathematicsand-reading-on-the-national-assessment-of-educational-progress.

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_naep.asp

Evaluation of the Comprehensive Technical Assistance Centers, FY 2012 Grantees 

1. Study Purpose: The Comprehensive Technical Assistance Centers is a federally funded 

program currently authorized under the Educational Technical Assistance Act of 2002. The 

Department awarded five-year grants in FY 2012 to 22 Comprehensive Technical 

Assistance Centers with the purpose to help state education agencies build their capacity to 

implement state-level initiatives and to support district- and school-level initiatives that 

improve educational outcomes for all students, close achievement gaps, and improve the 

quality of instruction. The FY 2014 appropriation for the Centers was $48.4 million. This 

study will inform the Department of Education, the Comprehensive Center program, and the 

larger field about the design, implementation, and outcomes of the Centers’ work. 

2. Key Questions Addressed:

 Program Design: How did the Centers design their work? How did Centers define 

capacity building? Did their definitions change over time? If so, how? What theories of 

action did Centers use to guide their general capacity-building work? Did the theories 

change over time? If so, how? How did Centers assess the needs of their 

constituencies? 

 Program Implementation: How did the Centers operate? What strategies did Centers 

employ to achieve their outcomes? To what extent did Centers implement technical 

assistance to their constituencies as planned? To what extent and how did Centers 

collaborate with each other? 

 Program Outcomes: What was the result of the Centers’ work? To what extent did 

Centers achieve their goals and objectives? 

3. Design: This evaluation is a multiyear descriptive study examining the Centers’ programs. 

Data on the Centers’ activities and outcomes will be collected during the FY 2015, FY 2016, 

and FY 2017 program years. The evaluation will describe how the individual Centers intend 

to build SEA capacity (their theories of action) and document what types of activities they 

actually conduct to build capacity. The evaluation plans to focus on Center projects in two 

priority areas: great teachers and leaders and early learning. Data collection will include: 

(1) the Centers’ management plans and technical assistance activity data; (2) interviews 

with staff from each Center; (3) interviews with technical assistance recipients; (4) a survey 

of Center staff; and (5) a survey of technical assistance recipients. Additionally, observations 

of technical assistance events will be conducted to inform project profiles that illustrate the 

strategies that the Centers use to support capacity building and achieve planned outcomes. 

This approach will yield a diverse set of data that can be analyzed and summarized using 

qualitative research methods and simple quantitative tabulations. 

4. Estimated or Actual Completion Date: A report describing findings is expected in 2018. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 172 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/other_techcenters12.asp

Implementation and Impact Evaluation of Race to the Top: Final Report 

1. Study Purpose: RTT is a competitive grant program of the Department that funds states and 

districts planning to implement comprehensive education reform in one or more core areas. 

Since its first awards in 2010, RTT has funded general state competitions, state 

competitions focused on early learning, and district competitions focused on personalized 

learning. With funds from the Recovery Act, the general state competition awarded 

approximately $4 billion to states in support of comprehensive K–12 education reform in 

several core areas, including teachers and leaders, standards and assessments, data 

systems, and school turnaround. This study examined the implementation of RTT and its 

relationship to student outcomes, focusing on the initial general state competition for RTT 

that began in 2010. 

2. Key Questions Addressed:

 Which policies and practices promoted by the RTT program do RTT states report using, 

and how do they compare to the policies and practices that non-RTT states report 

using? 

 Is receipt of an RTT grant related to improvement in student outcomes? 

3. Design: The RTT sample included all 50 states and DC. Data from interviews with all states 

and DC informed the first evaluation question. The second evaluation question was 

addressed using a short interrupted time series design with state-level NAEP data 

comparing, before and after the RTT competition, states that were awarded an RTT grant to 

states that applied for but were not awarded an RTT grant. 

4. Estimated or Actual Completion Date: The final report for RTT was released in October 

2016.

5. Key Findings from the Final Report for RTT:

 In four of six areas examined, 2010 RTT grantees reported using more policies and 

practices promoted by RTT than states that did not receive a grant: The four areas with 

differences were standards and assessments, teachers and leaders, school turnaround, 

and charter schools. The other two areas were state capacity and data systems. 

 2011 RTT grantees reported using more policies and practices promoted by RTT than 

states that did not receive a grant in one area, which was teachers and leaders. 

 Across all states, use of RTT-promoted policies and practices were highest in the data 

systems area and lowest in the teachers and leaders area: States reported using 

76 percent of the 8 RTT-promoted practices examined in data systems, but only 

26 percent of the 39 practices in teachers and leaders. 

 The relationship between RTT and student outcomes was not clear: Trends in student 

outcomes could be interpreted as providing evidence that RTT had a positive effect, a 

negative effect, or no effect. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_racetotop.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 173 

Implementation and Impact Evaluation of SIGs: Final Report 

1. Study Purpose: The SIG program is authorized through Title I of the ESEA and provides 

three-year awards to support turnaround in the nation’s persistently lowest-achieving 

schools. In FY 2009, the $546 million SIG appropriation was supplemented by $3 billion 

through the Recovery Act, for a total of $3.5 billion. SIG funds are distributed to states by 

formula based on Title I allocations. States then competitively award funds to districts 

applying on behalf of their eligible schools. Schools receiving SIG must implement one of 

four prescriptive intervention models: turnaround, transformation, closure, or restart. This 

study examined the implementation and impacts of SIG, focusing on the first cohort of SIG 

schools implementing intervention models beginning in the 2010–11 school year. 

2. Key Questions Addressed:

 Are SIG-funded schools using the improvement or turnaround strategies promoted by 

the four SIG intervention models, and how do they compare to strategies in schools not 

implementing a SIG-funded intervention model? How are states and districts supporting 

such efforts? 

 Does receipt of SIG funding to implement a school intervention model have an impact on 

outcomes for low-performing schools? 

 Is implementation of the four school intervention models related to improvement in 

outcomes for low-performing schools? 

3. Design: The SIG sample included about 500 schools in 60 districts from 22 states. This 

sample was purposively selected to support a regression discontinuity design to address the 

fourth evaluation question, exploiting cutoff rules that states used to identify their persistently 

lowest-achieving schools as eligible for SIG to implement one of the four intervention 

models. Data from state and district interviews, as well as school surveys from the SIG 

sample, were used to inform the third and fifth evaluation questions. Student- and schoollevel achievement data were also collected from administrative records up to the 2012–13 

school year to inform the fourth and fifth evaluation questions. 

4. Estimated or Actual Completion Date: The final report for SIG was released in January 

2017.

5. Key Findings from the Final Report for SIG:

 SIG schools implementing one of the four models (transformation, turnaround, restart, or 

closure) reported using more practices than other schools: SIG schools reported using 

an average of 23 out of 35 practices, whereas other schools reported using 20 practices. 

 Across all schools, use of SIG-promoted practices was highest in the area of 

comprehensive instructional reform strategies and lowest in the area of operational 

flexibility and support: Schools reported using 89 percent of the eight SIG-promoted 

practices examined in the comprehensive instructional reform strategies area, but only 

43 percent of the two practices in operational flexibility and support (the other two areas 

examined were increasing teacher and principal effectiveness, and increasing learning 

time and creating community-oriented schools). 

 Implementing any of the four SIG models had no significant impacts on math or reading 

test scores, high school graduation, or college enrollment. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 174 

 In elementary grades, student achievement gains did not differ across the four SIG 

models. In secondary grades, the turnaround model was associated with larger 

achievement gains than the transformation model. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_racetotop.asp

Implementation of Title I/II Program Initiatives 

1. Study Purpose: The Title I and Title II programs are part of the ESEA and are intended to 

help provide all students with equal access to education by providing financial assistance to 

schools and districts that have a high percentage of students from low-income families 

(Title I) and improving teacher and principal quality (Title II). The previous reauthorization of 

ESEA, the No Child Left Behind Act of 2001 (NCLB), increased accountability through the 

use of assessments for students, requirements for all teachers to be highly qualified, and 

funding for supports and interventions for schools not achieving adequate yearly progress 

for the entire school population as well as for various subgroups of students. Subsequently, 

a majority of states received ESEA Flexibility beginning in 2012, which allowed particular 

NCLB requirements to be waived in exchange for a commitment to implement various 

reform principles. ESEA was reauthorized as the ESSA in December 2015, and it offers 

states and districts more autonomy than under NCLB and ESEA Flexibility, specifically in the 

areas of accountability, educator evaluation systems, and teacher qualification 

requirements. 

This study is designed to provide relevant data from states, districts, schools, and teachers 

on the implementation of programs and policies related to Title I and Title II under NCLB and 

ESEA Flexibility (during the 2013–14 school year). It will also provide data from states and 

districts as they undergo the early stages of implementing the ESSA (during the 2017–18 

school year). 

2. Key Questions Addressed:

 What content standards and high school graduation requirements are states adopting, 

and what materials and resources do states, districts, and schools provide to help 

teachers implement the state content standards? 

 What assessments do states and districts use (in terms of assessment format and 

coverage of grade levels and content areas), and what materials and resources do 

states, districts, and schools provide to support the implementation of assessments and 

use assessment data? 

 How has student achievement changed over time? 

 What elements are included in states’ accountability systems? How do states and 

districts identify and reward their highest-performing schools, how do they identify and 

support their lowest-performing schools, and how do they offer differentiated support for 

those schools that are neither highest-performing nor lowest-performing? 

 How do states and districts evaluate teacher and principal effectiveness and assess 

equitable distribution of teachers and principals, and what supports do states, districts, 

and schools provide to improve teacher and principal effectiveness? 

3. Design: The study is descriptive and nationally representative. Data were collected from all 

50 states and the District of Columbia, a nationally representative sample of districts and 

schools, and teachers within those schools through surveys in the 2013–14 school year. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 175 

Subsequently, data will be collected from all 50 states and the District of Columbia as well 

as the same nationally representative sample of districts through surveys in the 2017–18 

school year. 

4. Estimated or Actual Completion Date: The first report, based on the findings from the 

2013–14 data collection, was released in January 2017.

5. Key Findings from the First Report:

 Most states adopted and most principals and teachers reported implementing state 

standards that focused on college- and career-readiness. All but one state had 

committed to having college- and career-ready standards in place by 2013–14. A 

majority of principals (67–69 percent, depending on subject) reported fully implementing 

state content standards, and most teachers reported receiving professional development 

relevant to state content standards (79 percent of teachers) and weekly use of aligned 

instructional activities (92 percent of teachers). 

 Many state assessments incorporated more sophisticated response formats to better 

assess students’ college- and career-readiness. In their reading/English language arts 

(ELA) summative assessments, many states (24–36, depending on grade level) reported 

using extended constructed-response formats, a type of response format intended to 

assess higher-order thinking skills. Nineteen states used this response format in math 

assessments. 

 States used ESEA flexibility to reset their accountability goals and to target a narrower 

set of schools for additional support. Forty-three states had received ESEA Flexibility for 

the 2013–14 school year. The most common accountability goal adopted by states with 

ESEA Flexibility (28 of the 43 states) was reducing by half the percentage of students 

and subgroups not proficient in 6–8 years. States with ESEA Flexibility identified 

5 percent of Title I schools as lowest performing and an additional 10 percent of Title I 

schools with substantial student achievement gaps, compared to non-Flexibility states 

that reported identifying 43 percent of Title I schools as lowest performing. 

 Almost all states adopted new laws or regulations related to educator evaluation 

systems between 2009 and 2014, and most districts reported full or partial 

implementation in 2013–14. Only four states had not adopted new teacher evaluation 

laws or regulations by 2014, and a majority (59 percent) of districts reported fully 

implementing, piloting, or partially implementing a new teacher evaluation system. 

However, few districts (18 percent) reported using evaluation system measures of 

student achievement growth and classroom practice consistent with emerging research. 

 Proficiency rates on the National Assessment of Educational Progress (NAEP) slightly 

increased from 2005 to 2015, with the largest increases in 4th and 8th grades and 

smaller or no increases in 12th grade. Overall proficiency rates increased by statistically 

significant levels of 4–5 percentage points in 4th and 8th grade reading and math and by 

2 percentage points in 12th grade reading. Statistically significant increases in 

proficiency were also evident for economically disadvantaged students in both subjects 

and across all three grades (by 4–7 percentage points), and in the large majority of 

individual states (46–51 states, depending on grade and subject). 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_titleI.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 176 

Progress and Challenges in Developing Tiered Quality Rating and Improvement Systems 

(TQRIS) in the Round 1 Race to the Top-Early Learning Challenge (ELC) States 

1. Study Purpose: ELC aims to improve early learning and development programs so children 

enter kindergarten ready to succeed. One of the key areas of reform for the program 

focuses on the design and implementation of Tiered Quality Rating and Improvement 

Systems (TQRIS) that can provide parents and stakeholders with information on the quality 

of early learning programs. Since ELC grantee states are in the process of developing, 

refining, and/or implementing their TQRIS, this evaluation will describe the current state 

plans and implementation status.

2. Key Question Addressed: 

 How are TQRIS ratings defined, collected, and generated in the nine Round 1 ELC 

states and how might the structure and implementation of the rating system influence the 

reliability and validity of the ratings? 

3. Design: This is a descriptive study that involved collecting and analyzing various data from 

the nine Round 1 ELC grantee states. The study collected and conducted a targeted review 

of documents describing the structure of TQRIS, including component measures and the 

quality indicators used to evaluate preschool programs, how these are combined to 

generate overall ratings, the availability of TQRIS and child outcome data, and policies 

regarding the administration of kindergarten entry assessments (KEAs) in each Round 1 

state. The study also included interviews with state administrators to confirm and clarify the 

information obtained from documents and, as needed, to gather information that could not 

be obtained from the document reviews. 

4. Estimated or Actual Completion Date: An evaluation brief is scheduled for completion in 

summer 2017. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_rtt.asp

Policy and Program Studies Service (PPSS)

TEACH Grant Study 

1. Study Purpose: The TEACH Grant program provides grants up to $4,000 a year to students 

who are completing coursework needed to begin teaching. If a recipient does not complete 

four years of service in a high-need subject at a high-need school within eight years after 

completing their coursework, their grant funds are converted to a direct unsubsidized loan. 

This study examines how institutions support and inform students who are eligible for 

TEACH Grants. In addition, the study examines why some participants do not meet TEACH 

Grant service requirements. Lastly, the study examines factors associated with grant 

recipients meeting and not meeting service requirements. 

2. Key Questions Addressed: 

 How are TEACH Grants administered in institutions of higher education? 

 Why do some TEACH Grant recipients fail to meet program service requirements? 

 What are the factors associated with TEACH Grant recipients meeting and not meeting 

the grant service requirements? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 177 

3. Design: The study includes surveys of 1) 479 institutions that administered at least 

10 TEACH Grants in the 2014–15 award year and 2) a sample of 500 TEACH Grant 

recipients. In addition, the study includes case studies of six institutions and an analysis of 

administrative data. 

4. Estimated Completion Date: The final report is scheduled for completion in summer 2017.

National Survey on High School Strategies Designed to Help At-Risk Students Graduate 

1. Study Purpose: This nationally representative survey of high school administrators is 

examining strategies that high schools use to reduce students’ likelihood of dropping out of 

high school and to increase their likelihood of attaining a high school credential. The survey 

seeks information on the prevalence of high school graduation strategies, the students who 

participate in them, and how high schools deliver services or interventions as part of that 

strategy. The Department will release a set of issue briefs based on the survey data to 

describe the prevalence and characteristics of dropout prevention strategies for at-risk youth 

and will compare high schools with high and low graduation rates, among other school 

characteristics. 

2. Key Questions Addressed: 

 What strategies are high schools implementing to help students stay in school and 

graduate? Do these strategies vary for high schools with high or low graduation rates? 

 How many students are served through each of these strategies? Are the strategies 

focused on particular student populations? 

 How do schools deliver services or interventions for each of the strategies? What 

specific services are provided, and who provides the services? 

3. Design: The study is conducting a web-based survey of a nationally representative sample 

of approximately 2,000 high school administrators. 

4. Estimated Completion Date: Issue briefs are scheduled for completion between winter 

2016–17 and spring 2017. 

Implementation Study of the Turnaround School Leaders Program 

1. Study Purpose: This study will examine the implementation of the Turnaround School 

Leaders Program (TSLP). This study seeks to generate information to help policymakers 

and practitioners who struggle with the challenges of developing leaders to turn around lowperforming schools and to add to the field’s general body of knowledge about developing 

turnaround leadership pipelines. 

2. Key Questions Addressed: 

 How do TSLP projects identify, develop, and support leaders for low-performing 

schools? 

 What role do project partners play in implementing projects and helping grantees to 

achieve project goals? 

 How have grantees modified projects to adapt to challenges or meet the demands of 

changing circumstances? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 178 

 How are grantees measuring the success of their TSLP projects, and do early outcome 

data show promising results? 

 How have the turnaround school leaders grants contributed to developing a sustainable, 

long-term pipeline of leaders for turnaround schools? 

3. Design: The study will include surveys of 12 TSLP cohort 1 grantees and case studies of 

seven TSLP cohort 1 grantees, including each grantees’ partners; an analysis of extant 

data, including grantee applications, early outcomes data, and other relevant project-specific 

data. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Study of School Climate Transformation Grants 

1. Study Purpose: The study will describe how states and school districts that participate in the 

School Climate Transformation Grant (SCTG) program are coordinating services and 

supports with certain other related federal programs administered by the Departments of 

Health and Human Services and Justice. 

2. Key Questions Addressed:

 What does coordination between SCTG and Project AWARE grantees look like? 

 What does coordination between SCTG and DOJ grantees look like? 

 What did grantees say about the value of coordination? 

 What were the challenges and lessons learned? 

3. Design: The study will conduct telephone interviews with representatives from 38 state and 

local grantees to explore the ways in which grantees coordinate services, the benefits that 

grantees have experienced from program coordination, and challenges and lessons learned. 

4. Estimated Completion Date: The report is scheduled for completion in late fall/winter 2017.

Study of Title I Schoolwide and Targeted Assistance Programs 

1. Study Purpose: The study will examine how Title I schoolwide programs use the schoolwide 

flexibility to design services and strategies to address the needs of low-achieving students 

and subgroups, and how such strategies compare to approaches used in targeted 

assistance programs. The study will include interviews and analysis of extant data in 

approximately 30 Title I schools, including both schoolwide and targeted assistance 

programs, as well as surveys of principals and district administrators for a representative 

sample of Title I schools. 

2. Key Questions Addressed: 

 How do schoolwide and targeted assistance programs use Title I funds to improve 

student achievement, particularly for low-achieving subgroups? 

 How do districts and schools make decisions about how to use Title I funds in 

schoolwide programs and targeted assistance programs? 

 To what extent do schoolwide programs commingle Title I funds with other funds or 

coordinate the use of Title I funds with other funds? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 179 

3. Design: The study will include both in-depth case studies and surveys of a nationally 

representative sample of 350 districts and 1,400 schools. The case studies will include 

approximately 35 Title I schools including both schoolwide and targeted assistance 

programs. Data collection for the case studies will include site visits and interviews with two 

to four staff members in each school and approximately two staff members in each district, 

as well as extant documents and data, including Title I budgets and plans. 

4. Estimated Completion Date: The report is scheduled for completion in spring 2018. 

Study of Digital Learning Resources for Instructing English Learners 

1. Study Purpose: This study will examine the use of digital learning resources (DLRs) to 

support the English language acquisition and academic achievement of English learners in 

K–12 education. The study will explore the range of DLRs that are available for use with 

English learner students, examine how districts and schools select and use these apps, and 

consult an expert panel of technology developers, practitioners, and education researchers 

for ideas on ways to improve the design and use of apps to support learning for English 

learner students. The study will culminate in a final report that presents findings from the 

study, as well as two short field-focused toolkits or guides for educators and technology 

developers that present key information from the study in a manner that will be accessible 

and useful for those audiences. 

2. Key Questions Addressed: 

 How do districts and teachers identify and select DLRs in general? How do districts and 

teachers identify and select DLRs specifically to support English learner students? 

 What are the types and characteristics of DLRs that districts most commonly report as 

used to support English learners? What are the types and characteristics of DLRs that 

teachers most commonly report that they use in instructing and structuring learning 

activities for their English learner students? 

 How do teachers of English learner students use DLRs in the instruction of English 

learner students? 

 To what extent do teachers receive professional development or other supports for 

effective use of DLRs for instruction? Which professional development approaches do 

teachers report to be most helpful in supporting their use of DLRs in instruction? 

 What are barriers to and supports for (1) the use of DLRs in instruction of English learner 

students and (2) the use of DLRs by students at home? How can districts, schools, and 

DLR developers address these? 

 How do districts and teachers define and measure the success of their use of technology 

to support English learner students? 

 How could developers and practitioners improve the usefulness of DLRs for instructing 

English learner students? 

3. Design: The study will explore the range of such apps that are available for use with English 

learner students, examine how districts and schools select and use these apps, and consult 

an expert panel of technology developers, practitioners, and education researchers for ideas 

on ways to improve the design and use of apps to support learning for English learner 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 180 

students. It will rely on six key components: (1) a literature review; (2) market research on 

existing DLRs for K–12 instruction; (3) survey of school districts; (4) survey of teachers; 

(5) case studies; and (6) an expert panel of technology developers, practitioners, and 

education researchers. 

4. Estimated Completion Date: The report is scheduled for completion in spring 2018. 

Study of the Title III Native American and Alaska Native Children in Schools Program

1. Study Purpose: The study will examine services and strategies for English learners in the 

Native American, Alaska Native, Native Hawaiian, and Pacific Islander communities that are 

supported through the Native American and Alaska Native Children in Schools (NAM) 

program. The study includes case study of 25 NAM grantees, and examines: (1) the types of 

services and strategies that NAM funds support to address the instructional needs of these 

communities and develop student proficiency in both English and (optionally) native 

languages; (2) how grantees plan, implement, and evaluate their respective projects; 

(3) how grantees coordinate and prioritize the use of NAM funds in relation to other federal, 

state, and local resources; and (4) how grantees use data and evidence to inform program 

implementation and meet U.S. Department of Education reporting requirements. 

2. Key Questions Addressed: 

 How does the NAM program support grantees in providing services to Native American, 

Alaska Native, Native Hawaiian, and Pacific Islander children, especially through teacher 

training and professional development, curriculum development, parent engagement, 

and English (especially academic English) and native language instruction? How do 

NAM grantees incorporate technology to support and/or preserve heritage languages 

through dual-language or other forms of English language instruction? How many 

students are served through each of these strategies? Are the strategies focused on 

particular student populations? 

 What challenges do NAM grantees face in providing funded services, and what steps 

have grantees taken to overcome these challenges? To what extent does the 

Department or other external entities provide support to overcome these challenges? 

What lessons have the grantees learned? 

 How do schools deliver services or interventions for each of the strategies? What 

specific services are provided, and who provides the services? 

 What are the roles and responsibilities among (as applicable) tribal entities, public 

schools, local education districts, and state agencies in implementing NAM grants and 

meeting federal reporting requirements? 

 What are NAM grant stakeholders’ perceptions of community participation and student 

engagement in language instruction and other educational programs? 

3. Design: The study will include site visits to NAM grantees to obtain detailed information 

regarding how they are implementing the NAM program and meeting the needs of their 

students and communities, as well as telephone surveys or interviews of local grant 

coordinators and state directors of Indian education to inform the case study site visits. The 

study will also utilize extant data, including grant applications.

4. Estimated Completion Date: The report is scheduled for completion in spring 2018. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 181 

Evaluation of the Title I, Part D Neglected or Delinquent Program 

1. Study Purpose: The study will examine the implementation of educational programs for 

children and youth in residential facilities and correctional institutions funded under Title I, 

Part D of ESEA. The study will be informed by surveys of state grantees and local 

subgrantees to examine the types of services and strategies that Part D funds support, how 

state and local agencies assist students in transitioning back to schools, how state 

correctional facilities implement institution-wide Part D projects, and how grantees assess 

the educational outcomes of participating students. 

2. Key Questions Addressed: 

 What types of services and strategies do Title I, Part D funds support for children and 

youth in correctional and child welfare settings? 

 How do correctional facilities and child welfare agencies assist students in transitioning 

back to districts and schools, including those outside their jurisdictions? 

 How do state correctional facilities plan and implement institution-wide Part D projects? 

 How do grantees assess the educational outcomes of students participating in 

Part D-funded educational programs? 

3. Design: The study will include a review of extant data, a review of literature related to 

programs for neglected and delinquent youth, surveys of state and local coordinators of 

Title I-Part D funded programs, and site visits to state agencies, school districts, correctional 

institutions, and child welfare facilities to obtain more detailed information on how grantees 

and subgrantees are implementing the Part D programs and how they are meeting the 

needs of their students. 

4. Estimated Completion Date: The report is scheduled for completion in spring 2018.

Evaluation of the Migrant Education Program 

1. Study Purpose: This study will examine how states, districts, and schools are providing 

instructional supports and assessing highly mobile migratory students, as well as examine 

state plans for implementing the new accountability requirements (as they pertain to 

migratory students) under the ESSA. The Migrant Education Program (MEP) is a state 

program, giving states flexibility in how they allocate funds to serve migratory students, 

allowing the local entities that serve migratory students to be both local educational

agencies (LEAs) and local operating agencies (LOAs). 

2. Key Questions Addressed: 

 How do state and local grantees respond to federal requirements for serving migratory 

children? 

 What services are provided to migratory students? 

 How do MEP grantees collaborate with other programs and agencies to address the 

needs of highly mobile students? 

 How do migrant programs support students in earning high school diplomas and 

equivalency, and preparing for postsecondary education and the workforce? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 182 

3. Design: The study will include survey of states, a nationally representative sample of district 

programs serving migrant children, site visits to state and local grantees. The contractor 

shall prepare a final report that integrates findings from the surveys and case studies as well 

as from an extant data analysis. 

4. Estimated Completion Date: The report is scheduled for completion in fall 2018.

Evaluation of the Indian Education LEA Grants Program 

1. Study Purpose: This study will examine the implementation of the Indian Education Local 

Education Agency (LEA) Grants Program funded under Title VI of the ESEA, as amended 

by the ESSA. The study will document the scope of activities funded by the Indian Education 

LEA Grants Program and will examine the LEA and tribal-level implementation of the grants. 

Specifically, the study will examine the processes used to identify and count eligible children 

and how grantees establish LEA program priorities and implement grant-funded services. 

2. Key Questions Addressed: 

 What strategies do grantees use to identify and count program-eligible children?

 How do grantees plan services?

 What services do Indian Education LEA Grants fund?

 How do grantees assess the project outcomes of students participating in Indian 

Education LEA Grants Program-funded educational programs? 

3. Design: This study will consist of four key components: (1) analysis of extant data including 

APRs and Electronic Application System for Indian Education (EASIE) data; (2) review of 

relevant literature; (3) survey of 1,300 coordinators of Indian Education LEA grant programs; 

and (4) case studies of nine districts. 

4. Estimated Completion Date: The report is scheduled for completion in fall 2018.

Study of Weighted Student Funding Systems 

1. Study Purpose: This study is examining weighted student funding (WSF) and school-based 

budgeting (SBB) systems, which are methods for providing funds to schools based on the 

numbers and, in the case of WSF, types of students they served. The study will examine 

how districts have implemented SBB and WSF systems for allocating funds to schools and 

how these districts and their schools compare with districts using traditional systems for 

allocating school resources. 

2. Key Questions Addressed: 

 How are resources allocated to schools in districts with SBB or WSF systems compared 

with districts with more traditional resource allocation practices? 

 In what ways do schools have autonomy and control over resource allocation decisions, 

and how does this vary between districts with SBB or WSF and other districts? 

 How has the implementation of WSF systems affected the distribution of funding 

provided to schools? 

 What challenges did districts and schools experience in implementing SBB and WSF, 

and how did they respond to those challenges? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2016 Annual Performance Report and FY 2018 Annual Performance Plan—U.S. Department of Education 183 

3. Design: To obtain detailed information about the implementation, benefits, and challenges of 

WSF systems, this study will conduct case studies in nine districts using such systems, 

including site visits that include in-person interviews and collection and analysis of extant 

documents and data such as school budgets and planning documents. In addition, the study 

will administer surveys to a nationally representative survey of districts and schools to 

enable the study to ground the case study data in the larger context of WSF, SBB, and 

traditional methods of resource allocation. 

4. Estimated Completion Date: The report is scheduled for completion in winter 2018–19.