APPENDICES

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 167 

Appendix E: Summary of Performance Evaluations Conducted 

During FY 2015 and Expected During FY 2016–17 

For a complete list of program evaluations and studies from the Office of Planning, Evaluation, 

and Policy Development, please visit 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html. For a complete list of evaluation 

studies of the National Center for Education Evaluation and Regional Assistance, please visit 

http://ies.ed.gov/ncee/projects/evaluation/index.asp. 

Evaluation Reports From FY 2015

National Center for Education Evaluation and Regional Assistance

School Choice 

Evaluation of the DC Opportunity Scholarship Program: An Early Look at Applicants and 

Participating Schools Under the SOAR Act 

1. Study Purpose: The April 2011 Scholarships and Opportunities for Results Act (SOAR Act) 

provided for a five-year continuation of a school choice program for low-income residents of 

Washington, DC. The program, still titled the Opportunity Scholarship Program (OSP), now 

provides annual scholarships of $8,000 (for grades K–8) or $12,000 (for grades 9–12) to 

enable low-income students to attend private schools in DC in lieu of the public schools 

already available to them. The new law also mandated another independent, rigorous 

evaluation of the effectiveness of the program. 

2. Key Questions Addressed:

 What is the impact of the OSP on student academic achievement and other measures of 

student success, overall and for subgroups of students identified in the statute as high 

priority? 

 What effect does the program have on student and parent perceptions of school safety 

and satisfaction, and on parents’ involvement in education of their children? 

 Does the program change students’ instructional environments and opportunities? 

3. Design: The evaluation will primarily compare outcomes of approximately 1,800 student 

applicants randomly assigned by lottery to either receive a scholarship or not receive a 

scholarship. Lotteries of program applicants were conducted in spring 2012 (cohort 1), 

spring 2013 (cohort 2), and spring 2014 (cohort 3). Data will be collected for three follow up 

years for each of the cohorts and, for students in both the scholarship and nonscholarship 

groups. The contractor will administer academic assessments, and conduct student, parent, 

and principal surveys each spring (spring 2013–16). Prior to the first impact report, 

descriptive reports based on application forms parents filled out when they applied to the 

OSP,principal surveys, and extant data will be released. 

4. Estimated or Actual Completion Date: The first descriptive report—describing the 

characteristics of participating schools and program applicants—was released in October 

2014. A second descriptive report is scheduled for release in early 2016 and a third in early 

spring 2016. The first impact report is scheduled for completion in summer 2016. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 168 

5. Key Findings from the First Report:

 Just over half of all DC private schools participated in the OSP, with current schools 

more likely to have published tuition rates above the OSP scholarship amounts than did 

participating schools in the past. 

 OSP applicants under the SOAR Act represent between 3 and 4 percent of the 

estimated 53,000 children in DC who meet the eligibility criterion. 

 A number of awarded scholarships go unused, with students from disadvantaged 

schools and families using awarded scholarships at lower rates than others. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154000/pdf/20154000.pdf 

Students with Disabilities 

School Practices and Accountability for Students with Disabilities 

1. Study Purpose: The focus of this study was on the inclusion of students with disabilities 

(SWDs) in school accountability systems and the variation in school practices in schools 

accountable and schools not accountable for the performance of the SWDs subgroup under 

the Elementary and Secondary Education Act (ESEA).

2. Key Questions Addressed:

 To what extent are schools accountable for the performance of the SWDs subgroup, and 

how does this accountability vary across schools and over time? 

 To what extent have schools accountable for the SWDs subgroup been identified as 

needing improvement? 

 How does school accountability for the SWDs subgroup relate to regular and special 

education practices for SWDs? 

3. Design: The evaluation relied on descriptive statistics to study patterns of school 

accountability across states and over time and to examine how school practices vary with 

school accountability for the SWDs subgroup. Data sources for the evaluation include extant 

data from the Department’s EDFacts database and 2011 surveys of principals and special 

education designees from elementary and middle schools in 12 states. 

4. Estimated or Actual Completion Date: The first two research questions for the study were 

addressed in an interim report released in May 2012 and an update of the 2012 interim 

report released in October 2013. A third report, addressing the third research question, was 

released in February 2015. 

5. Key Findings: This report addressed the first two key questions listed above for this study: 

 Across the 44 states with relevant data and DC, 35 percent of public schools were 

accountable for the performance of the SWDs subgroup in the 2009–10 school year, 

representing 59 percent of SWDs in those states. In those same 44 states and DC, 

62 percent of middle schools were accountable for SWDs performance, while 32 percent 

of elementary schools and 23 percent of high schools were accountable. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 169 

 In 31 states with relevant data, 56 percent of public schools were not accountable for the 

SWDs subgroup in any of the 4 years examined, in comparison with 23 percent of 

schools that were consistently accountable in each of the 4 years. 

 Among schools that were consistently accountable for the performance of the SWDs 

subgroup across 22 states during the 4 years, 56 percent were never identified for 

school improvement over this time period. By comparison, among schools that were 

consistently not accountable for SWDs subgroup performance in these states, 

80 percent were never identified for improvement. 

 When surveyed in 2011, elementary schools accountable for the SWDs subgroup were 

15.8 percentage points more likely than never-accountable elementary schools to report 

moving SWDs from self-contained settings to regular classrooms over the previous five 

years. Middle schools accountable for the SWDs subgroup were 16.7 percentage points 

more likely than never-accountable middle schools to report moving SWDs from selfcontained settings to regular classrooms over the previous five years. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154006/ 

Evaluation of the Teacher Incentive Fund: Implementation and Impacts of Pay-forPerformance After Two Years 

1. Study Purpose: The Teacher Incentive Fund (TIF) is authorized by Title V, Part D of ESEA 

and annual appropriations acts. The purpose of the TIF program is to develop and 

implement performance-based compensation systems (PBCSs) for teachers, principals, and 

other personnel in high-need schools. Research indicates that high quality teachers are 

critical to raising student achievement in low-performing schools, but schools most in need 

often have difficulty in attracting and retaining high-quality teachers. Performance pay is a 

policy promoted by the TIF program to improve the quality of teachers in high-need schools. 

This evaluation studies performance pay that provides substantial and differentiated bonus 

pay to high-performing teachers in low-performing schools with high-need students. 

2. Key Questions Addressed:

 What are the characteristics of all TIF districts and their performance-based 

compensation systems? What implementation experiences and challenges did TIF 

districts encounter? 

 How do teachers and principals in schools that did or did not offer pay-for-performance 

bonuses compare on key dimensions, including their understanding of TIF program 

features, exposure to TIF activities, allocation of time, and attitudes toward teaching and 

the TIF program? 

 What is the impact of pay-for-performance bonuses on students’ achievement on state 

assessments of math and reading? 

3. Design: Study schools were randomly assigned within a grant to either implement all 

components of the PBCS or the PBCS with a 1 percent across-the-board bonus in place of 

the differentiated effectiveness incentive component of the PBCS. Data collection will 

include a grantee survey, a survey of teachers and principals, teacher and principal school 

assignment records, student record information (such as student demographics and student 

test scores), and grantee interviews to document implementation information, as well as to 

conduct impact analyses. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 170 

4. Estimated or Actual Completion Date: A report on the implementation and early impacts 

after one year was released in September 2014. The second report on the ongoing 

implementation and impacts after two years was released in September 2015. 

5. Key Findings: The main findings among all TIF districts with 2010 awards are: 

 Ninety percent of all TIF districts in 2012–13 reported implementing at least 3 of the 4 

required components for teachers, and only about one-half (52 percent) reported 

implementing all four. This was a slight improvement from the first year of 

implementation. 

 Near the end of the second year of implementation, 65 percent of TIF districts reported 

that sustainability of the program was a major challenge. In contrast, no more than onethird of districts reported that other activities related to their program were a major 

challenge. 

For the subset of 10 districts that agreed to participate in a random assignment study, key 

findings on the effect of pay-for-performance on educators include the following: 

 Few evaluation districts structured pay-for-performance bonuses to align well with TIF 

guidance in the grant competition notice. 

 Educators’ understanding of key program components improved from the first to the 

second year, but many teachers still did not understand that they were eligible for a 

bonus. They also continued to underestimate how much they could earn from 

performance bonuses. 

 Offering educators pay-for-performance had small, positive impacts on their students’

reading achievement; impacts on students’ math achievement were not statistically 

significant but similar in magnitude. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154020/ 

New Findings on the Retention of Novice Teachers from Teaching Residency Programs 

1. Study Purpose: Teaching Residency Programs (TRPs) involve a year-long “clinical” 

experience (the “residency”) shadowing and co-teaching with an experienced mentor. TRPs 

also provide continued support and mentoring after participants become teachers of record. 

Before and during their residencies, participants in TRPs take coursework. The evaluation of 

TRPs, which focused on residency programs that received grants from the Teacher Quality 

Partnership (TQP) program, provides important descriptive and implementation information 

on TRPs, as well as information on the teacher retention outcomes of teachers who 

participate in TRPs. 

2. Key Questions Addressed:

 What are the characteristics of TRPs (e.g., length of overall program, criteria for 

selecting program participants)? What are the characteristics of participants in TRPs? 

 What are the retention rates of novice TRP teachers and their novice colleagues who did 

not go through TRPs? 

3. Design: Descriptive information concerning TQP grantees operating TRPs was collected 

through a survey administered in spring 2012. More detailed implementation information 

was collected through TRP program director interviews and surveys of residents and 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 171 

mentors, conducted within a subset of TRPs during spring 2012. Teacher mobility was 

tracked through district records and teacher surveys in order to examine retention in the 

profession, district, and school, among novice TRP and novice non-TRP teachers in a 

subset of six districts. 

4. Estimated or Actual Completion Date: A study report was released in November 2014. The 

follow-up evaluation brief was released in August 2015. 

5. Key Findings: The study’s evaluation brief updates the retention findings included in the 

study’s report. It examines the rates at which the residency teachers were retained in the 

same district and the same school as of fall 2013, thereby tracking two successive cohorts 

of teachers into their third or fourth year as a teacher of record. It found that: 

 TRP teachers were more likely to remain teaching in the same district than non-TRP 

teachers with similar teaching placements. Eighty-two percent of residency teachers and 

72 percent of other teachers remained in the same district from spring 2012 to fall 2013. 

 School-retention rates were similar between the two groups of teachers. Approximately 

61 percent of teachers in both groups remaining in the same school from spring 2012 to 

fall 2013. 

 TRP teachers who moved to different schools in the same district tended to join ones 

where a similar proportion of students were from low-income families, a lower 

percentage were black, and achievement was higher. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154015/

Teaching Residency Programs: A Multisite Look at a New Model to Prepare Teachers for 

High-Need Schools 

1. Study Purpose: Teaching Residency Programs (TRPs) involve a year-long “clinical” 

experience (the “residency”) shadowing and co-teaching with an experienced mentor. TRPs 

also provide continued support and mentoring after participants become teachers of record. 

Before and during their residencies, participants in TRPs take coursework. The evaluation of 

TRPs, which focused on residency programs that received grants from the Teacher Quality 

Partnership (TQP) program, provides important descriptive and implementation information 

on TRPs, as well as information on the teacher retention outcomes of teachers who 

participate in TRPs. 

2. Key Questions Addressed:

 What are the characteristics of TRPs (e.g., length of overall program, criteria for 

selecting program participants)? What are the characteristics of participants in TRPs? 

 What are the retention rates of novice TRP teachers and their novice colleagues who did 

not go through TRPs? 

3. Design: Descriptive information concerning TQP grantees operating TRPs was collected 

through a survey administered in spring 2012. More detailed implementation information 

was collected through TRP program director interviews and surveys of residents and 

mentors, conducted within a subset of TRPs during spring 2012. Teacher mobility was 

tracked through district records and teacher surveys in order to examine retention in the 

profession, district, and school, among novice TRP and novice non-TRP teachers in a 

subset of six districts. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 172 

4. Estimated or Actual Completion Date: The study report was released in November 2014. A 

follow-up evaluation brief was released in August 2015. 

5. Key Findings: The study’s report provides descriptive information regarding residency 

programs’ activities and participants. It also examines early teacher retention outcomes. It 

found that: 

 The residency programs provided a fieldwork experience, typically with a trained and 

experienced mentor teacher, along with integrated coursework. On average, teaching 

residents reported being fully in charge of instruction for 21 days during the first half of 

the residency and 37 days during the second half. Most mentor teachers had substantial 

teaching experience (10 years, on average), were trained by the residency program 

(averaging 37 hours of training), and had prior mentoring experience (an average of 

3.5 semesters). The residency programs included the equivalent of 10 courses on 

average, and most residents (83 percent) reported that their fieldwork reinforced what 

they learned in their coursework. 

 The residency programs somewhat broadened the pool of people entering the teaching 

profession in the participating districts. Novice teachers from the residency programs 

were more likely to report having worked in a full-time job other than teaching as 

compared to novice teachers from other preparation programs (72 percent versus 

63 percent). However, the groups had similar demographic characteristics (sex, 

race/ethnicity, and age). 

 Novice teachers from residency programs had similar retention rates to other novice 

teachers. Approximately 90 percent of teachers from both groups reported staying in the 

same district from spring 2012 to fall 2012; about 5 percent were no longer teaching. 

6. Links to Additional Information:

http://ies.ed.gov/pubsearch/pubsinfo.asp?pubid=NCEE20154002 

Other 

Are Low-Performing Schools Adopting Practices Promoted by School Improvement 

Grants (SIGs) ? 

1. Study Purpose: The SIG program is authorized through Title I of ESEA and provides 3-year 

awards to support turnaround in the nation’s persistently lowest-achieving schools. In 

FY 2009, the $546 million SIG appropriation was supplemented by $3 billion through the 

American Recovery and Reinvestment Act of 2009 (Recovery Act), for a total of $3.5 billion. 

SIG funds are distributed to states by formula based on Title I allocations. States then 

competitively award funds to districts applying on behalf of their eligible schools. Schools 

receiving SIG must implement one of four prescriptive intervention models: turnaround, 

transformation, closure, or restart. Previous research provides evidence that low-performing 

schools adopt some practices promoted by the four models, but little is known about how 

schools combine these practices. This study describes the individual SIG-promoted 

improvement practices adopted by low-performing schools. 

2. Key Questions Addressed: This report describes both the individual SIG-promoted 

improvement practices and the combinations of these practices that low-performing schools 

reported adopting. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 173 

3. Design: Findings are based on spring 2013 survey responses from 480 school 

administrators in 60 districts and 22 states. The 480 schools are all low-performing schools, 

with some receiving SIG and others not. 

4. Estimated or Actual Completion Date: The report was released in October 2014. 

5. Key Findings:

 Schools on average reported adopting 20 of 32 improvement practices promoted by the 

SIG transformation or turnaround models. 

 No school reported adopting all practices required under the transformation or 

turnaround models. 

 More than 96 percent of schools reported adopting each of the three most commonly 

adopted individual practices: using data to inform and differentiate instruction, increasing 

technology access for teachers or using computer-assisted instruction, and providing 

ongoing professional development that involves teachers working collaboratively or is 

facilitated by school leaders. 

 For 16 of the 32 practices examined, schools implementing a SIG model were 

statistically significantly more likely than schools not implementing one to report adopting 

that practice. 

 Almost every school reported adopting a unique combination of practices, but certain 

practices (for example, the three most commonly adopted practices listed above) were 

much more likely than others (for example, using financial incentives to recruit and retain 

effective teachers and principals) to be included in these combinations. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154001/

Building Teacher Capacity to Support English Language Learners in Schools Receiving 

SIGs 

1. Study Purpose: SIGs are authorized by Title I, Section 1003(g) of the ESEA. The purpose of 

the grants—awarded based on the Title I funding formula to states, which then competitively 

distribute the funds to districts applying on behalf of their eligible schools—is to support the 

turnaround of the nation’s persistently lowest-achieving schools. To qualify for the three-year 

grant, schools must (among other requirements) be willing to implement one of four 

prescribed intervention models: turnaround, restart, closure, or transformation. About 

$546 million was allocated in FY 2009 for SIG with a supplement of $3 billion from the 

Recovery Act. With the possibility of rollover funds, this amounts to a $3.5 billion injection 

into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This study 

will provide descriptively rich, primarily qualitative information for a small set of SIG schools 

with high proportions of English Language Learner (ELL) students that are receiving SIG in 

the first cohort to implement an intervention model beginning in the 2010–11 school year. 

2. Key Questions Addressed: This brief describes the efforts of SIG schools with high 

proportions of ELLs to improve teachers’ capacity for serving ELLs through staffing 

strategies and professional development. 

3. Design: This study employs a school-level case study design. A sample of 11 SIG schools 

with high percentages of ELLs (a median of 45 percent ELs) was purposively selected from 

four states to represent a range of geographic regions, urbanicities, school sizes, 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 174 

racial/ethnic compositions, socioeconomic statuses, SIG intervention models, and SIG 

funding levels, among other factors. Findings are based on ELL case study data collected in 

fall 2011, spring 2012, and fall 2012. 

4. Estimated or Actual Completion Date: The report was released in November 2014. 

5. Key Findings:

 Few schools reported leveraging staffing strategies to improve teacher capacity for 

serving ELLs. Administrators in 3 of the 11 schools reported considering ELL expertise 

and experience when hiring classroom teachers, while respondents in 2 of the 

11 schools reported that teachers’ ELL expertise and experience purposefully factored 

into assignment of teachers to specific classrooms. 

 Most teacher survey respondents (54 to 100 percent) in all 11 schools reported 

participating in ELL-related PD during the 2011–12 school year. On average, teachers 

reported that ELL-related PD accounted for less than 20 percent of their total PD hours. 

 Teacher survey respondents in schools that reported a greater PD focus on ELL-related 

topics, such as instructional strategies for advancing English proficiency or instructional 

strategies to use for ELLs within content classes, also generally appeared more likely to 

report that PD improved their effectiveness as teachers of ELLs. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154004/

Evaluation of the Regional Educational Laboratories: Final Report 

1. Study Purpose: The Regional Educational Laboratories (RELs) are a networked system of 

10 organizations that serve the educational needs of 10 designated regions across the 

United States. The Department is authorized by the Education Sciences Reform Act (ESRA) 

to award contracts to 10 RELs to support applied research, development, wide 

dissemination, and technical assistance activities. The REL program is administered by the 

Knowledge Utilization Division of NCEE. The FY12 appropriation for the REL program was 

$57.426 million. The national evaluation of the RELs is mandated under ESRA. 

2. Key Questions Addressed: 

 What activities did the RELs undertake to fulfill the missions specified in ERSA? 

 What were the technical quality and relevance of REL Fast Response Project reports 

published by IES and of the corresponding proposals? 

 What were the technical quality and relevance of REL impact study reports published by 

IES and of the corresponding proposals? 

 How relevant and useful were the REL technical assistance products to the needs of the 

states, localities, and policymakers in their regions? 

3. Design: This descriptive study relied on a combination of extant data, FY 2010 interviews 

with REL directors, and FY 2012 surveys of potential REL customers from state and local 

education agencies. Panels of experts met during FY 2010 and FY 2012 and rated the 

quality and relevance of REL Fast Response Project proposals and final reports and REL 

impact study proposals and final reports. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 175 

4. Estimated or Actual Completion Date: An interim report, based on interviews with REL 

directors and expert reviews of REL Fast Response Projects (FRPs), was released in 

September 2013. A final report, including findings from surveys of REL customers and 

expert reviews of REL impact studies, was released in April 2015. 

5. Key Findings from the Interim and Final Reports:

 REL staff members reported activities under each of the ten missions of the REL 

program specified in ESRA. The statement of work (SOW) for the REL contracts in place 

between 2006 and 2011 aligned explicitly with six of the ten statutory missions for the 

REL program. Four additional statutory missions were not explicitly in the SOW for the 

RELs, but RELs reported activities under those missions as well. 

 By December 1, 2009, the ten RELs had submitted 297 proposals to IES to conduct 

FRPs, of which 46 percent (137) were accepted for performance under the REL 

contracts. 

 The IES-published FRP reports received a mean quality rating of 3.81 on a 5-point 

scale, while the corresponding proposals received a mean quality rating of 3.24. Both of 

these means fell between the categories of “adequate” and “strong” quality. 

 The IES-published FRP reports received a mean relevance rating of 3.64 on a 5-point 

scale, while the corresponding proposals received a mean relevance rating of 3.39. Both 

of these means fell between the categories of “adequate” relevance and “relevant.”

 Expert panelists rated the 8 impact study reports selected for review as, on average, 

between “strong” and “very strong” in quality (4.10 on a 5-point scale, with 5 being the 

highest value), and between “relevant” and “very relevant” (4.06 on a 5-point scale). 

 Eighty-six percent of state educational agency administrators and 52 percent of school 

district administrators reported being “very familiar,” “somewhat familiar,” or “a little 

familiar” with the REL program. 

 Fifty percent of state administrators and 26 percent of district administrators who were at 

least “a little familiar” with the REL program were “very satisfied” with it. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154008/

State Capacity to Support the Turnaround of Low-Performing Schools 

1. Study Purpose: One objective of the Department’s SIG and Race to the Top (RTT) program 

is to help states enhance their capacity to support the turnaround of low-performing schools. 

This capacity may be important, given how difficult it is to produce substantial and sustained 

achievement gains in low-performing schools. There is limited existing research on the 

extent to which states have the capacity to support school turnaround and are pursuing 

strategies to enhance that capacity. This report documents states’ capacity to support 

school turnaround as of spring 2012 and spring 2013. It examines capacity issues for all 

states and for those that reported both prioritizing turnaround and having significant gaps in 

expertise to support it. 

2. Key Questions Addressed: This report documents states’ capacity to support school 

turnaround as of spring 2012 and spring 2013. It examines capacity issues for all states and 

for those that reported both prioritizing turnaround and having significant gaps in expertise to 

support it. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 176 

3. Design: Findings are based on interviews in spring 2012 and 2013 with administrators from 

49 states and the District of Columbia (12 Round 1 and 2 RTT states, 7 Round 3 RTT 

states, and 31 non-RTT states). 

4. Estimated or Actual Completion Date: The report was released in May 2015. 

5. Key Findings:

 More than 80 percent of states made turning around low-performing schools a high 

priority, but at least 50 percent found it very difficult to turn around low-performing 

schools. 

 38 states (76 percent) reported significant gaps in expertise for supporting school 

turnaround in 2012, and that number increased to 40 (80 percent) in 2013. 

 More than 85 percent of states reported using strategies to enhance their capacity to 

support school turnaround, with the use of intermediaries decreasing over time and the 

use of organizational or administrative structures increasing over time. 

 States that reported both prioritizing school turnaround and having significant gaps in 

expertise to support it were no more likely to report using intermediaries than other 

states but all 21 of these states reported having at least one organizational or 

administrative structure compared with 86 percent (25 of 29) of all other states. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154012/

State, District, and School Level Implementation of Reforms Promoted Under the 

Recovery Act 

1. Study Purpose: The Recovery Act provided an unprecedented $100 billion of funding for the 

Department. While the initial goal of this money was to deliver emergency funding for 

education and government services, the Recovery Act is also being used as an opportunity 

to spur innovation and reform at different levels of the education system. In turn, the 

Recovery Act provides a unique opportunity to foster school improvements and to learn from 

reform efforts. Although funds are being distributed through different grant programs, their 

goals and strategies are complementary, if not overlapping, as are likely recipients. For this 

reason, data collection and analysis took place across grant programs (i.e., was 

“integrated”), rather than separately for each set of grantees, allowing for a broad 

assessment of the Recovery Act as a whole. The Department seeks to understand through 

this evaluation how states, districts, and schools are working to implement the education 

reforms promoted by the Recovery Act. 

2. Key Questions Addressed:

Funding report: 

 At the state and local levels, who were the recipients of Recovery Act funds? To what 

extent did child poverty, state fiscal condition, student achievement, and other variables 

relate to funding? 

Interim and final reports: 

 To what extent were states, districts, and schools implementing the key education 

reform strategies promoted by the Recovery Act in 2010–11 in the areas of standards 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 177 

and assessment, data systems, educator workforce development, and support for low 

performing schools? 

 To what extent did 2010–11 implementation of key education reforms reflect progress 

since the Recovery Act funds were initially distributed in 2009–10?

 What were the greatest challenges experienced by states, districts, and schools working 

to implement key education reforms in the 2010–11 school year? 

3. Design: This study is primarily based on data from surveys of all 50 state education 

agencies (SEAs) and DC, and a nationally representative sample of school districts 

administered between spring 2011 and 2012. Survey respondents were the chief state 

school officer or other state agency officials designated by the chief as most knowledgeable 

about the topics in the survey. Descriptive and correlational analyses are used to answer the 

study’s research questions. 

4. Estimated or Actual Completion Date: A first report, titled “State and District Receipt of 

Recovery Act Funds,” was released in 2012. An interim report focused on state level 

implementation was released in January 2014. The final report focused on state, district, 

and school level implementation was released in September 2015. 

5. Key Findings: 

 At the state level, there was progress from 2009–10 to 2011–12 in each of the four areas 

of reform examined. Among the reforms measured, progress was greatest for SEA 

support for use of student achievement gains for principal evaluation (from 6 in 2009–10 

to 22 in 2011–12). However, in 2011–12 many more SEAs were carrying out reforms 

related to standards and assessments than were carrying out reforms related to 

educator evaluation and compensation or improving low-performing schools. 

 At the district and school levels, progress was uneven and varied by the reform area 

being examined. During the study period, there was an increase in the percentage of 

districts and schools that reported implementing standards-and-assessments-related 

reforms. However, there was a decrease in the percentage of districts that reported 

implementing reforms related to educator evaluation and compensation, and at the 

school level, implementation of these reforms largely remained flat. School-level but not 

district-level progress was seen for data-system-related reforms. 

 Across all levels, the most frequently reported reform implementation challenges were 

related to educator evaluation and compensation. The majority of SEAs reported 

difficulty measuring student growth for teachers of subjects with no standardized tests as 

a major challenge. Large percentages of districts and schools rated insufficient funding 

to provide performance-based compensation or differentiated compensation as major 

challenges. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154016/ 

Usage of Policies and Practices Promoted by Race to the Top and SIGs 

1. Study Purpose: RTT and SIG are signature Department programs in the Obama 

Administration, and both programs received substantial funding through the Recovery Act. 

As part of OMB’s FY 2010 Evaluation Initiative, IES proposed to conduct an impact 

evaluation of the RTT and SIG programs, focusing on the initial general state competition for 

RTT and the first cohort of SIG schools implementing intervention models beginning in the 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 178 

2010–11 school year. This is the first evaluation report of the Impact and Implementation 

Evaluation of RTT and SIG. 

2. Key Questions Addressed: 

 Which policies and practices promoted by the RTT program do RTT states report using, 

and how do they compare to the policies and practices that non-RTT states report 

using? 

 Are SIG-funded schools using the improvement strategies promoted by the four SIG 

intervention models, and how do they compare to strategies in schools not implementing 

a SIG-funded intervention model?

3. Design: The RTT sample includes all 49 states and DC. Data from spring 2012 interviews 

with all states and DC will inform the first key question. The SIG sample is purposively 

selected and includes about 470 schools in 60 districts from 22 states. Data from spring 

2012 state and district interviews, as well as spring 2012 school surveys from the SIG 

sample, will inform the second key question. 

4. Estimated or Actual Completion Date: The report was released in September 2015. 

5. Key Findings:

 Early (Round 1 and 2) RTT states used more policies and practices than non-RTT states 

in five of six areas examined: state capacity, standards and assessments, data systems, 

teachers and leaders, and charter schools (school turnaround was the exception). Later 

(Round 3) RTT states used more policies and practices than non-RTT states in just one 

area: teachers and leaders. 

 Use of policies and practices across RTT and non-RTT states was highest in the state 

capacity and data systems areas and lowest in the teachers and leaders area. 

 SIG schools used more practices than non-SIG schools in all four areas examined: 

comprehensive instructional reforms, teacher and principal effectiveness, learning time 

and community-oriented schools, and operational flexibility and support. 

 Use of practices among SIG and non-SIG schools was highest in the comprehensive 

instructional reforms area and lowest in the operational flexibility and support area. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154018/

Policy and Program Studies Service (PPSS)

Highly Qualified Teachers (HQT) Enrolled in Alternative Routes to Certification and 

Licensure 

1. Study Purpose: Title I of ESEA requires that all teachers of core academic subjects be 

highly qualified—meaning they have a bachelor’s degree, full state certification, and 

demonstrate expertise in the subject matter they teach—and that states work to ensure that 

poor and minority students are not taught by teachers who are inexperienced, out-of-field, or 

unqualified at higher rates than their peers. Under federal regulations (34 CFR § 200.56), 

the Department allows teachers who are enrolled in programs that provide alternate routes 

to certification or licensure to be considered highly qualified before meeting full state 

certification requirements. At the same time, alternate route programs have traditionally 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 179 

been used to address shortages, which are especially acute in urban areas, special 

education, and in certain content areas such as mathematics and science. This 

congressionally mandated report examines the extent to which students in four different 

subgroups are taught by teachers enrolled in alternative certification programs and classified 

as “highly qualified” under the ESEA: (1) students with disabilities, (2) English learners, 

(3) students in rural areas, and (4) students from low-income families. 

2. Key Questions Addressed: 

 What percentage of highly qualified teachers across the nation, in each state, and in 

each district are enrolled in an alternative certification program? 

 What percentage of highly qualified special education teachers across the nation, in 

each state, and in each district are enrolled in an alternative certification program? 

 What percentage of highly qualified Title III teachers across the nation, in each state, 

and in each district are enrolled in an alternative certification program? 

 What percentage of highly qualified teachers working in rural areas across the nation 

and in each state are enrolled in an alternative certification program? 

 What percentage of HQTs working in high-poverty areas across the nation and in each 

state are enrolled in an alternative certification program? 

3. Design: This report provides a descriptive analysis of data collected by the Department from 

states in summer and fall 2014 on the numbers of all teachers, special education teachers, 

and Title III teachers who are enrolled in alternative certification programs and deemed 

highly qualified. 

4. Actual Completion Date: The final report was released in June 2015. 

5. Key Findings: 

 While most states employed some HQTs who were enrolled in alternative route teacher 

preparation programs, these teachers made up a small proportion of total HQTs, both 

overall as well as for the four subgroups of HQTs examined in this study. 

 Overall, across 48 states and the District of Columbia, 1.5 percent of HQTs were 

enrolled in alternative route programs. The average percentage of HQTs enrolled in 

alternative route programs was 1.9 percent for special education teachers, 1.3 percent 

for Title III teachers, 2.3 percent for high-poverty school districts, and 1.3 percent for 

rural school districts. 

 A majority of the responding states reported that less than 1 percent of all HQTs were 

enrolled in alternative route programs. Four states reported that none of their HQTs were 

in alternative route programs, while 12 states had one or more HQT subgroups in which 

4 percent or more were enrolled in alternative route programs. 

 Most LEAs had no HQTs enrolled in alternative route programs. In the LEAs that did 

have such teachers, most had fewer than five such teachers, and they usually 

accounted for less than 4 percent of all HQTs in the LEA. 

 High-poverty school districts had higher percentages of HQTs enrolled in alternative 

route programs (2.3 percent), on average, than districts with low poverty rates 

(1.0 percent). 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 180 

 Across all rural school districts, there was a lower percentage of HQTs enrolled in 

alternative route programs than in all urban districts (1.3 percent vs. 2.2 percent, 

respectively). 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#tq

State and District Implementation of Education for Homeless Children and Youth 

Program 

1. Study Purpose: The Education for Homeless Children and Youth (EHCY) program provides 

grants to states and other grantees with the goal of ensuring that homeless children and 

youth have access to the same free, appropriate public education as do other children and 

youth. This study examined the implementation of the EHCY program at the state and 

school district levels based on surveys and analysis of extant data. 

2. Key Questions Addressed: 

 How do states allocate EHCY funds? 

 What are the roles and responsibilities of the state coordinator and district liaison? What 

services do districts provide to homeless children and youth? 

 How do states monitor and provide technical assistance to districts as part of the EHCY 

program? What technical assistance needs do state coordinators and district liaisons 

report? 

 What data do states and districts collect about homeless children and youth? 

 What do state coordinators and district liaisons perceive as barriers to school success 

for homeless children and youth? 

3. Design: The study surveyed all state coordinators for the ECHY program and a sample of 

448 district liaisons of EHCY districts for the 2010–11 school years. The study also 

examined state-submitted data on homeless children and youth. 

4. Actual Completion Date: The final report was released in February 2015. 

5. Key Findings:

 Sixteen states used a regional approach to suballocating EHCY funds; these 16 states 

accounted for 84 percent of all school districts that received EHCY funds and services. 

 State coordinators reported spending the most time on providing technical assistance to 

districts and coordinating with other organizations, while district liaisons reported 

spending the most time on identifying eligible students and ensuring that homeless 

students and families receive services. 

 District liaisons indicated that transportation, school supplies, and tutoring and 

supplemental instruction accounted for the largest expenditures of EHCY funds. 

 State coordinators were more likely to report using site visits and integrated monitoring 

visits in 2012 than in 1998, and the number who reported monitoring non-EHCY districts 

more than doubled. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 181 

 In addition to collecting required data such as homeless students’ achievement on state 

assessments, many states also collected other data such as graduation rates and 

attendance rates. 

 Barriers to school enrollment and attendance for homeless students that were most 

frequently identified by district liaisons were transportation and family or student 

preoccupation with survival needs. Other barriers included delays in obtaining school 

records and inability to complete school assignments because of the lack of an 

appropriate study area. 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#homeless

Evaluation of the Carol White Physical Education Program 

1. Study Purpose: The Carol M. White Physical Education Program (PEP) provides grants to 

school districts or community-based organizations (CBOs) to initiate, expand, and improve 

physical education for students in kindergarten through grade 12. In 2010, the Department 

revised PEP regulations to: include an increased focus on healthy eating habits and 

nutrition; require projects to conduct a needs assessment, and require that districts use four 

designated modules of the Centers for Disease Control’s School Health Index (SHI); use 

this assessment to develop project goals and plans to address identified weaknesses; 

encourage grantees to establish partnerships with community entities; encourage grantees 

to collect and use body mass index (BMI) data; and establish new performance measures 

and standard data collection methods. This report provides the results of the implementation 

study of the 76 projects (64 district-led and 12 CBO-led) funded by FY 2010 PEP grant 

awards—the first cohort of grantees under the revised PEP. 

2. Key Questions Addressed:

 What were the results of PEP grantees’ self-assessments of their physical activity, 

health, and nutrition policies and practices? 

 What physical activity and nutrition policy efforts did PEP grantees report? 

 What types of physical fitness and nutrition activities did PEP grantees report? 

 What role did community partnerships play in PEP projects? 

 What were PEP grantees’ experiences collecting and using BMI data?

 What implementation challenges and lessons learned did PEP grantees report? 

3. Design: This study used a mixed-methods research design that included surveys and case 

studies. Surveys of project directors were conducted in 2011 and 2013, the first and third 

years of the grant period. Case studies of five PEP projects (three districts and two CBOs) 

gathered more in-depth information from 59 interviewees regarding grantees’ experiences 

with community partners and BMI data collection. 

4. Actual Completion Date: The final report was released in February 2015. 

5. Key Findings: 

 Grantees most often reported weaknesses in their policies and programs related to the 

Health Education (43 percent) and Physical Education (32 percent) SHI modules. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 182 

 The greatest percentage of grantees made significant changes to physical activity 

policies concerning the use of a standards-based sequential physical education 

curriculum (59 percent) and policies involving recommendations or offers of physical 

activity through before- or after-school programs (55 percent). 

 About one-third of grantees made significant changes to nutrition policies aimed at 

reducing the availability of foods of minimal nutritional value, restricting the marketing of 

unhealthy foods at school, and requiring the adoption and implementation of strong 

nutritional standards for all foods sold and served in schools. 

 Grantees implemented more physical fitness, physical education, and nutrition-related 

activities during the PEP grant compared with before receipt of the grant, particularly in 

the areas of improving instruction, staff professional development, student engagement 

in physical activity, and curricula revisions. 

 The most frequently reported benefits of PEP partnerships were access to additional 

resources (reported by 90 percent of grantees with partnerships) and the ability to build 

on grantees’ own knowledge bases (86 percent). 

 Sixty-one percent of grantees that collected BMI data provided or planned to provide 

parents with information about their children’s BMI scores to help them take appropriate 

action. 

 The most common implementation challenge reported was executing the revised 

requirements for collecting data on the performance measures (83 percent). 

6. Link to Additional Information: 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html#safe

Summary of Performance Evaluations Expected During FY 2016 and 

FY 2017 

National Center for Education Evaluation and Regional Assistance

Literacy 

Early Childhood Language Development 

1. Study Purpose: Differences between the reading skills of disadvantaged children and their 

more advantaged peers have been measured nationally as early as kindergarten entry in 

the Early Childhood Longitudinal Study. The focus of this evaluation is the identification of 

classroom practices that are associated with improved student language development and 

comprehension. Such practices could be used in a future rigorous evaluation of these 

strategies. 

2. Key Question Addressed: 

 What classroom practices are associated with children’s development of language skills, 

background knowledge, and comprehension in prekindergarten through third grade? 

3. Design: The evaluation will analyze the relationships between the study’s observational 

measures of classroom practices and direct assessments of students collected during the 

2011–12 school year in 83 Title I schools. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 183 

4. Estimated or Actual Completion Date: The final report is expected in 2016. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/literacy_lang.asp 

Summary of Research Generated by Striving Readers on the Effectiveness of 

Interventions for Struggling Adolescent Readers

1. Study Purpose: Striving Readers was a discretionary grant program focused on raising 

reading achievement of middle school and high school students through intensive 

interventions for struggling readers and enhancing the quality of literacy instruction across 

the curriculum. The 2006 and 2009 cohorts of grantees were required to conduct rigorous, 

experimental evaluations. The Institute of Education Sciences (IES) provided technical 

assistance to support grantees in conducting evaluations that would meet What Works 

Clearinghouse (WWC) standards. This study also conducted a systematic review of the 

17 evaluations of ten different interventions funded by Striving Readers across the 

16 grants. 

2. Key Question Addressed: 

 What do the evaluations funded by the Striving Readers program tell us about effective 

interventions for improving the achievement of struggling adolescent readers? 

3. Design: The study is descriptive; it provided technical assistance to the local evaluations of 

Striving Readers grantees and also reviewed the completed evaluations against the WWC 

evidence standards. The report summaries the results of this systematic review of the 

evaluations. 

4. Estimated or Actual Completion Date: The final report was released on October 27, 2015. 

5. Key Findings: 

 Fifteen of the 17 evaluations of the interventions funded by the Striving Readers grant 

program met WWC evidence standards with or without reservations. 

 Based on findings from the 15 evaluations that met WWC evidence standards with or 

without reservations, four of the ten interventions funded by the Striving Readers grant 

program had positive, potentially positive, or mixed effects on reading achievement. 

 The Striving Readers studies not only expand the evidence base on effective reading 

interventions for adolescents but also increase the number of high quality effectiveness 

studies reviewed by the WWC on this topic. Three of the four interventions that had 

positive, potentially positive, or mixed effects on reading achievement had not previously 

been reviewed by the WWC. 

6. Link to Additional Information: 

http://ies.ed.gov/ncee/projects/evaluation/assistance_readers.asp

Pathways to Career or College 

Enhancing Advising to Improve College Fit in Upward Bound: Impacts on College 

Applications and Enrollment Intentions 

1. Study Purpose: Growing concern over college enrollment and completion rates has 

heightened interest in cost-effective strategies to improve the outcomes of low-income 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 184 

students, including those in high-profile college access programs. One aspect of this 

concern is that many low-income high school students fail to enroll in colleges that are well 

aligned with their qualifications, talents, and needs. This report provides the early impacts of 

a low-cost enhancement to current college advising approaches in Upward Bound that is 

designed to improve college fit and persistence. The approach includes professional 

development for Upward Bound advisors and user-friendly packets of information for 

students that demonstrate the value of considering a range of institutional indicators in their 

college application, search, and acceptance process.

2. Key Questions Addressed: 

 Can an enhanced college advising approach improve upon what Upward Bound 

grantees are already doing—i.e., does it have positive effects on the number of colleges 

to which Upward Bound participants apply, the quality/selectivity of the colleges in which 

they enroll, and their persistence? 

 In what types of grantees is this approach most effective and with what types of 

students? 

3. Design: About 200 Upward Bound grantees that volunteered were randomly assigned in 

spring 2015 so that half received the professional development and packets to begin using 

with their rising 2015–16 seniors (the treatment group) and half will receive the training and 

packets a year later to use with rising 2016–17 seniors (the control group). A survey 

administered in spring 2016 to seniors in both sets of projects will collect information about 

their college planning including the number of applications and acceptances. The study will 

also examine those seniors’ later enrollment and persistence in college. 

4. Estimated or Actual Completion Date: The first report on early impacts is expected in late 

2017.

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/pathways_upward.asp 

Is Access to Data Enough? The Effects of Providing Information on Student FAFSA 

Completion Directly to School Districts 

1. Study Purpose: Completing a Free Application for Federal Student Aid (FAFSA) is a key but 

challenging task for many students applying to college, particularly for low-income students. 

The Department has responded to concerns about the complexity of the FAFSA, including 

through the FAFSA Completion Project which started with a 20-district pilot in 2010 and 

expanded in 2012 to almost 100 more. One component of the project was consistent urging 

by federal officials and some technical assistance for participating districts to help them 

conduct FAFSA promotion and outreach activities districtwide. But the Department also 

provided districts with ongoing and direct access to federal data on which seniors had not 

submitted a FAFSA so that schools could provide targeted support to individual students. 

The evaluation tested whether that data access had an impact on FAFSA completion rates 

above and beyond the other FAFSA activities the districts might have implemented. The 

study was conducted by IES staff and designed to be low burden, and low-cost, depending 

only on the Department’s data for measuring effectiveness.

2. Key Question Addressed: 

 Does making the student-level data available improve FAFSA completion rates? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 185 

3. Design: Just over 60 districts that had more than one high school participated in the 

evaluation, for a total of 567 schools. IES randomly assigned the schools within each district 

to either an “early start” (treatment) group, which would have access to the FAFSA 

completion status for 2013 seniors, or a “later start” (control) group, which would have 

access for 2014 seniors. Because many districts had relatively small numbers of schools, 

pairs of schools were matched prior to random assignment to help ensure that important 

characteristics of schools were balanced between the treatment and control groups. At the 

beginning of the evaluation, districts provided the lists of all seniors in each school and 

these were matched against the Department’s federal student aid database so that FAFSA 

completion rates in March 2013 and September 2013 could be compared between the two 

groups of schools. 

4. Estimated or Actual Completion Date: The report is expected in early 2016. 

Upward Bound at 50: Reporting on Implementation Practices Today 

1. Study Purpose: Upward Bound is the oldest and largest of the federal college access 

programs targeted to low-income students and those who would represent the firstgeneration of college completers in their families. First established in 1965, the program 

currently serves more than 60,000 high school students at a cost of about $4,300 per youth 

with a wide array of academic and college transition support services. While much about the 

structure of Upward Bound and the services to be offered are prescribed in legislation, little 

is currently known about the intensity, duration, and mix of services provided by projects or 

about how they are delivered. Because of the importance of its mission, and the 

comprehensiveness and costs of its services, Upward Bound has long been of interest to 

policymakers. This report describes the approaches to providing program services as 

reported by Upward Bound project directors. 

2. Key Questions Addressed: 

 Within the core service areas of the program, where do projects focus their efforts? 

 How are services delivered to Upward Bound participants? 

 In what ways does the focus or delivery of services vary across different Upward Bound 

projects? 

3. Design: This descriptive report relies primarily on findings from a summer 2013 survey of all 

regular Upward Bound project directors. 

4. Estimated or Actual Completion Date: The report is expected in early 2016. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/pathways_trio.asp 

School Choice 

Evaluation of the DC Opportunity Scholarship Program: An Early Look at Applicants and 

Participating Schools Under the SOAR Act 

1. Study Purpose: The April 2011 SOAR Act provided for a five-year continuation of a school 

choice program for low-income residents of Washington, DC. The program, still titled the 

Opportunity Scholarship Program (OSP), now provides annual scholarships of $8,000 (for 

grades K–8) or $12,000 (for grades 9–12) to enable low-income students to attend private 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 186 

schools in DC in lieu of the public schools already available to them. The new law also 

mandated another independent, rigorous evaluation of the effectiveness of the program. 

2. Key Questions Addressed:

 What is the impact of the OSP on student academic achievement and other measures of 

student success, overall and for subgroups of students identified in the statute as high 

priority?

 What effect does the program have on student and parent perceptions of school safety 

and satisfaction, and on parents’ involvement in education of their children? 

 Does the program change students’ instructional environments and opportunities? 

3. Design: The evaluation will primarily compare outcomes of approximately 1,800 student 

applicants randomly assigned by lottery to either receive a scholarship or not receive a 

scholarship. Lotteries of program applicants were conducted in spring 2012 (cohort 1), 

spring 2013 (cohort 2), and spring 2014 (cohort 3). Data will be collected for three follow up 

years for each of the cohorts and, for students in both the scholarship and nonscholarship 

groups. The contractor will administer academic assessments, and conduct student, parent, 

and principal surveys each spring (spring 2013–16). Prior to the first impact report, 

descriptive reports based on application forms parents filled out when they applied to the 

OSP, principal surveys, and extant data will be released. 

4. Estimated or Actual Completion Date: The first descriptive report—describing the 

characteristics of participating schools and program applicants—was released in October 

2014. The second descriptive report is scheduled for release in late 2015 and the third in 

spring 2016. The first impact report is expected in summer 2016. 

5. Key Findings from the First Report:

 Just over half of all DC private schools participated in the OSP, with current schools 

more likely to have published tuition rates above the OSP scholarship amounts than did 

participating schools in the past. 

 OSP applicants under the SOAR Act represent between 3 and 4 percent of the 

estimated 53,000 children in DC who meet the eligibility criterion. 

 A number of awarded scholarships go unused, with students from disadvantaged 

schools and families using awarded scholarships at lower rates than others. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154000/pdf/20154000.pdf 

Parent Information and School Choice Evaluation 

1. Study Purpose: School choice is among the most visible K-12 education policy trends to 

emerge over the last two decades. For school choice to be effective, it seems critical that 

parents are able to navigate school choice systems and process large amounts of complex 

information about schools and application procedures to make informed choices. However, 

few studies have rigorously examined school choice information strategies or attempted to 

identify effective methods of information presentation. This evaluation is designed to 

address this significant gap in the literature. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 187 

2. Key Questions Addressed:

 What is the optimal way to present school choice information? (E.g., which presentation 

formats make school performance/achievement data easiest to understand? How should 

transportation options and location be conveyed?) 

 What is the right amount of information to provide to parents whom might be making a 

school choice? (E.g., is there a limit to the number of choice options and amount of 

information that parents can reasonably consider?) 

 How is school choice information best organized? (E.g., does the order in which 

information is presented shape how it is understood and used? Is there a way to make 

the information parents most often report caring about salient?) 

3. Design: The evaluation findings will be based on low cost quick turn-around “lab” based 

experiments. These studies will take advantage of opportunistic samples of low income 

parents in order to test a large number of potentially promising information strategies and to 

obtain findings within a relatively short period of time. 

4. Estimated or Actual Completion Date: A short user-friendly guide, based on the evaluation’s 

findings, that will help school districts and other providers of school choice information 

design or refine their own parent information materials is expected in October 2017. 

Students with Disabilities 

Evaluation of Response to Intervention Practices for Elementary School Reading 

1. Study Purpose: The focus of this evaluation is the implementation and impact of Response 

to Intervention (RtI) practices for struggling readers in elementary school. Response to 

Intervention (RtI) is a multistep approach to providing early and more intensive intervention 

and monitoring within the general education setting. IDEA permits some Part B special 

education funds to be used for “early intervening services” such as RtI, and also permit 

districts to use RtI to inform decisions regarding a child’s eligibility for special education 

under the category of specific learning disabilities. 

2. Key Questions Addressed:

 How do RtI practices for early-grade reading vary across schools? 

 How do schools experienced with RtI vary the intensity of reading instruction to children 

based on student benchmark reading performance? 

 What are the effects on grade 1–3 reading achievement of providing intensive 

interventions to children who are on the margin of identification for reading difficulties? 

3. Design: The evaluation is relying on a combination of descriptive data collection from school 

staff and regression discontinuity methods to address the research questions, and is 

focusing on practices in place during the 2011–12 school year. 

4. Estimated or Actual Completion Date: The final report was released in November 2015.
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 188 

5. Key Findings:

 Full implementation of the RtI framework in Grade 1–3 reading was reported by 86 

percent of the experienced elementary schools sampled for in-depth study, compared 

with 56 percent of a random sample of elementary schools from the same 13 states. 

 The schools sampled for in-depth study of their RtI practices provided more intense 

small-group instruction and more reading intervention services for groups of students 

reading below grade level than for groups of students reading at or above grade level. 

However, these schools’ implementation of RtI differed in some ways from descriptions 

in the prior literature: 

o In Grade 1, 45 percent of schools provided intervention services to some groups of 

students at all reading levels, rather than only for reading groups below grade level. 

o In Grade 1, 67 percent of schools provided at least some reading intervention during 

the core reading block, rather than only in addition to the core. 

 Grade 1 students who scored just below school-determined benchmarks on fall 

screening tests, and who were assigned to reading interventions, had lower spring 

reading scores than similar students in the same schools who were not assigned to 

those interventions. Unexplored, but plausible, factors that may be related to these 

negative effects include incorrect identification of students for intervention and a 

mismatch between the interventions provided and these students’ instructional needs. 

No statistically significant effects of assignment to reading interventions were found for 

students in Grades 2 or 3. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/disabilities_rti.asp

National Evaluation of the IDEA Technical Assistance and Dissemination Program: 

Interim Report 

1. Study Purpose: IDEA, which was most recently reauthorized in 2004, provides funds to 

assist states and local eduational agencies in making available a free, appropriate public 

education for children with disabilities. Funded at $12.6 billion in FY 2010, IDEA supports 

early intervention services for infants and toddlers, special education and related services 

for children ages 3 through 21, and early intervening services for students not in special 

education but in need of academic or behavioral support. IES is conducting studies under 

Section 664 of IDEA to assess the implementation and effectiveness of key programs and 

services supported under the law. 

As specified in IDEA Part D, the Technical Assistance and Dissemination (TA&D) program is 

to provide technical assistance, support model demonstration projects, disseminate useful 

information, and implement activities that are supported by scientifically based research to 

meet the needs of children with disabilities. The National Evaluation of the IDEA TA&D 

program is designed to describe the products and services provided by the TA&D program 

grantees, state and local needs for technical assistance, and the role that the TA&D 

program plays in meeting these needs and supporting implementation of IDEA. 

2. Key Questions Addressed:

 What are the primary technical assistance activities of the TA&D program? 

 What are states’ and other stakeholders’ needs for technical assistance and to what 

extent are these needs addressed by TA&D centers or other sources? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 189 

 To what extent are states and other stakeholders satisfied with the products and 

services received from TA&D program centers? 

3. Design: Data collection for the interim report included administering surveys to the national 

and regional level TA&D program grantees, all state IDEA Part B and Part C administrators, 

and a sample of state-level special education program staff. Data were collected between 

November 2012 and March 2013. State-level administrators and staff reported on their 

receipt of techincal assistance from TA&D program grantees during the 2010–11 school 

year. Grantees reported on their goals and activities from the beginning of their current grant 

through the interview date. The funding period for the centers included in the interim report 

varied, with the earliest end date in 2012 and the latest in 2014. Data collection for the final 

report included administering surveys to the 48 state level deaf-blind projects awarded 

grants in 2013 and to a sample of local level special and general education providers 

currently working with deaf-blind children. 

4. Estimated or Actual Completion Date: An interim report was released in October 2013. The 

final report is expected in October 2016. 

5. Key Findings from the Interim Report:

 TA&D centers most commonly reported providing technical assistance on the topics of 

parent and family involvement and data systems and use of data for improvement. 

States identified (a) “General Supervision/Monitoring,” (b) “early childhood transition,” 

(c) “financing of services/financing for special education,” and (d) “Response to 

Intervention” as the topics for which they had the greatest need for technical assistance 

in 2010–11.

 Many TA&D centers provide technical assistance on similar topics. For example, 

14 states received “high intensity” technical assistance (i.e., frequent training or 

consultation) on the same topic from 5 different centers. This evaluation was unable to 

establish whether such cases are indicators of inefficiency or of complementary and 

coordinated services. 

 State staff rated the majority of technical assistance experiences they had with TA&D 

centers as “very satisfactory” (71 percent). On average, customers receiving high 

intensity technical assistance were significantly more satisfied than those receiving lower 

intensity (i.e., infrequent training and consultation or web-only support). Satisfaction did 

vary to some degree depending on the special education topic being addressed. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144000/pdf/20144000.pdf 

Preparing for Life After High School 

1. Study Purpose: Phase I of the National Longitudinal Transition Study 2012 (NLTS 2012), 

also referred to as the Study of Transition Outcomes for Youth with Disabilities Phase I, is 

the third in a series examining the characteristics and school experiences of a nationally 

representative sample of youth with disabilities. NLTS 2012 focuses on youth ages 13 to 21 

(in December 2011), but also includes a small sample of students without disabilities to 

enable direct comparisons of students with and without individualized education programs 

(IEPs). It is part of the congressionally mandated National Assessment of the IDEA and is 

supported with funds authorized under Section 664 of IDEA. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 190 

2. Key Questions Addressed:

 What are the personal, family, and school characteristics of youth with disabilities in 

public schools across the country? 

 What regular education, special education, transition planning, and other relevant 

services and accommodations do youth with disabilities receive? 

 How do the services and accommodations differ from those of youth not served under 

IDEA, including those identified for services under Section 504 of the Rehabilitation Act? 

 How do the services and accommodations for youth with disabilities vary with the 

characteristics of youth? 

 How much have the services and accommodations of youth with disabilities changed 

over time? 

3. Design: Phase I of the study collected survey data (spring 2012–summer 2013) on a 

nationwide sample of 12,000 youth in school, of which 10,000 are students with IEPs across 

the federal disability categories. Surveys of both youth and their parents/guardians were 

administered. 

4. Estimated or Actual Completion Date: A report assessing existing research on the 

effectiveness of transition strategies for youth with disabilities was released August 2013. 

Three reports describing the survey results are expected in summer 2016. 

5. Key Findings from the First Report: A systematic review of the evidence on transition 

strategies was conducted, to inform development of the surveys and interpretation of the 

survey results. The review found: 

 Only 16 studies were designed and carried out with sufficient rigor to meet IES’ WWC 

standards. 

 Based on these studies, community-based work programs were found to have mixed 

effects on students’ employment outcomes and potentially positive effects on 

postsecondary education outcomes. Functional life skills development programs were 

found to have potentially positive effects on independent living outcomes although the 

extent of evidence was small. 

6. Link to Additional Information: 

http://ies.ed.gov/ncee/projects/evaluation/disabilities_ideatrans.asp 

Preschool Special Education Programs and Practices Supporting Children with 

Disabilities 

1. Study Purpose: Most recently authorized in 2004, IDEA provides funding to states to support 

special education and related services for children and youth with disabilities, including 

young children ages 3–5 years old. Currently there is limited information available on the 

curricula and interventions being used across states to support young children with 

disabilities. Phase I of the Evaluation of Preschool Special Education Practices is being 

conducted to address the primary objective of assessing the feasibility of conducting a largescale impact study of curricula or interventions promoting the literacy, language, and/or 

social-emotional skills of preschool-age children with disabilities. A secondary objective of 

the Phase I study is to provide nationally representative descriptive information about 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 191 

preschool special education programs and the specific curricula or interventions being 

delivered to preschool children with disabilities. 

2. Key Questions Addressed: 

 Which curricula and interventions are used nationally for preschool children with 

disabilities to promote learning of language, literacy, and social emotional skills? 

 How are states and school districts structuring programs to serve children ages 3 

through 5 with disabilities? 

 What staff resources are available to support the instruction of children ages 3 through 5 

with disabilities? 

3. Design: The study collected survey data from state and district grantees of IDEA funds to 

obtain nationally representative information on the programs, services, curricula and 

interventions available to children ages 3–5 identified for special education services. The 

survey sample included a nationally representative sample of 1,200 school district preschool 

special education coordinators and state Section 619 coordinators in all 50 states and DC. 

Additionally, an evidence review will be conducted to identify preschool curricula and 

interventions focused on improving literacy, language, and social emotional skills of 

preschool-aged children with disabilities for the design of impact study options. If an impact 

study is feasible to conduct, preparations for the impact study will occur under Phase I. 

4. Estimated or Actual Completion Date: A descriptive report on the survey findings is 

expected in 2017. 

5. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/disabilities_preschool.asp 

Study of Early Intervention and Special Education Services and Personnel 

1. Study Purpose: IDEA was most recently authorized in 2004 to promote a free appropriate 

public education for children with disabilities. Funded at $12.5 billion in FY 2015, IDEA 

supports early intervention services for infants and toddlers (IDEA Part C), special education 

and related services for children ages 3 through 21 (IDEA Part B), and early intervening 

services for students not in special education but in need of academic or behavioral support. 

Under Section 664 of IDEA 2004, IES is conducting studies to assess the implementation 

and effectiveness of key programs and services supported under the law. This study is 

supporting the analysis of extant data to examine early intervention and special education 

service delivery and the personnel providing services. 

2. Key Questions Addressed:

 How does early intervention service delivery vary across states? 

 How do special education and related services received by children and youth vary over 

time, across states, and by school characteristics? 

 How does the distribution of personnel providing special education services vary over 

time, across states, and by school characteristics? 

3. Design: This study includes new descriptive analysis of extant data available in public use or 

restricted formats. Among the data sources that are being used are cross-sectional Section 

618 data submitted by states to the Department and the NCES Schools and Staffing Survey. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 192 

4. Estimated or Actual Completion Date: The report is expected in summer 2016. 

5. Link to Additional Information: 

http://ies.ed.gov/ncee/projects/evaluation/disabilities_persserv.asp 

Teacher Effectiveness 

Elementary School Math Professional Development Impact Evaluation 

1. Study Purpose: Title II, Part A of the current ESEA provides monetary resources to improve 

teacher quality through professional development (PD). Currently there is relatively little 

rigorous empirical evidence on the components of an effective PD program. At the same 

time, there has been renewed policy interest in fostering a globally competitive STEM 

workforce, which will require teachers to effectively impart mathematics to their students. 

This study will present implementation and impact findings from a random assignment 

evaluation of an intensive math PD intervention focused on enhancing teacher content 

knowledge and integrating this knowledge into the classroom. 

2. Key Questions Addressed:

 What is the average impact on (a) teachers’ content knowledge, (b) teachers’ classroom 

practices, and (c) student achievement of offering a specialized professional 

development intervention (in this case, Intel Math, Mathematics Learning Communities, 

and Video Feedback Cycles) relative to “business-as-usual” professional development? 

 How is the professional development intervention implemented? 

3. Design: Approximately 200 volunteer fourth-grade math teachers from 6 districts around the 

country participated in this study during the 2013–14 school year. The intervention has three 

components: i) Intel Math (80 hours largely focused on developing teachers’ understanding 

of K–8 mathematics), ii) Mathematics Learning Communities (which offers teachers 

opportunities to connect their learning to the classroom through a professional learning 

community aligned to Intel Math that focuses on student thinking), iii) Video Feedback 

Cycles (which provide teachers with individualized feedback three different times based on 

videotaped lessons of their classroom teaching reviewed by certified analysts using the 

Mathematical Quality of Instruction (MQI) rubric). Within each participating school, fourthgrade teachers were randomly assigned to receive either the PD intervention or businessas-usual PD. Data collection includes measures of teacher knowledge and practice, and 

student achievement obtained from administrative records. 

4. Estimated or Actual Completion Date: The report is expected in 2016. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_mathpd.asp

Evaluation of the Teacher Incentive Fund 

1. Study Purpose: The Teacher Incentive Fund (TIF) is authorized by Title V, Part D of ESEA 

and annual appropriations acts. The purpose of the TIF program is to develop and 

implement performance-based compensation systems (PBCSs) for teachers, principals, and 

other personnel in high-need schools. Research indicates that high quality teachers are 

critical to raising student achievement in low-performing schools, but schools most in need 

often have difficulty in attracting and retaining high-quality teachers. Performance pay is a 

policy promoted by the TIF program to improve the quality of teachers in high-need schools. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 193 

This evaluation studies performance pay that provides substantial and differentiated bonus 

pay to high-performing teachers in low-performing schools with high-need students. 

2. Key Questions Addressed:

 What are the characteristics of all TIF grantee districts and their performance-based 

compensation systems? What implementation experiences and challenges did TIF 

districts encounter? 

 How do teachers and principals in schools that did or did not offer pay-for-performance 

bonuses compare on key dimensions, including their understanding of TIF program 

features, exposure to TIF activities, allocation of time, and attitudes toward teaching and 

the TIF program? 

 What is the impact of pay-for-performance bonuses on students’ achievement on state 

assessments of math and reading? 

3. Design: Study schools were randomly assigned within a grant to either implement all 

components of the PBCS or the PBCS with a 1 percent across-the-board bonus in place of 

the differentiated effectiveness incentive component of the PBCS. Data collection will 

include a grantee survey, a survey of teachers and principals, teacher and principal school 

assignment records, student record information (such as student demographics and student 

test scores), and grantee interviews to document implementation information, as well as to 

conduct impact analyses. 

4. Estimated or Actual Completion Date: The first report was released in September 2014. The 

second report was released in September 2015. The third and fourth reports, which will 

include updated impacts, are expected in 2016 and 2017. 

5. Key Findings from the Second Report:

The main findings among all TIF districts with 2010 awards are: 

 Ninety percent of all TIF districts in 2012–13 reported implementing at least 3 of the 

4 required components for teachers, and only about one-half (52 percent) reported 

implementing all four. This was a slight improvement from the first year of 

implementation. 

 Near the end of the second year of implementation, 65 percent of TIF districts reported 

that sustainability of the program was a major challenge. In contrast, no more than onethird of districts reported that other activities related to their program were a major 

challenge. 

For the subset of 10 districts that agreed to participate in a random assignment study, key 

findings on the effect of pay-for-performance on educators include the following: 

 Few evaluation districts structured pay-for-performance bonuses to align well with TIF 

guidance in the grant competition notice. 

 Educators’ understanding of key program components improved from the first to the 

second year, but many teachers still did not understand that they were eligible for a 

bonus. They also continued to underestimate how much they could earn from 

performance bonuses. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 194 

 Offering educators pay-for-performance had small, positive impacts on their students’

reading achievement; impacts on students’ math achievement were not statistically 

significant but similar in magnitude. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_incentive.asp

Impact Evaluation of Teacher and Leader Performance Evaluation Systems 

1. Study Purpose: Through the Race to the Top and Teacher Incentive Fund grant programs, 

as well the ESEA Flexibility waivers, the Department has emphasized the policy of 

evaluating educator effectiveness and providing educators useful and timely feedback 

needed to improve practice and, ultimately, student achievement. The current study is an 

intervention study designed to examine the implementation and impacts of a package of 

performance evaluation system components that include features that are currently 

promoted in policy discussions. The evaluation system components include measures of 

student achievement growth, classroom observations and measures of principal leadership. 

2. Key Questions Addressed:

 What is the impact of the performance evaluation system on student achievement? On 

teachers’ classroom practices? On teacher mobility? 

 What are districts’ and educators’ experiences with implementation? 

3. Design: Eight districts participated in the study. Within each district, a subset of 

approximately 15 schools were randomly assigned to receive the study’s treatment during 

2012–13 and 2013–14 or to participate only in the district’s usual performance evaluation 

system during the same time period. In treatment schools, each year, teachers received four 

rounds of classroom observations and feedback sessions, based on the Framework for 

Teaching or the CLASS, depending on district preference. Teachers also received valueadded scores in math and/or reading, when possible. Principals in study schools received 

feedback on their performance based on the VAL-ED, which was administered twice per 

study year. Data were collected on teacher and principal professional development 

experiences during 2012–13 and 2013–14, as well as their perceptions of the usefulness of 

performance feedback received. The study also collected student achievement data from 

district records for 2012–13 and 2013–14 and conducted classroom observations in 2013–

14. Teacher mobility and retention were measured through fall 2014. Implementation data 

collected include measures such as the frequency with which teachers were observed and 

received feedback, and the percentage of teachers who participated in a 360-degree 

assessment that is part of the principal performance evaluation system, in each of the two 

implementation years. The study will also look at the characteristics of the observation 

measures, VAL-ED ratings, and value-added scores, including the extent to which the 

measures differentiate performance. 

4. Estimated Completion Date: A report on the study’s first year of implementation is expected 

in 2016. A report on the study’s second year of implementation and impacts is expected in 

2017.

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_performance.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 195 

Study of the Distribution of Effective Teaching 

1. Study Purpose: Recent Department of Education initiatives, such as Race to the Top, the 

Teacher Incentive Fund, and the ESEA Flexibility policy, are designed in part to ensure that 

disadvantaged students have equal access to effective teaching. This study provides 

information about the extent that disadvantaged students receive less effective teaching 

than other students. The study also examines teacher mobility in participating districts and 

how patterns of mobility might contribute to unequal access. 

2. Key Questions Addressed:

 To what extent do disadvantaged students have equal access to effective teaching 

within school districts, and how does this change over time? 

 Is access to effective teaching related to different patterns of teacher hiring, retention, 

and mobility for high- and low-poverty schools? 

3. Design: The study is descriptive. It documents the distribution of effective teaching, as 

measured by value added, and changes in the distribution of effective teaching across the 

2008–09 through 2012–13 school years. The study also describes district polices designed 

to address inequitable distribution of effective teaching implemented during those years. 

Lastly, the study will examine teacher mobility patterns within participating districts. Data 

collection included the annual collection of district administrative records, including student 

achievement, to conduct value-added analyses as well as annual semi-structured interviews 

with district leadership to provide information on district policies. Data collection also 

included district personnel data to examine teacher mobility within participating districts. The 

study will be conducted in 29 geographically dispersed school districts. 

4. Estimated or Actual Completion Date: The first report on school years 2008–09 through 

2010–11 was released in November 2013. The final report on school years 2008–09 

through 2012–13 is expected in 2016. 

5. Key Findings from the First Report:

 The first report focused on access to effective teaching in 29 geographically dispersed 

school districts and found that: 

 On average, disadvantaged students had less access to effective teaching than 

relatively more advantaged students. Providing equal access to effective teaching for 

disadvantaged and other students would reduce the student achievement gap from 

28 percentile points to 26 percentile points in English/language arts (ELA) and from 

26 percentile points to 24 percentile points in math in a given year. 

 Access to effective teaching patterns for disadvantaged students were similar over the 

three years studied, 2008–09 through 2010–11. 

 Access to effective teaching varied across study districts. Access ranged from districts 

with equal access to districts with differences in access as large as 0.106 standard 

deviations of student test scores in ELA and 0.081 standard deviations of student test 

scores in math, favoring relatively more advantaged students. 

 Access to effective teaching was more related to the school assignment of students and 

teachers than the way that students were assigned to teachers within schools. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/tq_distribution.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 196 

Study of Teacher Preparation Experiences and Early Teacher Effectiveness 

1. Study Purpose: A primary focus of Title II, Part A, the Improving Teacher State Formula 

Grants program, is on the improvement of teacher quality Little research exists, however, to 

inform how best to prepare teachers for the classroom. The current study explores the 

variation in teachers’ preparation experiences. If feasible, it will examine whether the 

instructional skills that teachers learn about and have opportunities to practice in their 

preparation programs are associated with novice teachers’ effectiveness in the classroom. 

2. Key Questions Addressed:

 To what extent do teachers report participating in various preparation experiences? How 

much variation is there within and between programs? 

 Among novice teachers, what are the relationships between the teachers’ experience in 

their preparation programs and the achievement of students in their classroom?

3. Design: Approximately 3,200 novice teachers from grades 4 through 6 who are responsible 

for language arts and/or math instruction participated in the study. The study defines novice 

teachers as those in their first three years in the profession. In spring 2015, the teachers 

responded to a survey focused on their preparation experiences related to instructional skills 

that have been shown to be associated with teacher value-added scores. For each 

instructional skill, the teachers were asked about the nature of their learning experiences. If 

feasible, the study will compute value-added scores for teachers, based on students’ state 

math and English language arts tests, and examine the relationships between teacher 

preparation experiences and teacher value-added scores. Among novice teachers who have 

five or more English language learners in their classroom, the study will also examine the 

relationships between these teachers’ preparation experiences and teacher value added 

scores based on achievement of their English language learners, if feasible.

4. Estimated Completion Date: The report on this study is expected in 2017. 

5. Link to Additional Information: 

http://ies.ed.gov/ncee/projects/evaluation/tq_teacherprep_early.asp

Other 

Case Studies of Schools Receiving SIGs: Final Report 

1. Study Purpose: SIGs are authorized by Title I, Section 1003(g) of ESEA. The purpose of the 

grants—awarded based on the Title I funding formula to states, which then competitively 

distribute the funds to districts applying on behalf of their eligible schools—is to support the 

turnaround of the nation’s persistently lowest-achieving schools. To qualify for the three-year 

grant, schools must (among other requirements) be willing to implement one of four 

prescribed intervention models: turnaround, restart, closure, or transformation. About $546 

million was allocated in FY 2009 for SIG with a supplement of $3 billion from the Recovery 

Act. With the possibility of rollover funds, this amounts to a $3.5 billion injection into the SIG 

program during the 2010–11, 2011–12, and 2012–13 school years. This study will provide 

descriptively rich, primarily qualitative information for a small set of schools receiving SIG in 

the first cohort to implement an intervention model beginning in the 2010–11 school year. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 197 

2. Key Questions Addressed:

 What is the background and context of these persistently lowest-achieving schools? 

How do the leadership and staff in these schools define the performance problem, and 

to what do they attribute their problems? 

 What leadership styles do the principals of these persistently lowest-achieving schools 

exhibit? What actions do these schools engage in to try to improve their history of low 

performance? 

 What is the change process in these persistently lowest-achieving schools, particularly in 

terms of school climate and staff capacity? 

 What improvements do school stakeholders perceive during the three-year course of 

SIG, and do these improvements appear to be sustainable? 

3. Design: This study employs a school-level case study design. A core sample of 25 SIG 

schools was purposively selected from six states to represent a range of geographic 

regions, urbanicities, school sizes, racial/ethnic compositions, socioeconomic statuses, SIG 

intervention models, and SIG funding levels, among other factors. Data collection took place 

over three school years, beginning in spring 2011 and concluding in spring 2013, and 

included interviews with each state’s SIG leaders, a teacher survey, and site visits to the 

case study schools, which included analysis of fiscal records, as well as interviews and 

focus groups with district officials, principals, teachers, parents, union officials, external 

support providers, and students. 

4. Estimated or Actual Completion Date: Two descriptive reports were release in May 2011 

and October 2012. A report on the first year of SIG implementation was released in May 

2014. The final report is expected in spring 2016. 

5. Key Findings from the First Report: 

 The 25 schools differed in their community and fiscal context, their performance and 

reform history, and their interpretation of the causes of—and potential solutions for—

their performance problems. 

 Approaches to leadership varied across the 25 schools, with most principals exhibiting a 

mix of transformational, instructional, and strategic leadership qualities. 

 The 25 schools identified 11 improvement strategies and actions, most often increasing 

professional development activities, replacing the principal, and increasing learning time. 

Most of the schools did not perceive SIG as the primary impetus for change: just 7 of the 

25 schools experienced a disruption from past practice, and in 19 of the 25 schools, the 

improvement strategies and actions in the first year of SIG were a continuation of 

activities or plans that predated SIG. 

 All but one of the 25 schools perceived improvement in at least some areas, most often 

a safe and orderly school climate, and teacher collaboration. Schools that perceived the 

most improvements were more likely to have experienced a disruption from past 

practice, and to have principals with higher levels of strategic leadership. 

6. Link to Additional Information:

http://ies.ed.gov/ncee/projects/evaluation/other_schoolturnaround.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 198 

Evaluation of Investing in Innovation 

1. Study Purpose: Investing in Innovation (i3) is a competitive grant program that funds 

entitites with a record of improving student achievement in order to expand the 

implementation of, and investment in, evidence-based interventions to significantly improve 

student education outcomes. The i3 program supports three types of grants that are 

differentiated by level of prior evidence supporting the intervention, the scope of the 

proposed implemnetation, and the amount of funding to be provided. Scale-up and 

Validation grants require prior evidence of effectiveness, while Development grants support 

innovative strategies. 

Grantees receiving funds under this program are required to conduct an independent 

evaluation of their project and must agree, along with their independent evaluator, to 

cooperate with evaluation technical assistance provided by the Department and its 

contractor. The purpose of this technical assistance is to maximize the strength of the 

impact studies and the quality of their implementation data and performance feedback. 

2. Key Questions Addressed:

 What types of supports were provided to i3 local evaluators and how were these 

supports implemented? 

 Were the i3-funded evaluations well-designed and well-implemented? 

 What did the i3-funded evaluations add to the evidence base on educational 

interventions? 

3. Design: This study is descriptive and includes technical assistance to support the i3 local 

evaluators. The study also includes a systematic review of the completed evaluations 

against the WWC evidence standards as well as other criteria for assessing the strength of 

the impact and implementation aspects of the evaluations. 

4. Estimated or Actual Completion Date: The first report, which and will focus primarily on 

findings for the first cohort (FY 2010), is expected in 2016. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/assistance_ita.asp 

Evaluation of NAEP Achievement Levels 

1. Study Purpose: Under the provisions of P.L. 107-279, the Secretary is required to provide 

for continuing review of the National Assessment of Educational Progress (NAEP) 

assessment. The legislation identifies the issues to be addressed in the reviews, one of 

which includes the requirement to evaluate whether the NAEP achievement levels, 

established by the National Assessment Governing Board (NAGB), are “reasonable, valid, 

reliable and informative to the public.” Section 303(e)(2)(C) of the Education Sciences 

Reform Act of 2002 (P.L. 107-279) states that NAEP achievement levels shall be used on a 

trial basis until the Commissioner of the National Center for Education Statistics (NCES) 

determines, as a result of an evaluation under Sec. 303(f), that such levels are “reasonable, 

valid, and informative to the public.” This independent evaluation is intended to provide IES 

and the Commissioner of NCES with information necessary to inform the decision about 

whether the current trial status of the NAEP achievement levels can be removed or whether 

they should remain in trial status. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 199 

2. Key Objectives:

 Determine how “reasonable, valid, reliable and informative to the public” will be 

operationalized in this study. 

 Identify the kinds of objective data and research findings that will be examined. 

 Review and analyze extant information related to the study’s purpose. 

 Gather other objective information from relevant experts and stakeholders, without 

creating burden for the public through new, large-scale data collection. 

 Organize, summarize, and present the findings from the evaluation in a written report, 

including a summary that is accessible for nontechnical audiences, discussing the 

strengths/ weaknesses and gaps in knowledge in relation to the evaluation criteria. 

 Provide, prior to release of the study report, for an independent external review of that 

report for comprehensiveness, objectivity, and freedom from bias. 

3. Design: This study is focusing on the achievement levels used in reporting NAEP results for 

the reading and mathematics assessments in grades 4, 8, and 12. Specifically, the study is 

reviewing developments over the past decade in the ways achievement levels for NAEP are 

set and used and will evaluate whether the resulting achievement levels are “reasonable, 

valid, reliable, and informative to the public.” The study is relying on an independent 

committee of experts with a broad range of expertise related to assessment, statistics, social 

science, and education policy. The project is receiving oversight from the Board on Testing 

and Assessment (BOTA) and the Committee on National Statistics (CNSTAT) of the 

National Research Council. 

4. Estimated Completion Date: The report for this study is expected in 2016.

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_naep.asp

Implementation and Impact Evaluation of Race to the Top and SIGs: Final Report 

1. Study Purpose: RTT is a competitive grant program of the Department that funds states and 

districts planning to implement comprehensive education reform in one or more core areas. 

Since its first awards in 2010, RTT has funded general state competitions, state 

competitions focused on early learning, and district competitions focused on personalized 

learning. With funds from the Recovery Act, the general state competition awarded 

approximately $4 billion to states in support of comprehensive K–12 education reform in 

several core areas, including teachers and leaders, standards and assessments, data 

systems, and school turnaround. The SIG program is authorized through Title I of the ESEA

and provides three-year awards to support turnaround in the nation’s persistently lowestachieving schools. In FY 2009, the $546 million SIG appropriation was supplemented by 

$3 billion through the Recovery Act, for a total of $3.5 billion. SIG funds are distributed to 

states by formula based on Title I allocations. States then competitively award funds to 

districts applying on behalf of their eligible schools. Schools receiving SIG must implement 

one of four prescriptive intervention models: turnaround, transformation, closure, or restart. 

Both RTT and SIG received substantial funding through the Recovery Act. This study will 

examine the implementation and impacts of RTT and SIG, focusing on the initial general 

state competition for RTT and the first cohort of SIG schools implementing intervention 

models beginning in the 2010–11 school year. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 200 

2. Key Questions Addressed:

 Which policies and practices promoted by the RTT program do RTT states report using, 

and how do they compare to the policies and practices that non-RTT states report 

using? 

 Is receipt of an RTT grant related to improvement in student outcomes? 

 Are SIG-funded schools using the improvement or turnaround strategies promoted by 

the four SIG intervention models, and how do they compare to strategies in schools not 

implementing a SIG-funded intervention model? How are states and districts supporting 

such efforts? 

 Does receipt of SIG funding to implement a school intervention model have an impact on 

outcomes for low-performing schools? 

 Is implementation of the four school intervention models related to improvement in 

outcomes for low-performing schools? 

3. Design: The RTT sample will include all 50 states and DC. Data from interviews with all 

states and DC will inform the first evaluation question. The second evaluation question will 

be addressed using a short interrupted time series design with state-level NAEP data 

comparing, before and after the RTT competition, states that were awarded an RTT grant to 

states that applied for but were not awarded an RTT grant. The SIG sample will include 

about 500 schools in 60 districts from 22 states. This sample will be purposively selected to 

support a regression discontinuity design to address the fourth evaluation question, 

exploiting cutoff rules that states used to identify their persistently lowest-achieving schools 

as eligible for SIG to implement one of the four intervention models. Data from state and 

district interviews, as well as school surveys from the SIG sample will inform the third and 

fifth evaluation questions. Student- and school-level achievement data will also be collected 

from administrative records up to the 2012–13 school year to inform the fourth and fifth 

evaluation questions. 

4. Estimated or Actual Completion Date: The first report was released in September 2015. The 

final report is expected in fall 2016. 

5. Key Findings from the First Report:

 Early (Round 1 and 2) RTT states used more policies and practices than non-RTT states 

in five of six areas examined: state capacity, standards and assessments, data systems, 

teachers and leaders, and charter schools (school turnaround was the exception). Later 

(Round 3) RTT states used more policies and practices than non-RTT states in just one 

area: teachers and leaders. 

 Use of policies and practices across RTT and non-RTT states was highest in the state 

capacity and data systems areas and lowest in the teachers and leaders area. 

 SIG schools used more practices than non-SIG schools in all four areas examined: 

comprehensive instructional reforms, teacher and principal effectiveness, learning time 

and community-oriented schools, and operational flexibility and support. 

 Use of practices among SIG and non-SIG schools was highest in the comprehensive 

instructional reforms area and lowest in the operational flexibility and support area. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_racetotop.asp
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 201 

Implementation of Title I/II Program Initiatives 

1. Study Purpose: Title I and Title II programs are part of the ESEA and are intended to help 

provide all students with equal access to education by providing financial assistance to 

schools and districts which have a high percentage of students from low-income families 

(Title I) and improving teacher and principal quality (Title II). The previous reauthorization of 

ESEA, through the No Child Left Behind Act of 2001, increased accountability through the 

use of assessments for students, requirements for all teachers to be highly qualified, and 

funding for supports and interventions for schools not achieving adequate yearly progress 

for the entire school population as well as for various subgroups of students. Historically, 

there has been a congressionally mandated study of Title I that has also included 

information about Title II. Now that ESEA has been reauthorized, the second data collection 

for this study will be augmented to collect baseline information to evaluate the 

implementation of the new Title I and Title II provisions of ESSA. 

2. Key Questions Addressed:

 What content standards and high school graduation requirements are states adopting, 

and what materials and resources do states, districts, and schools provide to help 

teachers implement the state content standards? 

 What assessments do states and districts use (in terms of assessment format and 

coverage of grade levels and content areas), and what materials and resources do 

states, districts, and schools provide to support the implementation of assessments and 

use assessment data? 

 How has student achievement changed over time? 

 What elements are included in states’ accountability systems? How do states and 

districts identify and reward their highest-performing schools, how do they identify and 

support their lowest-performing schools, and how do they offer differentiated support for 

those schools that are neither highest-performing nor lowest-performing? 

 How do states and districts evaluate teacher and principal effectiveness and assess 

equitable distribution of teachers and principals, and what supports do states, districts, 

and schools provide to improve teacher and principal effectiveness? 

3. Design: The study is descriptive and nationally representative. Data were collected from all 

50 states, a nationally representative sample of districts and schools, and teachers within 

those schools through surveys in the 2013–14 school year. 

4. Estimated or Actual Completion Date: The first interim report, based on the findings from the 

baseline data collection, is expected in 2016. 

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_titleI.asp

Progress and Challenges in Developing Tiered Quality Rating and Improvement Systems 

(TQRIS) in the Round 1 Race to the Top-Early Learning Challenge (ELC) States 

1. Study Purpose: ELC aims to improve early learning and development programs so children 

enter kindergarten ready to succeed. One of the key areas of reform for the program 

focuses on the design and implementation of Tiered Quality Rating and Improvement 

Systems (TQRIS) that can provide parents and stakeholders with information on the quality 

of early learning programs. Since ELC grantee states are in the process of developing, 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 202 

refining, and/or implementing their TQRIS, this evaluation will describe the current state 

plans and implementation status.

2. Key Question Addressed: 

 How are TQRIS ratings defined, collected, and generated in the nine Round 1 ELC 

states and how might the structure and implementation of the rating system influence the 

reliability and validity of the ratings? 

3. Design: This is a descriptive study that involved collecting and analyzing various data from 

the nine Round 1 ELC grantee states. The study collected and conducted a targeted review 

of documents describing the structure of TQRIS, including component measures and the 

quality indicators used to evaluate preschool programs, how these are combined to 

generate overall ratings, the availability of TQRIS and child outcome data, and policies 

regarding the administration of kindergarten entry assessments (KEAs) in each Round 1 

state. The study also included interviews with state administrators to confirm and clarify the 

information obtained from documents and, as needed, gather information that could not be 

obtained from the document reviews. 

4. Estimated or Actual Completion Date: An evaluation brief is scheduled for completion in 

2016.

5. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_rtt.asp

Policy and Program Studies Service (PPSS)

Study of the Early Implementation of the ESEA Flexibility Provision 

1. Study Purpose: On September 23, 2011, President Obama announced the opportunity for 

states to request flexibility of certain ESEA provisions to help states move forward with 

efforts to improve student achievement and the quality of instruction in their schools. With 

input from broad and diverse stakeholders, the Department designed the ESEA Flexibility 

initiative so that states could build on their existing reform efforts to: (1) transition to collegeand career-ready standards and aligned assessments; (2) implement new systems of 

differentiated recognition, accountability, and support for schools; (3) evaluate and support 

teacher and principal effectiveness in more robust ways; and (4) reduce duplication and 

unnecessary burden on school districts by removing any duplicative and burdensome 

reporting requirements. The early implementation study will provide documentation of early 

state- and local-level implementation of the flexibility provision. This information is intended 

to inform federal and state policymakers and to support the Department’s monitoring and 

technical assistance to states. The study includes reviews of state and local documents; 

interviews with education officials at the state, district, and school level in early implementing 

states; and analysis of baseline student outcome and other data obtained through extant 

data sources. 

2. Key Questions Addressed:

 What prior experiences and rationales guided state approaches to accountability and 

support under the ESEA Flexibility initiative, including states’ selection of AMOs? 

 What are the primary components of state accountability and support systems under 

ESEA Flexibility? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 203 

 What modes of communication did states and districts use to inform and build local-level 

understanding of state systems under ESEA Flexibility? To what extent were these 

communication efforts perceived as effective? 

 How did local-level officials perceive state accountability systems under ESEA Flexibility, 

including the criteria being used to identify low-performing schools? 

 What approaches are states taking to identify and intervene in low-performing schools 

and districts? What improvement strategies did principals report implementing? 

 What structures or mechanisms were states and districts using to support lowperforming schools, and how did principals perceive these supports? 

 How did states and districts perceive their capacity to implement ESEA Flexibility 

provisions, and what factors did states and districts report as challenging and supporting 

their implementation efforts? 

3. Design: The primary data sources for this study included state ESEA Flexibility applications, 

state and district policy documents, and interviews with officials identified by state and 

district leaders as holding primary responsibility for accountability and support systems 

under ESEA Flexibility. 

4. Estimated Completion Date: The final report is scheduled for completion in winter 2015–16. 

State Efforts to Promote Equitable Access to Effective Teachers 

1. Study Purpose: This report provides a broad overview of state efforts, as of the 2011–12 

school year, to monitor equitable access to qualified and effective teachers among schools; 

develop and adopt multiple measures of teacher performance to rate teachers among at 

least three performance levels; and implement targeted strategies for promoting equitable 

access to qualified and effective teachers in schools serving high proportions of poor and/or 

minority students. The report examines the use of measures of teacher qualifications and 

teacher performance in the implementation of these strategies. 

2. Key Questions Addressed: 

 What measures did states use to monitor equitable access to qualified and effective 

teachers among schools? 

 To what extent were states developing or using multiple measures of teacher 

performance to rate teachers among at least three performance levels? 

 What strategies did states use to promote equitable access to qualified and effective 

teachers in schools serving high proportions of poor and/or minority students? 

3. Design: This report is based on telephone interviews with officials in SEAs in all states, the 

District of Columbia, and Puerto Rico. Study staff also reviewed extant sources including 

equity plans, Consolidated State Performance Reports, reports from federal monitoring 

visits, and other information on state websites. The interviews were conducted between 

August 2011 and January 2012, and the study reflects state efforts that were underway at 

that time, and predates implementation of the fall 2011 ESEA Flexibility initiative as well as 

the recent state equity plans submitted under the Excellent Educators for All initiative. 

4. Estimated Completion Date: The final report is scheduled for completion in winter 2015–16. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 204 

Evaluation of the Teacher Incentive Fund: Final Report 

1. Study Purpose: This study examines program implementation in the first two cohorts of TIF 

grantees (2006 and 2007), which included 33 grantees. All 33 grantees implemented 

performance pay systems for principals and other school administrators; 31 grantees also 

included teachers in their performance pay systems. The final report complements a 

previous 2012 report from this study on the early implementation of the program, and 

analyzes award payouts and educator perspectives on a variety of issues related to 

implementation. 

2. Key Questions Addressed: 

 How did the size and distribution of incentive awards vary across educators and 

grantees? 

 To what extent were incentive payments based on student achievement vs. other factors 

such as supports for improving practice, performance evaluations, and teaching in hardto-staff schools or subjects? 

 How did participating educators perceive the fairness and effectiveness of the 

performance pay system? 

3. Design: Data collection included surveys of a representative sample of teachers and 

principals in schools served by TIF projects; financial data for awards paid to educators; 

interviews with TIF project directors, teachers, principals, and other stakeholders; and 

reviews of extant documents. The surveys were conducted in spring 2011, and the incentive 

award payout data are primarily for the 2010–11 school year. 

4. Estimated Completion Date: The final report is scheduled for completion in winter 2015–16.

Study of Emerging Teacher Evaluation Systems 

1. Study Purpose: This study will provide descriptive information on the design and early 

implementation of teacher evaluation systems in eight school districts. The findings are 

intended to help other districts and states learn from the experiences of eight districts 

featured in the study, and apply the design and implementation lessons to their own work as 

it relates to teacher evaluation and support. 

2. Key Questions Addressed: 

 What are the purposes of the new teacher evaluation systems? With what stakeholders 

did districts consult and what measures were included when designing the new 

systems? 

 What steps did the districts take prior to full implementation to test the system and 

prepare teachers and staff to implement it? 

 How did the districts structure and conduct the classroom observation component of 

their teacher evaluation systems? 

 How did the districts analyze student achievement and other data to evaluate teacher 

performance? 

 How did the districts use teacher evaluation results to make personnel decisions? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 205 

 What administrative structures did districts use to support their new teacher evaluation 

systems? 

 What are the perceived early effects of the teacher evaluation systems on districts, 

principals, and teachers? 

3. Design: This descriptive study relies on interviews with key district administrators, principals, 

teachers, and representatives of community stakeholder groups, from eight districts, who 

were involved in the development and early implementation of the respective districts’ 

teacher evaluation system. Given the limited sample, the findings cannot be generalized to 

other districts.

4. Estimated Completion Date: The final report is scheduled for completion in winter 2015–16. 

National Evaluation of Title III Implementation: A Description of English Learner Student 

Achievement in Four Jurisdictions 

1. Study Purpose: Under ESEA Title III accountability provisions, annual measurable 

achievement objectives (AMAOs) are used to provide district-level and state-level summary 

information about EL students’ progress in attaining English proficiency and meeting state 

standards in English/language arts and mathematics. However, AMAOs provide only a 

partial understanding of EL performance because they do not control for the compositional 

shifts in the EL student population, disaggregate performance by subgroup characteristics 

(e.g., level of English language proficiency, poverty status), or provide continuous measures 

of student achievement. This report uses longitudinally linked student-level assessment data 

in four jurisdictions—New York, Texas, and two school districts in California—to examine 

student progress for consistent cohorts of ELs, former ELs, and non-ELs who could be 

followed for three years. 

2. Key Questions Addressed:

 Are EL students making progress in learning English? Does the amount of progress vary 

by student characteristics? 

 How long does it take for EL students to attain proficiency on the state ELP 

assessments, and to be redesignated as former English learners (ELs)? 

 Are EL students making progress toward meeting achievement targets on state content 

assessments in English/language arts and mathematics? 

 Are achievement gaps narrowing for EL students and for former EL students? 

3. Design: The study analyzed consistent cohorts of those students who were tested in the 

base year and in each of the two subsequent years (i.e., those with three consecutive years 

of assessment data). Most analyses examined a three-year period, from 2006–07 through 

2008–09 for New York and from 2005–06 through 2007–08 for the other three jurisdictions. 

The analytic sample of students is not representative of all students served by the four 

jurisdictions. Therefore, the findings cannot be generalized to all ELs, non-ELs, or former ELs 

in those states and districts, or to the nation as a whole. 

4. Estimated Completion Date: The final report is scheduled for completion in winter 2015–16. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 206 

Study of English Learners with Special Needs 

1. Study Purpose: The purpose of this study is to describe, on an exploratory scale, common 

issues in the identification of ELs for special education. The report will focus on four topics: 

procedures and practices used in identification; assessment data and instruments used to 

help determine ELs eligibility for special education; personnel, qualifications, and 

professional development to assist in identification; and criteria on which decisions are 

made to exit ELs with disabilities out of their language instruction programs. 

2. Key Questions Addressed: 

 What procedures and instruments are used to evaluate and identify ELs with disabilities? 

 What personnel are involved in the assessment and identification of ELs with disabilities 

(as distinct from non-ELs) and what are their roles? 

 What are typical criteria for exiting ELs with disabilities from language instruction 

educational programs? 

 What procedures, practices, and instruments are used to assess and identify ELs with 

disabilities, and how do these differ from those used with non-ELs? 

 What are the roles, backgrounds, and qualifications of school and district personnel 

involved in the assessment and identification of ELs with disabilities? 

 What challenges do districts and schools encounter in the assessment and identification 

of disabilities among ELs and what strategies do they use to overcome these 

challenges? 

 What procedures and practices do districts and schools use to exit ELs with disabilities 

from language instruction educational programs, and what are the challenges they face? 

3. Design: The study relied on a purposive sample of a diverse group of six case study 

districts, containing 18 schools total, which the study team visited in spring 2013. Study 

methods onsite largely consisted of interviews of a large number and variety of stakeholders 

involved in ESL, special education, and related support services at both the school district 

and school levels. Other data collected included special education data; published reports, 

guides, and regulations related to identification; and data on district and school websites. 

The data were coded to one or more of the three key questions and the findings were 

compared to those from a prior review of research. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2016. 

Feasibility Study on Improving the Quality of School-Level Expenditure Data 

1. Study Purpose: This feasibility study explores options for improving the completeness, 

consistency, and accuracy of school-level expenditures that would be most appropriate to 

include for assessing equity, as well as technical and operational challenges for including 

certain types of expenditures. The study also examines the experiences of states and 

districts that currently track school-level expenditures, including variations in definitions and 

practices used in these jurisdictions, the types of changes to accounting systems and 

practices that are required, and the potential cost of implementing such strategies. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 207 

2. Key Questions Addressed: 

 In states and districts that currently track expenditures at the school level, what types of 

personnel and nonpersonnel expenditures are included in the school-level data? 

 What is the quality of existing school-level expenditure data? What specific steps could 

be taken to improve the completeness, consistency, and accuracy of these data? 

 What changes would states and districts need to make to track expenditures at the 

school level if they do not currently do so? What costs have states and districts incurred 

to implement such data systems? 

3. Design: The study will explore options for improving the completeness, consistency, and 

accuracy of school-level expenditure reporting by: (1) convening an expert panel to identify 

specific challenges and potential solutions; (2) interviewing fiscal staff in five states and four 

school districts that have finance systems that track school-level finance data; and 

(3) collecting and analyzing available school-level expenditure data in the selected states 

and districts. 

4. Estimated Completion Date: The final report is scheduled for completion in summer 2016. 

Study of High School Strategies to Improve Graduation Rates 

1. Study Purpose: This nationally representative survey of high school administrators is 

examining strategies that schools are using to reduce students’ likelihood of dropping out of 

high school and to increase their likelihood of attaining a high school credential. The survey 

seeks information on what schools are implementing what activities with what students 

under what circumstances or conditions. The final report will use the survey data to describe 

the use and prevalence of various high school reform strategies to support at-risk youth and 

will compare high schools with high and low graduation rates. 

2. Key Questions Addressed: 

 What strategies are high schools implementing to help students stay in school and 

graduate? Do these strategies very for high schools with high or low graduation rates? 

 How many students are served through each of these strategies? Are the strategies 

focused on particular student populations? 

 How do schools deliver services or interventions for each of the strategies? What 

specific services are provided, and who provides the services? 

3. Design: The study is conducting a web-based survey of a nationally representative sample 

of approximately 2,000 high school administrators. 

4. Estimated Completion Date: The final report is scheduled for completion in summer 2016. 

Case Studies of the Implementation and Use of Kindergarten Entry Assessments (KEAs) 

1. Study Purpose: The purpose of the KEAs implementation case studies is to document the 

processes, accomplishments, challenges, and solutions of four states implementing KEAs 

and to share what they have learned with federal and state policymakers and the field. Of 

particular interest is to identify what is working well in states that are early adopters of KEAs. 

This information is needed to support the technical assistance efforts of the Department and 

to inform KEA efforts across the nation. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 208 

2. Key Questions Addressed:

 How have KEAs been developed and adopted by four states that were early to adopt a 

comprehensive KEA? 

 How are KEAs being implemented by 12 school districts within four states that were 

early adopters? 

 How do these four states and their districts and schools communicate and use KEA 

results to inform policy and practice? 

 What lessons did states, districts, and schools learn about KEA adoption, 

implementation, and use? 

3. Design: The nested sample design includes four states, 12 districts, and 24 schools. Data 

collection will include review of state and local documents, phone interviews with SEA 

preschool directors and professional development staff who facilitate district-level training, 

and in-person interviews with district administrators, principals, kindergarten teachers, and 

other staff involved in local administration of KEAs. 

4. Estimated Completion Date: The final report is scheduled for completion in summer 2016.

Study of Experiences and Needs of Rural Education Achievement Program (REAP) 

Grantees and Subgrantees 

1. Study Purpose: The purpose of this descriptive study is to examine how grantees and 

subgrantees use REAP funds provided through the Small, Rural School Achievement 

(SRSA) and Rural and Low-Income Schools (RLIS) programs—on their own or in 

combination with other federal funds—as well as to explore technical assistance needs 

related to REAP. The study will inform program operations, technical assistance to grantees, 

and future reauthorization of the programs. 

2. Key Questions Addressed:

 What are REAP grantees’ experiences of grant eligibility determination? To what extent 

do districts contact, or are they contacted by, the SEA regarding eligibility? 

 On what types of activities are REAP funds spent and in what proportions? What are 

grantees’ experiences of deciding how to target funds (e.g., processes and factors, 

personnel and stakeholders involved, integration in larger spending decisions)? Do 

districts perceive a need for greater flexibility in the use of federal Title VI funds? How 

does this vary by grantee characteristic? 

 How do grantees use REAP funds in combination with other federal program funds? To 

what extent are other Department or other federal programs (e.g., E-Rate, USDA Rural 

Development grants) considered or integrated? What are the challenges in spending 

REAP funds? 

 How many SRSA-eligible grantees use REAP flexibility? Does the use of flexibility differ 

by grantee characteristics? If flexibility is not used, why not? Exactly how do SRSA 

grantees tend to use flexibility? 

 What are the major challenges and technical assistance needs that REAP grantees face 

(e.g., eligibility determination and compliance with program requirements)? Are there 

technical assistance needs that grantees perceive to be unaddressed? For what? What 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 209 

is the frequency of technical assistance use by grantees? What is the purpose of such 

assistance, and who offers it? 

 Is there anything that districts or states would recommend changing about the REAP 

program administration or design? 

3. Design: The study consists of: 1) a survey of a sample of approximately 1,000 SRSA 

grantees and RLIS subgrantee districts; 2) telephone interviews with a sample of 30 SRSA 

grantees and RLIS subgrantees; and 3) telephone interviews with REAP coordinators in all 

states receiving REAP funds. Data collection will begin in winter 2015 and is expected to be 

completed by spring 2015. At the state level, the study will include interviews with all state 

REAP coordinators about state goals and priorities, the planning process for use of RLIS 

funds, the eligibility process for districts, management and distribution of SRSA and RLIS 

funds, and recommendations for the program. At the school district level, the study will 

include an online survey of a nationally representative sample of REAP coordinators about 

the REAP eligibility determination process, administration of REAP, and challenges and 

technical assistance needs related to REAP, as well as telephone interviews with a 

subsample of 30 districts. The interviews will include questions about program 

administration, technical assistance needs, and recommendations for changing the REAP 

program to better meet the needs of rural districts. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2016. 

A Study of Practices to Improve Learning in Kindergarten Through Third Grade 

1. Study Purpose: This study will summarize how (a) aligning preschool through third grade 

education and (b) differentiating instruction for children in kindergarten and first grade may 

build on the positive effects of preschool and help students in kindergarten through third 

grade (K-3) make cognitive, social-emotional, and academic gains. In addition, the study will 

conduct case studies of five programs that help improve cognitive, social-emotional, and/or 

academic outcomes for disadvantaged students in grades K-3 and that may build on the 

positive effects of preschool by using policies, programs, and practices from the two topic 

areas (a and b above). 

2. Key Questions Addressed: 

 What practices do educators and policymakers use to align preschool through thirdgrade (PK-3) education, and what considerations should stakeholders take into account 

when designing such PK-3 initiatives? 

 What are the findings and quality of research on the impact of differentiated instruction 

on children in kindergarten and first grade? 

 What are the characteristics (e.g., resources, personnel, staff characteristics, training, 

setting, population serviced) of PK-3 or differentiated instruction programs that aim to 

increase cognitive, social-emotional, or academic outcomes of students? 

 On what research, theory, and/or experiences did the designers of these programs base 

the program structure and content? 

 What are the challenges of implementing these programs, and how have staff and 

leaders tried to overcome these challenges? 

 How does the organization implementing the program ensure its sustainability? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 210 

3. Design: The literature review covers two topics: (1) preschool and K–3 alignment, and (2) 

differentiated instruction in kindergarten and first grade. The case studies will examine five 

sites that are implementing PK-3 education or differentiating instruction in kindergarten and 

first grade. The research time will interview principals, teachers, evaluators, and funders to 

understand programs’ characteristics, challenges and solutions, and the sustainability of the 

programs. 

4. Estimated Completion Date: The final case study report is scheduled for completion in winter 

2016–17.

Implementation Study of the Turnaround School Leaders Program 

1. Study Purpose: This study will examine the implementation of the Turnaround School 

Leaders Program (TSLP). The study will provide information on how grantees identify, 

develop, and support leaders and aspiring leaders of low-performing schools. The study will 

also examine how grantees: (1) maintain fidelity to the plans set forth in their original 

proposals; (2) use data to examine progress; and (3) work with project partners to meet 

goals. 

2. Key Questions Addressed: 

 How do TSLP projects identify, develop, and support leaders for low-performing 

schools? 

 What role do project partners play in implementing projects and helping grantees to 

achieve project goals? 

 To what extent is grantee implementation of their projects consistent with their original 

proposals? 

 How are grantees measuring the success of their TSLP projects, and do early outcome 

data show promising results? 

3. Design: The study will include case studies of five TSLP grantees, including each grantees’ 

partners; an analysis of extant data, including grantee applications, early outcomes data, 

and other relevant project-specific data; and surveys of all local coordinators of TSLP 

projects. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Study of School Climate Transformation Grants 

1. Study Purpose: The study will describe how states and school districts that participate in the 

School Climate Transformation Grant program are coordinating services and supports with 

certain other related federal programs administered by the Departments of Health and 

Human Services and Justice. 

2. Key Questions Addressed:

 What does coordination between SCTG and Project AWARE grantees look like? 

 What does coordination between SCTG and DOJ grantees look like? 

 What did grantees say about the value of coordination? 

 What were the challenges and lessons learned? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 211 

3. Design: The study will conduct telephone interviews with representatives from 38 state and 

local grantees to explore the ways in which grantees coordinate services, the benefits that 

grantees have experienced from program coordination, and challenges and lessons learned. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Study of Title I Schoolwide and Targeted Assistance Programs 

1. Study Purpose: The study will examine how Title I schoolwide programs use the schoolwide 

flexibility to design services and strategies to address the needs of low-achieving students 

and subgroups, and how such strategies compare to approaches used in targeted 

assistance programs. The study will include interviews and analysis of extant data in 

approximately 30 Title I schools, including both schoolwide and targeted assistance 

programs, as well as surveys of principals and district administrators for a representative 

sample of Title I schools. 

2. Key Questions Addressed: 

 How do schoolwide and targeted assistance programs use Title I funds to improve 

student achievement, particularly for low-achieving subgroups? 

 How do districts and schools make decisions about how to use Title I funds in 

schoolwide programs and targeted assistance programs? 

 To what extent do schoolwide programs commingle Title I funds with other funds or 

coordinate the use of Title I funds with other funds? 

3. Design: The study will include both in-depth case studies and surveys of a nationally 

representative sample of 470 districts and 1,410 schools. The case studies will include 

approximately 30 Title I schools including both schoolwide and targeted assistance 

programs and data collection for the case studies will include telephone interviews as well 

as extant documents and data, including Title I budgets and plans. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Study of Digital Learning Resources for Instructing English Learners (ELs) 

1. Study Purpose: This study will examine the use of digital learning resources (DLRs) to 

support the English language acquisition and academic achievement of ELs in K-12 

education. Specifically, the study will focus on DLRs that may be considered “apps” for 

enhancing instruction for EL students, including computer software, online programs, 

websites, mobile applications, and mobile computing devices. 

2. Key Questions Addressed: 

 How do districts and schools make decisions regarding investments in digital learning 

resources to support ELs, and how do they identify and select specific DLRs for ELs? 

What DLRs are most commonly used to support ELs? 

 How do teachers of EL students use DLRs in the instruction of ELs? Do they receive 

professional development or coaching in effective use of DLRs? If so, what professional 

development approaches do they find to be most helpful? 

 What barriers inhibit ELs’ use of existing DLRs in the classroom and at home, and how 

can districts, schools, and DLR developers address those issues? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 212 

 How do districts and schools define and measure the success of their use of technology 

to support ELs? 

 How could developers of DLRs and practitioners improve the usefulness of these tools 

for teaching EL students? 

3. Design: The study will explore the range of such apps that are available for use with EL 

students, examine how districts and schools select and use these apps, and consult an 

expert panel of technology developers, practitioners, and education researchers for ideas on 

ways to improve the design and use of apps to support learning for EL students. It will rely 

on six key components: (1) a literature review; (2) market research on existing DLRs for 

K–12 instruction; (3) survey of school districts; (4) survey of teachers; (5) case studies; and 

(6) an expert panel of technology developers, practitioners, and education researchers. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Study of the Title III Native American and Alaska Native Children in Schools Program

1. Study Purpose: The study will examine services and strategies for ELs in the Native 

American, Alaska Native, Native Hawaiian, and Pacific Islander communities that are 

supported through the Native American and Alaska Native Children in Schools (NAM) 

program. The study includes case study of 25 NAM grantees, and examines: (1) the types of 

services and strategies that NAM funds support to address the instructional needs of these 

communities and develop student proficiency in both English and (optionally) native 

languages; (2) how grantees plan, implement , and evaluate their respective projects; 

(3) how grantees coordinate and prioritize the use of NAM funds in relation to other federal, 

state and local resources; and (4) how grantees use data and evidence to inform program 

implementation and meet U.S. Department of Education reporting requirements. 

2. Key Questions Addressed:

 How does the NAM program support grantees in providing services to Native American, 

Alaska Native, Native Hawaiian, and Pacific Islander children, especially through teacher 

training and professional development, curriculum development, parent engagement, 

and English (especially academic English) and native language instruction? How do 

NAM grantees incorporate technology to support and/or preserve heritage languages 

through dual-language or other forms of English language instruction? How many 

students are served through each of these strategies? Are the strategies focused on 

particular student populations? 

 What challenges do NAM grantees face in providing funded services, and what steps 

have grantees taken to overcome these challenges? To what extent does the 

Department or other external entities provide support to overcome these challenges? 

What lessons have the grantees learned? 

 How do schools deliver services or interventions for each of the strategies? What 

specific services are provided, and who provides the services? 

 What are the roles and responsibilities among (as applicable) tribal entities, public 

schools, local education districts, and state agencies in implementing NAM grants and 

meeting federal reporting requirements? 

 What are NAM grant stakeholders’ perceptions of community participation and student 

engagement in language instruction and other educational programs? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2015 Annual Performance Report and FY 2017 Annual Performance Plan—U.S. Department of Education 213 

3. Design: The study will include site visits to NAM grantees to obtain detailed information 

regarding how they are implementing the NAM program, and meeting the needs of their 

students and communities, as well as telephone surveys or interviews of local grant 

coordinators and state directors of Indian education to inform the case study site visits. The 

study will also utilize extant data, including grant applications.

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017. 

Evaluation of the Title I, Part D Neglected or Delinquent Program 

1. Study Purpose: The study will examine the implementation of educational programs for 

children and youth in residential facilities and correctional institutions funded under Title I, 

Part D of ESEA. The study will be informed by surveys of state grantees and local 

subgrantees to examine the types of services and strategies that Part D funds support, how 

state and local agencies assist students in transitioning back to schools, how state 

correctional facilities implement institution-wide Part D projects, and how grantees assess 

the educational outcomes of participating students. 

2. Key Questions Addressed: 

 What types of services and strategies do Title I, Part D, funds support for children and 

youth in correctional and child welfare settings? 

 What types of services and strategies do Title I, Part D, funds support for children and 

youth in correctional and child welfare settings? 

 How do correctional facilities and agencies assist students in transitioning to districts 

and schools, including those outside their jurisdictions? 

 How do state correctional facilities plan and implement institutionwide Part D projects? 

 How do grantees assess the educational outcomes of students participating in Part Dfunded educational programs? 

3. Design: The study will include a review of extant data, a review of literature related to 

programs for neglected and delinquent youth, surveys of state and local coordinators of 

Title I-Part D funded programs, and site visits to state agencies, school districts, correctional 

institutions, and child welfare facilities to obtain more detailed information on how grantees 

and subgrantees are implementing the Part D programs and how they are meeting the 

needs of their students. 

4. Estimated Completion Date: The final report is scheduled for completion in fall 2017.