APPENDICES

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 120 

Appendix C: Summary of Performance Evaluations Conducted 

During FY 2014 and Expected During FY 2015–16

For a complete list of program evaluations and studies from the Office of Planning, Evaluation 

and Policy Development, please visit 

http://www2.ed.gov/about/offices/list/opepd/ppss/reports.html. For a complete list of evaluation 

studies of the National Center for Education Evaluation and Regional Assistance, please visit 

http://ies.ed.gov/ncee/projects/evaluation/index.asp. 

Evaluation Reports From FY 2014

National Center for Education Evaluation and Regional Assistance

School Choice 

Evaluation of the DC Opportunity Scholarship Program: An Early Look at Applicants and 

Participating Schools Under the SOAR Act 

1. Study Purpose: The April 2011 Scholarships and Opportunities for Results Act (SOAR Act) 

provided for a five-year continuation of a school choice program for low-income residents of 

Washington, DC. The program, still titled the Opportunity Scholarship Program (OSP), now 

provides annual scholarships of $8,000 (for grades K–8) or $12,000 (for grades 9–12) to enable 

low-income students to attend private schools in DC in lieu of the public schools already 

available to them. The new law also mandated another independent, rigorous evaluation of the 

effectiveness of the program. This report explores implementation of the DC OSP in the first two 

years after the reauthorization. 

2. Key Questions Addressed:

 Private schools: How many participate in the OSP and what are their characteristics? 

 Applicants: What is the nature of the demand being generated for the program among 

eligible students and families? What motivates families to apply to the OSP? How 

dissatisfied were they with schools before applying and what do they want most in a new 

school for their child? 

 Scholarships: To what extent is the OSP enabling students to enroll in private schools? 

3. Design: This descriptive report relies entirely on the application forms parents filled out when 

they applied to the OSP, school characteristics from the program operator’s school directory 

and NCES databases, and scholarship award and use records from the program operator. 

4. Estimated or Actual Completion Date: The final report was released in October 2014. 

5. Key Findings: 

 Just over half of all DC private schools participated in the OSP, with current schools more 

likely to have published tuition rates above the OSP scholarship amounts than did 

participating schools in the past. 

 OSP applicants under the SOAR Act represent between 3 and 4 percent of the estimated 

53,000 children in DC who meet the eligibility criterion. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 121 

 A number of awarded scholarships go unused, with students from disadvantaged schools 

and families using awarded scholarships at lower rates than others. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20154000/pdf/20154000.pdf 

Students with Disabilities 

Evaluation of the Personnel Development Program to Improve Services and Results for 

Children with Disabilities 

1. Study Purpose: This descriptive evaluation is of the Personnel Development Program (PDP) 

funded under IDEA, Title I, Part D, Subpart 2, Sec. 662. Projects funded under the program are 

designed to help address state-identified needs for personnel in special education, and help 

ensure that special education personnel are highly qualified and that teachers have the 

necessary knowledge and skills to provide appropriate instruction to students with disabilities. A 

portion of PDP grants is awarded to National Centers, which are to provide national capacitybuilding and scientifically based products and services to a variety of audiences. Grants are also 

awarded to specific institutions of higher education to develop courses of study for special 

education teachers and other service providers. These training grants can be used to improve 

the quality of personnel preparation programs and for stipends that support students enrolled in 

the programs. The PDP was funded at $88.299 million in FY 2012. 

2. Key Questions Addressed:

 What products were developed and services provided by the PDP National Centers funded 

between FY 2001 and FY 2007, and at what cost? 

 What were the quality and relevance/usefulness of documented materials and technical 

assistance provided by PDP National Centers funded between FY 2001 and FY 2007? 

 What were the characteristics of funded courses of study at IHEs awarded PDP training 

grants in FY 2006 or FY 2007? 

 How did funded courses of study use PDP training grant funding? 

 How many scholars enrolled in the funded courses of study, completed their programs, or 

dropped out before completion? 

 What were the quality and relevance/usefulness of new or significantly modified components 

for funded courses of study? 

 What became of courses of study that did not receive PDP training grant funding? 

3. Design: The evaluation relied on a combination of extant data and surveys of PDP grantees 

and applicants. Panels of experts rated the quality and relevance/usefulness of products and 

services from 12 National Centers and course-of-study components developed or significantly 

modified by recipients of PDP training grants awarded in FY 2006 or FY 2007. 

4. Estimated or Actual Completion Date: The report was released in winter 2014. 

5. Key Findings:

 For 15 products identified by national technical assistance center staff as their signature 

works and reviewed by independent panels of experts, the mean rating (on a scale of 1–5, 

with 5 being the highest) was 4.13 for quality and 4.25 for relevance/usefulness. For 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 122 

86 nonsignature products, the mean rating was 4.11 for quality and 3.91 for 

relevance/usefulness. 

 For 134 new or significantly modified components from 99 courses of study supported by 

PDP training grants, the mean quality rating (on a scale of 1–5, with 5 being the highest) 

was 3.71. 

 Thirty-four percent of courses of study that were not funded through FY 2006 and FY 2007 

training grant competitions were developed or maintained without PDP funding. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144007/

National Evaluation of the IDEA Technical Assistance and Dissemination Program: 

Interim Report 

1. Study Purpose: IDEA, which was most recently reauthorized in 2004, provides funds to assist 

states and local eduational agencies in making available a free, appropriate public education for 

children with disabilities. Funded at $12.6 billion in FY 2010, IDEA supports early intervention 

services for infants and toddlers, special education and related services for children ages 3 

through 21, and early intervening services for students not in special education but in need of 

academic or behavioral support. IES is conducting studies under Section 664 of IDEA to assess 

the implementation and effectiveness of key programs and services supported under the law. 

As specified in IDEA Part D, the Technical Assistance and Dissemination (TA&D) Program is to 

provide technical assistance, support model demonstration projects, disseminate useful 

information, and implement activities that are supported by scientifically based research to meet 

the needs of children with disabilities. The National Evaluation of the IDEA TA&D Program is 

designed to describe the products and services provided by the TA&D Program grantees, state 

and local needs for technical assistance, and the role that the TA&D Program plays in meeting 

these needs and supporting implementation of IDEA. 

2. Key Questions Addressed:

 What are the primary technical assistance activities of the TA&D Program? 

 What are states’ needs for technical assistance and to what extent are these needs 

addressed by TA&D centers or other sources? 

 For selected topics, to what extent are states satisfied with the products and services 

received from TA&D Program centers? 

3. Design: Data collection included administering surveys to TA&D Program grantees, all state 

IDEA Part B and Part C administrators, and a sample of state-level special education program 

staff. Data were collected between November 2012 and March 2013. State-level administrators 

and staff reported on their receipt of techincal assistance from TA&D Program grantees during 

the 2010–11 school year. Grantees reported on their goals and activities from the beginning of 

their current grant through the interview date. The funding period for the centers included in the 

interim report varied, with the earliest end date in 2012 and the latest in 2014. 

4. Estimated or Actual Completion Date: An interim report was released in October 2013. The 

final report is scheduled for completion in October 2016.
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 123 

5. Key Findings of Interim Report: 

 TA&D centers most commonly reported providing technical assistance on the topics of 

parent and family involvement and data systems and use of data for improvement. States 

identified (a) “General Supervision/Monitoring,” (b) “early childhood transition,” (c) “financing 

of services/financing for special education,” and (d) “Response to Intervention” as the topics 

for which they had the greatest need for technical assistance in 2010–11. 

 Many TA&D centers provide technical assistance on similar topics. For example, 14 states 

received “high intensity” technical assistance (i.e., frequent training or consultation) on the 

same topic from 5 different centers. This evaluation was unable to establish whether such 

cases are indicators of inefficiency or of complementary and coordinated services. 

 State staff rated the majority of technical assistance experiences they had with TA&D 

centers as “very satisfactory” (71 percent). On average, customers receiving high intensity 

technical assistance were significantly more satisfied than those receiving lower intensity 

(i.e., infrequent training and consultation or web-only support). Satisfaction did vary to some 

degree depending on the special education topic being addressed. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144000/pdf/20144000.pdf 

The Inclusion of Students with Disabilities in School Accountability Systems: An Update 

1. Study Purpose: The focus of this study is on the inclusion of students with disabilities (SWDs) 

in school accountability systems and the variation in school practices in schools accountable 

and schools not accountable for the performance of the SWD subgroup under ESEA.

2. Key Questions Addressed:

 To what extent are schools accountable for the performance of the SWD subgroup, and how 

does this accountability vary across schools and over time? 

 To what extent have schools accountable for the SWD subgroup been identified as needing 

improvement? 

 How does school accountability for the SWD subgroup relate to regular and special 

education practices for students with disabilities? 

3. Design: The evaluation is relying on descriptive statistics to study patterns of school 

accountability across states and over time and to examine how school practices vary with 

school accountability for the SWD subgroup. Data sources for the evaluation include extant data 

from the Department’s EDFacts database and 2011 surveys of principals and special education 

designees from elementary and middle schools in 12 states. 

4. Estimated or Actual Completion Date: This report, an update of the 2012 interim report, was 

released in October 2013. A third report, which will address the third key question listed above, 

will be released in winter 2015. 

5. Key Findings: 

This report addressed the first two key questions listed above for this study: 

 Across the 44 states with relevant data and DC, 35 percent of public schools were 

accountable for the performance of the SWD subgroup in the 2009–10 school year, 

representing 59 percent of SWDs in those states. In those same 44 states and DC, 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 124 

62 percent of middle schools were accountable for SWD performance, while 32 percent of 

elementary schools and 23 percent of high schools were accountable. 

 In 31 states with relevant data, 56 percent of public schools were not accountable for the 

SWD subgroup in any of the 4 years examined, in comparison with 23 percent of schools 

that were consistently accountable in each of the 4 years. 

 Among schools that were consistently accountable for the performance of the SWD 

subgroup across 22 states during the 4 years, 56 percent were never identified for school 

improvement over this time period. By comparison, among schools that were consistently 

not accountable for SWD subgroup performance in these states, 80 percent were never 

identified for improvement. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20134017/

Teacher Quality 

Access to Effective Teaching for Disadvantaged Students 

1. Study Purpose: Recent Department of Education initiatives, such as RTT, the Teacher 

Incentive Fund, and the ESEA Flexibility policy, are designed in part to ensure that 

disadvantaged students have equal access to effective teaching. This study provides 

information about the extent that disadvantaged students receive less effective teaching than 

other students. The study also examines teacher mobility in participating districts and how 

patterns of mobility might contribute to unequal access. 

2. Key Questions Addressed:

 To what extent do disadvantaged students have equal access to effective teaching within 

school districts, and how does this change over time? 

 Is access to effective teaching related to different patterns of teacher hiring, retention, and 

mobility for high- and low-poverty schools? 

 What policies are districts implementing that could promote an equitable distribution of 

effective teachers? 

3. Design: The study is descriptive. It documents the distribution of effective teaching, as 

measured by value added, and changes in the distribution of effective teaching across the 

2008–09 through 2012–13 school years. The study also describes district polices designed to 

address inequitable distribution of effective teaching implemented during those years. Lastly, 

the study will examine teacher mobility patterns within participating districts. Data collection 

included the annual collection of district administrative records, including student achievement, 

to conduct value-added analyses as well as annual semi-structured interviews with district 

leadership to provide information on district policies. Data collection also included district 

personnel data to examine teacher mobility within participating districts. The study will be 

conducted in 29 geographically dispersed school districts. 

4. Estimated or Actual Completion Date: The report was released in November 2013. 

5. Key Findings:

 On average, disadvantaged students had less access to effective teaching than relatively 

more advantaged students. Providing equal access to effective teaching for disadvantaged 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 125 

and other students would reduce the student achievement gap from 28 percentile points to 

26 percentile points in English/language arts (ELA), and from 26 percentile points to 

24 percentile points in math, in a given year. 

 Access to effective teaching patterns for disadvantaged students were similar over the three 

years studied, 2008–09 through 2010–11. 

 Access to effective teaching varied across study districts. Access ranged from districts with 

equal access to districts with differences in access as large as 0.106 standard deviations of 

student test scores in ELA and 0.081 standard deviations of student test scores in math, 

favoring relatively more advantaged students. 

 Access to effective teaching was more related to the school assignment of students and 

teachers than the way that students were assigned to teachers within schools. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144001/pdf/20144001.pdf

Do Disadvantaged Students Get Less Effective Teaching? Key Findings from Recent 

Institute of Education Sciences Studies 

1. Study Purpose: Recent Department of Education initiatives, such as RTT, the Teacher 

Incentive Fund, and the ESEA Flexibility policy, are designed in part to ensure that 

disadvantaged students have equal access to effective teaching. This brief provides information 

about the extent that disadvantaged students receive less effective teaching than other students 

by synthesizing findings from several IES-funded studies, including the report, “Access to 

Effective Teaching for Disadvantaged Students.” 

2. Key Questions Addressed:

 What do three IES-funded studies on teacher distribution conclude about equitable access 

to effective teaching? 

3. Design: This evaluation brief synthesizes the descriptive findings from three IES-funded 

studies on teacher distribution that have been peer-reviewed. The brief presents the findings 

from each study using the same approach, measuring whether disadvantaged students had less 

effective teaching on average than other students. The sample, collectively, spans 17 states. 

4. Estimated or Actual Completion Date: The evaluation brief was released in January 2014. 

5. Key Findings:

 Disadvantaged students received less effective teaching on average. Based on data from 

29 districts in grades 4–8 and two states in grades 4 and 5, disadvantaged students 

received less effective teaching in a given year than other students in those grades. The 

average disparity in teaching effectiveness was equivalent to about four weeks of learning 

for reading and two weeks for math. For context, the overall achievement gap for 

disadvantaged students in grades four through eight is equivalent to about 24 months in 

reading and 18 months in math. Study authors estimate differences in teaching 

effectiveness for one year represent 4 percent of the existing gap in reading and 2 to 

3 percent in math. 

 Access to effective teaching varied across districts. The size of the differences in effective 

teaching in a given year between disadvantaged and non-disadvantaged students varied 

across the 29 districts studied. The disparities for each district ranged from no statistically 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 126 

significant difference to a difference equivalent to 14 weeks of learning in reading and math 

in grades 4 through 8. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144010/pdf/20144010.pdf

Evaluation of the Teacher Incentive Fund: Implementation and Early Impacts of Pay-forPerformance After One Year 

1. Study Purpose: The Teacher Incentive Fund (TIF) is authorized by Title V, Part D of ESEA 

and annual appropriations acts. The purpose of the TIF program is to develop and implement 

performance-based compensation systems (PBCSs) for teachers, principals, and other 

personnel in high-need schools. Research indicates that high quality teachers are critical to 

raising student achievement in low-performing schools, but schools most in need often have 

difficulty in attracting and retaining high-quality teachers. Performance pay is a policy promoted 

by the TIF program to improve the quality of teachers in high-need schools. This evaluation 

studies performance pay that provides substantial and differentiated bonus pay to highperforming teachers in low-performing schools with high-need students. 

2. Key Questions Addressed:

 What are the characteristics of all TIF grantee districts and their performance-based 

compensation systems? What implementation experiences and challenges did TIF districts 

encounter? 

 How do teachers and principals in schools that did or did not offer pay-for-performance 

bonuses compare on key dimensions, including their understanding of TIF program 

features, exposure to TIF activities, allocation of time, and attitudes toward teaching and the 

TIF program? 

 What is the impact of pay-for-performance bonuses on students’ achievement on state 

assessments of math and reading? 

 How do pay-for-performance bonuses affect educator mobility, including whether mobility 

differs by educator effectiveness? 

 What performance-based compensation system features are associated with student 

achievement or educator mobility? 

3. Design: Study schools were randomly assigned within a grant to either implement all 

components of the PBCS or the PBCS with a 1 percent across-the-board bonus in place of the 

differentiated effectiveness incentive component of the PBCS. Data collection will include a 

grantee survey, a survey of teachers and principals, teacher and principal school assignment 

records, student record information (such as student demographics and student test scores), 

and grantee interviews to document implementation information, as well as to conduct impact 

analyses. 

4. Estimated or Actual Completion Date: The first report was released in September 2014. The 

second report, which will include impacts, is scheduled for completion in 2015. 

5. Key Findings: 

 Fewer than half of all 2010 TIF districts reported implementing all four required components 

of the TIF program, although most implemented three of the four components. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 127 

 Across all 2010 TIF grantees, districts expected to award a pay-for-performance bonus to 

more than 90 percent of eligible educators, with the average payout about 4 percent of the 

average educators’ salary. The districts expected a maximum pay-for-performance bonus 

for teachers that was twice as large as the average bonus, and a maximum bonus for 

principals that was 50 percent larger than the average bonus. 

 Many educators demonstrated a misunderstanding of the performance measures and the 

pay-for-performance bonuses used for TIF. 

 Most teachers and principals reported being satisfied with their professional opportunities, 

school environment, and the TIF program. Educators in schools that offered pay-forperformance bonuses tended to be less satisfied than those in schools that did not offer 

such bonuses. However, educators in schools offering pay-for-performance bonuses were 

more satisfied with the opportunity to earn additional pay, and a greater percentage 

indicated feeling increased pressure to perform due to the TIF program. 

6. Links to Additional Information: http://ies.ed.gov/ncee/pubs/20144019/pdf/20144019.pdf

Transfer Incentives for High-Performing Teachers: Final Results from a Multisite 

Randomized Experiment 

1. Study Purpose: This evaluation studies implementation of a policy, known to participating 

study school districts as the Talent Transfer Initiative (TTI), that provides incentives to identified 

high value-added teachers to teach in low-performing schools with high-need students. 

2. Key Questions Addressed:

 What can we learn from the implementation of TTI? Specifically, what can we learn about 

timing and scale of implementation, who transfers, and from where they transfer? 

 What were the intermediate impacts on participating schools? Specifically, how did TTI 

affect the dynamics within the school, such as the allocation of resources, staffing patterns, 

assignment of students to teachers and courses, and school climate? 

 What was TTI’s impact on student test scores? 

 What was TTI’s impact on teacher retention? 

3. Design: The study is being conducted in 10 school districts (168 school-grade teams in 

112 schools), and the design consists of segmenting the schools within districts to those eligible 

and not eligible for the treatment (the pay incentive). The treatment-eligible schools are 

randomly assigned to receive the treatment or not. Using value-added analysis, high-performing 

teachers teaching in the non-eligible schools are identified. The two-year treatment, conducted 

in school years 2009–10 and 2010–11 (in seven of the districts) and 2010–11 and 2011–12 (in 

an additional three districts), consists of hiring among the pool of those identified as high 

performing and interested in teaching in the treatment schools. The control schools follow 

normal hiring practices. Program transfer teachers receive a transfer incentive of $10,000 for 

each of the two years that they remain in the treatment school. Existing teachers in studyeligible schools that meet program criteria and remain in their school receive a retention 

payment of $5,000 a year. Data collection includes measures of teacher characteristics and 

hiring experiences, district/school hiring experiences and practices, and student achievement 

obtained from administrative records. 

4. Estimated or Actual Completion Date: The final report was released in November 2013. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 128 

5. Key Findings: The report examined the willingness of teachers to transfer when offered an 

incentive, teacher retention in the schools to which they transferred, and the impact on student 

achievement at the low-performing schools. The study found that: 

 The transfer incentive successfully attracted high value-added teachers to fill targeted 

vacancies. 

 The transfer incentive had a positive impact on teacher-retention rates during the payout 

period; retention of the high-performing teachers who transferred was similar to their 

counterparts in the fall immediately after the last payout. 

 The transfer incentive had a positive impact on math and reading achievement at the 

elementary school level in each of the two years after transfer. These impacts were 

equivalent to raising achievement by between 4 and 10 percentile points relative to all 

students in their home state. 

 There were no impacts—positive or negative—on achievement in middle schools. 

 Author calculations suggest that this transfer incentive intervention in elementary schools 

would save approximately $13,000 per grade per school compared to the cost of class size 

reduction aimed at generating the same size impacts. However, overall cost effectiveness 

can vary depending on a number of factors, such as teacher retention rates after the last 

installments of the incentive are paid out after the second year. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144003/pdf/20144003.pdf

Other 

Case Studies of Schools Receiving School Improvement Grants: Findings After the First 

Year of Implementation 

1. Study Purpose: School Improvement Grants are authorized by Title I, Section 1003(g) of the 

ESEA. The purpose of the grants—awarded based on the Title I funding formula to states, 

which then competitively distribute the funds to districts applying on behalf of their eligible 

schools—is to support the turnaround of the nation’s persistently lowest-achieving schools. To 

qualify for the three-year grant, schools must (among other requirements) be willing to 

implement one of four prescribed intervention models: turnaround, restart, closure, or 

transformation. About $546 million was allocated in FY 2009 for SIG with a supplement of 

$3 billion from ARRA. With the possibility of rollover funds, this amounts to a $3.5 billion 

injection into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This 

study will provide descriptively rich, primarily qualitative information for a small set of schools 

receiving SIG in the first cohort to implement an intervention model beginning in the 2010–11 

school year. 

2. Key Questions Addressed:

 What is the background and context of these persistently lowest-achieving schools? How do 

the leadership and staff in these schools define the performance problem, and to what do 

they attribute their problems? 

 What leadership styles do the principals of these persistently lowest-achieving schools 

exhibit? What actions do these schools engage in to try to improve their history of low 

performance? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 129 

 What is the change process in these persistently lowest-achieving schools, particularly in 

terms of school climate and staff capacity? 

 What improvements do school stakeholders perceive during the three-year course of SIG, 

and do these improvements appear to be sustainable? 

3. Design: This study employs a school-level case study design. A core sample of 25 SIG 

schools was purposively selected from six states to represent a range of geographic regions, 

urbanicities, school sizes, racial/ethnic compositions, socioeconomic statuses, SIG intervention 

models, and SIG funding levels, among other factors. Data collection took place over three 

school years, beginning in spring 2011 and concluding in spring 2013, and included interviews 

with each state’s SIG leaders, a teacher survey, and site visits to the case study schools, which 

included an analysis of fiscal records, as well as interviews and focus groups with district 

officials, principals, teachers, parents, union officials, external support providers, and students. 

4. Estimated or Actual Completion Date: The first report, based on the first year of data, was 

released in 2014. The final report, based on the second and third year of data, is scheduled for 

completion in 2015. 

5. Key Findings:

 Findings after the first year of implementation in the 25 “core” sample schools reveal that 

while all were low-performing, the schools differed in their community and fiscal contexts, 

performance and reform histories, interpretations of the causes of—and potential solutions 

for—their performance problems, and perceptions of improvement after the first year of SIG. 

 However, most schools did report that their improvement strategies and actions during the 

first year of SIG were a continuation of activities or plans that predated SIG, and few schools 

appeared to have experienced a disruption from past practice as of spring 2011. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144015/pdf/20144015.pdf 

A Focused Look at Rural Schools Receiving School Improvement Grants 

1. Study Purpose: School Improvement Grants are authorized by Title I, Section 1003(g) of the 

ESEA. The purpose of the grants—awarded based on the Title I funding formula to states, 

which then competitively distribute the funds to districts applying on behalf of their eligible 

schools—is to support the turnaround of the nation’s persistently lowest-achieving schools. To 

qualify for the three-year grant, schools must (among other requirements) be willing to 

implement one of four prescribed intervention models: turnaround, restart, closure, or 

transformation. About $546 million was allocated in FY 2009 for SIG with a supplement of 

$3 billion from ARRA. With the possibility of rollover funds, this amounts to a $3.5 billion 

injection into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This 

report will provide descriptively rich, primarily qualitative information for a small set of rural 

schools receiving SIG in the first cohort to implement an intervention model beginning in the 

2010–11 school year. 

2. Key Questions Addressed:

 What are the context and challenges of these rural SIG schools that are trying to turn 

around a history of low performance? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 130 

 How did these rural SIG schools perceive their rural context to influence the recruitment and 

retention of teachers and the engagement of parents, and what improvement actions did 

they implement in these two areas? 

3. Design: This study employs a school-level case study design. A sample of nine SIG schools 

located in rural areas was purposively selected from four states to represent a range of 

geographic regions, school sizes, racial/ethnic compositions, socioeconomic statuses, SIG 

intervention models, and SIG funding levels, among other factors. Data were collected in spring 

2012, and included interviews with each state’s SIG leaders, a teacher survey, and site visits to 

the case study schools, which included analysis of fiscal records, as well as interviews and 

focus groups with district officials, principals, teachers, parents, union officials, external support 

providers, and students. 

4. Estimated or Actual Completion Date: The report was released in 2014.

5. Key Findings: 

 Although rural SIG schools reported some challenges that nonrural SIG schools have also 

reported, such as low student motivation and staff morale, the rural schools reported 

additional challenges resulting from their schools’ remote locations and large catchment 

areas. For example, respondents reported that these rural characteristics affected the 

recruitment or retention of teachers and, to a lesser extent, parents’ involvement in the 

schools. 

 School and district administrators in eight of the nine schools suggested that long teacher 

commutes or isolated communities posed challenges to recruiting or retaining teachers. To 

counter these challenges, respondents in two schools reported offering direct support for 

teacher commutes (for example, gas stipends or vans), and respondents in three schools 

reported offering signing bonuses to incoming teachers. 

 School and district administrators and teaching staff in the nine schools mentioned multiple 

factors limiting parent involvement in school-based activities. Respondents from five schools 

perceived that a lack of access to transportation limited parent involvement, whereas 

respondents from three schools noted that the distance between schools and parents’

homes was a contributing factor. Four schools focused on hiring or expanding the role of 

parent liaisons to increase parent involvement. 

6. Links to Additional Information: http://ies.ed.gov/ncee/pubs/20144013/pdf/20144013.pdf 

A Focused Look at Schools Receiving School Improvement Grants That Have High 

Percentages of English Language Learner Students 

1. Study Purpose: School Improvement Grants are authorized by Title I, Section 1003(g) of the 

ESEA. The purpose of the grants—awarded based on the Title I funding formula to states, 

which then competitively distribute the funds to districts applying on behalf of their eligible 

schools—is to support the turnaround of the nation’s persistently lowest-achieving schools. To 

qualify for the three-year grant, schools must (among other requirements) be willing to 

implement one of four prescribed intervention models: turnaround, restart, closure, or 

transformation. About $546 million was allocated in FY 2009 for SIG with a supplement of 

$3 billion from ARRA. With the possibility of rollover funds, this amounts to a $3.5 billion 

injection into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This 

report will provide descriptively rich, primarily qualitative information for a small set of schools 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 131 

with high percentages of EL students, that are receiving SIG in the first cohort to implement an 

intervention model beginning in the 2010–11 school year. 

2. Key Questions Addressed:

 What are the context and challenges of these high-EL SIG schools that are trying to turn 

around a history of low performance? 

 How did these high-EL SIG schools approach the improvement process, and what supports 

did they provide to their ELs? 

 What capacity did these high-EL SIG schools have to address the unique needs of their 

ELs? 

3. Design: This study employs a school-level case study design. A sample of 11 SIG schools 

with high percentages of ELs was purposively selected from four states to represent a range of 

geographic regions, urbanicities, school sizes, racial/ethnic compositions, socioeconomic 

statuses, SIG intervention models, and SIG funding levels, among other factors. Data were 

collected in fall 2011, and included interviews with each state’s SIG leaders, a teacher survey, 

and site visits to the case study schools, which included analysis of fiscal records, as well as 

interviews and focus groups with district officials, principals, teachers, parents, union officials, 

external support providers, and students. 

4. Estimated or Actual Completion Date: The report was released in spring 2014. 

5. Key Findings: 

 Although all 11 schools reported providing specialized supports for EL students, the schools’ 

approaches to improvement during the initial phase of SIG appeared to include only 

moderate or limited attention to the unique needs of ELs.

 District and school administrators perceived challenges related to teachers’ expertise and 

skills in meeting the unique needs of ELs; however, teachers’ perceptions of their own 

capacity were more mixed. The capacity of the schools’ district offices to support ELs 

appeared to vary as well, with two small districts reporting no district-level staff with EL

training or experience and seven larger districts reporting district-level English-as-a-secondlanguage (ESL) departments with multiple trained staff members. 

 Schools that appeared to provide stronger attention to the unique needs of ELs in their 

improvement process were more likely to report having school staff dedicated to EL needs, 

such as EL coordinators, EL coaches, and ESL/bilingual teachers and tutors. Such schools 

also were more likely to be located in districts that reportedly provided expertise and an 

explicit focus on ELs within the context of SIG. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144014/pdf/20144014.pdf

Operational Authority, Support, and Monitoring of School Turnaround 

1. Study Purpose: The federal SIG program, to which $3 billion were allocated under ARRA, 

supports schools attempting to turn around a history of low performance. School turnaround 

also is a focus of RTT, another ARRA-supported initiative, which involved a roughly $4 billion 

comprehensive education reform grant competition for states. Given the size of these federal 

investments, in 2010 IES began to conduct a large-scale evaluation of RTT and SIG to better 

understand the implementation and impacts of these programs. The SIG component, in 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 132 

particular, focuses on a purposive sample of SIG-eligible schools, including (1) a group of 

schools that received SIG to implement one of four intervention models specified by the U.S. 

Department of Education and (2) a comparison group of schools from the same districts that 

were not implementing one of these four intervention models with SIG support. Though the 

results from this evaluation of SIG are not necessarily generalizable to SIG schools nationwide, 

they are nonetheless important because they add to the limited knowledge base about the 

implementation and impacts of SIG-funded school turnaround efforts. 

2. Key Questions Addressed: This report focuses on the implementation of SIG by examining 

three interrelated levers for school improvement: (1) school operational authority, (2) state and 

district support for turnaround, and (3) state monitoring of turnaround efforts. 

3. Design: SIG principles emphasize that school leaders should be given the autonomy to 

operate on matters such as staffing, calendars, and budgeting, but then also be appropriately 

supported and monitored by states and districts to ensure progress. It is thus of interest to 

document the actual policies and practices related to these three levers, and to see whether 

there are differences between study schools implementing a SIG-funded intervention model and 

comparison schools not implementing a SIG-funded intervention model. Findings are based on 

spring 2012 survey responses from 450 school administrators and interviews with 

administrators in the 60 districts and 21 of the 22 states where these schools are located. 

4. Estimated or Actual Completion Date: The report was released in January 2014. 

5. Key Findings:

 The most common area in which schools implementing and not implementing a SIG-funded 

intervention model reported having primary responsibility was their budgets (55 percent and 

54 percent). Fewer than half of the schools in both groups reported primary responsibility in 

the other seven operational areas examined, such as student discipline policies (38 percent 

and 35 percent), staffing (37 percent and 46 percent), assessment policies (25 percent and 

21 percent), and curriculum (18 percent and 16 percent). 

 The most common technical assistance and other supports for turnaround that states 

reported providing related to developing school improvement plans (20 of the 21 states 

interviewed) and identifying effective improvement strategies (19 of the 21 states 

interviewed). These two supports were also the ones districts and schools most frequently 

reported receiving. Schools implementing a SIG-funded intervention model were no more 

likely than non-implementing schools to report receiving supports in nine of twelve areas 

examined, including working with parents, school improvement planning, and recruiting or 

retaining teachers. 

 All 21 of the states interviewed reported being responsible for monitoring low-performing 

schools, although just 13 of them reported that districts were also responsible. State 

monitoring almost universally took the form of analyzing student data (21 states) and 

conducting site visits (20 states), and to a lesser extent having discussions with 

parents/community (16 states) and surveying school staff (12 states). Most states also 

reported that monitoring not only served accountability purposes, but also was used for 

formative purposes, such as to assess implementation fidelity (14 states) and identify 

additional supports for schools (14 states). 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144008/
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 133 

State Implementation of Reforms Promoted Under the Recovery Act 

1. Study Purpose: ARRA provided an unprecedented $100 billion of funding for the U.S. 

Department of Education. While the initial goal of this money was to deliver emergency funding 

for education and government services, ARRA is also being used as an opportunity to spur 

innovation and reform at different levels of the education system. In turn, ARRA provides a 

unique opportunity to foster school improvements and to learn from reform efforts. Although 

funds are being distributed through different grant programs, their goals and strategies are 

complementary, if not overlapping, as are likely recipients. For this reason, data collection and 

analysis took place across grant programs (i.e., was “integrated”), rather than separately for 

each set of grantees, allowing for a broad assessment of ARRA as a whole. The Department 

seeks to understand through this evaluation how states, districts, and schools are working to 

implement the education reforms promoted by ARRA. 

2. Key Questions Addressed: 

First report: 

 At the state and local levels, who were the recipients of ARRA funds? To what extent did 

child poverty, state fiscal condition, student achievement, and other variables relate to 

funding? 

Interim report: 

 To what extent did SEAs report implementing key reform strategies promoted by the 

Recovery Act in the 2010–11 school year? 

 How much of the 2010–11 school year implementation reflects progress since the Recovery 

Act? 

 What were the greatest reform implementation challenges for SEAs in the 2010–11 school 

year? 

Final report: 

 To what extent were states implementing the key education reform strategies promoted by 

the Recovery Act in 2010–11 in the areas of standards and assessment, data systems, 

educator workforce development, and support for low performing schools? 

 To what extent did 2010–11 implementation of key education reforms reflect progress since 

the Recovery Act funds were initially distributed in 2009–10?

 What were the greatest challenges experienced by states working to implement key 

education reforms in the 2010–11 school year? 

3. Design: This study is primarily based on data from surveys of all 50 SEAs and DC, and a 

nationally representative sample of school districts administered between spring 2011 and 

2012. Survey respondents were the chief state school officer or other state agency officials 

designated by the chief as most knowledgeable about the topics in the survey. Descriptive and 

correlational analyses are used to answer the study’s research questions.

4. Estimated or Actual Completion Date: A first report, titled “State and District Receipt of 

Recovery Act Funds,” was released in 2012. An interim report was released in January 2014. 

The final report is scheduled for release in 2015. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 134 

5. Key Findings of Interim Report: 

 Almost all SEAs provided guidance for choosing and implementing one of the four school 

intervention models the Department recommended to improve low performing schools, while 

only two reported supporting teacher evaluation models that included the complete set of 

criteria (e.g., use of student achievement gains) that the Recovery Act promoted. 

 Difficulty in measuring student growth for teachers of non-tested subjects was the challenge 

reported by the largest number of SEAs. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144011/pdf/20144011.pdf

State Requirements for Teacher Evaluation Policies Promoted by Race to the Top 

1. Study Purpose: Congress appropriated approximately $5.05 billion for the RTT program 

between 2009 and 2012, including approximately $4.35 billion through ARRA. The RTT initiative 

encouraged states to implement education policies in six core areas, including teacher 

evaluations. Evaluations are an important strategy to assess teacher quality, and there is 

growing consensus about the need for evaluation systems that could yield higher-quality 

information to improve teacher performance. An increasing body of evidence also suggests that 

some of the teacher evaluation policies promoted by RTT, such as using multiple measures and 

multiple rating categories, could help to produce more valid and reliable estimates of teacher 

quality. Given the substantial federal investment in RTT to promote certain teacher evaluation 

policies, it is important to learn about the policies that states are actually requiring. Many states 

have started to alter their policies, but knowledge of their progress remains limited. 

2. Key Questions Addressed: This brief describes the extent to which states required teacher 

evaluation policies aligned with the RTT initiative as of spring 2012.

3. Design: This brief examines the presence of state-level requirements for certain practices but 

not the actual district- or school-level implementation of such practices. Findings are based on 

interviews with administrators from 49 states and the District of Columbia (12 Round 1 and 2 

RTT states, 7 Round 3 RTT states, and 31 non-RTT states). 

4. Estimated or Actual Completion Date: The report was released in April 2014. 

5. Key Findings:

 States, on average, reported requiring less than half of eight teacher evaluation policies 

aligned with RTT priorities, although the number of policies required by RTT states was 

higher than non-RTT states (3.7 policies for Round 1 and 2 RTT states, 3.6 for Round 3 RTT 

states, and 2.2 for non-RTT states). 

 States’ reported teacher evaluation policies were most aligned with RTT priorities focused 

on using multiple measures to evaluate teacher performance (30 states); using multiple 

rating categories to classify teacher performance (31 states); and conducting annual 

evaluations (25 states). 

 States’ reported teacher evaluation policies were least aligned with RTT priorities focused 

on using evaluation results to inform decisions regarding career advancement (one state) 

and compensation (six states for annual salary increases, and five states for performancebased compensation). 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144016/pdf/20144016.pdf
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 135 

Policy and Program Studies Service (PPSS)

National Assessment of Career and Technical Education: Final Report

1. Study Purpose: The Carl D. Perkins Career and Technical Education Act of 2006 (Perkins IV)

was intended to raise the academic and technical rigor of secondary and postsecondary CTE 

instruction in order to prepare students for entry into high-skill, high-wage, or high-demand 

occupations. Perkins IV introduced new accountability requirements as well as a requirement 

that local subgrantees offer one or more programs of study (POS)—career pathways that help 

students make the transition from secondary to postsecondary education while pursuing an 

industry-recognized credential, postsecondary certificate, or degree. This final report of the 

congressionally mandated National Assessment of Career and Technical Education (NACTE) 

summarizes the most recent available data on the implementation of Perkins IV, as well as 

student participation and outcomes for CTE more generally. 

2. Key Questions Addressed: 

 Has student participation in CTE programs changed?

 How were Perkins IV funds allocated and used? 

 How were states and local subgrantees developing and implementing POS? 

 How were states implementing accountability provisions, and what measurement issues 

affect the validity and reliability of the accountability data that were reported? 

 Are educational and employment outcomes showing positive results for CTE participants? 

3. Design: This report summarizes data from studies commissioned for the NACTE, reviews of 

existing research, and analyses of extant data from state performance reports and from the 

National Center for Education Statistics. A study of Perkins IV implementation included surveys 

of all states and a representative sample of 2,041 LEAs and 1,006 IHEs in fall 2009. 

4. Completion Date: The final report was released in September 2014. 

5. Key Findings: 

 Secondary CTE coursetaking declined slightly from 1990 to 2009, while academic 

coursetaking increased. Some occupational areas saw large increases in CTE coursetaking, most notably health sciences and public services. 

 Subgrantees most commonly used Perkins IV funds for equipment, career guidance, and 

academic counseling. 

 Both state and local CTE directors reported incomplete compliance, as of 2008–09, with 

requirements that POS link secondary and postsecondary education by aligning course 

sequences. 

 States and local subgrantees are not required to report on POS participation and outcomes, 

and there are no national data on the number of students participating in POS or the 

outcomes they achieve. 

 States showed substantial variation in their definitions of CTE concentrators and in the 

specific measures they used for performance indicators. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 136 

 Studies that used quasi-experimental methods to control for student background found little 

or no relationship between CTE coursetaking and academic achievement. 

 Education and employment outcomes varied considerably by CTE field. For example, 

completion of a postsecondary degree or certificate in the same field was most common 

among high school graduates who concentrated in health sciences. 

6. Link to Additional Information: http://www2.ed.gov/rschstat/eval/sectech/nacte/careertechnical-education/final-report.pdf

Summary of Performance Evaluations Expected During FY 2015 and 

FY 2016

National Center for Education Evaluation and Regional Assistance

Literacy 

Effectiveness of Interventions for Improving Reading Achievement of Struggling 

Adolescent Readers: An Assessment and Summary of the Evidence 

1. Study Purpose: Striving Readers is a discretionary grant program focused on raising reading 

achievement of middle school and high school students through intensive interventions for 

struggling readers and enhancing the quality of literacy instruction across the curriculum. The 

2006 and 2009 cohorts of grantees were required to conduct rigorous, experimental 

evaluations. This study summarizes the evidence of the effectiveness of interventions aimed at 

struggling adolescent readers, including—but not limited to—the evaluations of the 16 Striving 

Readers grantees. 

2. Key Question Addressed: 

 What is the evidence of the effectiveness of interventions aimed at struggling adolescent 

readers? 

3. Design: The study is descriptive; it provided technical assistance to the local evaluations of 

Striving Readers grantees and also reviewed existing literature on interventions to raise reading 

achievement among struggling adolescent readers. The report will synthesize the evidence of 

the effectiveness of these interventions. 

4. Estimated or Actual Completion Date: The final report is scheduled for completion in winter 

2015.

Early Childhood Language Development 

1. Study Purpose: Differences between the reading skills of disadvantaged children and their 

more advantaged peers have been measured nationally as early as kindergarten entry in the 

Early Childhood Longitudinal Study. The focus of this evaluation is the identification of 

classroom practices that are associated with improved student language development and 

comprehension. Such practices could be used in a future rigorous evaluation of these 

strategies. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 137 

2. Key Question Addressed: 

 What classroom practices are associated with greater student progress in language 

development, background knowledge, and comprehension in prekindergarten through third 

grade? 

3. Design: The evaluation will analyze the relationships between the study’s observational 

measures of classroom practices and direct assessments of students collected during the 

2011–12 school year in 83 Title I schools. 

4. Estimated or Actual Completion Date: The final report is scheduled for completion in 2015. 

Evaluation of Response to Intervention Practices for Elementary School Reading 

1. Study Purpose: The focus of this evaluation is the implementation and impact of Response to 

Intervention (RtI) practices for struggling readers in elementary school. Response to 

Intervention (RtI) is a multistep approach to providing early and more intensive intervention and 

monitoring within the general education setting. IDEA permits some Part B special education 

funds to be used for “early intervening services” such as RtI, and also permit districts to use RtI 

to inform decisions regarding a child’s eligibility for special education under the category of 

specific learning disabilities. 

2. Key Questions Addressed:

 How do RtI practices for early-grade reading vary across schools? 

 How do schools experienced with RtI vary the intensity of reading instruction to children 

based on student benchmark reading performance? 

 What are the effects on grade 1–3 reading achievement of providing intensive interventions 

to children who are on the margin of identification for reading difficulties? 

3. Design: The evaluation is relying on a combination of descriptive data collection from school 

staff and regression discontinuity methods to address the research questions, and is focusing 

on practices in place during the 2011–12 school year. 

4. Estimated or Actual Completion Date: The final report from this evaluation is scheduled for 

completion in 2015. 

Pathways to Career or College

Upward Bound at 50: Reporting on Implementation Practices Today 

1. Study Purpose: Upward Bound is the oldest and largest of the federal college access 

programs targeted to low-income students and those who would represent the first-generation 

of college completers in their families. First established in 1965, the program currently serves 

more than 60,000 high school students at a cost of about $4,300 per youth with a wide array of 

academic and college transition support services. While much about the structure of Upward 

Bound and the services to be offered are prescribed in legislation, little is currently known about 

the intensity, duration, and mix of services provided by projects or about how they are delivered. 

Because of the importance of its mission, and the comprehensiveness and costs of its services, 

Upward Bound has long been of interest to policymakers. This report describes the approaches 

to providing program services as reported by Upward Bound project directors. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 138 

2. Key Questions Addressed: 

 Within the core service areas of the program, where do projects focus their efforts? 

 How are services delivered to Upward Bound participants? 

 In what ways does the focus or delivery of services vary across different Upward Bound 

projects? 

3. Design: This descriptive report relies primarily on findings from a summer 2013 survey of all 

regular Upward Bound project directors. 

4. Estimated or Actual Completion Date: The report is scheduled for completion in 2015. 

School Choice 

Evaluation of Conversion Magnet Schools 

1. Study Purpose: Since the mid-1970s, magnet schools have been critical to school districts’

efforts to implement voluntary desegregation plans and, in some cases, court desegregation 

orders. More recently, they have become an important component of public school choice as 

well as a strategy used by districts aiming to improve the achievement of all students, 

particularly students who are disadvantaged. Since 1985, the Office of Innovation and 

Improvement’s (OII) Magnet Schools Assistance Program (MSAP; funded at $100 million in 

FY 2010) has provided grants to school districts to support magnet programs with the specific 

goals of reducing, eliminating, or preventing minority group isolation, improving student 

achievement, and promoting diversity and increasing choice in public schools through the 

development of innovative educational methods and practices. 

Despite the popularity and longevity of this educational strategy, there have been few rigorous 

studies of the effects on important student outcomes, with mixed results. Drawing broad 

conclusions is particularly challenging because the structure and target population of magnet 

school programs are varied. This more targeted evaluation of magnet schools focuses on a 

single, common category of school receiving funding through MSAP: elementary schools that 

convert to become whole-school magnets. 

2. Key Questions Addressed: 

 How did student composition change, in terms of diversity and achievement, in 

neighborhood schools that converted to magnet schools? 

 To what extent were changes in diversity and achievement in these schools related to 

magnet conversion? 

3. Design: A feasibility study determined that there was a sufficient number of neighborhood 

schools that converted to magnet schools (conversion magnets) funded in the two most recent 

grant cycles to focus on these schools. School records data (student achievement scores, 

demographic characteristics, and school attended) were collected for the 2005–06 through 

2010–11 school years, three years before and up to four years after the magnet school 

conversion. Descriptive analyses are being conducted to examine changes in diversity and 

achievement before versus after conversion for the entire student population in the schools in 

our sample, as well as for relevant subgroups within the schools. We will also compare changes 

over time for the conversion magnet schools to those for other neighborhood schools in their 

districts (comparative interrupted time series analysis) to explore the hypothesis that any 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 139 

changes we see are associated with the conversion itself and not other efforts underway in the 

districts. 

4. Estimated or Actual Completion Date: The final report is scheduled for completion in 2015. 

Students with Disabilities 

Preparing for Life After High School 

1. Study Purpose: The National Longitudinal Transition Study 2012 (NLTS 2012), also referred 

to as the Study of Transition Outcomes for Youth with Disabilities, is the third in a series 

examining the characteristics and school experiences of a nationally representative sample of 

youth with disabilities. NLTS 2012 focuses on youth ages 13 to 21 (in December 2011), but also 

includes a small sample of students without disabilities to enable direct comparisons of students 

with and without individualized education programs (IEPs). It is part of the congressionally 

mandated National Assessment of the IDEA and is supported with funds authorized under 

Section 664 of IDEA. 

2. Key Questions Addressed:

 What are the personal, family, and school characteristics of youth with disabilities in public 

schools across the country? 

 What regular education, special education, transition planning, and other relevant services 

and accommodations do youth with disabilities receive? 

 How do the services and accommodations differ from those of youth not served under IDEA, 

including those identified for services under Section 504 of the Rehabilitation Act? 

 How do the services and accommodations for youth with disabilities vary with the 

characteristics of youth? 

 How much have the services and accommodations of youth with disabilities changed over 

time? 

3. Design: Phase I of the study collects survey data (spring 2012–summer 2013) on a 

nationwide sample of 12,000 youth in school, of which 10,000 are students with IEPs across the 

federal disability categories. Surveys of both youth and their parents/guardians will be 

administered. 

4. Estimated or Actual Completion Date: Two reports are scheduled for completion in summer 

2015. 

Teacher Quality 

Teaching Residency Programs: Description of a New Model for Preparing Teachers for 

High-Need Schools 

1. Study Purpose: Teaching Residency Programs (TRPs) involve a year-long “clinical” 

experience (the “residency”) shadowing and co-teaching with an experienced mentor. TRPs 

also provide continued support and mentoring after participants become teachers of record. 

Before and during their residencies, participants in TRPs take coursework. The evaluation of 

TRPs, which focuses on residency programs that have received grants from the Teacher 

Quality Partnership (TQP) Program, will provide important descriptive and implementation 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 140 

information on TRPs, as well as information on the teacher retention outcomes of teachers who 

participate in TRPs. 

2. Key Questions Addressed:

 What are the characteristics of TRPs (e.g., length of overall program, criteria for selecting 

program participants)? What are the characteristics of participants in TRPs? 

 What are the retention rates of novice TRP teachers and their novice colleagues who did not 

go through TRPs? 

3. Design: Descriptive information concerning TQP grantees operating TRPs was collected 

through a survey administered in spring 2012. More detailed implementation information was 

collected through TRP program director interviews and surveys of residents and mentors, 

conducted within a subset of TRPs during spring 2012. Teacher mobility will be tracked through 

district records and teacher surveys in order to examine retention in the profession, district, and 

school, among novice TRP and novice non-TRP teachers in a subset of six districts. 

4. Estimated or Actual Completion Date: The first report was released in November 2014. A 

follow-up evaluation brief is expected in 2015. 

5. Key Findings:

 In keeping with their intended purpose and the grant requirements, the residency programs 

provided a fieldwork experience, typically with a trained and experienced mentor teacher, 

along with integrated coursework. Residents reported an increase in the number of days 

fully in charge of instruction between the first and second halves of their residency 

(21 versus 37 days, on average). Most TRP mentors had substantial teaching experience 

(10 years, on average), were trained by the residency program (averaging 37 hours of 

training), and had prior mentoring experience (an average of 3.5 semesters). TRPs included 

the equivalent of 10 courses, on average, with core emphasis on content and pedagogy, 

classroom management, and student assessment and a lesser emphasis on child 

development and education philosophy. Most residents reported that their fieldwork 

reinforced what they learned in their coursework and that their coursework was wellintegrated with their residency classroom experiences (83 and 68 percent of residents, 

respectively). 

 The residency programs somewhat broadened the pool of people entering the teaching 

profession in the participating districts. Novice teachers in the study who had completed a 

TRP appeared more likely than their non-TRP peers to have made a distinct career change 

when they joined their programs. For example, they were more likely than non-TRP teachers 

to report having worked in a full-time job other than teaching (72 percent versus 63 percent). 

However, novice TRP and non-TRP teachers had similar demographic characteristics (sex, 

race/ethnicity, and age). 

 Novice teachers from residency programs had similar retention rates to other novice 

teachers. Focusing on teachers after their first or second years of teaching, about 

92 percent of TRP teachers and 90 percent of non-TRP teachers reported staying in the 

same district from spring 2012 to fall 2012; about 4 percent of TRP teachers and 6 percent 

of non-TRP teachers were no longer teaching. None of these were statistically significant 

differences. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 141 

Other 

Adoption of Policies and Practices Promoted by Race to the Top and School 

Improvement Grants 

1. Study Purpose: RTT and SIG are signature Department programs in the Obama 

Administration, and both programs received substantial funding through ARRA. As part of 

OMB’s FY 2010 Evaluation Initiative, IES proposed to conduct an impact evaluation of the RTT 

and SIG programs, focusing on the initial general state competition for RTT and the first cohort 

of SIG schools implementing intervention models beginning in the 2010–11 school year. This is 

the first evaluation report of the Impact and Implementation Evaluation of RTT and SIG. 

2. Key Questions Addressed: 

 Which policies and practices promoted by the RTT program do RTT states report adopting, 

and how do they compare to the policies and practices that non-RTT states report adopting? 

 Is receipt of an RTT grant related to improvement in student outcomes? 

 Are SIG-funded schools adopting the improvement or turnaround strategies promoted by the 

four SIG intervention models, and how do they compare to strategies in schools not 

implementing a SIG-funded intervention model? How are states and districts supporting 

such efforts? 

 Does receipt of SIG funding to implement a school intervention model have an impact on 

outcomes for low-performing schools? 

 Is implementation of the four school intervention models and the strategies prescribed by 

those models related to improvement in outcomes for low-performing schools? 

3. Design: The RTT sample will include all 50 states and DC. Data from interviews with all 

states and DC will inform the first evaluation question. The second evaluation question will be 

addressed using a short interrupted time series design with state-level NAEP data comparing, 

before and after the RTT competition, states that were awarded an RTT grant to states that 

applied for but were not awarded an RTT grant. 

The SIG sample will include about 525 schools in 60 districts from 22 states. This sample will be 

purposively selected to support a regression discontinuity design to address the fourth 

evaluation question, exploiting cutoff rules that states used to identify their persistently lowestachieving schools as eligible for SIG to implement one of the four intervention models. Data 

from state and district interviews, as well as school surveys from the SIG sample, will inform the 

third and fifth evaluation questions. Student- and school-level achievement data will also be 

collected from administrative records up to the 2012–2013 school year to inform the fourth and 

fifth evaluation questions. 

4. Estimated or Actual Completion Date: The first report on early implementation findings for 

SIG and RTT is scheduled for completion in 2015. The final report on implementation and 

impacts is expected in 2016. 

Are Low-Performing Schools Adopting Practices Promoted by School Improvement 

Grants? 

1. Study Purpose: The SIG program is authorized through Title I of ESEA and provides 3-year 

awards to support turnaround in the nation’s persistently lowest-achieving schools. In FY 2009, 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 142 

the $546 million SIG appropriation was supplemented by $3 billion through ARRA, for a total of 

$3.5 billion. SIG funds are distributed to states by formula based on Title I allocations. States 

then competitively award funds to districts applying on behalf of their eligible schools. Schools 

receiving SIG must implement one of four prescriptive intervention models: turnaround, 

transformation, closure, or restart. Previous research provides evidence that low-performing 

schools adopt some practices promoted by the four models, but little is known about how 

schools combine these practices. This study describes the individual SIG-promoted 

improvement practices adopted by low-performing schools. 

2. Key Questions Addressed: This brief describes both the individual SIG-promoted 

improvement practices and the combinations of these practices that low-performing schools 

reported adopting. 

3. Design: Findings are based on spring 2013 survey responses from 480 school administrators 

in 60 districts and 22 states. The 480 schools are all low-performing schools, with some 

receiving SIG and others not. 

4. Estimated or Actual Completion Date: The report was released in October 2014. 

5. Key Findings: 

 Schools on average reported adopting 20 of 32 improvement practices promoted by the SIG 

transformation or turnaround models. 

 No school reported adopting all practices required under the transformation or turnaround 

models. 

 More than 96 percent of schools reported adopting each of the three most commonly 

adopted individual practices: using data to inform and differentiate instruction, increasing 

technology access for teachers or using computer-assisted instruction, and providing 

ongoing professional development that involves teachers working collaboratively or is 

facilitated by school leaders. 

 For 16 of the 32 practices examined, schools implementing a SIG model were statistically 

significantly more likely than schools not implementing one to report adopting that practice. 

 Almost every school reported adopting a unique combination of practices, but certain 

practices (for example, the three most commonly adopted practices listed above) were much 

more likely than others (for example, using financial incentives to recruit and retain effective 

teachers and principals) to be included in these combinations. 

6. Link to Additional Information: http://ies.ed.gov/ncee/projects/evaluation/other_racetotop.asp 

Building Teacher Capacity to Support ELLs in Schools Receiving SIG 

1. Study Purpose: School Improvement Grants are authorized by Title I, Section 1003(g) of the 

ESEA. The purpose of the grants—awarded based on the Title I funding formula to states, 

which then competitively distribute the funds to districts applying on behalf of their eligible 

schools—is to support the turnaround of the nation’s persistently lowest-achieving schools. To 

qualify for the three-year grant, schools must (among other requirements) be willing to 

implement one of four prescribed intervention models: turnaround, restart, closure, or 

transformation. About $546 million was allocated in FY 2009 for SIG with a supplement of 

$3 billion from ARRA. With the possibility of rollover funds, this amounts to a $3.5 billion 

injection into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 143 

study will provide descriptively rich, primarily qualitative information for a small set of SIG 

schools with high proportions of EL students that are receiving SIG in the first cohort to 

implement an intervention model beginning in the 2010–11 school year. 

2. Key Questions Addressed: This brief describes the efforts of SIG schools with high 

proportions of ELs to improve teachers’ capacity for serving ELs through staffing strategies and 

professional development. 

3. Design: This study employs a school-level case study design. A sample of 11 SIG schools 

with high percentages of ELs (a median of 45 percent ELs) was purposively selected from four 

states to represent a range of geographic regions, urbanicities, school sizes, racial/ethnic 

compositions, socioeconomic statuses, SIG intervention models, and SIG funding levels, among 

other factors. Findings are based on EL case study data collected in fall 2011, spring 2012, and 

fall 2012. 

4. Estimated or Actual Completion Date: The report is scheduled for completion in spring 2015. 

Case Studies of Schools Receiving School Improvement Grants: Final Report 

1. Study Purpose: School Improvement Grants are authorized by Title I, Section 1003(g) of 

ESEA. The purpose of the grants—awarded based on the Title I funding formula to states, 

which then competitively distribute the funds to districts applying on behalf of their eligible 

schools—is to support the turnaround of the nation’s persistently lowest-achieving schools. To 

qualify for the three-year grant, schools must (among other requirements) be willing to 

implement one of four prescribed intervention models: turnaround, restart, closure, or 

transformation. About $546 million was allocated in FY 2009 for SIG with a supplement of 

$3 billion from ARRA. With the possibility of rollover funds, this amounts to a $3.5 billion 

injection into the SIG program during the 2010–11, 2011–12, and 2012–13 school years. This 

study will provide descriptively rich, primarily qualitative information for a small set of schools 

receiving SIG in the first cohort to implement an intervention model beginning in the 2010–11 

school year. 

2. Key Questions Addressed:

 What is the background and context of these persistently lowest-achieving schools? How do 

the leadership and staff in these schools define the performance problem, and to what do 

they attribute their problems? 

 What leadership styles do the principals of these persistently lowest-achieving schools 

exhibit? What actions do these schools engage in to try to improve their history of low 

performance? 

 What is the change process in these persistently lowest-achieving schools, particularly in 

terms of school climate and staff capacity? 

 What improvements do school stakeholders perceive during the three-year course of SIG, 

and do these improvements appear to be sustainable? 

3. Design: This study employs a school-level case study design. A core sample of 25 SIG 

schools was purposively selected from six states to represent a range of geographic regions, 

urbanicities, school sizes, racial/ethnic compositions, socioeconomic statuses, SIG intervention 

models, and SIG funding levels, among other factors. Data collection took place over three 

school years, beginning in spring 2011 and concluding in spring 2013, and included interviews 

with each state’s SIG leaders, a teacher survey, and site visits to the case study schools, which 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 144 

included analysis of fiscal records, as well as interviews and focus groups with district officials, 

principals, teachers, parents, union officials, external support providers, and students. 

4. Estimated or Actual Completion Date: The first report, based on the first year of data, was 

released in 2014. The final report, based on the second and third year of data, is scheduled for 

completion in 2015. 

5. Key Findings from the Interim Report:

 Findings after the first year of implementation in the 25 “core” sample schools reveal that 

while all were low-performing, the schools differed in their community and fiscal contexts, 

performance and reform histories, interpretations of the causes of—and potential solutions 

for—their performance problems, and perceptions of improvement after the first year of SIG. 

 However, most schools did report that their improvement strategies and actions during the 

first year of SIG were a continuation of activities or plans that predated SIG, and few schools 

appeared to have experienced a disruption from past practice as of spring 2011. 

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20144015/pdf/20144015.pdf 

Evaluation of the Regional Educational Laboratories: Final Report 

1. Study Purpose: The RELs are a networked system of 10 organizations that serve the 

educational needs of 10 designated regions across the United States. The Department is 

authorized by the Education Sciences Reform Act (ESRA) to award contracts to 10 RELs to 

support applied research, development, wide dissemination, and technical assistance activities. 

The REL program is administered by the Knowledge Utilization Division of the National Center 

for Education Evaluation and Regional Assistance (NCEERA) within the Department’s IES, 

which was established by ESRA in 2002. The FY 2012 appropriation for the REL program was 

$57.426 million. 

2. Key Questions Addressed: 

 What activities did the RELs undertake to fulfill the missions specified in ESRA? 

 What were the technical quality and relevance of REL Fast Response Project reports 

published by IES and of the corresponding proposals? 

 What were the technical quality and relevance of REL impact study reports published by IES 

and of the corresponding proposals? 

 How relevant and useful were the REL technical assistance products to the needs of the 

states, localities, and policymakers in their regions? 

3. Design: This descriptive study is relying on a combination of extant data, FY 2010 interviews 

with REL directors, and FY 2012 surveys of potential REL customers from state and local 

educational agencies. Panels of experts met during FY 2010 and FY 2012 and rated the quality 

and relevance of REL Fast Response Project proposals and final reports and REL impact study 

proposals and final reports. 

4. Estimated or Actual Completion Date: An interim report was released in 2013. The final 

report is scheduled for completion in winter 2015. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 145 

5. Key Findings from the Interim Report: 

 REL staff members reported activities under each of the 10 missions of the REL program 

specified in ESRA. The statement of work (SOW) for the REL contracts in place between 

2006 and 2011 aligned explicitly with 6 of the 10 statutory missions for the REL program. 

Four additional statutory missions were not explicitly in the SOW for the RELs, but RELs 

reported activities under those missions as well. 

 As of December 1, 2009, the 10 RELs had submitted 297 proposals to IES to conduct Fast 

Response Projects (FRPs), of which 46 percent (137) were accepted for performance under 

the REL contracts. 

 The IES-published FRP reports received a mean quality rating of 3.81 on a 5-point scale, 

while the corresponding proposals received a mean quality rating of 3.24. Both of these 

means fell between the categories of “adequate” and “strong” quality.

 The IES-published FRP reports received a mean relevance rating of 3.64 on a 5-point scale, 

while the corresponding proposals received a mean relevance rating of 3.39. Both of these 

means fell between the categories of “adequate” relevance and “relevant.”

6. Link to Additional Information: http://ies.ed.gov/ncee/pubs/20134014/ 

Implementation and Impact Evaluation of Race to the Top and School Improvement 

Grants 

1. Study Purpose: RTT is a competitive grant program of the Department that funds states and 

districts planning to implement comprehensive education reform in one or more core areas. 

Since its first awards in 2010, RTT has funded general state competitions, state competitions 

focused on early learning, and district competitions focused on personalized learning. With 

funds from ARRA, the general state competition awarded approximately $4 billion to states in 

support of comprehensive K–12 education reform in four core areas: teachers and leaders, 

standards and assessments, data systems, and school turnaround. The SIG program is 

authorized through Title I of the ESEA and provides three-year awards to support turnaround in 

the nation’s persistently lowest-achieving schools. In FY 2009, the $546 million SIG 

appropriation was supplemented by $3 billion through ARRA, for a total of $3.5 billion. SIG 

funds are distributed to states by formula based on Title I allocations. States then competitively 

award funds to districts applying on behalf of their eligible schools. Schools receiving SIG must 

implement one of four prescriptive intervention models: turnaround, transformation, closure, or 

restart. Both RTT and SIG received substantial funding through ARRA. This study will examine 

the implementation and impacts of RTT and SIG, focusing on the initial general state 

competition for RTT and the first cohort of SIG schools implementing intervention models 

beginning in the 2010–11 school year. 

2. Key Questions Addressed:

 Which policies and practices promoted by the RTT program do RTT states report adopting, 

and how do they compare to the policies and practices that non-RTT states report adopting? 

 Is receipt of an RTT grant related to improvement in student outcomes? 

 Are SIG-funded schools adopting the improvement or turnaround strategies promoted by the 

four SIG intervention models, and how do they compare to strategies in schools not 

implementing a SIG-funded intervention model? How are states and districts supporting 

such efforts? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 146 

 Does receipt of SIG funding to implement a school intervention model have an impact on 

outcomes for low-performing schools? 

 Is implementation of the four school intervention models and the strategies prescribed by 

those models related to improvement in outcomes for low-performing schools? 

3. Design: The RTT sample includes all 50 states and DC. Data from interviews with all states 

and DC will inform the first evaluation question. The second evaluation question will be 

addressed using a short interrupted time series design with state-level NAEP data. The SIG 

sample includes about 525 schools in 60 districts from 22 states, purposively selected to 

support a regression discontinuity design to address the fourth evaluation question. Data from 

state/district interviews and school surveys will inform the third and fifth evaluation questions. 

Administrative data on student and school achievement are being collected through the 

2012–13 school year to inform the fourth and fifth evaluation questions. 

4. Estimated or Actual Completion Date: The first report is scheduled for completion in 2015, 

and the final report is expected in 2016. 

State Capacity to Support the Turnaround of Low Performing Schools 

1. Study Purpose: RTT is a Department-sponsored competitive grant program that funds states 

and districts planning to implement comprehensive education reform in one or more core areas. 

Since its first awards in 2010, RTT has funded general state competitions, state competitions 

focused on early learning, and district competitions focused on personalized learning. With 

funds from ARRA, the general state competition awarded approximately $4 billion to states in 

support of comprehensive K–12 education reform in four core areas: teachers and leaders, 

standards and assessments, data systems, and school turnaround. This study examines the 

implementation of RTT, focusing on state capacity to support school turnaround. 

2. Key Questions Addressed: This brief describes states’ reported priorities for school 

turnaround, gaps in state-level expertise to support turnaround, and approaches to enhance 

capacity to support school turnaround. Consideration is given to whether these reports have 

changed between 2012 and 2013, and whether they differ for RTT and non-RTT states. 

3. Design: Findings are based on interviews in spring 2012 and 2013 with administrators from 

50 states and the District of Columbia (12 Round 1 and 2 RTT states, 7 Round 3 RTT states, 

and 32 non-RTT states). 

4. Estimated or Actual Completion Date: The report is scheduled for completion in 2015. 

Policy and Program Studies Service (PPSS)

State and District Implementation of Education for Homeless Children and Youth 

Program 

1. Study Purpose: The Education for Homeless Children and Youth (EHCY) program provides 

grants to states and other grantees with the goal of ensuring that homeless children and youth 

have access to the same free, appropriate public education as do other children and youth. This 

study examined the implementation of the EHCY program at the state and school district levels 

based on surveys and analysis of extant data. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 147 

2. Key Questions Addressed: 

 How do states allocate EHCY funds? 

 What are the roles and responsibilities of the state coordinator and district liaison? What 

services do districts provide to homeless children and youth? 

 How do states monitor and provide technical assistance to districts as part of the EHCY 

program? What technical assistance needs do state coordinators and district liaisons 

report? 

 What data do states and districts collect about homeless children and youth? 

 What do state coordinators and district liaisons perceive as barriers to school success for 

homeless children and youth? 

3. Design: The study surveyed all state coordinators for the ECHY program and a sample of 

448 district liaisons of EHCY districts for the 2010–11 school year. The study also examined 

state-submitted data on homeless children and youth. 

4. Estimated Completion Date: The final report is scheduled for completion in early 2015.

Evaluation of State and Local Implementation of Title III Standards, Assessment, and 

Accountability Systems 

1. Study Purpose: Under ESEA Title III accountability provisions, annual measurable 

achievement objectives (AMAOs) are used to provide district-level and state-level summary 

information about EL students’ progress in attaining English proficiency and meeting state 

standards in English/language arts and mathematics. This report uses a rigorous method for 

examining the performance and progress of EL students that controls for compositional shifts in 

the EL student population and is able to disaggregate performance by subgroup characteristics 

(e.g., poverty status, level of English language proficiency). 

2. Key Questions Addressed: 

 Are EL students making progress in learning English? Does the amount of progress vary by 

student characteristics? 

 How long does it take for EL students to attain proficiency on the state ELP assessments, 

and to be redesignated as former English learners? 

 Are EL students making progress toward meeting achievement targets on state content 

assessments in English/language arts and mathematics? 

 Are achievement gaps narrowing for EL students and for former EL students? 

3. Design: The study used longitudinally linked student-level assessment data in four 

jurisdictions—New York, Texas, and two school districts in California—to examine student 

progress for consistent cohorts of ELs, former ELs, and non-ELs who could be followed over a 

period of at least three years. Most analyses examined a three-year period, from 2006–07 

through 2008–09 for New York and from 2005–06 through 2007–08 for the other three 

jurisdictions. The analytic sample of students is not representative of all students served by the 

four jurisdictions, so the findings cannot be generalized to all ELs, non-ELs, or former ELs in those 

states and districts, or to the nation as a whole. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 148 

4. Estimated Completion Date: The final report is scheduled for completion in early 2015. 

Equitable Distribution of Effective Teachers: State and Local Responses to Federal 

Initiatives 

1. Study Purpose: This report provides a broad overview of state efforts, as of the 2011–12 

school year, to monitor equitable access to qualified and effective teachers among schools; 

develop and adopt multiple measures of teacher performance to rate teachers among at least 

three performance levels; and implement targeted strategies for promoting equitable access to 

qualified and effective teachers in schools serving high proportions of poor and/or minority 

students, including the use of measures of teacher qualifications and teacher performance in the 

implementation of these strategies. 

2. Key Questions Addressed: 

 What measures did states use to monitor equitable access to qualified and effective 

teachers among schools? 

 To what extent were states developing or using multiple measures of teacher performance 

to rate teachers among at least three performance levels? 

 What strategies did states use to promote equitable access to qualified and effective 

teachers in schools serving high proportions of poor and/or minority students? 

3. Design: This report is based on telephone interviews with officials in SEAs in all states, the 

District of Columbia, and Puerto Rico. Study staff also reviewed extant sources including equity 

plans, Consolidated State Performance Reports, reports from federal monitoring visits, and 

other information on state websites. 

4. Estimated Completion Date: The final report is scheduled for completion in early 2015.

Highly Qualified Teachers (HQT) Enrolled in Alternative Routes to Certification and 

Licensure 

1. Study Purpose: Title I of ESEA requires that all teachers of core academic subjects be highly 

qualified—meaning they have a bachelor’s degree, full state certification, and demonstrate 

expertise in the subject matter they teach—and that states work to ensure that poor and 

minority students are not taught by teachers who are inexperienced, out-of-field, or unqualified 

at higher rates than their peers. Under federal regulations (34 CFR § 200.56), the Department 

allows teachers who are enrolled in programs that provide alternate routes to certification or 

licensure to be considered highly qualified before meeting full state certification requirements. At 

the same time, alternate route programs have traditionally been used to address shortages, 

which are especially acute in urban areas, special education, and in certain content areas such 

as mathematics and science. This congressionally mandated report examines the extent to 

which students in four different subgroups are taught by teachers enrolled in alternative 

certification programs and classified as “highly qualified” under the ESEA: (1) students with 

disabilities, (2) English learners, (3) students in rural areas, and (4) students from low-income 

families. 

2. Key Questions Addressed: 

 What percentage of HQTs across the nation, in each state, and in each district are enrolled 

in an alternative certification program? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 149 

 What percentage of highly qualified special education teachers across the nation, in each 

state, and in each district are enrolled in an alternative certification program? 

 What percentage of highly qualified Title III teachers across the nation, in each state, and in 

each district are enrolled in an alternative certification program? 

 What percentage of HQTs working in rural areas across the nation and in each state are 

enrolled in an alternative certification program? 

 What percentage of HQTs working in high-poverty areas across the nation and in each state 

are enrolled in an alternative certification program? 

3. Design: This report provides a descriptive analysis of data collected by the Department from 

states in summer and fall 2014 on the numbers of all teachers, special education teachers, and 

Title III teachers who are enrolled in alternative certification programs and deemed highly 

qualified. 

4. Estimated Completion Date: The final report is scheduled for completion in early 2015. 

Evaluation of the Carol White Physical Education Program 

1. Study Purpose: The Carol M. White Physical Education Program (PEP) provides grants to 

school districts or community-based organizations (CBOs) to initiate, expand, and improve 

physical education for students in kindergarten through grade 12. In 2010, the Department 

revised PEP regulations to: include an increased focus on healthy eating habits and nutrition; 

require projects to conduct a needs assessment, and require that districts use four designated 

modules of the Center for Disease Control’s School Health Index (SHI); use this assessment to 

develop project goals and plans to address identified weaknesses; encourage grantees to 

establish partnerships with community entities; encourage grantees to collect and use body 

mass index (BMI) data; and establish new performance measures and standard data-collection 

methods. This report provides the results of the implementation study of the 76 projects (64 

district-led and 12 CBO-led) funded by FY 2010 PEP grant awards—the first cohort of grantees 

under the revised PEP. 

2. Key Questions Addressed: 

 What were the results of PEP grantees’ self-assessments of their physical activity, health, 

and nutrition policies and practices? 

 What physical activity and nutrition policy efforts did PEP grantees report? 

 What types of physical fitness and nutrition activities did PEP grantees report? 

 What role did community partnerships play in PEP projects? 

 What were PEP grantees’ experiences collecting and using BMI data?

 What implementation challenges and lessons learned did PEP grantees report? 

3. Design: This study used a mixed-methods research design that included surveys and case 

studies. Surveys of project directors were conducted in 2011 and 2013, the first and third years 

of the grant period. Case studies of five PEP projects (three districts and two CBOs) gathered 

more in-depth information from 59 interviewees regarding grantees’ experiences with 

community partners and BMI data collection. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2015. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 150 

Study of the Early Implementation of the ESEA Flexibility Provision 

1. Study Purpose: On September 23, 2011, President Obama announced the opportunity for 

states to request flexibility of certain ESEA provisions to help states move forward with efforts to 

improve student achievement and the quality of instruction in their schools. With input from 

broad and diverse stakeholders, the Department designed the ESEA Flexibility initiative so that 

states could build on their existing reform efforts to: (1) transition to college- and career-ready 

standards and aligned assessments; (2) implement new systems of differentiated recognition, 

accountability, and support for schools; (3) evaluate and support teacher and principal 

effectiveness in more robust ways; and (4) reduce duplication and unnecessary burden on 

school districts by removing any duplicative and burdensome reporting requirements. The early 

implementation study will provide documentation of early state- and local-level implementation 

of the flexibility provision. This information is intended to inform federal and state policymakers 

and to support the Department’s monitoring and technical assistance to states. The study 

includes reviews of state and local documents; interviews with education officials at the state, 

district, and school level in early implementing states; and analysis of baseline student outcome 

and other data obtained through extant data sources. 

2. Key Questions Addressed: 

States & Districts: 

 Approaches to accountability and support under ESEA Flexibility. What prior experiences 

and rationales guided state approaches to accountability and support under the ESEA 

Flexibility initiative, including states’ selection of annual measurable objectives (AMOs)? 

 Components of a system of accountability and support under ESEA Flexibility. What are the 

primary components of state accountability and support systems under ESEA Flexibility, 

including state processes for identifying reward schools and for identifying priority, focus, 

and other low-performing schools? 

 Intervention and support for low-performing schools and districts. What interventions and 

supports are states and districts implementing in low-performing schools, and what 

approaches are states taking to identify and intervene in low-performing districts? 

 Issues related to ESEA Flexibility implementation. To what extent and what challenges are 

states and districts experiencing related to ESEA Flexibility implementation, and to what 

extent and in what ways are states and districts communicating with stakeholders regarding 

ESEA Flexibility provisions? 

Schools: 

 To what extent did principals report understanding their state’s accountability system under 

ESEA Flexibility, and how did they perceive the communication efforts of their states and 

districts about the system? 

 How did principals perceive the criteria that their states were using to identify low-performing 

schools under ESEA Flexibility? 

 What improvement strategies did principals describe implementing? 

 What state and district support did principals report receiving to support their improvement 

efforts? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 151 

 What incentives or recognition for success did principals of reward schools report receiving 

from their states as a result of their reward designation? 

3. Design: The primary data sources for this study included state ESEA Flexibility applications, 

state and district policy documents, and interviews with officials identified by state and district 

leaders as holding primary responsibility for accountability and support systems under ESEA

Flexibility. The sample was purposefully selected to include officials representing 12 states, 

24 districts, and 36 schools. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2015.

Study of English Learners with Special Needs

1. Study Purpose: The purpose of this study is to describe, on an exploratory scale, common 

issues in the identification of ELs for special education. The report will focus on four topics: 

procedures and practices used in identification; assessment data and instruments used to bring 

ELs into special education; personnel, qualifications, and professional development to assist in 

identification; and instruments on which decisions are often made to exit ELs with disabilities out 

of their language instruction programs. 

2. Key Questions Addressed: 

 What procedures, practices, and instruments are used to assess and identify ELs with 

disabilities, and how do these differ from those used with non-ELs? 

 What are the roles, backgrounds, and qualifications of school and district personnel involved 

in the assessment and identification of ELs with disabilities? 

 What challenges do districts and schools encounter in the assessment and identification of 

disabilities among ELs and what strategies do they use to overcome these challenges? 

 What procedures and practices do districts and schools use to exit ELs with disabilities from 

language instruction educational programs, and what are the challenges they face? 

3. Design: The study relied on a purposive sample of a diverse group of six case study districts, 

containing 18 schools total, which the study team visited in spring 2013. Study methods onsite 

largely consisted of interviews of a large number and types of stakeholders involved in ESL, 

special education, and related support services at both the school district and school levels. 

Other data collected included special education data; published reports, guides, and regulations 

related to identification; and data on district and school websites. The data were coded to one or 

more of the four key questions and the findings were compared to those from a prior review of 

research. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2015.

Evaluation of the Teacher Incentive Fund: Final Report

1. Study Purpose: This study examines program implementation in the first two cohorts of TIF 

grantees (2006 and 2007), which included 33 grantees. All 33 grantees implemented 

performance pay systems for principals and other school administrators; 31 grantees also 

included teachers in their performance pay systems. The final report is a complement to a 2012 

report on early implementation of the program, and analyzes award payouts and educator 

perspectives on a variety of issues related to implementation and sustainability. 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 152 

2. Key Questions Addressed: 

 How did the size and composition of incentive awards vary across educators and grantees? 

 How did participating educators perceive the fairness and effectiveness of the performance 

pay systems? 

 Did grantees expect to continue their performance pay projects beyond the life of the TIF 

grant? 

3. Design: Data collection included surveys of a representative sample of teachers and 

principals in schools served by TIF projects; financial data for awards paid to educators; 

interviews with TIF project directors, teachers, principals, and other stakeholders; and reviews 

of extant documents. The surveys were conducted in spring 2011, and the incentive award 

payout data are primarily for the 2010–11 school year. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2015.

Study of Emerging Teacher Evaluation Systems

1. Study Purpose: This study will provide descriptive information on the design and early 

implementation of teacher evaluation systems in eight school districts. The findings are intended 

to help other districts and states learn from the experiences of eight districts featured in the 

study, and apply the design and implementation lessons to their own work as it relates to 

teacher evaluation and support. 

2. Key Questions Addressed: 

 What steps did the districts take to design a teacher evaluation system? What are the 

purported purposes of the new teacher evaluation systems in the case study districts? With 

what stakeholders did district staff consult when designing the new systems? What are the 

types of measures districts included in their teacher evaluation systems and how are each of 

those measures weighted? 

 What steps did the districts take prior to full implementation to test the system and prepare 

teachers and staff to implement it? 

 How do the districts structure and conduct the classroom observation component of their 

teacher evaluation systems? 

 How do the districts analyze student achievement and other data to evaluate teacher 

performance? 

 How do the districts use teacher evaluation results to make human resource decisions? To 

what extent are professional development decisions and opportunities tied to evaluation 

results? 

 What administrative structures do/did districts use to support their new teacher evaluation 

system? 

3. Design: This descriptive study relies on interviews with key district administrators, principals, 

teachers, and representatives of community stakeholder groups, from eight districts, who were 

involved in the development and early implementation of the respective districts’ teacher 

evaluation system. Given the limited sample, the findings cannot be generalized to other 

districts.
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 153 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2015.

Feasibility Study on Improving the Quality of School-Level Expenditure Data 

1. Study Purpose: This feasibility study explores options for improving the completeness, 

consistency, and accuracy of school-level expenditures that would be most appropriate to 

include for assessing equity, as well as technical and operational challenges for including 

certain types of expenditures. The study also examines the experiences of states and districts 

that currently track school-level expenditures, including variations in definitions and practices 

used in these jurisdictions, the types of changes to accounting systems and practices that are 

required, and the potential cost of implementing such strategies. 

2. Key Questions Addressed: 

 In states and districts that currently track expenditures at the school level, what types of 

personnel and non-personnel expenditures are included in the school-level data? 

 What is the quality of existing school-level expenditure data? What specific steps could be 

taken to improve the completeness, consistency, and accuracy of these data? 

 What changes would states and districts need to make to track expenditures at the school 

level if they do not currently do so? What costs have states and districts incurred to 

implement such data systems? 

3. Design: The study will explore options for improving the completeness, consistency, and 

accuracy of school-level expenditure reporting by: (1) convening an expert panel to identify 

specific challenges and potential solutions; (2) interviewing fiscal staff in five states and four 

school districts that have finance systems that track school-level finance data; and (3) collecting 

and analyzing available school-level expenditure data in the selected states and districts. 

4. Estimated Completion Date: The final report is scheduled for completion in summer 2015. 

National High School Reform Study 

1. Study Purpose: This nationally representative survey of high school administrators is 

examining strategies that schools are using to reduce students’ likelihood of dropping out of 

high school and to increase their likelihood of attaining a high school credential. The study 

examines dropout prevention strategies used by high schools, with an emphasis on those 

supported by the High School Graduation Initiative, authorized under Title I, Part H of ESEA, as 

amended. The survey seeks information on what schools are implementing what activities with 

what students under what circumstances or conditions. 

Data from the National High School Reform Study will inform a descriptive report on the 

strategies that high schools are using to help students graduate from high school, especially 

students at risk for dropping out and students in high schools with low graduation rates. 

Information from the survey will fill critical information gaps about the use and prevalence of 

high school reform strategies to support at-risk youth. 

2. Key Question Addressed: 

 What are the prevalence and characteristics of key high school reforms, especially dropout 

prevention strategies, operating in the nation’s public high schools, overall and in high 

schools with low graduation rates? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 154 

3. Design: A 30-minute survey administered online to a nationally representative sample of 

approximately 2,000 high school administrators.

4. Estimated Completion Date: Data collection is scheduled to begin in early 2015 and continue 

through June 2015. The final report is scheduled for completion in spring 2016. 

Study of Experiences and Needs of Rural Education Achievement Program (REAP) 

Grantees and Subgrantees 

1. Study Purpose: The purpose of this descriptive study is to examine how grantees and 

subgrantees use REAP funds provided through the Small, Rural School Achievement (SRSA) 

and Rural and Low-Income Schools (RLIS) programs—on their own or in combination with other 

federal funds—as well as to explore technical assistance needs related to REAP. The study will 

inform program operations, technical assistance to grantees, and future reauthorization of the 

programs. 

2. Key Questions Addressed: 

 What are REAP grantees’ experiences of grant eligibility determination? To what extent do 

districts contact, or are they contacted by, the SEA regarding eligibility? 

 On what types of activities are REAP funds spent and in what proportions? What are 

grantees’ experiences of deciding how to target funds (e.g., processes and factors, 

personnel and stakeholders involved, integration in larger spending decisions)? Do districts 

perceive a need for greater flexibility in the use of federal Title VI funds? How does this vary 

by grantee characteristic? 

 How do grantees use REAP funds in combination with other federal program funds? To 

what extent are other Department or other federal programs (e.g., E-Rate, USDA Rural 

Development grants) considered or integrated? What are the challenges in spending REAP 

funds? 

 How many SRSA-eligible grantees use REAP flexibility? Does the use of flexibility differ by 

grantee characteristics? If flexibility is not used, why not? Exactly how do SRSA grantees 

tend to use flexibility? 

 What are the major challenges and technical assistance needs that REAP grantees face 

(e.g., eligibility determination and compliance with program requirements)? Are there 

technical assistance needs that grantees perceive to be unaddressed? For what? What is 

the frequency of technical assistance use by grantees? What is the purpose of such 

assistance, and who offers it? 

 Is there anything that districts or states would recommend changing about the REAP 

program administration or design? 

3. Design: The study consists of: 1) a survey of a sample of approximately 1,000 SRSA 

grantees and RLIS subgrantee districts; 2) telephone interviews with a sample of 30 SRSA 

grantees and RLIS subgrantees; and 3) telephone interviews with REAP coordinators in all 

states receiving REAP funds. Data collection will begin in winter 2015 and is expected to be 

completed by spring 2015. At the state level, the study will include interviews with all state 

REAP coordinators about state goals and priorities, the planning process for use of RLIS funds, 

the eligibility process for districts, management and distribution of SRSA and RLIS funds, and 

recommendations for the program. At the school district level, the study will include an online 

survey of a nationally representative sample of REAP coordinators about the REAP eligibility 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 155 

determination process, administration of REAP, and challenges and technical assistance needs 

related to REAP, as well as telephone interviews with a subsample of 30 districts. The 

interviews will include questions about program administration, technical assistance needs, and 

recommendations for changing the REAP program to better meet the needs of rural districts. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2016. 

Case Studies of the Implementation and Use of Kindergarten Entry Assessments 

1. Study Purpose: The purpose of the KEAs implementation case studies is to document the 

processes, accomplishments, challenges, and solutions of four states implementing KEAs and 

to share what they have learned with federal and state policymakers and the field. Of particular 

interest is to identify what is working well in states that are early adopters of KEAs. This 

information is needed to support the technical assistance efforts of the Department and to 

inform KEA efforts across the nation. 

2. Key Questions Addressed: 

 How have KEAs been developed and adopted by four states that were early to adopt a 

comprehensive KEA? 

 How are KEAs being implemented by 12 school districts within four states that were early 

adopters? 

 How do these four states and their districts and schools communicate and use KEA results 

to inform policy and practice? 

 What lessons did states, districts, and schools learn about KEA adoption, implementation, 

and use? 

3. Design: The nested sample design includes four states, 12 districts, and 24 schools. Data 

collection will include review of state and local documents, phone interviews with SEA preschool 

directors and professional development staff who facilitate district-level training, and in-person 

interviews with district administrators, principals, kindergarten teachers, and other staff involved 

in local administration of KEAs. 

4. Estimated Completion Date: The final report is scheduled for completion in spring 2016.

Study on Sustaining the Positive Effects of Preschool 

1. Study Purpose: This study will accomplish two goals: (1) summarize what is known about 

policies, programs, and practices that can help students in grades K–3 build on the positive 

effects of preschool or make cognitive, social-emotional, and academic gains; and (2) provide 

detailed case study descriptions of five innovative programs that aim to help disadvantaged 

students sustain and build upon preschool’s positive effects. 

2. Key Questions Addressed: 

 What is known about policies, programs, and practices that can help students in 

grades K–3 build on the positive effects of preschool or make cognitive, social-emotional, 

and academic gains? 
APPENDICES

SUMMARY OF PERFORMANCE EVALUATIONS

FY 2014 Annual Performance Report and FY 2016 Annual Performance Plan—U.S. Department of Education 156 

 What are the characteristics of innovative programs (including school-based interventions, 

district-wide approaches, and comprehensive state policies) that sustain the positive effects 

of preschool for students, particularly for disadvantaged students? 

 Based on what research or experiences did the designers of these programs develop the 

program structures and content? 

 What are some of challenges of implementing these programs and how have staff and 

leaders tried to overcome these challenges? 

3. Design: The literature review consists of a two-stage systematic review of three topics: 

(1) preschool and K–3 alignment, (2) differentiated instruction, and (3) interventions to sustain 

the effects of preschool. Stage 1 will be a descriptive mapping review. Stage 2 will be an 

evidence review and will only apply for studies of the three topics that employed a rigorous 

design. The case studies will examine five sites at the elementary (K–3) level that are 

implementing policies, programs, or strategies related to one of the above three topic areas and 

have successfully demonstrated gains for disadvantaged students (e.g., economically 

disadvantaged children; children who are learning English as their second language; and 

children who come from homeless, neglected, or migrant populations) in cognitive, socialemotional, or academic domains. 

4. Estimated Completion Date: The literature review is scheduled for completion in summer 

2015. The final case study report is scheduled for completion in fall 2016.