Student Achievement: Schools Use Multiple Strategies to Help Students Meet Academic Standards, Especially Schools with Higher Proportions of Low-Income and Minority Students
Released: 2009-11-16T00:00:00

Letter

Background

NCLBA reauthorized the Elementary and Secondary Education Act of 1965 (ESEA) and built upon accountability requirements created under a previous reauthorization, the Improving America’s Schools Act of 1994 (IASA). Under ESEA, as amended, Congress sought to improve student learning by incorporating academic standards and assessments in the requirements placed on states. Academic standards, which describe what students should know and be able to do at different grade levels in different subjects, help guide school systems in their choice of curriculum and help teachers plan for classroom instruction. Assessments, which states use to measure student progress in achieving the standards, are required to be administered by states.

NCLBA further strengthened some of the accountability requirements contained in ESEA, as amended. Specifically, NCLBA’s accountability provisions require states to develop education plans that establish academic standards and performance goals for schools to meet AYP and lead to 100 percent of their students being proficient in reading, math, and science by 2014. This proficiency must be assessed annually in reading and math in grades 3 through 8 and periodically in science, whereas assessments were required less frequently under the IASA. Under NCLBA, schools’ assessment data generally must be disaggregated to assess progress toward state proficiency targets for students in certain designated groups, including low-income students, minority students, students with disabilities, and those with limited English proficiency. Each of these groups must make AYP in order for the school to make AYP. Schools that fail to make AYP for 2 or more consecutive years are required to implement various improvement measures identified in NCLBA, and these measures are more extensive than those required under IASA. Education, which has responsibility for general oversight of NCLBA, reviews and approves state plans for meeting AYP requirements. As we have previously reported, Education had approved all states’ plans—fully or conditionally—by June 2003.

NCLBA also recognizes the role of teachers in providing a quality education by requiring states to ensure that all teachers in core academic subjects are “highly qualified.” Under this requirement, teachers generally must have a bachelor’s degree, be fully certified, and demonstrate their knowledge of the subjects they teach. Previously, there were no specific requirements regarding teacher quality under ESEA, as amended.

Principals and Teachers Used a Variety of Instructional Practices to Help Students Meet Standards, and Many of These Practices Were Used More Frequently at Schools with Higher Proportions of Low- Income and Minority Students

According to our analysis of NLS-NCLB data from Education, most principals reported their schools focused on multiple instructional practices in their voluntary school improvement efforts. These strategies were used more often at schools with higher proportions of low-income students (“high-poverty schools”) and schools with higher proportions of minority students (“high-minority schools”) than at schools with lower proportions of low-income students (“low-poverty schools”) and schools with lower proportions of minority students (“low-minority schools”). Likewise, the survey of math teachers in California, Georgia, and Pennsylvania indicates teachers were using many different instructional practices in response to their state tests, and teachers at high-poverty and high-minority schools were more likely than teachers at low-poverty and low-minority schools to have been increasing their use of some of these practices. Some researchers we spoke with suggested that differences in the use of these instructional practices exist because schools with low- poverty or low-minority student populations might generally be meeting accountability standards and, therefore, would need to try these strategies less frequently.

Principals at High-Poverty and High-Minority Schools Emphasized Certain School Improvement Strategies More Than Principals at Other Schools

According to nationally representative data from Education’s NLS-NCLB, in school year 2006-2007 most principals focused on multiple strategies in their school improvement efforts. The survey asked principals the extent to which their schools were focusing on ten different strategies in their voluntary school improvement initiatives. The three most common strategies were: (1) using student achievement data to inform instruction and school improvement; (2) providing additional instruction to low- achieving students; and (3) aligning curriculum and instruction with standards and/or assessments. (See fig. 1.) Nearly all school principals placed a major or moderate focus on three or more surveyed strategies in their school improvement efforts, and over 80 percent of principals placed a major or moderate focus on six or more strategies. However, as Education’s report on the survey data cautioned, the number of improvement strategies emphasized was not necessarily an indication of the intensity or quality of the improvement efforts.

While nearly all principals responded that they used multiple improvement strategies, there were statistically significant differences in principals’ responses across a range of school characteristics, including percentage of the school’s students receiving free or reduced price lunch (poverty), percentage of minority students, the school’s location, and AYP status. For example, when comparing schools across poverty levels, we found that principals at high-poverty schools were two to three times more likely than principals at low-poverty schools to focus on five particular strategies in their school improvement efforts: Restructuring the school day to teach core content areas in greater depth; Increasing instructional time for all students (e.g., by lengthening the school day or year, shortening recess); Providing extended-time instructional programs (e.g., before-school, after- school, or weekend instructional programs); Implementing strategies for increasing parents’ involvement in their children’s education; and Increasing the intensity, focus, and effectiveness of professional development.

Likewise, when comparing schools across minority levels, we found that principals at high- and moderate-minority schools were approximately two to three times more likely than principals at low-minority schools to make six particular school improvement strategies a major or moderate focus of their school improvement efforts. For instance, principals at schools with a high percentage of minority students were more than three times as likely as principals at schools with a low percentage of minority students to provide extended-time instruction such as after-school programs. A school’s location was associated with differences in principals’ responses about the strategies they used as well: principals at rural schools were only about one-third to one-half as likely as central city schools to make five of these school improvement strategies a moderate or major focus of their school improvement efforts.

When we compared principal responses based on AYP status, there was some evidence of a statistically significant association between AYP status and the extent to which principals focused these strategies in their school improvement efforts, but it was limited when the other variables such as poverty and minority were taken into account. AYP status had some correlation with the demographic characteristics of poverty and minority, and those characteristics explained the patterns of principals’ responses more fully than the AYP characteristic. However, our analysis generally showed that schools that had not made AYP were more likely to make six of these school improvement strategies a moderate or major focus of their school improvement plan than schools that had made AYP. Additionally, Education reported that schools identified for improvement under NCLBA—that is, schools that have not made AYP for two or more consecutive years—were engaged in a greater number of improvement efforts than non-identified schools. Therefore, principals of the non- identified schools may have been less likely than principals of identified schools to view specific strategies as a major or moderate focus.

We spoke with several researchers about the results of our analysis of the principals’ responses, especially at high-poverty and high-minority schools. While the researchers could not say with certainty the reasons for the patterns, they noted that high-poverty and high-minority schools tend to be most at risk of not meeting their states’ standards, so that principals at those schools might be more willing to try different approaches. Conversely, the researchers noted that principals at schools meeting standards would not have the same incentives to adopt as many school improvement strategies.

Most Math Teachers in Three Surveyed States Have Increased Their Use of Certain Instructional Practices in Response to State Tests, especially in High-Poverty and High- Minority Schools

The RAND survey of elementary and middle school math teachers in California, Georgia and Pennsylvania showed that in each of the three states at least half of the teachers reported increasing their use of certain instructional practices in at least five areas as a result of the statewide math test (see fig. 2). For example, most teachers in Pennsylvania responded that due to the state math test they: (1) focused more on standards, (2) emphasized assessment styles and formats, (3) focused more on subjects tested, (4) searched for more effective teaching methods, and (5) spent more time teaching content.

As we did with the survey responses of principals, we analyzed the teacher survey data to determine whether math teachers’ responses differed by school characteristics for poverty, minority, location, and AYP status. As with the principals’ responses, we found that elementary and middle school math teachers in high-poverty and high-minority schools were more likely than teachers in low-poverty and low-minority schools to report increasing their use of certain instructional practices, and this pattern was consistent across the three states (see fig. 3). For example, 69 percent of math teachers at high-poverty schools in California indicated they spent more time teaching test-taking strategies as opposed to 38 percent of math teachers in low-poverty schools. In Georgia, 50 percent of math teachers in high-poverty schools reported offering more outside assistance to non- proficient students in contrast to 26 percent of math teachers in low- poverty schools. Fifty-one percent of math teachers at high-poverty schools in Pennsylvania reported focusing more attention on students close to proficiency compared to 23 percent of math teachers doing so in low poverty schools.

Similar to what our poverty analysis showed, survey responses provided some evidence that math teachers in high-minority schools were more likely than those in low-minority schools to change their instructional practices. Math teachers at high-minority schools in each of the three states, as compared to those at low-minority schools, were more likely to: rely on open-ended tests in their own classroom assessments; increase the amount of time spent teaching mathematics by replacing non- instructional activities with mathematics instruction; focus on topics emphasized in the state math test; and teach general test-taking strategies.

We also analyzed the RAND data with regard to school location and a school’s AYP status, but results from these characteristics were not significant for as many instructional practices.

As we did regarding the survey responses of principals, we spoke to several researchers, including the authors of the three-state teacher study, regarding possible reasons for the patterns we saw in the teacher survey data. The researchers we spoke with provided similar possible reasons for the patterns in the teacher survey as they did for patterns in the principal survey. For instance, the researchers noted that high-poverty and high- minority schools are more likely to be at risk of failing to meet the state standards, which might prompt teachers to try different approaches. On the other hand, the researchers stated that teachers at those schools meeting the standards would not have the same incentives to change their instructional practices.

Research Shows That Standards-based Accountability Systems Can Influence Instructional Practices through Standards and Assessments in Both Positive and Negative Ways

Research shows that using a standards-based curriculum that is aligned with corresponding instructional guidelines can positively influence teaching practices. Specifically, some studies reported changes by teachers who facilitated their students developing higher-order thinking skills, such as interpreting meaning, understanding implied reasoning, and developing conceptual knowledge, through practices such as multiple answer problem solving, less lecture and more small group work. Additionally, a few researchers we interviewed stated that a positive effect of NCLBA’s accountability provisions has been a renewed focus on standards and curriculum.

However, some studies indicated that teachers’ practices did not always reflect the principles of standards-based instruction and that current accountability policies help contribute to the difficulty in aligning practice with standards. Some research shows that, while teachers may be changing their instructional practices in response to standards-based reform, these changes may not be fully aligned with the principles of the reform. That research also notes that the reliability in implementing standards in the classroom varied in accordance with teachers’ different beliefs in and support for standards-based reform as well as the limitations in their instructional capabilities. For example, one observational study of math teachers showed that, while teachers implemented practices envisioned by standards-based reform, such as getting students to work in small groups or using manipulatives (e.g., cubes or tiles), their approaches did not go far enough in that students were not engaged in conversations about mathematical or scientific concepts and ideas. To overcome these challenges, studies point to the need for teachers to have opportunities to learn, practice, and reflect on instructional practices that incorporate the standards, and then to observe their effects on student learning. However, some researchers have raised concerns that current accountability systems’ focus on test scores and mandated timelines for achieving proficiency levels for students do not give teachers enough time to learn, practice, and reflect on instructional practices and may discourage some teachers from trying ambitious teaching practices envisioned by standards-based reform.

Another key element of a standards-based accountability system is assessments, which help measure the extent to which schools are improving student learning through assessing student performance against the standards. Some researchers note that assessments are powerful tools for managing and improving the learning process by providing information for monitoring student progress, making instructional decisions, evaluating student achievement, and evaluating programs. In addition, assessments can also influence instructional content and help teachers use or adjust specific classroom practices. As one synthesis concluded, assessments can influence whether teachers broaden or narrow the curriculum, focus on concepts and problem solving—or emphasize test preparation over subject matter content.

In contrast, some of the research and a few experts we interviewed raised concerns about testing formats that do not encourage challenging teaching practices and instructional practices that narrow the curriculum as a result of current assessment practices. For example, depending on the test used, research has shown that teachers may be influenced to use teaching approaches that reflect the skills and knowledge to be tested. Multiple choice tests tend to focus on recognizing facts and information while open-ended formats are more likely to require students to apply critical thinking skills. Conclusions from a literature synthesis conducted by the Department of Education stated that “ teachers respond to assessment formats used, so testing programs must be designed and administered with this influence in mind. Tests that emphasize inquiry, provide extended writing opportunities, and use open-ended response formats or a portfolio approach tend to influence instruction in ways quite different from tests that use closed-ended response formats and which emphasize procedures.” We recently reported that states have most often chosen multiple choice items over other item types of assessments because they are cost effective and can be scored within tight time frames. While multiple choice tests provide cost and time saving benefits to states, the use of multiple choice items make it difficult, if not impossible, to measure highly complex content. Other research has raised concerns that, to avoid potential consequences from low-scoring assessment results under NCLBA, teachers are narrowing the curriculum being taught—sometimes referred to as “teaching to the test”—either by spending more classroom time on tested subjects at the expense of other non-tested subjects, restricting the breadth of content covered to focus only on the content covered by the test, or focusing more time on test-taking strategies than on subject content.

Research Highlights Some Potentially Successful Practices for Improving Student Achievement, although Experts Contend That Methodological Issues Constrain Reaching Definitive Conclusions about What Works

Our literature review found some studies that pointed to instructional practices that appear to be effective in raising student achievement. But, in discussing the broader implications of these studies with the experts that we interviewed, many commented that, taken overall, the research is not conclusive about which specific instructional practices improve student learning and achievement.

Some researchers stated that this was due to methodological issues in conducting the research. For example, one researcher explained that, while smaller research studies on very specific strategies in reading and math have sometimes shown powerful relationships between the strategy used and positive changes in student achievement, results from meta- analyses of smaller studies have been inconclusive in pointing to similar patterns in the aggregate. A few other researchers stated that the lack of empirical data about how instruction unfolds in the classroom hampers the understanding about what works in raising student performance.

A few researchers also noted that conducting research in a way that would yield more conclusive results is difficult. One of the main difficulties, as explained by one researcher, is the number of variables a study may need to examine or control for in order to understand the effectiveness of a particular strategy, especially given the number of interactions these variables could have with each other. One researcher mentioned cost as a challenge when attempting to gather empirical data at the classroom level, stating “teaching takes place in the classroom, but the expense of conducting classroom-specific evaluations is a serious barrier to collecting this type of data.” Finally, even when research supports the efficacy of a strategy, it may not work with different students or under varying conditions. In raising this point, one researcher stated that “educating a child is not like making a car” whereby a production process is developed and can simply be repeated again and again. Each child learns differently, creating a challenge for teachers in determining the instructional practices that will work best for each student.

Some of the practices identified by both the studies and a few experts as those with potential for improving student achievement were: Differentiated instruction. In this type of instruction, teaching practices and plans are adjusted to accommodate each student’s skill level for the task at hand. Differentiated instruction requires teachers to be flexible in their teaching approach by adjusting the curriculum and presentation of information for students, thereby providing multiple options for students to take in and process information. As one researcher described it, effective teachers understand the strategies and practices that work for each student and in this way can move all students forward in their learning and achievement.

More guiding, less telling. Researchers have identified two general approaches to teaching: didactic and interactive. Didactic instruction relies more on lecturing and demonstrations, asking short answer questions, and assessing whether answers are correct. Interactive instruction focuses more on listening and guiding students, asking questions with more than one correct answer, and giving students choices during learning. As one researcher explained, both teaching approaches are important, but some research has shown that giving students more guidance and less direction helps students become critical and independent thinkers, learn how to work independently, and assess several potential solutions and apply the best one. These kinds of learning processes are important for higher-order thinking. However, implementing “less instruction” techniques requires a high level of skill and creativity on the part of the teacher.

Promoting effective discourse. An important corollary to the teacher practice of guiding students versus directing them is effective classroom discussion. Research highlights the importance of developing students’ understanding not only of the basic concepts of a subject, but higher-order thinking and skills as well. To help students achieve understanding, it is necessary to have effective classroom discussion in which students test and revise their ideas, and elaborate on and clarify their thinking. In guiding students to an effective classroom discussion, teachers must ask engaging and challenging questions, be able to get all students to participate, and know when to provide information or allow students to discover it for themselves.

Additionally, one synthesis of several experimental studies examining practices in elementary math classrooms identified two instructional approaches that showed positive effects on student learning. The first was cooperative learning in which students work in pairs or small teams and are rewarded based on how well the group learns. The other approach included programs that helped teachers introduce math concepts and improve skills in classroom management, time management, and motivation. This analysis also found that using computer-assisted instruction had moderate to substantial effects on student learning, although this type of instruction was always supplementary to other approaches or programs being used.

We found through our literature review and interviews with researchers that the issue of effective instructional practices is intertwined with professional development. To enable all students to achieve the high standards of learning envisioned by standards-based accountability systems, teachers need extensive skills and knowledge in order to use effective teaching practices in the classroom. Given this, professional development is critical to supporting teachers’ learning of new skills and their application. Specifically, the research concludes that professional development will more likely have positive impacts on both teacher learning and student achievement if it: Focuses on a content area with direct links to the curriculum; Challenges teachers intellectually through reflection and critical problem Aligns with goals and standards for student learning; Lasts long enough so that teachers can practice and revise their Occurs collaboratively within a teacher learning community—ongoing teams of teachers that meet regularly for the purposes of learning, joint lesson planning, and problem solving; Involves all the teachers within a school or department; Provides active learning opportunities with direct applications to the Is based on teachers’ input regarding their learning needs.

Some researchers have raised concerns about the quality and intensity of professional development currently received by many teachers nationwide. One researcher summarized these issues by stating that professional development training for teachers is often too short, provides no classroom follow up, and models more “telling than guiding” practices. Given the decentralized nature of the U.S. education system, the support and opportunity for professional development services for teachers varies among states and school districts, and there are notable examples of states that have focused resources on various aspects of professional development. Nevertheless, shortcomings in teachers’ professional development experiences overall are especially evident when compared to professional development requirements for teachers in countries whose students perform well on international tests, such as the Trends in International Mathematics and Science Study and the Program for International Student Assessment. For example, one study showed that fewer than 10 percent of U.S. math teachers in school year 2003-04 experienced more than 24 hours of professional development in mathematics content or pedagogy during the year; conversely, teachers in Sweden, Singapore, and the Netherlands are required to complete 100 hours of professional development per year.

Agency Comments and Our Evaluation

We provided a copy of our draft report to the Secretary of Education for review and comment. Education’s written comments, which are contained in appendix V, expressed support for the important questions that the report addresses and noted that the American Recovery and Reinvestment Act of 2009 included $250 million to improve assessment and accountability systems. The department specifically stated that the money is for statewide data systems to provide information on individual student outcomes that could help enable schools to strengthen instructional practices and improve student achievement. However, the department raised several issues about the report’s approach. Specifically, the department commented that we (1) did not provide the specific research citations throughout the report for each of our findings or clearly explain how we selected our studies; (2) mixed the opinions of education experts with our findings gleaned from the review of the literature; (3) did not present data on the extent to which test formats had changed or on the relationship between test format and teaching practices when discussing our assessment findings; and (4) did not provide complete information from an Education survey regarding increases and decreases in instructional time.

As stated in the beginning of our report, the list of studies we reviewed and used for our findings are contained in appendix IV. We provide a description in appendix I of our criteria, the types of databases searched, the types of studies examined (e.g., experimental and nonexperimental) and the process by which we evaluated them. We relied heavily on two literature syntheses conducted by the Department of Education— Standards in Classroom Practice: Research Synthesis and The Influence of Standards on K-12 Teaching and Student Learning: A Research Synthesis, which are included in the list. These two syntheses covered, in a more comprehensive way than many of the other studies that we reviewed, the breadth of the topics that we were interested in and included numerous research studies in their reviews. Many of the findings in this report about the research are taken from the conclusions reached in these syntheses. However, to make this fact clearer and more prominent, we added this explanation to our abbreviated scope and methodology section on page 5 of the report.

Regarding the use of expert opinion, we determined that obtaining the views of experts about the research we were reviewing would be critical to our understanding its broader implications. This was particularly important given the breadth and scope of our objectives. The experts we interviewed, whose names and affiliations are listed in appendix III, are prominent researchers who conduct, review, and reflect on the current research in the field, and whose work is included in some of the studies we reviewed, including the two literature syntheses written by the Department of Education and used by us in this study. We did not consider their opinions “conjecture” but grounded in and informed by their many years of respected work on the topic. We have been clear in the report as to when we are citing expert opinion, the research studies, or both.

Regarding the report section discussing the research on assessments, it was our intent to highlight that, according to the research, assessments have both positive and negative influences on classroom teaching practices, not to conclude that NCLBA was the cause of either. Our findings in this section of the report are, in large part, based on conclusions from the department’s syntheses mentioned earlier. For example, The Influence of Standards on K-12 Teaching and Student Learning: A Research Synthesis states “… tests matter—the content covered, the format used, and the application of their results—all influence teacher behavior.” Furthermore, we previously reported that states most often have chosen multiple choice assessments over other types because they can be scored inexpensively and their scores can be released prior to the next school year as required by NCLBA. That report also notes that state officials and alignment experts said that multiple choice assessments have limited the content of what can be tested, stating that highly complex content is “difficult if not impossible to include with multiple choice items.” However, we have revised this paragraph to clarify our point and provide additional information.

Concerning the topic of narrowing the curriculum, we agree with the Department of Education that this report should include a fuller description of the data results from the cited Education survey in order to help the reader put the data in an appropriate context. Hence, we have added information to that section of the report. However, one limitation of the survey data we cite is that it covers changes in instructional time for a short time period—from school year 2004-05 to 2006-07. In the its technical comments, the Department refers to its recent report, Title I Implementation: Update on Recent Evaluation Findings for a fuller discussion of this issue. The Title I report, while noting that most elementary teachers reported no change from 2004–05 to 2006–07 in the amount of instructional time that they spent on various subjects, also provides data over a longer, albeit earlier period time period, from 1987–88 to 2003–04, from the National Center on Education Statistics Schools and Staffing Survey. In analyzing this data, the report states that elementary teachers had increased instructional time on reading and mathematics and decreased the amount of time spent on science and social studies during this period. We have added this information as well. Taken together, we believe these data further reinforce our point that assessments under current accountability systems can have, in addition to positive influences on teaching, some negative ones as well, such as the curriculum changes noted in the report, even if the extent of these changes is not fully known.

Education also provided technical comments that we incorporated as appropriate.

We are sending copies of this report to the Secretary of Education, relevant congressional committees, and other interested parties. The report also is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-7215 or ashbyc@gao.gov. Contact points for our Office of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix VI.

Appendix I: Scope and Methodology

To address the objectives of this study, we used a variety of methods. To determine the types of instructional practices schools and teachers are using to help students achieve state academic standards and whether those practices differ by school characteristics, we used two recent surveys of principals and teachers. The first survey, a nationally- representative survey from the Department of Education’s (Education) National Longitudinal Study of No Child Left Behind (NLS-NCLB) conducted by the RAND Corporation (RAND), asked principals the extent to which their schools were focusing on certain strategies in their voluntary school improvement efforts. Education’s State and Local Implementation of the No Child Left Behind Act Volume III— Accountability Under NCLB: Interim Report included information about the strategies emphasized by principals as a whole, and we obtained from Education the NLS-NCLB database to determine the extent to which principals’ responses differed by school characteristic variables. We conducted this analysis on school year 2006-2007 data by controlling for four school characteristic variables: (1) the percentage of a school’s students receiving free or reduced price lunch (poverty); (2) the percentage of students who are a racial minority (minority); (3) whether the school is in an urban, urban fringe (suburban), or rural area (school location); and (4) the school’s adequate yearly performance (AYP) status.

We analyzed data from a second RAND survey, which was a three-state survey sponsored by the National Science Foundation that asked math teachers in California, Georgia, and Pennsylvania how their classroom teaching strategies differed due to a state math test. RAND selected these states to represent a range of approaches to standards-based accountability and to provide some geographic and demographic diversity; the survey data is representative only for those three states individually. RAND’s report on the three-state survey data included information about how teachers within each of the three states had changed their teaching practices due to a state accountability test. RAND provided us with descriptive data tables based on its school year 2005-2006 survey data; we analyzed the data to measure associations between the strategies used and the school characteristic variables. We requested tables that showed this information for teachers in all schools, and separately for teachers in different categories of schools (elementary and middle schools) and by the school characteristics of poverty, minority, school location and AYP status. We obtained from RAND standard error information associated with the estimates from the different types of schools and thus were able to test the statistical significance of differences in likelihood between what teachers from different types of schools reported.

As part of our analyses for both surveys, we reviewed documentation and performed electronic testing of the data obtained through the surveys. We also conducted several interviews with several researchers responsible for the data collection and analyses and obtained information about the measures they took to ensure data reliability. On the basis of our efforts to determine the reliability of the data, we determined the data from each of these surveys were sufficiently reliable for the purposes of our study.

We reviewed existing literature to determine what researchers have found regarding the effect of standards-based accountability systems on instructional practices, and practices that work in raising student achievement. To identify existing studies, we conducted searches of various databases, such as the Education Resources Information Center, Proquest, Dialog EDUCAT, and Education Abstracts. We also asked all of the education researchers that we interviewed to recommend additional studies. From these sources, we identified 251 studies that were relevant to our study objectives about the effect of standards-based accountability systems on instructional practices and instructional practices there are effective in raising student achievement. We selected them according to the following criteria: covered the years 2001 through 2008 and were either experimental or quasi-experimental studies, literature syntheses, or studied multiple sites. We selected the studies for our review based on their methodological strength, given the limitations of the methods used, and not necessarily on whether the results could be generalized. We performed our searches from August 2008 to January 2009.

To assess the methodological quality of the selected studies, we developed a data collection instrument to obtain information systematically about each study being evaluated and about the features of the evaluation methodology. We based our data collection and assessments on generally accepted social science standards. We examined factors related to the use of comparison and control groups; the appropriateness of sampling and data collection methods; and for syntheses, the process and criteria used to identify studies. A senior social scientist with training and experience in evaluation research and methodology read and coded the methodological discussion for each evaluation. A second senior social scientist reviewed each completed data collection instrument and the relevant documentation to verify the accuracy of every coded item. This review identified 20 selected studies that met GAO’s criteria for methodological quality.

We supplemented our synthesis by interviewing prominent education researchers identified in frequently cited articles and through discussions with knowledgeable individuals. We also conducted interviews with officials at the U.S. Department of Education, including the Center on Innovation and Improvement, and the Institute on Education Sciences’ National Center for Education Evaluation and Regional Assistance, as well as other educational organizations. We also reviewed relevant federal laws and regulations.

Appendix II: Analyses of the Relationship between School Characteristics and Principals’ Focus on School Improvement Strategies

In order to analyze the National Longitudinal Study of No Child Left Behind (NLS-NCLB) principal survey conducted by the RAND Corporation, we analyzed strategies on which principals most often focused, taking into account the percentage of a school’s students receiving free or reduced price lunch (poverty), the percentage of students who are a racial minority (minority), whether the school is in an urban, suburban, or rural area (school location), and the school’s adequate yearly performance (AYP) status (see table 1). Our analyses used “odds ratios,” generally defined as the ratio of the odds of an event occurring in one group compared to the odds of it occurring in another group, to express differences in the likelihoods of schools with different characteristics using these strategies. We used odds ratios rather than percentages because they are more appropriate for statistical modeling and multivariate analysis. Odds ratios indicate how much higher (when they are greater than 1.0) or lower (when they are less than 1.0) the odds were that principals would respond that a given strategy was a major or moderate focus. We included a reference category for the school characteristics (low minority, low poverty, and central city) in the top row of table 1, and put comparison groups beneath those reference categories, as indicated by the column heading in the second row (high-minority, high- poverty, or rural schools). As an example, the third cell in the “high- minority schools” column indicates that principals in high-minority schools were 2.65 times more likely to make “implementing new instructional approaches or curricula in reading/language arts/English” a focus of their school improvement efforts. In another example, the odds that principals would “restructure the school day to teach core content areas in greater depth (e.g., establishing a literacy block)” were 2.8 times higher for high-poverty schools than low poverty schools, as seen in the sixth cell under “high-poverty schools.” Those cells with an asterisk indicate statistically significant results; that is, we have a high degree of confidence that the differences we see are not just due to chance but show an actual difference in the survey responses. See appendix I for further explanation of our methodology.

Appendix III: List of Education Researchers

Appendix IV: Studies Meeting GAO’s Criteria for Methodological Quality

Laura. S Hamilton; Brian M Stecher; Jennifer Linn Russell; Julie A. Marsh; Jeremy Miles

“Strong States, Weak Schools: The Benefits and Dilemmas of Centralized Accountability”

Quasi-experimental design with matched groups; multiple regressions used with data.

Literature review using a best-evidence synthesis (related to a meta-analysis)

Appendix V: Comments from the Department of Education

Appendix VI: GAO Contact and Staff Acknowledgments

Cornelia M. Ashby (202) 512-7215 or ashbyc@gao.gov.

Staff Acknowledgments

Janet Mascia (Assistant Director), Bryon Gordon (Assistant Director), and Andrew Nelson (Analyst-in-Charge) managed all aspects of the assignment. Linda Stokes and Caitlin Tobin made significant contributions to this report in all aspects of the work. Kate van Gelder contributed to writing this report, and Ashley McCall contributed to research for the report. Luann Moy, Justin Fisher, Cathy Hurley, Douglas Sloane, and John Smale Jr. provided key technical support, and Doreen Feldman and Sheila R. McCoy provided legal support. Mimi Nguyen developed the graphics for the report.