Student Achievement: Schools Use Multiple Strategies to Help Students Meet Academic Standards, Especially Schools with Higher Proportions of Low-Income and Minority Students
Published: 2009-11-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The federal government has invested billions of dollars to improve student academic performance, and many schools, teachers, and researchers are trying to determine the most effective instructional practices with which to accomplish this. The Conference Report for the Consolidated Appropriations Act for Fiscal Year 2008 directed GAO to study strategies used to prepare students to meet state academic achievement standards. To do this, GAO answered: (1) What types of instructional practices are schools and teachers most frequently using to help students achieve state academic standards, and do those instructional practices differ by school characteristics? (2) What is known about how standards-based accountability systems have affected instructional practices? (3) What is known about instructional practices that are effective in improving student achievement? GAO analyzed data from a 2006-2007 national survey of principals and 2005-2006 survey of teachers in three states, conducted a literature review of the impact of standards-based accountability systems on instructional practices and of practices that are effective in improving student achievement, and interviewed experts.
section_title: What GAO Found
paragraphs: Nationwide, most principals focused on multiple strategies to help students meet academic standards, such as using student data to inform instruction and increasing professional development for teachers, according to our analysis of data from a U.S. Department of Education survey. Many of these strategies were used more often at high-poverty schools--those where 75 percent or more of the students were eligible for the free and reduced-price lunch program--and high-minority schools--those where 75 percent or more of students were identified as part of a minority population, than at lower poverty and minority schools. Likewise, math teachers in California, Georgia, and Pennsylvania increased their use of certain instructional practices in response to their state tests, such as focusing more on topics emphasized on assessments and searching for more effective teaching methods, and teachers at high-poverty and high-minority schools were more likely than teachers at lower-poverty schools and lower-minority schools to have made these changes, according to GAO's analysis of survey data collected by the RAND Corporation. Some researchers suggested that differences exist in the use of these practices because schools with lower poverty or lower minority student populations might generally be meeting accountability requirements and therefore would need to try these strategies less frequently. Research shows that standards-based accountability systems can influence instructional practices in both positive and negative ways. For example, some research notes that using a standards-based curriculum that is aligned with corresponding instructional guidelines can facilitate the development of higher order thinking skills in students. But, in some cases, teacher practices did not always reflect the principles of standards-based instruction, and the difficulties in aligning practice with standards were attributed, in part, to current accountability requirements. Other research noted that assessments can be powerful tools for improving the learning process and evaluating student achievement, but assessments can also have some unintended negative consequences on instruction, including narrowing the curriculum to only material that is tested. Many experts stated that methodological issues constrain knowing more definitively the specific instructional practices that improve student learning and achievement. Nevertheless, some studies and experts pointed to instructional practices that are considered to be effective in raising student achievement, such as differentiated instruction. Professional development for teachers was also highlighted as important for giving teachers the skills and knowledge necessary to implement effective teaching practices.

section_title: Letter
paragraphs: The federal government has invested billions of dollars to help schools  meet requirements of the No Child Left Behind Act of 2001 (NCLBA) to  improve student academic performance in reading, math, and science. To  this end, many schools, teachers, and researchers are trying to determine  the most effective instructional practices to improve student achievement.  Instructional practices refer to school or district-level improvement  strategies, such as aligning curriculum with academic standards,  restructuring the school day, or providing additional professional  development to teachers. Instructional practices can also refer to  classroom teaching practices like assigning more homework or searching  for more effective teaching methods. Little is known about the extent to  which instructional practices have changed in response to NCLBA’s  accountability requirements, whether these practices vary by type of  school, and the extent to which some practices have proven to be more  effective than others.
Under NCLBA, states are required to develop challenging student  academic achievement standards, administer tests based on those  standards (standards-based assessments) to measure student proficiency,  and develop targets for performance on these tests. Specifically, NCLBA  requires states to develop a plan to ensure that their students are making  adequate yearly progress (AYP) toward proficiency in reading, math, and  science by 2014 for students collectively and in key student subgroups,  including low-income and minority students.
While NCLBA creates requirements for student proficiency, it generally  allows states to determine how best to meet those requirements. The  Conference Report accompanying the Consolidated Appropriations Act for  Fiscal Year 2008 directed that GAO conduct a study of strategies used to  prepare students to meet state academic achievement standards. In  response, we agreed with the Senate and House Appropriations  Committees, the Senate Committee on Health, Education, Labor and  Pensions, and the House Committee on Education and Labor to address  the following questions:  1.  What types of instructional practices are schools and teachers most  frequently using to help students achieve state academic standards,  and do those instructional practices differ by school characteristics?   2.  What is known about how standards-based accountability systems  such as that in NCLBA have affected instructional practices?  3.  What is known about instructional practices that are effective in  improving student achievement?
To answer these questions, we analyzed data from two recent surveys of  principals and teachers that were conducted by the RAND Corporation  (RAND). The first survey, the nationally representative National  Longitudinal Study of No Child Left Behind (NLS-NCLB), was sponsored  by the U.S. Department of Education (Education) and asked principals the  extent to which their schools were focusing on certain strategies in their  school improvement efforts. We conducted an analysis of the school year  2006-2007 survey data on school improvement strategies by controlling for  school characteristic variables, such as the percentage of a school’s  students receiving free or reduced price lunch (poverty); the percentage of  students who are a racial minority (minority); whether the school is in an  urban, urban fringe or large town, or rural area (school location); and the  school’s AYP performance status. The second survey, a three-state survey  sponsored by the National Science Foundation (NSF), asked elementary  and middle school teachers in California, Georgia, and Pennsylvania how  their classroom teaching strategies differed due to a state math test.
RAND selected these states to represent a range of approaches to  standards-based accountability and to provide some geographic and  demographic diversity. Using school year 2005-2006 data from the three- state survey, which is representative only of those three states  individually, we measured associations between the teacher responses and  the school characteristic variables. As part of these survey analyses, we  reviewed documentation and performed electronic testing of the data  obtained through the surveys and conducted interviews with the primary  RAND researchers responsible for the data collection and analysis. We  determined the survey data were sufficiently reliable for the purposes of  our study. To answer questions two and three, we conducted a literature  review and synthesis. We supplemented our synthesis by interviewing  prominent education researchers identified in frequently cited articles and  through discussions with other knowledgeable individuals. We also  reviewed relevant federal laws and regulations.
We conducted our work from July 2008 to November 2009 in accordance  with all sections of GAO’s Quality Assurance Framework that are relevant  to our objectives. The framework requires that we plan and perform the  engagement to obtain sufficient and appropriate evidence to meet our  stated objectives and to discuss any limitations in our work. We believe  that the information and data obtained, and the analysis conducted,  provide a reasonable basis for any findings and conclusions in this  product.
subsections: section_title: Background
paragraphs: NCLBA reauthorized the Elementary and Secondary Education Act of 1965  (ESEA) and built upon accountability requirements created under a  previous reauthorization, the Improving America’s Schools Act of 1994  (IASA). Under ESEA, as amended, Congress sought to improve student  learning by incorporating academic standards and assessments in the  requirements placed on states. Academic standards, which describe what  students should know and be able to do at different grade levels in  different subjects, help guide school systems in their choice of curriculum  and help teachers plan for classroom instruction. Assessments, which  states use to measure student progress in achieving the standards, are  required to be administered by states.
NCLBA further strengthened some of the accountability requirements  contained in ESEA, as amended. Specifically, NCLBA’s accountability  provisions require states to develop education plans that establish  academic standards and performance goals for schools to meet AYP and  lead to 100 percent of their students being proficient in reading, math, and  science by 2014. This proficiency must be assessed annually in reading and  math in grades 3 through 8 and periodically in science, whereas  assessments were required less frequently under the IASA. Under NCLBA,  schools’ assessment data generally must be disaggregated to assess  progress toward state proficiency targets for students in certain  designated groups, including low-income students, minority students,  students with disabilities, and those with limited English proficiency. Each  of these groups must make AYP in order for the school to make AYP.  Schools that fail to make AYP for 2 or more consecutive years are required  to implement various improvement measures identified in NCLBA, and  these measures are more extensive than those required under IASA.  Education, which has responsibility for general oversight of NCLBA,  reviews and approves state plans for meeting AYP requirements. As we  have previously reported, Education had approved all states’ plans—fully  or conditionally—by June 2003.
NCLBA also recognizes the role of teachers in providing a quality  education by requiring states to ensure that all teachers in core academic  subjects are “highly qualified.” Under this requirement, teachers generally  must have a bachelor’s degree, be fully certified, and demonstrate their  knowledge of the subjects they teach. Previously, there were no specific  requirements regarding teacher quality under ESEA, as amended.
section_title: Principals and Teachers Used a Variety of Instructional Practices to Help Students Meet Standards, and Many of These Practices Were Used More Frequently at Schools with Higher Proportions of Low- Income and Minority Students
paragraphs: According to our analysis of NLS-NCLB data from Education, most  principals reported their schools focused on multiple instructional  practices in their voluntary school improvement efforts. These strategies  were used more often at schools with higher proportions of low-income  students (“high-poverty schools”) and schools with higher proportions of  minority students (“high-minority schools”) than at schools with lower  proportions of low-income students (“low-poverty schools”) and schools  with lower proportions of minority students (“low-minority schools”).  Likewise, the survey of math teachers in California, Georgia, and  Pennsylvania indicates teachers were using many different instructional  practices in response to their state tests, and teachers at high-poverty and  high-minority schools were more likely than teachers at low-poverty and  low-minority schools to have been increasing their use of some of these  practices. Some researchers we spoke with suggested that differences in  the use of these instructional practices exist because schools with low- poverty or low-minority student populations might generally be meeting  accountability standards and, therefore, would need to try these strategies  less frequently.
subsections: section_title: Principals at High-Poverty and High-Minority Schools Emphasized Certain School Improvement Strategies More Than Principals at Other Schools
paragraphs: According to nationally representative data from Education’s NLS-NCLB,  in school year 2006-2007 most principals focused on multiple strategies in  their school improvement efforts. The survey asked principals the extent  to which their schools were focusing on ten different strategies in their  voluntary school improvement initiatives. The three most common  strategies were: (1) using student achievement data to inform instruction  and school improvement; (2) providing additional instruction to low- achieving students; and (3) aligning curriculum and instruction with  standards and/or assessments. (See fig. 1.) Nearly all school principals  placed a major or moderate focus on three or more surveyed strategies in  their school improvement efforts, and over 80 percent of principals placed  a major or moderate focus on six or more strategies. However, as  Education’s report on the survey data cautioned, the number of  improvement strategies emphasized was not necessarily an indication of  the intensity or quality of the improvement efforts.
While nearly all principals responded that they used multiple improvement  strategies, there were statistically significant differences in principals’  responses across a range of school characteristics, including percentage of  the school’s students receiving free or reduced price lunch (poverty),  percentage of minority students, the school’s location, and AYP status.  For example, when comparing schools across poverty levels, we found  that principals at high-poverty schools were two to three times more likely  than principals at low-poverty schools to focus on five particular strategies  in their school improvement efforts:    Restructuring the school day to teach core content areas in greater depth;  Increasing instructional time for all students (e.g., by lengthening the  school day or year, shortening recess);    Providing extended-time instructional programs (e.g., before-school, after- school, or weekend instructional programs);  Implementing strategies for increasing parents’ involvement in their  children’s education; and  Increasing the intensity, focus, and effectiveness of professional  development.
Likewise, when comparing schools across minority levels, we found that  principals at high- and moderate-minority schools were approximately two  to three times more likely than principals at low-minority schools to make  six particular school improvement strategies a major or moderate focus of  their school improvement efforts. For instance, principals at schools with  a high percentage of minority students were more than three times as  likely as principals at schools with a low percentage of minority students  to provide extended-time instruction such as after-school programs. A  school’s location was associated with differences in principals’ responses  about the strategies they used as well: principals at rural schools were  only about one-third to one-half as likely as central city schools to make  five of these school improvement strategies a moderate or major focus of  their school improvement efforts.
When we compared principal responses based on AYP status, there was  some evidence of a statistically significant association between AYP status  and the extent to which principals focused these strategies in their school  improvement efforts, but it was limited when the other variables such as  poverty and minority were taken into account. AYP status had some  correlation with the demographic characteristics of poverty and minority,  and those characteristics explained the patterns of principals’ responses  more fully than the AYP characteristic. However, our analysis generally  showed that schools that had not made AYP were more likely to make six  of these school improvement strategies a moderate or major focus of their  school improvement plan than schools that had made AYP. Additionally,  Education reported that schools identified for improvement under  NCLBA—that is, schools that have not made AYP for two or more  consecutive years—were engaged in a greater number of improvement  efforts than non-identified schools. Therefore, principals of the non- identified schools may have been less likely than principals of identified  schools to view specific strategies as a major or moderate focus.
We spoke with several researchers about the results of our analysis of the  principals’ responses, especially at high-poverty and high-minority schools.  While the researchers could not say with certainty the reasons for the  patterns, they noted that high-poverty and high-minority schools tend to be  most at risk of not meeting their states’ standards, so that principals at  those schools might be more willing to try different approaches.  Conversely, the researchers noted that principals at schools meeting  standards would not have the same incentives to adopt as many school  improvement strategies.
section_title: Most Math Teachers in Three Surveyed States Have Increased Their Use of Certain Instructional Practices in Response to State Tests, especially in High-Poverty and High- Minority Schools
paragraphs: The RAND survey of elementary and middle school math teachers in  California, Georgia and Pennsylvania showed that in each of the three  states at least half of the teachers reported increasing their use of certain  instructional practices in at least five areas as a result of the statewide  math test (see fig. 2). For example, most teachers in Pennsylvania  responded that due to the state math test they: (1) focused more on  standards, (2) emphasized assessment styles and formats, (3) focused  more on subjects tested, (4) searched for more effective teaching methods,  and (5) spent more time teaching content.
As we did with the survey responses of principals, we analyzed the teacher  survey data to determine whether math teachers’ responses differed by  school characteristics for poverty, minority, location, and AYP status. As  with the principals’ responses, we found that elementary and middle  school math teachers in high-poverty and high-minority schools were more  likely than teachers in low-poverty and low-minority schools to report  increasing their use of certain instructional practices, and this pattern was  consistent across the three states (see fig. 3). For example, 69 percent of  math teachers at high-poverty schools in California indicated they spent  more time teaching test-taking strategies as opposed to 38 percent of math  teachers in low-poverty schools. In Georgia, 50 percent of math teachers in  high-poverty schools reported offering more outside assistance to non- proficient students in contrast to 26 percent of math teachers in low- poverty schools. Fifty-one percent of math teachers at high-poverty  schools in Pennsylvania reported focusing more attention on students  close to proficiency compared to 23 percent of math teachers doing so in  low poverty schools.
Similar to what our poverty analysis showed, survey responses provided  some evidence that math teachers in high-minority schools were more  likely than those in low-minority schools to change their instructional  practices. Math teachers at high-minority schools in each of the three  states, as compared to those at low-minority schools, were more likely to:  rely on open-ended tests in their own classroom assessments;  increase the amount of time spent teaching mathematics by replacing non- instructional activities with mathematics instruction;  focus on topics emphasized in the state math test; and  teach general test-taking strategies.
We also analyzed the RAND data with regard to school location and a  school’s AYP status, but results from these characteristics were not  significant for as many instructional practices.
As we did regarding the survey responses of principals, we spoke to  several researchers, including the authors of the three-state teacher study,  regarding possible reasons for the patterns we saw in the teacher survey  data. The researchers we spoke with provided similar possible reasons for  the patterns in the teacher survey as they did for patterns in the principal  survey. For instance, the researchers noted that high-poverty and high- minority schools are more likely to be at risk of failing to meet the state  standards, which might prompt teachers to try different approaches. On  the other hand, the researchers stated that teachers at those schools  meeting the standards would not have the same incentives to change their  instructional practices.
section_title: Research Shows That Standards-based Accountability Systems Can Influence Instructional Practices through Standards and Assessments in Both Positive and Negative Ways
paragraphs: Research shows that using a standards-based curriculum that is aligned  with corresponding instructional guidelines can positively influence  teaching practices. Specifically, some studies reported changes by  teachers who facilitated their students developing higher-order thinking  skills, such as interpreting meaning, understanding implied reasoning, and  developing conceptual knowledge, through practices such as multiple  answer problem solving, less lecture and more small group work.  Additionally, a few researchers we interviewed stated that a positive effect  of NCLBA’s accountability provisions has been a renewed focus on  standards and curriculum.
However, some studies indicated that teachers’ practices did not always  reflect the principles of standards-based instruction and that current  accountability policies help contribute to the difficulty in aligning practice  with standards. Some research shows that, while teachers may be  changing their instructional practices in response to standards-based  reform, these changes may not be fully aligned with the principles of the  reform. That research also notes that the reliability in implementing  standards in the classroom varied in accordance with teachers’ different  beliefs in and support for standards-based reform as well as the limitations  in their instructional capabilities. For example, one observational study of  math teachers showed that, while teachers implemented practices  envisioned by standards-based reform, such as getting students to work in  small groups or using manipulatives (e.g., cubes or tiles), their approaches  did not go far enough in that students were not engaged in conversations  about mathematical or scientific concepts and ideas. To overcome these  challenges, studies point to the need for teachers to have opportunities to  learn, practice, and reflect on instructional practices that incorporate the  standards, and then to observe their effects on student learning. However,  some researchers have raised concerns that current accountability  systems’ focus on test scores and mandated timelines for achieving  proficiency levels for students do not give teachers enough time to learn,  practice, and reflect on instructional practices and may discourage some  teachers from trying ambitious teaching practices envisioned by  standards-based reform.
Another key element of a standards-based accountability system is  assessments, which help measure the extent to which schools are  improving student learning through assessing student performance against  the standards. Some researchers note that assessments are powerful tools  for managing and improving the learning process by providing information  for monitoring student progress, making instructional decisions,  evaluating student achievement, and evaluating programs. In addition,  assessments can also influence instructional content and help teachers use  or adjust specific classroom practices. As one synthesis concluded,  assessments can influence whether teachers broaden or narrow the  curriculum, focus on concepts and problem solving—or emphasize test  preparation over subject matter content.
In contrast, some of the research and a few experts we interviewed raised  concerns about testing formats that do not encourage challenging teaching  practices and instructional practices that narrow the curriculum as a result  of current assessment practices. For example, depending on the test  used, research has shown that teachers may be influenced to use teaching  approaches that reflect the skills and knowledge to be tested. Multiple  choice tests tend to focus on recognizing facts and information while  open-ended formats are more likely to require students to apply critical  thinking skills. Conclusions from a literature synthesis conducted by the  Department of Education stated that “ teachers respond to assessment  formats used, so testing programs must be designed and administered with  this influence in mind. Tests that emphasize inquiry, provide extended  writing opportunities, and use open-ended response formats or a portfolio  approach tend to influence instruction in ways quite different from tests  that use closed-ended response formats and which emphasize  procedures.” We recently reported that states have most often chosen  multiple choice items over other item types of assessments because they  are cost effective and can be scored within tight time frames. While  multiple choice tests provide cost and time saving benefits to states, the  use of multiple choice items make it difficult, if not impossible, to measure  highly complex content. Other research has raised concerns that, to  avoid potential consequences from low-scoring assessment results under  NCLBA, teachers are narrowing the curriculum being taught—sometimes  referred to as “teaching to the test”—either by spending more classroom  time on tested subjects at the expense of other non-tested subjects,  restricting the breadth of content covered to focus only on the content  covered by the test, or focusing more time on test-taking strategies than on  subject content.
section_title: Research Highlights Some Potentially Successful Practices for Improving Student Achievement, although Experts Contend That Methodological Issues Constrain Reaching Definitive Conclusions about What Works
paragraphs: Our literature review found some studies that pointed to instructional  practices that appear to be effective in raising student achievement. But, in  discussing the broader implications of these studies with the experts that  we interviewed, many commented that, taken overall, the research is not  conclusive about which specific instructional practices improve student  learning and achievement.
Some researchers stated that this was due to methodological issues in  conducting the research. For example, one researcher explained that,  while smaller research studies on very specific strategies in reading and  math have sometimes shown powerful relationships between the strategy  used and positive changes in student achievement, results from meta- analyses of smaller studies have been inconclusive in pointing to similar  patterns in the aggregate. A few other researchers stated that the lack of  empirical data about how instruction unfolds in the classroom hampers  the understanding about what works in raising student performance.
A few researchers also noted that conducting research in a way that would  yield more conclusive results is difficult. One of the main difficulties, as  explained by one researcher, is the number of variables a study may need  to examine or control for in order to understand the effectiveness of a  particular strategy, especially given the number of interactions these  variables could have with each other. One researcher mentioned cost as a  challenge when attempting to gather empirical data at the classroom level,  stating “teaching takes place in the classroom, but the expense of  conducting classroom-specific evaluations is a serious barrier to collecting  this type of data.” Finally, even when research supports the efficacy of a  strategy, it may not work with different students or under varying  conditions. In raising this point, one researcher stated that “educating a  child is not like making a car” whereby a production process is developed  and can simply be repeated again and again. Each child learns differently,  creating a challenge for teachers in determining the instructional practices  that will work best for each student.
Some of the practices identified by both the studies and a few experts as  those with potential for improving student achievement were:    Differentiated instruction. In this type of instruction, teaching practices  and plans are adjusted to accommodate each student’s skill level for the  task at hand. Differentiated instruction requires teachers to be flexible in  their teaching approach by adjusting the curriculum and presentation of  information for students, thereby providing multiple options for students  to take in and process information. As one researcher described it,  effective teachers understand the strategies and practices that work for  each student and in this way can move all students forward in their  learning and achievement.
More guiding, less telling. Researchers have identified two general  approaches to teaching: didactic and interactive. Didactic instruction  relies more on lecturing and demonstrations, asking short answer  questions, and assessing whether answers are correct. Interactive  instruction focuses more on listening and guiding students, asking  questions with more than one correct answer, and giving students choices  during learning. As one researcher explained, both teaching approaches  are important, but some research has shown that giving students more  guidance and less direction helps students become critical and  independent thinkers, learn how to work independently, and assess  several potential solutions and apply the best one. These kinds of learning  processes are important for higher-order thinking. However, implementing  “less instruction” techniques requires a high level of skill and creativity on  the part of the teacher.
Promoting effective discourse. An important corollary to the teacher  practice of guiding students versus directing them is effective classroom  discussion. Research highlights the importance of developing students’  understanding not only of the basic concepts of a subject, but higher-order  thinking and skills as well. To help students achieve understanding, it is  necessary to have effective classroom discussion in which students test  and revise their ideas, and elaborate on and clarify their thinking. In  guiding students to an effective classroom discussion, teachers must ask  engaging and challenging questions, be able to get all students to  participate, and know when to provide information or allow students to  discover it for themselves.
Additionally, one synthesis of several experimental studies examining  practices in elementary math classrooms identified two instructional  approaches that showed positive effects on student learning. The first was  cooperative learning in which students work in pairs or small teams and  are rewarded based on how well the group learns. The other approach  included programs that helped teachers introduce math concepts and  improve skills in classroom management, time management, and  motivation. This analysis also found that using computer-assisted  instruction had moderate to substantial effects on student learning,  although this type of instruction was always supplementary to other  approaches or programs being used.
We found through our literature review and interviews with researchers  that the issue of effective instructional practices is intertwined with  professional development. To enable all students to achieve the high  standards of learning envisioned by standards-based accountability  systems, teachers need extensive skills and knowledge in order to use  effective teaching practices in the classroom. Given this, professional  development is critical to supporting teachers’ learning of new skills and  their application. Specifically, the research concludes that professional  development will more likely have positive impacts on both teacher  learning and student achievement if it:    Focuses on a content area with direct links to the curriculum;    Challenges teachers intellectually through reflection and critical problem    Aligns with goals and standards for student learning;    Lasts long enough so that teachers can practice and revise their    Occurs collaboratively within a teacher learning community—ongoing  teams of teachers that meet regularly for the purposes of learning, joint  lesson planning, and problem solving;  Involves all the teachers within a school or department;    Provides active learning opportunities with direct applications to the  Is based on teachers’ input regarding their learning needs.
Some researchers have raised concerns about the quality and intensity of  professional development currently received by many teachers  nationwide. One researcher summarized these issues by stating that  professional development training for teachers is often too short, provides  no classroom follow up, and models more “telling than guiding” practices.  Given the decentralized nature of the U.S. education system, the support  and opportunity for professional development services for teachers varies  among states and school districts, and there are notable examples of states  that have focused resources on various aspects of professional  development. Nevertheless, shortcomings in teachers’ professional  development experiences overall are especially evident when compared to  professional development requirements for teachers in countries whose  students perform well on international tests, such as the Trends in  International Mathematics and Science Study and the Program for  International Student Assessment. For example, one study showed that  fewer than 10 percent of U.S. math teachers in school year 2003-04  experienced more than 24 hours of professional development in  mathematics content or pedagogy during the year; conversely, teachers in  Sweden, Singapore, and the Netherlands are required to complete 100  hours of professional development per year.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a copy of our draft report to the Secretary of Education for  review and comment. Education’s written comments, which are contained  in appendix V, expressed support for the important questions that the  report addresses and noted that the American Recovery and Reinvestment  Act of 2009 included $250 million to improve assessment and  accountability systems. The department specifically stated that the money  is for statewide data systems to provide information on individual student  outcomes that could help enable schools to strengthen instructional  practices and improve student achievement. However, the department  raised several issues about the report’s approach. Specifically, the  department commented that we (1) did not provide the specific research  citations throughout the report for each of our findings or clearly explain  how we selected our studies; (2) mixed the opinions of education experts  with our findings gleaned from the review of the literature; (3) did not  present data on the extent to which test formats had changed or on the  relationship between test format and teaching practices when discussing  our assessment findings; and (4) did not provide complete information  from an Education survey regarding increases and decreases in  instructional time.
As stated in the beginning of our report, the list of studies we reviewed  and used for our findings are contained in appendix IV. We provide a  description in appendix I of our criteria, the types of databases searched,  the types of studies examined (e.g., experimental and nonexperimental)  and the process by which we evaluated them. We relied heavily on two  literature syntheses conducted by the Department of Education— Standards in Classroom Practice: Research Synthesis and The Influence  of Standards on K-12 Teaching and Student Learning: A Research  Synthesis, which are included in the list. These two syntheses covered, in  a more comprehensive way than many of the other studies that we  reviewed, the breadth of the topics that we were interested in and  included numerous research studies in their reviews. Many of the findings  in this report about the research are taken from the conclusions reached  in these syntheses. However, to make this fact clearer and more  prominent, we added this explanation to our abbreviated scope and  methodology section on page 5 of the report.
Regarding the use of expert opinion, we determined that obtaining the  views of experts about the research we were reviewing would be critical  to our understanding its broader implications. This was particularly  important given the breadth and scope of our objectives. The experts we  interviewed, whose names and affiliations are listed in appendix III, are  prominent researchers who conduct, review, and reflect on the current  research in the field, and whose work is included in some of the studies  we reviewed, including the two literature syntheses written by the  Department of Education and used by us in this study. We did not consider  their opinions “conjecture” but grounded in and informed by their many  years of respected work on the topic. We have been clear in the report as  to when we are citing expert opinion, the research studies, or both.
Regarding the report section discussing the research on assessments, it  was our intent to highlight that, according to the research, assessments  have both positive and negative influences on classroom teaching  practices, not to conclude that NCLBA was the cause of either. Our  findings in this section of the report are, in large part, based on  conclusions from the department’s syntheses mentioned earlier. For  example, The Influence of Standards on K-12 Teaching and Student  Learning: A Research Synthesis states “… tests matter—the content  covered, the format used, and the application of their results—all  influence teacher behavior.” Furthermore, we previously reported that  states most often have chosen multiple choice assessments over other  types because they can be scored inexpensively and their scores can be  released prior to the next school year as required by NCLBA. That report  also notes that state officials and alignment experts said that multiple  choice assessments have limited the content of what can be tested, stating  that highly complex content is “difficult if not impossible to include with  multiple choice items.” However, we have revised this paragraph to clarify  our point and provide additional information.
Concerning the topic of narrowing the curriculum, we agree with the  Department of Education that this report should include a fuller  description of the data results from the cited Education survey in order to  help the reader put the data in an appropriate context. Hence, we have  added information to that section of the report. However, one limitation of  the survey data we cite is that it covers changes in instructional time for a  short time period—from school year 2004-05 to 2006-07. In the its technical  comments, the Department refers to its recent report, Title I  Implementation: Update on Recent Evaluation Findings for a fuller  discussion of this issue. The Title I report, while noting that most  elementary teachers reported no change from 2004–05 to 2006–07 in the  amount of instructional time that they spent on various subjects, also  provides data over a longer, albeit earlier period time period, from 1987–88  to 2003–04, from the National Center on Education Statistics Schools and  Staffing Survey. In analyzing this data, the report states that elementary  teachers had increased instructional time on reading and mathematics and  decreased the amount of time spent on science and social studies during  this period. We have added this information as well. Taken together, we  believe these data further reinforce our point that assessments under  current accountability systems can have, in addition to positive influences  on teaching, some negative ones as well, such as the curriculum changes  noted in the report, even if the extent of these changes is not fully known.
Education also provided technical comments that we incorporated as  appropriate.
We are sending copies of this report to the Secretary of Education,  relevant congressional committees, and other interested parties. The  report also is available at no charge on the GAO Web site at  http://www.gao.gov.
If you or your staff have any questions about this report, please contact me  at (202) 512-7215 or ashbyc@gao.gov. Contact points for our Office of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix VI.
section_title: Appendix I: Scope and Methodology
paragraphs: To address the objectives of this study, we used a variety of methods. To  determine the types of instructional practices schools and teachers are  using to help students achieve state academic standards and whether  those practices differ by school characteristics, we used two recent  surveys of principals and teachers. The first survey, a nationally- representative survey from the Department of Education’s (Education)  National Longitudinal Study of No Child Left Behind (NLS-NCLB)  conducted by the RAND Corporation (RAND), asked principals the extent  to which their schools were focusing on certain strategies in their  voluntary school improvement efforts. Education’s State and Local  Implementation of the No Child Left Behind Act Volume III— Accountability Under NCLB: Interim Report included information about  the strategies emphasized by principals as a whole, and we obtained from  Education the NLS-NCLB database to determine the extent to which  principals’ responses differed by school characteristic variables. We  conducted this analysis on school year 2006-2007 data by controlling for  four school characteristic variables: (1) the percentage of a school’s  students receiving free or reduced price lunch (poverty); (2) the  percentage of students who are a racial minority (minority); (3) whether  the school is in an urban, urban fringe (suburban), or rural area (school  location); and (4) the school’s adequate yearly performance (AYP) status.
We analyzed data from a second RAND survey, which was a three-state  survey sponsored by the National Science Foundation that asked math  teachers in California, Georgia, and Pennsylvania how their classroom  teaching strategies differed due to a state math test. RAND selected these  states to represent a range of approaches to standards-based  accountability and to provide some geographic and demographic diversity;  the survey data is representative only for those three states individually.  RAND’s report on the three-state survey data included information about  how teachers within each of the three states had changed their teaching  practices due to a state accountability test. RAND provided us with  descriptive data tables based on its school year 2005-2006 survey data; we  analyzed the data to measure associations between the strategies used and  the school characteristic variables. We requested tables that showed this  information for teachers in all schools, and separately for teachers in  different categories of schools (elementary and middle schools) and by the  school characteristics of poverty, minority, school location and AYP  status. We obtained from RAND standard error information associated  with the estimates from the different types of schools and thus were able  to test the statistical significance of differences in likelihood between what  teachers from different types of schools reported.
As part of our analyses for both surveys, we reviewed documentation and  performed electronic testing of the data obtained through the surveys. We  also conducted several interviews with several researchers responsible for  the data collection and analyses and obtained information about the  measures they took to ensure data reliability. On the basis of our efforts to  determine the reliability of the data, we determined the data from each of  these surveys were sufficiently reliable for the purposes of our study.
We reviewed existing literature to determine what researchers have found  regarding the effect of standards-based accountability systems on  instructional practices, and practices that work in raising student  achievement. To identify existing studies, we conducted searches of  various databases, such as the Education Resources Information Center,  Proquest, Dialog EDUCAT, and Education Abstracts. We also asked all of  the education researchers that we interviewed to recommend additional  studies. From these sources, we identified 251 studies that were relevant  to our study objectives about the effect of standards-based accountability  systems on instructional practices and instructional practices there are  effective in raising student achievement. We selected them according to  the following criteria: covered the years 2001 through 2008 and were either  experimental or quasi-experimental studies, literature syntheses, or  studied multiple sites. We selected the studies for our review based on  their methodological strength, given the limitations of the methods used,  and not necessarily on whether the results could be generalized. We  performed our searches from August 2008 to January 2009.
To assess the methodological quality of the selected studies, we developed  a data collection instrument to obtain information systematically about  each study being evaluated and about the features of the evaluation  methodology. We based our data collection and assessments on generally  accepted social science standards. We examined factors related to the use  of comparison and control groups; the appropriateness of sampling and  data collection methods; and for syntheses, the process and criteria used  to identify studies. A senior social scientist with training and experience in  evaluation research and methodology read and coded the methodological  discussion for each evaluation. A second senior social scientist reviewed  each completed data collection instrument and the relevant  documentation to verify the accuracy of every coded item. This review  identified 20 selected studies that met GAO’s criteria for methodological  quality.
We supplemented our synthesis by interviewing prominent education  researchers identified in frequently cited articles and through discussions  with knowledgeable individuals. We also conducted interviews with  officials at the U.S. Department of Education, including the Center on  Innovation and Improvement, and the Institute on Education Sciences’  National Center for Education Evaluation and Regional Assistance, as well  as other educational organizations. We also reviewed relevant federal laws  and regulations.
section_title: Appendix II: Analyses of the Relationship between School Characteristics and Principals’ Focus on School Improvement Strategies
paragraphs: In order to analyze the National Longitudinal Study of No Child Left  Behind (NLS-NCLB) principal survey conducted by the RAND  Corporation, we analyzed strategies on which principals most often  focused, taking into account the percentage of a school’s students  receiving free or reduced price lunch (poverty), the percentage of students  who are a racial minority (minority), whether the school is in an urban,  suburban, or rural area (school location), and the school’s adequate yearly  performance (AYP) status (see table 1). Our analyses used “odds ratios,”  generally defined as the ratio of the odds of an event occurring in one  group compared to the odds of it occurring in another group, to express  differences in the likelihoods of schools with different characteristics  using these strategies. We used odds ratios rather than percentages  because they are more appropriate for statistical modeling and  multivariate analysis. Odds ratios indicate how much higher (when they  are greater than 1.0) or lower (when they are less than 1.0) the odds were  that principals would respond that a given strategy was a major or  moderate focus. We included a reference category for the school  characteristics (low minority, low poverty, and central city) in the top row  of table 1, and put comparison groups beneath those reference categories,  as indicated by the column heading in the second row (high-minority, high- poverty, or rural schools). As an example, the third cell in the “high- minority schools” column indicates that principals in high-minority  schools were 2.65 times more likely to make “implementing new  instructional approaches or curricula in reading/language arts/English” a  focus of their school improvement efforts. In another example, the odds  that principals would “restructure the school day to teach core content  areas in greater depth (e.g., establishing a literacy block)” were 2.8 times  higher for high-poverty schools than low poverty schools, as seen in the  sixth cell under “high-poverty schools.” Those cells with an asterisk  indicate statistically significant results; that is, we have a high degree of  confidence that the differences we see are not just due to chance but show  an actual difference in the survey responses. See appendix I for further  explanation of our methodology.
section_title: Appendix III: List of Education Researchers
section_title: Appendix IV: Studies Meeting GAO’s Criteria for Methodological Quality
subsections: section_title: Laura. S Hamilton; Brian M Stecher; Jennifer Linn Russell; Julie A. Marsh; Jeremy Miles
paragraphs: “Strong States, Weak Schools: The  Benefits and Dilemmas of  Centralized Accountability”
Quasi-experimental design  with matched groups;  multiple regressions used  with data.
Literature review using a  best-evidence synthesis  (related to a meta-analysis)
section_title: Appendix V: Comments from the Department of Education
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
paragraphs: Cornelia M. Ashby (202) 512-7215 or ashbyc@gao.gov.
subsections: section_title: Staff Acknowledgments
paragraphs: Janet Mascia (Assistant Director), Bryon Gordon (Assistant Director), and  Andrew Nelson (Analyst-in-Charge) managed all aspects of the  assignment. Linda Stokes and Caitlin Tobin made significant contributions  to this report in all aspects of the work. Kate van Gelder contributed to  writing this report, and Ashley McCall contributed to research for the  report. Luann Moy, Justin Fisher, Cathy Hurley, Douglas Sloane, and John  Smale Jr. provided key technical support, and Doreen Feldman and Sheila  R. McCoy provided legal support. Mimi Nguyen developed the graphics for  the report.