No Child Left Behind Act: States Face Challenges Measuring Academic Growth
Published: 2006-07-27T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The No Child Left Behind Act (NCLBA) requires that states improve academic performance so that all students reach proficiency in reading and mathematics by 2014 and that achievement gaps close among student groups. States set annual proficiency targets using an approach known as a status model, which calculates test scores 1 year at a time. Some states have interest in using growth models that measure changes in test scores over time to determine if schools are meeting proficiency targets. The Chairman of the Committee on Education and the Workforce asked GAO to testify on its recent report on measuring academic growth. Specifically, this testimony discusses (1) how many states are using growth models and for what purposes, (2) how growth models can measure progress toward achieving key NCLBA goals, and (3) what challenges states face in using growth models especially to meet the law's key goals. While growth models may be defined as tracking the same students over time, GAO used a definition that also included tracking the performance of schools and groups of students. In comments on the report, Education said that this definition could be confusing. GAO used this definition of growth to reflect the variety of approaches states were taking.
section_title: What GAO Found
paragraphs: Nearly all states were using or considering growth models for a variety of purposes in addition to their status models as of March 2006. Twenty-six states were using growth models, and another 22 were either considering or in the process of implementing them. Most states using growth models measured progress for schools and for student groups, and 7 also measured growth for individual students. States used growth models to target resources for students that need extra help or award teachers bonuses based on their school's performance. Certain growth models are capable of tracking progress toward the goals of universal proficiency by 2014 and closing achievement gaps. For example, Massachusetts uses its model to set targets based on the growth that it expects from schools and their student groups. Schools can make adequate yearly progress (AYP) if they reach these targets, even if they fall short of reaching the statewide proficiency targets set with the state's status model. Tennessee designed a model that projects students' test scores and whether they will be proficient in the future. In this model, if 79 percent of a school's students are predicted to be proficient in 3 years, the school would reach the state's 79 percent proficiency target for the current school year (2005-2006). States face challenges measuring academic growth, such as creating data and assessment systems to support growth models and having personnel to analyze and communicate results. The use of growth models to determine AYP may also challenge states to make sure that students in low-performing schools receive needed assistance. U.S. Department of Education (Education) initiatives may help states address these challenges. Education started a pilot project for states to use growth models that meet the department's specific criteria, including models that track progress of individual students, to determine AYP. Education also provided grants to states to track individual test scores over time.

section_title: Letter
paragraphs: I am pleased to be here today to discuss our report, which describes how  states use growth models to measure academic performance and how  these models can measure progress toward achieving key goals of the   No Child Left Behind Act of 2001 (NCLBA). With annual expenditures  approaching $13 billion dollars for Title I alone, NCLBA represents the  federal government’s single largest investment in the education of the   48 million students who attend public schools. The NCLBA—the most  recent reauthorization of the Elementary and Secondary Education Act of  1965—requires states to improve academic performance so that all  students are proficient by 2014 and achievement gaps among groups such  as economically disadvantaged students close. The upcoming  reauthorization of the law presents an opportunity to discuss some key  issues associated with the act.
To measure whether schools are making adequate yearly progress (AYP)  toward having all students proficient by 2014, states set annual proficiency  targets using an approach known as a status model, which calculates test  scores 1 year at a time. With status models, states or districts determine  whether schools make AYP based on performance for the year while  generally not taking into account how much better or worse the school did  than during the previous year. Thus, a school that is showing significant  improvement in student achievement but has too few students at the  proficient level would not likely make AYP.
In addition to determining whether schools meet proficiency targets, some  states have interest in also recognizing schools that make progress toward  NCLBA goals. Growth models can measure progress in achievement or  proficiency over time and vary in complexity, such as calculating annual  progress in a school’s average test scores from year to year; estimating test  score progress while taking into account how factors such as student  background may affect such progress; or projecting future scores based on  current and prior years’ results. While growth models are sometimes  defined as tracking the same students over time, because of the  committee’s interest in the range of models states are using to measure  academic improvement, we define a growth model as a model that  measures changes in proficiency levels or test scores of a student, group,  grade, school, or district for 2 or more years. We included models that  track schools and student groups in order to provide a broad assessment  of options that may be available to states.
My testimony today will focus on how growth models may provide useful  information on academic performance. Specifically, I will discuss (1) how  many states are using growth models and for what purposes, (2) how  growth models can measure progress toward achieving key NCLBA goals,  and (3) what challenges states face in using growth models especially to  meet the law’s key goals.
My written statement is drawn from our recent report on growth models,  which we completed for the committee. For this report, we conducted a  survey of all states to determine whether they were using growth models.  We conducted telephone interviews with state and local education agency  officials in eight states that collectively use a variety of growth models,  and conducted site visits to California, Massachusetts, North Carolina and  Tennessee. For Massachusetts and Tennessee we analyzed student-level  data from selected schools to illustrate how their models measure  progress toward key NCLBA goals. We conducted this work in accordance  with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: The NCLBA requires states to set challenging academic content and  achievement standards in reading or language arts and mathematics and  determine whether school districts and schools make AYP toward meeting  these standards. To make AYP, schools generally must:    show that the percentage of students scoring at the proficient level  or higher meets the state proficiency target for the school as a  whole and for designated student groups,  test 95 percent of all students and those in designated groups, and    meet goals for an additional academic indicator, such as the state’s  graduation rate.
The purpose of Title I Part A is to improve academic achievement for  disadvantaged students. Schools receiving Title I federal funds that do not  make AYP for 2 or more years in a row must take action to assist students,  such as offering students the opportunity to transfer to other schools or  providing additional educational services like tutoring.
States measure AYP using a status model that determines whether or not  schools and students in designated groups meet proficiency targets on  state tests 1 year at a time. States generally used data from the 2001-2002  school year to set the initial percentage of students that needed to be  proficient for a school to make AYP, known as a starting point. From this  point, they set annual proficiency targets that increase up to 100 percent  by 2014. For example, for schools in a state with a starting point of 28  percent to achieve 100 percent by 2014, the percentage of students who  scored at or above proficient on the state test would have to increase by   6 percentage points each year, as shown in figure 1. Schools that do not  reach the state target will generally not make AYP.
The law indicates that states are expected to close achievement gaps, but  does not specify annual targets to measure progress toward doing so.  States, thus, have flexibility in the rate at which they close these gaps. To  determine the extent that achievement gaps are closing, states measure  the difference in the percentage of students in designated student groups  and their peers that reach proficiency. For example, an achievement gap  exists if 40 percent of a school’s non-economically disadvantaged students  were proficient compared with only 16 percent of economically  disadvantaged students, a gap of 24 percentage points. To close the gap,  the percentage of students in the economically disadvantaged group that  reaches proficiency would have to increase at a faster rate than that of  their peers.
If a school misses its status model target in a single year, the law includes  a “safe harbor” provision that provides a way for schools that are showing  significant increases in proficiency rates of student groups to make AYP.  Safe harbor measures academic performance in a way that is similar to  certain growth models do and allows a school to make AYP by reducing  the percentage of students in designated student groups that were not  proficient by 10 percent, so long as the groups also show progress on  another academic indicator. For example, in a state with a status model  target of 40 percent proficient, a school could make AYP under safe harbor  if 63 percent of a student group was not proficient compared to 70 percent  in the previous year.
section_title: Nearly All States Reported Using or Considering Growth Models to Measure Academic Performance
paragraphs: Twenty-six states reported using growth models in addition to using their  status models to track the performance of schools, designated student  groups, or individual students, as reported in our March 2006 survey.  Additionally, nearly all states are considering the use of growth models  (see fig. 2).
Of the 26 states using growth models, 19 states reported measuring  changes for schools and student groups, while 7 states reported measuring  changes for schools, student groups, and individuals, as shown in table 1.
For example, Massachusetts used a model that measures growth for the  school as a whole and for designated student groups. The state awards  points to schools in 25-point increments for each student, depending on  how students scored on the state test. Schools earn 100 points for each  student who reaches proficiency, but fewer points for students below  proficiency. The state averages the points to award a final score to  schools. Growth in Massachusetts is calculated by taking the difference in  the annual scores that a school earns between 2 years. Figure 3 illustrates  the growth a school can make from one year to the next as measured by  Massachusetts model.
Tennessee reported using a growth model that sets different goals for each  individual student based on the students’ previous test scores. The goal is  the score that a student would be expected to receive, and any difference  between a student’s expected and actual score is considered that student’s  amount of yearly growth, as shown in figure 4.
In addition, Tennessee’s model, known as the Tennessee Value-Added  Assessment System, estimates the unique contribution—the value added— that the teacher and school make to each individual student’s growth in  test scores over time. The state then uses that amount of growth, the  unique contribution of the school, and other information to determine  whether schools are below, at, or above their level of expected  performance. The model also grades schools with an A, B, C, D, or F,  which is considered a reflection of the extent to which the school is  meeting its requirements for student learning.
Seventeen of the 26 states using growth models reported that their models  were in place before the passage of the NCLBA during the 2001-2002  school year, and the remaining 9 states implemented them after the law  was passed. States used them for purposes such as rewarding effective  teachers and designing intervention plans for struggling schools. For  example, North Carolina used its model as a basis to decide whether  teachers receive bonus money. Tennessee used its value-added model to  provide information about which teachers are most effective with which  student groups. In addition to predicting students’ expected scores on  state tests, Tennessee’s model was used to predict scores on college  admissions tests, which is helpful for students who want to pursue higher  education. In addition, California used its model to identify schools  eligible for a voluntary improvement program.
section_title: Certain Growth Models Can Measure Progress toward Key NCLBA Goals
paragraphs: Certain growth models can measure progress in achieving key NCLBA  goals of reaching universal proficiency by 2014 and closing achievement  gaps. While states developed growth models for purposes other than  NCLBA, states such as Massachusetts and Tennessee have adjusted their  state models to use them to meet NCLBA goals. The Massachusetts model  has been used to make AYP determinations as part of the state’s  accountability plan in place since 2003. Tennessee submitted a new model  to Education for the growth models pilot that differs from the value-added  model described earlier. This new model gives schools credit for students  projected to reach proficiency within 3 years in order to meet key NCLBA  goals. Our analysis of how models in Massachusetts and Tennessee can  measure progress toward the law’s two key goals is shown in table 2.
Massachusetts designed a model that can measure progress toward the  key goals of NCLBA by setting targets for the improvement of schools and  their student groups that increase over time until all students are  proficient in 2014. Schools can get credit for improving student proficiency  even if, in the short term, the requisite number of students has yet to reach  the state’s status model proficiency targets. For example, figure 5  illustrates a school that is on track to make AYP annually through 2014 by  reaching its growth targets. While these growth targets increase at a faster  pace than the state’s annual proficiency target until 2014, they do provide  the school with an additional measure by which it can make AYP.
The model also measures whether achievement gaps are closing by setting  targets for designated student groups, similar to how it sets targets for  schools as a whole. Schools that increase proficiency too slowly—that is,  do not meet status or growth targets—will not make AYP. For example,  one selected school in Massachusetts showed significant gains for several  designated student groups that were measured against their own targets.  However, the school did not make AYP because gains for one student  group were not sufficient. This group—students with disabilities—fell  short of its growth target, as shown in figure 6.
Tennessee developed a different model that can also measure progress  toward the NCLBA goals of universal proficiency and closing achievement  gaps. Tennessee created a new version of the model it had been using for  state purposes to better align with NCLBA. Referred to as a projection  model, this approach projects individual student’s test scores into the  future to determine when they may reach the state’s status model  proficiency targets.
In order to make AYP under this proposal, a school could reach the state’s  status model targets by counting as proficient in the current year those  students who are predicted to be proficient in the future. The state  projects scores for elementary and middle school students 3 years into the  future to determine if they are on track to reach proficiency, as follows:  fourth-grade students projected to reach proficiency by seventh  grade,  fifth-grade students projected to reach proficiency by eighth grade,  and    sixth-, seventh-, and eighth-grade students projected to reach  proficiency on the state’s high school proficiency test.
These projections are based on prior test data and assume that the student  will attend middle or high schools with average performance (an  assumption known as average schooling experience).  At our request,  Tennessee provided analyses for students in several schools that would  make AYP under the proposed model. To demonstrate how the model  works, we selected students from a school and compared their actual  results in fourth grade (panel A) with their projected results for seventh  grade (panel B) (see fig. 7).
Tennessee’s proposed model can also measure achievement gaps. Under  NCLBA, a school makes AYP if all student groups meet the state  proficiency target. In Tennessee’s model, whether the achievement gap is  potentially closed would be determined through projections of students’  performance in meeting the state proficiency target.
section_title: States Face Challenges in Implementing Growth Models
paragraphs: States generally face challenges in collecting and analyzing the data  required to implement growth models including models that would meet  the law’s goals. In addition, using growth models can present risks for  states if schools are designated as making AYP while still needing  assistance to progress. Education has initiatives that may help states  address these challenges.
States must have certain additional data system requirements to  implement growth models, including models that would meet NCLBA  requirements.
First, a state’s ability to collect comparable data over at least 2 years is a  minimum requirement for any growth model. States must ensure that test  results are comparable from one year to the next and possibly from one  grade to the next, both of which are especially challenging when test  questions and formats change. Second, the capacity to collect data across  time and schools is also required to implement growth models that use  student-level data. This capacity often requires a statewide system to  assign unique numbers to identify individual students. Developing and  implementing these systems is a complicated process that includes  assigning numbers, setting up the system in all schools and districts, and  correctly matching individual student data over time, among other steps.  Third, states need to ensure that data are free from errors in their  calculations of performance. While ensuring data accuracy is important  for status models, doing so is particularly important for growth models,  because errors in multiple years can accumulate, leading to unreliable  results.
States also need greater research and analysis expertise to use growth  models as well as support for people who need to manage and  communicate the model’s results. For example, Tennessee officials told us  that they have contracted with a software company for several years  because of the complexity of the model and its underlying data system.  Florida has a contract with a local university to assist it with assessing  data accuracy, including unique student identifiers required for its model.  In addition, states will incur training costs as they inform teachers,  administrators, media, legislators, and the general public about the  additional complexities that occur when using growth models. For  example, administrators in one district in North Carolina told us that their  district lacks enough specialists who can explain the state’s growth model  to all principals and teachers in need of guidance and additional training.
Using growth models can present risks for states if schools are designated  as making AYP while still needing assistance to progress. On the basis of  growth model results, some schools would make AYP even though these  schools may have relatively low-achieving students. As a result, some  students in Title I schools may be disadvantaged by not receiving  federally-required services.
In two Massachusetts districts that we analyzed, 23 of the 59 schools that  made AYP did so based on the state’s growth model, even though they did  not reach the state’s status model proficiency rate targets in 2003-2004.  Consequently, these schools may not be eligible to receive services  required under NCLBA for schools in need of improvement, such as  tutoring and school choice. Because these schools would need to sustain  high growth rates in order to achieve universal proficiency by 2014, it is  likely that their students would benefit from additional support.
In Tennessee, 47 of the 353 schools that had not made AYP in the   2004-2005 school year would do so under the state’s proposed projection  model. One school that would be allowed to make AYP under the  proposed model was located in a high-poverty, inner-city neighborhood.  That school receives Title I funding, as two-thirds of its students are  classified as economically disadvantaged. The school was already  receiving services required under NCLBA to help its students. If the school  continues to make AYP under the growth model, these services may no  longer be provided.
Education’s initiatives, such as the growth model pilot project, may  facilitate growth model implementation. In November 2005, Education  announced a pilot project for states to submit proposals for using a growth  model—one that meets criteria established by the department—along with  their status model, to determine AYP. While NCLBA does not specify the  use of growth models for making AYP determinations, the department  started the pilot to evaluate how growth models might help schools meet  NCLBA proficiency goals and close achievement gaps.
For the growth model pilot project, each state had to demonstrate how its  growth model proposal met Education’s criteria, many of which are  consistent with the legal requirements of status models. In addition to  those requirements, Education included criteria that the proposed models  track student progress over time and have an assessment system with  tests that are comparable over time. Of the 20 proposals, Education  approved 2 states—North Carolina and Tennessee—to use growth models  to make AYP determinations in the 2005-2006 school year. States may  submit proposals for the pilot again this fall.
In addition to meeting all of the criteria, Education and peer reviewers  noted that Tennessee and North Carolina had many years of experience  with data systems that support growth models. These states must report to  Education the number of schools that made AYP on the basis of their  status and growth models. Education expects to share the results with  other states, Congress, and the public after it assesses the effects of the  pilot.
In addition to the growth model pilot project, Education awarded nearly  $53 million in grants to 14 states for the design and implementation of  statewide longitudinal data systems—systems that are essential for the  development of student-level growth models. While independent of the  pilot project, states with a longitudinal data system—one that gathers data  such as test scores on the same student from year to year—will be better  positioned to implement a growth model than they would have been  without it. Education intended the grants to help states generate and use  accurate and timely data to meet reporting requirements, support decision  making, and aid education research, among other purposes. Education  plans to disseminate lessons learned and solutions developed by states  that received grants.
section_title: Conclusion
paragraphs: While status models provide a snapshot of academic performance, growth  models can provide states with more detailed information on how schools’  and students’ performance has changed from year to year. Growth models  can recognize schools whose students are making significant gains on  state tests but are still not proficient. Educators can use information about  the academic growth of individual students to tailor interventions to the  needs of particular students or groups. In this respect, models that  measure individual students’ growth provide the most in-depth and useful  information, yet the majority of the models currently in use are not  designed to do this.
Through its approval of Massachusetts’ model and the growth model pilot  program, Education is proceeding prudently in its effort to allow states to  use growth models to meet NCLBA requirements. Education is allowing  only states with the most advanced models that can measure progress  toward NCLBA goals to use the models to determine AYP. Under the pilot  project, which has clear goals and criteria that requires states to compare  results from their growth model with status model results, Education is  poised to gain valuable information on whether or not growth models are  overstating progress or whether they appropriately give credit to fast- improving schools.
While growth models may be defined as tracking the same students over  time, GAO used a definition that also includes tracking the performance of  schools and groups of students. In comments on our report, Education  expressed concern that this definition may confuse readers because it is  very broad and includes models that compare changes in scores or  proficiency levels of schools or groups of students. GAO used this  definition of growth to reflect the variety of approaches states are taking  to measure academic progress.
Mr. Chairman, this concludes my prepared statement. I would be happy to  respond to any questions that you or members of the committee may have.
section_title: GAO Contact and Staff Acknowledgments
paragraphs: For more information on this testimony, please call Marnie S. Shaul at  (202) 512-7215. Individuals making key contributions to this testimony  include Blake Ainsworth, Karen Febey, Harriet Ganson, Shannon Groff,  Andrew Huddleston, Jason Palmer, and Rachael Valliere.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. It may be reproduced and distributed in its entirety without further  permission from GAO. However, because this work may contain copyrighted images or  other material, permission from the copyright holder may be necessary if you wish to  reproduce this material separately.