No Child Left Behind Act: States Face Challenges Measuring Academic Growth That Education's Initiatives May Help Address
Released: 2006-07-27T00:00:00

Letter

Background

The No Child Left Behind Act of 2001 increased the federal government’s role in kindergarten-12th grade education by setting two key goals: to reach universal proficiency so that all students score at the proficient level of achievement—as defined by the states—by 2014, and to close achievement gaps between high- and low-performing students, especially those in designated groups: students who are economically disadvantaged, are members of major racial or ethnic groups, have learning disabilities, or have limited English proficiency.

With these two key goals in mind, NCLBA requires states to set challenging academic content and achievement standards in reading or language arts and mathematics to determine whether school districts and schools make AYP toward meeting these standards.

Education has responsibility for general oversight of the NCLBA. As part of this oversight, Education is responsible for reviewing and approving state plans for meeting AYP requirements. As we have reported, it approved all states’ plans—fully or conditionally—by June 2003. It also reviews state systems of standards and assessments to ensure they are aligned with the law’s requirements. As of April 2006, Education had approved these systems for Delaware, South Carolina, and Tennessee and was in the process of reviewing them in other states.

Status Models

States measure AYP using a status model that determines whether or not schools and students in designated groups meet proficiency targets on state tests 1 year at a time. To make AYP, schools must show that the percentage of students scoring at the proficient level or higher meets the state proficiency target for the school as a whole and for designated student groups, test 95 percent of all students and those in designated groups, and meet goals for an additional academic indicator (which can be chosen by each individual state for elementary and middle schools but must be the state-defined graduation rate in high schools).

States generally used data from the 2001-2002 school year to set the initial percentage of students that needed to be proficient for a school to make AYP, known as a starting point, as prescribed in the NCLBA and Education’s guidance. Using these initial percentages, states then set annual proficiency targets that increase up to 100 percent by 2014. For example, for schools in a state with a starting point of 28 percent to achieve 100 percent by 2014, the percentage of students who scored at or above proficient on the state test would have to increase by 6 percentage points each year, as shown in figure 1. Setting targets for increasing proficiency through 2014 does not ensure that schools will raise student performance to these levels. Instead, the targets provide a goal, and schools that do not reach the goal will generally not make AYP.

School districts with schools receiving federal funds under Title I Part A that do not make AYP for 2 or more years in a row must take action to assist students, such as offering students the opportunity to transfer to other schools or providing additional educational services like tutoring. School districts with schools that meet these criteria must set aside an amount equal to 20 percent of their Title I funds to provide these services and spend up to that amount depending on how much demand exists for these services to be provided. These schools, in consultation with their districts, are also required to implement a plan to improve their students’ achievement.

The law indicates that states are expected to close achievement gaps, but does not specify annual targets to measure progress toward doing so. States thus have flexibility in the rate at which they close these gaps. To determine the extent that achievement gaps are closing, states measure the difference in the percentage of students in designated student groups and their peers that reach proficiency. Using a hypothetical example, figure 2 shows how closing achievement gaps between economically disadvantaged students and their peers would be reported.

In this example, 40 percent of the school’s non-economically disadvantaged students were proficient compared with only 16 percent of disadvantaged students in 2002, a gap of 24 percentage points. To close the gap, the percentage of students in the economically disadvantaged group that reaches proficiency would have to increase at a faster rate than that of their peers. By 2014, the gap is eliminated, with both groups at 100 percent proficient.

Safe Harbor

If a school misses its status model target, the law also provides a way for it to make AYP if it significantly increases the proficiency rates of student groups that do not meet the proficiency target. The law includes a provision, known as safe harbor, which allows a school to make AYP by reducing the percentage of students in designated student groups that were not proficient by 10 percent, so long as it also shows progress on another academic indicator. Safe harbor measures academic performance similar to certain growth models, according to one education researcher. For example, in a state with a status model target of 40 percent proficient, a school could make AYP under safe harbor if 63 percent of a student group were not proficient compared to 70 percent in the previous year. See figure 3.

Growth Models

In contrast to status models that measure the percentage of students at or above proficiency in a school 1 year at a time, growth models measure change in achievement or proficiency over time. Some of these models show changes in achievement for schools and student groups using students’ average scores. Other models provide more detailed information on how individual students progress over time. Growth models can enable school officials to monitor the year-to-year changes in performance of students across many levels of achievement, including those who may be well below or well above proficiency. They may also be used to predict test scores in future years based on current and prior performance.

While definitions of growth models vary, for this report, GAO defines a growth model as a model that measures changes in proficiency levels or test scores of a student, group, grade, school, or district for 2 or more years. Some definitions restrict the use of the term “growth models” to refer only to those models that measure changes for the same students over time. GAO included models in this report that track different groups of students in order to provide a broad assessment of options that may be available to states. Growth models can be designed to measure successive groups of students (for example, students in the third grade class in 2006 with students in the third grade class in 2005) or track a cohort of students over time (for example, students in the fourth grade in 2006 with the same students in the third grade in 2005).

School-level growth models track changes in the percentage of students that reach proficiency or their achievement scores over time. For example, the charts in figure 4 show how two hypothetical schools measure their proficiency with a status model and with a measure of progress over time.

In the case of Washington Middle School, a growth model shows a decline in performance, while a status model indicates that the school exceeded the state proficiency target of 40 percent. This school was able to make AYP even though its proficiency rate decreased. In contrast, the use of a growth model with Adams Elementary School shows that the school improved its performance, but its status model results indicate that the school did not meet the 40 percent proficiency target. That school did not make AYP, even though its proficiency rate increased. Thus, the type of model used could lead to different perspectives on how schools are performing.

Individual-level growth models track changes in proficiency or achievement for individual students over time. For example, individual student growth can be measured by comparing the difference between a student’s test scores in 2 consecutive years. A student may score 300 on a test in one year and 325 on the test in the next year, resulting in an increase of 25 points. These scores could then be averaged to measure school-level results as in the previous example. Individual student growth can also be measured over more than 2 years to identify longer-terms trends in performance. Additionally, growth can be projected into the future to predict when a student may reach proficiency, and that information may be used to target interventions to students who would otherwise continue to perform below standard.

Nearly All States Were Using or Considering Growth Models to Supplement Other Measures of Academic Performance

Nearly all states were using or considering growth models to track performance, as of March 2006. Although NCLBA requires states to use status models to determine whether schools make AYP, the 26 states with growth models reported using them for state purposes such as identifying schools in need of extra assistance. Seventeen of these states had growth models in place prior to NCLBA.

Half of the States Were Using Growth Models, and Most Remaining States Were Considering Them

Twenty-six states reported using growth models in addition to using their status models to track the performance of schools, designated student groups, or individual students, in our survey as of March 2006 (see figure 5). Additionally, nearly all states are considering the use of growth models: 20 of 26 states that used one growth model were also considering or in the process of implementing another growth model, and 22 of 25 states that did not use growth models were considering or in the process of implementing them to provide more detailed information about school, group, or student performance.

States Were Using Growth Models for State, Rather than Federal, Purposes

Seventeen of the 26 states using growth models reported that their models were in place before the passage of the NCLBA during the 2001-2002 school year, and the remaining 9 states implemented them after the law was passed, as shown in figure 8.

Once NCLBA was enacted, states were required to develop plans to show how they would meet federal requirements for accountability as measured by whether their schools made AYP. Education approved these plans, but generally did not permit states to include growth models. According to Education officials, since NCLBA requires that states make AYP determinations on the basis of the percentage of students who are proficient at one point in time—rather than the increase or decrease in that percentage over time—growth models were considered inconsistent with the goals of the act. For example, California began using its model, called the Academic Performance Index, in the 1999-2000 school year to set yearly growth targets for schools. These targets were based on combined test scores for reading/language arts, mathematics, and other subjects. However, according to officials at the California Department of Education, California’s model, developed prior to NCLBA, was not designed to explicitly achieve the law’s key goals of universal proficiency by 2014 or closing achievement gaps. Further, a California Department of Education official explained that because the model did not report scores from reading, math, and other subjects separately, California was not approved to make AYP determinations using its model. In contrast, Massachusetts’ growth model was in place prior to NCLBA passage and then was adapted to align explicitly with the law’s key goals. Education approved Massachusetts’ AYP plan, allowing the state to use both its status model and growth model to determine AYP.

Instead of using growth models to make AYP determinations, states used them for other purposes, such as rewarding effective teachers and designing intervention plans for struggling schools. For example, North Carolina used its model as a basis to decide whether teachers receive bonus money. Tennessee used its value-added model to provide information about which teachers are most effective with which student groups. In addition to predicting students’ expected scores on state tests, Tennessee’s model was used to predict scores on college admissions tests, which is helpful for students who want to pursue higher education. In addition, California used its model to identify schools eligible for a voluntary improvement program.

The type of growth model used has implications for how results may be applied. California’s model provides information about the performance of its schools, enabling the state to distinguish higher-performing from lower- performing schools. However, the model does not provide information about individual teachers or students. In contrast, Tennessee’s model does provide information about specific teachers and students, allowing the state to make inferences about how effective its teachers are. While California may use its results for interventions in schools, Tennessee may use its results to target interventions to individual students.

Certain Growth Models Provide Useful Information on the Extent That Schools Are Achieving Key NCLBA Goals but May Overlook Some Low- Performing Schools if Used for AYP

Certain growth models measure the extent that schools and students are achieving key NCLBA goals. While the use of growth models may allow states to recognize gains schools are making toward the law’s goals, it may also put students in some lower-performing schools at risk for not receiving additional federal assistance. While states developed growth models for purposes other than NCLBA, states such as Massachusetts and Tennessee have adjusted their state models to use them to meet NCLBA goals. The Massachusetts model has been used to make AYP determinations as part of the state’s accountability plan in place since 2003. This model is approved by Education in part because it complies with the key goal of universal proficiency by 2014. Tennessee submitted a new model to Education for the growth model pilot project that differs from the value-added model we describe earlier. The value-added model, developed several years prior to NCLBA, gives schools credit for students who exceeded their growth expectations. The new model gives schools credit for students projected to reach proficiency within 3 years in order to comply with the key NCLBA goal of showing that students are on track to reach proficiency by 2014.

Certain Growth Models Measure Progress in Achieving Key NCLBA Goals of Universal Proficiency by 2014 and Closing Achievement Gaps

Like status models, certain growth models can measure progress in achieving key NCLBA goals of reaching universal proficiency by 2014 and closing achievement gaps. Our analysis of how models in Massachusetts and Tennessee can track progress toward the law’s two key goals is shown in table 2.

Our analysis of data from selected schools in those states demonstrates how these models measure progress toward the key goals. One school in Massachusetts had a baseline score of 27.4 points in math. Its growth target for the following 2-year cycle was 12.1, requiring it to reach 39.5 points by 2004. In comparison, the state’s target using its status model was 60.8 points in 2004. The growth target was set at 12.1 because, if the school’s points increased this much in each of the state’s six cycles, the school would have 100 points by 2014. In so doing, it would reach universal proficiency in that year, as is seen in figure 9.

In fact, the school scored 42.6 in 2004, thus exceeding its target of 39.5. The school also showed significant gains for several designated student groups that were measured against their own targets. However, the school did not make AYP because gains for one student group were not sufficient. This group—students with disabilities—showed gains of 9.3 points resulting in a final score of 23.6 points, short of its growth target of 28.6. Figure 10 compares this school’s baseline, target, and first cycle results for the school as a whole and for selected student groups.

Massachusetts has designed a model that can measure progress toward the key goal of NCLBA by setting growth targets for each year until all students are proficient in 2014. Schools like the one mentioned above can get credit for improving student proficiency even if, in the short term, the requisite number of students have yet to reach the current status model proficiency targets. The model also measures whether achievement gaps are closing by setting targets for designated student groups, similar to how it sets targets for schools as a whole. Schools that increase proficiency too slowly—that is, do not meet status or growth targets—will not make AYP.

Tennessee developed a different model that also measures progress toward the NCLBA goals of universal proficiency and closing achievement gaps. Tennessee created a new version of the model it had been using for state purposes to better align with NCLBA. Referred to as a projection model, this approach projects individual student’s test scores into the future to determine when they may reach the state’s status model proficiency targets in the future. This model was accepted as part of Education’s pilot project, allowing the state to use it to make AYP determinations in the 2005-2006 school year.

In order to make AYP under this proposal, a school could reach the state’s status model targets by counting as proficient those students who are predicted to be proficient in the future. The state projects scores for elementary and middle school students 3 years into the future to determine if they are on track to reach proficiency, as follows: fourth grade students projected to reach proficiency by seventh grade, fifth grade students projected to reach proficiency by eighth grade, and sixth, seventh, and eighth grade students projected to reach proficiency on the state’s high school proficiency test.

These projections are based on prior test data and are not based on student characteristics. Also, the projections are based on the assumption that the student will attend middle or high schools with average performance (an assumption known as average schooling experience), and allow the student’s current school to count them as proficient in the current year if they are projected to be proficient in the future.

Tennessee estimated that of its 1,341 elementary and middle schools, 47 schools that did not make AYP using its status model would be able to make AYP under its proposed model that gives schools credit for students projected to be proficient in the future. At our request, Tennessee provided analyses for students in several schools that would make AYP under the proposed model. To demonstrate how the model works, we selected students from a school and compared their actual results in fourth grade (Panel A) with their projected results for seventh grade (Panel B) (see figure 11).

Some students who were not proficient based on their scores in 2004-2005 were projected to be proficient by the time they reach later grades. For example, student A did not score at the proficient level in fourth grade but was projected to score at the proficient level in seventh grade. The state has proposed to determine whether schools make AYP by using the percentage of students who are projected to be proficient (like student A) in the future, instead of the percentage of students presently proficient. For example, if 79 percent of an elementary school’s students are projected to be proficient on future math tests, the school will make AYP for the state’s 79 percent target in the 2005-2006 school year, regardless of the percentage of students in that school that are currently proficient.

Tennessee’s proposed model can also measure achievement gaps. Under NCLBA, a school makes AYP if all student groups meet the state proficiency target. For example, a school could have a 20 percentage point gap for one group if, for example, 59 percent of students with limited English proficiency were proficient compared to 79 percent of their peers. While results based on projections may show that achievement gaps are closing, gaps would actually be closed only if the projections were realized.

Giving Credit for Growth May Overlook Some Low- Performing Schools in the Near Term

Using these models to measure progress, states could recognize improvement by allowing some schools to make AYP even though the schools may have relatively low achieving students. These schools may have a long way to go before reaching 100 percent proficiency and will need to increase student proficiency at a faster rate than schools making AYP under a status model. If a school that receives funds under Title I is unable to sustain this rate of progress, it may have difficulty reaching universal proficiency by 2014. In addition, if a school that did not meet status model targets but made AYP by meeting growth model targets, its students may not qualify for additional assistance provided for by NCLBA. Schools that receive Title I funds and that do not make AYP for 2 consecutive years are identified for improvement. According to some school district officials, it may be helpful not to be identified for improvement because they can devise their own interventions instead of implementing school transfer programs or working with state-approved supplemental educational service providers. While delaying these interventions may disadvantage students in some Title I schools, reducing the number of schools identified for improvement could allow for greater concentration of dollars in the lowest-performing schools.

In Massachusetts, of the 134 schools in the two districts we analyzed, 23 of the 59 schools that made AYP did so based on the state’s growth model even though they did not reach the state’s status model proficiency rate targets in 2003-2004. The state had its growth model approved by Education as part of its accountability plan and therefore was able to determine that these 23 schools made AYP. One of these schools served a high-minority, low-income population and missed the state proficiency target in English/Language Arts of 75.6 points for the school as a whole and for each of its student groups. For example, one student group, students with disabilities, scored 44.3 points, missing the target by 31.3. However, this school made AYP, because the school as a whole and each of its student groups had shown enough improvement to meet their growth targets—including the group of students with disabilities that improved by 6.8 points.

In Tennessee—of the 1,341 schools for which the state made AYP determinations in the 2004-2005 school year—47 of the 353 schools (13.3 percent) that had not made AYP would do so if the state’s proposed projection model were applied. However, some of these schools have many other indicators of needing assistance. For example, one school that would be allowed to make AYP under the proposed model was located in a high-poverty, inner-city neighborhood. That school receives Title I funding, as two-thirds of its students are classified as economically disadvantaged. The school was already receiving services required under NCLBA to help its students. If it makes AYP 2 years in a row, these services may no longer be required.

Additionally, estimates of future proficiency often rely on certain assumptions. In the case of Tennessee’s proposed model, a key assumption is that students would receive an average schooling experience in the years between when the data were measured and when the final projection is made. According to Tennessee officials, an average schooling experience is defined as one in which a student receives instruction in a school whose performance is the average of all schools in the state. To the extent that a student attends a school with performance that is significantly different from average, actual performance is likely to deviate from the estimates, rendering those estimates relatively less reliable. Moreover, by allowing a school to count students’ future proficiency in the current year, the Tennessee proposal may only be delaying a school’s inability to meet status model targets and forestalling needed assistance.

States Face Challenges In Measuring Year-to- Year Growth That Education’s Pilot Program and Data System Grants May Help Address

States face challenges in implementing growth models that Education’s initiatives may help address. Challenges states face include the extent that states’ data and assessment systems will support the models, whether the models can generate valid and reliable results, and states’ expertise to use, manage and communicate results about growth. These challenges are generally similar to those faced by states in implementing status models but are accentuated, because growth models measure progress over multiple years and thus require more data and systems designed to track data over time. Education’s growth model pilot program and data system grants may make it possible for more states to meet AYP requirements using a growth model, but greater usage largely depends upon improving states’ data and assessment systems.

Requirements of Growth Models Pose Challenges for Implementation

One challenge states face in using growth models is the ability to collect comparable data over at least 2 years, a minimum requirement for any growth model. States must ensure that test results are comparable from one year to the next and possibly from one grade to the next, both of which are especially challenging when test questions and formats change. Depending on the type of model, states may incorporate scores from 2, 3, or even more prior years. Officials from 13 states that were implementing or considering the use of growth models told us that they need to consider their state’s ability to make comparisons from one year to the next before their model could be operational. Other states that are implementing new data systems or assessments may have to wait a few years before they have enough data to assess progress from one year to the next. For example, one of those state officials said that his state will need at least 3 years of test data in order to set realistic multiyear growth targets for its proposed growth model. Some states currently using growth models, such as Florida and Ohio, have been collecting and comparing student data for several years.

A significant challenge to implementing growth models that use student- level data is the capacity to collect these data across time and schools. This capacity often requires a statewide system to assign unique numbers to identify individual students. At least 37 states have systems with unique numbers as of April 2006, according to officials with the Data Quality Campaign (a nonprofit organization that helps states improve data quality). Developing and implementing these systems is a complicated process that includes assigning numbers, setting up the system in all schools and districts, and correctly matching individual student data over time, among other steps. For example, school staff must have students’ unique numbers when students change schools. However, Education officials have cited cases of school staff assigning a new number for a student instead of locating the student’s original number. Additionally, peer reviewers for Education’s growth model pilot project cited concerns about the ability of 3 states to correctly match student data from year to year. Some states have contracted with outside organizations to assist them in establishing these systems. In addition, one model provides a “teacher effect score” as an estimate of the impact that individual teachers have on individual students’ academic achievement, thus requiring even more information.

Ensuring data are free from errors is important for calculations using status models and growth models. Doing so is more important when using growth models, because errors in multiple years can accumulate, leading to unreliable results. Fourteen state officials cited concerns about the design and reliability of growth models in areas ensuring data accuracy and measuring progress.

States also need greater research and analysis expertise to use growth models as well as support for people who need to manage and communicate the model’s results. For example, Tennessee officials told us that they have contracted with a software company for several years because of the complexity of the model and its underlying data system. Florida has a contract with a local university to assist it with assessing data accuracy, including unique student identifiers required for its model. In addition, states will incur training costs as they inform teachers, administrators, media, legislators, and the general public about the additional complexities that occur when using growth models. For example, administrators in one district in North Carolina told us that personnel issues are their main concerns with using growth models. Their district lacks enough specialists who can explain the state’s growth model to all principals and teachers in need of guidance and additional training. In an effort to address their limited capacity, district officials told us they have been collaborating with neighboring districts to share training resources regarding the state’s growth model.

Education Has Two Programs to Help States Develop Growth Models

In November 2005, Education announced a pilot project for states to submit proposals for using a growth model—one that meets criteria established by the department—along with their status model, to determine AYP. Education officials told us that the department is conducting its pilot project under authority provided in the law that, upon request from a state, allows the Secretary to waive certain requirements in the NCLBA. While the NCLBA does not specify the use of growth models for making AYP determinations, the department started the pilot in part to gain information on how these models might help schools achieve the law’s key goals. According to Education officials, 7 states had already requested to use growth models for AYP determinations before the department invited states to submit growth model proposals.

For the growth model pilot project, each state had to demonstrate how its growth model proposal met Education’s criteria, referred to as “core principles” outlined in its November 2005 announcement. While many of these criteria are consistent with the legal requirements of status models, tracking student progress over time and having an assessment system with tests that are comparable over time are new (see table 3).

Twenty states submitted proposals to Education by the February 17, 2006 deadline. Education reviewed proposals from the 14 states that planned to make AYP determinations for the 2005-2006 school year and forwarded 8 of them for peer review. In May 2006, Education approved North Carolina and Tennessee to use their proposed growth models to make AYP determinations for the 2005-2006 school year. Education noted that those states met all of the department’s criteria, such as reaching the key NCLBA goals of universal proficiency and closing achievement gaps. Additionally, Education and peer reviewers noted that those states had many years of experience with data systems that support calculating results using growth models. The 6 states whose proposals had received peer review were invited to resubmit proposals in September 2006. Other states that had submitted proposals for the 2006-2007 school year, and those that had not previously submitted proposals were invited to do so by November 1, 2006, for potential implementation in the 2006-2007 school year.

While Tennessee received unconditional approval to implement its proposed growth model, peer reviewers noted they were concerned that Tennessee’s use of “average school experience” is likely to result in inaccurate projections, especially for disadvantaged students. This is because many students attend schools in districts that are struggling, and the schools they are likely to attend 3 years out could provide them with a school experience that is markedly below average. For this reason, Education requested that the state, after it implements the model, provide data to compare actual results with its projections.

North Carolina received approval as long as its system of standards and assessments was approved by July 1, 2006. Reviewers of the state’s proposal noted that the state proposed to average student results for calculating growth, instead of examining growth results of all students, in direct violation of Education’s criteria. According to Education, the state changed its original approach so that growth would account for all students and would not use averages.

Six states had proposals that were peer-reviewed but not approved. The department cited a variety of reasons for not approving these proposals, including that they did not lead to universal proficiency by 2014, applied growth calculations to nonproficient students only (instead of all students), used a margin of error on individual test results that would likely lead to students’ being counted as proficient when in fact they were not, and proposed annually resetting growth targets. Education is allowing these states to resubmit their proposals for review later in 2006. If approved then, they can use growth models to make AYP determinations in the 2006-2007 school year.

Approved states must report to Education the number of schools that made AYP on the basis of their status and growth models. Education expects to share the results with other states, Congress, and the public after it assesses the effects of the pilot.

In addition to the growth model pilot project, Education announced in April 2005 a competition for grants for the design and implementation of statewide longitudinal data systems. While independent of the pilot project, states with a longitudinal data system—one that gathers data on the same student from year to year—will be better positioned to implement a growth model than they would have been without it. Many states applied to participate in the growth model pilot project or received a grant (see table 4).

Longitudinal data systems link data, such as test scores and enrollment patterns, of individual students over time. Education intended the grants to help states generate and use accurate and timely data to meet reporting requirements, support decision making, and aid education research, among other purposes. Education received applications from 45 states for the 3-year grants, and in November 2005, Education awarded a total of $52.8 million in grants to 14 states. States receiving grants must submit annual and final reports on the status of the development and the implementation of these systems. Education plans to disseminate lessons learned and solutions developed by states that received grants.

Concluding Observations

While status models provide a snapshot of academic performance, growth models can provide states with more detailed information on how schools’ and students’ performance has changed from year to year. Growth models can recognize schools whose students are making significant gains on state tests but are still not proficient and may provide incentives for schools with mostly proficient students to make greater improvements. Educators can use the growth models of individual students to tailor interventions to the needs of particular students or groups. In this respect, models that measure individual students’ growth provide the most in- depth and useful information, yet most of the models currently in use are not designed to do this.

Through its approval of Massachusetts’ model and the growth model pilot program, Education is proceeding prudently in its effort to allow states to use growth models to meet NCLBA requirements. Education is allowing only states with the most advanced models that can measure progress toward NCLBA goals to use the models to determine AYP. If schools are allowed to make AYP by getting credit for growth, some lower-performing schools will make AYP and the opportunity for school improvements the federal law prescribes to help students may be missed. However, if schools that show the most growth but do not meet status model targets are permitted to make AYP, states could target Title I school improvement on their lowest-performing schools.

By proceeding with a pilot project with clear goals and criteria and by requiring states to compare results from their growth model with status model results, Education is poised to gain valuable information on whether or not growth models are overstating progress or whether they appropriately give credit to fast-improving schools.

Agency Comments and Our Evaluation

We obtained written comments on a draft of this report from the Department of Education. Education’s comments are reproduced in appendix III. Education also provided additional technical comments, which have been included in the report as appropriate.

Education commented that it appreciates our concluding observation that the department “is poised to gain valuable information on whether or not growth models are overstating progress or whether they appropriately give credit to fast-improving schools.” Education expressed concern that the definition of growth models used in the report may confuse readers because it is very broad and includes models that compare changes in scores or proficiency levels of schools or groups of students. To inform its pilot project, Education used research that defines the term “growth model” to refer to models that track the growth of individual students.

For the purposes of this report, we defined growth models to include models that track growth of schools, groups of students, and individual students over time. While we acknowledge that some research exists to define growth models as tracking the same students over time, other research exists to show that there are different ways of classifying models that states use or could potentially use. As such, the definition used in this report reflects the variety of approaches states are taking to measure academic progress.

As agreed with your staff, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to the Secretary of Education and other interested parties. We will also make copies available to others upon request. In addition, the report will be made available at no charge on GAO’s Web site at http://www.gao.gov.

Please contact me at (202) 512-7215 if you or your staff have any questions about this report. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors are listed in appendix IV.

Appendix I: Scope and Methodology

To address the objectives of this study, we used a variety of methodological models. We interviewed experts in the field of measuring academic achievement as well as state, district, and school officials. We also reviewed documentation from states’ Web sites, and examined published studies that detailed characteristics and policy issues of states’ models. We conducted a series of interviews in selected states with officials who had a variety of experiences and viewpoints on growth models. In four of those states—California, Massachusetts, North Carolina, and Tennessee—we interviewed officials at the state, district, and school levels so that we could obtain a variety of perspectives on growth models. We selected those states for in-depth interviews based on diverse characteristics of their respective models, all of which were in place prior to the No Child Left Behind Act of 2001 (NCLBA).

To address the first objective, we surveyed state education agencies in the 50 states, the District of Columbia, and Puerto Rico and reviewed documentation from states’ accountability workbooks. States reported to us whether they were using or considering the use of a growth model to measure academic achievement. The surveys were conducted using self- administered electronic questionnaires sent in an e-mail to all 52 states beginning January 13, 2006. We closed the survey on March 16, 2006, after the 51st respondent had replied. Puerto Rico did not complete the survey.

The survey asked respondents to indicate, first, whether the state was currently using a growth model. GAO classified school-level models, like improvement models, as growth models for the purposes of this report. Some restrict the use of the term “growth models” to refer only to those that measure changes for the same group of students or individual students over time (see, for example, Council of Chief State School Officers, Policymakers’ Guide to Growth Models for School Accountability: How do Accountability Models Differ? Washington, D.C.: Oct. 2005). GAO included school-level models in this study to provide a broader assessment of options that may be available to states. If the state was using a growth model, we asked about its characteristics, whether the state was considering use of an additional model, whether the state planned to apply to Education’s growth model pilot program, and how the results from its model were used. If the state was not using a growth model, we asked whether it was considering doing so. We also asked about characteristics of the model under consideration and about key issues that must be addressed in order for it to be implemented. In some cases, we asked additional questions in e-mails and in phone interviews. The other methods we used to learn about states’ models included reviewing documentation from states’ Web sites and examining published studies that detailed characteristics and policy issues of states’ models.

To address the second objective, we analyzed data from selected schools from two states, Massachusetts and Tennessee. These states were chosen based on a variety of factors, including expert recommendation, their use of different growth models, geographic diversity, and data availability. Within these states, we selected schools that were in urban, suburban, and rural areas.

For Massachusetts, for one urban district and one suburban district, we selected the median school (as measured by the schools’ index values) among schools that had shown growth but had not made adequate yearly progress in the 2004-2005 school year.

For Tennessee, for one urban district and one rural district, we selected schools that were used in the state’s growth model pilot project proposal.

State officials from Massachusetts provided individual student data to GAO from the two selected school districts. GAO reviewed the state’s adequate yearly progress and growth model calculations and replicated school-level index and calculations from student and statewide data. State officials from Tennessee provided analyses its contractor had performed, also using individual student data. In both cases, GAO conducted an assessment of the reliability of these data and found the data to be sufficiently reliable for illustrating how growth models measure progress toward key goals of NCLBA. These assessments included electronic testing of data fields and interviews with state officials and in Tennessee’s case, the contractor as well. These interviews consisted of questions regarding the history of the data system, system audits and security, and possible threats to the systems, among other topics. GAO’s assessments also included reviews of documentation regarding the data systems.

To address the third objective, we used data from the survey and information provided to us by Education and state officials. We reviewed documentation related to Education’s growth model pilot project and proposals submitted by several states. We interviewed Education and state officials about the pilot project, including criteria for selection and processes for review and approval. We conducted our work between June 2005 and May 2006 in accordance with generally accepted government auditing standards.

Appendix II: Selected Data from GAO’s Survey of States’ Use or Consideration of Growth Models

The tables below provide specific information on characteristics of states’ growth models (as of the 2005-2006 school year), as reported on the survey. This information includes the grades in which growth models were reported, the level at which growth models were reported, the measures of achievement used to determine growth in test scores, and the characteristics of the assessments used to compare students’ test scores.

Growth Model Reporting by Grade

States using growth models varied as to whether or not they used test scores from consecutive grades. Seventeen states reported using growth models in consecutive grades, while 9 states reported using them in nonconsecutive grades. For example, Tennessee uses test scores from grades 4 through 12, while Vermont uses grades 5, 8, and 10. Whether states used test scores from consecutive grades may depend on the type of model they used. The states that reported measuring individual student growth used test scores in consecutive grades (for example, grades 3 through 12 or 4 through 10). In contrast, the 19 states that use school-level information in their growth model calculations varied in the combination of grades they used in their models: 11 of those 19 states used growth models in three or more 8 used a variety of grade combinations.

For each state with a growth model, table 5 lists the grades in which the state reports school growth and indicates whether the model measures individual student growth.

Level of Reporting Growth Model Results

States with growth models reported results for schools but varied in terms of reporting results at other levels, such as the individual student or school district. Table 6 lists the different levels at which states with growth models reported results.

Measure of Achievement in Growth Models

The measure of achievement in growth models indicates the methods states use to compare individual and group scores to determine the amount of growth. Table 7 outlines the measures that each state with a growth model used to determine how growth is reported.

Characteristics of Assessments Used in Growth Models

Growth models rely on data from state proficiency tests and measure growth with a variety of characteristics as shown in table 8.

Appendix III: Comments from the Department of Education

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

Blake Ainsworth (Assistant Director), Jason Palmer (Analyst-in-Charge), and Dan Alspaugh (Analyst-in-Charge) managed the assignment. Karen Febey, Shannon Groff, and Robert Miller made significant contributions to this report, in all aspects of the work. Kathy Larin, Harriet Ganson, Lise Levie, Beth Morrison, and Rachael Valliere provided analytic assistance. Luann Moy provided support with the survey. Anna Maria Ortiz and Beverly Ross provided analytic assistance with measuring school results related to key NCLBA goals. Jim Rebbe provided legal support and Mimi Nguyen developed the report’s graphics.

Related GAO Products

No Child Left Behind Act: Improved Accessibility to Education’s Information Could Help States Further Implement Teacher Qualification Requirements. GAO-06-25. Washington, D.C.: Nov. 21, 2005.

No Child Left Behind Act: Education Could Do More to Help States Better Define Graduation Rates and Improve Knowledge about Intervention Strategies. GAO-05-879. Washington, D.C.: Sept. 20, 2005.

No Child Left Behind Act: Most Students with Disabilities Participated in Statewide Assessments, but Inclusion Options Could Be Improved. GAO-05-618. Washington, D.C.: July 20, 2005.

Charter Schools: To Enhance Education’s Monitoring and Research, More Charter School-Level Data Are Needed. GAO-05-5. Washington, D.C.: Jan. 12, 2005.

No Child Left Behind Act: Education Needs to Provide Additional Technical Assistance and Conduct Implementation Studies for School Choice Provision. GAO-05-7. Washington, D.C.: Dec. 10, 2004.

No Child Left Behind Act: Improvements Needed in Education’s Process for Tracking States’ Implementation of Key Provisions. GAO-04-734. Washington, D.C.: Sept. 30, 2004.

No Child Left Behind Act: Additional Assistance and Research on Effective Strategies Would Help Small Rural Districts. GAO-04-909. Washington, D.C.: Sept. 23, 2004.

Special Education: Additional Assistance and Better Coordination Needed among Education Offices to Help States Meet the NCLBA Teacher Requirements. GAO-04-659. Washington, D.C.: July 15, 2004.

Student Mentoring Programs: Education’s Monitoring and Information Sharing Could Be Improved. GAO-04-581. Washington, D.C.: June 25, 2004.

No Child Left Behind Act: More Information Would Help States Determine Which Teachers Are Highly Qualified. GAO-03-631. Washington, D.C.: July 17, 2003.

Title I: Characteristics of Tests Will Influence Expenses; Information Sharing May Help States Realize Efficiencies. GAO-03-389. Washington, D.C.: May 8, 2003.