Title I: Education Needs to Monitor States' Scoring of Assessments
Released: 2002-04-01T00:00:00

Letter

Background

The original Title I legislation was passed in 1965, but the 1994 reauthorization of ESEA mandated fundamental changes to the Title I program. One of the key changes involved the development of state systems of standards and assessments to ensure that students served by Title I were held to the same standards of achievement as all other children. Prior to 1994, some states had already implemented assessment systems, but these tended to be norm-referenced—students’ performance was judged in relation to the performance of other students. The 1994 legislation required assessments that were criterion-based—students’ performance was to be judged against an objective standard. Every state applying for Title I funds since 1994 agreed to implement the changes described in the 1994 law and to bring its assessment systems into compliance. States are also required to develop a definition of adequate yearly progress based on the assessments to hold schools accountable for educational progress. To help states that could not meet the proposed 2001 timeline, Education had authority to grant timeline waivers and compliance agreements to states under certain conditions. In its 2001 ESEA reauthorization, Congress increased testing requirements for states as well as the consequences for not improving test scores in schools and did not eliminate any of the requirements of the 1994 legislation. As shown in table 1, the 1994 and 2001 legislative requirements for assessment and accountability concern developing standards for content and performance; measuring improvement; implementing and administering assessments, including assessing students with limited English proficiency; reporting assessment data; and applying consequences for not meeting performance goals.

Almost all states employ contractors to perform services to help them meet these requirements. Among states that we interviewed, contractors included private companies, universities, nonprofit organizations, and individual consultants. These entities were hired to provide services that may include assessment development, administration, scoring, analysis, and reporting of results. Some of these entities can provide combinations of services to states, such as test development and test scoring. States are responsible for monitoring contractor performance.

Congress allowed states to phase in the 1994 ESEA requirements over time, giving states until the beginning of the 2000-01 school year to fully implement them with the possibility of limited extensions. Education is responsible for determining whether or not a state is in compliance with these requirements and is authorized under ESEA, to give states more time to implement the requirements as long as states are making adequate progress toward this goal. States submit evidence to Education showing that their system for assessing students and holding schools accountable meets Title I requirements. Education has contracted with individuals with expertise in assessments and Title I to review this evidence. The experts provide Education with a report on the status of each state regarding the degree to which a state’s system for assessing students meets the requirements and deserves approval. Using this and other information, the Secretary sends each state a decision letter that summarizes the experts’ review and communicates whether a state is in full compliance, in need of a timeline waiver, or more seriously, a compliance agreement. Education may withhold funds if a state does not meet the terms of its compliance agreement. The 1994 legislation was not specific in the amount of administrative funds that could be withheld from states failing to meet negotiated timelines, but the 2001 legislation states that Education must withhold 25 percent of state administrative funds until the state meets the 1994 requirements including the terms of any timeline waivers or compliance agreements.

In June 2000, we issued a report on states’ efforts to ensure compliance with key Title I requirements. At that time, we expressed concern about the number of states that were not positioned to meet the deadlines in the 1994 law. To increase compliance, we made two recommendations. We recommended that the Department of Education should (1) facilitate among states the exchange of information and best practices regarding the implementation of Title I requirements and (2) implement additional measures to improve research on the effectiveness of different services provided through Title I to improve student outcomes. Education continues to work on the implementation of these recommendations. In addition, we said that Congress should consider requiring that states’ definitions of adequate yearly progress apply to disadvantaged children, as well as to the overall student population. The 2001 legislation does require that states apply adequate yearly progress requirements and report on the results by subgroups, including students in poverty, with disabilities, and with limited English proficiency.

Most States Are Not in Compliance with the 1994 Title I Accountability and Assessment Requirements

As of March 2002, 17 states were in compliance with the 1994 Title I assessment requirements; however, 35 were not. (See table 2.) Departmental approval of timeline waivers to give states more time to reach compliance has been granted for 30 states. Education has asked five states to enter into compliance agreements that will establish the final date by which they must be in compliance before losing Title I funding. Among other requirements, states that are not in compliance have most frequently not met the specific requirements to assess all students and break out assessment data by subcategories of students. The 2001 legislation requires states to implement additional assessments through 2008, thus substantially augmenting current assessment requirements. Education has published a notice of proposed negotiated rulemaking in the Federal Register and has solicited comments from outside parties in preparation for establishing state compliance standards for the 2001 legislation.

When Education determines that a state is not in compliance with the 1994 Title I assessment requirements, it may grant the state a timeline waiver for meeting those requirements. A waiver may not exceed 3 years. Education officials indicate that the agency grants waivers to states that have a history of success in implementing significant portions of their assessment systems, have a clear plan with a definite timeline for complying with the Title I requirements, and have the capacity to carry out the plan and thus meet those requirements. When a state requests a waiver, it must provide Education with a plan that includes a timeline for addressing deficiencies in the state’s assessment system. Education reviews this information to decide whether the waiver should be granted and its duration. So far, Education has granted timeline waivers to 30 states. (see table 3.)

A compliance agreement is deemed necessary when Education determines that a state will not complete the implementation of its assessment system in a timely manner. According to Education officials, a state requiring a compliance agreement generally does not have a history of successful implementation, has not met a significant number of Title I requirements, and does not have a plan in place for meeting those requirements. Education recommends a compliance agreement so that a state may continue to receive Title I funds. Before Education may enter into a compliance agreement, a public hearing must be held in which the state has the burden of persuading Education that full compliance with the Title I requirements is not feasible until a future date. The state must be able to attain compliance within 3 years of the signing of the compliance agreement by the state and Education. The state then negotiates the terms of the agreement with Education. Education’s written findings and the terms of the compliance agreement are published in the Federal Register. A state that enters into a compliance agreement to address requirements of the 1994 Title I law and subsequently fails to meet the requirements of the agreement can be subject to loss of some state Title I administrative funds. Education is presently working on five compliance agreements (Alabama, Idaho, Montana, West Virginia, and the District of Columbia) and has held public hearings for each.

The 2001 reauthorization of ESEA was signed into law on January 8, 2002. The act provides states not in compliance with the 1994 Title I requirements at the time of the signing of the 2001 legislation with a 90-day period that started on January 8, 2002 to negotiate changes in the dates by which they must be in compliance with the 1994 requirements. After the conclusion of this 90-day period, the legislation prohibits further extensions for compliance with the 1994 requirements. States failing to meet these negotiated timelines will be subject to loss of some of their Title I administrative funds. According to senior Education officials, this loss could be significant to states, as many use federal program administrative funds to pay the salaries of state department of education staff.

A review of documents from Education shows that noncompliant states have most commonly not met two Title I requirements—assessing all students and breaking out assessment data by subcategories of students. Title I does not permit states to exempt any student subgroup from their assessments and Education’s guidance states that individual exemptions may be permitted by the states in extraordinary circumstances. Nonetheless, many states allow substantial exemptions for students with disabilities and limited English proficiency. Several states reported that they have only recently amended laws that prohibited testing of some students with limited English proficiency. Title I also requires states, local districts, and schools to report the performance of students overall and in a variety of subcategories. These categories are gender, race, ethnicity, English proficiency status, migrant status, disability status, and economic disadvantage. Many states disaggregated data for some but not all of these categories. Documents from Education show that data for the disabled, migrant, and economically disadvantaged subcategories are the most common subgroups excluded from state, district, and school reports. In addition, many states lag in other areas, such as aligning assessments to state content standards.

To achieve compliance with the 2001 legislation, states will need to add new standards and increase assessment efforts, as detailed in table 1. In responding to our survey, 48 states reported that they have developed content standards in science, but only 16 reported having annual assessments for math and 18 reported annual assessments for reading in all grades 3 through 8. In addition, states will not have the 2 to 3 year timeline waivers available to them as they had when they worked to meet the 1994 requirements. New 2001 requirements listed in table 1 have deadlines that vary according to the requirement, and the Secretary of Education can give states 1 additional year from those deadlines to meet the new requirements, but only in case of a “natural disaster or a precipitous and unforeseen decline in the financial resources of the state.” Since the majority of states have not met the requirements of the 1994 law, it appears that many states may not be well-situated as they work to meet the schedule for implementing new requirements that build upon the 1994 requirements.

States Cited Support and Coordination as Furthering Compliance

States successful in meeting key Title I requirements attributed their success primarily to four factors. These factors were (1) the efforts of state leaders to make Title I compliance a priority; (2) coordination between staff of different agencies and levels of government; (3) obtaining buy-in from local administrators, educators, and parents; and (4) the availability of state-level expertise. Survey respondents identified inadequate funding as an obstacle to compliance. The state Title I officials we interviewed said that their states’ commitment of resources to norm- referenced assessments that conflicted with the 1994 Title I requirements contributed to this obstacle.

Almost 80 percent of the respondents identified state leaders’ efforts as a factor that facilitated their meeting the 1994 Title I requirements. In every state that had attained compliance with the Title I requirements, the officials that we interviewed said that the governor, legislators, or business leaders made compliance with the Title I requirements a high-priority. States described the development of high-level committees, new state legislation, and other measures to raise the visibility and priority of this issue. For example, one governor spearheaded a plan that used commissions to develop content standards and assessments aligned with those standards. Some state officials we interviewed reported that efforts by state department of education leaders resulted in major organizational changes in the state education department. For example, according to one Title I Director, the state changed the organizational structure and reporting relationships of state offices to organize them by function rather than by funding streams and to enhance coordination; according to another Title I Director, state leaders who did not support changes necessary to achieve compliance with Title I were replaced with staff that did support the changes.

In responding to our survey, over 80 percent of the Title I officials identified the ability of staff or agencies to coordinate their efforts with one another as a factor that helped them meet requirements. In our interviews, state officials cited the necessity of coordination between state and local staff working in the areas of assessment, instruction, and procurement. Two of the states we interviewed specifically noted that when the assessment office shared a physical location with the Title I office, coordination was easier and the ability to achieve compliance with Title I was enhanced.

Title I and other officials we interviewed in those states that had met the 1994 Title I assessment requirements noted that they had made great efforts to obtain buy-in from other state officials, local administrators, educators, and the public. They said that efforts to ensure buy-in paved the way for changes meant to ensure compliance with the assessment and accountability requirements of the1994 legislation. Several officials we interviewed reported holding public meetings and focus groups to obtain input from parents, teachers and local administrators regarding how the state should implement Title I requirements. They also reported conducting public relations campaigns to educate the public about the importance of complying with Title I requirements for standards-based assessment. One state, for example, conducted 6 years of focus groups and hearings and conducts a conference annually to allow local education officials to gain advice from experts regarding any concerns or problems they are having in implementing Title I requirements.

In responding to our survey, over 80 percent of state Title I directors identified the availability of state level expertise as a factor that facilitated their efforts to meet Title I requirements. State officials we interviewed reported that training for teachers and district personnel was often needed to apply new content standards in the classroom and to administer assessments correctly. Two states, for example, used regional centers to educate local staff on assessments and standards.

Fifty percent of survey respondents identified inadequate funding as an obstacle in moving toward compliance and noncompliant states cited this problem more often than compliant ones. In our interviews, Title I and assessment officials from noncompliant states reported that progress toward compliance with Title I requirements was stalled because of investments they had made in assessment systems that predated and conflicted with the requirements of the 1994 Title I reauthorization. Respondents said that they had made substantial investments of time and money in systems of assessment that often relied upon norm-referenced assessments and did not meet the 1994 requirement for criterion-based tests. They noted that it took their states several years to change from the old system of assessment to one meeting the requirements specified by the 1994 reauthorizing legislation. According to the officials we interviewed, building support to start again on another system and obtaining the funding made it more difficult to make the necessary changes in a timely manner. In addition, one survey respondent from a very small state noted that due to the state’s size it has a small number of staff and does not have the technical expertise needed to develop a new system, thus hampering the state’s ability to meet the requirements.

Most States are Taking Some Action To Ensure Accurate Scoring, Justification for Exemptions, and Appropriate Accommodations, but Actions are Limited

Most states are taking some action to ensure that Title I assessments are scored accurately, that any exemptions for students with limited English proficiency are justified, and that students are receiving appropriate accommodations when these are needed to gather an accurate assessment of their abilities. Most states hire a contractor to score Title I assessments and about two-thirds of these states monitored the scoring performed by the contractor. Some states that hire contractors have found errors in the scoring the contractors did, and in some cases, these errors have had serious negative consequences for schools and students. Most states reported taking some actions to ensure that students with limited English proficiency and disabilities received appropriate accommodations during testing. Education is redesigning its current compliance and monitoring program to better monitor states’ implementation of Title I.

Problems Found in Contractor Scoring

According to our survey results, most states (44) hire a contractor for test scoring, but 16 of these states identified no monitoring mechanism to ensure the accuracy of their contractor’s scoring and reporting. Among those states that did report one or more monitoring mechanisms, 15 reported that they monitored the contractor’s scoring by comparing a sample of original student test results to the contractor’s results. A few states also reported, in interviews with us, that they compared their most recent test scores with those from previous years and looked for significant variations that suggested potential errors in scoring. However, in our interviews, some assessment officials indicated that they use this type of monitoring rather informally.

The problems identified in assessment scoring suggest that these approaches do not always provide adequate assurance of complete and accurate results. Indeed, several of the states that use contractors to score tests reported that they have had problems with errors in scoring whether or not they had monitoring measures in place. In some cases, contractors marked correct answers as incorrect and in other cases the contractors calculated the scores incorrectly. The errors were discovered by a number of individuals, including local district officials, parents, and state agency staff. These scoring errors had impacts on students, families, and school and district resources. Based on erroneous scores calculated by a contractor, one state sent thousands of children to summer school in the mistaken belief that their performance was poor enough to meet the criterion for summer intervention. In addition to disrupting families’ summer plans and potentially preventing student promotions, this may have drawn resources away from other necessary activities. In another case, based on a contractor’s erroneous scoring, a state incorrectly identified several schools as “in need of improvement,” a designation that carries with it both bad publicity and extra expense, for example, districts may have to fund the needed improvements.

A few state officials that we interviewed told us that they have begun instituting processes to check the accuracy of scoring. For example, three states said that they had hired individuals who were experts in test scoring or they hired other third parties to conduct independent audits. States that were in compliance with 1994 Title I assessment requirements generally had more complete monitoring systems, including measures such as technical advisory committees to review results, conduct site visits, and follow a sample of tests through the scoring and reporting process. In contrast, several states indicated they are still relying on contractor self- monitoring to ensure accurate scoring.

Although Education is obligated under the Federal Managers’ Financial Integrity Act of 1982 and the Single Audit Act to ensure that states that receive federal funds comply with statutory and regulatory requirements to monitor contractors, it currently takes limited action regarding states’ monitoring of assessment contractors. Education’s inspector general has reported deficiencies in an important vehicle for such oversight - - Education’s compliance reviews of state programs. The compliance reviews are conducted on a 4-year cycle and include an on-site visit that lasts 1 week. Specifically, the OIG cited insufficient time to conduct the reviews, lack of knowledge among Education staff about areas they were reviewing, and a lack of consistency in how the reviews were conducted. Senior Education officials told us the department is redesigning the current compliance and monitoring program used for its on-site visits to better focus on outcomes and accountability in Title I and that it is addressing the OIG’s recommendations. However, a senior Education official who is working on the redesign of the compliance reviews told us that the current draft plans did not include specific checks on state monitoring of assessment scoring. Confidence in the accuracy of test scoring is critical to acceptance of the test results’ use in assessing school performance.

Some Actions Taken To Ensure Exemptions Justified and Accommodations Appropriate

According to our surveys and interviews, 33 states have taken at least minimal actions to ensure any exemptions for students with limited English proficiency are justified and 41 states take actions to ensure accommodations for students with disabilities are appropriate. Most states reported that they had developed standards for districts to follow in accommodating these students so that assessments could yield accurate measures of their performance. However, states reported few actions that would ensure that these guidelines were being followed. For example, 17 states reported that they compare the number of students with limited English proficiency tested within a given year against the number for the previous year. They used this comparison as their means of verifying that the numbers of students receiving exemptions were reasonable. As the pool of students in a particular school can change substantially from year to year, this comparison has obvious limitations. Moreover, students’ status, for example with respect to English proficiency, can change from year to year. Similarly, 37 states reported using an annual comparison of the number of students with disabilities being tested as a check for appropriate accommodations. However, it is not evident how such comparisons would allow states to ascertain the appropriateness of the accommodations.

Survey results and interviews did indicate that more states are taking actions to monitor accommodations for students with disabilities than for students with limited English proficiency. For example, while 25 states reported that they had standards for accommodating students with limited English proficiency, 36 had standards for accommodating students with disabilities. The state officials that we interviewed told us that this was because districts built upon steps they had taken under the Individuals with Disabilities Education Act (IDEA) to document the accommodations needed by students with disabilities. In general, states said that the districts have more experience and technical expertise for assessing and supporting students with disabilities because of working under IDEA for many years. In contrast, some states lacked consistent standards for identifying students with limited English proficiency and more states were still working to develop alternate assessments or accommodations for these students. Augmenting the 1994 requirements, the new 2001 legislation requires that states annually assess the language proficiency of students with limited English proficiency by the 2002-03 school year. States do conduct cyclical monitoring of the implementation of all their programs that might be used to assess the appropriateness of district policy and practice with regard to testing accommodations. However, in a recent review,we found that states varied dramatically in the frequency of their on-site visits. The average time between visits to districts ranged from 2 years or less (6 states) to more than 7 years (17 states).

Conclusions

This snapshot of the states’ status with respect to the 1994 Title I requirements suggests that many states may not be well-positioned to meet the requirements added in 2001. Only 17 states were in compliance with the assessment requirements of the 1994 law in March of 2002; therefore, the majority of states will still be working on meeting the 1994 requirements as they begin work toward meeting the new requirements.

In addition, despite the enhanced emphasis on assessment results, states still appear to be struggling with ensuring that assessment data are complete and correct. The 1994 and 2001 ESEA reauthorizations raised student assessments to a new level of importance. The assessments are intended to help ensure that all students, including those who have disabilities and those who have limited English proficiency are meeting challenging standards. In addition, assessment results are a key part of the mechanism for holding both schools and states accountable for improving educational performance. Thus, ensuring the completeness and accuracy of assessment data is central to measuring students’ progress and ensuring accountability. Without adequate oversight of assessment scoring, efforts to identify and improve low-performing schools could be hindered by lack of confidence in assessment results or uncertainty regarding whether particular schools have been appropriately identified for improvement. Education’s current monitoring does not include specific oversight of how states ensure the quality of scoring contractors’ work, but Education’s revision of its monitoring process provides the agency with an opportunity to help states ensure that scoring done by contractors is accurate.

Recommendation

To enhance confidence in state assessment results, we recommend that when the Department of Education monitors state compliance with federal programs, it include checks for contractor monitoring related to Title I, Part A. Specifically, Education should include in its new compliance reviews a check on the controls states have in place to ensure proper test scoring and the effective implementation of these controls by states.

Agency Comments

We provided Education with a draft of this report for review. The Department’s official comments are printed in appendix II. In its comments, Education agreed with our recommendation. Education also provided us with technical comments that we incorporated in the report as appropriate.

We are sending copies of this report to appropriate congressional committees and other interested parties. If you have any questions about this report, please contact me on (202) 512-7215 or Betty Ward-Zukerman on (202) 512-2732. Key contributors to this report were Mary Roy, G. Paul Chapman II, Laura Pan Luo, Corinna Nicolaou, and Patrick DiBattista.

Appendix I: Scope and Methodology

We conducted this review in conjunction with our partners in the Domestic Working Group. The Domestic Working Group’s objective is to allow officials in the federal, state, and local governmental audit communities to interact on a personal and informal basis on various topics of mutual concern. The group consists of 18 (6 federal, 6 state, and 6 local) top officials and is intended to complement the work of the intergovernmental audit forums and other professional associations.

For this review, the Texas State Auditor's Office conducted a detailed assessment of data quality at the state and local levels in Texas, while the Department of Education's Office of Inspector General did so at the state and local levels in California and conducted additional work on control processes at the Department of Education. In Pennsylvania, the Pennsylvania Department of Auditor General conducted work at the state level and the Philadelphia Controller's Office pursued the same goal within the city of Philadelphia.

To complement these efforts, GAO surveyed all states and conducted detailed interviews with several regarding their experiences in implementing major provisions of Title I. Specifically, we reviewed three key questions: (1) the status of states’ compliance with key 1994 Title I assessment requirements; (2) factors that have hindered or helped states move toward meeting the requirements; and (3) the actions states are taking to ensure that Title I assessments are scored accurately, exemptions for students with limited English proficiency are justified, and students with disabilities are accommodated during testing according to federal regulations.

We obtained information on the first objective from the Department of Education. We met with Education officials and obtained updated listings of compliance throughout the audit. In addition, we reviewed state decision letters, peer reviews of state assessment systems, and reports completed or commissioned by Education’s Planning and Evaluation Service.

To address the second and third questions, we used both a state survey of Title I directors and detailed interviews with state Title I officials and other state officials who played a key role in Title I compliance - often assessment officials and sometimes Special Education, program evaluation, and information technology officials. We sent the survey to all 50 state directors and to the District of Columbia and Puerto Rico. We received 50 completed surveys. We followed up with 19 states to clarify and expand on questions in the interview related to contracting for the scoring of tests. We interviewed officials from 5 states that had assessment systems approved by Education and 3 states that were still trying to attain compliance. We also interviewed two expert reviewers, Education officials with responsibility for Title I and program review, three officials at Education’s regional assistance centers, and officials at the Council of Chief of State School Officers. We coordinated our work and findings with our audit partners, who provided us with information relative to their states’ activities.