Reading First: States Report Improvements in Reading Instruction, but Additional Procedures Would Clarify Education's Role in Ensuring Proper Implementation by States
Released: 2007-03-23T00:00:00

Letter

Background

Reading First, which was enacted as part of NCLBA, aims to assist states and local school districts in establishing reading programs for students in kindergarten through third grade by providing funding through 6-year formula grants. The goal of the program is to ensure that every student can read at grade level or above by the end of third grade. To that end, Reading First provides funds and technical assistance to states and school districts to implement programs supported by scientifically-based reading research (SBRR), increase teacher professional development based on this research, and select and administer reading assessments to screen, diagnose, and monitor the progress of all students. NCLBA defines SBRR as research that (1) uses systematic, empirical methods that draw on observation or experiment; (2) involves rigorous data analyses that test stated hypotheses and justify general conclusions; (3) relies on measurements or observational methods that are valid; and (4) has been accepted by a peer-reviewed journal or approved by a panel of independent experts. Further, NCLBA requires states to adopt reading programs that contain the five essential components of reading-- (1) phonemic awareness; (2) phonics; (3) vocabulary development; (4) reading fluency, including oral reading skills; and (5) reading comprehension strategies.

While Education has responsibility for overseeing the Reading First program and states’ implementation and compliance with statutory and program requirements, NCLBA places restrictions on what Education officials can require states to do. Specifically, Education is not authorized to mandate, direct, control, or endorse any curriculum designed to be used in elementary or secondary schools. Further, when Education was formed in 1979, Congress was concerned about protecting state and local responsibility for education and, therefore, placed limits in Education’s authorizing statute on the ability of Education officials to exercise any direction, supervision, or control over the curriculum or program of instruction, the selection of textbooks or personnel, of any school or school system.

Every state could apply for Reading First funds, and states were required to submit a state plan for approval that demonstrates how they will ensure that statutory requirements will be met by districts. Education, working in consultation with the National Institute for Literacy (NIFL), as required in NCLBA, established an expert review panel composed of a variety of reading experts to evaluate state plans and recommend which plans should be approved. In these plans, states were required to describe how they would assist districts in selecting reading curricula supported by SBRR, valid and reliable reading assessments, and professional development programs for K-3rd grade teachers based on SBRR. The law does not call for Education to approve or disapprove particular reading programs or curricula identified in state plans. When appropriate, the peer review panel was to also recommend clarifications or identify changes it deemed necessary to improve the likelihood of a state plan’s success. NCLBA requires that Education approve each state’s application only if it meets the requirements set forth in the law.

Reading First allows states to reserve up to 20 percent of their funds for professional development; technical assistance; and planning, administrative, and reporting activities. For example, states can use their funds to develop and implement a professional development program to prepare K-3rd teachers in all essential components of reading instruction. One model for supporting teachers’ reading instruction involves hiring a Reading Coach who works with teachers to implement reading activities aligned with SBRR. Almost all states require Reading First schools to have a Reading Coach tasked with supporting teachers and principals with instruction, administering assessments, and interpreting assessment data.

States that receive Reading First grants are required to conduct a competitive sub-grant process for eligible school districts and must distribute at least 80 percent of the federal Reading First grants they receive to districts. NCLBA and Education guidance provides states with flexibility to set eligibility criteria for school districts so that eligible districts are among those in the state that have the highest number or percentage of K-3rd grade students reading below grade level and (1) have jurisdiction over an empowerment zone or enterprise community, (2) have a significant number or percentage of schools identified as in need of improvement, or (3) are among the districts in the state that have the highest number or percentages of children counted as poor and school- aged for the purposes of Title I. NCLBA establishes priorities that states must consider when awarding a Reading First sub-grant, while also allowing states to establish other priority areas. For instance, NCLBA requires that the state sub-grant process give priority to districts with at least 15 percent of students or 6,500 children from families with incomes below the poverty line, but states also have some flexibility to establish additional priorities, such as a demonstrated commitment to improving reading achievement. The sub-grant process along with the criteria at each stage is summarized in figure 1.

Districts are required to use their sub-grant funds to carry out certain activities identified in NCLBA. For example, districts must use these funds to select and implement reading programs based on SBRR that include the essential components of reading instruction, to select and implement diagnostic reading assessment tools, and to provide professional development opportunities for teachers. Additionally, districts are permitted to use Reading First funds in support of other activities, such as training parents and tutors in the essential components of reading instruction.

States are required to report to Education annually on the implementation of Reading First, including their progress in reducing the number of students who are reading below grade level. Additionally, states are required to submit a mid-point progress report to Education at the end of the third year of the grant period. These mid-point progress reports are subject to review by the same expert peer review panel that evaluated state applications. If Education determines, after submission and panel review of a state’s mid-point progress report and on the basis of ongoing Education monitoring, that a state is not making significant progress, Education has the discretion to withhold further Reading First grant payments from that state. While these state reports to Education are intended to provide information on the effectiveness of Reading First, Education is also required to contract with an independent organization outside Education for a rigorous and scientifically-valid, 5-year, national evaluation of the program, with a final report scheduled to be issued in 2007.

The Reading First program has relied on several key contractors to perform a number of program functions. For example, Education officials hired RMC Research Corporation, a company that provides research, evaluation, and related services to educational and human services clients, to provide technical assistance to states and districts that have received Reading First funding. According to Education officials, RMC contractors were tasked initially with providing specific, individualized guidance on the application process to state officials who requested it. RMC later became the national coordinator for the contract overseeing the National Center for Reading First Technical Assistance and its three regional subsidiaries: the Eastern Regional Reading First Technical Assistance Center (ERRFTAC) in Tallahassee, Florida; the Central Regional Reading First Technical Assistance Center (CRRFTAC) in Austin, Texas; and the Western Regional Reading First Technical Assistance Center (WRRFTAC) in Eugene, Oregon. In this role, RMC staff provides support to the TACs and their employees, as well as weekly coordination among the TACs, and regular training seminars. Operated out of universities recognized by Education officials for their expertise in SBRR and related areas, the centers began operations in 2003 and are responsible for providing an array of technical assistance activities to states, including national and regional conferences, training and professional development, products and materials, and liaisons to national reading experts. Education officials also contracted with Learning Point Associates to provide technical assistance to states as they launched their sub-grant competitions.

Once Reading First sub-grants had been awarded to local districts, Education contracted with the American Institutes for Research (AIR), a behavioral and social science research organization, to conduct annual monitoring visits to each state. These visits incorporate sessions with state officials, as well as visits to a few districts in each state and are designed to assess states’ and districts’ compliance with their approved plans. After each monitoring visit, AIR representatives submit a report, including any findings of non-compliance, to Reading First officials. Reading First officials are to forward these reports to the cognizant state officials.

States Reported Changes as Well as Improvements in Reading Instruction Since the Inception of Reading First

States reported that there have been a number of changes and improvements in reading instruction since the implementation of Reading First. There has been an increased emphasis on the five key components of reading, assessments, and professional development with more classroom time being devoted to reading activities. However, according to publishers we interviewed, there have been limited changes to instructional material. Similarly, many states that approved reading programs for districts to choose from report few changes to their lists of approved programs.

States Reported That Reading Instruction Changed or Improved Following the Implementation of Changes under Reading First

In responding to our survey, 69 percent of all states reported great or very great improvement in reading instruction since inception of Reading First. One area in which states reported a change that may have contributed to improvement of reading was the degree to which classroom instruction explicitly incorporated the five key components. In our survey, at least 39 states reported that Reading First schools had incorporated each of the five required components of reading into curriculum to a great or very great degree as a result of Reading First. State and local officials we talked to during some of our site visits reinforced this opinion and in particular noted that Reading First teachers had awareness of and were more focused on the five components. In addition, the increased time devoted to reading activities under Reading First may have contributed to improvement. Several district officials we met with told us they were including a protected, uninterrupted block of time for reading instruction of 90 minutes or more per day—which the department’s Guidance for the Reading First Program lists as a key element of an effective reading program. Education’s Reading First Implementation Evaluation: Interim Report (The Interim Report) also found that Reading First teachers reported allocating over 90 minutes per day, on average, for a designated reading block.

States officials reported improvement in reading instruction resulting from the use of assessments. In responding to our survey, one state official said, “One of the strengths of the Reading First program has been its strong adherence to SBRR and to the use of valid and reliable assessments in guiding instruction and program evaluation.” A number of state and local officials we interviewed reported that the use of assessments changed after Reading First, especially in the way that teachers use data from these assessments to better inform reading instruction. Specifically, district officials we talked to during our site visits reported that teachers review students’ assessment results to determine the areas in which they need more targeted instruction. One official also reported that assessment data can sometimes be used to identify successful teachers from whom other teachers can learn teaching techniques, with one official asserting that “Reading First has and is making a great impact on teachers’ instructional practices, techniques, and strategies.” Also, according to Education’s Interim Report, researchers estimated that 83 percent of Reading First teachers cited assessment results essential to organizing instructional groups, 85 percent cited the results essential to determining progress on skills, and 75 percent cited the results essential to identifying students who need reading intervention.

According to our survey, most states also reported that the assessments they used differed greatly or very greatly from the ones they used prior to Reading First. States reported a wide variety of reading assessments on their state-approved lists, with over 40 different assessments listed. By far, the most frequently approved assessment was Dynamic Indicators of Basic Early Literacy Skills (DIBELS), approved by 45 states. Also, a few states reported to us that they were moving toward a more uniform systematic assessment system for the first time, whereas previously each school could choose which assessment it would use. Some state and district officials told us that having a more uniform and systematic assessment was beneficial, because, for instance, it allowed the officials to track and compare reading scores more easily.

Professional development is another area in which state officials noted improvement. All states reported improvement in professional development as a result of Reading First, with at least 41 states reporting that professional development for reading teachers improved greatly or very greatly in each of five key instructional areas. Further, a considerable majority of states reported great or very great increases in the frequency of professional development and the resources devoted to it, 45 and 39, respectively. One state reported, “The provision of funding to be used to support statewide professional development efforts for K-3 reading has been an important aspect of the program.” The Interim Report on the Reading First program highlights that a vast majority of Reading First teachers had received training on the five key components of reading. In our site-visits, district officials confirmed that, for the most part, teachers in their Reading First classrooms had received training. However, in responding to our survey, 19 states did report some challenges in training of 100 percent of Reading First teachers, with teacher turnover cited by 12 states as the reason some Reading First teachers might not have taken any type of Reading First training. Figure 2 summarizes reported improvements in professional development for teachers.

Professional development was provided by a variety of federal, state, and private sources. Staff from the TACs and officials from at least one state reported providing professional development to districts customized to the individual district’s needs and perceived future needs. Education’s Interim Report on Reading First implementation noted that state Reading First coordinators in 33 states reported that state staff chose and organized all statewide professional development efforts and played a key role in selecting professional development topics for districts and schools. In addition, publishers we spoke with told us they often provide training to acclimate teachers to their products. Certain publishers of major commercial reading programs and assessments told us that since the implementation of Reading First, districts demand much more training. Specifically, according to some of the publishers and TAC staff we spoke with, districts have been interested in more in-depth workshops on particular topics such as teaching techniques and using and interpreting assessments.

Finally, another aspect of professional development pertinent to Reading First is the presence of a Reading Coach. State and district officials reported that Reading Coaches receive training that better enables them to assist schools. Education’s Interim Report found that each Reading Coach worked with an average of 1.2 schools and with 21 teachers to help implement activities aligned with SBRR.

Publishers Reported Making Few Changes to Reading Programs, and States Reported Adopting a Variety of Approaches to Select Reading Programs

Three of the four major publishers of reading programs we spoke with reported that they had not made significant changes to the content of their reading programs as a result of Reading First. Two publishers stated that they made minor changes to their reading materials to make more explicit how the content of the existing programs align with the five components emphasized in Reading First. Two of them reported that they made changes to their programs based on the National Reading Panel’s findings, which was prior to the enactment of Reading First. For example, representatives of one company stated that they launched a new reading program based on the findings of the National Reading Panel that takes into account the requirements of Reading First. Despite limited changes to the actual instructional material, all the publishers noted a greater emphasis on assessing the efficacy of their reading programs as a result of Reading First. In an effort to measure the effectiveness of their programs, the publishers reported devoting more effort to research and to evaluate how effective their reading programs were at raising reading assessment scores.

States followed two main approaches in selecting reading programs for districts —22 identified a state-approved list of programs for districts to select, while the other 29 did not have a state-approved list, thereby requiring districts in those states to self-select reading programs and determine, with some state oversight and subject to state approval, whether they satisfy the requirements of SBRR. Of the 22 states with approved lists, reading program publishers most frequently represented on the lists were Houghton Mifflin, McGraw-Hill, and Harcourt (see table 1). At the school level, Education found in its Interim Report that these three reading program publishers were also the most frequently used, estimating that between 11 and 23 percent of schools used programs from one of them.

Additionally, of the 22 states that identified a list of approved core reading programs for Reading First, 8 already had a list of approved core reading programs for adoption by all schools in their state prior to Reading First. Only two of these states reported removing reading programs—a total of six—from their lists because they did not meet Reading First requirements. According to Education’s Interim Report, an estimated 39 percent of Reading First schools reported adopting a new core reading program at the beginning of the 2004-2005 school year in which they received their Reading First grant, in contrast with an estimated 16 percent of non-Reading First Title I schools.

States used a variety of sources to help them identify and select reading programs that met Reading First’s criteria. For example, 15 of the 22 states with state-approved lists reported using the Consumer’s Guide to Evaluating A Core Reading Program Grades K-3: A Critical Elements Analysis to make this decision. Other frequently used resources include criteria in the state’s application for Reading First, information obtained at Reading First Leadership Academies provided by Education, and other states’ approved lists. Based on responses to our survey, the table below summarizes approaches states used to develop their approved lists (see table 2).

Based on our survey results, 25 of the 29 states reporting that they did not have a list of approved core reading programs said they provided guidance for districts and schools to identify core reading programs. Fifteen of these states reported directing districts and schools to conduct a review of reading programs using A Consumer’s Guide to Evaluating a Core Reading Program. Other states reported providing a variety of guidance to districts to help them select reading programs supported by SBRR, including referring them to the approved lists of other states and reviews conducted by academic experts.

States Awarded Reading First Sub- Grants to School Districts Using a Variety of Different Criteria, and Some States Reported Difficulties with Implementation

States varied in how they exercised their flexibility to set additional eligibility and award criteria as allowed by the Reading First program, and some states reported difficulty with implementing key aspects of the Reading First program while other states did not. In the areas in which they were given flexibility, states used a variety of criteria for determining eligibility and in awarding sub-grants to eligible districts, such as awarding grants to districts that had previously received federal reading dollars. Education reported that over 3,400 school districts were eligible to apply for Reading First sub-grants in the states’ first school year of funding. Of these districts, nearly 2,100 applied for and nearly 1,200 received Reading First sub-grants in the states’ first school year of funding. In addition, 22 states reported that it was difficult or very difficult to help districts with reading scores that had not improved sufficiently. On the other hand, 28 states reported that it was easy or very easy to determine whether districts’ applications met criteria for awarding sub-grants.

States Varied in How they Exercised Their Flexibility to Set Additional School District Sub-grant Eligibility and Award Criteria

States varied in how they exercised their flexibility to set school district eligibility criteria for sub-grants. The Reading First program provides states with some flexibility to define eligibility criteria within the statutory guidelines. For instance, while Reading First requires that states target districts with students in kindergarten through third grade reading below grade level, states have flexibility to set eligibility criteria based on the percentage and/or number of these students within districts. While 34 states reported electing to base eligibility on a percentage of schools with students reading below grade level, 18 states reported electing to base eligibility on a number of students reading below grade level. After applying eligibility criteria, Education reported that states determined that over 3,400 school districts were eligible to apply for Reading First sub- grants for states’ first school year of funding, or about 20 percent of all school districts nationwide. However, the percentage of eligible districts varied greatly across the states, ranging from about 3 to 93 percent.

Of those districts eligible to apply, 62 percent, or nearly 2,100 districts, did so, as summarized in figure 3 below. States reported a variety of reasons why eligible school districts did not apply such as the prescriptive nature of the program, differences in educational philosophy, and inadequate resources for the application process. For example, officials from a few states reported that some districts did not have the capacity to write the grant application. An official from one state reported that some districts did not have the time and the staff to complete the sub-grant process. Furthermore, an official from another state reported that the application process was too lengthy and time-consuming to complete.

Nineteen states reported in our survey that they exercised flexibility in establishing priorities when awarding Reading First sub-grants. States set a variety of additional priorities for awarding grants to school districts. For instance, six states reported that they gave priority to districts that already had other grants, such as Early Reading First grants, or indicated that they could somehow use their Reading First funds in combination with other resources to maximize the number of students reading at grade level. In contrast, two states gave priority to districts that had not received other grant funding. In addition, two states gave priority to districts based on the population of Native Americans or students with limited English proficiency. After applying selection criteria, states awarded Reading First sub-grants to about 34 percent or nearly 1,200 school districts for states’ first school year of funding. This represented about 56 percent of the 2,100 eligible districts that applied and nearly 7 percent of all school districts nationwide for states’ first school year of funding (see fig. 3).

Some States Reported Difficulty Implementing Key Aspects of the Program

Some states reported difficulty in implementing key aspects of the Reading First program. Twenty-two states reported that it was either difficult or very difficult to help districts with reading scores that had not improved sufficiently. Officials from one state said that this was difficult because it requires close examination of students reading deficiencies and the commitment of school leadership. Officials from another state reported some difficulty in improving selected reading skills of students with limited English proficiency, which are concentrated in pockets around the state. Seventeen states reported that it was either difficult or very difficult to assess how districts applied SBRR in choosing their reading program. Finally, seven states reported difficulty implementing four or more of six key program aspects listed in our survey and shown in figure 4. Officials from one of these states told us that the difficulty with implementation was due to the newness of the program for which everything had to be developed from scratch.

On the other hand, states reported ease implementing other key aspects. In particular, 28 states reported that it was easy or very easy to determine whether districts’ applications met criteria for awarding sub-grants. For example, states are required to determine whether districts will adhere to the key components of the program, such as developing a professional development program or using reading assessments to gauge performance. Several states we interviewed suggested that it was easy to make this determination because some of the Reading First requirements were already in place in their states before Reading First was implemented. For example, some state officials we interviewed mentioned using reading assessments prior to Reading First. In addition, officials in one state told us that they already had a professional development program in place to train teachers on the state’s reading program. Twenty-four states reported that it was easy or very easy to identify reading programs based on SBRR.

Education Provided a Wide Range of Guidance, Assistance, and Oversight, Generally Satisfying State Officials, but Education Lacked Controls to Guard against Mandating or Endorsing Curricula and Did Not Provide Written Monitoring Procedures

Education officials provided states a wide variety of guidance, assistance, and oversight, but Education lacked written procedures to guide its interactions with the states and provided limited information on its monitoring procedures. Education’s guidance and assistance included written guidance, preparatory workshops, feedback during the application process, and feedback from monitoring visits. Additionally, guidance and assistance were provided by Education’s contractors, including the regional technical assistance centers. For the most part, state officials characterized the guidance and assistance they received from Education officials and contractors, especially the regional technical assistance centers, as being helpful or very helpful, and many also reported relying on the expertise of Reading First officials in other states. However, Education lacked controls to ensure that its officials did not endorse or otherwise mandate or direct states to adopt particular reading curricula. For example, according to state officials, Education officials and contractors made suggestions to some states to adopt or eliminate certain reading programs, assessments, or professional development providers. In addition, some state officials reported a lack of clarity about key aspects of the annual monitoring process, including time frames and expectations of states in responding to monitoring findings.

Education Provided a Wide Range of Guidance and Assistance

Education provided a variety of written and informal guidance and assistance to states to help them prepare their applications. For example, three months after the enactment of NCLBA in January 2002, Education issued two key pieces of written guidance to states pertaining to the Reading First program and grant application process: the Guidance for the Reading First Program and Criteria for Review of State Applications. Education officials also sponsored three Reading Leadership Academies in the early part of 2002. The Academies were forums for state education officials to obtain information and build their capacity to implement key aspects of the Reading First program, including professional development and the application of SBRR. Education contracted with RMC Research Corporation to provide technical assistance to states related to the grant application process. States reported seeking guidance from RMC on various aspects of the Reading First application, in particular the use of instructional assessments (17 states) and instructional strategies and programs (14 states). Throughout the application process, both Education and RMC officials were available to address states’ questions. In particular, Education officials provided feedback to states on the results of expert review panel evaluations of their applications. Consequently, a large number of states reported that Education required them to address issues in their applications, most commonly related to the use of instructional assessments (33 states) and instructional strategies and programs (25 states). See figure 5 for issues raised about state applications. Forty-eight states reported that they needed to modify their application at least once, and 27 reported modifying them three or more times.

Once grants were awarded, Education continued to provide assistance and contracted with RMC Research to oversee three regional TACs to help states implement Reading First. RMC established three TACs affiliated with state university centers in Florida, Texas, and Oregon, which RMC and TAC officials told us were selected based on their expertise in one or more areas central to the success of the Reading First program, such as professional development or reading assessment. Each technical assistance center was responsible for providing comprehensive support to each of the states in its geographic region (see fig. 6). States reported that they looked to these centers for guidance on a variety of issues, especially creating professional development criteria, using reading assessments, and helping districts with reading scores that had not improved sufficiently.

According to TAC staff, some of the most common requests they receive pertain to the use and interpretation of assessment data and use of Reading Coaches. TAC staff also told us that they catalog recurring issues or problems.

In addition, according to one RMC official and some state officials, the TACs provided support to states during implementation to help them supplement their capacity and expertise in evaluating whether or not reading programs proposed by districts were based on SBRR. For instance, staff from the TAC in Florida explained that some states in their region had asked for assistance in evaluating reading programs that had been in use prior to Reading First to gauge their compliance with the requirements of Reading First. Staff from the TAC emphasized that in reviewing these reading programs, they used the criteria in each state’s approved state plan as the criteria for determining compliance with Reading First requirements. Officials in one state explained that while the staff at their state educational agency (SEA) possessed the knowledge necessary to conduct reviews of reading programs, scarce state staff resources would have made it difficult to conclude the reviews in the short time frame available. Though Education officials were aware of and initially condoned the TAC review process, Education officials advised all TACs to discontinue reviews of programs—to avoid the appearance of impropriety—after allegations were raised about Reading First officials expressing preference for specific reading programs. (Table 3 provides a summary of the types of guidance and assistance provided by Education and its contractors.)

During the application and implementation phases of the Reading First program, many states came to rely on other unofficial sources of guidance, including other states’ Reading First officials, in addition to the written guidance provided by Education. For example, as noted earlier, among the 22 states that had an approved list of reading programs for Reading First districts, 15 reported using A Consumer’s Guide to Evaluating a Core Reading Program to assist them in reviewing potential reading programs. In addition, officials from 21 states reported that other states’ Reading First Coordinators provided great or very great help during the Reading First state grant application process. Further, a number of state officials reported using the information from other states’ websites, such as approved reading programs, to help inform their own decisions pertaining to the selection of reading programs. One state official explained, “With our limited infrastructure and dollars, we were never able to muster the resources needed to run an in-house programs review,” and further that, “It worked well for us to use the programs and materials review results from larger states that ran rigorous review processes.” Another state official reported that the state did not feel equipped to apply the principles of SBRR in evaluating reading programs and responded by comparing one state’s review and subsequent list of reading programs to those of a few other states to make judgments about allowable programs.

States Were Generally Satisfied with the Guidance and Assistance Available to Them

Most states reported making use of and being satisfied with the primary sources of guidance available to them over the course of the Reading First application and implementation processes. For example, 46 states reported making use of the two key pieces of Education’s written guidance in preparing their Reading First applications. A majority of states also reported that these pieces of guidance provided them with the information needed to adequately address each of the key application components. For example, over 40 states reported that the guidance related to the definition of sub-grant eligibility and selection criteria for awarding sub-grants helped them adequately address these areas in their application. However, officials in eight states reported that the guidance on the use of instructional assessments did not provide them with the information needed to adequately address this area. (See fig. 7.)

Overall, most state officials were also satisfied with the level of assistance they received from Education staff and their contractors in addressing issues related to the Reading First application and implementation processes. For example, state officials in 39 states reported that Education staff were of great or very great help during the application or implementation process. Additionally, officials from 48 states reported that Education officials were helpful or very helpful in addressing states’ implementation-related questions, which frequently dealt with using reading assessments and helping districts with reading scores that had not improved sufficiently. A number of state officials reported to us that they appreciated the guidance and attention they received from Reading First officials at Education. For example, one state Reading First Coordinator reported, “the U.S. Department of Education personnel have been wonderful through the process of implementing Reading First. I can’t say enough about how accessible and supportive their assistance has been.” Another state official remarked that the state’s efforts to make reading improvements “would have been impossible without their [Education officials and contractors] guidance and support.” Even officials from one state who had a disagreement with Education over its suggestion to eliminate a certain reading program characterized most of the guidance they received from Reading First officials as “excellent.” However, one state official reported feeling that the technical assistance workshops have served as conduits for Education officials to send messages about the specific reading programs and assessments they prefer. Another state official reported that, “core programs and significant progress have not been defined” and that “SBRR programs are not clearly designated.”

According to responses obtained to our survey, the three TACs also provided a resource for states seeking advice on issues pertaining to the implementation of their Reading First programs. Specifically, 41 states cited the Centers as helpful or very helpful in addressing states’ inquiries related to the implementation of Reading First. In addition, on a variety of key implementation components, more state officials reported seeking information from their regional TACs than they did from Education officials (see table 4).

Education Lacked Written Procedures to Guide Its Interactions with States

We found that Education developed no written guidance, policies, or procedures to direct or train Education officials or contractors regarding their interactions with the states. Federal agencies are required under the Federal Managers’ Financial Integrity Act of 1982 to establish and maintain internal controls to provide a reasonable assurance that agencies achieve objectives of effective and efficient operations, reliable financial reporting, and compliance with applicable laws and regulations. When executed effectively, internal controls work to ensure compliance with applicable laws and regulations by putting in place an effective set of policies, procedures, and related training. We found that Education had not developed written guidance or training to guide managers on how to implement and comply with statutory provisions prohibiting Education officials from directing or endorsing state and local curricular decisions. Department officials told us that it was their practice that program managers should consult the Office of General Counsel if they had questions regarding interactions with grantees. Reading First officials told us that it was their approach to approve each state’s method and rationale for reviewing or selecting reading programs as outlined in each state’s plan and that state compliance with program requirements, including adherence to the principles of SBRR, would then be assessed using the provisions of these plans as the criteria. Similarly, officials from Education’s contractors responsible for conducting monitoring visits told us that they were instructed by Education to use state plans as the criteria for gauging states’ compliance with Reading First reading program requirements, but that they were provided no formal written guidance or training. A senior Education attorney who is currently working with Reading First program officials told us that he was not aware that they had used this approach and that he felt that the statutory requirements should also play an important role in the monitoring process. Following the publication of the IG’s report in September, Education’s Office of General Counsel has provided training to senior management on internal control requirements and has begun working with the Reading First office to develop procedures to guide the department’s activities.

Despite the statutory prohibition against mandating or endorsing curricula and the department’s stated approach to rely on state plans, and the processes articulated in them, to assess compliance, states reported to us several instances in which Reading First officials or contractors appeared to intervene to influence their selection of reading programs and assessments. For example, officials from four states reported receiving suggestions from Education or its contractors to adopt specific reading programs or assessments. Specifically, two states reported that it was suggested that they adopt a particular reading assessment. Similarly, Education’s first IG report also documented one instance in which Reading First officials at Education worked in concert with state consultants to ensure that a particular reading program was included on that state’s list of approved reading programs.

In addition, states reported that Education officials or contractors suggested that they eliminate specific reading programs or assessments related to Reading First. Specifically, according to our survey results, officials from 10 states reported receiving suggestions that they eliminate specific programs or assessments. In some cases, the same program was cited by officials from more than one state. In one instance, state officials reported that Education officials alerted them that expert reviewers objected to a reading program that was under consideration but not named explicitly in the state’s application. An official from a different state reported receiving suggestions from Education officials to eliminate a certain reading program, adding that Education’s justification was that it was not aligned with SBRR. In another instance, state officials pointed out that they had adopted a program that was approved by other states, according to the procedures in their approved state plan, but were told by Education officials that it should be removed from their list and that Education would subsequently take a similar course of action with regard to those other states as well. Also, Education officials did not always rely on the criteria found in state plans as the basis for assessing compliance. We found, for example, one summary letter of findings from a monitoring report in which Education officials wrote that “Two of the monitored districts were implementing reading programs that did not appear to be aligned with scientifically based reading research.” Officials we spoke to in that state told us that they did not feel that they had been assessed on the basis of the procedures outlined in the state’s plan, but rather that the reading program itself was being called into question. The IG also found that Reading First officials communicated to several states against the use of certain reading programs or assessments, including Rigby and Reading Recovery.

Officials from a few states also reported being contacted by Education regarding district Reading First applications or reading programs. For example, officials from four states reported being contacted by an Education official about a district application under consideration and one of those states also reported being approached by staff from one of the regional technical assistance centers or another contractor for the same reason. Officials from each of these states indicated that the reason they were contacted stemmed from the reading programs being used by the districts in question. In a few cases, state officials reported being contacted by Education officials regarding the state’s acceptance of a reading program or assessment that was not in compliance with Reading First. In one instance, state officials reported that Education contacted them outside of the normal monitoring process after they had obtained information from a national Reading First database maintained by a non- profit research organization that districts in the state were using a specific reading program.

Five states also reported receiving recommendations from Reading First officials or contractors to change some of the professional development providers proposed in their original grant applications. When asked about the specific providers identified for elimination, three of the states indicated that the providers identified for elimination were in-state experts. In one case, a state was told that the review panel cited a lack of detail about the qualifications of the state’s proposed professional development consultants.

We also found that while Education officials laid out an ambitious plan to annually monitor every state, they failed to develop written procedures guiding its monitoring visits. For example, Education did not establish timelines for submitting final reports to states following monitoring visits, specifically how and when state officials were expected to follow up with Education officials regarding findings from the monitoring visits. As a result, states did not always understand monitoring response procedures, timelines, and expectations. While we found that most state officials we spoke with understood that they were to be monitored with the use of their state plans as the criteria, they did not always understand what was required of them when responding to monitoring findings. For example, one state official reported being unaware that the state was supposed to respond to Education officials about findings from its monitoring report. An official from another state maintained that he/she was unclear about the process the state was to follow to respond to findings, and that no timeline for responding was provided to him/her. Furthermore, one state reported that findings were not delivered in a timely manner, and another state reported that Education did not address the state’s responses to the monitoring findings. Key aspects of an effective monitoring program include communicating to individuals responsible for the function any deficiencies found during the monitoring.

Conclusions

The Reading First program, according to state coordinators, has brought about changes and improvements to the way teachers, administrators, and other education professionals approach reading instruction for children in at-risk, low-performing schools during the critical years between kindergarten and third grade. To assist states in implementing this large, new federal reading initiative, Education has provided a wide range of guidance, assistance, and oversight, that, for the most part, states have found helpful. However, Education failed to develop comprehensive written guidance and procedures to ensure that its interactions with states complied with statutory provisions. Specifically, Education lacked an adequate set of controls to ensure that Reading First’s requirements were followed, while at the same time ensuring that it did not intervene into state and local curricular decisions. We concur with the Education IG’s recommendations that the Department develop a set of internal procedures to ensure that federal statutes and regulations are followed and we feel it is important for the Secretary to follow up on these recommendations to ensure that they are properly implemented. Additionally, we feel it is important for the department to have clear procedures in place to guide departmental officials in their dealings with state and local officials. While Education’s stated approach was to rely on state plans as its criteria for enforcing Reading First’s requirements, states reported several instances in which it appears that Education officials did attempt to direct or endorse state and local curricular decisions. Such actions would prevent states from exercising their full authority under the law and would violate current statutory restrictions. Balancing Reading First’s requirements and the limits placed on the department requires Education to have clear, explicit, and well-documented procedures to guide its interactions with the states. Failure to do so places the department at risk of violating the law and leaves it vulnerable to allegations of favoritism.

Additionally, while Education’s annual monitoring effort for Reading First is ambitious, it did not provide clear guidelines and procedures to states. As a result, states were not always aware of their roles and responsibilities in responding to findings of non-compliance, and Education was not always consistent in its procedures to follow up with states to resolve findings and let states know if they had taken proper actions. Key aspects of an effective monitoring program include transparency and consistency. Letting all states know in a timely manner whether or not their plans to address deficiencies are adequate is important to ensure that findings are dealt with in an appropriate, timely and clear manner.

Recommendations

In addition to addressing the IG’s recommendations to develop internal (1) policies and procedures to guide program managers on when to solicit advice from General Counsel and (2) guidance on the prohibitions imposed by section 103(b) of the DEOA, we recommend that, in order to ensure that the department complies with statutory prohibitions against directing, mandating, or endorsing state and local curricular decisions, the Secretary of Education also establish control procedures to guide departmental officials and contractors in their interactions with states, districts, and schools.

In addition, to help the department conduct effective monitoring of the Reading First program, we recommend that the Secretary of Education establish and disseminate clear procedures governing the Reading First monitoring process. In particular, Education should delineate states’ rights and responsibilities and establish timelines and procedures for addressing findings.

Agency Comments

We provided a draft of this report to the Department of Education and received written comments from the agency. In its comments, included as appendix III of this report, Education agreed with our recommendations and indicated that it will take actions to address them.

Specifically, Education said it will provide written guidance to all departmental staff to remind them of the importance of impartiality in carrying out their duties and not construing program statutes to authorize the department to mandate, direct, or control curriculum and instruction, except to the extent authorized by law. On February 7, 2007, the Secretary of Education issued a memorandum to senior officers reminding them that it is important to maintain objectivity, fairness, and professionalism when carrying out their duties. The Secretary’s memorandum also emphasizes the importance of adhering to the statutory prohibitions against mandating, directing, and controlling curriculum and instruction, and strongly encourages managers to consult with Education’s Office of General Counsel early on to identify and resolve potential legal issues.

Also, according to Education’s written comments on our draft report and the Secretary’s February 7, 2007, memorandum to senior officers, annual training will be required on internal controls and this training will address statutory prohibitions against mandating, directing or controlling local curriculum and instruction decisions.

Regarding its monitoring process for Reading First, in its comments, Education said that it will develop and disseminate guidelines to states outlining the goals and purposes of its monitoring efforts, revise the monitoring protocols, and develop timelines and procedures on states’ rights and responsibilities for addressing monitoring findings. Education also included in its response a summary of its actions and planned actions to address recommendations from the department’s Office of Inspector General’s recent report on the implementation of the Reading First program.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies of this report to appropriate congressional committees, the Secretary of Education, and other interested parties. Copies will also be made available upon request. In addition, the report will be available at no charge on GAO’s Web site at http://www.gao.gov. If you or your staff have any questions about the report, please contact me at (202) 512-7215. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

Our objective was to answer the following questions: (1) What changes have occurred to reading instruction since the inception of Reading First? (2) What criteria have states used to award Reading First sub-grants to districts, and what, if any, difficulty did states face in implementing the program? (3) What guidance, assistance, and oversight did Education provide states related to the Reading First program? To answer these questions, we collected both qualitative and quantitative information about the Reading First program from a variety of sources. We conducted a Web- based survey of the Reading First Directors in all 50 states and the District of Columbia. We also obtained and analyzed data from the Department of Education for each state on Reading First districts’ eligibility, applications and awards for states’ first school year of funding. The first school year of funding varied across states. Twenty-five states received their first year of funding in the 2002-2003 school year. Twenty-five states received their first year of funding in the 2003-2004 school year. To assess the reliability of this data, we talked to agency officials about data quality control procedures and reviewed relevant documentation. We excluded two states because of reporting inconsistencies, but determined that the data for the other states were sufficiently reliable for the purposes of this report. We also conducted semi-structured follow-up interviews with Reading First Directors in 12 states, mostly over the telephone. We conducted site visits to 4 of the 12 states. During the site visits, we met with state officials, local program administrators, and state-level technical assistance providers, as well as school officials from individual schools, including teachers, principals, and Reading First coaches. In identifying local sub-grant recipients to meet with in each state, we sought to incorporate the perspectives of urban, rural, and suburban school districts. We selected the 12 states to have diversity in a variety of factors, including geographic distribution, grant size, poverty rate, percentage of students reading at or below grade level, urban and rural distinctions, the presence of a statewide list of approved reading programs, and whether states had reported that they received guidance from Education officials advocating for or against particular reading programs or assessments. For both the survey and follow-up interviews, to encourage candid responses, we promised to provide confidentiality. As a result, state survey responses will be provided primarily in summary form or credited to unnamed states, and the states selected for follow-up interviews will not be specifically identified. Furthermore, in order to adequately protect state identities, we are unable to provide the names of particular reading programs or assessments Education officials or contractors suggested a state use or not use. We did not attempt to verify allegations made by state or local officials in their survey responses or during interviews or otherwise make any factual findings about Education’s conduct.

We also visited or talked with administrators from each of the three regional Reading First Technical Assistance Centers, located in Florida, Texas and Oregon, as well as RMC Research, the federal contractor tasked with administering the contract with the technical assistance centers. We also interviewed several publishers and other providers of reading curricula and assessments, to obtain their views about changes Reading First has prompted in states, districts, and schools. We chose these providers to reflect the perspectives of large, commercial reading textbook programs that are widely represented nationwide on states’ lists of approved programs, as well as some other selected providers of reading curricula, including some that have filed complaints related to Reading First. We also interviewed Education officials about the implementation of the Reading First program.

To obtain a better understanding of state program structure, as well as the nature of interactions between Education officials and state grantees, we reviewed state grant files, monitoring reports, and related correspondence for the 12 states where we conducted follow-up interviews. In addition, we reviewed NCLBA language authorizing Reading First, as well as statements of work articulating the responsibilities of the regional technical assistance centers and the contractor tasked with providing assistance to states in conducting local sub-grant competitions. We conducted our work from December 2005 through January 2007 in accordance with generally accepted government auditing standards.

Survey of States

To better understand state implementation of the Reading First program, we designed and administered a Web-based survey of the Reading First Directors in all 50 states and the District of Columbia. The survey was conducted between June and July 2006 with 100 percent of state Reading First Directors responding. The survey included questions about curriculum; professional development; and state Reading First grant eligibility, application, award, and implementation processes. The survey contained both closed- and open-ended questions. For the open-ended questions, we used content analysis to classify and code the responses from the states such as the publishers on states’ approved lists. We had two people independently code the material, then reconciled any differences in coding. Because this was not a sample survey, there are no sampling errors. However, the practical difficulties of conducting any survey may introduce nonsampling errors, such as variations in how respondents interpret questions and their willingness to offer accurate responses. We took steps to minimize nonsampling errors, including pre- testing draft instruments and using a Web-based administration system. Specifically, during survey development, we pre-tested draft instruments with one expert reviewer and Reading First Directors in four states during April and May 2006. In the pre-tests, we were generally interested in the clarity of the questions and the flow and layout of the survey. For example, we wanted to ensure definitions used in the survey were clear and known to the respondents, categories provided in closed-ended questions were complete and exclusive, and the ordering of survey sections and the questions within each section was appropriate. On the basis of the pre- tests, the Web instrument underwent some slight revisions. A second step we took to minimize nonsampling errors was using a Web-based survey. By allowing respondents to enter their responses directly into an electronic instrument, this method automatically created a record for each respondent in a data file and eliminated the need for and the errors (and costs) associated with a manual data entry process. To further minimize errors, programs used to analyze the survey data were independently verified to ensure the accuracy of this work.

Appendix II: Descriptions of Assessment Types

Reading First Assessment Requirements and Descriptions

Reading programs under Reading First must include rigorous assessments with proven validity and reliability.

Assessments must measure progress in the five essential components of reading instruction and identify students who may be at risk for reading failure or who are already experiencing reading difficulty.

Reading programs under Reading First must include screening assessments, diagnostic assessments, and classroom-based instructional assessments of progress.

Appendix III: Comments from the Department of Education

Appendix IV: GAO Contacts and Staff Acknowledgments

Acknowledgments

Bryon Gordon, Assistant Director, and Tiffany Boiman, Analyst-in-Charge, managed this engagement and made significant contributions to all aspects of this report. Sonya Phillips, Sheranda Campbell, Janice Ceperich, and Andrew Huddleston also made significant contributions. Jean McSween provided methodological expertise and assistance. Sheila McCoy and Richard Burkard delivered legal counsel and analysis. Susannah Compton, Charlie Willson, and Scott Heacock assisted with message and report development.