Education Research: Education Should Improve Assessments of R&amp;D Centers, Regional Labs, and Comprehensive Centers
Released: 2002-01-24T00:00:00

Letter

Background

The R&D Centers, Regional Labs, and Comprehensive Centers share responsibility with other programs created by the Congress for education research, research-based activities, and technical assistance. Many of these programs are located in six different offices throughout Education. (See fig. 4.)

For example, the Individuals with Disabilities Education Act established a special education research and innovation program as well as technical assistance centers to improve services and results for children with disabilities. The Rehabilitation Act established a National Institute for Disability and Rehabilitation Research. The Office of Special Education and Rehabilitative Services administers these programs. The amount of funding received by these programs in fiscal year 2000 ranged from $70,000 for 11 American Overseas Research Centers to $86.5 million for the National Institute on Disability and Rehabilitation Research.

OERI is Education’s lead office for educational research and development. Its goals are to promote quality and equity in education by funding research; developing new learning materials, teaching techniques, and methods of organizing schools; demonstrating and evaluating promising educational practices; disseminating research-based information; and collecting data related to schools in the United States and other nations. Unlike other Education offices, OERI’s activities span all grade levels, from preschool through adult education, and all major content areas of instruction.

The R&D Centers, established in the 1960s to increase fundamental knowledge in education, are administered by OERI. Over the years, legislative changes have repositioned their placement in Education. Most recently, the Educational Research, Development, Dissemination and Improvement Act of 1994 reorganized OERI, implementing measures that changed the way the R&D Centers related to Education. First, the act created the National Educational Research Policy and Priorities Board to work with the Assistant Secretary of OERI to establish a long-term national agenda for research, development, and dissemination activities. Unlike previous boards, this board was charged with improving research priorities and developing standards for evaluating OERI research, including that done by the R&D Centers. Second, the act established five national research institutes within OERI, each with its own research focus, and placed the R&D Centers, as well as field-initiated studies and other research-related programs, under the appropriate institute. The Congress is expected to start the reauthorization of OERI in 2002; the last reauthorization was in 1994 and was intended to be on a 5-year cycle. Figure 5 shows the locations of the R&D Centers.

Legislation has changed the focus of the Regional Labs over time. Originally established to resemble the labs funded by the Atomic Energy Commission, such as the Los Alamos National Laboratory, the Congress created the Regional Labs in 1965 to conduct long-term activities to address national educational problems. However, funding to support these activities was never made available, limiting the scope of the Regional Labs to smaller-scale projects. Further, in the late 1970s, the Congress’s negative reactions to federally supported curriculum projects prompted Education and the Regional Labs to discontinue all large-scale nationally oriented curriculum projects. As a result, Regional Labs developed an increasingly regional agenda. In 1994, the Congress gave the governing board of each Regional Lab sole responsibility for determining if the Regional Lab fulfilled the terms of its contract with Education and its regional agenda. The Congress mandated that each governing board reflects a balanced representation of states in the region, as well as interests and concerns of regional constituencies, including teachers and education researchers. Figure 6 shows the states included in the regions of the 10 Regional Labs.

Created in 1994, the Comprehensive Centers were established more recently than the R&D Centers and the Regional Labs. The Improving America’s Schools Act of 1994 consolidated the functions of 48 categorical technical assistance centers that supported programs authorized under the ESEA, including Title I, Migrant Education, and Indian Education, into 15 Comprehensive Centers. The Congress created the Comprehensive Centers to support comprehensive, cross-program assistance as envisioned in the law, and placed them under the Office of Elementary and Secondary Education (OESE) and the Office of Bilingual Education and Minority Languages Affairs (OBEMLA). Figure 7 show the states included in the regions served by the Comprehensive Centers.

Region XV includes: American Samoa, Commonwealth of the Northern Mariana Islands, Federated States of Micronesia (Chuuk, Kosrae, Pohnpei, and Yap), Guam, Hawaii, Republic of Marshall Islands, and the Republic of Palau.

The amount of resources allocated to the R&D Centers, Regional Labs, and Comprehensive Centers differs, but the overall investment in these programs is modest. For example, to operate in fiscal year 2000, each R&D Center received from $1.5 million to $6.6 million, each Regional Lab received from $3.8 million to $8.6 million, while each Comprehensive Center received from $0.9 million to $2.7 million. The core budget for all these programs totaled about $130 million. Table 1 shows the key features of the R&D Centers, Regional Labs, and Comprehensive Centers.

Because of the importance of education research, research-based activities and technical assistance in improving schools, many studies have focused on the R&D Centers, Regional Labs, and Comprehensive Centers. Education has funded various assessments of the R&D Centers and the Regional Labs and has recently conducted an evaluation of the Comprehensive Centers. In addition, the R&D Centers and the Regional Labs have been studied and discussed by numerous independent organizations, including the National Academy of Sciences, the Brookings Institution, the RAND Corporation, and the National Educational Research Policy and Priorities Board. Several studies, including the 1992 National Academy of Sciences report and a 2000 Brookings Institute report, have concluded that the funds available to OERI to support research have been, and continue to be, insufficient to support long-term, large-scale efforts.

Activities of the Three Programs Reflect Legislative Mandates and Support Education’s Agenda, but Education Has Limited Control over Regional Labs

Laws define different missions and roles for the R&D Centers, Regional Labs, and Comprehensive Centers, and these differences are reflected in how these programs spend their money. R&D Centers focus on national research priorities, such as student assessment. Although both Regional Labs and Comprehensive Centers have a regional orientation, Regional Labs focus on meeting the needs of the regions. Comprehensive Centers focus on assisting customers in their regions implement federal education agendas, such as helping to close the achievement gaps for federally targeted groups like disadvantaged students. Education uses funding documents and program oversight to direct and prioritize the activities of the R&D Centers, Regional Labs, and Comprehensive Centers and shape the agendas of the R&D Centers and Comprehensive Centers. Unlike R&D Centers and Comprehensive Centers, the Regional Labs have governing boards. Because these governing boards determine regional agendas and oversee Regional Lab activities, Education has limited ability to shape the agendas of Regional Labs or ensure accountability for their products and services.

Laws Mandate Missions and Activities of the R&D Centers, Regional Labs, and Comprehensive Centers

The Congress created a separate primary focus for the R&D Centers, Regional Labs, and Comprehensive Centers and gave them the responsibility of performing specific activities. Because of differences in their mandates, the R&D Centers, Regional Labs, and Comprehensive Centers have different roles in supporting Education’s research agenda. The Educational Research, Development, Dissemination, and Improvement Act of 1994 places the R&D Centers under education research institutes in OERI, each of which addresses a specific content area, and requires them to carry out the purposes for which the institutes were created by conducting research and development. In contrast, rather than requiring the Regional Labs to address a particular content area, the act requires them to use research-based knowledge to address the issues in the regions they serve and assigns them an expansive array of activities to perform. Like the Regional Labs, the Comprehensive Centers have extensive mandates that require the Comprehensive Centers to focus on certain customers by giving priority to schools with schoolwide programs and the highest number of children in poverty. The law, however, allows Education to guide the general direction of the mandated activities. Table 2 provides the primary focus of the R&D Centers, Regional Labs, and Comprehensive Centers and summarizes their mandated activities.

The spending patterns of the R&D Centers, Comprehensive Centers, and Regional Labs reflect their mandates and missions. For example, the Congress authorized the R&D Centers to conduct research and development in order to increase fundamental knowledge of central issues in education. To support this mission, they reported spending 73 percent of the $35.5 million in core funding they received from OERI in fiscal year 2000 on research and an additional 14 percent on development and dissemination. The Congress created the Comprehensive Centers to help state educational agencies, school districts, and schools within an assigned region implement federal elementary and secondary school programs by providing technical assistance and training. Accordingly, the Comprehensive Centers reported spending most of their fiscal year 2000 $28.6 million core funds from OESE 83 percent on technical assistance.

Regional Labs were authorized to conduct a wide range of research-based activities, including applied research, development, dissemination, and technical assistance. Their spending reflected these purposes: the Regional Labs reported spending 25 percent of their fiscal year 2000 $65.2 million core funds from OERI on research, 17 percent on development, 16 percent on dissemination, and 21 percent on technical assistance.

Most R&D Centers, three Regional Labs, and two Comprehensive Centers used funds from other organizations, including other federal agencies, state and local educational agencies, and foundations, to expand work they were performing for Education. For example, one R&D Center, the National Center for Postsecondary Improvement, used funding from the Pew Charitable Trust to extend the scope of a study examining the impact of state and university admission standards on secondary schools and students. Similarly, 10 R&D Centers reported that they leveraged additional money from other federal and state agencies and not-for-profit organizations to promote Education’s research agenda. Two of these R&D Centers reported receiving more funding from these other sources of funding than from OERI. Three Regional Labs also reported receiving funds from non-OERI sources, with these funds composing between 1 to 27 percent of their budgets. Two Comprehensive Centers reported receiving external funding and these funds accounted for 0.3 to 2 percent of those Centers’ funding.

Education’s Ability to Shape R&D Center, Regional Lab, and Comprehensive Center Agendas Varies

Education shapes the activities of the R&D Centers, Regional Labs, and Comprehensive Centers through its funding documents and program monitoring. Education uses funding documents, such as grant announcements and statements of work, to guide and direct activities included in the mandates and to obtain help in implementing department activities. In addition, Education assigns program officers to oversee the activities of the R&D Centers, Regional Labs, and Comprehensive Centers.

Although priorities in R&D Center cooperative agreements are broad and do not impose particular methods for researching a topic, they are specific enough to shape the direction and breadth of the R&D Centers’ research agenda. For example, in spelling out the priorities for the R&D Center for enhancing young children’s development and learning, Education identified topics, theories, and research areas that the Center should address. Similarly in spelling out the priorities for the R&D Centers for meeting the needs of diverse student populations, Education identified topics, theories, and student populations to be studied.

In addition, the R&D Centers’ cooperative agreements require that a portion of R&D Centers’ funds be spent on tasks defined by OERI. R&D Centers’ cooperative agreements require them to reserve 5 percent of their core funds to carry out OERI initiated activities that assist OERI in carrying out its responsibilities. For example, OERI may require an R&D Center to write a briefing paper or conduct a research project.

Contracts between Education and the Regional Labs give the Regional Labs control over most of their activities, but also give Education the opportunity to guide some of their work. Regional Labs identify the critical issues in their region and develop plans to address these issues. However, these contracts also allow Education to assign each Regional Lab a broad specialty area for example, early childhood education or educational technology—that reflects the national education agenda and is aligned with a dominant theme of an OERI research institute with which they are associated. In making Regional Labs responsible for a particular specialty area, the contracts required that they (1) conduct development, applied research, and dissemination in that area; (2) keep abreast of developments in their designated field; (3) provide subject area expertise to other labs; and (4) work cooperatively with OERI institutes as appropriate. Although Education defines the general type of activities for Regional Labs in specialty areas, the governing boards determine the focus of the activities and the extent to which they met the requirements of the contract. Additionally, the contracts require Regional Labs to work together on areas of concern to all Regional Labs, such as how to effectively disseminate their products and develop a telecommunications network. Regional Labs reported spending about 25 percent of their funds on these required national activities and on other activities with a national purpose. Unlike the cooperative agreements for the R&D Centers, Regional Lab contracts do not require them to do work to support OERI activities. Prior to 1995, the contracts required that the Regional Labs spend 1 percent of their core funds from OERI to support OERI. However, this requirement was dropped because of objections from the Regional Labs. Currently, Regional Labs may agree to perform work for OERI in return for additional funding.

Although the Comprehensive Centers are not research entities, Education shapes their activities by setting priorities for them in funding documents. Such priorities include, for example, meeting with school district officials to review and provide advice on district procedures for meeting federal requirements and assisting school districts in the development of student assessments. In addition, Comprehensive Centers are required to engage in common activities identified by Education such as conducting annual conferences on school improvement activities that promote Education’s agenda.

In addition to funding documents, program officers who administer grants and contracts may play an important role in influencing activities of the R&D Centers and Comprehensive Centers by ensuring that work performed is consistent with work proposed and that funds are being used as effectively and efficiently as possible. Program officers are supposed to perform the following types of activities: (1) help to develop funding documents and ensure that R&D Centers and Comprehensive Centers are in compliance with these documents; (2) review progress reports, financial reports, and products; and (3) approve dissemination plans, staffing changes, and activities funded by other sources. Program officers reported using these oversight functions to ensure that the activities of R&D Centers and Comprehensive Centers are consistent with their proposals. For example, Education program officials reported that they identified activities that were inconsistent with those in the R&D Centers’ proposals and subsequently negotiated alternative activities, indicated where collaboration between some R&D Centers would be beneficial, and encouraged R&D Centers to drop nonpromising lines of research. Program officers also may play a role in determining supplemental and future funding decisions.

Even though regional governing boards are responsible for the oversight of Regional Labs, Education’s program officers, nonetheless, may still have the potential to influence the activities of these labs. A program officer assigned to a Regional Lab described their role as that of “critical friends” who use their professional expertise and interpersonal relationships with Regional Labs’ staff to influence the activities of the Regional Labs. In addition, Regional Lab program officers, like R&D Center program officers, may play a role in determining supplemental and future funding decisions.

Time and authority may limit program officers’ ability to exercise their influence. In OERI, one program officer is generally assigned to each R&D Center and Regional Lab. The program officers we interviewed reported spending about 50 percent of their time on monitoring activities related to the R&D Centers and Regional Labs because of other assigned responsibilities. Because the R&D Centers and Regional Labs are complex organizations and prolific producers of products and services, officers have to be very selective in targeting their own time. Only one program officer is assigned to monitor all 15 Comprehensive Centers.

Regional Labs’ Governance Structure Limits Education’s Ability to Provide Product Oversight and Shape Their Agendas

The Congress has consistently given education program oversight to either a federal agency—usually Education or to the states, since the states are generally responsible for the education of their students. The Regional Labs are unlike other federal education programs because neither the federal government nor state governments have oversight responsibility for programs. Specifically, Education has little control over Regional Labs because regional boards govern them. This occurs in spite of the fact that the Regional Labs get the largest share of the federal dollars devoted to these three programs that conduct or support research. The law requires that the Regional Labs establish governing boards with regional representatives. The regional boards have sole responsibility for determining the regional agenda and for determining whether the Labs are fulfilling the responsibility of their contracts, even though these contracts are funded by Education. Even for the national specialty areas, Education sets only wide parameters while the governing boards determine specific activities. Education limits its communications with Regional Labs mainly to administrative issues, according to a director we interviewed at a Regional Lab. In addition, Education’s Regional Lab program officers told us their comments on Regional Lab work and products were only advisory.

The law requires each Regional Lab to establish a governing board that reflects both a balanced representation of the states in the region and the concerns of regional constituencies and includes teachers and education researchers. In addition, Education requires that every chief state school officer in the area served by the Regional Labs be offered an opportunity to serve on the board or to designate a representative. These safeguards, however, do not assure that priorities of each state in the region will be met. A variety of factors potentially make achieving balanced regional and state representation challenging. First, Regional Labs are not given guidance on how to obtain balance between states and regional interests, and regions, unlike states, have no formal governing body to establish educational priorities. Second, the governing boards, not the states, have the authority to determine how members are nominated and selected. Finally, states within regions vary substantially in size, population density, poverty levels, and ethnic composition. These factors may determine state educational priorities in a way that would make them vary widely. For example, California—a diverse and large state in size and population—is included with Utah, Nevada and Arizona in one region. New York also a diverse and large state—is included with Vermont and other New England states in another region.

Collaboration and Coordination among the R&D Centers, Regional Labs, Comprehensive Centers, and Education Is Ongoing

R&D Centers, Regional Labs, and Comprehensive Centers provided many examples of collaborative and coordinated activities. R&D Centers, Regional Labs, and Comprehensive Centers are required to collaborate and coordinate with each other and Education. The programs reported they are most likely to engage in collaborative and coordinated activities when they share a common interest in a specific student population, such as English language learners, or a specific topic, such as assessment. Partnerships between programs, common memberships in consortia, and staff members who are employed by more than one of these organizations facilitate collaboration and coordination and help leverage resources. Education plays a proactive role in promoting collaboration and coordination by including requirements for certain activities as part of its funding documents and in its ongoing negotiations with these organizations. However, certain factors differences in student populations, funding uncertainty, and competition reduce opportunities for collaboration and coordination.

R&D Centers, Regional Labs, and Comprehensive Centers Collaborate and Coordinate in Various Ways

R&D Centers, Regional Labs, and Comprehensive Centers collaborate and coordinate with each other and Education as required by law and the documents that control their funding. Certain types of collaboration and coordination flow naturally from overlapping needs, interests, and resources. For example, Regional Labs and Comprehensive Centers with populations of students with limited English proficiency would naturally make use of the R&D Centers, Regional Labs, and Comprehensive Centers that have expertise in that area.

R&D Centers, Regional Labs, and Comprehensive Centers reported a variety of collaborative efforts for fiscal year 2000, including joint projects and training. Joint projects included: Two Comprehensive Centers (the Northern California Comprehensive Center and the Southern California Comprehensive Center) worked together to produce teleconferences to help low performing schools. Ten Regional Labs and an R&D Center (the National Center for Early Development and Learning) produced a training guide entitled Continuity in Early Childhood: A Framework for Home, School, and Community Linkages.

An R&D Center (the Center for Research on the Education of Students Placed At-Risk) and a Regional Lab (the Northwest Regional Lab) produced a joint publication about parent involvement in schools.

A Regional Lab (the Northwest Regional Lab) and an R&D Center (the Center for Research on Evaluation, Standards, and Student Testing) created the Classroom Assessment Tool Kit.

Examples of training included: A Comprehensive Center (the Southern California Comprehensive Center) trained other Comprehensive Centers to teach instructors how to coach children learning to read.

An R&D Center (the Center for Research on Evaluation, Standards, and Student Testing) trained a Regional Lab (WestEd) to use a data collection tool.

An R&D Center (the National Center for Improving Student Learning and Achievement in Mathematics and Science) worked with a Comprehensive Center (the Region VI Comprehensive Center) to conduct a professional development project by teaching people to train math and science teachers.

A Regional Lab (the Southwest Educational Development Laboratory) provided training to a staff member of a Comprehensive Center (the Southeast Comprehensive Center) in the use of Flashlight and Compass: A Collection of Tools to Promote Instructional Coherence a tool for establishing teacher study groups.

R&D Center, Regional Lab, and Comprehensive Center staff also provided many examples of coordination efforts meant to ensure that each was aware of the others’ projects. Some examples include the following: As a follow-up to Education’s National Awards for Model Professional Development, three Regional Labs, WestEd, the North Central Regional Lab, and the Mid-Continent Regional Lab, studied how the awarded districts supported districtwide teacher and student learning. An R&D Center, the Center for the Study of Teaching and Policy, shared its data on resource allocation among some of the same districts and contributed to the research questions and design. The Regional Labs shared the findings with Education.

An R&D Center, Center for Research on Education, Diversity, and Excellence, reviewed work on Spanish Writing Assessment done by a Regional Lab, Northwest Regional Lab.

Legislative Requirements and Common Interests Foster Collaboration and Coordination

Legislation requires that the Regional Labs and Comprehensive Centers collaborate and coordinate with each other and with the R&D Centers, but their mandates differ in the amount of collaboration and coordination they require. The Regional Labs are required to collaborate and coordinate with each other, Education-funded technical assistance providers, and OERI institutes, and to share and plan joint activities with other Education- funded state, and federal programs. They are also required to establish a network for sharing information, planning activities involving multiple regions, and working on national projects. The Comprehensive Centers are required to share information, coordinate services, and work cooperatively with the Regional Labs, R&D Centers, Education’s regional offices, state and local educational agencies, and all other Education- funded research, development, dissemination, and technical assistance programs. The R&D Centers do not have specific legislative requirements to collaborate and coordinate with other Education-funded programs; however, they are required to do so in their cooperative agreements.

R&D Centers, Regional Labs, and Comprehensive Centers collaborate and coordinate when they have an interest in the same student population. The following examples illustrate how student populations provide a focal point for collaboration and coordination.

A member of the Mid-Continent Regional Lab, which includes states with large numbers of Native Americans, sat on the steering committee of the National Research Center on the Gifted and Talented, an R&D Center. This committee oversaw the production of a publication on talented American Indian and Alaskan Native students.

Two R&D Centers, the Center for Research on Education, Diversity, and Excellence and the National Center for Improving Student Learning and Achievement in Mathematics and Science, published a newsletter on issues related to how diverse students learn math and science.

All 15 of the Comprehensive Centers created an Internet mailing list about English-language learners to share information from their regions, identify staff with proficiency in meeting the needs of English- language learners, and disseminate information.

Interlacing Organizational Relationships Generally Facilitate Collaboration and Coordination

The structures of the R&D Centers, Regional Labs, and Comprehensive Centers foster collaboration and coordination among them and other entities. In some cases, partnerships may encourage collaboration. In other cases, collaboration occurs because one single entity operates both an R&D Center, Regional Lab, and/or Comprehensive Center.

Partnerships may encourage collaboration by establishing bridges between programs. R&D Centers, Regional Labs, and Comprehensive Centers are programs that operate through parent organizations universities, not-for-profit organizations, and educational agencies. These parent organizations create consortia—formal partnerships with other universities, not-for-profit organizations, for-profit corporations, and educational agencies—to run R&D Centers and Comprehensive Centers. Participants in partnerships that run R&D Centers range from 2 universities to 29 universities and not-for-profit organizations. Figure 8 shows one R&D Center that is a consortium of 5 universities and the affiliations they have with other R&D Centers. (Parent Organization) (NCPI)

Center for Research on Education, Diversity and Excellence (CREDE)

Unlike R&D Centers and Comprehensive Centers, the parent organizations that run Regional Labs do not form partnerships with other universities, not-for-profits, or educational agencies to run the Regional Labs. They may, however, be asked by a Comprehensive Center or R&D Center to enter into a partnership in order to provide specialized services. We found four parent organizations that operated Regional Labs that had formed partnerships with Comprehensive Centers. Figure 9 shows an example of such a partnership.

Participants in partnerships that run Comprehensive Centers range from three to eight universities, not-for-profit organizations, and educational agencies. Most parent organizations that run Comprehensive Centers partner with other organizations to obtain expert services in specialized areas, such as migrant education or Indian education. Because each of these specialized partners may work with as many as four Comprehensive Centers, these partnerships may establish bridges that forge coordination in particular topical areas. For example, ESCORT, a former categorical technical assistance center, specializes in migrant education and partners with four Comprehensive Centers to provide services in that area.

Collaboration also results when a single organization operates both an R&D Center, Regional Lab, or a Comprehensive Center. For example, staff from the Western Regional Lab and the Northern California Comprehensive Center participated in a workgroup that developed a guide on how schools could better obtain student perspectives and suggestions to improve school planning. As shown in figure 9, both the Western Regional Lab and the Northern California Comprehensive Center are run by the same parent organization. Five of the 10 parent organizations that run a Regional Lab also run a Comprehensive Center. One of the 12 parent organizations that operates an R&D Center also operates a Comprehensive Center.

These interlacing organizational relationships allow R&D Centers, Regional Labs, and Comprehensive Centers to leverage resources. For example, an Education official working with the Comprehensive Centers told us that through their parent organizations and partnerships Comprehensive Centers are able to leverage the personnel and expertise needed to perform their work. In those cases where parent organizations run two programs, we were told that staff divide their time between programs to leverage expertise. Similarly, a director of a parent organization that runs a Regional Lab and a Comprehensive Center stated that the Comprehensive Center draws upon experts assigned primarily to other projects to obtain skills needed to implement particular activities.

Individual relationships, such as staff members holding multiple appointments within R&D Centers, Regional Labs, or Comprehensive Centers also facilitate collaboration and coordination among R&D Centers, Regional Labs, and Comprehensive Centers. For example, a principal investigator for the Consortium for Policy Research in Education, an R&D Center, also works on projects for the National Center for Improving Student Learning and Achievement in Mathematics and Science, another R&D Center.

Education Plays a Proactive Role in Encouraging Collaboration and Coordination

The funding agreements between Education and the R&D Centers, Regional Labs, and Comprehensive Centers reflect Education’s interpretation and implementation of legislative requirements for collaboration and coordination. The cooperative agreements for the R&D Centers require them to collaborate and coordinate with Regional Labs, Comprehensive Centers, other federal programs, policy institutions, and advocacy groups. R&D Centers also agree, in their funding documents, to conduct an annual meeting to share research with Education and other research and development programs, and collaborate with OERI. Regional Lab contracts with Education require that they participate in at least two meetings a year convened to discuss issues related to Education-funded programs. In addition, Education’s contracts for Regional Labs require Regional Lab representatives to meet with OERI annually and chairpersons of the governing boards to meet with OERI when their contract begins. Regional Labs also have an option in their contracts with Education that allows them to earn supplemental funds by agreeing to perform work in collaboration and coordination with OERI, including sponsoring meetings and panels and writing briefs. Cooperative agreements for the Comprehensive Centers require them to meet with seven different Education-funded programs, including the Regional Labs, to discuss collaboration and coordination; plan a national conference; engage in a common project to improve teaching; and collaborate with each other and local and state educational agencies.

Cooperative agreements for the R&D Centers and Comprehensive Centers also outline Education’s responsibilities to facilitate collaboration and coordination. For example, the funding agreement for the Comprehensive Centers specifies that Education officials will work with Comprehensive Centers in planning conferences and identifying areas for collaboration and coordination. The funding agreements that Education has with Regional Labs do not outline Education’s responsibilities for facilitating collaboration and coordination.

Education officials may also identify appropriate areas for collaboration and coordination. For example, an institute director told us that she contacted an R&D Center in another institute to discuss possible areas for collaboration and coordination with the R&D Center in her institute. Similarly, some Comprehensive Center officials stated that the program officer assigned to them identified areas for cross-program collaboration and coordination and communicated with them frequently. For example, the program officer suggested collaborating and coordinating on the creation of a common framework for working with low performing schools in Comprehensive Center regions.

Differences in Student Population and Topics, Funding Uncertainty, and Competition Reduce Opportunities for Collaboration and Coordination

Despite the efforts made by the R&D Centers, Regional Labs, Comprehensive Centers, and Education in fostering collaboration and coordination, barriers exist. R&D Centers, Regional Labs, and Comprehensive Centers cited differences in student populations and topics, uncertain funding, and competition as hindrances to collaboration and coordination. R&D Centers have different research focuses and conduct research on different topics. Regional Labs and Comprehensive Centers serve diverse geographical areas with different interests. These differences potentially reduce opportunities for collaboration and coordination. For example, an R&D Center with a focus on postsecondary education would have little or no reason to collaborate or coordinate on substantive issues with other R&D Centers, Regional Labs, and Comprehensive Centers that focus on research related to elementary and secondary education. Funding uncertainties also make collaboration and coordination difficult. Directors said they were often reluctant to write collaborative activities into their proposals because they did not know which organizations would win future funding competitions. Competition also limits collaboration and coordination. The education research and technical assistance business is a competitive industry. Like others in competitive industries, R&D Center, Regional Lab, and Comprehensive Center staff are protective of ideas and practices that give them advantages over other organizations that they perceive as competitors for future sources of funding.

Education Needs to Improve Its Assessments of the R&D Centers, Regional Labs, and Comprehensive Centers

The recently funded evaluations of the R&D Centers, Regional Labs, and Comprehensive Centers provided limited information on outcomes of the activities conducted by these programs. OERI is required to use peer review to evaluate the R&D Centers and Regional Labs. Peer review is well accepted and widely used for assessing the merit of research proposals and the scientific soundness of research. Unlike the R&D Centers, research is only a relatively small part of what Regional Labs do. Their other activities dissemination and technical assistance would have been more appropriately evaluated using methods other than peer review. The peer review processes that Education used to evaluate the R&D Centers and Regional Labs have shortcomings that limited the usefulness of the findings. First, the peer review process used has the potential for bias because the R&D Centers and the Regional Labs selected most of the products that were reviewed. Second, the processes were cumbersome. For example, Education required each member of a review team to write an individual assessment report and to review all contracts, contract modifications, progress reports, and annual updates for a three-year period. In addition, with regard to the Comprehensive Centers, the customer satisfaction survey evaluations of the Comprehensive Centers did not provide information on individual centers.

Recent Peer Reviews of the R&D Centers and Regional Labs Provided Limited Information

Traditionally, Education has used peer review to evaluate activities carried out by OERI, including those conducted by the R&D Centers and the Regional Labs. The Educational Research, Development, Dissemination, and Improvement Act of 1994 codified this practice by requiring OERI to develop peer review standards for evaluating and assessing the performance of recipients of grants, cooperative agreements, and contracts that exceed $100,000, as well as for selecting proposals for funding and identifying exemplary and promising educational programs.

Historically, peer review has been used extensively in the selection of proposed research projects and, to a lesser extent, to evaluate research and development programs. Peer review entails an independent assessment of the technical or scientific merit of research by peers who are scientists with knowledge and expertise equal to that of the researchers whose work they review. It is sometimes used to evaluate research when the ultimate outcomes of the research are unpredictable.

Peer review may be appropriate for evaluating some activities conducted by the R&D Centers but other evaluation techniques are better suited for evaluating the many activities of the R&D Centers and Regional Labs. According to the National Academy of Sciences, although peer review is well suited for assessing theory-driven research with potential long-term effects, it is less appropriate for assessing applied research, development, technical assistance, and dissemination efforts in which outcomes are anticipated and can be measured over a relatively short period of time.The R&D Centers and Regional Labs engage in many research activities that are designed to achieve practical outcomes. Evaluation methods that measure outcomes customer surveys, comparisons with similar programs, and controlled case studies may be better suited for evaluating these activities.

Education’s peer review processes have the potential for bias and were cumbersome. Both of these conditions limited the usefulness of findings in addressing key issues.

The self-selection of materials for review by R&D Centers and Regional Labs did not provide a representative cross section of their products and services. R&D Center staff were involved in deciding which products were to be reviewed and each Regional Lab nominated five or six major products or services, two of which were selected for review. While this approach allowed reviewers an in-depth look at major program initiatives, it did not provide reviewers with a cross section of the R&D Centers or Regional Labs’ work, nor did it allow them to generalize from these works to other activities.

Selecting as reviewers for the R&D Centers and Regional Labs some individuals who have previously evaluated the merits of the grant applications or proposals raised questions about objectivity. It is likely that the individuals who selected these organizations as grant or contract recipients might want their original choices validated.

Requiring peers to have a broad mix of skills made selection and scheduling of reviewers difficult. Unlike other agencies that select peers solely on the basis of their expertise in the area of work, OERI requires that review panels include individuals with a broad range of knowledge and experience. For example, OERI standards require peer review panels to include individuals with in-depth knowledge of education policy or practice and in-depth knowledge of theories and methods of study related to the subject area. These requirements complicated the identification of peer review panels.

The amount of material to be reviewed was extensive in terms of the short time frames of the assessments and the complexity of the organizations. Over a short period of time—for example, 5 days on site for the review of a Regional Lab, with half that time devoted to data gathering and the other half to writing the reports—all reviewers were required to read immense amounts of material, including funding documents, statements of work, proposals, and progress reports, to learn, in detail, how the programs performed their work and to write individual reports. Some reviewers complained that they only had time to “scratch the surface,” and that much of the material they had to review was repetitive. If responsibilities could have been shared, peer reviewers would have been able to gather more in-depth knowledge.

In some cases, materials chosen for review by the R&D Centers were incomplete. Reviewers of some programs noted that materials addressing the rationale, hypotheses, controls, and usefulness of studies were often insufficient for them to make informed judgements.

Assessments took place midway in the funding cycles. The reviews generally took place during the third year of a 5-year contract. As a result, reviewers were hesitant to be critical in reports because many studies were on going and therefore could improve by the end of the contract.

The dual purposes of the peer reviews inhibited candor. On one hand, Education designed the reviews to be formative evaluations— evaluations that were to focus on the performance of the programs in terms of their missions and the technical quality of their products. In this regard, reviews were to examine the overall quality of the work of the R&D Centers and Regional Labs, the extent to which R&D Centers and Regional Labs performed their work on time and met professional standards. However, the reviews were also designed to assess the usefulness, outcomes, and effects of their work to help OERI determine if the R&D Centers and Regional Labs merited continued funding. The peer review process depended exclusively upon expert opinion rather than directly measuring how useful the research was or its effects. Reviewers discussed “the potential” value of activities and were not able to predict the ability of the entities to contribute substantially to the field. Moreover, they did not believe their findings should have influence over funding decisions, which affected what they wrote in their reports.

Evaluation of the Comprehensive Centers Did Not Provide Comparable Information on Individual Centers

Education’s evaluation of the Comprehensive Centers met the requirements of the law but provided little information that would help Education determine if each Comprehensive Center was meeting the needs of its customers. The Elementary and Secondary Education Act requires the Secretary of Education to collect information about the availability and quality of services provided by the Comprehensive Centers and to conduct surveys to determine if populations served by the Comprehensive Centers are satisfied with their access to services and the quality of those services. As part of the year 2000 evaluation, a contractor surveyed the satisfaction of customers who had participated in either of two activities offered by a Comprehensive Center. These activities were selected from among all the activities offered by the Comprehensive Center because they were long-term or intensive. The contractors surveyed customers by randomly selecting them from a list prepared by each Comprehensive Center. The contractor also surveyed a nationally representative sample of state and district officials that they identified as a likely pool of customers for the centers and a sample of gatekeepers— individuals that had requested or negotiated for services on behalf of their school or school district. In randomly selecting customers to survey, the contractor did not choose a number large enough from each Comprehensive Center’s list to allow any reliable generalizations to be made about a particular Comprehensive Center. Likewise, the sample of potential customers and gatekeepers was not suitable to draw conclusions about an individual Comprehensive Center. The inclusion of representative activities and customers for each Comprehensive Center would have increased the cost of the mail survey. However, not drawing a sample that was representative at the center level reduced the usefulness of the evaluation because Education could not identify variation across Comprehensive Centers or obtain information to improve practices at individual Comprehensive Centers.

Conclusions

The Regional Labs are unlike most federal education programs because neither the federal government nor state governments have oversight responsibility for their programs. Under the current structure, Education is accountable for the activities it funds through the Regional Labs, but current laws limit its ability to oversee those activities. Not only is federal oversight limited, states also have limited control over the regional agenda. Although the requirement that governing boards have a balanced representation of states in the region may ensure state input for the agenda of the Regional Lab, the regional priorities that the Regional Labs serve may not correspond to the educational priorities of all states in the regions. Thus, neither the states nor Education can ensure that the Regional Labs are meeting the needs of the states.

Congressionally mandated peer reviews of the R&D Centers and Regional Labs have produced limited information about the overall performance of each organization, the services they provide, or the extent to which teaching and learning are improved by the products R&D Centers and Regional Labs produce. In part, this was because reviewers could not divide the tasks among themselves, as each reviewer was required to do a full, independent assessment. In addition, given the present practice allowing the Regional Lab directors to choose products and services for review, the potential exists for bias and therefore calls into question the quality of the assessments. As a result, Education lacks information that would be useful in making funding decisions or improving the performance of each organization. Unless standard program evaluation techniques, such as customer surveys or controlled case studies, are introduced into the evaluation process of these entities, these problems are likely to continue.

The value of the mandated year 2000 evaluations of the Comprehensive Centers was limited. We recognize that addressing this problem would involve expanding the sample sizes and, for the consumer survey, the number of activities selected. However, currently, neither the Comprehensive Centers themselves nor Education can determine if the customers of a particular Comprehensive Center are satisfied with some or all of its products and services. As a result, problems at any given center could go unchecked. Moreover, Education cannot assess the relative strengths and weakness of individual Comprehensive Centers so it can improve the services in all centers and make better funding decisions.

Matters for Congressional Consideration

If the Congress wishes to ensure greater accountability to a governmental entity for the Regional Labs, it could consider either giving Education responsibility for determining the regional agenda and overseeing the products and services of the Regional Labs or Education could provide these funds to states, possibly as part of a larger formula grant, for subsequent distribution by each state. This would give states greater control in purchasing research-based educational products and services.

If the Congress wants to increase the usefulness of the assessments of the R&D Centers and Regional Labs, the Congress should consider allowing Education to use methods other than peer review when such methods are more appropriate than peer review for evaluating the activities of R&D Centers and Regional Labs.

Recommendations for Executive Action

To improve the assessments of the R&D Centers and Regional Labs, we recommend that the Secretary of Education direct the Assistant Secretary of OERI to use random selection of projects, services and products to be reviewed when conducting future evaluations of R&D Centers and Regional Labs, and revise the peer review standards to allow for division of labor and greater concentration on assessing the quality of projects, services and products.

To improve the performance and usefulness of the Comprehensive Centers, the Secretary of Education should direct the Assistant Secretary of Elementary and Secondary Education and the Assistant Secretary of Bilingual Education and Minority Languages Affairs to design future evaluations of the Comprehensive Centers to produce findings pertaining to individual Comprehensive Centers.

Agency Comments

We provided a draft of this report to the Department of Education for comment. Education’s Executive Secretariat confirmed by e-mail that Department officials had reviewed the draft and had no comments except for a few technical clarifications regarding funding.

We are sending copies of this report to the Secretary of Education, the appropriate congressional offices, and other interested parties. If you have any questions, please contact me at (202) 512-7015 or Eleanor L. Johnson (202) 512-7209. Other contributors can be found in appendix I.

Appendix I: GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to those named above, Malcolm Drewery, Jr., Tahra N. Edwards, Richard B. Kelley, and Sarah Moorhead made key contributions to this report.