61428 Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations

DEPARTMENT OF EDUCATION

34 CFR Part 701

RIN 1850–AA52

Standards for Conduct and Evaluation

of Activities Carried out by the Office

of Educational Research and

Improvement (OERI)—Designation of

Exemplary and Promising Programs

AGENCY: Department of Education.

ACTION: Final regulation.

SUMMARY: The Assistant Secretary for

Educational Research and Improvement

(OERI) is establishing final regulations

pursuant to the ‘‘Educational Research,

Development, Dissemination, and

Improvement Act of 1994.’’ The

regulations are intended to provide

quality assurance that programs

designated by the Department of

Education as either exemplary or

promising have met criteria that will

allow educators, professional

organizations, and others to use these

programs with confidence.

DATES: These regulations take effect

December 17, 1997. However, affected

parties do not have to comply with the

information collection requirement in

§ 701.4 until the Department of

Education publishes in the Federal

Register notification of the compliance

date and the control number assigned by

the Office of Management and Budget

(OMB) to this information collection

requirement. Publication of the control

number notifies the public that OMB

has approved this information

collection requirement under the

Paperwork Reduction Act of 1995.

FOR FURTHER INFORMATION CONTACT:

Stephen O’Brien, U.S. Department of

Education, 555 New Jersey Avenue,

NW, Room 502B, Washington, D.C.

Telephone: (202) 219–2141. Internet:

(StevelO’Brien@ed.gov). Individuals

who use a telecommunication device for

the deaf (TDD) may call the Federal

Information Relay Service (FIRS) at 1–

800–877–8339 between 8 a.m. and 8

p.m., Eastern time, Monday through

Friday.

Individuals with disabilities may

obtain this document in an alternate

format (e.g., Braille, large print,

audiotape, or computer diskette) on

request to the person listed in the

preceding paragraph.

SUPPLEMENTARY INFORMATION: On March

31, 1994, President Clinton signed

Public Law 103–227, which includes

Title IX, the Educational Research,

Development, Dissemination, and

Improvement Act of 1994 (the Act). The

Act restructured OERI and provided it

with a broad mandate to conduct an

array of research, development,

dissemination, and improvement

activities aimed at strengthening the

education of all students.

The Act directed the Assistant

Secretary to develop, in consultation

with the National Educational Research

Policy and Priorities Board (the Board),

the highest standards of professional

excellence necessary to govern the

conduct and evaluation of all research,

development, and dissemination

activities carried out by the OERI. The

legislation requires that the standards be

developed in three phases.

In the first phase, standards were

promulgated to establish the peer

review process and evaluation criteria to

be used for reviewing applications for

grants and cooperative agreements and

proposals for contracts. The Department

published final regulations setting out

these standards on September 14, 1995

(60 FR 47808). The regulations in this

announcement address the second

phase of development by establishing

the criteria for panels to use in

reviewing potentially exemplary and

promising educational programs. The

Assistant Secretary will later publish

proposed regulations for phase three of

the standards, which will govern how

OERI evaluates performance of its

recipients of grants, contracts, and

cooperative agreements.

The OERI legislation requires that

expert panels be established to review

educational programs submitted by

individuals or organizations. The

legislation also provides that the

Secretary may identify educational

programs for the panels to review. The

statute requires the panels to

recommend to the Secretary those

programs that should be designated as

exemplary or promising and

disseminated through the Department’s

National Education Dissemination

System. The law requires that each

panel consist of appropriately qualified

experts and practitioners and requires

the Secretary to develop standards that

describe the procedures the panels will

use in reviewing the educational

programs. Section 941(a)(3) of the law

broadly defines educational programs to

include educational polices, research

findings, practices, and products.

Educational programs may range in size

and complexity from an individual

instructional program—such as an

elementary school science program—to

a comprehensive reform initiative

involving multiple goals and

participants. Programs at all levels of

education—preschool, elementary,

secondary, and postsecondary—are

eligible for consideration.

In determining whether an

educational program should be

recommended as exemplary or

promising, each panel is required by the

Act to consider: (a) Whether, based on

empirical data, the program is effective

and should be designated as exemplary

or (b) whether there is sufficient

evidence to demonstrate that the

program shows promise for improving

student achievement and should be

designated as promising. The Act

expressly states that a panel shall not

eliminate a program from consideration

based on the lack of one type of

supporting data such as test scores.

The evaluation process set forth in the

final regulations will ensure that

programs disseminated by the

Department are high-quality, researchbased programs that have provided

evidence indicating they have improved

teaching, learning, or both, or has

demonstrated other worthy educational

performance outcomes. The

Department’s dissemination system is

designed to make information about

these promising and exemplary

programs available to the public as

quickly as possible. The system will

enable the Department to respond to all

forms of requests for information and

assistance, and to support the

applications of research and best

practice. The system will use electronic

networking and the capabilities of:

—National Research Institutes;

—Educational Resources Information

Center (ERIC);

—Regional Educational Laboratories;

—Department-supported dissemination

and technical assistance providers;

—National Library of Education;

—Eisenhower Regional Consortia and

Clearinghouse, and

—Other public and private nonprofit

entities, including education

associations and networks.

Until recently, the Department

validated exemplary programs through

its Program Effectiveness Panel (PEP)

and disseminated them through the

National Diffusion Network (NDN).

Since this program no longer exists,

with the adoption of these standards the

Department will evaluate and

disseminate promising educational

programs in addition to exemplary

programs. The Department will also

work in partnership with constituency

groups who have expertise in the

specific topic areas represented by the

expert panels to develop coordinated

procedures to maximize their

involvement in this work.

On June 3, 1996, the Secretary

published a notice of proposed

rulemaking (NPRM) for this part in the
Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations 61429

Federal Register (61 FR 27990–27993).

These final regulations contain three

major changes from the NPRM. These

changes are fully explained in the

‘‘Analysis of Comments and Changes’’

elsewhere in this preamble. The changes

pertain to the standing panel; the

distinction between ‘‘promising’’ and

‘‘exemplary’’; and the factors listed

under the criteria expert panels will use

to evaluate programs.

Analysis of Comments and Changes

In response to the Secretary’s

invitation in the NPRM, seven parties

submitted comments on the proposed

regulations. This included comments

from individual members of two pilot

panels (math/science and gender equity)

that were appointed by the Secretary to

field test the expert panel process. In

addition to the public comment,

comments from the Board’s

Subcommittee on Standards are

addressed as required by the legislation.

The full Board approved the final

regulations at a meeting on September

26, 1997. An analysis of the comments

and of the changes in the regulations

since publication of the NPRM follows.

Major issues are grouped according to

subject with appropriate sections of the

regulations referenced in parentheses.

Technical and other minor changes—

and suggested changes the Secretary is

not legally authorized to make under the

applicable statutory authority—are not

addressed.

Eligibility (§ 701.3)

Comments: One commenter asked for

clarification on who is eligible to submit

educational programs for designation as

promising or exemplary. Specifically,

this commenter asked whether

federally-funded entities, such as the

Regional Laboratories, will be required

to go through this process; whether local

agencies that receive Federal funding

through states, such as under Title I of

the Elementary and Secondary

Education Act (ESEA), may submit

programs on their own; whether

sponsors need to be invited to submit or

may submit on their own initiative; and

whether for-profit entities may submit.

Discussion: The law provides that

‘‘individuals’’ or ‘‘organizations’’ may

submit educational programs for review.

Since the law is silent on the specific

nature of the organizations, the

Secretary believes that for-profit

agencies would be eligible to submit

programs for review. With respect to the

OERI-funded Regional Educational

Laboratories, the law provides that the

Secretary may identify those programs

for panel review. In addition, the

Secretary believes that the Laboratories

could submit one or more of their

programs on their own initiative. The

question of whether local agencies that

receive Federal funding through a State

or Federal entity, such as under Title I

of the ESEA, can submit on their own

or must go through their funding

agency, will be addressed in

administrative guidance.

Changes: None.

Content of Submissions (§ 701.4)

Comments: Three commenters made

suggestions about this section. Two

commenters believed that requiring

funding and staffing information was

burdensome and not germane to the

designation of a program as promising

or exemplary. One commenter believed

that this section should require program

sponsors to submit specific materials

related to content and methods. Another

commenter believed that this section

should include the requirement that the

program include evidence of

sustainability of improvement with

targeted student populations.

Discussion: The Secretary believes

that funding and staffing information

should be included to help determine

whether an educational program should

be recommended as either exemplary or

promising. The Secretary agrees that

sponsors should be required to submit

information or materials specific to

content and methods, as available and

appropriate. The Secretary believes that

the evidence of sustainability of student

improvement should be evaluated by

peer reviewers in accordance with

§ 701.22.

Changes: Section 701.4(b)(7) has been

renumbered as § 701.4(b)(8) and a new

§ 701.4(b)(7) has been added to include

a provision for specific materials

relevant to content and methods.

Procedures for Submitting Educational

Programs (New § 701.5)

Comments: One commenter believed

that the regulations should contain more

specificity about the procedure for

submitting programs to the expert

panels. This commenter requested

specifics on who receives the

submissions and whether they may be

submitted at any time or only on

specific dates.

Discussion: The Secretary agrees that

the general submission procedures

should be included in regulations. A

sponsor seeking the exemplary or

promising designation for its

educational program may submit its

program at any time for consideration to

the Assistant Secretary, who will assign

the submitted program to the

appropriate panel for review. The

individual expert panels will set

appropriate timelines for program

submissions. In addition, the Assistant

Secretary will periodically establish and

announce in the Federal Register

specific topic areas of high priority.

Sponsors of educational programs in

these areas will be invited to submit

them for consideration.

Changes: A new § 701.5 has been

added to include general procedures for

submitting educational programs for

review by an expert panel.

Establishment of Panels (§ 701.10)

Comments: The Board’s

Subcommittee on Standards

recommended a change to the expert

panel system. The Subcommittee

thought that the structure of having

members of the expert panels drawn

from a separate standing panel of

educational experts was an unwieldy,

overly-complicated structure. The Board

recommended that the expert panels be

formed separately from a standing

panel, which would instead provide an

administrative oversight and monitoring

function for the expert panels.

Discussion: The Secretary agrees that

the expert panel should be formed

separately from a standing panel. The

Secretary will determine the feasibility

of establishing a separate standing panel

for the oversight and monitoring

functions referred to by the Board—

functions which are administrative in

nature and could also be performed by

OERI staff. Elimination of a reference to

a standing panel in the regulations

would not alter the composition and

function of the expert panels as outlined

in the NPRM.

Changes: Section 701.10(a) has been

removed, § 701.10(b) has been revised,

§ 701.11 has been removed, § 701.12(a)

has been revised, and § 701.12 has also

been renumbered as § 701.11.

Panel Membership (§§ 701.11 and

701.12)

Comments: One commenter observed

that §§ 701.11 and 701.12 in the NPRM

did not explicitly state that those

serving on the panels would represent

both the community of practice and that

of research. One commenter believed

that each panel should include one or

more members with evaluation

expertise in order to help evaluate

evidence of effectiveness.

Discussion: The Secretary agrees with

these comments.

Changes: A sentence has been added

at the end of the renumbered § 701.11(a)

(formerly § 701.12(a)) stating that the

membership of the expert panels will

represent both the community of

practice and the community of research.

Additionally, §§ 701.11(b)(3) and
61430 Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations

701.11(b)(4) have been renumbered as

§§ 701.11(b)(4) and 701.11(b)(5),

respectively; and a new provision for

§ 701.11(b)(3) has been added to include

the selection of at least one individual

with expertise in evaluating educational

programs.

Difference Between Promising and

Exemplary Programs (§ 701.21)

Comments: Five comments were

received on the distinction between

promising and exemplary programs. As

proposed in § 701.21, the distinction

was based upon the generalizability of

the educational programs. Promising

programs had to meet each of the

criteria of educational effectiveness in

§ 701.22 (success, quality, educational

significance, and usefulness to others)

with respect to only one ‘‘context or

population.’’ Exemplary programs had

to meet each of the criteria ‘‘with

respect to multiple contexts or multiple

populations.’’

Two commenters believed that the

distinction should stay the way it was

in the NPRM, although one of those

suggested some clarifying language.

However, three commenters questioned

the distinction on the basis that it was

too narrowly and artificially drawn and

did not reflect the commonly

understood meaning of the words

‘‘promising’’ and ‘‘exemplary.’’ In this

regard, one commenter believed that

promising programs should not have to

meet every criterion in § 701.22 at the

same level as exemplary programs. Two

commenters believed that promising

programs should have to meet the

criteria at the same level as exemplary,

but that the evidence required of

promising programs should be less

stringent and that exemplary programs

should be held to a higher standard of

evidence.

Discussion: The Secretary agrees with

those commenters who questioned the

proposed distinction and advised OERI

to give a more common sense meaning

to the terms ‘‘promising’’ and

‘‘exemplary.’’ The Secretary believes

that the distinction between promising

and exemplary programs specified in

legislation is sufficient to cover these

concerns. The Secretary relies upon the

expert judgment of the expert panel

reviewers in determining the nature and

weight of evidence necessary to

designate a program as either promising

or exemplary, and in applying the

criteria listed in § 701.22 in making this

determination.

Changes: A revision has been made to

the distinction between ‘‘promising’’

and ‘‘exemplary.’’

Criteria (§ 701.22)

Comments: Five commenters

provided comments on this section and

suggested revisions to either the

wording of the criteria or to the content

of the factors listed under each criterion

or both. These comments included

comments from one member of the

math/science pilot panel and three

members of the gender equity pilot

panel. Although the math/science panel

member did provide comments specific

to the proposed criteria and factors, the

consensus of this panel was that the

expert panel process would be better

served if each panel developed its own

factors specific to the content or

discipline or both under review by the

individual panel. One commenter

suggested that the word ‘‘replicability’’

would better capture the concept for the

criterion entitled ‘‘usefulness to others.’’

In addition, OERI’s Board (The

Subcommittee on Standards) thought

that the regulations should be as simple

as possible and should give the expert

panels as much discretion as possible in

evaluating programs submitted for

review.

Discussion: In addition to the math/

science and gender equity panels, the

Secretary will establish pilot panels in

technology and early reading in the next

year. The Secretary has determined that

until the work of all four pilot panels is

concluded, the regulation should retain

only the four criteria outlined in the

NPRM in § 701.22 and allow each panel

the flexibility to establish its own

individual factors under each criterion

that are specific to its content or

discipline. The fact that the comments

from the public suggested various

changes to the factors underscores the

desirability of this approach. While the

final regulations will therefore no longer

require the expert panels to apply the

factors listed in the NPRM, the Secretary

encourages each panel to look at these

factors as suggested examples. The

Secretary will review the factors

developed by all of the panels to see if

the criteria set forth in the final

regulations need to be modified.

Changes: The factors specified under

each of the four criteria have been

eliminated and the criterion,

‘‘usefulness to others’’ has been changed

to ‘‘replicability.’’

Assessment of Educational Impact

In the NPRM the Secretary requested

comments on whether the proposed

regulations would require transmission

of information that is being gathered by

or is available from any other agency or

authority of the United States.

Based on the response to the NPRM

and on its own review, the Department

has determined that the regulations in

this document do not require

transmission of information that is being

gathered by or is available from any

other agency or authority of the United

States.

Electronic Access to This Document

Anyone may view this document, as

well as all other Department of

Education documents published in the

Federal Register, in text or portable

document format (pdf) on the World

Wide Web at either of the following

sites:

http://ocfo.ed.gov/fedreg.htm

http://www.ed.gov/news.html

To use the pdf you must have the

Adobe Acrobat Reader Program with

Search, which is available free at either

of the previous sites. If you have

questions about using the pdf, call the

U.S. Government Printing Office toll

free at 1–888–293–6498.

Anyone may also view these

documents in text copy only on an

electronic bulletin board of the

Department. Telephone: (202) 219–1511

or, toll free, 1–800–222–4922. The

documents are located under Option

G—Files/Announcements, Bulletins and

Press Releases.

Note: The official version of this document

is the document published in the Federal

Register.

List of Subjects in 34 CFR Part 701

Education, Educational research,

Reporting and recordkeeping

requirements.

(Catalog of Federal Domestic Assistance

Number does not apply)

Dated: November 11, 1997.

Ricky T. Takai,

Acting Assistant Secretary for Educational

Research and Improvement.

The Secretary amends chapter VII of

title 34 of the Code of Federal

Regulations by adding a new part 701 to

read as follows:

PART 701—STANDARDS FOR

CONDUCT AND EVALUATION OF

ACTIVITIES CARRIED OUT BY THE

OFFICE OF EDUCATIONAL

RESEARCH AND IMPROVEMENT

(OERI)—DESIGNATION OF

EXEMPLARY AND PROMISING

PROGRAMS

Subpart A—General

Sec.

701.1 What is the purpose of these

standards?

701.2 What definitions apply?

701.3 Who is eligible to submit an

educational program for review?
Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations 61431

701.4 What must a program sponsor submit

for review?

701.5 What are the procedures for

submitting an educational program for

review by an expert panel?

Subpart B—Selection of Panel Members

701.10 How are panels established?

701.11 How is the membership of expert

panels determined?

Subpart C—The Expert Panel Review

Process

701.20 How does an expert panel evaluate

programs?

701.21 What is the difference between an

exemplary and a promising program?

701.22 What criteria are used to evaluate

programs for exemplary or promising

designation?

Authority: 20 U.S.C. 6011(i), unless

otherwise noted.

Subpart A—General

§ 701.1 What is the purpose of these

standards?

(a) The standards in this part

implement section 941(d) of the

Educational Research, Development,

Dissemination, and Improvement Act of

1994.

(b) These standards are intended to

provide quality assurance that

educational programs designated by the

U.S. Department of Education as either

exemplary or promising have met

criteria that will allow educators,

professional organizations, and others to

use these programs with confidence.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.2 What definitions apply?

The following definitions apply to

this part:

Assistant Secretary means the

Assistant Secretary for the Office of

Educational Research and Improvement.

Educational programs mean

educational policies, research findings,

practices, and products.

Program sponsor means a party

submitting an educational program for

designation by the Secretary as either

promising or exemplary.

Secretary means the Secretary of the

Department of Education or an official

or employee of the Department acting

for the Secretary under a delegation of

authority.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.3 Who is eligible to submit an

educational program for review?

Any public or private agency,

organization or institution, or an

individual may submit an educational

program for review.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.4 What must a program sponsor

submit for review?

(a) To have an educational program

considered for designation as exemplary

or promising, a sponsor must submit to

the Secretary a description of the

program, program materials, and a

discussion of the program that is

responsive to the criteria in § 701.22.

(b) Information submitted must

include, to the extent relevant to the

particular program,—

(1) A program abstract of 250 words

or less;

(2) A description of the salient

features of the program;

(3) A description of the program’s

philosophy and history;

(4) Site information, including

demographics;

(5) A description of evaluation results;

(6) Funding and staffing information;

(7) Specific materials relevant to

content and methods, as appropriate;

and

(8) Organization name, address,

telephone and fax numbers, e-mail

address (if available), and contact

person.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.5 What are the procedures for

submitting an educational program for

review by an expert panel?

(a) An applicant seeking the

exemplary or promising designation for

its educational program may submit its

program at any time for consideration to

the Assistant Secretary, who will assign

the submitted program to the

appropriate expert panel for review.

(b) The Assistant Secretary will

periodically establish and announce in

the Federal Register specific topic areas

of high priority. Sponsors of educational

programs in these areas will be invited

to submit their programs for

consideration.

(c) The individual expert panels will

set appropriate timelines for reviewing

program submissions.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

Subpart B—Selection of Panel

Members

§ 701.10 How are panels established?

The Assistant Secretary selects

individuals, based on their areas of

expertise, to serve on expert panels in

specific topic areas for the purpose of

reviewing and evaluating educational

programs and recommending, to the

Secretary, those programs that should be

designated as exemplary or promising.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.11 How is the membership of expert

panels determined?

(a) For the review of each program or

group of programs, the Assistant

Secretary establishes an expert panel.

The membership of the expert panels

will represent both the community of

practice and the community of research.

(b) In establishing the membership of

each expert panel, the Assistant

Secretary—

(1) Selects individuals who have indepth knowledge of the subject area or

content of the program or group of

programs to be evaluated;

(2) Selects at least one current teacher,

principal, or other school-based or

community-based professional;

(3) Selects at least one individual with

expertise in evaluating educational

programs;

(4) Ensures that no more than onethird of the panel members are

employees of the Federal Government;

and

(5) Ensures that each panel member

does not have a conflict of interest, as

determined in accordance with

paragraph (c) of this section, with

respect to any educational program the

panel member is asked to review.

(c) Panel members are considered

employees of the U.S. Department for

the purposes of conflicts of interest

analysis and are subject to the

provisions of 18 U.S.C. 208, 5 CFR

2635.502, and the Department’s policies

used to implement those provisions.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

Subpart C—The Expert Panel Review

Process

§ 701.20 How does an expert panel

evaluate programs?

(a) Each panel member shall—

(1) Independently review each

program based on the criteria in

§ 701.22;

(2) Provide written comments based

on an analysis of the strengths and

weaknesses of the program according to

the criteria;

(3) Participate in site visits or other

verification activities, if appropriate;

and

(4) Participate in a meeting of the

expert panel, if appropriate, to discuss

the reviews.

(b) A panel may not eliminate an

educational program from consideration

based solely on the fact that the program

does not have one specific type of

supporting data, such as test scores.

(c) Each expert panel shall make a

recommendation to the Secretary as to
61432 Federal Register / Vol. 62, No. 221 / Monday, November 17, 1997 / Rules and Regulations

whether the program is exemplary,

promising, or neither.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.21 What is the difference between an

exemplary and a promising program?

(a) In determining whether an

educational program should be

recommended as exemplary or

promising, the panel shall consider—

(1) Whether, based on empirical data,

the program is effective and should be

designated as exemplary; or

(2) Whether there is sufficient

evidence to demonstrate that the

program shows promise for improving

student achievement and should be

designated as promising.

(b) The Secretary relies upon the

judgment and expertise of peer

reviewers, as established in § 701.11, to

determine the nature and extent of

evidence required to distinguish

between promising and exemplary

programs and to apply the four criteria

established in § 701.22, and their own

individual factors under each criterion

in making this determination.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

§ 701.22 What criteria are used to evaluate

programs for exemplary or promising

designation?

The Secretary establishes the

following evaluation criteria for expert

panels to use in determining whether an

educational program should be

recommended as exemplary, promising,

or neither:

(a) Evidence of success.

(b) Quality of the program.

(c) Educational significance.

(d) Replicability.

(Authority: 20 U.S.C. 6011(i)(2)(B)(iii) and

(E), 6041(d))

[FR Doc. 97–30051 Filed 11–14–97; 8:45 am]

BILLING CODE 4000–01–P