Vol. 78 Tuesday, 

No. 156 August 13, 2013 

Part II 

Department of Education 

34 CFR Parts 75 and 77 

Direct Grant Programs and Definitions That Apply to Department 

Regulations; Final Rule 


ehiers on DSK2VPTVN1PROD with RULES_2
49338 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

DEPARTMENT OF EDUCATION 

34 CFR Parts 75 and 77 

[Docket ID ED–2012–OII–0026] 

RIN 1890–AA14 

Direct Grant Programs and Definitions 

That Apply to Department Regulations 

AGENCY: Department of Education. 

ACTION: Final regulations. 

SUMMARY: The Secretary amends the 

regulations in the Education Department 

General Administrative Regulations 

(EDGAR) to: improve the Department’s 

ability to evaluate the performance of 

discretionary grant programs and 

grantee projects; support, where 

appropriate, projects supported by 

evidence of effectiveness; review grant 

applications using selection factors that 

promote the Secretary’s policy 

objectives related to project evaluation, 

sustainability, productivity, and strategy 

to scale; and reduce burden on grantees 

in selecting implementation sites, 

implementation partners, or evaluation 

service providers for their proposed 

projects. These amendments will allow 

the Department to be more effective and 

efficient when selecting grantees in 

discretionary grant competitions, 

provide higher-quality data to the 

Congress and the public, and better 

focus applicants on the goals and 

objectives of the programs to which they 

apply for grants. 

DATES: These regulations are effective 

September 12, 2013. 

FOR FURTHER INFORMATION CONTACT: Erin 

McHugh, U.S. Department of Education, 

400 Maryland Avenue SW., Room 

4W319, LBJ, Washington, DC 20202. 

Telephone: (202) 401–1304 or by email: 

erin.mchugh@ed.gov. 

If you use a telecommunications 

device for the deaf (TDD) or a text 

telephone (TTY), call the Federal Relay 

Service (FRS), toll free, at 1–800–877– 

8339. 

SUPPLEMENTARY INFORMATION:

Executive Summary 

Purpose of This Regulatory Action: 

The purpose of this action is to amend 

EDGAR to improve the quality and 

effectiveness of grant-making decisions 

and reduce the burden on applicants 

and grantees. These amendments will 

help align the Department’s grant 

process with the Secretary’s policy 

objectives and allow Department 

programs to design grant competitions 

to achieve those objectives. These 

amendments will also increase the 

flexibility for applicants and grantees to 

both reduce burden on applicants and 

grantees and improve the quality of data 

generated and reported by grantees. The 

authority to amend EDGAR is 20 U.S.C. 

1221e–3 and 3474. 

Summary of the Major Provisions of 

This Regulatory Action: These rules: 

1. Allow the Secretary, in the 

application notice for a grant 

competition, to establish performance 

measurement requirements for grantees 

(New § 75.110); 

2. Revise requirements for project 

evaluations submitted to the 

Department by grantees and for 

continuation of a multi-year project to 

incorporate performance measurement 

requirements for grantees (Amended 

§§ 75.253 and 75.590); 

3. Authorize grantees to procure 

implementation sites without regard to 

the procurement procedures in parts 74 

and 80 and use small purchase 

procedures to procure evaluation 

service providers and providers of 

services that are essential to the success 

of a proposed grant, provided the site or 

service provider is identified in the 

grant application (New § 75.135); 

4. Allow the Secretary, through an 

announcement in the Federal Register, 

to authorize grantees under particular 

programs to award subgrants to directly 

carry out programmatic activities. The 

possible subgrantees and the program 

activities they would carry out must be 

identified and described in the grantees’ 

applications or selected through a 

competitive process set out in 

subgranting procedures established by 

the grantee (New § 75.708); 

5. Add one new selection criterion 

and amend two existing selection 

criteria that the Department may use to 

evaluate applications. The new criterion 

is used to assess the extent to which a 

proposed project could be brought to 

scale. The amendments to the general 

selection criteria also include the 

addition of five new factors to 

§ 75.210(h) (Quality of the Project 

Evaluation) that could be used to assess 

how well a proposed project evaluation 

would produce evidence about the 

project’s effectiveness. We also revised 

one factor and added six new factors to 

§ 75.210(c) (Quality of the Project 

Design) (Amended §§ 75.209 and 

75.210); 

6. Authorize program offices to 

consider the effectiveness of proposed 

projects under a new priority that could 

be used as an absolute, competitive 

preference, or invitational priority (New 

§ 75.266); and 

7. Allow the Secretary to fund data 

collection periods after the end of the 

substantive work of a project so that 

project outcomes could be assessed 

using data from the entire project period 

(Amended §§ 75.250 and 75.251). 

Costs and Benefits: The Secretary 

believes that these regulations do not 

impose significant costs on entities that 

would receive assistance through 

Department of Education programs. Any 

costs imposed on applicants by these 

regulations are limited to the paperwork 

burden involved in preparing an 

application and keeping records needed 

to track progress on meeting 

performance measures. The benefits of 

implementing them outweigh any costs 

incurred by applicants. 

The benefits of the amendments in 

these regulations for the use of 

performance measures, baseline data, 

and performance targets established by 

the Department or by grantees 

themselves are that the Department 

would collect meaningful data that 

could be used to select applications for 

funding and assess the success of 

individual projects. The Department 

will also use these data to report to the 

Congress and the Office of Management 

and Budget (OMB) on the success of the 

grant programs in achieving their 

legislative objectives. The Department’s 

strengthened capability to evaluate the 

success of Department programs should 

help improve the effectiveness of those 

programs and improve transparency 

about how public funds are expended, 

without imposing additional costs on 

grantees or other parties. 

Additionally, these final regulations 

add a new § 75.135 and amend § 75.708 

regarding subgranting and competition 

exceptions. These sections will reduce 

costs, increase benefits, and potentially 

improve project quality by removing 

barriers that impede grantees from 

working with, either through a contract 

or a subgrant, implementation partners 

and service providers identified in 

funded applications. These final 

regulations will relieve grantees of the 

costs of administering competitions 

without reducing accountability or 

increasing the risk of improper use of or 

accounting for grant expenditures. 

These regulations also provide the 

Department with greater flexibility in 

conducting grant competitions by 

allowing for the use of selection criteria 

that: (1) Are closely aligned with 

program objectives and priorities, and 

(2) promote policy objectives such as 

project evaluation, sustainability, 

productivity, and strategy to scale. 

Thus, these amendments will benefit 

applicants, the Department, and the 

public by allowing the Secretary to 

establish selection criteria that are 

concise and closely aligned with the 

goals and objectives of a particular grant 

competition and are focused more 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49339 

closely and coherently on the intended 

programmatic and policy outcomes. 

Because the new, more specific criteria 

will be used instead of the more generic 

criteria currently in EDGAR, the 

regulations will generate these benefits 

without increasing the costs for 

applicants, grantees, or the Department. 

On December 14, 2012, the Secretary 

published a notice of proposed 

rulemaking (NPRM) for these 

amendments in the Federal Register (77 

FR 74392). 

This document includes three 

revisions from the NPRM. We discuss 

changes from the NPRM in greater detail 

in the Analysis of Comments and 

Changes. Specifically, we have revised 

§ 75.135 to make it clear that grantees 

may exercise the competition exception 

when procuring services from entities in 

instances where the entity is identified 

in the funded application. We have also 

revised the definitions of ‘‘strong 

evidence of effectiveness’’ and 

‘‘moderate evidence of effectiveness’’ in 

§ 77.1 to clarify that only studies with 

unfavorable outcomes that were so 

substantial as to call into question the 

potential effectiveness of the proposed 

project would disqualify the evidence 

from meeting the condition in the 

definitions. We do not discuss minor 

technical or editorial changes. 

Public Comment: In response to our 

invitation in the NPRM, 38 parties 

submitted comments on the proposed 

regulations. We group major issues 

according to subject. Analysis of 

Comments and Changes: An analysis of 

the comments and of any changes in the 

regulations since publication of the 

NPRM follows. 

Information Regarding Performance 

Measurement—§ 75.110 

Comments: One commenter agreed 

with the proposed amendment to create 

§ 75.110, stating that establishing 

performance measurement requirements 

in a notice inviting applications for a 

competition would both increase the 

likelihood of obtaining more robust data 

on grantee performance and increase the 

number of rigorous evaluation studies in 

the field. 

Some commenters agreed with the 

proposed amendment but requested 

clarification on key points. One 

commenter expressed concerns that 

performance measures beyond those 

related to student achievement would 

not be considered. Another commenter 

suggested differentiating between 

performance measures and outcomes 

data, indicating that performance 

measures help grantees continuously 

improve their projects, while outcomes 

data are useful in evaluating the success 

of their projects. The commenter also 

suggested developing a list of indicators 

for applicants to use when defining and 

adopting their own measures of success. 

Another commenter noted the 

importance of aligning performance 

measures with program goals and taking 

into account the size and scope of each 

proposed project when evaluating the 

quality of the performance measures. 

Two commenters expressed support 

for the proposed amendment, but they 

suggested that special considerations be 

made for applicants with limited 

capacity to analyze and collect data and 

recommended that these applicants be 

permitted to use grant funds and 

additional planning time in order to 

meet the performance measurement 

requirement. 

One commenter expanded on the idea 

of allowing grantees to use grant funds 

for performance measurement by 

suggesting the inclusion of a provision 

for performance measurement expenses 

in part 75, subpart F. 

Discussion: We agree with the 

commenter that the proposed 

amendment on performance 

measurement will strengthen the quality 

of data provided by grantees on their 

projects. Grantees typically report 

performance measures specific to their 

projects. Because those performance 

measures vary significantly, even among 

projects supported under one 

competition, it is very difficult for the 

Department to track the overall success 

of a program without performance 

measures that apply to all projects 

funded under a particular program’s 

competition. By requiring standard 

performance measurements in a notice 

inviting applications, and by retaining 

the applicant’s ability to set additional 

project-specific measures, we are more 

likely to obtain data that are meaningful 

both to evaluate the overall program and 

the quality of each grant funded under 

a competition for that program. This 

allows us to more effectively measure 

each program’s effectiveness, as 

required by the Government 

Performance and Results Act of 1993 

(GPRA), and will provide the 

Department a mechanism to hold 

grantees accountable for their 

performance and their success in 

implementing their grants. 

We also agree that it is important to 

consider performance measures beyond 

those related to student achievement. 

While Department programs share a 

common focus on improving academic 

achievement for students, many 

programs focus on factors that may not 

directly relate to students, such as 

professional development for teachers 

or engaging parents and family members 

in the school community. With this in 

mind, § 75.110 gives the Department 

flexibility to set standard performance 

measurement requirements for all types 

of programs, not just programs that 

measure student performance, while 

continuing to invite applicants to set 

additional project-specific measures. 

Therefore, this regulation will allow the 

use of a variety of performance 

measures. While we explicitly require 

that grantees collect and report on 

GPRA-mandated performance measures, 

which may be focused on student 

achievement, grantees retain the 

discretion to establish additional 

performance measures uniquely related 

to the objectives of their proposed 

projects. 

We recognize that some grantees may 

have limited capacity to meet the 

performance measurement requirement 

and acknowledge that this may appear 

to disadvantage small local educational 

agencies (LEAs), rural LEAs, community 

colleges, and small nonprofit 

organizations in particular. We suggest 

that when preparing an application, an 

applicant assess its needs and develop 

its proposed budget accordingly. For 

example, an applicant that lacks 

sufficient resources to collect and 

analyze data on its own may request 

funding to obtain data collection and 

evaluation services from external 

providers. Neither current regulations 

nor these new regulations prohibit an 

applicant from including in its project 

budget support for data collection and 

analysis. If an applicant decides to 

procure these services from a contractor, 

the applicant must meet the 

procurement requirements authorized 

under new § 75.135(b), including 

identifying the proposed contractor in 

the application. See discussion of 

§ 75.135 under Procurement and 

Subgrant Process for Entities Named in 

Applications. 

We agree that performance measures 

should be aligned with the goals of the 

Department program and that useful 

measures will take into account the 

expected scope and size of each 

proposed project. Regarding the 

comment suggesting that we amend 

§ 75.110 to list specific program-aligned 

performance measures, we do not think 

§ 75.110 is the most appropriate 

platform for enumerating specific, 

program-aligned performance measures. 

The purpose of the amendments to 

§ 75.110 is to permit the Department to 

establish performance measures in the 

notice inviting applications and to 

establish standard performance 

measurement requirements that all 

applicants for a particular Department 

program must use, while still allowing 


ehiers on DSK2VPTVN1PROD with RULES_2
49340 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

applicants the flexibility to suggest 

other, more project-specific, 

performance measures. Also, given the 

variety of programs to which these 

regulations apply, we do not think it is 

appropriate to prescribe a list of 

indicators in the regulation. We think 

that Department program officials are in 

the best position to establish 

appropriate performance measurement 

indicators for particular grant 

competitions and need the discretion to 

change the measures as the program 

evolves. In addition, more detailed 

information on indicators for a 

particular Department program will be 

provided in each notice inviting 

applications than can be provided 

through the use of generic performance 

measures listed in a regulation. 

Finally, we agree that performance 

measures and outcomes data are two 

separate terms, but we want to clarify 

that both are necessary and important to 

the continuous improvement and 

success of a grant. ‘‘Performance 

measure’’ is defined as any quantitative 

indicator, statistic, or metric used to 

gauge program or project performance. 

Thus, a performance measure is a unit 

for measuring outcome data. By 

selecting the appropriate measures, we 

can ensure that the outcome data 

collected by grantees are relevant to 

program performance and that the 

Department has the data needed to 

report program performance information 

to the Congress under GPRA. Further, 

we expect that grantees will collect 

outcome data not only at the end of a 

project, but in the interim as well. 

Formative outcome data are collected 

and analyzed throughout the project 

period and are useful for the continuous 

improvement of the project, while 

summative outcome data are collected 

and analyzed at the end of the project 

period and are useful when evaluating 

the project’s overall impact. 

Performance measures are expected to 

inform both types of outcome data. 

Changes: None. 

Comments: Some commenters 

expressed concern that the proposed 

changes to § 75.110 would unfairly 

disadvantage small, rural, and 

economically disadvantaged LEAs that 

may have limited access to data or 

limited resources to collect data. 

Commenters requested that we clarify 

how performance measures, baseline 

data, performance targets, and 

performance data will be set, and by 

whom they will be set. 

Some commenters expanded on this 

suggestion, requesting that we solicit 

LEAs or field experts for input on 

defining performance measures, 

baseline data, performance targets, and 

performance data. 

Some commenters did not agree with 

the proposed change and expressed 

concern that it would prove too costly 

and burdensome for grantees. One 

commenter did not agree with the 

proposed regulation because, according 

to the commenter, the performance 

measurement requirement would be too 

costly and would not ultimately 

improve services for students. 

Discussion: While we recognize that 

all applicants may not have equal 

resources to collect and report 

performance measurement data prior to 

receiving a grant, each applicant should 

assess its capacity when writing its 

application and develop its budget 

proposal accordingly. An applicant may 

include funds in its project budget to 

support data collection and analysis. 

Applicants can use the exception in 

§ 75.135 to procure the needed expertise 

to collect the appropriate data and 

evaluate the outcomes under the 

measures established for the 

competition. 

Performance measures must be 

aligned to the goals of the program, 

which are based on the law and the 

Department’s regulations and policies. 

As such, performance measures for a 

particular program are generally set by 

the Department officials responsible for 

the program. We appreciate the 

opinions of LEA representatives and 

field experts and encourage interested 

parties to comment on notices of 

proposed priorities; however, 

performance measures must ultimately 

align with program goals so the 

Department can measure the 

effectiveness of its programs. 

Gathering reliable and valid 

information on project outcomes is an 

integral part of determining which 

processes, products, strategies, and 

practices are working for students and 

which are not. While these final 

regulations may require grantees to use 

a portion of project funds on measuring 

performance, we consider it to be an 

important investment that will 

ultimately lead grantees to more 

successful results and thereby improve 

results for students and help the 

Department report more meaningful 

information to the Congress on the 

benefits of the Department’s programs, 

as required under GPRA. 

Changes: None. 

Procurement and Subgrant Process for 

Entities Named in Applications— 

§ 75.135 Competition Exception for 

Implementation Sites, Implementation 

Partners, or Evaluation Service 

Providers; and § 75.708 Prohibition on 

Subgrants 

Comments: Several commenters 

expressed strong support for the 

proposed changes to §§ 75.135 and 

75.708. Many of these commenters 

recommended that the Secretary allow 

grantees to make subgrants without 

approval from the Secretary. The 

commenters stated that individual 

grantees are better positioned than the 

Secretary to determine whether they 

need to make subgrants to carry out 

their projects, what types of entities may 

receive subgrants, and how the 

subgrants would be made. One 

commenter suggested revising the 

regulation to provide that subgrants 

should always be allowed unless the 

Department decides to prohibit it in 

certain circumstances. The commenter 

thought that formulating the regulation 

in this manner would encourage publicprivate partnerships while preserving 

the Secretary’s authority to prohibit 

subgranting when necessary. One 

commenter argued that providing direct 

authority to grantees to identify and 

administer subgrants would reduce the 

administrative burden of seeking 

approval from the Department. Another 

commenter indicated this flexibility is 

necessary to mitigate implementation 

delays in instances when the 

publication of the notice inviting 

applications in the Federal Register is 

not timely. The commenter noted that 

State educational agencies (SEAs) 

particularly need this flexibility and 

suggested adding a new paragraph that 

allows a State’s Chief School Officer to 

determine the types of entities that may 

receive subgrants and the procedures for 

making subgrants within the State. 

Some commenters also recommended 

that the regulation specifically identify 

SEAs, institutions of higher education 

(IHEs), and nonprofit organizations as 

types of entities that may be awarded a 

subgrant. One commenter proposed 

adding for-profit entities as a type of 

entity that may be awarded a subgrant. 

The commenter noted the inclusion of 

for-profit entities is particularly 

important considering that many grants 

are designed around a product or 

service that will be provided by a forprofit entity and emphasized that 

grantees should identify partners or 

providers based on the needs of their 

projects without consideration for the 

corporate status of a partner or provider. 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49341 

Discussion: We appreciate the 

commenters’ support for the changes 

regarding subgranting. However, we 

decline to make the revisions suggested 

because it is prudent and necessary for 

the Department to maintain control over 

when § 75.708 is used. The Department 

must ensure that subgrants are only 

authorized and used in a manner and 

under circumstances that are consistent 

with the requirements and purposes of 

authorizing statutes. This objective can 

only be met if the Department retains 

control over authorizing the grantee’s 

use of subgrants. We note however, that 

under § 75.708 the Secretary will 

indicate through an announcement in 

the Federal Register whether subgrants 

can be made to entities identified in an 

approved application or can be made to 

entities selected through a competitive 

process included in the grantee’s 

application. Thus, in lieu of requiring a 

grantee to identify all entities that will 

receive subgrants in the approved 

application, the Secretary may allow a 

grantee to use a competitive process that 

it describes in the approved application 

to determine the entities that will 

receive subgrants. 

With regard to the comment 

suggesting that for-profit entities be 

allowed to receive subgrants, we note 

that grantees already have the authority 

to enter into contracts with for-profit 

entities. Additionally, we think that 

procurements are the appropriate 

vehicle for grantees to use to secure 

goods and services from for-profit 

entities. For that reason, we decline to 

revise the regulations to allow subgrants 

to for-profit entities. However, we agree 

that there may be circumstances under 

which a product or service provided by 

a for-profit entity is integral to 

implementation of a project. As a result, 

we have revised § 75.135 to include 

entities that will provide a product or 

service that would, if removed from the 

grant, have a detrimental effect on the 

successful implementation of the grant. 

Changes: We have revised § 75.135(b) 

to clarify that when entering into a 

contract for data collection, data 

analysis, evaluation services, or 

essential services, as defined in 

paragraph (f) of this section, an 

applicant may select a provider using 

the informal, small-purchase 

procurement procedures in 34 CFR 

80.36(d)(1), regardless of whether that 

applicant would otherwise be subject to 

that part or whether the evaluation 

contract would meet the standards for a 

small purchase order, if— 

(1) The contract is with the data 

collection, data analysis, evaluation 

service, or essential service provider; 

(2) The data collection, data analysis, 

evaluation service, or essential service 

provider that the applicant proposes to 

use is identified in the application for 

the grant; and 

(3) The data collection, data analysis, 

evaluation service, or essential service 

provider is identified in the application 

in order to meet a statutory, regulatory, 

or priority requirement related to the 

competition. 

We have also added paragraph (f) to 

state that, for the purposes of this 

section, essential service means a 

product or service directly related to the 

grant that would, if not provided, have 

a detrimental effect on the grant. 

Comment: One commenter 

recommended that the Department 

broadly implement the authority to 

allow subgrants. The commenter 

suggested that the Ready To Learn (RTL) 

program is particularly well-suited for 

the use of subgrants given that 

subrecipients of RTL grantees are often 

responsible for the development and 

production of educational programming 

that is integral to the grant. According 

to the commenter, subgranting will both 

continue to ensure close monitoring of 

funds and foster close collaboration that 

will further project objectives. 

Discussion: As noted elsewhere in our 

responses to comments in this section, 

the Department cannot establish a 

universal rule allowing the use of 

subgrants because program statutes 

define differently whether subgrants 

may be used and in what circumstances 

they may be used. Additionally, 

subgrants, even when not prohibited by 

a statute, may not always be appropriate 

for a particular program. 

Changes: None. 

Comments: Two commenters stated 

that, although the proposed 

amendments in § 75.135 that permit 

exceptions to the procurement 

procedures are beneficial and they 

support them as written, some grantees 

may be subject to State or local laws that 

require specific procurement 

procedures. One commenter explained 

that such State laws and requirements 

negate the benefits of the proposed 

amendment and suggested revising the 

language to minimize this consequence. 

Discussion: The proposed 

amendments in § 75.135 relax 

requirements that otherwise apply to 

grantees under parts 74 and 80. They do 

not, as the commenters noted, eliminate 

a grantee’s responsibilities to comply 

with their own procurement 

requirements and State and local laws 

that exceed those required by 

Department regulations. State 

governments may follow their own 

procurement requirements, subject only 

to the requirement that they must 

include in their contracts all clauses 

required by Federal statutes, Executive 

orders, and implementing regulations. 

We note that some SEAs have adopted 

some of the requirements in § 80.36. 

Other State, local, and Indian tribal 

government grantees must comply with 

the minimum requirements in 34 CFR 

80.36. Non-governmental grantees must 

comply with the minimum procurement 

requirements in 34 CFR 74.41–74.48. 

These final regulations do not change 

other applicable financial management 

and procurement requirements in 34 

CFR parts 74 and 80, including those 

that require State agencies to follow 

their own procurement policies and 

procedures (34 CFR 80.36(a)) or that 

generally require grantees to maintain 

procurement procedures that prohibit 

conflicts of interest. The continued 

applicability of these requirements in 

parts 74 and 80 of EDGAR is crucial to 

ensuring accountability for the use of 

Federal funds by grantees. 

Changes: None. 

Comments: Many commenters 

expressed strong support for the 

revisions to § 75.135 and § 75.708 and 

agreed that the exemption from 

procurement requirements for selecting 

implementation sites or partner entities, 

and the use of small purchase 

procedures to select evaluation service 

providers identified in grant 

applications will improve the 

implementation and outcome of grants 

funded by the Department. One 

commenter noted that the proposed 

amendment would be particularly 

beneficial to SEAs because it will 

support more efficient use of resources 

and ensure grant activities are 

implemented on a timely basis. 

One commenter expressed general 

support for the proposed amendment 

but recommended revising it to include 

a competition exception for products or 

services identified in the application 

that are unique and essential, meaning 

that the use of an alternative product or 

service would be detrimental to the 

implementation of the project. The 

commenter also suggested adding a 

condition that allows the use of 

simplified procurement procedures for 

products or services that are not core to 

the implementation of the project when 

the costs of conducting a competition 

would be excessive in relation to the 

amount of grant funds that would be 

awarded in the contract. The commenter 

also stated that local and State 

procurement requirements would still 

apply, so these revisions would 

eliminate an additional Federal 

requirement in these instances but 


ehiers on DSK2VPTVN1PROD with RULES_2
49342 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

would not remove those existing 

protections. 

Discussion: We appreciate the 

commenters’ support for the proposed 

amendment to § 75.135 and agree it will 

be beneficial to grantees. With regard to 

the recommendation that we revise the 

regulation to allow grantees to use the 

competition exception for products and 

services identified in the application 

that are unique, we recognize that grant 

projects may be designed around such 

products or services. Therefore, we have 

revised paragraph (b) of § 75.135 to 

include an exception for entities that 

will provide a product or service that 

would, if not provided, have a 

detrimental effect on the grant. 

However, we decline to revise the 

regulation to reduce the competition 

requirements for products or services 

that are not identified in the application 

or core to the implementation of the 

project because we do not think such a 

revision is consistent with the intent of 

the change. We do not intend this 

change to limit competition in instances 

when full and open competition is 

practical. We note, however, that the 

simplified acquisition threshold already 

provides grantees some flexibility in 

competition requirements for 

procurements under $100,000. OMB has 

proposed to raise this threshold to 

$150,000 in its proposed amendments to 

title 2 of the CFR. See the OMB 

proposal, Reform of Federal Policies 

Relating to Grants and Cooperative 

Agreements; Cost Principles and 

Administrative Requirements (Including 

Single Audit Act), published on Friday, 

February 1, 2013, at 78 FR 7282. Thus, 

regardless of the exemption authorized 

in these final regulations, applicants 

will have greater flexibility to use small 

purchase procedures when the 

procurement threshold is raised. 

Therefore, it is not necessary or 

appropriate to make the changes 

suggested. 

Also, as noted earlier, these 

regulations relax certain procurement 

requirements that otherwise apply to 

grantees under parts 74 and 80. Grantees 

should be aware, however, that these 

amendments do not eliminate a 

grantee’s responsibilities to comply with 

its own procurement requirements and 

State and local laws to the extent that 

those requirements and laws exceed the 

minimum requirement in parts 74 and 

80. 

Changes: We have revised § 75.135(b) 

to clarify that when entering into a 

contract for data collection, data 

analysis, evaluation services, or 

essential services, as defined in 

paragraph (f) of this section, an 

applicant may select a provider using 

the informal, small-purchase 

procurement procedures in 34 CFR 

80.36(d)(1), regardless of whether that 

applicant would otherwise be subject to 

that part or whether the evaluation 

contract would meet the standards for a 

small purchase order, if— 

(1) The contract is with the data 

collection, data analysis, evaluation 

service, or essential service provider; 

(2) The data collection, data analysis, 

evaluation service, or essential service 

provider that the applicant proposes to 

use is identified in the application for 

the grant; and 

(3) The data collection, data analysis, 

evaluation service, or essential service 

provider is identified in the application 

in order to meet a statutory, regulatory, 

or priority requirement related to the 

competition. 

We have also added paragraph (f) to 

state that, for the purposes of this 

section, essential service means a 

product or service directly related to the 

grant that would, if not provided, have 

a detrimental effect on the grant. 

Comment: None. 

Discussion: Based on the comments 

received, we revised § 75.135(b) to 

expand the circumstances under which 

grantees may use small purchase 

procedures. We have also revised 

§ 75.135(c) to require applicants, who 

utilize this exception, to certify that 

they followed the small purchase 

procedures. The small purchase 

procedures, while not as extensive as 

the full procurement requirements set 

out in Parts 74 and 80, provide 

important protections to the Federal 

interest in the prudent and allowable 

use of grant funds. By requiring 

applicants that utilize this exception to 

certify that they followed the small 

purchase procedures, we provide 

increased assurance that the protections 

to Federal grant funds offered by those 

procedures are, in fact, in place. 

Changes: We have revised the 

certification requirement in § 75.135(c) 

to require grantees that relied on the 

exceptions of § 75.135(b) to certify that 

they used small purchase procedures to 

obtain a product or service if the 

applicant relied on the exception 

authorized in this section to procure the 

product or service. 

Amendments Relating to Evidence— 

§ 75.210(c) Quality of the Project 

Design, § 75.210(h) Quality of the 

Evaluation, § 75.266 Consideration for 

Applications Supported by Strong or 

Moderate Evidence, § 75.590 Evaluation 

by the Grantee, and § 77.1 Definitions 

That Apply to All Department 

Programs 

Comments: Many commenters 

strongly supported the proposed 

definitions of ‘‘strong evidence of 

effectiveness’’ and ‘‘moderate evidence 

of effectiveness’’ in § 77.1(c). One 

commenter applauded the Department 

for expanding the focus on evidencebased practices and stated that this 

effort will result in higher quality grant 

applications and outcomes. One 

commenter noted that the strength of 

evidence used to support a project’s 

effectiveness should be rigorous, 

objective, and pertinent to the goals of 

the project. 

A few of these commenters suggested 

amendments or clarifications to these 

definitions. One commenter 

recommended clarifying that a study, in 

order to meet these definitions, need 

only meet the appropriate standards 

outlined in the What Works 

Clearinghouse (WWC) and not 

necessarily be reviewed by the WWC or 

posted on the WWC Web site. Three 

commenters noted that these 

definitions, as proposed, present a risk 

that a study could meet the definition 

even if the effects are: (1) On trivial or 

developer-created outcomes; (2) 

artificially inflated or likely a result of 

chance; or (3) so small in size as to be 

of little importance. These commenters 

recommended revising the definitions to 

clarify that the study must be of 

sufficient duration and sample size to 

represent a valid test and to require that 

the study find a significant favorable 

outcome based on a measure of clear 

policy importance. One of these 

commenters further suggested requiring 

that the study have a substantial and 

important effect on improving student 

achievement or student growth, closing 

achievement gaps, decreasing dropout 

rates, increasing high school graduation 

rates, or increasing college enrollment 

and competition rates. The commenters 

stated that such changes avoid the 

loophole of classifying programs that 

lack policy or practical importance as 

evidence-based programs. 

Discussion: We agree with the 

commenters that the evidence used to 

support a project’s effectiveness should 

be objective and pertinent to the goals 

of the project. However, we also 

recognize that at the various stages of a 

proposed project’s development, 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49343 

1What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

currently found at the following link: http://

ies.ed.gov/ncee/wwc/DocumentSum.aspx?sid=19. 

2See definition for full description of the two 

possible conditions. 

different types of evidence are available 

to assess the effectiveness of a project. 

That is why we include definitions for 

four levels of evidence: ‘‘Strong 

evidence of effectiveness,’’ ‘‘moderate 

evidence of effectiveness,’’ ‘‘evidence of 

promise,’’ and ‘‘strong theory.’’ We 

establish these definitions in order to 

develop an understanding for applicants 

of what is required to meet each level 

of evidence. Combined, these four levels 

of evidence allow the Department to 

support effective projects (strong or 

moderate evidence of effectiveness) or 

projects that have a high potential to be 

effective (evidence of promise or strong 

theory). 

We appreciate the commenters’ 

concerns regarding the definitions of 

‘‘strong evidence of effectiveness’’ and 

‘‘moderate evidence of effectiveness.’’ 

However, we do not consider the 

proposed changes necessary as the 

definitions already safeguard against the 

risks identified by the commenters. 

With regard to ensuring the outcome 

has policy and practical importance, 

both of these definitions refer to the 

effect on a ‘‘relevant outcome.’’ The 

definition of ‘‘relevant outcome’’ 

explains that it is the ultimate outcome 

of the proposed process, product, 

strategy, or practice and should be 

consistent with the specific goals of the 

Department program. Thus, the 

references to ‘‘relevant outcome’’ would 

not allow studies that only show effects 

on trivial outcomes to meet the 

requirements of the definitions. 

Further, the WWC Procedures and 

Standards Handbook 1 explains that a 

study that does not include a valid or 

reliable outcome measure, or does not 

provide adequate information to 

determine whether it uses an outcome 

that is valid or reliable, would not meet 

WWC Evidence Standards. Because the 

WWC Evidence Standards are 

incorporated in these definitions, a 

study that only includes an outcome 

measure created by the evaluator or 

developer with weak or no validity or 

reliability data would not meet the 

requirements of the definitions. 

With regard to requiring that a study 

has an adequate sample size to ensure 

that the effect is not artificially inflated, 

a result of chance, or so small it is of 

little importance, both definitions refer 

to the definitions of a ‘‘large sample’’ 

and a ‘‘multi-site sample.’’ In order for 

any study to meet the requirements of 

‘‘strong evidence of effectiveness,’’ that 

study would need to have used a ‘‘large 

sample’’ and a ‘‘multi-site sample.’’ 

Under the definition of ‘‘moderate 

evidence of effectiveness,’’ a study must 

meet, among other requirements, one of 

the following two conditions: (1) WWC 

Evidence Standards without 

reservations; or (2) WWC Evidence 

Standards with reservations.2 Although 

a small study that meets WWC Evidence 

Standards ‘‘without reservations’’ would 

meet one of these two conditions, a 

study that meets WWC Evidence 

Standards ‘‘with reservations’’ could 

only meet the requirements of 

‘‘moderate evidence of effectiveness’’ if 

it used a ‘‘large sample’’ and a ‘‘multisite sample.’’ We think the inclusion of 

‘‘large sample’’ and ‘‘multi-site sample’’ 

in the definitions of ‘‘strong evidence of 

effectiveness’’ and ‘‘moderate evidence 

of effectiveness’’ appropriately mitigate 

the risks identified by the commenters. 

With regard to ensuring that studies 

are of sufficient duration to meet the 

requirements, we note that the WWC 

Evidence Standards do not require a 

minimum study length. More 

importantly, because it is not clear that 

requiring a minimum study length is 

appropriate or necessary, we decline to 

revise the definitions to include such a 

requirement. 

Changes: None. 

Comment: None. 

Discussion: In § 77.1(c), we have 

revised the definitions of ‘‘moderate 

evidence of effectiveness’’ and ‘‘strong 

evidence of effectiveness’’ by adding the 

phrase ‘‘and overriding’’ to the second 

parenthetical in paragraphs (a) and (b) 

of both definitions. We add this phrase 

to clarify that only studies with 

unfavorable outcomes that were so 

substantial as to call into question the 

potential effectiveness of the proposed 

project would disqualify the evidence 

from meeting the condition in the 

definitions. 

Changes: We have revised the second 

parenthetical in paragraphs (a) and (b) 

of the definitions of ‘‘moderate evidence 

of effectiveness’’ and ‘‘strong evidence 

of effectiveness’’ to add the phrase ‘‘and 

overriding.’’ The parenthetical now 

reads ‘‘with no statistically significant 

and overriding unfavorable impacts on 

that outcome for relevant populations in 

the study or in other studies of the 

intervention reviewed by and reported 

on by the What Works Clearinghouse.’’ 

Comments: One commenter expressed 

concern that the proposed definitions 

related to evidence would stifle 

innovation and that providing special 

consideration for projects supported by 

evidence of effectiveness would limit 

the pool of applications for a 

competition. Another commenter stated 

that such consideration is not 

appropriate for all programs and the 

proposed amendment appears to be an 

attempt to turn all projects funded by 

the Department into Investing in 

Innovation (i3) projects. 

Discussion: The establishment of 

procedures to provide special 

consideration for projects supported by 

strong or moderate evidence of 

effectiveness provides the Secretary a 

mechanism to support effective projects 

and offer incentives to the field for 

building an evidence base on the 

effectiveness of the processes, products, 

strategies, and practices that are, or will 

be used, in education. However, as 

noted in our response to other 

comments in this discussion, we 

recognize that different types of 

evidence are available at the various 

stages of a proposed project’s 

development and that there are some 

areas where strong or moderate 

evidence of effectiveness is not yet 

available. As such, we agree that it 

would not be appropriate for the 

Secretary to consider whether a project 

is supported by strong or moderate 

evidence of effectiveness for all 

Department programs. The Secretary 

will only provide special consideration 

for projects supported by strong or 

moderate evidence of effectiveness in 

programs where such evidence exists or 

where such incentives are meaningful. 

When such levels of evidence do not 

exist, Department program officials may 

consider whether using ‘‘evidence of 

promise’’ or ‘‘strong theory’’ would be 

more appropriate for spurring 

innovation. Thus, we do not think 

providing special consideration in 

program areas that do have these levels 

of evidence would preclude robust 

competition or stifle innovation. 

Changes: None. 

Comments: Three commenters 

expressed support for the establishment 

of procedures to provide special 

consideration for projects supported by 

strong or moderate evidence of 

effectiveness. However, these 

commenters suggested clarifying that 

the special consideration be given to 

both existing projects supported by 

strong or moderate evidence of 

effectiveness and new projects that are 

proposing to adopt or adapt models 

supported by strong or moderate 

evidence of effectiveness. 

Discussion: The definitions of ‘‘strong 

evidence of effectiveness’’ and 

‘‘moderate evidence of effectiveness’’ in 

§ 77.1(c) indicate that the study needs to 

be of the effectiveness of the proposed 

process, product, strategy, or practice. 


ehiers on DSK2VPTVN1PROD with RULES_2
49344 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

These definitions also clarify that the 

study must overlap with the 

populations and settings in the 

proposed project. Therefore, a new 

project that is adopting the model of the 

process, product, strategy, or practice in 

the study meets the definitions. An 

applicant proposing a new process, 

product, strategy, or practice that is 

adapting or changing the model from 

what was in the study would need to 

explain how the study supports the 

adapted version of the model. Thus, a 

study may be used to support an 

adaptation of the model in the study so 

long as the applicant can provide a 

justification that the proposed project’s 

efficacy necessitates the adaptation, and 

is based upon the evidence and theory 

supported by the original study. 

Given the variety of programs to 

which these regulations apply, we do 

not think it is appropriate for the 

Department to determine at what single 

point an adaptation would make the 

study no longer credible for supporting 

the effectiveness of the proposed 

project. However, any programs 

providing special consideration for 

projects supported by strong or 

moderate evidence of effectiveness 

would provide instructions to 

applicants on the information they need 

to submit to demonstrate that they meet 

the applicable evidence level. 

Changes: None. 

Comment: One commenter 

recommended providing special 

consideration only for projects 

supported by strong or moderate 

evidence of effectiveness through the 

establishment of a separate competition, 

as opposed to ‘‘an across the board 

competitive preference.’’ 

Discussion: Section 75.266 authorizes 

the Secretary to establish a separate 

competition or provide a competitive 

preference for applications supported by 

strong or moderate evidence of 

effectiveness. We decline to limit the 

Secretary to providing special 

consideration through a separate 

competition because that process may 

not be appropriate for all Department 

programs. Given the variety of programs 

to which these regulations apply, it is 

important that we provide sufficient 

flexibility for determining which 

programs require, and how the Secretary 

would consider, strong or moderate 

evidence of effectiveness. 

Change: None. 

Comment: One commenter noted that 

projects funded by the Department 

should produce evaluations that meet 

defined standards but questioned 

whether the WWC Evidence Standards 

were appropriate considering the 

burden associated with conducting 

evaluations that are designed to meet 

those standards. Specifically, the 

commenter expressed concern that 

small or rural LEAs would not have the 

capacity to conduct such evaluations 

and that the Department’s use of 

selection factors promoting WWC 

Evidence Standards would favor large 

research organizations over LEAs. The 

commenter further stated that it is 

contradictory for the Department to use 

selection factors that promote 

evaluations more rigorous than required 

by the program. To address these 

concerns, the commenter recommended 

revising § 75.210(h)(2)(viii)–(x) to 

require that the proposed project 

evaluation meets the next level higher 

or equivalent level of the prior evidence 

supporting the proposed project’s 

effectiveness. 

Discussion: The WWC is an initiative 

of the Department’s Institute of 

Education Sciences (IES) and serves as 

a central and trusted source of scientific 

evidence for what works in education. 

Although we recognize the WWC 

Evidence Standards primarily refer to 

randomized controlled trial (RCT) and 

quasi-experimental design (QED) 

studies, we also note that these designs 

are the most rigorous and defensible 

methods for producing unbiased 

evidence of project effectiveness. 

We agree with the commenter that 

conducting project evaluations that are 

designed to meet the WWC Evidence 

Standards requires planning and 

resources. However, because an 

applicant may obtain an evaluation 

service provider to conduct the project 

evaluation through a contract and may 

include these activities and costs in its 

proposed project budget, the use of the 

factors in § 75.210(h)(2)(viii)–(x) would 

not inherently disadvantage an 

applicant that lacks the internal 

capacity to conduct such evaluations. 

We also note that § 75.210(h) (Quality of 

the Project Evaluation) is only one 

criterion among multiple criteria used to 

evaluate applications. The Department 

considers each program’s purpose, 

goals, and applicant pool when deciding 

which selection criteria and factors to 

use in a given year’s competition. By 

creating these factors under § 75.210(h) 

(Quality of the Project Evaluation), the 

Department has the option—not the 

obligation—to use them to encourage 

applicants to propose project 

evaluations that would meet WWC 

Evidence Standards. Consequently, the 

Department will use these factors when 

appropriate for a particular competition 

and will not use them when doing so 

would conflict with required program 

evaluations. 

We decline to replace these factors 

with a factor that would allow a 

proposed project evaluation to be the 

equivalent level of the prior evidence 

supporting the proposed project’s 

effectiveness. In general, to provide the 

public the greatest return on its 

investment, evaluations funded by the 

Department should build on prior 

research, as appropriate. Although we 

recognize the importance of replicating 

results of a past study, we think it is 

important for applicants to propose 

project evaluations that increase the 

level of evidence of the proposed 

project’s effectiveness, as appropriate. 

By providing the flexibility to select 

among the various factors under 

§ 75.210(h) (Quality of the Project 

Evaluation), the Department has the 

discretion to select factors that are 

appropriate for the areas of study and 

research goals for a particular program. 

Changes: None. 

Comments: Two commenters 

indicated they had no objections to the 

proposed changes to §§ 75.210, 75.266, 

and 77.1 regarding evidence of 

effectiveness and WWC Evidence 

Standards, but cautioned the 

Department to be prudent in their use in 

discretionary grant competitions. One 

commenter stated that lack of evidence 

should not be the sole rationale for 

deciding not to make a grant to a 

particular applicant and suggested that 

the new regulations should not be used 

to establish a high threshold for 

evidence of effectiveness in areas where 

the amount of evidence on existing 

practice is not strong, particularly in 

areas that are difficult to measure, such 

as school climate or efforts to reduce 

administrative burden or build capacity. 

Similarly, another commenter 

recommended that programs establish 

thresholds for evidence of effectiveness 

that are commensurate with the quantity 

and quality of existing evidence in the 

field. 

Discussion: We agree that the new 

regulations in §§ 75.210, 75.266, and 

77.1 regarding evidence of effectiveness 

should only be used when appropriate 

for a particular program. We are making 

changes to these regulations to support 

evidence-based grant making in areas 

where evidence exists and to provide 

incentives and opportunities to build 

the body of evidence of effectiveness in 

education. 

Changes: None. 

Comments: Several commenters noted 

the distinction between a ‘‘project’’ and 

a ‘‘strategy within a project.’’ The 

commenters stated that it might be more 

appropriate for the Department to 

evaluate the effectiveness of an 

individual strategy used by a grantee 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49345 

rather than attempt to measure the 

effectiveness of a project as a whole. 

The commenters suggested revising the 

proposed selection criteria to clarify that 

programs or strategies could be used 

when evaluating a project’s 

effectiveness. 

Discussion: An applicant may propose 

to evaluate different strategies within a 

project using different evaluation design 

methods. For example, an applicant 

may propose a pre-post analysis to 

assess progress of one strategy within its 

project and a more rigorous evaluation 

design for another strategy within its 

project. Despite the flexibility we allow 

an applicant in designing the proposed 

project evaluation, under § 75.590 

(Evaluation by the grantee), the entire 

project being supported by Federal 

funds must be evaluated. 

Changes: None. 

Comments: A few commenters stated 

that they could not support the 

inclusion of selection factors that 

consider evidence of effectiveness 

because it would competitively 

disadvantage certain types of applicants, 

including those with limited resources 

or those that serve student populations 

that have unique needs. 

Two commenters specifically 

suggested that the consideration of 

effectiveness would present a 

disadvantage to community colleges. 

One commenter discussed three 

challenges for community colleges that 

make it difficult for them to conduct 

rigorous evaluations. According to the 

commenter, rigorous evaluation designs 

(1) distract from community colleges’ 

missions to provide access to education 

for all students; (2) often require 

approval of an Institutional Review 

Board (IRB), and many community 

colleges have not established IRBs; and 

(3) require signed consent from 

participants, which creates additional 

complications and concerns regarding 

student access to educational programs 

or support services. Both commenters 

stated that most studies posted on the 

WWC Web site focus on K–12 education 

and that existing research around 

community colleges is insufficient for 

them to compete if factors related to 

evidence of effectiveness are used by the 

Department. To address this concern, 

one commenter recommended creating a 

special track of priority funding for 

empirical research on community 

colleges in all of the Department’s 

postsecondary programs. 

One commenter suggested modifying 

the new selection factors to exempt 

programs with ‘‘historical evidence of 

benefit to students.’’ The commenter 

stated that programs that equalize 

educational opportunity among lowincome, first-generation college students 

who, in large part, are from 

underrepresented groups, should not be 

required to use quantitative research to 

determine their effectiveness. 

One commenter stated that programs 

providing parental training and 

engagement services would be 

disadvantaged by selection factors 

related to evidence of effectiveness 

because such programs require a focus 

on individual parent and family needs. 

The commenter expressed concern that 

the use of these selection factors, or any 

special consideration given for evidence 

of effectiveness, would limit which 

entities could apply to a particular 

program without providing a clear 

benefit to children and their families. 

Another commenter suggested that the 

selection factors referring to ‘‘evidence 

of promise’’ and ‘‘strong theory’’ be the 

only selection factors related to 

evidence of effectiveness used for 

implementation-based grants. 

Discussion: As noted elsewhere in our 

response to comments in this 

discussion, we agree that the selection 

factors relating to evidence of 

effectiveness, whether they fall under 

§ 75.210(c) (Quality of the Project 

Design) or § 75.210(h) (Quality of the 

Project Evaluation), should only be used 

when appropriate for a particular 

program. We include these selection 

factors to support evidence-based grant 

making in areas where evidence exists 

and to provide incentives and 

opportunities to build the body of 

evidence of effectiveness in education. 

Because the Department has the 

discretion to select factors that are 

appropriate for the areas of study and 

research goals of a particular program, 

and therefore would not select factors 

that would require applicants to provide 

evidence of effectiveness in areas that 

have not been widely researched, we 

decline to remove these factors. 

Additionally, we do not think the 

amount of research for a particular area 

of education on the WWC Web site is a 

reason not to add these factors to the 

regulations. A study does not need to be 

reviewed by the WWC or posted on the 

WWC Web site to meet the WWC 

Evidence Standards. Department 

program officials could use research 

available on the WWC Web site or from 

other sources to inform their decision 

on whether these selection factors are 

appropriate for the particular program. 

Further, we note that these factors 

address evidence of effectiveness and 

evaluation of effectiveness at various 

levels. Two of the factors refer to 

grantees proposing evaluation designs 

that meet WWC Evidence Standards 

with or without reservations, but we 

also include two other factors that refer 

to ‘‘evidence of promise’’ and ‘‘strong 

theory.’’ Including four levels of 

evidence provides the Department the 

opportunity to consider the level of 

evidence available in the field for the 

types of projects to be funded by the 

relevant program and the capacity of 

potential applicants to design 

evaluations that would assess the 

effectiveness of a project at these 

different levels. 

With regard to the other issues raised 

by the commenters, we recognize that 

rigorous evaluation designs require 

grantees to compare individuals 

participating in the project to those who 

are not participating in the evaluation. 

However, requiring more rigorous 

evaluation designs does not contradict 

the educational mission of serving all 

students because evaluating the 

effectiveness of a particular intervention 

is necessary to understand which 

interventions effectively improve 

student outcomes. Although funds that 

support evaluation services cannot also 

support direct services to students, 

investing in evaluation allows entities to 

focus finite resources on only those 

processes, products, strategies, or 

practices that are most effective in 

improving student outcomes. Therefore, 

we do not think evaluating the 

effectiveness of a project using a 

rigorous design would impede an entity 

from carrying out its educational 

mission. Further, because an applicant 

may contract to obtain an evaluation 

service provider that has access to an 

IRB to conduct the project evaluation, 

and because the applicant may include 

these activities or activities related to 

accessing an independent IRB or 

establishing its own IRB to support the 

project evaluation and their costs in its 

proposed budget, we do not think 

applicants that lack their own IRBs are 

disadvantaged. Similarly, because under 

the Common Rule for the Protection of 

Human Subjects, an IRB can modify or 

waive requirements for written consent, 

and the costs for activities to obtain 

written consent from participants may 

be included in the proposed budget, we 

do not think a specific type of entity is 

inherently disadvantaged by the use of 

selection factors that encourage 

applicants to propose rigorous 

evaluations of their projects’ 

effectiveness. 

With regard to the comments 

recommending that these factors not be 

used for programs that historically have 

benefited students and that we only 

allow the use of ‘‘evidence of promise’’ 

and ‘‘strong theory’’ for implementation 

grants, we reiterate the importance of 

the Department supporting the 


ehiers on DSK2VPTVN1PROD with RULES_2
49346 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

improvement of information available to 

practitioners and policymakers about 

which practices work, for which types 

of students, and in which contexts. 

These selection factors support that goal 

by providing incentives to applicants for 

grants to build an evidence base on the 

effectiveness of the processes, products, 

strategies, and practices that are, or will 

be used, in education. 

Changes: None. 

Comments: A few commenters stated 

that the Department should not support 

evaluations using research designs that 

include control groups. Two 

commenters stated that experimental 

designs are unethical because they 

require grantees to withhold treatment 

from students in public education. 

Another commenter felt that it was 

inappropriate to deny services to 

students simply to accommodate 

research and evaluation, particularly 

when the Congress authorized and 

funded the program to provide services 

to students. One commenter further 

expressed concern that favoring such 

designs would provide an advantage to 

large research organizations over LEAs 

that lack the capacity to conduct such 

evaluations. 

In addition to concerns about placing 

students or teachers in ‘‘experimental’’ 

versus ‘‘control’’ groups, one commenter 

cited challenges regarding the use of 

RCTs. Specifically, the commenter 

noted that a pure control condition is 

rare because fidelity of implementation 

can significantly impact the 

effectiveness of a process, product, 

strategy, or practice. The commenter 

indicated that how well a particular 

process, product, strategy, or practice 

works depends on the conditions under 

which it is implemented, and the costs 

of observation and metrics to determine 

the fidelity of that implementation are 

significant. The commenter also noted 

that, because products and services are 

constantly changing and improving, the 

products or services are frequently no 

longer available in the format or version 

that was studied by the time an 

evaluation is complete. The commenter 

concluded that equal consideration of 

alternative study designs, such as 

regression discontinuity designs, is 

needed to ensure the Department does 

not limit the development, 

implementation, and evaluation of 

innovative projects. 

One commenter expressed concern 

that the selection factors that refer to 

‘‘evidence of promise’’ and ‘‘strong 

theory’’ would be used to make all 

implementation projects into 

randomized research projects with 

control groups. According to the 

commenter, these selection factors 

would skew successful applications 

toward projects conducting research 

studies and away from projects 

providing services to students and 

teachers. The commenter stated that to 

require such project evaluations is not 

consistent with legislative intent and 

would not result in improved student 

outcomes. Another commenter made a 

similar statement that the proposed 

amendments regarding evidence of 

effectiveness and evaluation should not 

be used for the TRIO programs because 

they would undermine the intent of the 

Higher Education Opportunity Act of 

2008. 

Discussion: We appreciate the 

commenters’ concerns about whether it 

is ethical to evaluate the effectiveness of 

a project using a randomized 

experimental design. In order to ensure 

ethical research, the Department, under 

the Common Rule for the Protection of 

Human Subjects in Research (34 CFR 

97), does not permit covered human 

subjects research to be initiated until it 

has been reviewed by an IRB and 

receives the Department’s protection of 

human subjects clearance. Although we 

recognize that conducting these types of 

evaluations can be difficult, we also 

recognize that random assignment of 

entities (students, teachers, schools, or 

other units of analysis) to a treatment or 

control group is the most effective way 

to eliminate plausible competing 

explanations for observed differences 

between treated and non-treated 

individuals or groups (i.e., the estimated 

treatment effect). Two common 

strategies used in randomized 

experiments in education that are 

designed to address this ethical concern 

are the use of a ‘‘wait-list’’ control group 

and the assignment of schools, rather 

than students, to the treatment group. 

Despite the challenges in conducting 

such evaluations, we consider it 

important to provide a mechanism for 

the Department, when appropriate, to 

use these selection factors to encourage 

grantees to conduct evaluations of their 

projects that will improve the 

information available to practitioners 

and policymakers about which 

processes, products, strategies, and 

practices work, for which types of 

students, and in which contexts. 

We disagree that the regulation would 

favor large research institutions over 

other types of applicants. Applicants 

that do not feel they possess adequate 

resources to carry out a rigorous 

evaluation of their proposed projects 

may contract with an evaluation service 

provider. Applicants can use the 

exception in § 75.135 to procure the 

needed expertise to implement a 

rigorous evaluation. 

We disagree with the comment that 

the use of selection factors referring to 

‘‘evidence of promise’’ and ‘‘strong 

theory’’ requires all implementation 

projects to become randomized research 

projects with control groups. We define 

‘‘strong theory’’ to mean a rationale for 

the proposed process, product, strategy, 

or practice that includes a ‘‘logic 

model’’ (as defined in § 77.1(c)). The 

development of a logic model and the 

associated rationale does not require a 

grantee to conduct a randomized 

experiment. Similarly, although a 

grantee may use a QED or RCT to meet 

the ‘‘evidence of promise’’ definition, a 

grantee could also use a correlational 

study with statistical controls. Thus, 

neither evidence level requires the use 

of a treatment and control group. See 

§ 77.1(c) for definitions of these terms. 

Although we appreciate that products 

and services change over time and may 

no longer be available in the format or 

version that was studied by the time an 

evaluation is complete, these selection 

factors are intended to provide 

incentives to applicants for grants to 

build an evidence base on the 

effectiveness of the processes, products, 

strategies, and practices that are, or will 

be used, in education. As noted 

elsewhere in our discussion of 

comments related to evidence of 

effectiveness, an applicant may use a 

study of a product or service to support 

an adaptation of it so long as the 

applicant can provide a justification that 

the proposed project’s efficacy 

necessitates the adaptation, and is based 

upon the evidence and theory supported 

by the original study. This same concept 

applies to the potential for learning from 

a project evaluation of a product or 

service that may no longer be available 

in the format or version that was studied 

by the time an evaluation is complete. 

Moreover, the selection factor regarding 

‘‘evidence of promise’’ does allow 

consideration of alternative study 

designs. 

With regard to commenters’ concerns 

about the Department requiring the use 

of evaluation designs that are in conflict 

with a program’s statute, the 

Department does not propose or require 

grantees to use grant funds in a manner 

that is prohibited by statute. As noted 

elsewhere in our responses to comments 

in this section, the Department has 

discretion in determining which 

selection factors are most appropriate 

for a particular program’s purpose and 

goals. Therefore, the Department would 

not use a selection factor that is in 

conflict with a program’s governing 

statute, purpose, or goals. 

Changes: None. 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49347 

Comments: Several commenters 

recommended including a selection 

factor under § 75.210(h) (Quality of the 

Project Evaluation) to promote 

evaluative methods for small service 

delivery programs. Specifically, the 

commenters requested the addition of a 

selection factor for studies that assess or 

use best practices for service delivery 

strategies using small-scale pilots. 

Discussion: We agree that a selection 

factor that encourages project 

evaluations of pilot strategies that may 

be best practices is of value, and we 

have included § 75.210(h)(2)(xii) for this 

purpose. This selection factor supports 

project evaluations that clearly 

articulate the key components and 

outcomes of the grant-supported 

process, product, strategy, or practice, as 

well as the measurable threshold for 

acceptable implementation. 

Implementation studies that articulate 

the key components of the proposed 

project and the measureable threshold 

for acceptable implementation of the 

key components are necessary to 

disseminate information about and 

replicate best practices. These studies 

also could be used to evaluate a pilot of 

service delivery strategy. Because 

§ 75.210(h) (Quality of the Project 

Evaluation) and other existing factors 

under this criterion provide for the type 

of evaluation proposed by the 

commenter, we do not think it is 

necessary to create a factor for the 

evaluation of a specific type of project. 

Changes: None. 

Selection Criteria Based on Statutory or 

Regulatory Provisions—§ 75.209 

Comment: One commenter agreed 

with the proposed changes to § 75.209 

but suggested including a clause that 

explicitly allows for successful 

applicants to be able to adjust their 

projects based on a continuous 

improvement model. The commenter 

noted that this change would allow 

grantees to use formative evidence and 

research to adjust their projects as 

needed, resulting in better outcomes 

overall. 

Discussion: We agree that continuous 

improvement models are useful. In fact, 

grantees currently address their lessons 

learned during the implementation of 

the project and discuss how they can 

continuously improve their projects in 

their annual performance reports to the 

Department. The regulations do not 

prohibit a grantee from adjusting its 

project as needed, provided that the 

scope or objectives of the project are not 

changed. Our intent in the changes to 

§ 75.209 is to provide the Secretary the 

flexibility to use selection criteria 

related to a program’s statute or 

regulations in notices inviting 

applications. 

Changes: None. 

General Selection Criteria—§ 75.210 

Comments: Many commenters 

expressed general concern over the 

proposed changes to § 75.210(c) (Quality 

of the Project Design) without focusing 

on any one proposed factor. Some noted 

that the proposed selection factors 

under § 75.210(c) are not widely 

applicable to all Department programs 

and that some selection factors may 

unfairly disadvantage some applicants. 

For example, one commenter asserted 

that the proposed selection factors will 

not improve student outcomes and are 

therefore unnecessary. Another 

commenter expressed concern that the 

proposed selection factors allow the 

Secretary too much discretion when 

designing competitions and that the 

intent and purpose of some alreadyestablished programs could be 

compromised. 

Alternatively, one commenter 

suggested that many of the proposed 

selection factors rely too much on peer 

reviewer subjectivity and further that 

inter-rater reliability between peer 

reviewers would be difficult to achieve 

if these factors are used in a 

competition. 

Discussion: We agree that each 

selection factor in § 75.210(c) (Quality of 

the Project Design) is not applicable to 

each Department program. There is no 

requirement that the Department use 

each selection factor listed in § 75.210(c) 

(Quality of the Project Design) for every 

program or competition. We rely on 

Department program officials to choose 

the selection factors for their programs 

prudently, with the capacity of 

applicants in mind, such that the 

selection factors used will appropriately 

match the goals of the program. 

As part of the discretionary grant 

process, we depend on peer reviewers 

for their objectivity and professional 

expertise. The Department trains peer 

reviewers on the details of the particular 

program, and monitors peer reviewer 

discussions to ensure that reviewers 

make scoring decisions based solely on 

the selection criteria provided in the 

notice inviting applications. While the 

Department takes these steps to support 

inter-rater reliability, we also rely on the 

professional judgment and expertise of 

peer reviewers when evaluating 

applications. 

Although some factors may not 

directly relate to student achievement, 

we disagree with the comment that the 

new selection factors in § 75.210(c) 

(Quality of the Project Design) will not 

improve student achievement. Each 

factor requires applicants to approach 

the design of their projects in ways that 

will increase efficiency, productivity, 

and overall success. Increased student 

achievement will result from a 

Department competition’s use of 

selection factors that better allow 

grantees to implement their projects 

effectively. 

Changes: None. 

Comment: One commenter agreed 

with the proposed change to 

§ 75.210(c)(2)(xvi) regarding integration 

of a proposed project with similar or 

related efforts. The commenter stated 

that emphasizing integration efforts 

within the grantee’s community would 

increase the likelihood of a successful 

project. The commenter noted that this 

proposed factor is particularly 

appropriate for public charter schools, 

given their unique positions in their 

communities. 

Some commenters expressed concern 

that the proposed change to 

§ 75.210(c)(2)(xvi) would disadvantage 

nonprofit entities. These commenters 

noted that nonprofit entities do not 

necessarily have control over State or 

Federal funding streams and may have 

difficulty securing willing community 

partners. One commenter expanded on 

this concern, and suggested that we 

clarify that the integration of existing 

funding streams only be considered to 

the extent practicable for applicants. 

Similarly, another commenter noted 

that in some high-need areas, 

opportunities to partner with funding 

organizations simply do not exist. 

One commenter suggested that 

§ 75.210(c)(2)(xvi) regarding increased 

efficiency and productivity and (xxvii) 

regarding using nonpublic funds or 

resources to build on similar or related 

efforts be used only as competitive 

preference priorities. The commenter 

stressed that the proposed selection 

factors could disadvantage small and 

rural LEAs, and potentially eliminate 

applicants with otherwise strong 

responses to the criterion due solely to 

their inability to secure other sources of 

funding. The commenter also warned 

that a nonpublic entity may have its 

own agenda when agreeing to partner 

with an applicant, possibly 

complicating the nature of the 

partnership. 

One commenter expressed concern 

that nonpublic funds and resources 

were given favor in § 75.210(c)(2)(xxvii). 

The commenter argued that integration 

of resources is important regardless of 

their source, be it public or private. 

Based on this argument, the commenter 

suggested the selection factor be 

removed. 


ehiers on DSK2VPTVN1PROD with RULES_2
49348 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

Discussion: We agree that the 

proposed selection factors will increase 

the likelihood of grantee success in 

implementing their projects. Requiring 

an applicant to create partnerships with 

community, State, or Federal partners 

establishes early on that project 

sustainability is an important factor for 

success, in the event that the proposed 

project is awarded grant funds. 

We recognize that establishing 

partnerships with community 

organizations may be challenging for 

some grantees, particularly those based 

in high-need areas. However, we would 

like to clarify that, under amended 

selection factor § 75.210(c)(2)(xvi), a 

grantee is not limited to using 

organizations based in its community 

when selecting partners. Grantees may 

choose to integrate or build on the 

related efforts of other programs based 

anywhere in the country, assuming that 

the goals and efforts of such programs 

align appropriately with those of the 

grant. 

We also recognize that small and rural 

LEAs may experience challenges when 

responding to the selection factor. We 

would like to stress that this change to 

the selection criterion in § 75.210(c) 

(Quality of Project Design) does not 

require its use in each Department 

competition. The Department chooses 

appropriate selection factors based on 

the intended goals of the program and 

the expected capacity of applicants. 

We disagree that changing the 

proposed selection factors into 

competitive preference priorities would 

prove more favorable for small and rural 

LEAs. Competitive preference priorities, 

by providing grantees who address them 

with an advantage over those who 

choose not to do so, can be decisive in 

determining which applicants receive 

grants. Therefore, it is not clear that 

considering the integration of other 

funding streams as a competitive 

preference priority, as opposed to a 

selection factor, would address the 

commenter’s concern. 

Nonprofit entities, while typically not 

in a position to control State or Federal 

funding streams, do have options 

available to them to address 

§ 75.210(c)(2)(xvi). This selection factor 

considers the extent to which an 

applicant has secured partners to build 

on similar efforts. A nonprofit entity, if 

it were eligible to apply for funding in 

a Department program, could 

collaborate with an LEA, which is likely 

receiving State and Federal funding, on 

a mutually beneficial project that aligns 

with the goals of the Department 

program. 

While we agree that 

§ 75.210(c)(2)(xxvii) regarding 

integrating with or building on related 

efforts may not be appropriate for some 

Department programs, we are retaining 

it because there are Department 

programs in which it would be 

beneficial. Because the use of this 

selection factor is not required for use 

in all Department programs or 

competitions, we think the best 

approach to addressing this concern is 

for the Department to use the selection 

factor in only those programs for which 

it is appropriate. 

In response to the concern that a 

nonpublic entity may take advantage of 

an applicant and complicate the nature 

of the partnership to promote its own 

agenda, applicants applying to a 

competition in which 

§ 75.210(c)(2)(xxvii) is a selection factor 

should use their best judgment in 

evaluating potential partners and only 

enter into formal relationships with 

entities that share their goals. This is 

generally a prudent practice, whether 

the applicant is choosing to partner with 

a public or a nonpublic entity, and 

should be followed in any competition 

that requires an applicant to work with 

a partner, even if § 75.210(c)(2)(xxvii) is 

not a selection factor. 

Changes: None. 

Comment: Some commenters praised 

the proposed selection factor 

§ 75.210(c)(2)(xxiv) regarding resources 

for operating a project beyond the length 

of the grant. Commenters also supported 

§ 75.210(c)(2)(xxv), which asks 

applicants to describe the potential and 

planning for the incorporation of project 

purposes and activities into the ongoing 

work of the grant. These commenters 

stated that grant funds should not be 

awarded in cases where long-term 

funding is needed but not secured and 

that asking an applicant to explicitly 

address how it plans to continue the 

project after the completion of a grant 

award will help to ensure long-term 

success. 

Two commenters expressed 

agreement with the proposed changes to 

§ 75.210(c)(2)(xxiv) and (xxv), but 

suggested some further modifications. 

One commenter suggested that we 

consider whether applicants have 

effectively worked to build a market for 

educational services. Another 

commenter suggested that when 

considering the extent to which an 

applicant has secured resources to 

sustain the project beyond the grant 

period, we also consider whether the 

applicant has demonstrated evidence of 

broad stakeholder commitment to the 

project. 

One commenter agreed that it is 

critical that grantees plan their projects 

with sustainability in mind but did not 

agree with the addition of selection 

factors § 75.210(c)(2)(xxiv) and (xxv), 

arguing that the current state of the 

economy is not stable enough to ensure 

that resources committed at the time an 

award is made would still be available 

at the end of a grant period. Another 

commenter noted that the proposed 

changes could disadvantage community 

colleges and proposed that we consider 

an applicant’s cost per student when 

using proposed selection factors related 

to sustainability. The commenter stated 

that applicants working with fewer 

resources per student need more 

flexibility in adhering to the 

requirements outlined in selection 

factor § 75.210(c)(2)(xxiv). 

Discussion: We agree that long-term 

planning and broad stakeholder support 

are integral to a grantee’s successful 

project. The amendments to § 75.210(c) 

(Quality of the Project Design) will 

allow for flexibility when assessing an 

applicant’s plan to sustain its project 

after the grant period ends. With added 

flexibility in § 75.210(c), the Department 

may choose to fund applications that 

have a strong focus in effective and 

sustainable practices. 

We recognize that some applicants, 

such as community colleges, may 

operate with fewer resources per 

student than other types of applicants. 

However, the regulations do not 

prohibit such an applicant from 

requesting funds in its budget proposal 

to support the proposed project fully. If 

an applicant assesses its resources and 

finds that it requires more funds per 

student to carry out the project and 

address selection factor 

§ 75.210(c)(2)(xxiv), that applicant 

should plan its budget accordingly. It is 

important that an applicant have the 

discretion to determine how best to 

address its sustainability needs. For 

example, an applicant may design its 

project to include strategies that build 

its capacity to implement project 

activities more efficiently, which in turn 

would support sustainability after the 

grant. 

The proposed selection factors related 

to sustainability are designed with the 

current economic climate in mind. As a 

few commenters noted, applicants 

should be actively planning on how 

those realities will affect their proposed 

projects. The intent of selection factors 

§ 75.210(c)(2)(xxiv) and (xxv) is to 

encourage applicants to engage in this 

planning process and comprehensively 

plan how their projects could be 

implemented beyond the grant period if 

such projects were funded. 

Finally, regarding the 

recommendation that we include a 

factor considering whether an applicant 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49349 

effectively worked to build a market for 

educational services, we note that we 

have added a new selection criterion, 

§ 75.210(i) (Strategy to Scale), which 

includes selection factor 

§ 75.210(i)(2)(v), that considers whether 

an applicant demonstrates an unmet 

demand for the process, product, 

strategy, or practice that will enable the 

applicant to reach the level of scale that 

is proposed in the application. This 

factor recognizes work that an applicant 

would do in advance of the project to 

build a market for educational services. 

Changes: None. 

Comment: Two commenters noted 

specifically the importance of 

productivity and efficiency, stating that 

selection factor § 75.210(c)(2)(xxvi) is 

especially appropriate given the current 

climate of limited resources and high 

expectations for success. 

Discussion: We agree that 

productivity and efficiency have 

become increasingly important factors 

to consider in recent years. Allowing the 

Secretary to evaluate whether a 

proposed project is efficient and 

productive will ensure that Department 

funds are used as effectively and 

prudently as possible. 

Changes: None. 

Comments: Two commenters agreed 

with the addition of a new selection 

criterion, § 75.210(i) (Strategy to Scale), 

to consider an applicant’s ability to 

successfully scale a project at the 

regional or national level. One 

commenter noted that the proposed 

addition would specifically benefit 

charter management organizations and 

support them in scaling successful 

strategies, and the other noted the 

importance of sharing best practices 

broadly. 

Another commenter expressed 

support for the selection criterion in 

§ 75.210(i) but requested that we allow 

for-profit entities, as well as nonprofit 

entities, to partner with grantees to 

bring their projects to scale during the 

grant period. The commenter stated that 

scaling has not historically been an area 

of expertise for entities providing 

educational services and that for-profit 

entities are well-suited to provide 

needed support. 

Discussion: We agree that, in many 

Department programs, an applicant’s 

ability to scale a proposed process, 

product, strategy, or program is very 

important. As the Department begins 

and continues to use this selection 

criterion, we expect potential applicants 

will devote resources and supports to 

focus on the processes, products, 

strategies, and programs that have 

greater potential to scale. 

Should a grantee decide that it needs 

additional help in the area of scalability, 

that grantee is not obligated to seek 

assistance from only nonprofit entities. 

The proposed selection criterion, as 

written, does not explicitly refer to the 

types of entities with which a grantee 

may choose to work. We recognize that 

some for-profit entities may be 

particularly well-positioned to help 

grantees achieve scale, and encourage 

each grantee, to the extent it requires 

additional support, to seek out partners 

that are best suited to meet the needs of 

their projects. 

Changes: None. 

Comments: Many commenters noted 

that while the strategy to scale is an 

important criterion to consider for new 

projects, it is not applicable to programs 

that have already established successful 

practices at a national scale or to 

programs that are already widely 

available to students. 

Conversely, some commenters 

expressed concern that the proposed 

selection criterion § 75.210(i) (Strategy 

to Scale) would not be feasible for small 

LEAs, rural LEAs, or community 

colleges. One commenter requested 

revising the language of the proposed 

criterion to ‘‘Feasibility of Replication’’ 

and placing it as a selection factor under 

§ 75.210(h) (Quality of the Project 

Evaluation). This commenter also 

suggested that the subject of scaling a 

project is better suited to an IES grant. 

Another commenter noted that the 

selection criterion should instead be 

used only as a selection factor in 

specific circumstances because an 

applicant’s capacity to scale is not a 

useful consideration for all Department 

programs. Another commenter added 

that some programs are, by definition, 

small and community based and that 

the use of this criterion would adversely 

affect such programs. 

One commenter did not agree with 

the proposed selection criterion 

concerning an applicant’s strategy to 

scale, noting that increasing focus on 

scaling projects to regional and national 

levels would decrease focus on student 

outcomes at the local level. The 

commenter also points out that many 

projects are effective because they are 

planned with a specific place in mind, 

and scaling such projects could prove 

ineffective. 

Discussion: We recognize that the 

proposed selection criterion may not be 

applicable to Department programs that 

are already well established. We stress 

that Department program officials are in 

the best position to decide which 

selection criteria and factors fit the goals 

of their programs. When preparing 

notices inviting applications, the 

Department will continue to consider 

the strengths and needs of likely 

applicants, and will choose selection 

criteria that are appropriate to the 

program’s purpose, goals, and applicant 

pool. Likewise, if the Department 

concludes that the nature of the program 

or types of applicants are not conducive 

to scaling, then the Department may 

decide not to include the selection 

criterion in the notice inviting 

applications for the program. 

If the Department concludes the use 

of this criterion is consistent with the 

program’s purpose and goals then 

applicants that better address the 

criterion will likely receive more points 

for the criterion than applicants that 

address it poorly. We recognize that 

some types of applicants may not 

typically design or implement projects 

that include activities to support 

effectively scaling a proposed process, 

product, strategy, or practice; however, 

any applicant responding to a notice 

inviting applications that includes this 

criterion may consider partnering with 

others to take the proposed process, 

product, strategy, or practice to scale. 

We do not agree with the suggestion 

to change selection criterion § 75.210(i) 

(Strategy to Scale) into a selection factor 

under selection criterion § 75.210(h) 

(Quality of the Project Evaluation). The 

nuances needed to make the freestanding selection criterion useful and 

adaptable to a variety of Department 

programs would be lost if it were rewritten as a selection factor under 

another criterion. It is important that a 

grantee experiencing success be able to 

share information about its project and 

support broad implementation of it to 

ensure that best practices are widely 

accessible and more easily replicated in 

the field. We think that by including 

§ 75.210(i) (Strategy to Scale) as a 

selection criterion, as opposed to a 

selection factor within a selection 

criterion, we are able to provide clearer 

guidance to applicants on effective 

scaling methodology and feasible 

replication. 

We disagree that including a selection 

criterion that considers an applicant’s 

ability to effectively scale its proposed 

process, product, strategy, or practice 

would decrease focus on student 

outcomes at the local level. By choosing 

to consider one selection criterion, the 

Department does not diminish the 

influence of other selection criteria 

under consideration. For example, if 

Department program officials choose to 

consider § 75.210(c) (Quality of the 

Project Design) and § 75.210(i) (Strategy 

to Scale), a successful applicant would 

be expected to respond effectively to 

both criteria. That applicant would 


ehiers on DSK2VPTVN1PROD with RULES_2
49350 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

therefore need to explain why its project 

design is effective in increasing student 

outcomes in its current setting and 

explain its capacity to scale. While 

§ 75.210(i) (Strategy to Scale) primarily 

considers how well an applicant could 

implement its proposed process, 

product, strategy, or practice in a variety 

of settings and populations, it remains 

one piece among many to be considered 

as part of the competition process. 

Changes: None. 

Maximum Funding Period—§ 75.250 

Comments: Many commenters 

expressed support for the change to this 

regulation because it will improve the 

quality of the data available to 

determine whether educational 

activities improve teaching and 

learning. However, one of these 

commenters stated that the option for 

funding for continued evaluation should 

be guaranteed. The commenter also 

suggested that grantees be allowed to 

use funds to hire qualified data 

management personnel and consultants 

to develop data architecture and data 

storage capacity. 

Discussion: We appreciate the 

commenters’ support for this regulation. 

However, we cannot guarantee 

continued data collection periods for all 

programs and grantees because this 

option is only possible in cases where 

there is authority for evaluation 

activities and sufficient funds have been 

appropriated for the program. Because 

these conditions may not be met for all 

programs or in all years, we cannot 

guarantee a continued data collection 

period for all programs and grantees. 

With regard to the recommendation 

that the Department allow grantees to 

use grant funds to support personnel or 

contracts to assist with data collection, 

we note that, under the current 

regulations and cost principles, 

applicants may include such costs in 

their proposed budgets to contract for 

these services so long as they are 

necessary to the performance 

measurement and evaluation of the 

project. 

Changes: None. 

Comment: One commenter 

recommended clarifying whether the 

Secretary could approve a data 

collection period without providing 

additional funds. The commenter 

explained that in some cases grantees 

may need both additional time and 

funds in order to complete performance 

measurement activities but that there 

are also instances when a grantee may 

need only additional time. 

Discussion: Under § 75.261, a grantee 

may request a no-cost extension of its 

project period to complete approved 

project activities. Thus, the regulations 

already allow grantees to request 

additional time to complete 

performance measure and other project 

activities without additional funding, so 

long as the appropriation accounts 

remain available. Funds obligated on a 

fiscal year basis remain available in 

grant accounts for five fiscal years after 

the expiration of the fiscal year for 

which the funds were obligated by the 

Federal government. 31 U.S.C. 1552(a). 

Thus, both obligated and unobligated 

grant funds generally remain available 

to grantees during no cost extensions to 

fund continued collection of data after 

the end of a project period. 

The amended regulations in § 75.250 

allow the Secretary to approve a data 

collection period with or without 

additional funds. Prior to the approval 

of a data collection period, we would 

assess with the grantee the appropriate 

duration of the data collection period 

and whether additional funds are 

necessary to complete the data 

collection, reporting, and analysis that 

would occur during that period. 

Changes: None. 

Comment: One commenter stated that 

data collection is not the only valid 

reason for extending a grant period and 

suggested revising the regulations to 

allow extensions on the basis of 

effectiveness and to aid in a project’s 

transition to a new funding stream. 

Discussion: These amendments apply 

to discretionary grant programs that 

award funds on the basis of a 

competitive process. As such, it would 

not be appropriate for the Department to 

award additional funds to a grantee to 

conduct a new project or transition to a 

new funding stream outside of the 

competitive process. 

Changes: None. 

Continuation of a Multi-Year Project 

After the First Budget Period—§ 75.253 

Comments: Three commenters 

expressed support for the change in this 

regulation. One commenter stated that 

the change would improve the use of 

performance measurement and 

evaluation. However, one of these 

commenters requested additional 

information on the meaning of 

‘‘substantial progress.’’ Another 

commenter urged establishing programspecific evaluation requirements that 

balance the need for valid evidence of 

effectiveness with the need to limit 

burden on grantees. 

Discussion: We appreciate the 

commenters’ support for the amended 

regulation. Given the variety of 

programs to which these regulations 

apply, a more detailed definition of 

‘‘substantial progress’’ would not be 

practical or helpful. The Department 

will establish program-specific 

evaluation requirements in the notice 

inviting applications and will consider 

the program’s purpose, goals, and 

applicant pool when deciding the 

evaluation requirements to use in a 

given year’s competition. As part of this 

process, the Department must consider 

the burden of the information collection 

associated with the application and 

program requirements and receive 

approval under the Paperwork 

Reduction Act of 1995 from OMB to 

collect that information. Because 

current law requires programs to 

consider the burden associated with 

information collection, we do not think 

it is necessary to make the change 

requested by the commenter. 

Changes: None. 

Other Comments Not Directly Related 

to the Proposed Rule 

Comment: One commenter stated that 

the amendments were unclear and 

would produce nothing of value for 

students in this country. 

Discussion: Although we recognize 

these amendments may not directly 

affect students, we disagree with the 

assessment that they would not produce 

anything of value. These amendments 

are designed to support the successful 

implementation of projects funded by 

the Department and improve their 

performance measures, which will in 

turn benefit students served by the 

projects. The proposed amendments 

also allow the Department to be more 

effective and efficient when selecting 

discretionary grantees, to provide 

higher-quality data to the Congress and 

the public about the effectiveness of 

Department programs, and to reduce 

administrative burden on applicants 

and grantees. 

Changes: None. 

Comment: One commenter expressed 

concern that students with disabilities 

are underserved. 

Discussion: We appreciate the 

commenter’s concern and note that 

section 504 of the Rehabilitation Act of 

1973 and the Department’s section 504 

implementing regulations prohibit 

discrimination on the basis of disability 

for entities receiving financial assistance 

from the Department. In addition, the 

Department enforces Title II of the 

Americans with Disabilities Act (ADA), 

as well as the regulations implementing 

Title II of the ADA, which prohibit 

discrimination on the basis of disability 

by public entities. Finally, section 427 

of the General Education Provisions Act 

(GEPA) addresses equitable access by 

requiring all applicants to provide a 

statement that identifies access barriers 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49351 

to participation, which can include 

barriers to participation by individuals 

with disabilities, in their projects and 

identifies solutions to overcome those 

barriers. 

Moreover, the Department’s Office of 

Special Education and Rehabilitative 

Services (OSERS) is committed to 

improving results and outcomes for 

people with disabilities of all ages. 

OSERS provides a wide array of 

supports to parents and individuals, 

school districts, and states in three main 

areas: Special education, vocational 

rehabilitation, and research. Within 

OSERS, the Office of Special Education 

Programs (OSEP) supports a 

comprehensive array of programs and 

projects authorized by the Individuals 

with Disabilities Education Act (IDEA) 

that improve results for infants, 

toddlers, children, and youth with 

disabilities. 

Changes: None. 

Comment: One commenter 

recommended adding language to 

§§ 76.722 and 80.40 to clarify that, 

although a grantee may require 

subrecipients to submit reports in a 

manner and format that enable the 

grantee to comply with Department 

requirements, an SEA must not do so in 

a manner that would place financial or 

programmatic burden on the 

subrecipient or require a subrecipient to 

provide data that is readily available to 

the SEA by other means. The 

commenter noted that monitoring 

subrecipients is vital to the successful 

implementation of a grant, but a grantee 

should not use it to usurp autonomy of 

subrecipients or to require the use of 

specific financial software that could be 

costly or burdensome to small entities. 

Discussion: In the preamble of the 

NPRM, we discussed on page 74392 the 

Department’s retrospective analysis of 

existing regulations and requested 

comment on other regulations within 

EDGAR that may be in need of 

modification. We appreciate this 

commenter’s concerns regarding 

§§ 76.722 and 80.40 and will use the 

feedback to further inform and plan our 

retrospective review efforts. 

Changes: None. 

Executive Orders 12866 and 13563 

Regulatory Impact Analysis 

Under Executive Order 12866, the 

Secretary must determine whether this 

regulatory action is ‘‘significant’’ and, 

therefore, subject to the requirements of 

the Executive order and subject to 

review by the Office of Management and 

Budget (OMB). Section 3(f) of Executive 

Order 12866 defines a ‘‘significant 

regulatory action’’ as an action likely to 

result in a rule that may— 

(1) Have an annual effect on the 

economy of $100 million or more, or 

adversely affect a sector of the economy, 

productivity, competition, jobs, the 

environment, public health or safety, or 

State, local, or tribal governments or 

communities in a material way (also 

referred to as an ‘‘economically 

significant’’ rule); 

(2) Create serious inconsistency or 

otherwise interfere with an action taken 

or planned by another agency; 

(3) Materially alter the budgetary 

impacts of entitlement grants, user fees, 

or loan programs or the rights and 

obligations of recipients thereof; or 

(4) Raise novel legal or policy issues 

arising out of legal mandates, the 

President’s priorities, or the principles 

stated in the Executive order. 

This final regulatory action is a 

significant regulatory action subject to 

review by OMB under section 3(f) of 

Executive Order 12866. 

We have also reviewed these 

regulations under Executive Order 

13563, which supplements and 

explicitly reaffirms the principles, 

structures, and definitions governing 

regulatory review established in 

Executive Order 12866. To the extent 

permitted by law, Executive Order 

13563 requires that an agency— 

(1) Propose or adopt regulations only 

on a reasoned determination that their 

benefits justify their costs (recognizing 

that some benefits and costs are difficult 

to quantify); 

(2) Tailor its regulations to impose the 

least burden on society, consistent with 

obtaining regulatory objectives and 

taking into account—among other things 

and to the extent practicable—the costs 

of cumulative regulations; 

(3) In choosing among alternative 

regulatory approaches, select those 

approaches that maximize net benefits 

(including potential economic, 

environmental, public health and safety, 

and other advantages; distributive 

impacts; and equity); 

(4) To the extent feasible, specify 

performance objectives, rather than the 

behavior or manner of compliance a 

regulated entity must adopt; and 

(5) Identify and assess available 

alternatives to direct regulation, 

including economic incentives—such as 

user fees or marketable permits—to 

encourage the desired behavior, or 

provide information that enables the 

public to make choices. 

Executive Order 13563 also requires 

an agency ‘‘to use the best available 

techniques to quantify anticipated 

present and future benefits and costs as 

accurately as possible.’’ The Office of 

Information and Regulatory Affairs of 

OMB has emphasized that these 

techniques may include ‘‘identifying 

changing future compliance costs that 

might result from technological 

innovation or anticipated behavioral 

changes.’’ 

We are issuing these final regulations 

only on a reasoned determination that 

their benefits justify their costs. In 

choosing among alternative regulatory 

approaches, we selected those 

approaches that maximize net benefits. 

Based on the analysis that follows, the 

Department believes that these proposed 

regulations are consistent with the 

principles in Executive Order 13563. 

We also have determined that this 

regulatory action does not unduly 

interfere with State, local, or tribal 

governments in the exercise of their 

governmental functions. 

In accordance with both Executive 

orders, the Department has assessed the 

potential costs and benefits, both 

quantitative and qualitative, of this 

regulatory action. The potential costs 

associated with this regulatory action 

are those resulting from statutory 

requirements and those we have 

determined as necessary for 

administering the Department’s 

programs and activities. 

Paperwork Reduction Act of 1995 

These regulations do not contain any 

information collection requirements. 

Intergovernmental Review 

These regulations subject to the 

requirements of Executive Order 12372 

and the regulations in 34 CFR part 79. 

One of the objectives of the Executive 

order is to foster an intergovernmental 

partnership and a strengthened 

federalism. The Executive order relies 

on processes developed by State and 

local governments for coordination and 

review of proposed Federal financial 

assistance. 

This document provides early 

notification of our specific plans and 

actions for these regulations. 

Assessment of Educational Impact 

In the NPRM we requested comments 

on whether the proposed regulations 

would require transmission of 

information that any other agency or 

authority of the United States gathers or 

makes available. 

Based on the response to the NPRM 

and on our review, we have determined 

that these final regulations do not 

require transmission of information that 

any other agency or authority of the 

United States gathers or makes 

available. 


ehiers on DSK2VPTVN1PROD with RULES_2
49352 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

Accessible Format: Individuals with 

disabilities can obtain this document in 

an accessible format (e.g., braille, large 

print, audiotape, or compact disc) on 

request to the program contact person 

listed under FOR FURTHER INFORMATION

CONTACT. 

Electronic Access to This Document: 

The official version of this document is 

the document published in the Federal 

Register. Free Internet access to the 

official edition of the Federal Register 

and the Code of Federal Regulations is 

available via the Federal Digital System 

at: www.gpo.gov/fdsys. At this site you 

can view this document, as well as all 

other documents of this Department 

published in the Federal Register, in 

text or Adobe Portable Document 

Format (PDF). To use PDF you must 

have Adobe Acrobat Reader, which is 

available free at the site. 

You may also access documents of the 

Department published in the Federal 

Register by using the article search 

feature at: www.federalregister.gov. 

Specifically, through the advanced 

search feature at this site, you can limit 

your search to documents published by 

the Department. You may also view this 

document in text [or PDF] at the 

following site: 

List of Subjects 

34 CFR Part 75 

Accounting, Copyright, Education, 

Grant programs—education. 

34 CFR Part 77 

Education, Grant programs— 

education. 

Dated: August 6, 2013. 

Arne Duncan, 

Secretary of Education. 

For the reasons discussed in the 

preamble, the Secretary amends parts 75 

and 77 of title 34 of the Code of Federal 

Regulations as follows: 

PART 75—DIRECT GRANT 

PROGRAMS 

■ 1. The authority citation for part 75 

continues to read as follows: 

Authority: 20 U.S.C. 1221e–3 and 3474, 

unless otherwise noted. 

■ 2. Add a new § 75.110 to read as 

follows: 

§ 75.110 Information regarding 

performance measurement. 

(a) The Secretary may establish in an 

application notice for a competition one 

or more performance measurement 

requirements, including requirements 

for performance measures, baseline 

data, or performance targets, and a 

requirement that applicants propose in 

their applications one or more of their 

own performance measures, baseline 

data, or performance targets. 

(b) If an application notice requires 

applicants to propose project-specific 

performance measures, baseline data, or 

performance targets, the application 

must include the following, as required 

by the application notice: 

(1) Performance measures. How each 

proposed performance measure would 

accurately measure the performance of 

the project and how the proposed 

performance measure would be 

consistent with the performance 

measures established for the program 

funding the competition. 

(2) Baseline data. (i) Why each 

proposed baseline is valid; or 

(ii) If the applicant has determined 

that there are no established baseline 

data for a particular performance 

measure, an explanation of why there is 

no established baseline and of how and 

when, during the project period, the 

applicant would establish a valid 

baseline for the performance measure. 

(3) Performance targets. Why each 

proposed performance target is 

ambitious yet achievable compared to 

the baseline for the performance 

measure and when, during the project 

period, the applicant would meet the 

performance target(s). 

(c) If the application notice 

establishes performance measurement 

requirements, the applicant must also 

describe in the application— 

(1)(i) The data collection and 

reporting methods the applicant would 

use and why those methods are likely to 

yield reliable, valid, and meaningful 

performance data; and 

(ii) If the Secretary requires applicants 

to collect data after the substantive work 

of a project is complete regarding the 

attainment of certain performance 

targets, the data collection and reporting 

methods the applicant would use during 

the post-performance period and why 

those methods are likely to yield 

reliable, valid, and meaningful 

performance data. 

(2) The applicant’s capacity to collect 

and report reliable, valid, and 

meaningful performance data, as 

evidenced by high-quality data 

collection, analysis, and reporting in 

other projects or research. 

(Authority: 20 U.S.C. 1221e–3 and 3474) 

■ 3. Add a new undesignated center 

heading ‘‘Competition Exceptions’’ in 

subpart C immediately before the 

undesignated center heading ‘‘State 

Comment Procedures’’. 

■ 4. Add a new § 75.135 to subpart C 

under the undesignated center heading 

‘‘Competition Exceptions’’ to read as 

follows: 

§ 75.135 Competition exception for 

proposed implementation sites, 

implementation partners, or service 

providers. 

(a) When entering into a contract with 

implementation sites or partners, an 

applicant is not required to comply with 

the competition requirements in 34 CFR 

74.43 or 80.36(c), as applicable, if— 

(1) The contract is with an entity that 

agrees to provide a site or sites where 

the applicant would conduct the project 

activities under the grant; 

(2) The implementation sites or 

partner entities that the applicant 

proposes to use are identified in the 

application for the grant; and 

(3) The implementation sites or 

partner entities are included in the 

application in order to meet a 

regulatory, statutory, or priority 

requirement related to the competition. 

(b) When entering into a contract for 

data collection, data analysis, evaluation 

services, or essential services, an 

applicant may select a provider using 

the informal, small-purchase 

procurement procedures in 34 CFR 

80.36(d)(1), regardless of whether that 

applicant would otherwise be subject to 

that part or whether the evaluation 

contract would meet the standards for a 

small purchase order, if— 

(1) The contract is with the data 

collection, data analysis, evaluation 

service, or essential service provider; 

(2) The data collection, data analysis, 

evaluation service, or essential service 

provider that the applicant proposes to 

use is identified in the application for 

the grant; and 

(3) The data collection, data analysis, 

evaluation service, or essential service 

provider is identified in the application 

in order to meet a statutory, regulatory, 

or priority requirement related to the 

competition. 

(c) If the grantee relied on the 

exceptions under paragraph (a) or (b) of 

this section, the grantee must certify in 

its application that any employee, 

officer, or agent participating in the 

selection, award, or administration of a 

contract is free of any real or apparent 

conflict of interest and, if the grantee 

relied on the exceptions of paragraph (b) 

of this section, that the grantee used 

small purchase procedures to obtain the 

product or service. 

(d) A grantee must obtain the 

Secretary’s prior approval for any 

change to an implementation site, 

implementation partner, or data 

collection, data analysis, evaluation 

service, or essential service provider, if 

the grantee relied on the exceptions 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49353 

1What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

2What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

under paragraph (a) or (b) of this section 

to select the entity. 

(e) The exceptions in paragraphs (a) 

and (b) of this section do not extend to 

the other procurement requirements in 

34 CFR part 74 and 34 CFR part 80 

regarding contracting by grantees and 

subgrantees. 

(f) For the purposes of this section, 

essential service means a product or 

service directly related to the grant that 

would, if not provided, have a 

detrimental effect on the grant. 

(Authority: 20 U.S.C. 1221e–3 and 3474) 

■ 5. Revise § 75.209 to read as follows: 

§ 75.209 Selection criteria based on 

statutory or regulatory provisions. 

The Secretary may establish selection 

criteria and factors based on statutory or 

regulatory provisions that apply to the 

authorized program, which may 

include, but are not limited to criteria 

and factors that reflect— 

(a) Criteria contained in the program 

statute or regulations; 

(b) Criteria in § 75.210; 

(c) Allowable activities specified in 

the program statute or regulations; 

(d) Application content requirements 

specified in the program statute or 

regulations; 

(e) Program purposes, as described in 

the program statute or regulations; or 

(f) Other pre-award and post-award 

conditions specified in the program 

statute or regulations. 

(Authority: 20 U.S.C. 1221e–3 and 3474) 

■ 6. Amend § 75.210 by: 

■ A. Revising the introductory text. 

■ B. Revising paragraph (c)(2)(xvi). 

■ C. Adding paragraphs (c)(2)(xxiv) 

through (xxix). 

■ D. Adding paragraphs (h)(2)(viii) 

through (xii). 

■ E. Adding a new paragraph (i). 

The revisions and additions read as 

follows. 

§ 75.210 General selection criteria. 

In determining the selection criteria to 

evaluate applications submitted in a 

grant competition, the Secretary may 

select one or more of the following 

criteria and may select from among the 

list of optional factors under each 

criterion. The Secretary may define a 

selection criterion by selecting one or 

more specific factors within a criterion 

or assigning factors from one criterion to 

another criterion. 

* * * * * 

(c) * * * 

(2) * * * 

(xvi) The extent to which the 

proposed project will integrate with or 

build on similar or related efforts to 

improve relevant outcomes (as defined 

in 34 CFR 77.1(c)), using existing 

funding streams from other programs or 

policies supported by community, State, 

and Federal resources. 

* * * * * 

(xxiv) The extent to which the 

applicant demonstrates that it has the 

resources to operate the project beyond 

the length of the grant, including a 

multi-year financial and operating 

model and accompanying plan; the 

demonstrated commitment of any 

partners; evidence of broad support 

from stakeholders (e.g., State 

educational agencies, teachers’ unions) 

critical to the project’s long-term 

success; or more than one of these types 

of evidence. 

(xxv) The potential and planning for 

the incorporation of project purposes, 

activities, or benefits into the ongoing 

work of the applicant beyond the end of 

the grant. 

(xxvi) The extent to which the 

proposed project will increase efficiency 

in the use of time, staff, money, or other 

resources in order to improve results 

and increase productivity. 

(xxvii) The extent to which the 

proposed project will integrate with or 

build on similar or related efforts in 

order to improve relevant outcomes (as 

defined in 34 CFR 77.1(c)), using 

nonpublic funds or resources. 

(xxviii) The extent to which the 

proposed project is supported by 

evidence of promise (as defined in 34 

CFR 77.1(c)). 

(xxix) The extent to which the 

proposed project is supported by strong 

theory (as defined in 34 CFR 77.1(c)). 

* * * * * 

(h) * * * 

(2) * * * 

(viii) The extent to which the methods 

of evaluation will, if well-implemented, 

produce evidence about the project’s 

effectiveness that would meet the What 

Works Clearinghouse Evidence 

Standards without reservations.1

(ix) The extent to which the methods 

of evaluation will, if well-implemented, 

produce evidence about the project’s 

effectiveness that would meet the What 

Works Clearinghouse Evidence 

Standards with reservations.2

(x) The extent to which the methods 

of evaluation will, if well-implemented, 

produce evidence of promise (as defined 

in 34 CFR 77.1(c)). 

(xi) The extent to which the methods 

of evaluation will provide valid and 

reliable performance data on relevant 

outcomes. 

(xii) The extent to which the 

evaluation plan clearly articulates the 

key components, mediators, and 

outcomes of the grant-supported 

intervention, as well as a measurable 

threshold for acceptable 

implementation. 

(i) Strategy to scale. (1) The Secretary 

considers the applicant’s strategy to 

scale the proposed project. 

(2) In determining the applicant’s 

capacity to scale the proposed project, 

the Secretary considers one or more of 

the following factors: 

(i) The applicant’s capacity (e.g., in 

terms of qualified personnel, financial 

resources, or management capacity) to 

bring the proposed project to scale on a 

national or regional level (as defined in 

34 CFR 77.1(c)) working directly, or 

through partners, during the grant 

period. 

(ii) The applicant’s capacity (e.g., in 

terms of qualified personnel, financial 

resources, or management capacity) to 

further develop and bring to scale the 

proposed process, product, strategy, or 

practice, or to work with others to 

ensure that the proposed process, 

product, strategy, or practice can be 

further developed and brought to scale, 

based on the findings of the proposed 

project. 

(iii) The feasibility of successful 

replication of the proposed project, if 

favorable results are obtained, in a 

variety of settings and with a variety of 

populations. 

(iv) The mechanisms the applicant 

will use to broadly disseminate 

information on its project so as to 

support further development or 

replication. 

(v) The extent to which the applicant 

demonstrates there is unmet demand for 

the process, product, strategy, or 

practice that will enable the applicant to 

reach the level of scale that is proposed 

in the application. 

(vi) The extent to which the applicant 

identifies a specific strategy or strategies 

that address a particular barrier or 

barriers that prevented the applicant, in 

the past, from reaching the level of scale 

that is proposed in the application. 

■ 7. Revise § 75.250 to read as follows: 

§ 75.250 Maximum funding period. 

(a) The Secretary may approve a 

project period of up to 60 months to 

perform the substantive work of a grant. 

(b) The Secretary may approve a data 

collection period for a grant for a period 

of up to 72 months after the end of the 

project period and provide funding for 


ehiers on DSK2VPTVN1PROD with RULES_2
49354 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

the data collection period for the sole 

purpose of collecting, analyzing, and 

reporting performance measurement 

data regarding the project. The Secretary 

may inform applicants of the Secretary’s 

intent to approve data collection periods 

in the application notice published for 

a competition or may decide to fund 

data collection periods after grantees 

have started their project periods. 

(Authority: 20 U.S.C. 1221e–3 and 3474.) 

■ 8. Amend § 75.251 by revising the 

section heading and adding a new 

paragraph (c) to read as follows: 

§ 75.251 Budget periods. 

* * * * * 

(c) If the Secretary funds a multi-year 

data collection period, the Secretary 

may fund the data collection period 

through separate budget periods and 

fund those budget periods in the same 

manner as those periods are funded 

during the project period. 

■ 9. Amend § 75.253 by— 

■ A. Revising paragraph (a)(2). 

■ B. Adding a new paragraph (a)(5). 

■ C. Redesignating paragraphs (b) 

through (e) as paragraphs (c) through (f). 

■ D. Adding a new paragraph (b). 

■ E. Revising newly redesignated 

paragraph (f). 

The revisions and additions read as 

follows: 

§ 75.253 Continuation of a multi-year 

project after the first budget period. 

(a) * * * 

(2) The grantee has either— 

(i) Made substantial progress in 

achieving— 

(A) The goals and objectives of the 

project; and 

(B) If the Secretary established 

performance measurement requirements 

for the grant in the application notice, 

the performance targets in the grantee’s 

approved application; or 

(ii) Obtained the Secretary’s approval 

for changes to the project that— 

(A) Do not increase the amount of 

funds obligated to the project by the 

Secretary; and 

(B) Enable the grantee to achieve the 

goals and objectives of the project and 

meet the performance targets of the 

project, if any, without changing the 

scope or objectives of the project. 

* * * * * 

(5) The grantee has maintained 

financial and administrative 

management systems that meet the 

requirements in 34 CFR 74.21 or 80.20, 

as appropriate. 

(b) In deciding whether a grantee has 

made substantial progress, the Secretary 

may consider any information relevant 

to the authorizing statute, a criterion, a 

priority, or a performance measure, or to 

a financial or other requirement that 

applies to the selection of applications 

for new grants. 

* * * * * 

(f) Unless prohibited by the program 

statute or regulations, a grantee that is 

in the final budget period of its project 

period may seek continued assistance 

for the project as required under the 

procedures for selecting new projects for 

grants. 

* * * * * 

■ 10. Add § 75.266 to subpart D to read 

as follows: 

§ 75.266 What procedures does the 

Secretary use if the Secretary decides to 

give special consideration to applications 

supported by strong or moderate evidence 

of effectiveness? 

(a) As used in this section, ‘‘strong 

evidence of effectiveness’’ is defined in 

34 CFR 77.1(c); 

(b) As used in this section, ‘‘moderate 

evidence of effectiveness’’ is defined in 

34 CFR 77.1(c); and 

(c) If the Secretary determines that 

special consideration of applications 

supported by strong or moderate 

evidence of effectiveness is appropriate, 

the Secretary may establish a separate 

competition under the procedures in 34 

CFR 75.105(c)(3), or provide 

competitive preference under the 

procedures in 34 CFR 75.105(c)(2), for 

applications supported by: 

(1) Evidence of effectiveness that 

meets the conditions set out in 

paragraph (a) of the definition of ‘‘strong 

evidence of effectiveness’’ in 34 CFR 

77.1; 

(2) Evidence of effectiveness that 

meets the conditions set out in either 

paragraph (a) or (b) of the definition of 

‘‘strong evidence of effectiveness’’ in 34 

CFR 77.1; or 

(3) Evidence of effectiveness that 

meets the conditions set out in the 

definition of ‘‘moderate evidence of 

effectiveness.’’ 

(Authority: 20 U.S.C. 1221e–3 and 3474.) 

■ 11. Revise § 75.590 to read as follows. 

§ 75.590 Evaluation by the grantee. 

(a) If the application notice for a 

competition required applicants to 

describe how they would evaluate their 

projects, each grantee under that 

competition must demonstrate to the 

Department that— 

(1) The evaluation meets the 

standards of the evaluation in the 

approved application for the project; 

and 

(2) The performance measurement 

data collected by the grantee and used 

in the evaluation meet the performance 

measurement requirements of the 

approved application. 

(b) If the application notice for a 

competition did not require applicants 

to describe how they would evaluate 

their projects, each grantee must 

provide information in its performance 

report demonstrating— 

(1) The progress made by the grantee 

in the most recent budget period, 

including progress based on the 

performance measurement requirements 

for the grant, if any; 

(2) The effectiveness of the grant, 

including fulfilling the performance 

measurement requirements of the 

approved application, if any; and 

(3) The effect of the project on the 

participants served by the project, if 

any. 

(Authority: 20 U.S.C. 1221e–3 and 3474.) 

■ 12. Amend § 75.708 by: 

■ A. Revising the section heading. 

■ B. Revising paragraph (a). 

■ C. Redesignating paragraph (b) as 

paragraph (e). 

■ D. Adding new paragraphs (b), (c) and 

(d). 

The revision and additions read as 

follows. 

§ 75.708 Subgrants. 

(a) A grantee may not make a subgrant 

under a program covered by this part 

unless authorized by statute or by 

paragraph (b) of this section. 

(b) The Secretary may, through an 

announcement in the Federal Register, 

authorize subgrants when necessary to 

meet the purposes of a program. In this 

announcement, the Secretary will— 

(1) Designate the types of entities, e.g., 

State educational agencies, local 

educational agencies, institutions of 

higher education, and nonprofit 

organizations, to which subgrants can be 

awarded; and 

(2) Indicate whether subgrants can be 

made to entities identified in an 

approved application or, without regard 

to whether the entity is identified in an 

approved application, have to be 

selected through a competitive process 

set out in subgranting procedures 

established by the grantee. 

(c) If authorized under paragraph (b) 

of this section, a subgrant is allowed if 

it will be used by that entity to directly 

carry out project activities described in 

that application. 

(d) The grantee, in awarding subgrants 

under paragraph (b) of this section, 

must— 

(1) Ensure that subgrants are awarded 

on the basis of an approved budget that 

is consistent with the grantee’s 

approved application and all applicable 

Federal statutory, regulatory, and other 

requirements; 


ehiers on DSK2VPTVN1PROD with RULES_2
Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 49355 

1What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

2What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

3What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

4What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

5What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

(2) Ensure that every subgrant 

includes any conditions required by 

Federal statute and executive orders and 

their implementing regulations; and 

(3) Ensure that subgrantees are aware 

of requirements imposed upon them by 

Federal statute and regulation, 

including the Federal antidiscrimination laws enforced by the 

Department. 

* * * * * 

PART 77— DEFINITIONS THAT APPLY 

TO DEPARTMENT REGULATIONS 

■ 13. The authority citation for part 77 

is revised to read as follows: 

Authority: 20 U.S.C. 1221e–3 and 3474, 

unless otherwise noted. 

■ 14. Amend § 77.1(c) by adding the 

following definitions in alphabetical 

order: 

§ 77.1 Definitions that apply to all 

Department programs. 

* * * * * 

(c) * * * 

Ambitious means promoting 

continued, meaningful improvement for 

program participants or for other 

individuals or entities affected by the 

grant, or representing a significant 

advancement in the field of education 

research, practices, or methodologies. 

When used to describe a performance 

target, whether a performance target is 

ambitious depends upon the context of 

the relevant performance measure and 

the baseline for that measure. 

* * * * * 

Baseline means the starting point 

from which performance is measured 

and targets are set. 

* * * * * 

Evidence of promise means there is 

empirical evidence to support the 

theoretical linkage(s) between at least 

one critical component and at least one 

relevant outcome presented in the logic 

model for the proposed process, 

product, strategy, or practice. 

Specifically, evidence of promise means 

the conditions in paragraphs (a) and (b) 

of this section are met: 

(i) There is at least one study that is 

a— 

(A) Correlational study with statistical 

controls for selection bias; 

(B) Quasi-experimental study that 

meets the What Works Clearinghouse 

Evidence Standards with reservations; 1

or 

(C) Randomized controlled trial that 

meets the What Works Clearinghouse 

Evidence Standards with or without 

reservations.2

(ii) The study referenced in paragraph 

(a) found a statistically significant or 

substantively important (defined as a 

difference of 0.25 standard deviations or 

larger), favorable association between at 

least one critical component and one 

relevant outcome presented in the logic 

model for the proposed process, 

product, strategy, or practice. 

* * * * * 

Large sample means an analytic 

sample of 350 or more students (or other 

single analysis units) who were 

randomly assigned to a treatment or 

control group or 50 or more groups 

(such as classrooms or schools) that 

contain 10 or more students (or other 

single analysis units) and that were 

randomly assigned to a treatment or 

control group. 

* * * * * 

Logic model (also referred to as theory 

of action) means a well-specified 

conceptual framework that identifies 

key components of the proposed 

process, product, strategy, or practice 

(i.e., the active ‘‘ingredients’’ that are 

hypothesized to be critical to achieving 

the relevant outcomes) and describes 

the relationships among the key 

components and outcomes, theoretically 

and operationally. 

* * * * * 

Moderate evidence of effectiveness 

means one of the following conditions 

is met: 

(i) There is at least one study of the 

effectiveness of the process, product, 

strategy, or practice being proposed that 

meets the What Works Clearinghouse 

Evidence Standards without 

reservations,3 found a statistically 

significant favorable impact on a 

relevant outcome (with no statistically 

significant and overriding unfavorable 

impacts on that outcome for relevant 

populations in the study or in other 

studies of the intervention reviewed by 

and reported on by the What Works 

Clearinghouse), and includes a sample 

that overlaps with the populations or 

settings proposed to receive the process, 

product, strategy, or practice. 

(ii) There is at least one study of the 

effectiveness of the process, product, 

strategy, or practice being proposed that 

meets the What Works Clearinghouse 

Evidence Standards with reservations,4

found a statistically significant favorable 

impact on a relevant outcome (with no 

statistically significant and overriding 

unfavorable impacts on that outcome for 

relevant populations in the study or in 

other studies of the intervention 

reviewed by and reported on by the 

What Works Clearinghouse), includes a 

sample that overlaps with the 

populations or settings proposed to 

receive the process, product, strategy, or 

practice, and includes a large sample 

and a multi-site sample (NOTE: multiple 

studies can cumulatively meet the large 

and multi-site sample requirements as 

long as each study meets the other 

requirements in this paragraph). 

* * * * * 

Multi-site sample means more than 

one site, where site can be defined as an 

LEA, locality, or State. 

* * * * * 

National level describes the level of 

scope or effectiveness of a process, 

product, strategy, or practice that is able 

to be effective in a wide variety of 

communities, including rural and urban 

areas, as well as with different groups 

(e.g., economically disadvantaged, racial 

and ethnic groups, migrant populations, 

individuals with disabilities, English 

learners, and individuals of each 

gender). 

* * * * * 

Performance measure means any 

quantitative indicator, statistic, or 

metric used to gauge program or project 

performance. 

Performance target means a level of 

performance that an applicant would 

seek to meet during the course of a 

project or as a result of a project. 

* * * * * 

Quasi-experimental design study 

means a study using a design that 

attempts to approximate an 

experimental design by identifying a 

comparison group that is similar to the 

treatment group in important respects. 

These studies, depending on design and 

implementation, can meet What Works 

Clearinghouse Evidence Standards with 

reservations 5 (they cannot meet What 

Works Clearinghouse Evidence 

Standards without reservations). 

Randomized controlled trial means a 

study that employs random assignment 

of, for example, students, teachers, 


ehiers on DSK2VPTVN1PROD with RULES_2
49356 Federal Register / Vol. 78, No. 156 / Tuesday, August 13, 2013 / Rules and Regulations 

6What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

7What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

8What Works Clearinghouse Procedures and 

Standards Handbook (Version 2.1, September 2011), 

which can currently be found at the following link: 

http://ies.ed.gov/ncee/wwc/ 

DocumentSum.aspx?sid=19. 

classrooms, schools, or districts to 

receive the intervention being evaluated 

(the treatment group) or not to receive 

the intervention (the control group). The 

estimated effectiveness of the 

intervention is the difference between 

the average outcome for the treatment 

group and for the control group. These 

studies, depending on design and 

implementation, can meet What Works 

Clearinghouse Evidence Standards 

without reservations.6

Regional level describes the level of 

scope or effectiveness of a process, 

product, strategy, or practice that is able 

to serve a variety of communities within 

a State or multiple States, including 

rural and urban areas, as well as with 

different groups (e.g., economically 

disadvantaged, racial and ethnic groups, 

migrant populations, individuals with 

disabilities, English learners, and 

individuals of each gender). For an LEAbased project, to be considered a 

regional-level project, a process, 

product, strategy, or practice must serve 

students in more than one LEA, unless 

the process, product, strategy, or 

practice is implemented in a State in 

which the State educational agency is 

the sole educational agency for all 

schools. 

Relevant outcome means the student 

outcome(s) (or the ultimate outcome if 

not related to students) the proposed 

process, product, strategy, or practice is 

designed to improve; consistent with 

the specific goals of a program. 

* * * * * 

Strong evidence of effectiveness 

means one of the following conditions 

is met: 

(i) There is at least one study of the 

effectiveness of the process, product, 

strategy, or practice being proposed that 

meets the What Works Clearinghouse 

Evidence Standards without 

reservations,7 found a statistically 

significant favorable impact on a 

relevant outcome (with no statistically 

significant and overriding unfavorable 

impacts on that outcome for relevant 

populations in the study or in other 

studies of the intervention reviewed by 

and reported on by the What Works 

Clearinghouse), includes a sample that 

overlaps with the populations and 

settings proposed to receive the process, 

product, strategy, or practice, and 

includes a large sample and a multi-site 

sample (Note: multiple studies can 

cumulatively meet the large and multisite sample requirements as long as each 

study meets the other requirements in 

this paragraph). 

(ii) There are at least two studies of 

the effectiveness of the process, product, 

strategy, or practice being proposed, 

each of which: Meets the What Works 

Clearinghouse Evidence Standards with 

reservations,8 found a statistically 

significant favorable impact on a 

relevant outcome (with no statistically 

significant and overriding unfavorable 

impacts on that outcome for relevant 

populations in the studies or in other 

studies of the intervention reviewed by 

and reported on by the What Works 

Clearinghouse), includes a sample that 

overlaps with the populations and 

settings proposed to receive the process, 

product, strategy, or practice, and 

includes a large sample and a multi-site 

sample. 

Strong theory means a rationale for 

the proposed process, product, strategy, 

or practice that includes a logic model. 

[FR Doc. 2013–19390 Filed 8–12–13; 8:45 am] 

BILLING CODE 4000–01–P 


ehiers on DSK2VPTVN1PROD with RULES_2