October 1, 2014

Fourth Draft OMB Package, Version Two – Part B

(revised)

PPSS TO 10: Studies of Rural Education Achievement Program (REAP) Grantees

Subtask 3.5: Prepare Fourth Draft OMB Clearance Package

Contract Number GS-10F-0554N; Order Number ED-PEP-11-O-0090

SRI Project #P21494

Submitted to:

Andrew Abrams

Policy and Program Studies Service

U.S. Department of Education

400 Maryland Avenue, SW

Washington, DC 20202

Prepared by:

SRI International

Kyra Caspary, Chris Padilla, Nancy Adelman, Rebecca Schmidt, Erica Harbatkin, and Kaily Yee
1

SUPPORTING STATEMENT, PART B

PAPERWORK REDUCTION ACT SUBMISSION

B. Collection of Information Employing Statistical Methods

B.1. Sampling Design

Potential Respondent Universe

As discussed in Part A, the study will include a survey of a representative sample of REAP districts, 

interviews with REAP coordinators in all states receiving REAP funds in school year 2014–15,

1

and 

interviews with REAP coordinators in 30 districts receiving REAP funds. The REAP program office is 

interested in learning about REAP districts’ needs, their experiences with eligibility determination, how 

they use REAP funds, and the challenges and technical assistance needs they face. The proposed survey 

of 1,000 districts is appropriate to provide descriptive information on these questions using a 

representative sample of the universe of SRSA and RLIS grantees (see section B.2 for a discussion of the 

degree of precision this sample will provide).

Exhibit 1 provides an estimate of the sampling universe of REAP districts in school year 2014–15 using 

the numbers of districts in school year 2013–14. Nationwide, two-thirds of REAP districts received SRSA 

funds and one-third were eligible for RLIS funds in the 2013–14 school year. The REAP Office can provide 

lists of SRSA grantee districts and RLIS-eligible districts by November 2014 (in time for sampling), but 

does not maintain RLIS subgrantee lists. Therefore, the sampling frame for the survey comprises all 

districts that receive SRSA funds in school year 2014–15 and all districts that are eligible for RLIS funds in 

school year 2014–15. Although the number of districts actually receiving RLIS funds will be slightly lower 

than the number of RLIS-eligible districts, the RLIS take-up rate in school year 2013–14 was 

approximately 96 percent. The research team expects a similar take-up rate in school year 2014–15, so 

there should be few RLIS-eligible districts that are sampled for the survey and do not receive funds. The 

email invitation to REAP coordinators in RLIS districts will verify that the district received funds in school 

year 2014–15. Districts that did not receive funds will be excluded from the sample, and the research 

team will randomly select another district from the same stratum to take their place. 

 

1

All but two states (Hawaii and Vermont) had districts that received REAP funds in the 2013–14 school year. The 

research team anticipates that the number of states with REAP districts in 2014–15 will be close to 48. 
2

Exhibit 1. Universe of REAP Grantees by Grantee Type in 2013–142

RLIS and SRSA districts are concentrated in different parts of the United States, with the greatest 

number of SRSA districts in the Midwest and the greatest number of RLIS districts in the South (see 

Exhibit 2). In addition, SRSA districts are eligible to exercise a provision called REAP Flex that allows 

greater flexibility in the use of Title funds. In 2013–14, 49 percent of SRSA districts used the REAP Flex 

provision.

 

2

The research team will update this exhibit with 2014–15 data when they are available from the REAP Office.

REAP

(Rural Education Achievement Program) 

~6,300 districts (48 states)

SRSA

(Small, Rural School Achievement 

grants) 

4,299 districts

SRSA Flex 

~2,098 districts

SRSA non-Flex 

~2,201 districts

RLIS

(Rural and Low-Income School 

program) 

~2,088 districts
3

Exhibit 2. Total Number of SRSA and RLIS Districts per Region in 2013–143

Region Census Division

Number of RLIS 

Eligible Districts

Number of SRSA 

Districts

Midwest

East North Central 341 622

West North Central 170 1,094

Total 511 1,716

Northeast

Middle Atlantic 113 214

Northeast 75 270

Total 188 484

South

East South Central 363 18

South Atlantic 319 22

West South Central 471 793

Total 1,153 833

West

Mountain 114 689

Pacific 122 577

Total 236 1,266

Total 2,088 4,299

Sample Selection Process

Survey of REAP Districts

Starting with the approximately 6,300 districts that receive SRSA funds or are eligible for RLIS funds in

school year 2014–15, the survey sample selection process will ultimately identify approximately 

1,000 districts that are representative of each geographic region and grantee type. Upon receipt of all 

state lists of RLIS eligible and SRSA grantee districts in school year 2014–15, the research team will 

select a stratified random sample of districts stratified by grantee type and census division.4There are 

two program type categories and nine census divisions, resulting in 18 strata (see Exhibit 3).

The target sample for the study is 1,004 districts. Because the REAP office is interested in learning about 

the particular experiences of SRSA grantees in choosing whether or not to exercise REAP Flex and how 

they use this provision, the study will allocate two-thirds of the sample to SRSA districts (668 districts) 

and one-third to RLIS districts (336 districts). Because the Flex use rate is approximately 50 percent, this 

should yield a sample of approximately 334 SRSA districts that exercise Flex and 334 that do not. 

Because the lists of REAP-Flex users will not be available from the REAP Program Office to create the 

sampling frame, the study will not stratify SRSA grantees based on Flex use but will include questions 

about districts’ use of REAP Flex on the survey. 

The study’s Technical Working Group has stressed that rural districts vary greatly depending on the part 

of the country in which they are located. Therefore, to ensure adequate numbers of districts to make 

statements about REAP districts by region, the study will allocate equal numbers of sampled districts to 

each region within each program type. This means the study will sample 84 RLIS districts and 167 SRSA 

 

3

The research team will update this exhibit when 2014–15 data become available. The researchers expect the 

number of districts in each stratum to be similar in school year 2014–15.

4

In analysis and reporting, the study will weight responses to the total number of districts in each stratum.
4

districts in each region, as shown in Exhibit 3. Within each region and program type, the study will 

allocate the districts to each census division proportionally based on the number of districts in the 

sampling universe. This sampling design ensures that the sample will have adequate precision to report 

on survey results by both program type and by region and should yield a sample of SRSA districts of 

which approximately half exercise REAP Flex, allowing the study to report on survey results for REAP 

Flex users and nonusers. Therefore, the researchers will be able to report on the responses of RLIS 

subgrantees, SRSA Flex users and SRSA grantees who did not use Flex at the national level. The 

researchers will also be able to report the responses of REAP districts in each of the four regions, but 

sample size is not adequate to allow the study to report responses of RLIS or SRSA districts within each 

region. Because the number of RLIS and SRSA districts varies greatly by census division and some strata 

will have few sampled districts, the study will not report the results of the survey by census division. The 

study is stratifying at the census division level to ensure sampling at least one district per populated 

stratum for purposes of the follow-up interviews, as described below. Likewise, the study will report on 

the responses of Flex districts at the national level, not broken down by region, because the predicted 

number of sampled Flex districts in some regions is too low to report results with precision. 

Exhibit 3. Estimated Sample of Districts by Region and Grantee Type5

Region Census Division

Number of RLIS 

Eligible Districts

Number of SRSA 

Districts

Estimated 

Number of 

Sampled REAPFlex Districts

Midwest

East North Central 56 61 13

West North Central 28 106 57

Total 84 167 70

Northeast

Middle Atlantic 50 74 21

Northeast 34 93 17

Total 84 167 38

South

East South Central 26 4 0

South Atlantic 23 4 3

West South Central 34 159 114

Total 83 167 117

West

Mountain 41 91 30

Pacific 43 76 57

Total 84 167 87

Total 336 668 312

Follow-Up Interviews

The research team will use a similar stratification scheme to select a subsample of districts for follow-up 

interviews. This sample will not be representative of REAP districts overall. Instead, the words of the 

districts administrators will be used to illustrate findings from the survey. For this reason, the research 

team will begin with the sample of surveyed districts and select a subsample of 30 to be interviewed. 

The research team will select one district in each census division for RLIS grantees, and two districts in 

each census division for SRSA grantees, for a total of 27 districts. The study will rely on REAP Flex use in 

 

5

The research team will update the numbers in this table when 2014–15 data are available from the REAP Office.
5

2013–14 to guide the selection of the two SRSA districts within each census division, with an effort to 

select one district that used Flex in 2013–14 and one that did not, although this will not be possible in all 

strata. For example, none of the SRSA districts in the east south central census division used Flex in 

2013-14 school year, so in this stratum the study will select two SRSA districts, neither of which used 

Flex in 2013-14. The study will also select three extra districts for a total of 30 districts. The researchers 

will select one extra RLIS district in the west south central census division, which has the largest number 

of RLIS-eligible districts. The researchers will also select two extra SRSA districts (one that exercised

REAP Flex in 2013-14 and one that did not ) in the west north central region, which has the largest 

number of SRSA grantee districts. 

Finally, the research team will also conduct in-depth interviews with each of the state-level REAP 

coordinators (among states that received REAP funds in 2013-14).

B.2. Procedures for the Collection of Information

Once the study has determined the sample of approximately 1,000 districts for the survey, the research 

team will ask the state REAP coordinator in each of the 48 participating states to identify the respondent 

and contact information for each sampled district in his/her state, via a template (Appendix F). The 

study expects that this person will commonly be the district superintendent or a federal grant 

coordinator. 

The research team will email a link to a secure web-based version of the survey to these individuals in 

December 2014. The survey will remain open through April 2015. While the survey is open, the 

researchers will send weekly email reminders to the district contacts who have not yet responded. In 

May, the researchers will download the survey data and begin cleaning and analysis. Appendix A 

contains a full draft of the survey instrument.

For the follow-up district interviews, the research team will randomly select 30 of these district REAP 

coordinators (one in each stratum, and an extra district in the largest strata within each program type) 

to schedule an approximately 45-minute phone interview.6These phone interviews will be conducted 

from January 2015 through March 2015. During the same time period, the research team will also 

interview one state-level REAP coordinator for approximately 45 minutes in each of the 48 states 

receiving REAP funds in 2013–14.

Statistical Methodology and Estimation Procedures

The interviews with state and district REAP coordinators involve qualitative data collection. Statistical 

methodology is not applicable. The research team will quantify the qualitative data collected from all 

state REAP coordinators when this quantification can be done reliably.

The study will examine descriptive statistics on each survey question. For example, the researchers will 

produce tables with the frequency and percentage of districts responding “Yes” to each subtopic in this 

survey question: “Were any of the following people involved in deciding how to spend your [RLIS][SRSA] 

funds?” These data will answer the question of which personnel are involved in deciding how to target 

REAP funds. Researchers will weight responses to the total number of districts in each stratum, and will 

report the standard error of each reported percentage in an appendix.

Where appropriate, the study will also examine survey responses by the district characteristics (see 

Exhibit 4). The source of these variables is either the eligibility spreadsheets maintained by the federal 

REAP Program Office (“Eligibility Spreadsheets”) or the Common Core of Data (CCD).

 

6

If there are empty strata in the 2014–15 school year, the research team will select additional SRSA districts in the 

next largest stratum.
6

Exhibit 4. Source of District Characteristics

Variable Source

Grantee type Eligibility Spreadsheets

Region CCD

Size CCD

Percent of students and families in poverty Eligibility Spreadsheets

Award Amount* Eligibility Spreadsheets

*Available only for SRSA grantees

The PPSS guidelines for categorizing district size classify districts with 2,500 or fewer students as “small.” 

However, this category applies to more than 90 percent of the districts that receive REAP funds. 

Therefore, the research team will further classify “small” districts into two groups based on the median 

average daily attendance (ADA) for the districts with 2,500 or fewer students. Based on the 2013–14 

SRSA grantee and RLIS-eligible districts, the three categories for district size would be: medium or large

(more than 2,500 students with average daily attendance), small (346 to 2,500 students), and smallest

(345 or fewer students). Award amount is not available from the REAP Program Office as it is for SRSA 

grantees. States may allocate RLIS funds to districts by competition or by formula. States that choose to

allocate RLIS funds by formula may use ADA or an alternative formula. Through state interview, the 

study will confirm or refute the impression of REAP Program Office staff that all states allocate RLIS 

funds by formula using ADA. If confirmed,, ADA may serve as a proxy for award amount for RLIS districts.

Degree of Accuracy Needed

With an expected 85 percent response rate, the sample size described above will allow the study to 

provide estimates with a margin of error of no greater than plus or minus 6.1 percentage points for RLIS 

districts and for each group of SRSA Flex users and nonusers, 4.2 for SRSA districts overall, 3.4

percentage points for the REAP program as a whole, and 6.9 percentage points for the REAP program 

within each census region. These margin of error values were obtained by simulating the survey results 

20 times and averaging the 20 margin of error estimates. The standard deviation of the margin of error 

estimates was less than 0.03 percentage points.

Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems anticipated.

Use of Periodic Data Collection

The survey will be administered and the data collected only once. State and district coordinators will be 

interviewed only once. 

B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse

Response Rate

SRI has extensive experience in administering surveys and carrying out interviews in schools and districts 

for research purposes. SRI will provide each selected state and district with a letter describing the study 

and its importance for the field (see Appendix E). These letters will also include the purpose of the 

survey, information on why the district was selected, and how to learn more about the study. The 

research team will use additional key access strategies such as having a designated researcher as the 

primary contact for each district; sending email reminders weekly; making weekly reminder phone calls 

beginning in the fourth week of data collection; and sending a mail reminder, if necessary to 

communicate with the districts; and providing opportunities for district contacts to ask questions about 
7

the study. The research team anticipates an 85 percent response rate — a realistic goal given two past 

surveys of REAP districts —and will persist with these efforts until this response rate is achieved or the 

data collection period ends. A survey of district administrators in districts receiving RLIS funds in 2007–

08 conducted by Berkeley Policy Associates achieved a response rate of 84 percent (U.S. Department of 

Education, 2010). A 2005–06 survey of districts eligible for SRSA conducted by the Urban Institute 

achieved a response rate of 94 percent (U.S. Department of Education, 2007).

Generalizability of the Sample

The research design for the survey relies on a simple random sample stratified by grantee type and 

census division intended to capture descriptive information. As such, the findings from the survey will be 

generalizable to all districts that received REAP funds in school year 2013–14. It will also be 

generalizable to each geographic region and grantee type, when weighted to account for the 

stratification design. 

State interviews will be conducted with REAP coordinators in all states that received REAP funds in 

school year 2014–15. The findings will describe the practices and experiences of this population of state 

coordinators. The district interviews will not be generalizable but can and will be used to illustrate 

survey findings.

B.4. Test of Procedures and Methods

The research team has conducted internal pretesting of protocol items to ensure clarity. Additionally, 

the research team confirmed that all protocols are aligned with the research questions, ensuring the 

protocols will capture all necessary information. In April of 2014 the research team piloted the survey 

with seven randomly selected district administrators across the four census regions to approximate the 

average respondent.

Many of the protocol questions have been adapted from relevant questions used in other REAP and SRI 

studies. For example, questions related to technical assistance have their roots in questions developed 

and used as part of a past study of RLIS districts (U.S. Department of Education, 2010). Questions related 

to the use of REAP Flex started with items used in a past study of districts eligible to exercise REAP Flex 

(U.S. Department of Education, 2007).

After the pilot participants tested the instruments (i.e., took the survey), the researchers conducted 

phone conversations with each of them to discuss clarity of wording and flow, how they interpreted the 

questions, and any other issues that may come up. The researchers revised the instruments based on 

this feedback.

B.5. Consultations on Statistical Aspects of the Design

The research team consulted with Dr. Harold Javitz, Distinguished Scientist at SRI International, on 

sampling for the survey. He can be reached at 650-859-5274.

Agency

Andrew Abrams of the U.S. Department of Education is the Contracting Officer’s Representative for the 

study. He can be reached at 202-401-1232.

Contractors

SRI International will be responsible for data collection and analysis, under the direction of 

Kyra Caspary, who can be reached at 503-477-4228.
8

References

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and 

Program Studies Service. (2007). Evaluation of Flexibility under No Child Left Behind: Volume III—The 

Rural Education Achievement Program (REAP Flex). Washington, D.C.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and 

Program Studies Service (2010). Evaluation of the Implementation of the Rural and Low-Income 

School (RLIS) Program: Final Report. Washington, D.C.