June 18, 2014

Fourth Draft OMB Package, Version Two – Part B

PPSS TO 10: Studies of Rural Education Achievement Program (REAP) Grantees

Subtask 3.5: Prepare Fourth Draft OMB Clearance Package

Contract Number GS-10F-0554N; Order Number ED-PEP-11-O-0090

SRI Project #P21494

Submitted to:

Andrew Abrams

Policy and Program Studies Service

U.S. Department of Education

400 Maryland Avenue, SW

Washington, DC 20202

Prepared by:

SRI International

Kyra Caspary, Chris Padilla, Nancy Adelman, Rebecca Schmidt, Erica Harbatkin, and Kaily Yee
1

SUPPORTING STATEMENT, PART B

PAPERWORK REDUCTION ACT SUBMISSION

B. Collection of Information Employing Statistical Methods

B.1. Sampling Design

Potential Respondent Universe

As discussed in Part A, the study will include a survey of a representative sample of REAP districts, 

interviews with REAP coordinators in all states receiving REAP funds in school year 2014–15,

1

and 

interviews with REAP coordinators in 30 districts receiving REAP funds. The REAP program office is 

interested in learning about REAP districts’ needs, their experiences of eligibility determination, how 

they use REAP funds, and the challenges and technical assistance needs they face. The proposed survey 

of 1,000 districts is appropriate to provide descriptive information on these questions using a 

representative sample of the universe of SRSA and RLIS grantees (see section B.2 for a discussion of the 

degree of precision this sample will provide).

Exhibit 1 provides an estimate of the sampling universe of REAP districts in school year 2014–15 using 

the numbers of districts in school year 2013–14. Nationwide, two-thirds of REAP districts received SRSA 

funds and one-third were eligible for RLIS funds in the 2013–14 school year. The REAP Office can provide 

lists of SRSA grantee districts and RLIS-eligible districts by November 2014 (in time for sampling), but 

does not maintain RLIS subgrantee lists. Therefore, the sampling frame for the survey comprises all 

districts that receive SRSA funds in school year 2014–15 and all districts that are eligible for RLIS funds in 

school year 2014–15. Although the number of districts actually receiving RLIS funds will be slightly lower 

than the number of RLIS-eligible districts, the RLIS take-up rate in school year 2013–14 was 

approximately 96 percent. The research team expects a similar take-up rate in school year 2014–15, so 

there should be few RLIS-eligible districts that are sampled for the survey and do not receive funds. The 

email invitation to REAP coordinators in RLIS districts will verify that the district received funds in school 

year 2014–15. Districts that did not receive funds will be excluded from the sample, and the research 

team will randomly select another district from the same stratum to take their place. 

 

1

All but two states (Hawaii and Vermont) had districts that received REAP funds in the 2013–14 school year. The 

research team anticipates that the number of states with REAP districts in 2014–15 will be close to 48. 
2

Exhibit 1. Universe of REAP Grantees by Grantee Type in 2013–142

RLIS and SRSA districts are concentrated in different parts of the United States, with the greatest 

number of SRSA districts in the Midwest and the greatest number of RLIS districts in the South (see 

Exhibit 2). In addition, SRSA districts are eligible to exercise a provision called REAP Flex that allows 

greater flexibility in the use of Title funds. In 2013–14, 49 percent of SRSA districts used the REAP Flex 

provision.

 

2

The research team will update this exhibit with 2014–15 data when they are available from the REAP Office.

REAP

(Rural Education Achievement Program) 

~6,300 districts (48 states)

SRSA

(Small, Rural School Achievement 

grants) 

4,299 districts

SRSA Flex 

~2,098 districts

SRSA non-Flex 

~2,201 districts

RLIS

(Rural and Low-Income School 

program) 

~2,088 districts
3

Exhibit 2. Total Number of SRSA and RLIS Districts per Region in 2013–143

Region Division

Number of RLIS 

Eligible Districts

Number of SRSA 

Districts

Midwest

East North Central 341 622

West North Central 170 1,094

Total 511 1,716

Northeast

Middle Atlantic 113 214

Northeast 75 270

Total 188 484

South

East South Central 363 18

South Atlantic 319 22

West South Central 471 793

Total 1,153 833

West

Mountain 114 689

Pacific 122 577

Total 236 1,266

Total 2,088 4,299

Sample Selection Process

Starting with the approximately 6,300 districts that receive SRSA funds or are eligible for RLIS funds in

school year 2014–15, the survey sample selection process will ultimately identify approximately 

1,000 districts that are representative of each geographic region and grantee type. Upon receipt of all 

state lists of RLIS eligible and SRSA grantee districts in school year 2014–15, the research team will 

select a simple random sample of districts stratified by grantee type and census division. There are two 

program type categories and nine census divisions, resulting in 18 strata (see Exhibit 3).

The target sample for the study is 1,004 districts. Because the REAP office is interested in learning about 

the particular experiences of SRSA grantees in choosing whether or not to exercise REAP Flex and how 

they use this provision, the study will allocate two-thirds of the sample to SRSA districts (668 districts) 

and one-third to RLIS districts (336 districts). Because the Flex use rate is approximately 50 percent, this 

should yield a sample of approximately 334 SRSA districts that exercise Flex and 334 that do not. 

Because the lists of REAP-Flex users will not be available from the REAP Program Office to create the 

sampling frame, the study will not stratify SRSA grantees based on Flex use. 

The study’s Technical Working Group has stressed that rural districts vary greatly depending on the part 

of the country in which they are located. Therefore, to ensure adequate numbers of districts to make 

statements about REAP districts by region, the study will allocate equal numbers of districts to each 

region within each program type. This means the study will sample 84 RLIS districts and 167 SRSA 

districts in each region, as shown in Exhibit 3. Within each region and program type, the study will 

allocate the districts to each division proportionally based on the number of districts in the sampling 

universe. This sampling design ensures that the sample will have adequate precision to report on survey 

results by both program type and by region and should yield a sample of SRSA districts of which 

 

3

The research team will update this exhibit when 2014–15 data become available. The researchers expect the 

number of districts in each stratum to be similar in school year 2014–15.
4

approximately half exercise REAP Flex, allowing the study to report on survey results for REAP Flex users 

and nonusers. Therefore, the researchers will be able to report on the responses of RLIS subgrantees, 

SRSA Flex users and SRSA grantees who did not use Flex at the national level. The researchers will also 

be able to report the responses of REAP districts in each of the four regions, but sample size is not 

adequate to allow the study to report responses of RLIS or SRSA districts within each region. Because 

the number of RLIS and SRSA districts varies greatly by division and some strata will have few sampled 

districts, the study will not report the results of the survey by division. The study is stratifying at the 

division level to ensure sampling at least one district per populated stratum for purposes of the followup interviews, as described below. Likewise, the study will report on the responses of Flex districts at the 

national level, not broken down by region, because the predicted number of sampled Flex districts in 

some regions is too low to report results with precision. 

Exhibit 3. Estimated Sample of Districts by Region and Grantee Type4

Region Division

Number of RLIS 

Eligible Districts

Number of SRSA 

Districts

Estimated 

Number of 

Sampled REAPFlex Districts

Midwest

East North Central 56 61 13

West North Central 28 106 57

Total 84 167 70

Northeast

Middle Atlantic 50 74 21

Northeast 34 93 17

Total 84 167 38

South

East South Central 26 4 0

South Atlantic 23 4 3

West South Central 34 159 114

Total 83 167 117

West

Mountain 41 91 30

Pacific 43 76 57

Total 84 167 87

Total 336 668 312

The research team will use a similar stratification scheme to select districts for follow-up interviews to 

ensure the study includes interviews with districts in each region of the country; however, it will select 

only 30 districts, and they will be selected within census division.

This sample will not be representative of REAP districts overall. Instead, the words of the districts 

administrators will be used to illustrate findings from the survey. For this reason, the research team will 

begin with the sample of surveyed districts and select a subsample of 30 to be interviewed. The research 

team will select one district in each census division for RLIS grantees, and two districts in each division 

for SRSA grantees, for a total of 27 districts. The study will rely on REAP Flex use in 2013–14 to guide the 

selection of the two SRSA districts, with an effort to select one district that used Flex in 2013–14 and 

one that did not within each division, although this will not be possible in all strata. For example, none of 

the SRSA districts in the east south central division used Flex in 2013-14 school year, so in this stratum 

 

4

The research team will update the numbers in this table when 2014–15 data are available from the REAP Office.
5

the study will select two SRSA districts, neither of which used Flex in 2013-14. The study will also select

three extra districts for a total of 30 districts. The researchers will select one extra RLIS district in the 

west south central division, which has the largest number of RLIS-eligible districts. The researchers will 

also select two extra SRSA districts (one that exercised REAP Flex in 2013-14 and one that did not ) in the 

west north central region, which has the largest number of SRSA grantee districts. 

Finally, the research team will also select one REAP coordinator at the state level in each of the states 

that received REAP funds for an in-depth interview. 

B.2. Procedures for the Collection of Information

Once the study has determined the sample of approximately 1,000 districts for the survey, the research 

team will identify the respondent and contact info for each district. The research team will first ask the 

REAP Program Office for the contact name, phone number, and email address for the REAP coordinator 

in sampled SRSA districts. The REAP Program Office has provided the research team with a database of 

state REAP coordinators. For RLIS districts and SRSA districts where the REAP Program Office is unable to 

provide this information, the research team will ask the REAP coordinator in each state to identify a

REAP contact person for each sampled district and provide contact information; the study expects that 

this person will commonly be the district superintendent or a federal grant coordinator. 

The research team will email a link to a secure web-based version of the survey to these individuals in 

December 2014. The survey will remain open through April 2015. While the survey is open, the 

researchers will send weekly email reminders to the district contacts who have not yet responded. In 

May, the researchers will download the survey data and begin cleaning and analysis. Appendix A 

contains a full draft of the survey instrument.

The research team will also randomly select 30 of these district REAP coordinators (one in each stratum, 

and an extra district in the largest strata within each program type) to schedule an approximately 45-

minute phone interview with one district administrator per district.5These phone interviews will be 

conducted from January 2015 through March 2015. During the same time period, the research team will 

also interview one state-level REAP coordinator for approximately 45 minutes in each of the 48 states 

receiving REAP funds in 2013–14.

Statistical Methodology and Estimation Procedures

The interviews with state and district REAP coordinators involve qualitative data collection. Statistical 

methodology is not applicable. The research team will quantify the qualitative data collected from all 

state REAP coordinators when this quantification can be done reliably.

The study will examine descriptive statistics on each survey question. For example, the researchers will 

produce tables with the frequency and percentage of districts responding “Yes” to each subtopic in this 

survey question: “Were any of the following people involved in deciding how to spend your [RLIS][SRSA] 

funds?” These data will answer the question of which personnel are involved in deciding how to target 

REAP funds. Researchers will weight responses to the total number of districts in each stratum, and will 

report the standard error of each reported percentage in an appendix.

Where appropriate, the study will also examine survey responses by the district characteristics (see 

Exhibit 4). The source of these variables is either the eligibility spreadsheets maintained by the federal 

REAP Program Office (“Eligibility Spreadsheets”) or the Common Core of Data (CCD).

 

5

If there are empty strata in the 2014–15 school year, the research team will select additional SRSA districts in the 

next largest stratum.
6

Exhibit 4. Source of District Characteristics

Variable Source

Grantee type Eligibility Spreadsheets

Region CCD

Size CCD

Percent of students and families in poverty Eligibility Spreadsheets

Award Amount* Eligibility Spreadsheets

*Available only for SRSA grantees

The PPSS guidelines for categorizing district size classify districts with 2,500 or fewer students as “small.” 

However, this category applies to more than 90 percent of the districts that receive REAP funds. 

Therefore, the research team will break down “small” districts further into two groups of average daily 

attendance, at the median of average daily attendance for the districts with 2,500 or fewer students. 

Based on the 2013–14 SRSA grantee and RLIS-eligible districts, the three categories for district size 

would be: medium or large (more than 2,500 students in average daily attendance), small (346 to 

2,500 students), and smallest (345 or fewer students). For RLIS districts, ADA may serve as a proxy for 

award amount, which is not available from the REAP Program Office as it is for SRSA grantees.

Degree of Accuracy Needed

With an expected 85 percent response rate, the sample size described above will allow the study to 

provide estimates with a margin of error of no greater than plus or minus 6.1 percentage points for RLIS 

districts and for each group of SRSA Flex users and nonusers, 4.1 for SRSA districts overall, and 3.5

percentage points for the REAP program as a whole.

Unusual Problems Requiring Specialized Sampling Procedures

There are no unusual problems anticipated.

Use of Periodic Data Collection

The survey will be administered and the data collected only once. State and district coordinators will be 

interviewed only once. 

B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse

Response Rate

SRI has extensive experience in administering surveys and carrying out interviews in schools and districts 

for research purposes. SRI will provide each selected state and district with a letter describing the study 

and its importance for the field (see Appendix E). These letters will also include the purpose of the 

survey, information on why the district was selected, and how to learn more about the study. The 

research team will use additional key access strategies such as having a designated researcher as the 

primary contact for each district; sending email reminders weekly; using multiple methods (i.e., phone, 

email, mail if necessary) to communicate with the districts; and providing opportunities for district 

contacts to ask questions about the study. The research team anticipates an 85 percent response rate —

a realistic goal given two past surveys of REAP districts. A survey of district administrators in districts 

receiving RLIS funds in 2007–08 conducted by Berkeley Policy Associates achieved a response rate of 84 

percent (U.S. Department of Education, 2010). A 2005–06 survey of districts eligible for SRSA conducted 

by the Urban Institute achieved a response rate of 94 percent (U.S. Department of Education, 2007).
7

Generalizability of the Sample

The research design for the survey relies on a simple random sample stratified by grantee type and 

census division intended to capture descriptive information. As such, the findings from the survey will be 

generalizable to all districts that received REAP funds in school year 2013–14. It will also be 

generalizable to each geographic region and grantee type, when weighted to account for the 

stratification design. 

State interviews will be conducted with REAP coordinators in all states that received REAP funds in 

school year 2014–15. The findings will describe the practices and experiences of this population of state 

coordinators. The district interviews will not be generalizable and will be used to illustrate survey 

findings.

B.4. Test of Procedures and Methods

The research team has conducted internal pretesting of protocol items to ensure clarity. Additionally, 

the research team confirmed that all protocols tie back to the research questions, ensuring the 

protocols will capture all necessary information. In April of 2014 the research team piloted the survey 

with seven randomly selected district administrators to approximate the average respondent.

Many of the protocol questions have been adapted from relevant questions used in other REAP and SRI 

studies. For example, questions related to technical assistance have their roots in questions developed 

and used as part of a past study of RLIS districts (U.S. Department of Education, 2010). Questions related 

to the use of REAP Flex started with items used in a past study of districts eligible to exercise REAP Flex 

(U.S. Department of Education, 2007).

After the pilot participants tested the instruments (i.e., took the survey), the researchers conducted 

phone conversations with each of them to discuss clarity of wording and flow, how they interpreted the 

questions, and any other issues that may come up. The researchers revised the instruments based on 

this feedback.

B.5. Consultations on Statistical Aspects of the Design

The research team consulted with Dr. Harold Javitz, Distinguished Scientist at SRI International, on 

sampling for the survey. He can be reached at 650-859-5274.

Agency

Andrew Abrams of the U.S. Department of Education is the Contracting Officer’s Representative for the 

study. He can be reached at 202-401-1232.

Contractors

SRI International will be responsible for data collection and analysis, under the direction of 

Kyra Caspary, who can be reached at 503-477-4228.
8

References

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and 

Program Studies Service. (2007). Evaluation of Flexibility under No Child Left Behind: Volume III—The 

Rural Education Achievement Program (REAP Flex). Washington, D.C.

U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and 

Program Studies Service (2010). Evaluation of the Implementation of the Rural and Low-Income 

School (RLIS) Program: Final Report. Washington, D.C.