### June 18 , 2014



# Fourth **Draft OMB Package , Version Two**  **– Part B** 

**PPSS TO 10: Studies of Rural Education Achievement Program (REAP) Grantees**  **Subtask 3 .**  **5 : Prepare**  **Fourth Draft OMB**  **Clearance Package** Contract Number GS - 10F - 0554N; Order Number ED - PEP - 11 - O -0090 SRI Project # P 21494

**Submitted to:** Andrew Abrams Policy and Program Studies Service U.S. Department of Education 400 Maryland Avenue, SW Washington, DC 20202 **Prepared by:** SRI International Kyra Caspary, Chris Padilla, Nancy Adelman, Rebecca Schmidt, Erica Harbatkin , and Kaily Yee1 

## SUPPORTING STATEMENT, PART B **PAPERWORK REDUCTION ACT SUBMISSION B. Collection of Information Employing**  **Statistical Methods B.1. Sampling Design**  Potential Respondent Universe As discussed in Part A, t he study will include a survey of a representative sample of REAP districts, interviews with REAP coordinators in all states receiving REAP funds in school year 2014– 15 , 1 and interviews with REAP coordinators in 30 districts receiving REAP funds. The REAP program office is interested in learning about REAP districts’ needs, their experiences of eligibility determination, how they use REAP funds, and the challenge s and technical assistance needs they face. The proposed survey of 1,000 districts is appropriate to provide descriptive information on these questions using a representative sample of the universe of SRSA and RLIS grantees (see section B.2 for a discussio n of the degree of precision this sample will provide). Exhibit 1 provides an estimate of the sampling universe of REAP districts in school year 2014– 15 using the numb ers of districts in school year 2013– 14. Nationwide, two -thirds of REAP districts received SRSA funds and one -third were eligible for RLIS funds in the 2013 – 14 school year. The REAP Office can provide lists of SRSA grantee districts and RLIS - eligible distr icts by November 2014 (in time for sampling), but does not maintain RLIS subgrantee lists. Therefore, t he sampling frame for the survey comprises all districts that receive SRSA funds in school year 2014– 15 and all districts that are eligible for RLIS fund s in school year 2014– 15. Although the number of districts actually receiving RLIS funds will be slightly lower than the number of RLIS - eligible districts, the RLIS take - up rate in school year 2013 – 14 was approximately 96 percent. The research team expects a similar take - up rate in school year 2014– 15, so there should be few RLIS - eligible districts that are sampled for the survey and do not receive funds. The email invitation to REAP coordinators in RLIS districts will verify that the district received fund s in school year 2014– 15. Districts that did not receive funds will be excluded from the sample, and the research team will randomly select another district from the same stratum to take their place.

1 All but two states (Hawaii and Vermont) had districts that received REAP funds in the 2013– 14 school year. The research team anticipates that the number of states with REAP districts in 2014– 15 will be close to 48.2 

### Exhibit **1 .**  **Universe of REA P Grantees by Grantee Type in 2013**  **– 14**  **2**

RLIS and SRSA districts are concentrated in different parts of the United States, with the greatest number of SRSA districts in the Midwest and the greatest number of RLIS districts in the South (see Exhibit 2 ). In addition, SRSA districts are eligible to exercise a provision called REAP Flex that allows greater flexibility in the use of Title funds. In 2013– 14, 49 percent of SRSA districts used the REAP Flex provision .

2 The research team will update this exhibit with 2014 – 15 data when they are available from the REA P Office. REAP (Rural Education Achievement Program) ~6,300 districts (48 states)

SRSA (Small, Rural School Achievement

grants) 4,299 districts

SRSA Flex ~2,098 districts SRSA non - Flex ~2,201 districts RLIS

(Rural and Low- Income School

program) ~2,088 districts3 

### Exhibit **2 . Total**  **N umber of SRSA and RLIS**  **D istricts per**  **R egion**  **in 2013 –** **14 3**  **Region Division**  **Number of RLIS E**  **ligible D**  **istricts Number of SRSA**  **D istricts**  **Midwest** East North Central 341 622

West North Central 170 1,094

Total 511 1,716 **Northeast** Middle Atlantic 113 214

Northeast 75 270

Total 188 484 **South** East South Central 363 18

South Atlantic 319 22 West South Central 471 793

Total 1, 153 833 **West** Mountain 114 689

Pacific 122 577

Total 236 1,266 **Total 2,088**  **4,299** 

### Sample Selection Process Starting with the approximately 6,300 districts that receive SRSA funds or are eligible for RLIS funds in school year 2014– 15, the survey sample selection process will ultimately identify approximately 1,000 districts that are representative of each geographic region and grantee type. Upon receipt of all state lists of RLIS eligible and SRSA grantee districts in school year 2014– 15, the research team will select a simple random sample of districts stratified by grantee type and census division. There are two program type categories and nine c ensus divisions, resulting in 18 strata (see Exhibit 3 ) . The target sample for the study is 1,004 districts. Because t he REAP office is interested in learning about the particular experiences of SRSA grantees in choosing whether or not to exercise REAP Flex and how they use this provision , the study will allocate two -thirds of the sample to SRSA districts (668 districts) and one -third to RLIS districts (336 districts) . Because the Flex use rate is approximately 50 percent, this should yield a sample of a pproximately 334 SRSA districts that exercise Flex and 334 that do not. Because the lists of REAP - Flex users will not be available from the REAP Program Office to create the sampling frame, the study will not stratify SRSA grantees based on Flex use. The study’s Technical Working Group has stressed that rural districts vary greatly depending on the part of the country in which they are located . Therefore, t o ensure adequate numbers of districts to make statements about REAP districts by region, the study will allocate equal numbers of districts to each region within each program type. This means the study will sample 84 RLIS districts and 167 SRSA districts in each region, as shown in Exhibit 3 . Within each region and program type, the study will allocate the districts to each division proportionally based on the number of districts in the sampling universe. This sampling design ensures that the sample will have adequ ate precision to report on survey results by both program type and by region and should yield a sample of SRSA districts of which
3 The research team will update this exhibit when 2014– 15 data become available. The researchers expect the number of districts in each stratum to be similar in school year 2014– 15.4

approximately half exercise REAP Flex, allowing the study to report on survey results for REAP Flex users and nonusers. Therefore, the researchers will be able to report on the responses of RLIS subgrantees, SRSA Flex users and SRSA grantees who did not use Flex at the national level. The researchers will also be able to report the responses of REAP districts in each of the four regions, but sample size is not adequate to allow the study to report responses of RLIS or SRSA districts within each region. Because the number of RLIS and SRSA districts varies greatly by division and some strata will have few sampled districts, the stud y will not report the results of the survey by division. The study is stratifying at the division level to ensure sampling at least one district per populated stratum for purposes of the follow up interviews, as described below. Likewise, the study will re port on the responses of Flex districts at the national level, not broken down by region, because the predicted number of sampled Flex districts in some regions is too low to report results with precision .



### Exhibit **3 . Estimated**  **S ample of**  **D istricts by**  **R egion and**  **G rantee**  **T ype**  **4**

**Region Division**  **Number of RLIS E**  **ligible D**  **istricts Number of SRSA**  **D istricts**  **Estimated**

**N umber of**  **S ampled REAP** **** **Flex D**  **istricts Midwest**  East North Central 56 61 13

West North Central 28 106 57

Total 84 167 70 **Northeast** Middle Atlantic 50 74 21

Northeast 34 93 17

Total 84 167 38 **South** East South Central 26 4 0

South Atlantic 23 4 3 West South Central 34 159 114

Total 83 167 117 **West** Mountain 41 91 30

Pacific 43 76 57

Total 84 167 87 **Total 336**  **668 312**  The research team will use a similar stratification scheme to select districts for follow - up interviews to ensure the study includes interviews with districts in each region of the country; however, it will select only 30 districts, and they will be selected within census division . T his sample will not be representative of REAP districts overall. Instead, the words of the districts administrators will be used to illustrate findings from the survey. For this reason, the research team will begin with the sample of surveyed districts and select a subsample of 30 to be interviewed. The research team will select one district in each census division for RLIS grantees, and two districts in each division for SRSA grantees , for a total of 27 d istricts . The study will rely on REAP Flex use in 2013– 14 to guide the selection of the two SRSA districts, with an effort to select one district that used Flex i n 2013 – 14 and one that did not within each division , although this will not be possible in all strata. For example, none of the SRSA districts in the east south central division used Flex in 2013 - 14 school year, so in this stratum

4 The research team will update the numbers in this table when 2014– 15 data are available from the REAP Office.5

the study will select two SRSA districts, neither of which used Flex in 2013 - 14 . The study will also select three extra districts for a total of 30 districts . The researchers will select one extra RLIS district in the west south central division , which has the largest number of RLIS - eligible districts. The researchers will also select two extra SRSA districts ( one that exe rcis ed REAP Flex in 2013 - 14 and one that did not ) in the west north central region , which has the largest number of SRSA grantee districts . Finally, the research team will also select one REAP coordinator at the state level in each of the states that rec eived REAP funds for an in - depth interview. 

### B.2. Procedures for the Collection of Information Once the study has determined the sample of approximately 1,000 districts for the survey, the research team will identify the respondent and contact info for eac h district. The research team will first ask the REAP Program Office for the contact name, phone number, and email address for the REAP coordinator in sampled SRSA districts. The REAP Program Office has provided the research team with a database of state R EAP coordinators. For RLIS districts and SRSA districts where the REAP Program Office is unable to provide this information, the research team will ask the REAP coordinator in each state to identify a REAP contact person for each sampled district and provi de contact information; the study expects that this person will commonly be the district superintendent or a federal grant coordinator. The research team will email a link to a secure w eb - based version of the survey to these individuals in December 2014. The survey will remain open through April 2015. While the survey is open, the researchers will send weekly email reminders to the district contacts who have not yet responded. In May, the researchers will download the survey data and begin cleaning and ana lysis. Appendix A contains a full draft of the survey instrument. The research team will also randomly select 30 of these district REAP coordinators (one in each stratum, and an extra district in the largest strat a within each program type) to schedule an approximately 45 - minute phone interview with one district administrator per district. 5 These phone interviews will be conducted from January 2015 through March 2015. During the same time period, the research team will also interview one state - level REAP co ordinator for approximately 45 minutes in each of the 48 states receiving REAP funds in 2013 – 14. 

### Statistical Methodology and Estimation Procedures The interviews with state and district REAP coordinators involve qualitative data collection. Statistical methodology is not applicable. The research team will quantify the qualitative data collected from all state REAP coordinators when this quantification can be done reliably. The study will examine descriptive statistics on each survey question. For example , the researchers will produce tables with the frequency and percentage of districts responding “Yes” to each subtopic in this survey question: “Were any of the following people involved in deciding how to spend your [RLIS][SRSA] funds?” These data will an swer the question of which personnel are involved in de ciding how to target REAP funds. Researchers will weight responses to the total number of districts in each stratum, and will report the standard error of each reported percentage in an appendix. Where appropriate, the study will also examine survey responses by the district characteristics (see Exhibit 4 ) . The source of these variables is either t he eligibility spreadsheets maintained by the federal REAP Program Office (“Eligibility Spreadsheets”) or the Common Core of Data (CCD).

5 If there are empty strata in the 2014– 15 school year, the research team will select additional SRSA districts in the next largest stratum.6 

### Exhibit **4 . Source of District Characteristics**  **Variable Source**  Grantee type Eligibility Spreadsheets Region CCD Size CCD Percent of students and families in poverty Eligibility Spreadsheets Award Amount* Eligibility Spreadsheets *Available only for SRSA grantees The PPSS guidelines for categorizing district size classify districts with 2,500 or fewer students as “small.” However, this category applies to more than 90 percent of the districts that receive REAP funds. Therefore, the research team will break down “small” districts further into two groups of average daily attendance, at the median of average daily attendance for the districts with 2,500 or fewer students. Based on the 2013 – 14 SRSA grantee and RLIS - eligible districts, the three categories for district size would be: medium or large ( more than 2,500 students in average daily attendance), small (346 to 2,500 students) , and small est (345 or fewer students). For RLIS districts, ADA may serve as a proxy for award amount, which is not available from the REAP Program Office as it is for SRSA grantees. 

### Degree of Accuracy Needed With an expected 85 percent response rate, the sample size described above will allow the study to provide estimates with a margin of error of no greater than plus or minus 6.1 percentage points for RLIS districts and for each group of SRSA Flex users and nonusers, 4.1 for SRSA districts overall, and 3.5 percentage points for the REAP program as a whole. 

### Unusual Problems Requiring Specialized Sampling Procedures There are no unusual problems anticipated. 

### Use of Periodic Data Collection The survey will be administered and the data collected only once. State and district coordinators will be interviewed only once. 

### B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse Response Rate SRI has extensive experience in ad ministering surveys and carrying out interviews in schools and districts for research purposes. SRI will provide each selected state and district with a letter describing the study and its importance for the field (see Appendi x E) . These letters will also include the purpose of the survey, information on why the district was selected, and how to learn more about the study. The research team will use additional key access strategies such as having a designated researcher as the primary contact for each distr ict; sending email reminders weekly; using multiple methods (i.e., phone, email, mail if necessary) to communicate with the districts; and providing opportunities for district contacts to ask questions about the study. The research team anticipates an 85 p ercent response rate — a realistic goal given two past surveys of REAP districts. A survey of district administrators in districts receiving RLIS funds in 2007–08 conducted by Berkeley Policy Associates achieved a response rate of 84 percent (U.S. Departme nt of Education, 2010). A 2005–06 survey of districts eligible for SRSA conducted by the Urban Institute achieved a response rate of 94 percent (U.S. Department of Education, 2007 ).7



### Generalizability of the Sample The research design for the survey relies o n a simple random sample stratified by grantee type and census division intended to capture descriptive information. As such, the findings from the survey will be generalizable to all districts that received REAP funds in school year 2013– 14 . It will also be generalizable to each geographic region and grantee type , when weighted to account for the stratification design . State interviews will be conducted with REAP coordinators in all states that received REAP funds in school year 2014– 15 . The findings will describe the practices and experiences of this population of state coordinators. The district interviews will not be generalizable and will be used to illustrate survey findings. 

### B.4. Test of Procedures and Methods The research team has condu cted internal pretesting of protocol items to ensure clarity. Additionally, the research team confirmed that all protocols tie back to the research questions, ensuring the protocols will capture all necessary information. In April of 2014 the research team piloted the survey with seven randomly selected district administrators to approximate the average respondent. Many of the protocol questions have been adapted from relevant questions used in other REAP and SRI studies. For example, questions related to t echnical assistance have their roots in questions developed and used as part of a past study of RLIS districts (U.S. Department of Education, 2010) . Questions related to the use of REAP Flex started with items used in a past study of districts eligible to exercise REAP Flex (U.S. Department of Education, 2007). After the pilot participants tested the instruments (i.e., took the survey), the researchers conducted phone conversations with each of them to discuss clarity of wording and flow, how they interpreted the questions, and any other issues that may come up. The researchers revised the instruments based on this feedback. 

### B.5. **Consultations on Statistical Aspects of the Design** The research team consulted with Dr. Harold Javitz, Distinguished Sci entist at SRI International, on sampling for the survey . He can be reached at 650 - 859 - 5274. 

### Agency Andrew Abrams of the U.S. Department of Education is the Contracting Officer’s Representative for the study. He can be reached at 202 -401 - 1232. 

### Contractors SRI International will be responsible for data collection and analysis, under the direction of Kyra Caspary, who can be reached at 503 -477 -4228 .8



### R**eferences** U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy a nd Program Studies Service. (2007 ) . Evaluation of Flexibility under No Child Left Behind: Volume III — The Rural Education Achievement Program (REAP Flex) . Washington, D.C. U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service (2010) . Evaluation of the Implementation of the Rural and Low- Income School (RLIS) Program: Final Report . Washington, D.C.