### September 22 , 201 4 

# U.S. Department of Education

**PPSS TO 13. High School Reform Study**  **Information Package** 

### September 22 , 201 4 

# OMB Clearance Package

PPSS Task Order 13. High School Reform Study

Contract Number GS
- 10F
-05 5 4N, Order # ED

- PEP
- 11
-O
-0090/TO13 SRI Project #P21496

**Submitted to:** Joanne Bogart Policy and Program Studies Service U.S. Department of Education 400 Maryland Avenue, SW Washington, DC 20202 **Prepared by:** Christine Padilla, Ellen Schiller, and Rebecca Schmidt, SRI International Deborah Herget, RT I InternationalP21496, Task 4: Draft OMB Package 4 

## Revised Supporting Statement, Part B **Paperwork Reduction Act Submission B. Collection of Information Employing**  **Statistical Methods B.1. Sampling Design**  Potential Respondent Universe The study will include a survey of a nationally representative sample of high school principals. The study will examine the prevalence and characteristics of high school reform strategi es, particularly dropout prevention strategies. The proposed survey of 2,0 88 schools is appropriate to provide descriptive information on these questions based on a representative sample of the universe of public high schools (see section B.2 for a discuss ion of the degree of precision this sample will provide). Selection of the sample and administration of the survey will be carried out by the Research Triangle Institute (RTI ). The sampling frame for this survey is based on the Common Core of Data (CCD)
. Public high schools provid ing instruction to 12 th

-grade students in the fall of 201 0 will be included unless ( 1 ) the lowest offered grade is 11 th grade or higher, ( 2 ) there are fewer than five stud ents in grades 9 through 12 , (3 ) the percentage of students enrolled in grades 9 through 12 is under 20 percent of the total school enrollment and the total num ber of stud ents in grades 9 through 12 is fewer than 20, or the school name contains one of nine keywords indicating juvenile d etention center or hospital. Of the 103,813 total schools listed in the 2010
– 11 CCD, 22,447 high schools meet the criteria to be included in the sampling frame. 

### Sample Selection Process The sample will be selected using a stratified random sampling approach. The research team at RTI classified s chool s in the sampling frame into locale (urban, suburban, and rural) and graduation rate (high and low) strata. RTI combined the 12 locale codes provided in the CCD into three broad er groups. Exhibit 2 shows the three values of the locale sampling strata and their relationship to the 12 locale codes available in the CCD.P21496, Task 4: Draft OMB Package 5 

### Exhibit **2 . Sampling Locale and CCD Locale Codes**  **Sampling Locale**  **CCD Locale** Urban 11
- City, Large Territory Urban 12
- City, Midsize Territory Urban 13
- City, Small Territory Suburban 21
- Suburb, Large Territory Suburban 22
- Suburb, Midsize Territory Suburban 23
- Suburb, Small Territory Suburban 31
- Town, Fringe Territory Suburban 32
- Town, Distant Territory Suburban 33
- Town, Remote Territory Rural 41
- Rural, Fringe Census

- defined Rural Territory Rural 42
- Rural, Distant Census

- defined Rural Territory Rural 43
- Rural, Remote Census

-defined Rural Territory Graduation rates for each school were gathered from the ED Facts data bases, which is a central repository of performance data supplied by K

- 12 state education agencies. To identify the two graduation rate strata ( high and low ), t he RTI research team established a threshold of 80 percent to define high graduation rates based on the all rate

_ 1011 variable on the ED Facts website . The decision to use an 80 percent threshold was made because approximately 50 percent of schools reported graduation rates of 80 percent or less . Because schools often reported graduation rates in categories (e.g., 80 to 85 percent ) rather than a specific number, some additional decision rules were necessary. There were three categories that included 80 percent as a lower bound ( 80 to 85 percent; 80 to 89 percent; and greater than or equal to 80 percent ). All schools included in any of these three categories were included in the “high graduation rate” category. There were 3, 302 schools without graduation rate information in ED Facts . RTI used an imputation approach to assign these schools to either the high or low graduation rate stratum. The imputation process began by examining the distribution of the high/low graduation rate classification for the 19, 145 schools by sampling loca le. Exhibit 3 shows the distribution of these 19, 145 schools by sampling locale and graduation rate classification. 

### Exhibit **3 :**  **Initial Distribution of HSRS Sampling Frame Schools by Sampling Strata** 

**Graduation Rate** **\ Locale**  **Rural Suburban**  **Urban Total**  **Total** 7,828 7,0 5 7 4,2 6 0 19, 1 45 **High** 5,35 1 4 , 449 1,74 8 11,548 **Low** 2,4 77 2,6 08 2,5 12 7, 597 The percentage of schools classified as high graduation rate was calculated separately for each locale sampling stratum; 68. 4 percent of rural schools were classified as high graduation rate, 6 3 percent of suburban schools were classified as high graduation rate, and 4 1.0 percent of urban schools were classified as high graduation rate. Each of the 3, 302 schools with unknown graduation rates w asP21496, Task 4: Draft OMB Package 6

randomly assigned to the high graduation rate stratum with probability 68. 4 if the school was classified as rural, with probability 6 3.0 if the school was classified as suburban, and with probability 4 1.0 if the school was classified as urban. T he final distribution of the 22, 447 schools by strata is shown in Exhibit
. 

### Exhibit **4 .**  **Final Distribution of HSRS Sampling Frame Schools by Sampling S** **trata**

**Graduation Rate** **\ Locale**  **Rural Suburban**  **Urban Total**  **Total** 9, 141 8, 180 5, 1 26 22, 447 **High** 6,260 5, 1 64 2, 086 13, 510 **Low** 2,881 3,0 16 3,040 8,937


### B.2. Procedures for Collection of Information Initial Contact and Follow up Procedures for collecting data for the High School Reform Study ( HSRS ) will commence upon receipt of OMB approval anticipated November 2014. Once OMB approval is received, a letter will be sent to school districts to notify them that the study team will co ntact selected schools in their district in about a week. If it is known that a school district requires a research application prior to contacting schools, that application will be submitted along with the district letter. Project staff will be prepared t o comply with all district protocols and will not contact schools until district approval is received. The district letter will include a toll

-free number that they can call to alert project staff if there are requirements to be met before contacting schoo ls. If project staff are already in contact with schools and are informed by school or school district staff that specific district requirements must be met before schools may commit to participating in a research study, school contact will cease until the district requirements are met and approval is received to resume contact with the school
. Data collection with school administrators will commence about one week after district letters are sent. A letter and study fact sheet will be mailed to the principa l that will emphasize the importance of the study and will provide the URL, user ID, and password to access the online survey . School administrators will be informed that they may comp lete the 30

- minute survey or ask a designee to do so. A designee may be any person at the school who is knowledgeable about school policies and programs. If they prefer to have someone else complet e the form, the administrator will be asked to share the login information on the notification letter. The first question on the su rvey will ask for the name and title of the person completing the survey . A toll

-free number and email address will be provided so school staff may contact project staff with questions as needed. Copies of letters and the fact sheet can be found in Appendix B
. Prompting calls will begin with nonrespondin g schools after about three weeks to answer any questions about the study, remind administrators that they may delegate the survey to another knowledgeable staff member, and encourage participation. Administrators may co mplete the survey as an interview by phone should they prefer a phone inte rview to an online survey
. Up to three mailings will be sent to the school over the data collection period to remind staff to comp lete the online survey. Email re minders will be sent every 10 days.P21496, Task 4: Draft OMB Package 7

The research team will email a link to a secure web

- based version of the survey to the school principal in eac h of the 2,088 sampled schools o n December 1, 2014 . The survey will remain open through June 1, 2015. In June, the researchers will download the survey data and begin cleaning and analysis. A full draft of the survey is included in Appendix A. 

### Statistical Methodology and Estimation Procedures The study will examine descriptive statistics on each survey question. F or example, the researchers will produce tables with the frequency and percentage of schools responding “Yes” to each question that asks if a particular strategy is available at the school such as: “In the 2014

- 15 school year, does your school have formal adult mentor(s)?” These data will answer the question about the prevalence and key high school reforms in the nations’ public high schools. Researchers will weight responses to the total number of schools in each stratum, and will report the standard error of each reported percentage in the appendix. Where appropriate, the study may also examine survey responses by school characteristics such as school type (e.g., regular school, alternative school) and school enrollment. The source of these variables is ei ther the Common Core of Data (CCD) or survey background questions. 

### Degree of Accuracy Needed RTI determined sample sizes by estimating the number of responding sample members to achieve certain power requirements, adjusting that number upward to account for 25 percent school nonparticipation and for design effects arising out of potential weight adjustments 1 as well as imputation misclassification. RTI estimated precision requirements using four assumptions. First, statistical tests will employ an alpha of 0.05 and will have a required power of 80 percent . Second , the underlying proportion was assumed to be .30 for purposes of calculat ing sample sizes for estimates and tests of proportions. Third, the design effect will be less than 1.6. Finally, the resea rchers expect a 25 percent nonresponse or non participation rate by schools. The selected total sample size of 2,088 schools is designed to support comparisons between schools from any two strata such that there is 80 percent power to detect a 15 percent di fference in proportions. The sample of 2,088 is allocated equally to each of the six strata so that 348 schools will be sampled from each stratum in order to obtain 261 responding schools per stratum. These sample size calculations are sensitive to assumpt ions about nonresponse rates and to misclassification rates associated with imputation of graduation rate status. A reserve sample will be established, comprising 174 schools in each of the six sampling strata, so that additional schools are available if t he nonresponse rate is substantially higher than 25 percent . The reserve sample also could be used to supplement the initial sample of 2,088 schools if the imputation classification was found to have a high error rate. However, misclassified schools must n ot be excluded from data collection; rather, additional schools could be added from the reserve sample in order to try to obtain 261 responding schools in each sampling stratum. The degree to which this may be possible depends on actual field data collection experiences and costs. 

### Use of Periodic Data Collection The survey will only be administered once during the 2014
– 15 school year
.

1 Design effects quantify the extent to which the survey’s sampling error varies from the sampling error expected when using a simple random sample.P21496, Task 4: Draft OMB Package 8



### B.3. Methods for Maximizing Response Rate and Dealing with Nonresponse Response Rate The approach for gaining coo peration from school administrators includes a series of mailings and emails with follow

- up telephone calls to prompt for outstanding incomplete surveys. Letters to school districts and schools will be printed on Department letterhead and signed by high

- ra nking officials at the U.S. Department of Education’s Office of Planning, Evaluation and Policy Development as well as the High School Graduation Initiative (HSGI). The inclusion of an HSGI signature on the letter will lend legitimacy to the study as many sampled schools and school districts will be familiar with the HSGI grant program that supports dropout prevention and reentry efforts. With an estimated 2,088 schools in the sample and an expected 80 percent participation rate (see B.2), 1,670 school adm inistrators should complete the survey. A six
- month data collection is planned to allow school administrators to participate at a time most convenient to them. At the end of data collection, a school analytic weight will be constructed that will account for school nonresponse. 

### Generalizability of the Sample The research design for the survey relies on a simple random sample stratified by locale and graduation rate intended to capture descriptive information . As such, the findings from the survey will be ge neralizable to all public high schools within the sampling frame as well as each locale and schools with high and low graduation rates. 

### B.4. Test of Procedures and Methods The research team has conducted internal pretesting of survey items to ensure clarity. Additionally, the research team confirmed that all items are related to the research questions, ensuring the s urvey will capture all necessary information. Many of the survey questions have been adapted from relevant questions used in other national surveys of high school reforms . For example, some questions related to credit recovery programs have their roots in questions on the Fast Response Survey System
–Alternative Schools and Programs Survey (2008) and Distance Education Survey (2010)
. The research team will pilot the survey items with nine regionally dispersed district administrators in July. After the pilot participants test the instruments (i.e., take the survey), the researchers will have phone conversations with each participant to discuss clarity of wording and flow, interpret ations of the questions, and any other issues that emerge . The researchers will revise the instruments based on this feedback. 

### B.5. Consultations on Statistical Aspects of the Design The research team consulted with Dr. David Wilson, Senior Research Scientist at RTI International, on sampling for the survey. He can be reached at 919

- 541
- 6990. 

### Agency Joanne Bogart of the U.S. Department of Education is the Contracting Officer’s Representative for the study. She c an be reached at 202
- 205
- 7855. 

### Contractors SRI International and RTI will be responsible for data collection , under the direction of Ellen Schiller, who can be reached at 703
- 247
- 8503
.P21496, Task 4: Draft OMB Package 9



### References Alliance for Excellent Education. (2013). The economic b enefits of increasing the h igh school graduation rate for p ublic school students in the United States . Washington, DC: Alliance for Excellent Education. Amos, J. (2008). Dropouts, diplomas, and dollars: U.S. h igh schools and the n ation’s economy. Washington, DC: Alliance for Excellent Education. Carnevale, A. P., Smith, N. , & Strohl, J. (2010). Help wanted: Projections of jobs and education requirements through 2018. Washington, DC: Center on Education and the Workforce, Georgetown University. R umberger, R. W. (2013). Poverty and high school d ropouts : The impact of family and community poverty on high school dropouts, The SES Indicator (6) 2, May, 2013. Stetser, M. C., & Stillwell, R. (2014). Public high school four

-year on - time graduation rates and event dropout rates: School years 2010

-11 and 2011

- 12, First look (NCES 2014

-391) . Washington, DC: National Center for Education Statistics, U.S. Department of Education.