CONTRACT: ED - IES - 12 - C - 0003



## Supporting Statement for OMB Clearance of a Survey of **Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model** 

**Section B**
**August 16, 2013 Revised September 6, 2013**  **Revised October 8, 2013 Revised October 16, 2013** ii



### Contents Section B: Collection of Information ................................ ................................ ................................ ........ 1 B1. Respondent Universe and Sampling Methods ................................ ................................ ................ 1 B2. Procedures for the Collection of Information ................................ ................................ .................. 2 B3. Methods To Maximize Response Rates and Deal With Nonresponse ................................ ......... 4 B4. Tests of Procedures or Methods to be Undertaken ................................ ................................ ......... 4 B5. Individuals Consulted on Statistical Aspects of the Design ................................ .......................... 5 References ................................ ................................ ................................ ................................ ................... 5 

### List of Tables Table 1. Characteristics of ru ral SIG transformation schools vary in Cohort 1 ..........................2 The following a ttachments are stored in the “ NW S07” folder in the “ REL Northwest Memos and Correspondence ” library : Attachment B : Copy of Survey and Online Survey Screen Shots Attachment C : Intro Letter Invitation and Follow - Up EmailsSupporting Statement for OMB Clearance 1



## Supporting Statement for OMB Clearance of a Survey **of Principals of Rural Schools Receiving School**  **Improvement Grants and Using the Transformation Model**  **Section B: Collection of Information B**  **1. Respondent Universe and Sampling Methods**  The Regional E ducational L aboratory (REL) Northwest, with assistance from Policy Studies Associates (PSA) , plan s to examine how rural schools implement School Improvement Grants (SIGs) . These competitive grants have provide d two cohorts of low - performing schools with federal funding for school improv e me nt and require schools to implement one of four school improveme nt models: the tran sformation model, which involves a variety of reforms; the turnaround model, which involves primarily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school. Collectin g data from rural SIG schools is essential to this study, which seeks to inform regional and national supports for future school improvement efforts . REL Northwest was established under the 2012

–2017 Regional Education Laboratories Program. The current authorization for the program is under the Education Sciences Reform Act of 2002, Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES’) National Center for Education Evaluation and Regional Assistance. IES has approved this study plan that collects principal (or proxy) survey data about the implementation of the transformation model under SIGs . The universe for this descriptive study is relatively small ( i.e., 211 schools across the nation). In order to have a large enough number of respondents to examine the data disaggregated by school characteristics, such as geographic location and student demographics, this study surveys the entire universe of rural C ohort 1 SIG schools ( N = 211 ) and, therefore, doe s not use any sampling procedures. We identified these 211 Cohort 1 rural schools across the nation that implemented the SIG transformation model using the SIG baseline database and the N ational C enter on E d ucation S tatistics preliminary 2011 data . Hurlburt, Le Floch, Therriault, and Cole (2011) compiled the SIG baseline database based on p ublicly available information. The SIG baseline database is available on the U.S. Department of Education website. We defined rural schools as those given locale codes 31 through 43 by the National Center for Education Statistics (NCES) 1 .
1 NCES has developed locale codes to describe a school’s location ranging from “large urban” to “remote rural.” The codes are based on the physical location represented by an address that is matched against a geographic database maintained by the Census Bur eau.Supporting Statement for OMB Clearance 2

B ecause SIG funding goes to schools serving large numbers of low- income students (i.e. , those eligible for free or reduced - price lunch) , ru ral SIG transformation schools have high levels of poverty; ho wever, these schools vary considerably by ethnicity (table 1 ). **Table 1 . Characteristics of rural SIG transformation schools vary in**  **Cohort 1**  **Number of s**  **chools Percentage** 

**non -** **White Percentage** 

**Native A merican**  **Percentage**

**free and reduced**   ** price lunch** 31 Town, Fringe 6 41% 0% 47% 32 Town, Distant 33 60% 5% 68% 33 Town, Remote 16 75% 18% 81% 41 Rural, Fringe 48 70% 7% 72% 42 Rural, Distant 62 55% 16% 71% 43 Rural, Remote 46 77% 72% 78% 31 –43 All Schools 211 65% 24% 72% Source: Author ’ s analysis of SIG baseline database
. The schools are spread throughout the United States. Of the 50 states, 42 had Cohort 1 rural SIG transformation schools. California and three southern states had the largest number of rural SIG transformation schools: 19 schools in Florida, 12 in California, 12 in South Carolina, and 10 in North Carolina. The remaining states had fewer th an 10 rural SIG transformation schools. The s e survey data ha ve not been collected previously. We expect a response rate of 75 percent . More details about the response rate are included in B3. The survey is included in attachment B. 

### B **2. Procedures for**  **the Collection of Information** This one - time data collection by REL Northwest and PSA seeks responses from the universe of 211 rural SIG schools in Cohort 1 implementing the transformation model. Therefore, we will not use any sampl ing methods. Our procedu re for collecting information includes several steps: a pilot survey, an initial invitation letter, the actual survey, and up to five follow - up contacts. By the end of the 60 days of public notice through the Federal Register, w e will have already conducted a pilot of the survey instrument with nine or fewer individuals from rural SIG schools that are us ing the transformation model and received Cohort 1 funding . This pilot, to be conducted in October 2013, will allow us to analyze pilot participant responses item -by- item and revise the survey to ens ure that all items and instructions are relevant and easily understood . Cohort 1 principals who participate in the pilot will have the opportunity to update their online pilot survey responses when the actual survey is administered. After we receive OMB clearance, we will begin actual data collection. To conduct the survey, we will first send an introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents of districts, and principals of rural SIG transformation schools ( attachment C ) . The letter emphasize s the impo rtance of the study. ToSupporting Statement for OMB Clearance 3

provide further incentive to complete the survey, the letter inform s recipients that those who respond to the s urvey will receive a link to the published study and will be invited to participate in multiple webinars to discuss the study results. The letter will also inform respondents that REL Northwest/PSA will mail and email principals an online survey link within one week . Principals who do not complete the online survey will be offered a W ord document after several reminder email s, in case they prefer responding to this version of the survey. They may return the word document via fax or email. If participan ts wish, they may request a paper document in the mail, and we will provide a paper document that is saddle stitched to avoid the pages becoming separated. Either version should take approximately 20 minutes to complete. Finally, the letter informs recipients that principals who are unfamiliar with SIG at their current schools will only fill out the beginning of the survey. This portion of the survey asks if the principal is familiar with SIG implementation at his or her school. If the principal is not familiar with SIG impl ementation, the survey provides instructions for designating a proxy to complete the survey and ends without asking any remaining survey items . The letter also explain s that current principals who were at their schools during the time period being studied will also be allowed to designate a proxy if it will help the school complete the survey promptly. They can do so by responding to the email. One week after sending the intr oductory letter, we will email an invit ation to Cohort 1 rural SIG school principals to inviting them to complete the online survey. The online survey will be available via Survey Gizmo, a survey administration software program . We will use procedure s allowing respondent s to log in and out of their survey as often as needed to complete it. Two weeks later after three reminder emails, nonrespond e nts will be offered a W ord doc version of the survey that they can return via fax or email. We will maintain all survey data in our password - protected servers under secure conditions. We will conduct up to five follow - up emails and/or telephone calls to nonrespondents to achieve at least a 75 percent response rate. If the principal is wil l ing we will conduct the survey via phone , if we reach that principal by phone in order to reach our 75 percent response rate target . To clean the survey data, we will import all data into SPSS software. Our initial cleaning efforts will focus on identifying principals for whom we have missing data. We expect missing data to be minimal since the online version of the survey will prompt principals to fill in any missing responses before they submit the survey. However, some principals may opt to take the paper version, so these su rveys may contain missing data. If this is the case, we will contact principals to fill in these missing data. Although we will encourage principals to complete all items on the su r vey and will explain its importance, their participation is voluntary. We understand that some principals may decline to respond to all of the survey items. Once the missing data are retrieved, we will continue our data processing. Our survey data processing procedures will include a visual review of any word document surveys, d ata editingSupporting Statement for OMB Clearance 4

as necessary, and a review of initial frequencies to ensure that responses are within acceptable ranges and there are no data entry errors. 

### B **3. Methods**  **T o Maximize Response Rates and Deal**  **W ith Nonresponse**  Our target response rate is 75 percen t . We have set this target based on other similar studies. For example, the most recent Schools and Staffing Survey (SASS) of principals by the National Center of Education Statistics had a response rate of 79 percent for both public school and Bureau of Indian Education principals; however, this estimate may be high since the SASS had local survey coordinators (Battle, 2009). In addition, this report did not disaggregate the response rates for rural school principals. We suspect that response rates would be somewhat lower for rural school principals. For example, PSA assisted with data collection for a survey of rural district administrators, which achieved a response rate of 74 percent (Zhang, 2008). Our population is principals in low - performing rural schools. These principals may have more job stress and be less likely to respond than those in nonrural, higher performing schools . Therefo re , we have set our target at 75 pe rcent, a lower rate than the SASS and similar to Zhang (2008) . We will use several methods to maximize response rates and address nonresponse. First, we have minimized the burden for respondents by (1) requesting informati on that is generally available to school leadership, including principals, other administrators, and teacher - leaders directly involved in SIG implementation and (2) requesting the minimum amount of information required to answer the study questions. As di scussed in B2, we will send an initial introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents in districts, and principals in selected rural SIG transformation schools. Several other studies have found these introductory letters improved survey response rates ( Cook, Heath, & Thompson, 2000; Kaplowitz, Hadlock, & Levine, 2004 ). We will also send reminder emails as described previously . If needed after multiple reminders, we are prepared to administer the survey by phone. A s discussed previously, principals who are new to their schools or who are too busy to complete the survey may designate a proxy. At some SIG schools, district or teacher leaders are very involved with SIG implementation ( Scott, McMurrer, McIntosh, & Dibner, 2012 ) . These individuals may have more time to respond to the survey and, therefore, increase return rates. 

### B **4. Tests of Procedures or Methods to be Undertaken**  Survey items in this data collection are adapted from a p reviously conducted survey as part of an evaluation of SIG in Oregon ( Scott, Davis, & Krasnoff , 2012). In addition, a s described previously, we will conduct a pilot of the survey instr ument wi th nine or fewer participantsSupporting Statement for OMB Clearance 5

during the 60 - day period of public notice through the Federal Register . We will randomly select nine principals of rural SIG schools that rece ived their SIG funds in cohort 1. The pilot will follow the survey procedures described for the actual study. Add itionally, pilot participants will be asked to comment on whether the survey correspondence clearly conveyed the purpose of the study and whether the survey instructions and items were clear and easy to follow. Cohort 1 principals who participate in the pi lot will have the opportunity to update their online pilot survey responses when the actual survey is administered. 

### B5. **Individuals Consulted on Statistical A spects of the D**  **esign** Two technical experts will play an important role in providing insight an d guidance in support of this study of the rural SIG transformation model . The y include :  Tom Dee, Professor, Graduate School of Education, Stanford University, 650 . 723 . 6847  Hans Bos, Vice President, Education, American Institute s for Research, 707 . 701 . 3004 Both individuals are members of REL Northwest’s Technical Working Group (TWG). The main responsibility of TWG members is to provide written feedback on study plans and/or study reports and to meet as a group via telephone with the research team to cl arify feedback and provide additional recommendations . The TWG members also are available to advise the researchers as questions come up, speak with REL staff about future research project ideas, and generally lend their expertise as needed. All TWG member s are experts who are distinguished in their fields and extremely knowledgeable about research design and methods. 

### References Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007

–08 Schools and Staffing Survey ( First Look, NCES 2009 - 323). Re trieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf Cook, C., Heath, F., & Thompson, R.L. (2000). A meta- analysis of response rates in w eb - or Internet-based surveys. Educational and Psychological Measurement , 60 (6) , 821
–836. Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG

- e ligible and SIG

- a warded s chools (NCEE 2011 - 4019). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf Kaplowitz, M.D., Hadlock, T. D., & Levine, R. (2004). A comparison of w eb and mail survey response rates. Public Opinion Quarterly, 68 (1) , 94
– 101 .Supporting Statement for OMB Clearance 6

Scott, C . , Davis, A ., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011

– 2012. Portland, OR: Education Northwest. Scott, C., McMurrer, J., McIntosh, S . , & Dibner, K. (2012). Opportunities and ob stacles: Implementing stimulus

-

funded School Improvement Grants in Maryland, Michigan, and Idaho . Washington, DC: Center on Education Policy. Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from C enter on Education Policy website: http://www.cep dc.org/displayDocument.cfm?DocumentID=308