CONTRACT: ED
- IES
- 12
- C
- 0003



## Supporting Statement for OMB Clearance of a Survey of **Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model** 

**Section B August 16, 2013**  **Revised September 6, 2013 Revised October 8, 2013**  **Revised October 16, 2013 Revised December 1**  **3 , 2013** ii 

### Contents Section B: Collection of Information
................................ ................................ ................................ ........ 1 B1. Respondent Universe and Sampling Methods
................................ ................................ ................ 1 B2. Procedures for the Collection of Information
................................ ................................ .................. 2 B3. Methods To Maximize Response Rates and Deal With Nonresponse ................................ ......... 4 B4. Tests of Procedures or Methods to be Undertaken ................................ ................................ ......... 5 B5. Individuals Consulted on Statistical Aspects of the Design
................................ .......................... 5 References ................................ ................................ ................................ ................................ ................... 6 

### List of **t ables**  Table 1. Characteristics of rural SIG transformation schools vary in Cohort 1 ..........................2 The following attachments are stored in the “ NW S07” folder in the “ REL Northwest Memos and Correspondence ” library : Attachment B : Copy of Survey and Online Survey Screen Shots Attachment C : Intro Letter Invitation and Follow
- Up Emails Attachment F: Pilot Survey ReportSupporting s tatement for OMB c learance 1



## Supporting Statement for OMB Clearance of a Survey **of Principals of Rural Schools Receiving School Improvement Grants and Using the**  **Transformation Model Section B: Collection of**  **i nformation**  B 1. Respondent u niverse and s ampling m ethods The Regional E ducational L aboratory (REL) Northwest, with assistance from Policy Studies Associates (PSA) , plan s to examine how rural schools implement School Improvement Grants (SIGs)
. These competitive grants have provide d two cohorts of low
- performing schools with federal funding for school improve me nt and require schools to implement one of four school improvement models: the tran sformation model, which involves a variety of reforms; the turnaround model, which involves pr imarily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school. Collecting data from rural SIG schools is essential to this study, which seeks to inform regional and national supports fo r future school improvement efforts . REL Northwest was established under the 2012

–2017 Regional Education Laboratories Program. The current authorization for the program is under the Education Sciences Reform Act of 2002, Part D, Section 174, (20 U.S.C. 9 564), administered by the Institute of Education Sciences’ (IES’) National Center for Education Evaluation and Regional Assistance. IES has approved this study plan that collects principal (or proxy) survey data about the implementation of the transformati on model under SIGs
. The universe for this descriptive study is relatively small ( i.e., 211 schools across the nation). In order to have a large enough number of respondents to examine the data disaggregated by school characteristics, such as geographic l ocation and student demographics, this study surveys the entire universe of rural Cohort 1 SIG schools ( N = 211 ) and, therefore, doe s not use any sampling procedures. We identified these 211 Cohort 1 rural schools across the nation that implemented the SIG transformation model using the SIG baseline database and the N ational C enter on E ducation S tatistics preliminary 2011 data . Hurlburt, Le Floch, Therriault, and Cole (2011) compiled the SIG baseline database based on p ublicly available information. The SIG baseline database is available on the U.S. Department of Education website. We defined rural schools as those given locale codes 31 through 43 by the National Center for Education Statistics (NCES) 1 .

1 NCES has developed locale codes to describe a school’s location ranging from “large urban” to “remote rural.” The codes are based on the physical location represented by an address that is matched against a geographic database maintained by the Census Bur eau.Supporting s tatement for OMB c learance 2

B ecause SIG funding goes to schools serving large numbers of low
- income students (i.e. , those eligible for free or reduced
- price lunch) , rural SIG transformation schools have high levels of poverty; ho wever, these schools vary considerably by ethnicity (table 1 ). **Table 1 . Characteristics of rural SIG tran**  **sformation schools vary in Cohort**  **1 Number of**  **s chools**  **Percentage**

**non -** **White Percentage** 

**Native A merican**  **Percentage**

**free or**  **reduced**   ** price lunch** 31 Town, Fringe 6 41% 0% 47% 32 Town, Distant 33 60% 5% 68% 33 Town, Remote 16 75% 18% 81% 41 Rural, Fringe 48 70% 7% 72% 42 Rural, Distant 62 55% 16% 71% 43 Rural, Remote 46 77% 72% 78% 31
–43 All Schools 211 65% 24% 72% Source: Author ’ s analysis of SIG baseline database
. The schools are spread throughout the United States. Of the 50 states, 42 had Cohort 1 rural SIG transformation schools. California and three southern states had the largest number of rural SIG transformation schools: 19 schools in Florida, 12 in California, 12 in South Carolina, and 10 in North Carolina. The remaining states had fewer th an 10 rural SIG transformation schools. The s e survey data ha ve not been collected previously. We expect a response rate of 75 percent . More details about the response rate are included in B3. The survey is included in attachment B. B 2. Procedures for th e collection of i nformation This one
- time data collection by REL Northwest and PSA seeks responses from the universe of 211 rural SIG schools in Cohort 1 implementing the transformation model. Therefore, we will not use any sampl ing methods. Our procedure for collecting information includes several steps: a pilot survey, an initial invitation letter, the actual survey, and up to five follow
- up contacts. Between October 23 and November 8, 2013, we conducted a pilot of the survey instrument with seven princi pals from rural SIG schools that used the transformation model and received Cohort 1 funding . This pilot allow ed us to analyze pilot participant responses item by item and revise the survey to ens ure that all items and instructions we re relevant and easily understood . The pilot also allowed us to test and revise our data collection procedures. A memo describing the pilot is include d in attachment F , and more information about the pilot is in section B4. Cohort 1 principals who participate d in the pilot will have the opportunity to update their online pilot survey responses when the actual survey is administered. After we receive OMB clearance, we will begin actual data collection. To conduct the survey, we will first send an introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents of districts, and principals of rural SIG transformation schools ( attachment C ) . The letter emphasize s the impo rtance of the study. To provide further incentive to complete the survey, the letter inform s recipients that those who respond to the survey will receive a link to the published study and will be invited toSupporting s tatement for OMB c learance 3

participate in multiple webin ars to discuss the study results. The letter will also inform respondents that REL Northwest/PSA will email principals an online survey link within one week. Finally, the letter informs recipients that principals who are unfamiliar with SIG at their current schools will only fill out the beginning of the survey. This portion of the survey asks if the principal is familiar with SIG implementation at his or her school. If the principal is not familiar with SIG impl ementation, the survey provides instruc tions for designating a proxy to complete the survey and ends without asking any remaining survey items . The letter also explain s that current principals who were at their schools during the time period being studied will be allowed to designate a proxy if it will help the school complete the survey promptly. They can do so by responding to the email. One week after sending the intr oductory letter, we will email an invit ation to Cohort 1 rural SIG school principals inviting them to complete the online survey. The online survey will be available via Survey Gizmo, a survey administration software program . We will use procedures allowing respondent s to log in and out of their survey as often as needed to complete it. We will send principals two reminders t o complete the survey using this online format. I f return rates are less than 50 percent after the second reminder to complete the survey, we will offer a gift card to increase the response rate. We will do this for three reasons. First, our pilot showed that offering an incentive (an Amazon electronic gift card) after several reminders increase d the response rate from 44 percent to 77 percent. Se veral research studies support the use of incentives (Armstrong, 1975; Church, 1993; James & Bolstein, 1992 ), p articularly for online surveys of principals (Jacob & Jacob, 2012) . Principals who completed the survey before the second reminder will be sent the incentive retroactively. After three reminder emails, nonrespond e nts will be offered a W ord doc version of the survey in case they prefer responding to this version of the survey. They may return the W ord document via fax or email. If participants wish, they may request a paper document in the mail, and we will provide a paper document that is saddle stitched t o avoid the pages becoming separated. All versions should take approximately 20 minutes to complete, based on our survey pilot . We will maintain all survey data in our password
- protected servers under secure conditions. We will conduct up to five follow
- up emails and/or telephone calls to nonrespondents to achieve at least an 80 percent response rate. If the principal is wil l ing we will conduct the survey via phone , in order to reach the 80 percent response rate . To clean the survey data, we will import all data into SPSS software. Our initial cleaning efforts will focus on identifying principals for whom we have missing data. We expect missing data to be minimal since the online version of the survey will prompt principals to fill in any missing response s before they submit the survey. However, some principals may opt to take the paper version, so these surveys may contain missing data. If this is the case, we will contact principals to fill in these missing data. Although we will encourage principals to complete all items on theSupporting s tatement for OMB c learance 4

su r vey and will explain its importance, their participation is voluntary. We understand that some principals may decline to respond to all of the survey items. Once the missing data are retrieved, we will continue our data proces sing. Our survey data
processing procedures will include a visual review of any word document surveys, data editing as necessary, and a review of initial frequencies to ensure that responses are within acceptable ranges and there are no data entry errors. B 3. Methods t o m aximize response rates and d eal w ith n onresponse Our target response rate is 80 percent . We have set this target based on other similar studies. For example, the most recent Schools and Staffing Survey (SASS) of principals by the National Center of Education Statistics had a response rate of 79 percent for both public school and Bureau of Indian Education principals (Battle, 2009). We suspect that response rates would be somewhat different for rural school principals. For example, PSA assisted with data collection for a survey of rural district administrators, which achieved a response rate of 74 percent (Zhang, 2008). Our population is principals in low
- performing rural schools. These principals may have more job stress and be less likely to respond than those in nonrural, higher performing schools . Therefo re, because we set our target at 80 percent, w e will use several methods to maximize response rates and address nonresponse. First, we have minimized the bu rden for respondents by (1) requesting information that is generally available to school leadership, including principals, other administrators, and teacher
leaders directly involved in SIG implementation and (2) requesting the minimum amount of informatio n required to answer the study questions. Second, as discussed in B2, we will send an initial introductory letter to superintendents and school turnaround leaders in state departments of education, superintendents in districts, and principals in selected rural SIG transformation schools. Several other studies have found these introductory letters improved survey response rates (Cook, Heath, & Thompson, 2000; Kaplowitz, Hadlock, & Levine, 2004). We will also send reminder emails as described previously. If needed after multiple reminders, we are prepared to administer the survey by phone. Third, we will offer an incentive to principals who have not completed the survey after two reminders. Se veral research reviews show that incentives often increase respons e rates (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992 ) . More importantly, a recent experimental study of an online principal survey showed that principals’ response rates, in particular, were statistically significantly higher when principals wer e offered an incentive (Jacob & Jacob, 2012). Finally, a s discussed previously, principals who are new to their schools or who are too busy to complete the survey may designate a proxy. At some SIG schools, district or teacher leaders areSupporting s tatement for OMB c learance 5

very involved with SIG implementation ( Scott, McMurrer, McIntosh, & Dibner, 2012 ) . These individuals may have more time to respond to the survey and, therefore, increase return rates. B 4. Tests of p rocedures or m ethods to be u ndertaken Survey items in this data collection are adapted from a previously conducted survey as part of an evaluation of SIG in Oregon ( Scott, Davis, & Krasnoff , 2012). In addition, a s described previously, we conducted a pilot of the survey instrument from October 23 through November 8, 2013 . We randomly selected nine principals of rural SIG schools that rece ived their SIG funds in C ohort 1 . Seven principals responded by November 8, 2013 . The pilot follow ed the survey procedures described in this document for the actual study. Additionally, pilot participants were asked to comment on whether the survey correspondence clearly conveyed the purpose of the study and whether the survey instructions and items were clear and easy to follow. Cohort 1 principals who participate d in the pilot will have the o pportunity to update their online pilot survey responses when the actual survey is administered. A more detailed description of the p ilot is included in attachment F. B5. Individuals c onsulted on s tatistical a spects of the d esign Two technical experts will play an important role in providing insight and guidance in support of this study of the rural SIG transformation model . The y include :  Tom Dee, Professor, Graduate School of Education, Stanford University, 650 . 723 . 6847  Hans Bos, Vice President, Education, American Institute s for Research, 707 . 701 . 3004 Both individuals are members of REL Northwest’s Technical Working Group (TWG). The main responsibility of TWG members is to provide written feedback on study plans and/or study repo rts and to meet as a group via telephone with the research team to clarify feedback and provide additional recommendations . The TWG members also are available to advise the researchers as questions come up, speak with REL staff about future research project ideas, and generally lend their expertise as needed. All TWG members are experts who are distinguished in their fields and extremely knowledgeable about research design and methods.Supporting s tatement for OMB c learance 6

References Armstrong, S.J. (1975). Monetary incentives in mail surveys
. Public Opinion Quarterly, 39 (1), 111

– 116. Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007

–08 Schools and Staffing Survey ( First Look, NCES 2009
- 323). Re trieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A meta
analysis. Public Opinion Quarterly, 57 (1), 62

–79. Cook, C., Heath, F., & Thompson, R.L. (2000). A meta
- analysis of response rates in w eb
- or Internet
-based surveys. Educational and Psychological Measurement , 60 (6) , 821

–836. Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG

- e ligible and SIG

- a warded s chools (NCEE 2011
- 4019). Retrieved from U.S. Department of Education, In stitute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf Jacob, R.T., & Jacob, B. (2012). Pre
-notification, incentives, and survey modality: An experimental test of methods to increase survey response rates of school principals. Journal of Research on Educational Effectiveness, 5 (4) , 401

–418. James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow
- up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54 (3) , 346

–361. Kaplowitz, M.D., Hadlock, T. D., & Levine, R. (2004). A comparison of w eb and mail survey response rates. Public Opinion Quarterly, 68 (1) , 94

– 101 . Scott, C . , Davis, A ., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011

– 2012. Portland, OR: Education Northwest. Scott, C., McMurrer, J., McIntosh, S
. , & Dibner, K. (2012). Opportunities and obstacles: Implementing stimulus

-

funded School Improvement Grants in Maryland, Michigan, and Idaho . Washington, DC: Center on Education Policy. Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from Center on Education Policy website: http://www.cep
dc.org/displayDocument.cfm?DocumentID=308