CONTRACT: ED-IES-12-C-0003

Supporting Statement for OMB Clearance of a Survey of 

Principals of Rural Schools Receiving School Improvement 

Grants and Using the Transformation Model 

Section B

August 16, 2013

Revised September 6, 2013

Revised October 8, 2013

Revised October 16, 2013

Revised December 13, 2013
ii

Contents

Section B: Collection of Information........................................................................................................ 1

B1. Respondent Universe and Sampling Methods................................................................................ 1

B2. Procedures for the Collection of Information.................................................................................. 2

B3. Methods To Maximize Response Rates and Deal With Nonresponse......................................... 4

B4. Tests of Procedures or Methods to be Undertaken......................................................................... 5

B5. Individuals Consulted on Statistical Aspects of the Design.......................................................... 5

References ................................................................................................................................................... 6

List of tables

Table 1. Characteristics of rural SIG transformation schools vary in Cohort 1..........................2

The following attachments are stored in the “NW S07” folder in the “REL Northwest Memos 

and Correspondence” library:

Attachment B: Copy of Survey and Online Survey Screen Shots

Attachment C: Intro Letter Invitation and Follow-Up Emails

Attachment F: Pilot Survey Report
Supporting statement for OMB clearance 1

Supporting Statement for OMB Clearance of a Survey of Principals of 

Rural Schools Receiving School Improvement Grants and Using the 

Transformation Model

Section B: Collection of information

B1. Respondent universe and sampling methods

The Regional Educational Laboratory (REL) Northwest, with assistance from Policy Studies 

Associates (PSA), plans to examine how rural schools implement School Improvement Grants 

(SIGs). These competitive grants have provided two cohorts of low-performing schools with

federal funding for school improvement and require schools to implement one of four school 

improvement models: the transformation model, which involves a variety of reforms; the 

turnaround model, which involves primarily replacing staff; the restart model, which involves

becoming a charter school; and the closure model, which closes the school. Collecting data from 

rural SIG schools is essential to this study, which seeks to inform regional and national supports 

for future school improvement efforts.

REL Northwest was established under the 2012–2017 Regional Education Laboratories Program. 

The current authorization for the program is under the Education Sciences Reform Act of 2002, 

Part D, Section 174, (20 U.S.C. 9564), administered by the Institute of Education Sciences’ (IES’) 

National Center for Education Evaluation and Regional Assistance. IES has approved this study 

plan that collects principal (or proxy) survey data about the implementation of the 

transformation model under SIGs.

The universe for this descriptive study is relatively small (i.e., 211 schools across the nation). In 

order to have a large enough number of respondents to examine the data disaggregated by 

school characteristics, such as geographic location and student demographics, this study 

surveys the entire universe of rural Cohort 1 SIG schools (N = 211) and, therefore, does not use 

any sampling procedures. We identified these 211 Cohort 1 rural schools across the nation that 

implemented the SIG transformation model using the SIG baseline database and the National 

Center on Education Statistics preliminary 2011 data. Hurlburt, Le Floch, Therriault, and Cole 

(2011) compiled the SIG baseline database based on publicly available information. The SIG 

baseline database is available on the U.S. Department of Education website. We defined rural 

schools as those given locale codes 31 through 43 by the National Center for Education Statistics 

(NCES)1.

 

1 NCES has developed locale codes to describe a school’s location ranging from “large urban” to “remote rural.” The 

codes are based on the physical location represented by an address that is matched against a geographic database 

maintained by the Census Bureau.
Supporting statement for OMB clearance 2

Because SIG funding goes to schools serving large numbers of low-income students (i.e., those 

eligible for free or reduced-price lunch), rural SIG transformation schools have high levels of 

poverty; however, these schools vary considerably by ethnicity (table 1).

Table 1. Characteristics of rural SIG transformation schools vary in Cohort 1

Number of 

schools

Percentage

non-White

Percentage

Native American

Percentage

free or reducedprice lunch

31 Town, Fringe 6 41% 0% 47%

32 Town, Distant 33 60% 5% 68%

33 Town, Remote 16 75% 18% 81%

41 Rural, Fringe 48 70% 7% 72%

42 Rural, Distant 62 55% 16% 71%

43 Rural, Remote 46 77% 72% 78%

31–43 All Schools 211 65% 24% 72%

Source: Author’s analysis of SIG baseline database.

The schools are spread throughout the United States. Of the 50 states, 42 had Cohort 1 rural SIG 

transformation schools. California and three southern states had the largest number of rural SIG 

transformation schools: 19 schools in Florida, 12 in California, 12 in South Carolina, and 10 in 

North Carolina. The remaining states had fewer than 10 rural SIG transformation schools.

These survey data have not been collected previously. We expect a response rate of 75 percent. 

More details about the response rate are included in B3. The survey is included in attachment B.

B2. Procedures for the collection of information

This one-time data collection by REL Northwest and PSA seeks responses from the universe of 

211 rural SIG schools in Cohort 1 implementing the transformation model. Therefore, we will 

not use any sampling methods. Our procedure for collecting information includes several steps: 

a pilot survey, an initial invitation letter, the actual survey, and up to five follow-up contacts.

Between October 23 and November 8, 2013, we conducted a pilot of the survey instrument with 

seven principals from rural SIG schools that used the transformation model and received 

Cohort 1 funding. This pilot allowed us to analyze pilot participant responses item by item and 

revise the survey to ensure that all items and instructions were relevant and easily understood.

The pilot also allowed us to test and revise our data collection procedures. A memo describing 

the pilot is included in attachment F, and more information about the pilot is in section B4. 

Cohort 1 principals who participated in the pilot will have the opportunity to update their 

online pilot survey responses when the actual survey is administered.

After we receive OMB clearance, we will begin actual data collection. To conduct the survey, we 

will first send an introductory letter to superintendents and school turnaround leaders in state 

departments of education, superintendents of districts, and principals of rural SIG 

transformation schools (attachment C). The letter emphasizes the importance of the study. To 

provide further incentive to complete the survey, the letter informs recipients that those who 

respond to the survey will receive a link to the published study and will be invited to 
Supporting statement for OMB clearance 3

participate in multiple webinars to discuss the study results. The letter will also inform 

respondents that REL Northwest/PSA will email principals an online survey link within one 

week.

Finally, the letter informs recipients that principals who are unfamiliar with SIG at their current 

schools will only fill out the beginning of the survey. This portion of the survey asks if the 

principal is familiar with SIG implementation at his or her school. If the principal is not familiar 

with SIG implementation, the survey provides instructions for designating a proxy to complete 

the survey and ends without asking any remaining survey items. The letter also explains that 

current principals who were at their schools during the time period being studied will be 

allowed to designate a proxy if it will help the school complete the survey promptly. They can 

do so by responding to the email.

One week after sending the introductory letter, we will email an invitation to Cohort 1 rural SIG 

school principals inviting them to complete the online survey. The online survey will be 

available via Survey Gizmo, a survey administration software program. We will use procedures 

allowing respondents to log in and out of their survey as often as needed to complete it. We will 

send principals two reminders to complete the survey using this online format.

If return rates are less than 50 percent after the second reminder to complete the survey, we will 

offer a gift card to increase the response rate. We will do this for three reasons. First, our pilot 

showed that offering an incentive (an Amazon electronic gift card) after several reminders 

increased the response rate from 44 percent to 77 percent. Several research studies support the 

use of incentives (Armstrong, 1975; Church, 1993; James & Bolstein, 1992), particularly for 

online surveys of principals (Jacob & Jacob, 2012). Principals who completed the survey before 

the second reminder will be sent the incentive retroactively.

After three reminder emails, nonrespondents will be offered a Word doc version of the survey 

in case they prefer responding to this version of the survey. They may return the Word 

document via fax or email. If participants wish, they may request a paper document in the mail, 

and we will provide a paper document that is saddle stitched to avoid the pages becoming 

separated. All versions should take approximately 20 minutes to complete, based on our survey 

pilot. We will maintain all survey data in our password-protected servers under secure 

conditions. We will conduct up to five follow-up emails and/or telephone calls to 

nonrespondents to achieve at least an 80 percent response rate. If the principal is willing we will 

conduct the survey via phone, in order to reach the 80 percent response rate.

To clean the survey data, we will import all data into SPSS software. Our initial cleaning efforts 

will focus on identifying principals for whom we have missing data. We expect missing data to 

be minimal since the online version of the survey will prompt principals to fill in any missing 

responses before they submit the survey. However, some principals may opt to take the paper 

version, so these surveys may contain missing data. If this is the case, we will contact principals 

to fill in these missing data. Although we will encourage principals to complete all items on the 
Supporting statement for OMB clearance 4

survey and will explain its importance, their participation is voluntary. We understand that 

some principals may decline to respond to all of the survey items.

Once the missing data are retrieved, we will continue our data processing. Our survey dataprocessing procedures will include a visual review of any word document surveys, data editing

as necessary, and a review of initial frequencies to ensure that responses are within acceptable 

ranges and there are no data entry errors.

B3. Methods to maximize response rates and deal with nonresponse

Our target response rate is 80 percent. We have set this target based on other similar studies. 

For example, the most recent Schools and Staffing Survey (SASS) of principals by the National 

Center of Education Statistics had a response rate of 79 percent for both public school and 

Bureau of Indian Education principals (Battle, 2009). We suspect that response rates would be 

somewhat different for rural school principals. For example, PSA assisted with data collection 

for a survey of rural district administrators, which achieved a response rate of 74 percent

(Zhang, 2008). Our population is principals in low-performing rural schools. These principals 

may have more job stress and be less likely to respond than those in nonrural, higher 

performing schools. Therefore, because we set our target at 80 percent, we will use several 

methods to maximize response rates and address nonresponse.

First, we have minimized the burden for respondents by (1) requesting information that is 

generally available to school leadership, including principals, other administrators, and teacherleaders directly involved in SIG implementation and (2) requesting the minimum amount of 

information required to answer the study questions.

Second, as discussed in B2, we will send an initial introductory letter to superintendents and 

school turnaround leaders in state departments of education, superintendents in districts, and 

principals in selected rural SIG transformation schools. Several other studies have found these 

introductory letters improved survey response rates (Cook, Heath, & Thompson, 2000;

Kaplowitz, Hadlock, & Levine, 2004). We will also send reminder emails as described 

previously. If needed after multiple reminders, we are prepared to administer the survey by 

phone.

Third, we will offer an incentive to principals who have not completed the survey after two 

reminders. Several research reviews show that incentives often increase response rates 

(Armstrong, 1975; Church, 1993; James, & Bolstein, 1992). More importantly, a recent 

experimental study of an online principal survey showed that principals’ response rates, in 

particular, were statistically significantly higher when principals were offered an incentive 

(Jacob & Jacob, 2012).

Finally, as discussed previously, principals who are new to their schools or who are too busy to 

complete the survey may designate a proxy. At some SIG schools, district or teacher leaders are 
Supporting statement for OMB clearance 5

very involved with SIG implementation (Scott, McMurrer, McIntosh, & Dibner, 2012). These 

individuals may have more time to respond to the survey and, therefore, increase return rates.

B4. Tests of procedures or methods to be undertaken

Survey items in this data collection are adapted from a previously conducted survey as part of 

an evaluation of SIG in Oregon (Scott, Davis, & Krasnoff, 2012). In addition, as described

previously, we conducted a pilot of the survey instrument from October 23 through November 

8, 2013. We randomly selected nine principals of rural SIG schools that received their SIG funds 

in Cohort 1. Seven principals responded by November 8, 2013. The pilot followed the survey 

procedures described in this document for the actual study. Additionally, pilot participants 

were asked to comment on whether the survey correspondence clearly conveyed the purpose of 

the study and whether the survey instructions and items were clear and easy to follow. Cohort 1 

principals who participated in the pilot will have the opportunity to update their online pilot 

survey responses when the actual survey is administered. A more detailed description of the 

pilot is included in attachment F.

B5. Individuals consulted on statistical aspects of the design

Two technical experts will play an important role in providing insight and guidance in support 

of this study of the rural SIG transformation model. They include:

 Tom Dee, Professor, Graduate School of Education, Stanford University, 650.723.6847

 Hans Bos, Vice President, Education, American Institutes for Research, 707.701.3004

Both individuals are members of REL Northwest’s Technical Working Group (TWG). The main 

responsibility of TWG members is to provide written feedback on study plans and/or study 

reports and to meet as a group via telephone with the research team to clarify feedback and 

provide additional recommendations. The TWG members also are available to advise the 

researchers as questions come up, speak with REL staff about future research project ideas, and 

generally lend their expertise as needed. All TWG members are experts who are distinguished 

in their fields and extremely knowledgeable about research design and methods.
Supporting statement for OMB clearance 6

References

Armstrong, S.J. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39(1), 111–

116.

Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and 

secondary school principals in the United States: Results from the 2007–08 Schools and Staffing 

Survey (First Look, NCES 2009-323). Retrieved from U.S. Department of Education, 

National Center for Education Statistics website: 

http://nces.ed.gov/pubs2009/2009323.pdf

Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A metaanalysis. Public Opinion Quarterly, 57(1), 62–79.

Cook, C., Heath, F., & Thompson, R.L. (2000). A meta-analysis of response rates in web- or 

Internet-based surveys. Educational and Psychological Measurement, 60(6), 821–836.

Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications 

and SIG-eligible and SIG-awarded schools (NCEE 2011-4019). Retrieved from U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance website: 

http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf

Jacob, R.T., & Jacob, B. (2012). Pre-notification, incentives, and survey modality: An 

experimental test of methods to increase survey response rates of school principals. 

Journal of Research on Educational Effectiveness, 5(4), 401–418.

James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow-up mailings on 

the response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3), 

346–361.

Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004). A comparison of web and mail survey 

response rates. Public Opinion Quarterly, 68(1), 94–101.

Scott, C., Davis, A., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011–

2012. Portland, OR: Education Northwest.

Scott, C., McMurrer, J., McIntosh, S., & Dibner, K. (2012). Opportunities and obstacles: 

Implementing stimulus-funded School Improvement Grants in Maryland, Michigan, and Idaho. 

Washington, DC: Center on Education Policy.

Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. 

Retrieved from Center on Education Policy website: http://www.cepdc.org/displayDocument.cfm?DocumentID=308