CONTRACT: ED-IES-12-C-0003

Supporting Statement for OMB Clearance of a Survey of 

Principals of Rural Schools Receiving School Improvement 

Grants and Using the Transformation Model

Section A

August 16, 2013

Revised September 6, 2013

Revised October 8, 2013

Revised Oct 16, 2013

Revised December 13, 2013

Revised December 30, 2013
ii

Contents

Section A: Justification...............................................................................................................................1

A1. Circumstances Necessitating Collection of Information............................................................... 1

A2. How, by Whom, and for What Purpose Information Is to Be Used............................................ 3

A3. Use of Automated, Electronic, Mechanical or Other Technological Collection 

Techniques ..................................................................................................................................... 5

A4. Efforts to Identify Duplication.......................................................................................................... 6

A5. Sensitivity to Burden on Small Entities............................................................................................ 7

A6. Consequence to Federal Program or Policy Activities if the Collection Is Not 

Conducted or Is Conducted Less Frequently............................................................................ 7

A7. Special Circumstances........................................................................................................................ 7

A8. Federal Register Announcement and Consultation....................................................................... 7

A9. Payment or Gift to Respondents....................................................................................................... 8

A10. Confidentiality of the Data .............................................................................................................. 9

A11. Additional Justification for Sensitive Questions ........................................................................ 10

A12. Estimates of Hour Burden ............................................................................................................. 10

A13. Estimate for the Total Annual Cost Burden to Respondents or Record Keepers .................. 11

A14. Estimates of Annualized Costs to the Federal Government..................................................... 11

A15. Reasons for Program Changes or Adjustments.......................................................................... 11

A16. Plans for Tabulation and Publication........................................................................................... 11

A17. Approval to Not Display the Expiration Date for OMB Approval.......................................... 15

A18. Exception to the Certification Statement..................................................................................... 16

References ................................................................................................................................................. 17
iii

List of tables

Table 1. Administration times ................................................................................................................ 11

Table 2. Timeline ..................................................................................................................................... 15

The following attachments are stored in the “NW S07” folder in the “REL Northwest Memos 

and Correspondence” library, with the exceptions of Attachments D and E, which we will add 

later since the Federal Register notices cannot be submitted until this document is approved by 

IES:

Attachment A: ESRA

Attachment B: Copy of Survey and Online Survey Screen Shots

Attachment C: Intro Letter Invitation and Follow-Up Emails

Attachment D: 60-Day Federal Register Notice (to be added after the notice is submitted)

Attachment E: 30-Day Federal Register Notice (to be added after the notice is submitted)

Attachment F: Pilot Survey Report

Attachment G: Affidavits of Nondisclosure
Supporting statement for OMB clearance 1

Supporting Statement for OMB Clearance of a Survey of 

Principals of Rural Schools Receiving School Improvement 

Grants and Using the Transformation Model

Section A: Justification

A1. Circumstances necessitating collection of information

This data collection is authorized by the Educational Sciences Reform Act (ESRA) of 2002 (see 

attachment A). Part D, Section 174(f)(2) of ESRA states that as part of its central mission and 

primary function, each regional educational laboratory “shall support applied research by ... 

developing and widely disseminating, including through Internet-based means, scientifically 

valid research, information, reports, and publications that are usable for improving academic

achievement, closing achievement gaps, and encouraging and sustaining school improvement, 

to schools, districts, institutions of higher education, educators (including early childhood 

educators and librarians), parents, policymakers, and other constituencies, as appropriate, 

within the region in which the regional educational laboratory is located.”

REL Northwest and our partner Policy Studies Associates (PSA), the authors of this study, are

interested in how rural schools implement School Improvement Grants (SIGs). These 

competitive federal grants have provided funds to two cohorts of low-performing schools to 

increase student achievement. The grants require schools to implement one of four school 

improvement models: the transformation model, which involves a variety of reforms; the 

turnaround model, which involves primarily replacing staff; the restart model, which involves

becoming a charter school; and the closure model, which closes the school. 

This descriptive study collects and reports on data from rural school principals across the nation 

that received federal SIGs in Cohort 1 and used the transformation model. This information 

collection is essential to a study by the Regional Educational Laboratory (REL) Northwest, 

which serves five Northwest states (Alaska, Idaho, Montana, Oregon, and Washington). Most

rural Northwest schools with SIG funding (96 percent) used the transformation model to 

improve. REL Northwest and its partner PSA are collecting survey data about the 

implementation of the transformation model to inform regional and national supports for 

future school improvement efforts. Data collection will occur once in spring 2014. IES has 

approved the study plan.

Statement of need and background information

To date, there is little systematic evidence about the implementation of the SIG program. In the 

case of rural SIG settings, this lack of research evidence is particularly problematic. Rural 

schools often face steep challenges when trying to implement the kinds of staff replacement and 

on-site professional development practices promoted by the SIG program (Klein, 2010). This 

study addresses these challenges by examining the level of implementation of the SIG 

transformation model in rural regions, identifying the organizations—states, districts, 
Supporting statement for OMB clearance 2

universities, and other entities—that assist schools in these implementation efforts, and 

exploring perceived gaps in assistance. 

SIG funding mechanisms are not new. Congress put them in place as part of the No Child Left 

Behind Act (NCLB) of 2001 (2002), the reauthorization of the Elementary and Secondary 

Education Act (ESEA). However, in 2009 the American Recovery and Reinvestment Act (ARRA) 

substantially increased funding for these grants. With this increase, the U.S. Department of 

Education issued new guidance, dramatically changing SIG requirements (U.S. Department of 

Education, Office of Elementary and Secondary Education, 2011).

Under this guidance, federal funds pass to states as usual, but states are now charged with 

assisting schools in implementing one of four school improvement models (U.S. Department of 

Education, Office of Elementary and Secondary Education [OESE], 2011): (1) transformation, 

replacing the principal and implementing a number of strategies to improve the school; (2) 

turnaround, replacing the principal and 50 percent of staff members and providing operational 

flexibility needed to implement a variety of school improvement strategies; (3) restart, 

reopening as a school operated by a charter or education management organization; and (4) 

closure, closing the school and ensuring that students transfer to a high-performing school.

Because the transformation model requires less disruption and staff member replacement than 

other models, some consider it the most flexible of the four (Klein, 2010). Perhaps as a result of 

this flexibility, most grantee schools—and 96 percent of rural schools nationally—have chosen

this model (Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch, 2012). 

Therefore, our study focuses on the transformation model in rural settings. The transformation 

model requires the following 11 activities (U.S. Department of Education, OESE, 2011):

1. Replace the principal who led the school prior to the transformation model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals 

that take into account data on student growth, as well as other factors

3. Identify and reward school leaders, teachers, and other staff members who improved 

student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote continuous use of student data in order to inform and differentiate instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve 

student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related 

support from the LEA, the SEA, or an external organization
Supporting statement for OMB clearance 3

Study design, data collection, and timing

This descriptive study collects and analyzes principals’ reports of SIG transformation model 

implementation in their rural schools. It uses a one-time survey of the 211 Cohort 1 principals of 

rural schools implementing the SIG transformation model (attachment B). The survey will occur 

in spring 2014. It also incorporates existing data from the SIG baseline database available on the 

U.S. Department of Education website, including each school’s National Center for Education 

Statistics (NCES) locale code (i.e., the geographic characteristics of the school), percentage of 

students eligible for free or reduced-price lunch (i.e., poverty levels among students), and 

percentages of non-White and Native American students. The study uses the data to address 

the following research questions:

1) How do principals of rural SIG transformation schools rate the extent to which their 

school implemented the required activities in the transformation model?

2) What do principals report were the challenges to implementation of each activity?

3) What types of technical assistance providers assisted schools, which particular 

transformation activities did they assist with, and how did the principals rate the 

sufficiency of this assistance?

4) Do principal ratings of implementation, challenges, and technical assistance providers 

vary by school geographic or demographic characteristics?

A2. How, by whom, and for what purpose information is to be used

To make good decisions about ongoing school transformation efforts and future similar school 

improvement initiatives, the nation’s education leaders need more information about rural SIG 

implementation of the transformation model. More specifically, the data from this study will be 

used by REL Northwest and our state contacts to provide information to federal policymakers

during ongoing efforts to reauthorize ESEA. These policymakers need such information as they 

consider changing or retaining language around the requirements for the continued use of SIGs 

under ESEA, which are likely to continue to provide funds to schools for school improvement, 

as well as language incorporating elements of the ESEA waivers involving school improvement 

efforts (U.S. Department of Education, 2012). We will also use these data to provide information 

to state policymakers in states with similar legislation around school improvement. 

Furthermore, we will use these data to inform education practitioners, particularly those who

provide support for school improvement. These practitioners need the data to identify gaps in 

technical assistance for rural schools seeking to improve and uncover opportunities to provide 

additional and/or different services to schools.

Why is it crucial for these policymakers and practitioners to consider changes in policies and 

practices for transforming rural schools? Despite the fact that the transformation model requires 

less disruption and staffing changes, many education leaders anticipate that rural schools, in 

particular, will have difficulty meeting the requirements of the transformation model. For 

example, recruiting and retaining high-quality staff members is known to be more difficult in 

rural areas (Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Monk, 2007). In addition, 

finding technical assistance providers who can work with rural schools can be challenging 

(Corbett, 2011); transforming school schedules to allow for extended learning time can be 
Supporting statement for OMB clearance 4

difficult due to the longer bus rides and higher transportation costs (Sandel & Bhat, 2008); and 

rural parents can be mistrustful of outside technical assistance providers—such as state 

employees or university professors—who seem determined to change their schools (Owens, 

Richerson, Murphy, Jageleweski, & Rossi, 2007). However, there is little empirical information 

about the implementation of the transformation model in rural schools.

This study will provide education policymakers and practitioners with crucial information, 

including: specifying the extent to which the transformation activities were implemented and 

the challenges to implementation and identifying which activities were supported by technical 

assistance providers and how sufficient principals found this support. The study will do this 

primarily through a specially designed survey of the 211 rural school principals implementing 

SIG transformation models in Cohort 1 (funded from 2010 through 2013, with an option to carry 

over funds to 2014). We have designed this survey (attachment B) based on our review of the 

literature on SIG and rural schools and on our staff members’ past work on SIG evaluation. 

The first section of the survey asks principals when they became an administrator at their school

and if they are familiar with SIG implementation at their school. If the principal is new to the 

school and does not know about the SIG implementation, the survey will prompt the principal 

to provide contact information for the prior principal or for a designated proxy, who is 

identified as a staff member who was employed during the SIG funding period and served in 

one of the following roles:

 A school-level administrator, such as an assistant principal, who participated in the 

school’s SIG

 A teacher who was responsible for assisting with the implementation of the school’s 

SIG

 A district administrator responsible for school improvement

The second section of the survey asks principals to rate their school’s implementation of the 11 

required transformation model activities using a four-point scale that reflects the stages of 

implementation developed by Fixsen, Blase, Naoom, and Wallace (2009). Transformation 

activities include:

1

1. Replace the principal who led the school prior to commencement of the transformation 

model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals 

that take into account data on student growth, as well as other factors

 

1

In order to ensure that each survey item represents only one SIG transformation activity, we split several activities 

into multiple survey items. To ensure that the principal is knowledgeable about all activities, we eliminated those 

activities that are district rather than principal responsibilities, such as principal replacement and evaluation. We also 

eliminated the activity, “Ensure that the school receives ongoing, intensive technical assistance and related support 

from the LEA, the SEA, or an external organization” because we ask principals about this assistance in a separate 

section of the survey.
Supporting statement for OMB clearance 5

3. Identify and reward school leaders, teachers, and other staff members who improved 

student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote the continuous use of student data in order to inform and differentiate 

instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve 

student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related 

support from the LEA, the SEA, or an external organization

We ask about other school improvement activities in which rural schools may engage (through 

an open-ended item), and we also ask principals to indicate which activities they believe were 

essential to core efforts to improve the school.

Next, the principals will be asked to select one or more challenges for each activity. We will 

allow principals to select multiple responses for this survey item, since some items may be 

related to one another. For example, “insufficient funding to implement activities” may mean 

that staff members are overworked and, therefore, also have “insufficient staff time to 

implement activities.” Principals may select both responses if appropriate. They will also have 

the option of selecting “NA (no challenges)” or “NA (did not implement this activity).” Finally, 

in an open-ended item, principals may list other challenges to implementing the activities.

The next section of the survey asks principals the extent to which they agree that technical 

assistance from other entities was sufficient to help the school implement the required 

transformation activities. Participants will rate the sufficiency on a four-point agree/disagree 

scale. The choice of entities includes their district, their state education agency, a partner in 

higher education, or another technical assistance provider. This list is based on the entities that 

other studies have found assisted with SIG most often (Corbett, 2011; Scott, McMurrer, 

McIntosh, & Dibner, 2012). If a principal selects “another technical assistance provider,” the 

survey will prompt the principal to fill in the name of the organization that provided the

technical assistance. The principals will also be able to elaborate on their responses regarding 

the sufficiency of the assistance from each entity. 

A3. Use of automated, electronic, mechanical, or other technological collection techniques

Initially, we will invite principals to complete an online survey (with electronic submission of 

responses through Survey Gizmo). If return rates are less than 50 percent after the second 

reminder to complete the survey, we will offer an electronic gift card to increase the response 

rate. We will do this for three reasons. First, our pilot showed that offering an incentive (an 

Amazon electronic gift card) after several reminders increased the response rate from 44 percent 
Supporting statement for OMB clearance 6

to 77 percent. Several research studies support the use of incentives (Armstrong, 1975; Church, 

1993; James, & Bolstein, 1992), particularly for online surveys of principals (Jacob & Jacob, 2012). 

Principals who completed the survey before the second reminder will be sent the incentive 

retroactively.

Those that do not respond after three reminders will have the option of completing a paper 

version of the survey that they may email, fax, or mail back to our partner, PSA. Either version 

will take about the same amount of time to complete (20 minutes). PSA will host a website 

where respondents can access the link to the online survey. Attachment C contains the advance 

mail notification letter, the invitation to participate in the survey, and the follow-up notification 

email/letters.

We have decided to first invite principals to respond to the survey online and then, after three 

reminders, to allow nonrespondents a paper option in case principals preferred this format.

Kaplowitz, Hadlock, and Levine (2004) found that a web survey application achieved similar 

response rates to a mailed hard copy questionnaire when both were preceded by an advance 

mail notification. In addition, Millar and Dillman (2011) found that a combined use of multiple 

response-inducing techniques (web and mail) produced the best response rates.

Study results will be available via an online link to the study on the IES and REL Northwest 

websites. Participating principals and their districts will be invited to participate in webinars to 

discuss the results of the study. In these webinars, participants will share ideas about 

implementing the SIG transformation model and similar school improvement efforts. They will 

also discuss the supports provided by technical assistance providers and how to leverage 

additional supports to address challenges to school improvement. This participation will be 

entirely voluntary and is presented as a way of motivating principals to participate.

A4. Efforts to identify duplication

This one-time study collects unique information that is not available elsewhere and is not being 

collected by other studies. To date, most discussions of rural SIG have combined all rural 

schools regardless of their other geographic and demographic characteristics and have not 

delved deeply into the implementation of the required activities in the transformation model 

(e.g., Dee, 2012; Rosenberg, 2011; Scott, 2012; Scott et al., 2012). While these past studies offered 

important information about SIG, the current study fills a need for information about rural 

schools’ implementation of the required activities in the transformation model. Ongoing 

national studies of SIG, such as Hurlburt and colleagues (2011, 2012), focus on all types of 

schools and on the four school turnaround models broadly, not on characteristics of rural 

schools and the required activities in the transformation model. These studies do not provide 

information about how states and other technical assistance providers might support specific 

school transformation activities in rural settings. 

The American Institutes for Research and Mathematica are currently conducting an in-depth 

descriptive study of 25 SIG schools, including 10 in rural areas. While this study will provide an 
Supporting statement for OMB clearance 7

in-depth look at some rural schools, it relies on less than 5 percent of rural SIG schools and will 

not generalize across rural SIG schools.

A5. Sensitivity to burden on small entities

This project collects data from principals of schools, which by definition are small entities. We 

have minimized the burden for respondents by (1) requesting information that is generally 

available to school leadership, including principals, other administrators, and teacher leaders 

directly involved in SIG implementation, and (2) requesting the minimum amount of 

information required to answer the study questions. Answering these questions will provide 

policymakers and practitioners with information needed to make good decisions about the SIG 

transformation model and similar school improvement efforts. For example, items about the 

geographic and demographic characteristics of the schools are not included in the survey

because this information is available through the SIG baseline database. 

As shown in section “A12. Total Annual Cost Burden to Respondents,” we estimate a 20-minute

total time burden on each respondent. In addition, we will allow principals to designate a 

proxy, if it helps minimize the burden on the school.

A6. Consequence to Federal program or policy activities if the collection is not conducted or is 

conducted less frequently

This one-time data collection has the potential to inform federal, state, and local agency 

programs and policies related to turning around low-performing schools both through future 

SIG allocations and other efforts such as NCLB waivers. Without this study, policymakers and 

their supporters would have little information about the unprecedented $3 billion that went into 

SIG in 2009 through the ARRA—about six times what Congress had allocated the previous year 

(Scott et al., 2012). Of these funds, more than $300 million went to rural transformation in the

first cohort of schools (Hurlburt et al., 2011). In addition, the types of efforts required by federal 

guidance to turn around low-performing schools may be similar in the future. This study 

informs this future work on other school improvement efforts that are similar to SIG.

A7. Special circumstances

There are no special circumstances. The request fully complies with the regulations.

A8. Federal Register announcement and consultation

Federal Register announcement

ED will publish Federal Register Notices to allow both a 60-day and 30-day public comment 

period. The REL will assist ED in addressing any public comments received. 

There has been no comments received under the 60 day comment period notice.
Supporting statement for OMB clearance 8

Consultations outside the agency

Prior to this request for OMB approval, the research team received expert feedback on this 

survey and study from researchers outside the agency through REL Northwest’s Technical 

Working Group (TWG). The two TWG members who contributed to this study were 

distinguished experts in their fields and extremely knowledgeable about research design and 

methods: Tom Dee of Stanford University and Hans Bos of American Institutes for Research.

These TWG members provided written feedback on study plans and/or reports and met via 

telephone with the research team to clarify their feedback and provide additional

recommendations.

A9. Payment or gift to respondents

As described in section A3, if return rates are less than 50 percent after the second reminder to 

complete the survey, we will offer an electronic gift card to increase the response rate. We will 

do this for three reasons. First, during the pilot, response rates were low until we offered a gift 

card from Amazon. Upon the first request, only two principals (22 percent) completed the 

survey. After two reminders, four principals (44 percent) had completed the survey and one 

had partially completed it. PSA staff decided to provide an Amazon gift card as an incentive. 

Several principals responded positively to the incentive. The principal, who had only partially 

completed the survey, finished the survey. Two others also completed the pilot survey for a 

response rate of 77 percent. More details about the pilot are contained in attachment F.

Second, we reviewed the literature and found several reviews of survey research that showed

incentives increased response rates (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992). In 

addition, a recent experimental study of the impact of incentives on the return rate of a 

principal survey (Jacob & Jacob, 2012) influenced our thinking. In this experimental survey of 

1,177 principals, offering an incentive had a statistically significant positive impact on the return 

rate: 48 percent of principals offered an incentive with the online survey returned the survey 

compared to 22 percent who were not offered the incentive.

Third, REL guidance recommends the use of incentives for groups that are surveyed frequently 

(Sloan, Ingels, & Burghardt, 2012). Because our participants are principals, they are likely to be 

surveyed frequently. Research shows that 50 percent of principals, in general, receive four or 

more survey requests per year and 25 percent receive more than seven per year (Jacob, Scott, &

Bowers, 2008). 

Therefore, based on the results of our pilot, the research literature, and REL guidance on survey 

incentives, we will offer an incentive to principals. In order to avoid offering the incentive 

unnecessarily, we will first ask principals to complete the survey without offering an incentive. 

Then, if at the second reminder the return rate is less than 50 percent (as it was in the pilot), then 

we will offer a $20 Amazon gift card. Principals, who already completed the survey without an 

incentive, will receive the $20 Amazon gift card retroactively as a thank you. We chose $20 as 

the amount of the gift card because this is the amount recommended in REL guidance (Sloan, 

Ingels, & Burghardt, 2012).
Supporting statement for OMB clearance 9

In addition to the gift card, participants will receive a link to the published study and 

invitations to several webinars to discuss the study. While this is not a payment or gift, it will 

allow participants to have a voice in the discussion of study results. During the webinar, 

participants will share information about implementing SIG and similar school improvement 

efforts. They will also discuss the supports provided by technical assistance providers and how 

to leverage additional supports for schools to address challenges to school improvement. We

believe this opportunity will motivate people to complete the survey.

A10. Confidentiality of the data

REL Northwest and PSA will be following the new policies and procedures required by the 

Education Sciences Reform Act of 2002, Title I, Part E, Section 183: “All collection, maintenance, 

use, and wide dissemination of data by the Institute” are required to “conform with the 

requirements of section 552 of title 5, United States Code, the confidentiality standards of 

subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act 

(20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights 

and Privacy Act, and the Protection of Pupil Rights Amendment. Subsection (c) of section 183 

referenced above requires the Director of IES to “develop and enforce standards designed to 

protect the confidentiality of persons in the collection, reporting, and publication of data.” 

Subsection (d) of section 183 prohibits the disclosure of individually identifiable information,

and makes any publishing or communicating of individually identifiable information by 

employees or staff a felony.

REL Northwest and PSA will protect the confidentiality of all information collected for the 

study and will use it for research purposes only. No information that identifies any study 

participant will be released. Information from participating institutions and respondents will be 

presented at aggregate levels in reports. Unique identification numbers will be assigned to each 

participating school and used to identify all survey responses. The ID number/name association 

files will be kept secure in a confidential file separate from the data analysis file. No information 

identifying respondents will be included in the study data files or reports. Data from the online 

survey software system will be downloaded and deleted from the online system within one 

week after the data collection ends. These data files will then be stripped of any identifying 

information. Information will be reported in aggregate so that individual responses are not 

identifiable. All identification lists will be destroyed at the end of the project.

The research team is trained to follow strict guidelines for soliciting consent, administering data 

collection instruments, and preserving respondent confidentiality. All members of the research 

team will have successfully completed the Collaborative Institutional Training Initiative (CITI) 

course in the Protection of Human Research Subjects through Liberty IRB. A copy of the 

affidavit of nondisclosure is provided in attachment G and will be signed by each researcher 

who will have access to the data.

All study materials will include the following language:
Supporting statement for OMB clearance 10

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, 

Part E, Section 183, responses to this data collection will be used only for statistical purposes. 

The reports prepared for this study will summarize findings across the sample and will not 

associate responses with a specific district or individual. We will not provide information that 

identifies you or your district to anyone outside the study team, except as required by law. Any 

willful disclosure of such information for nonstatistical purposes, without the informed consent of 

the respondent, is a class E felony.

A11. Additional justification for sensitive questions

No questions of a highly sensitive nature are included in the survey or interview, and we have 

received Institutional Review Board approval for this study.

A12. Estimates of hour burden

We estimate the total reporting burden associated with this data collection to be 61.58 hours and 

$2,609.90 (table 1). This includes 63 principals designating a proxy and 169 survey respondents 

completing the survey (i.e., 80 percent of 211). More details about these activities are included 

below in table 1. We estimated a principal’s hourly wage using May 2012 Bureau of Labor 

Statistics National Occupational Employment and Wage Estimates 

(http://www.bls.gov/oes/current/oes_nat.htm).

Designating a proxy

We anticipate that approximately 63 principals (i.e., 30 percent of 211) may designate a proxy to 

fill out the survey because they are either new to the school or too busy to fill out the survey. 

We based this estimate on the national average of 20 percent principal attrition yearly (Battle, 

2010), plus an additional 10 percent of principals who may be too busy. We anticipate that the 

designation of proxies will add about 5 minutes to the burden on schools. Designating a proxy 

adds 5.25 hours to the burden and $222.50.

Survey

We will invite all 211 principals or designated proxies of our target population (i.e., the Cohort 1 

rural SIG transformation schools) to participate in the survey. The target response rate for these 

individuals is 80 percent, and the approximate time required for each respondent to complete 

the online survey is 20 minutes on average, based on the pilot and on a similar survey (Scott, 

Davis, & Krasnoff, 2012). The total hours for the survey are 56.33, and the total cost is $2,387.41.

The most recent Schools and Staffing Survey of principals by the National Center of Education 

Statistics had a response rate of 79 percent for both public school and Bureau of Indian 

Education principals (Battle, 2009). This report did not disaggregate the response rates for rural 

school principals. But, we suspect that response rates would be somewhat lower for rural school 

principals. For example, the response rate of the rural district administrators’ survey was 74

percent (Zhang, 2008). Therefore, because we set our target at 80 percent, we will use several 

methods to maximize response rates and address nonresponse, such as offering an incentive as 

described in A9 and the use of online data collection described in A3.
Supporting statement for OMB clearance 11

Table 1: Administration times

Reporting 

method

Number of 

respondents

Average 

time 

(hours)

Total burden

(hours)

Hourly rate Estimated cost

Designating a 

proxy 63 5 minutes 5.25 $42.38 $ 222.50 

Survey 169 20 minutes 56.33 $42.38 $2,387.41

Total 61.58 $2,609.90

A13. Estimate for the total annual cost burden to respondents or record keepers

There are no direct start-up costs to respondents.

A14. Estimates of annualized costs to the Federal Government

The total estimated cost for this study is $373,250. Although the data collection occurs only 

once, these costs cover a two-year study period; therefore, the annual cost of the study is 

$186,625. We estimated these costs using the amounts budgeted in the REL Northwest Year 2 

plan and approved by IES. These costs include planning the study, administering the survey, 

collecting and cleaning data (including the use of an incentive), and analyzing and reporting

results. In addition to the evaluation costs, there are personnel costs of several Federal 

employees involved in the oversight and analysis of information collection that amount to an 

annualized cost of $5,000 for Federal labor.

A15. Reasons for program changes or adjustments

This is a new study.

A16. Plans for tabulation and publication

We will tabulate and analyze the data by the four research questions. We will address the first 

three questions by providing the frequencies of survey responses for each transformational 

activity by level of implementation, challenges, and technical assistance provider. We will also 

explore the first three research questions using open-ended items on the survey. The fourth 

research question requires inferential statistics. Plans for tabulation and analysis are described 

in more detail below.

Research question 1: How do principals of rural SIG transformation schools rate the extent to 

which their school implemented the required activities in the transformation model?

We will address the first research question primarily by calculating frequencies of survey 

responses for each transformation activity by the extent of implementation, (i.e., N/A, 

planning/developing, early or partial implementation, or full implementation). The categories of 

levels of implementation are defined more fully in the survey (attachment B). We will also 

provide descriptive statistics about the percentages of principals that rate these activities as 

essential to their core efforts to improve schools.
Supporting statement for OMB clearance 12

We will use one open-ended item to add details about the implementation of activities related 

to school improvement. The item asks principals to elaborate on their responses regarding 

transformation model activities that are fully implemented in their school and essential to core 

efforts to improve the school. To examine the item, we will develop codes inductively (Mayring, 

2000). To develop codes, the two study authors will review approximately a third of the data 

independently and create codes. They will then compare these codes and agree on final codes. 

Next, they will both independently code the remaining data and compare results. If they cannot 

agree on the code for a particular piece of data, a third researcher will determine the final code.

Research question 2: What do principals report were the challenges to implementation of each 

activity?

As in research question one, we will address this question primarily through frequencies of 

survey responses about challenges. Because some activities may pose multiple challenges, 

principals may select all challenges that apply (i.e., insufficient funding, insufficient staff 

expertise, insufficient staff time, insufficient technological capacity/equipment, lack of teacher

support, lack of district support/guidance, other).

Principals will also be given the opportunity to elaborate on their responses regarding 

challenges. To conduct an exploratory analysis of this qualitative data, we will develop codes 

inductively (Mayring, 2000). To create codes, the two study authors will review approximately 

one third of the data independently. They will then compare these codes and agree on final 

codes. Next, they will both independently code the remaining data and compare results. If they 

cannot agree on the code for a particular piece of data, a third researcher will determine the 

final code.

Research question 3: What types of technical assistance providers assisted schools, which 

particular transformation activities did they assist with, and how did the principals rate the 

sufficiency of this assistance?

We will provide two types of descriptive statistics to explore this question. First, we will 

provide frequencies for the principals reporting that their school received assistance on each of 

the transformation activities for particular types of providers (i.e., districts, states, universities, 

and other). Then, we will provide descriptive statistics representing the degree to which 

principals whose schools received this assistance agreed that the assistance was sufficient.

To provide more detail about technical assistance providers, we will examine the open-ended 

response that asks principals to provide the name(s) of their technical assistance organization(s)

We will list the top 10 organizations in the main body of the report and the complete list of 

organizations in the technical appendix.

Finally, open-ended items about each type of technical assistance provider will allow principals 

to elaborate on their responses regarding the sufficiency of the technical assistance from each 
Supporting statement for OMB clearance 13

type of provider. We will analyze this qualitative data using deductive coding (Mayring, 2000). 

As described previously, two study authors will review the data independently, then compare 

and agree on final codes. Disagreements, if any, will be settled by a third researcher.

Research question 4: Do principal ratings of implementation, challenges, and technical 

assistance providers vary by school geographic or demographic characteristics?

To examine variation in stages of implementation and in assistance by the geographic and 

demographic characteristics of schools, we will use the Kruskal Wallis H test. The Kruskal 

Wallis H test is a nonparametric test that allows the comparison of mean ranks of more than 

two categorical groups. We have chosen to use this test because, in our study, the dependent 

variables of interest (our survey responses about the stages of implementation and about 

sufficiency of assistance) are ordinal and the independent variables of interest (geographic and 

demographic characteristics of schools) are nominal. We will conduct four series of Kruskal 

Wallis H tests:

1) Implementation by geographic characteristics of schools, (i.e., by locale codes 

representing the type of rural area a school is in, such as a rural remote area versus a 

distant town)

2) Sufficiency of assistance by geographic characteristics of schools

3) Implementation by demographic characteristics of schools (i.e., by whether the school 

serves primarily white, Latino, or Native American students)

4) Sufficiency of assistance by demographic characteristics of schools

We discuss each of these series of analyses in more detail below.

To examine variations in reported implementation by geographic characteristics, we will 

conduct the H tests for each item’s response category by each of the six locale codes (31, 32, 33, 

41, 42, and 43). However, any cells with fewer than five responses will be eliminated from the 

analysis or will be collapsed if possible, (e.g., strongly agree and agree could be combined and 

strongly disagree and disagree could be combined). In each analysis, the ordinal Likert scale 

responses for the survey item will serve as the dependent variable. The independent or 

“grouping” variable will include the six locale codes. If the H test is statistically significant, we 

will perform a post hoc Dunn’s test to determine which specific groups were different from one 

another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test 

statistics for each analysis. To determine statistical significance we will use the BenjaminiHochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by geographic characteristics, we will also 

conduct H tests for the items about assistance. For these items, we will consider assistance from 

each type of provider (district, state, university, and other technical assistance providers) 

separately. We expect that in many instances this may result in cells with fewer than five 

responses. Those items with small cell sizes will be eliminated from the analysis. In each 

analysis, the ordinal Likert scale responses for the survey item will serve as the dependent 
Supporting statement for OMB clearance 14

variable. The independent or “grouping” variable will include the six locale codes. If the H test 

is statistically significant, we will perform a post hoc Dunn’s test to determine which specific 

groups were different from one another. We will display mean ranks, degrees of freedom, H 

test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance 

we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & 

Hochberg, 1995).

To examine variations in reported implementation by demographic characteristics, we will 

classify the schools in four ways: those that are more than 50 percent Latino (28 schools), those 

that are more than 50 percent Native American (50 schools), those that are more than 50 percent 

White (68 schools), and those that do not have a single racial majority (64 schools).2 We will use 

Kruskal Wallis H tests to examine variations in the data on implementation by student 

demographic characteristics. We will conduct the H tests for each item’s response category by 

each of the four demographic categories, although any cells with fewer than five responses will 

be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the 

survey item will serve as the dependent variable. The independent or “grouping” variable will 

include the four demographic categories. If the H test is statistically significant, we will perform 

a post hoc Dunn’s test to determine which groups differed from one another. We will display 

mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a 

technical appendix. To determine statistical significance we will use the Benjamini-Hochberg 

correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by demographic characteristics, we will also use 

Kruskal Wallis H tests. As in the previously described analysis of implementation, we will 

consider assistance from each type of provider (district, state, university, and other technical 

assistance providers) separately. We expect that in many instances we will need to eliminate 

analysis for cells with fewer than five responses. In each analysis, the ordinal Likert scale 

responses for the survey item will serve as the dependent variable. The independent or 

“grouping” variable will include the four demographic categories. If the H test is statistically 

significant, we will perform a post hoc Dunn’s test to determine which groups differed from 

one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test 

statistics for each analysis in a technical appendix. To determine statistical significance we will 

use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & 

Hochberg, 1995).

Publications

For this study, we will create two publications. A What’s Happening? report will be targeted to 

our national practitioner and policymaker audience, particularly state agencies, districts, 

universities, and other technical assistance providers involved in supporting school 

improvement efforts similar to those in SIG transformation schools. This report will contain 

graphic summaries of the study and its findings. 

 

2 One school had missing data for student ethnicity in the SIG baseline database.
Supporting statement for OMB clearance 15

A Stated Briefly report will summarize findings that may appeal to regional and state 

organizations that are interested in improving schools but that do not provide direct technical 

assistance to schools—particularly those organizations that set school improvement policies and 

guidance. The Stated Briefly report should help these organizations set policy and create 

guidance that supports aspects of transformation that fewer participants reported were fully 

implemented, as well as potentially reducing or remediating such reported challenges. 

Both What’s Happening and Stated Briefly reports will be available on the REL Northwest website 

and the IES website. As stated earlier, geographic and demographic analyses will not be 

conducted when fewer than five respondents are included. This will reduce the risk of 

deductive disclosure.

Timeline

The timeline for the study, including the publications, is presented in table 2.

Table 2. Timeline

Timeline Milestone Date for deliverable

January 2013–April 2013 Develop data collection protocol and 

instruments

NA

April 2, 2013 Submit draft study plan to TWG, 

which includes study methodology, 

data collection activities, and 

analysis plan

NA

April 30, 2013 Submit draft study plan to IES, 

revised in response to TWG 

feedback

Draft plan to IES April 30, 

2013

July 24, 2013 Submit final study plan to IES Final plan to IES June 14, 

2013

October 2013 OMB submission of 60-day Federal 

Register Notice

NA

March–May 2014 Data collection NA

June 2014 Data analysis NA

July 2014 Write reports NA

September 2014 Submit draft What’s Happening?

report to IES

Draft What’s Happening?

report, 12 months after 

approval

November 2014 Submit final What’s Happening?

report to IES

Final What’s Happening?

report, 13–14 months after 

approval

December 2015 Submit draft Stated Briefly to IES Draft Stated Briefly, 

15 months after approval

January 2015 Submit final Stated Briefly to IES Final Stated Briefly, 

16 months after approval

A17. Approval to not display the expiration date for OMB approval

This is not applicable. The instrument will display the expiration date for OMB approval. 
Supporting statement for OMB clearance 16

A18. Exception to the certification statement

We do not seek any exceptions to the certification statement.
Supporting statement for OMB clearance 17

References

Armstrong, S.J. (1975). Monetary incentives in mail surveys. Public Opinion Quarterly, 39(1), 111–

116.

Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and 

secondary school principals in the United States: Results from the 2007–08 Schools and Staffing 

Survey (First Look, NCES 2009-323). Retrieved from U.S. Department of Education, 

National Center for Education Statistics website: 

http://nces.ed.gov/pubs2009/2009323.pdf

Battle, D. (2010). Principal attrition and mobility: Results from the 2008–09 principal follow-up survey

(First Look, NCES 2010-337). Retrieved from U.S. Department of Education, National 

Center for Education Statistics website: http://nces.ed.gov/pubs2010/2010337.pdf

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful 

approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 

57(1), 289–300.

Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A metaanalysis. Public Opinion Quarterly, 57(1), 62–79.

Corbett, J. (2011). Lead turnaround partners: How the emerging marketplace of lead turnaround 

partners is changing school improvement. Lincoln, IL: Academic Development Institute, 

Center on Innovation and Improvement.

Dee, T. (2012). School turnarounds: Evidence from the 2009 stimulus (NBER Working Paper No. 

17990). Cambridge MA: National Bureau of Economic Research.

Fixsen, D.L., Blase, K.A., Naoom, S.F., & Wallace F. (2009). Core implementation components.

Research on Social Work Practice, 19(5), 531–540.

Hammer, P.C., Hughes, G., McClure, C., Reeves, C., & Salgado, D. (2005). Rural teacher 

recruitment and retention practices: A review of the research literature, national survey of rural 

superintendents, and case studies of programs in Virginia. Nashville, TN: Edvantia, 

Appalachia Educational Laboratory.

Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications 

and SIG-eligible and SIG-awarded schools (NCEE 2011-4019). Retrieved from U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance website: 

http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf
Supporting statement for OMB clearance 18

Hurlburt, S., Therriault, S.B., & Le Floch, K.C. (2012). School Improvement Grants: Analyses of state 

applications and eligible and awarded schools (NCEE 2012-4060). Retrieved from U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance website: 

http://ies.ed.gov/ncee/pubs/20124060/pdf/20124060.pdf

Jacob, R.T., & Jacob, B. (2012). Pre-notification, incentives, and survey modality: An 

experimental test of methods to increase survey response rates of school principals. 

Journal of Research on Educational Effectiveness, 5(4), 401–418.

Jacob, R.T., Scott, L., & Bowers, A. (2008, April). Evaluation and implementation of web-surveys in 

school-based settings. Paper presented at the annual meeting of the American Educational 

Research Association, New York, NY.

James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow-up mailings on 

the response rate and response quality in mail surveys. Public Opinion Quarterly, 54(3), 

346–361.

Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004). A comparison of web and mail survey 

response rates. Public Opinion Quarterly, 68(1), 94–101.

Klein, A. (2010, May 21). School turnaround models draw bipartisan concern. Education Week, 

29(33). Retrieved from 

http://www.edweek.org/ew/articles/2010/05/21/33turnaround.h29.html

Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1(2). 

Retrieved from http://www.qualitativeresearch.net/index.php/fqs/article/view/1089/2386

Millar, M.M., & Dillman, D.A. (2011). Improving response to web and mixed-mode surveys. 

Public Opinion Quarterly, 75(2), 249–269. 

Monk, D.H. (2007). Recruiting and retaining high-quality teachers in rural areas. Future of 

Children, 17(1), 155–174.

No Child Left Behind Act of 2001. (2002). Pub. L. No. 107-110, 115 Stat. 1425.

Owens, J.S., Richerson, L., Murphy, C.E., Jageleweski, A., & Rossi, L. (2007). The parent 

perspective: Informing the cultural sensitivity of parenting programs in rural 

communities. Child & Youth Care Forum, 36(5/6), 179–194.

Rosenberg, S.J. (2011). The challenges in implementing school improvement grant models in rural high 

schools (Master’s thesis). Duke University, Durham, NC.
Supporting statement for OMB clearance 19

Sandel, K., & Bhat, S. (2008). Financing and sustaining out-of-school time programs in rural 

communities. Retrieved from Finance Project website: 

http://www.financeproject.org/publications/ostruralyouth.pdf

Scott, C., Davis, A., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011–

2012. Portland, OR: Education Northwest.

Scott, C., McMurrer, J., McIntosh, S., & Dibner, K. (2012). Opportunities and obstacles: 

Implementing stimulus-funded School Improvement Grants in Maryland, Michigan, and Idaho. 

Washington, DC: Center on Education Policy.

Scott, G. (with Sirois, E., Spicer, S., Arsenault, J., King, M., Sorbello, S., & Steel-Lowney, B.). 

(2012). School Improvement Grants: Education should take additional steps to enhance 

accountability for schools and contractors (GAO-12-373). Washington DC: U.S. Government 

Accountability Office.

Sloan, M., Ingels, J., & Burghardt, J. (2012). Analytical and technical support for advancing education 

evaluation: How to put together an OMB supporting statement. Princeton, NJ: Mathematica 

Policy Research.

U.S. Department of Education. (2012, June 7). ESEA flexibility (Updated ed.). Retrieved from 

http://www.ed.gov/esea/flexibility/documents/esea-flexibility.doc

U.S. Department of Education, Office of Elementary and Secondary Education. (2011). Guidance 

on fiscal year 2010 School Improvement Grants under section 1003(g) of the Elementary and 

Secondary Education Act of 1965. Retrieved from 

http://www2.ed.gov/programs/sif/sigguidance02232011.pdf

Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. 

Retrieved from Center on Education Policy website: http://www.cepdc.org/displayDocument.cfm?DocumentID=308