CONTRACT: ED-IES-12-C-0003

Supporting Statement for OMB Clearance of a Survey of 

Principals of Rural Schools Receiving School Improvement 

Grants and Using the Transformation Model

Section A

August 16, 2013

Revised September 6, 2013

Revised October 8, 2013

Revised Oct 16, 2013
ii

Contents

Section A: Justification...............................................................................................................................1

A1. Circumstances Necessitating Collection of Information............................................................... 1

A2. How, by Whom, and for What Purpose Information Is to Be Used............................................ 3

A3. Use of Automated, Electronic, Mechanical or Other Technological Collection 

Techniques ..................................................................................................................................... 5

A4. Efforts to Identify Duplication .......................................................................................................... 6

A5. Sensitivity to Burden on Small Entities............................................................................................ 6

A6. Consequence to Federal Program or Policy Activities if the Collection Is Not 

Conducted or Is Conducted Less Frequently............................................................................ 7

A7. Special Circumstances........................................................................................................................ 7

A8. Federal Register Announcement and Consultation....................................................................... 7

A9. Payment or Gift to Respondents....................................................................................................... 8

A10. Confidentiality of the Data .............................................................................................................. 8

A11. Additional Justification for Sensitive Questions .......................................................................... 9

A12. Estimates of Hour Burden ............................................................................................................... 9

A13. Estimate for the Total Annual Cost Burden to Respondents or Record Keepers .................. 10

A14. Estimates of Annualized Costs to the Federal Government..................................................... 10

A15. Reasons for Program Changes or Adjustments.......................................................................... 10

A16. Plans for Tabulation and Publication........................................................................................... 10

A17. Approval to Not Display the Expiration Date for OMB Approval.......................................... 15

A18. Exception to the Certification Statement..................................................................................... 15

References ................................................................................................................................................. 16
iii

List of Tables

Table 1. Administration times ................................................................................................................ 10

Table 2. Timeline ..................................................................................................................................... 15

The following attachments are stored in the “NW S07” folder in the “REL Northwest Memos 

and Correspondence” library, with the exceptions of Attachments D and E which we will add 

later since the Federal Register notices cannot be submitted until this document is approved by 

IES:

Attachment A: ESRA

Attachment B: Copy of Survey and Online Survey Screen Shots

Attachment C: Intro Letter Invitation and Follow-Up Emails

Attachment D: Affidavits of Nondisclosure

Attachment E: IRB Approval Form
Supporting Statement for OMB Clearance 1

Supporting Statement for OMB Clearance of a Survey of 

Principals of Rural Schools Receiving School Improvement 

Grants and Using the Transformation Model

Section A: Justification

A1. Circumstances Necessitating Collection of Information

This data collection is authorized by the Educational Sciences Reform Act (ESRA) of 2002 (see 

attachment A). Part D, Section 174(f)(2) of ESRA states that as part of its central mission and 

primary function, each regional educational laboratory “shall support applied research by ... 

developing and widely disseminating, including through Internet-based means, scientifically 

valid research, information, reports, and publications that are usable for improving academic

achievement, closing achievement gaps, and encouraging and sustaining school improvement, 

to schools, districts, institutions of higher education, educators (including early childhood 

educators and librarians), parents, policymakers, and other constituencies, as appropriate, 

within the region in which the regional educational laboratory is located.”

REL Northwest and our partner Policy Studies Associates (PSA), the authors of this study, are

interested in how rural schools implement School Improvement Grants (SIGs). These 

competitive federal grants have provided funds to two cohorts of low-performing schools to 

increase student achievement. The grants require schools to implement one of four school 

improvement models: the transformation model, which involves a variety of reforms; the 

turnaround model, which involves primarily replacing staff; the restart model, which involves

becoming a charter school; and the closure model, which closes the school. 

This descriptive study collects and reports on data from rural school principals across the nation 

that received federal SIGs in Cohort 1 and used the transformation model. This information 

collection is essential to a study by the Regional Educational Laboratory (REL) Northwest, 

which serves five Northwest states (Alaska, Idaho, Montana, Oregon, and Washington). Most

rural Northwest schools with SIG funding (96 percent) used the transformation model to 

improve. REL Northwest and its partner PSA are collecting survey data about the 

implementation of the transformation model to inform regional and national supports for 

future school improvement efforts. Data collection will occur once in spring 2014. IES has 

approved the study plan.

Statement of Need and Background Information

To date, there is little systematic evidence about the implementation of the SIG program. In the 

case of rural SIG settings, this lack of research evidence is particularly problematic. Rural 

schools often face steep challenges when trying to implement the kinds of staff replacement and 

on-site professional development practices promoted by the SIG program (Klein, 2010). This 

study addresses these challenges by examining the level of implementation of the SIG 

transformation model in rural regions, identifying the organizations—states, districts, 
Supporting Statement for OMB Clearance 2

universities, and other entities—that assist schools in these implementation efforts, and 

exploring perceived gaps in assistance. 

SIG funding mechanisms are not new. Congress put them in place as part of the No Child Left 

Behind Act (NCLB) of 2001 (2002), the reauthorization of the Elementary and Secondary 

Education Act (ESEA). However, in 2009 the American Recovery and Reinvestment Act (ARRA) 

substantially increased funding for these grants. With this increase, the U.S. Department of 

Education issued new guidance, dramatically changing SIG requirements.

Under this guidance, federal funds pass to states as usual, but states are now charged with 

assisting schools in implementing one of four school improvement models (U.S. Department of 

Education, Office of Elementary and Secondary Education [OESE], 2011): (1) transformation, 

replacing the principal and implementing a number of strategies to improve the school; (2) 

turnaround, replacing the principal and 50 percent of staff members and providing operational 

flexibility needed to implement a variety of school improvement strategies; (3) restart, 

reopening as a school operated by a charter or education management organization; and (4) 

closure, closing the school and ensuring that students transfer to a high-performing school.

Because the transformation model requires less disruption and staff member replacement than 

other models, some consider it the most flexible of the four (Klein, 2010). Perhaps as a result of 

this flexibility, most grantee schools—and 96 percent of rural schools nationally—have chosen

this model (Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch, 2012). 

Therefore, our study focuses on the transformation model in rural settings. The transformation 

model requires the following 11 activities (U.S. Department of Education, OESE, 2011):

1. Replace the principal who led the school prior to the transformation model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals 

that take into account data on student growth, as well as other factors

3. Identify and reward school leaders, teachers, and other staff members who improved 

student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote continuous use of student data in order to inform and differentiate instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve 

student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related 

support from the LEA, the SEA, or an external organization

Study Design, Data Collection, and Timing

This descriptive study collects and analyzes principals’ reports of SIG transformation model 

implementation in their rural schools. It uses a one-time survey of the 211 Cohort 1 principals of 
Supporting Statement for OMB Clearance 3

rural schools implementing the SIG transformation model (attachment B). The survey will occur 

in spring 2014. It also incorporates existing data from the SIG baseline database available on the 

U.S. Department of Education website, including each school’s National Center for Education 

Statistics (NCES) locale code (i.e., the geographic characteristics of the school), percentage of 

students eligible for free or reduced-price lunch (i.e., poverty levels among students), and 

percentages of non-White and Native American students. The study uses the data to address 

the following research questions:

1) How do principals of rural SIG transformation schools rate the extent to which their 

school implemented the required activities in the transformation model?

2) What do principals report were the challenges to implementation of each activity?

3) What types of technical assistance providers assisted schools, which particular 

transformation activities did they assist with, and how did the principals rate the 

sufficiency of this assistance?

4) Do principal ratings of implementation, challenges, and technical assistance providers 

vary by school geographic or demographic characteristics?

A2. How, by Whom, and for What Purpose Information Is To Be Used

To make good decisions about ongoing school transformation efforts and future similar school 

improvement initiatives, the nation’s education leaders need more information about rural SIG 

implementation of the transformation model. More specifically, the data from this study will be 

used by REL Northwest and our state contacts to provide information to federal policymakers

during ongoing efforts to reauthorize ESEA. These policymakers need such information as they 

consider changing or retaining language around the requirements for the continued use of SIGs 

under ESEA, which are likely to continue to provide funds to schools for school improvement, 

as well as language incorporating elements of the ESEA waivers involving school improvement 

efforts (U.S. Department of Education, 2012). We will also use these data to provide information 

to state policymakers in states with similar legislation around school improvement. 

Furthermore, we will use these data to inform education practitioners, particularly those who

provide support for school improvement. These practitioners need the data to identify gaps in 

technical assistance for rural schools seeking to improve and uncover opportunities to provide 

additional and/or different services to schools.

Why is it crucial for these policymakers and practitioners to consider changes in policies and 

practices for transforming rural schools? Despite the fact that the transformation model requires 

less disruption and staffing changes, many education leaders anticipate that rural schools, in

particular, will have difficulty meeting the requirements of the transformation model. For 

example, recruiting and retaining high-quality staff members is known to be more difficult in 

rural areas (Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Monk, 2007). In addition, 

finding technical assistance providers who can work with rural schools can be challenging 

(Corbett, 2011); transforming school schedules to allow for extended learning time can be 

difficult due to the longer bus rides and higher transportation costs (Sandel & Bhat, 2008); and 

rural parents can be mistrustful of outside technical assistance providers—such as state 
Supporting Statement for OMB Clearance 4

employees or university professors—who seem determined to change their schools (Owens, 

Richerson, Murphy, Jageleweski, & Rossi, 2007). However, there is little empirical information 

about the implementation of the transformation model in rural schools.

This study will provide education policymakers and practitioners with crucial information, 

including: specifying the extent to which the transformation activities were implemented and 

the challenges to implementation and identifying which activities were supported by technical 

assistance providers and how sufficient principals found this support. The study will do this 

primarily through a specially designed survey of the 211 rural school principals implementing 

SIG transformation models in Cohort 1 (funded from 2010 through 2013, with an option to carry 

over funds to 2014). We have designed this survey (attachment B) based on our review of the 

literature on SIG and rural schools and on our staff members’ past work on SIG evaluation. 

The first section of the survey asks principals how long they have been a principal and when 

they became an administrator at their school. If the principal was not an administrator at the 

school during the SIG funding period, a follow-up item will ask whether the principal knows 

how the school implemented the SIG. If the principal is new to the school and does not know 

about the SIG implementation, the survey will prompt the principal to provide contact 

information for the prior principal or for a designated proxy, who is identified as a staff 

member who was employed during the SIG funding period and served in one of the following 

roles:

 A school-level administrator, such as an assistant principal, who participated in the 

school’s SIG

 A teacher who was responsible for assisting with the implementation of the school’s 

SIG

 A district administrator responsible for school improvement

The second section of the survey asks principals to rate their school’s implementation of the 11 

required transformation model activities using a four-point scale that reflects the stages of 

implementation developed by Fixsen, Blase, Naoom, and Wallace (2009). Transformation 

activities include:

1

1. Replace the principal who led the school prior to commencement of the transformation 

model

2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals 

that take into account data on student growth, as well as other factors

3. Identify and reward school leaders, teachers, and other staff members who improved 

student outcomes, and identify and remove those who did not

4. Provide staff with ongoing, high-quality, job-embedded professional development

 

1

In order to ensure that each survey item represents only one SIG transformation activity, we split several activities 

into multiple survey items. We also eliminated the activity, “Ensure that the school receives ongoing, intensive 

technical assistance and related support from the LEA, the SEA, or an external organization” because we ask 

principals about this assistance in a separate section of the survey.
Supporting Statement for OMB Clearance 5

5. Implement strategies designed to recruit, place, and retain staff

6. Use data to identify and implement a new instructional program

7. Promote the continuous use of student data in order to inform and differentiate 

instruction

8. Establish schedules and strategies that provide increased learning time

9. Provide ongoing mechanisms for family and community engagement

10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve 

student outcomes

11. Ensure that the school receives ongoing, intensive technical assistance and related 

support from the LEA, the SEA, or an external organization

We also want to gather information about challenges to implementation. In the second section 

of the survey, the principals will be asked to select one or more challenges for each activity. We 

will allow principals to select multiple responses for this survey item, since some items may be 

related to one another. For example, “insufficient funding” may mean that staff members are 

overworked and, therefore, also have “insufficient staff time” to implement the model fully. 

Principals may select both responses if appropriate. They will also have the option of selecting 

“NA (no challenges).”

This section of the survey also includes two open-ended items. The first item gathers 

information about which activities were important to the school’s improvement efforts. The 

second will collect information about other school improvement activities that rural schools 

may engage in.

The third and last section of the survey asks principals to identify the entities that assisted them 

as they implemented the required transformation model activities. We will allow principals to 

check all that apply. The choice of entities includes their district, their state education agency, a 

partner in higher education, or another technical assistance provider. This list is based on the 

entities that other studies have found assisted with SIG most often (Corbett, 2011; Scott, 

McMurrer, McIntosh, & Dibner, 2012). If a principal selects “another technical assistance 

provider,” the survey will prompt the principal to fill in the name of the organization that 

provided the technical assistance. Then, for each entity that the principal selects, the principal 

will indicate the degree to which he or she agrees that assistance from this entity was sufficient. 

Participants will rate the sufficiency on a four-point agree/disagree scale.

A3. Use of Automated, Electronic, Mechanical or Other Technological Collection 

Techniques

Initially, we will invite principals to complete an online survey (with electronic submission of 

responses through Survey Gizmo). Those that do not respond to this initial invitation and three 

reminders will have the option of completing a paper version of the survey (with electronic,

faxed, or handwritten submission of responses) that they may email, fax, or mail back to our 

partner PSA. Either version will take about the same amount of time to complete (20 minutes). 
Supporting Statement for OMB Clearance 6

PSA will host a website where respondents can access the link to the online survey. Attachment 

C contains the advance mail notification letter, the invitation to participate in the survey, and 

the follow-up notification email/letters.

We have decided to first invite principals to respond to the survey online and then, after three 

reminders, to allow nonrespondents a paper option in case principals preferred this format.

Kaplowitz, Hadlock, and Levine (2004) found that a web survey application achieved similar

response rates to a mailed hard copy questionnaire when both were preceded by an advance 

mail notification. In addition, Millar and Dillman (2011) found that a combined use of multiple 

response-inducing techniques (web and mail) produced the best response rates.

Study results will be available via an online link to the study on the IES and REL Northwest 

websites. Participating principals and their districts will be invited to participate in webinars to 

discuss the results of the study. In these webinars, participants will share ideas about 

implementing the SIG transformation model and similar school improvement efforts. They will 

also discuss the supports provided by technical assistance providers and how to leverage 

additional supports to address challenges to school improvement. This participation will be 

entirely voluntary and is presented as a way of motivating principals to participate.

A4. Efforts to Identify Duplication

This one-time study collects unique information that is not available elsewhere and is not being 

collected by other studies. To date, most discussions of rural SIG have combined all rural 

schools regardless of their other geographic and demographic characteristics and have not 

delved deeply into the implementation of the required activities in the transformation model 

(e.g., Dee, 2012; Rosenberg, 2011; Scott, 2012; Scott et al., 2012). While these past studies offered 

important information about SIG, the current study fills a need for information about rural 

schools’ implementation of the required activities in the transformation model. Ongoing 

national studies of SIG, such as Hurlburt and colleagues (2011, 2012), focus on all types of 

schools and on the four school turnaround models broadly, not on characteristics of rural 

schools and the required activities in the transformation model. These studies do not provide 

information about how states and other technical assistance providers might support specific 

school transformation activities in rural settings. 

The American Institutes for Research and Mathematica are currently conducting an in-depth 

descriptive study of 25 SIG schools, including 10 in rural areas. While this study will provide an 

in-depth look at some rural schools, it relies on less than 5 percent of rural SIG schools and will 

not generalize across rural SIG schools.

A5. Sensitivity to Burden on Small Entities

This project collects data from principals of schools, which by definition are small entities. We 

have minimized the burden for respondents by (1) requesting information that is generally 
Supporting Statement for OMB Clearance 7

available to school leadership, including principals, other administrators, and teacher leaders 

directly involved in SIG implementation, and (2) requesting the minimum amount of 

information required to answer the study questions. Answering these questions will provide 

policymakers and practitioners with information needed to make good decisions about the SIG 

transformation model and similar school improvement efforts. For example, items about the 

geographic and demographic characteristics of the schools are not included in the survey

because this information is available through the SIG baseline database. 

As shown in section “A12. Total Annual Cost Burden to Respondents,” we estimate a 20-minute

total time burden on each respondent. In addition, we will allow principals to designate a 

proxy, if it helps minimize the burden on the school.

A6. Consequence to Federal Program or Policy Activities if the Collection Is Not 

Conducted or Is Conducted Less Frequently

This one-time data collection has the potential to inform federal, state, and local agency 

programs and policies related to turning around low-performing schools both through future 

SIG allocations and other efforts such as NCLB waivers. Without this study, policymakers and 

their supporters would have little information about the unprecedented $3 billion that went into 

SIG in 2009 through the ARRA—about six times what Congress had allocated the previous year 

(Scott et al., 2012). Of these funds, more than $300 million went to rural transformation in the

first cohort of schools (Hurlburt et al., 2011). In addition, the types of efforts required by federal 

guidance to turn around low-performing schools may be similar in the future. This study 

informs this future work on other school improvement efforts that are similar to SIG.

A7. Special Circumstances

There are no special circumstances. The request fully complies with the regulations.

A8. Federal Register Announcement and Consultation

Federal Register Announcement

ED will publish Federal Register Notices to allow both a 60-day and 30-day public comment 

period. The REL will assist ED in addressing any public comments received. 

Consultations Outside the Agency

The research team will seek the expertise of persons outside the agency through REL 

Northwest’s Technical Working Group (TWG). The main responsibility of TWG members is to 

provide written feedback on study plans and/or reports and to meet via telephone as a group

with the research team to clarify that feedback and provide additional recommendations. The 

TWG members are also available to advise the research alliances as questions arise, speak with 

REL staff about future research project ideas, and generally lend their expertise as needed. In 

some cases, TWG members have also provided input on technical assistance projects. This work

is not expected to exceed six days in 2013, and each TWG member will receive a stipend of 
Supporting Statement for OMB Clearance 8

$3,000, to be dispersed as deliverables are met. All TWG members are experts who are 

distinguished in their fields and extremely knowledgeable about research design and methods. 

Their contracts have been approved by IES.

Three TWG members will provide insight and guidance in support of this study of the rural SIG 

transformation model. They suggested revisions to the study plan and the survey instrument 

prior to this request for OMB approval. They include:

 Tom Dee, Stanford University

 Hans Bos, American Institutes for Research

A9. Payment or Gift to Respondents

We will not provide any payment or gifts to participants. Participants will, however, receive a 

link to the published study and invitations to several webinars to discuss the study. While this 

is not a payment or gift, it will allow participants to have a voice in the discussion of study

results. During the webinar, participants will share information about implementing SIG and 

similar school improvement efforts. They will also discuss the supports provided by technical 

assistance providers and how to leverage additional supports for schools to address challenges 

to school improvement. We believe this opportunity will motivate people to complete the 

survey.

A10. Confidentiality of the Data

REL Northwest and PSA will be following the new policies and procedures required by the 

Education Sciences Reform Act of 2002, Title I, Part E, Section 183: “All collection, maintenance, 

use, and wide dissemination of data by the Institute” are required to “conform with the 

requirements of section 552 of title 5, United States Code, the confidentiality standards of 

subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act 

(20 U.S.C. 1232g, 1232h).” These citations refer to the Privacy Act, the Family Educational Rights 

and Privacy Act, and the Protection of Pupil Rights Amendment. Subsection (c) of section 183 

referenced above requires the Director of IES to “develop and enforce standards designed to 

protect the confidentiality of persons in the collection, reporting, and publication of data.” 

Subsection (d) of section 183 prohibits the disclosure of individually identifiable information,

and makes any publishing or communicating of individually identifiable information by 

employees or staff a felony.

REL Northwest and PSA will protect the confidentiality of all information collected for the 

study and will use it for research purposes only. No information that identifies any study 

participant will be released. Information from participating institutions and respondents will be 

presented at aggregate levels in reports. Unique identification numbers will be assigned to each 

participating school and used to identify all survey responses. The ID number/name association 

files will be kept secure in a confidential file separate from the data analysis file. No information 

identifying respondents will be included in the study data files or reports. Data from the online 
Supporting Statement for OMB Clearance 9

survey software system will be downloaded and deleted from the online system within one 

week after the data collection ends. These data files will then be stripped of any identifying 

information. Information will be reported in aggregate so that individual responses are not 

identifiable. All identification lists will be destroyed at the end of the project. 

The research team is trained to follow strict guidelines for soliciting consent, administering data 

collection instruments, and preserving respondent confidentiality. All members of the research 

team will have successfully completed the Collaborative Institutional Training Initiative (CITI) 

course in the Protection of Human Research Subjects through Liberty IRB. A copy of the 

affidavit of nondisclosure is provided in attachment D for each researcher who will have access 

to the data. 

All study materials will include the following language:

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, 

Part E, Section 183, responses to this data collection will be used only for statistical purposes. 

The reports prepared for this study will summarize findings across the sample and will not 

associate responses with a specific district or individual. We will not provide information that 

identifies you or your district to anyone outside the study team, except as required by law. Any 

willful disclosure of such information for nonstatistical purposes, without the informed consent of 

the respondent, is a class E felony.

A11. Additional Justification for Sensitive Questions

No questions of a highly sensitive nature are included in the survey or interview. 

Attachment E contains copies of the approved Institutional Review Board clearance forms.

A12. Estimates of Hour Burden

We estimate the total reporting burden associated with this data collection to be 57.92 hours and 

$2,454.51 (table 1). This includes 63 principals designating a proxy and 158 survey respondents 

completing the survey (i.e., 75 percent of 211). More details about these activities are included 

below in table 1. We estimated a principal’s hourly wage using May 2012 Bureau of Labor 

Statistics National Occupational Employment and Wage Estimates 

(http://www.bls.gov/oes/current/oes_nat.htm).

Designating a Proxy

We anticipate that approximately 63 principals (i.e., 30 percent of 211) may designate a proxy to 

fill out the survey because they are either new to the school or too busy to fill out the survey. 

We based this estimate on the national average of 20 percent principal attrition yearly (Battle, 

2010), plus an additional 10 percent of principals who may be too busy. We anticipate that the 

designation of proxies will add about 5 minutes to the burden on schools. Designating a proxy 

adds 5.25 hours to the burden and $222.50.
Supporting Statement for OMB Clearance 10

Survey

We will invite all 211 principals or designated proxies of our target population (i.e., the Cohort 1 

rural SIG transformation schools) to participate in the survey. The target response rate for these 

individuals is 75 percent, and the approximate time required for each respondent to complete 

the online survey is 20 minutes on average, based on a similar survey (Scott, Davis, & Krasnoff, 

2012). The total hours for the survey are 52.67, and the total cost is $2,232.01.

The most recent Schools and Staffing Survey of principals by the National Center of Education 

Statistics had a response rate of 79 percent for both public school and Bureau of Indian 

Education principals (Battle, 2009). This report did not disaggregate the response rates for rural 

school principals. But, we suspect that response rates would be somewhat lower for rural school 

principals. For example, the response rate of the rural district administrators’ survey was 74

percent (Zhang, 2008). Therefore, we have set our target at 75 percent.

 Table 1: Administration times

Reporting 

Method

Number of 

Respondents

Average 

Time 

(hours)

Total Burden

(hours)

Hourly Rate Estimated Cost

Designating a 

proxy 63 5 minutes 5.25 $42.38 $ 222.50 

Survey 158 20 minutes 52.67 $42.38 $2,232.01 

Total 57.92 $2,454.51 

A13. Estimate for the Total Annual Cost Burden to Respondents or Record 

Keepers

There are no direct start-up costs to respondents.

A14. Estimates of Annualized Costs to the Federal Government

The total estimated cost for this study is $373,250. Although the data collection occurs only 

once, these costs cover a two-year study period; therefore the annual cost of the study is 

$186,625. We estimated these costs using the amounts budgeted in the REL Northwest Year 2 

plan and approved by IES. These costs include planning the study, administering the survey, 

collecting and cleaning data, and analyzing and reporting results. In addition to the evaluation 

costs, there are personnel costs of several Federal employees involved in the oversight and 

analysis of information collection that amount to an annualized cost of $5,000 for Federal labor. 

A15. Reasons for Program Changes or Adjustments

This is a new study.

A16. Plans for Tabulation and Publication
Supporting Statement for OMB Clearance 11

We will tabulate and analyze the data by the four research questions. We will address the first 

three questions by providing the frequencies of survey responses for each transformational 

activity by level of implementation, challenges, and technical assistance provider. We will also 

explore the first three research questions using open-ended items on the survey. The fourth 

research question requires inferential statistics. Plans for tabulation and analysis are described 

in more detail below.

Research question 1: How do principals of rural SIG transformation schools rate the extent to 

which their school implemented the required activities in the transformation model?

We will address the first research question primarily by calculating frequencies of survey 

responses for each transformation activity by the extent of implementation, (i.e., N/A, 

planning/developing, early or partial implementation, or full implementation). The categories of 

levels of implementation are defined more fully in the survey (attachment B). We will also 

provide descriptive statistics about the percentages of principals that rate these activities as 

essential to their core efforts to improve schools.

We will use one open-ended item to add details about the implementation of activities related 

to school improvement. The item asks principals to list any additional school improvement 

activities their school implemented during the SIG funding period. To examine the item, we

will develop codes inductively (Mayring, 2000), but expect these codes to include additional 

school improvement activities and those strategies identified in our review of the literature 

about rural schools. To develop codes, the two study authors will review approximately one

third of the data independently and create codes. They will then compare these codes and agree 

on final codes. Next, they will both independently code the remaining data and compare 

results. If they cannot agree on the code for a particular piece of data, a third researcher will 

determine the final code. 

Research question 2: What do principals report were the challenges to implementation of each 

activity?

As in research question one, we will address this question primarily through frequencies of 

survey responses about challenges. Because some activities may pose multiple challenges, 

principals may select all challenges that apply (i.e., insufficient funding, insufficient staff 

expertise, insufficient staff time, insufficient technological capacity/equipment, lack of teacher 

support, lack of district support/guidance, other). Principals selecting “other” as a challenge will 

have the opportunity to specify the other challenges their school experienced. To conduct an 

exploratory analysis of this qualitative data, we will develop codes inductively (Mayring, 2000). 

To create codes, the two study authors will review approximately one third of the data 

independently. They will then compare these codes and agree on final codes. Next, they will 

both independently code the remaining data and compare results. If they cannot agree on the 

code for a particular piece of data, a third researcher will determine the final code. 
Supporting Statement for OMB Clearance 12

Research question 3: What types of technical assistance providers assisted schools, which 

particular transformation activities did they assist with, and how did the principals rate the 

sufficiency of this assistance?

We will provide two types of descriptive statistics to explore this question. First, we will 

provide frequencies for the principals reporting that their school received assistance on each of 

the transformation activities for particular types of providers (i.e., districts, states, universities, 

and other). Then, we will provide descriptive statistics representing the degree to which 

principals whose schools received this assistance agreed that the assistance was sufficient.

To provide more detail about technical assistance providers, we will examine the open-ended 

response that asks principals to provide the name(s) of their technical assistance organization(s)

We will list the top 10 organizations in the main body of the report and the complete list of 

organizations in the technical appendix.

Research question 4: Do principal ratings of implementation, challenges, and technical 

assistance providers vary by school geographic or demographic characteristics?

To examine variation in stages of implementation and in assistance by the geographic and 

demographic characteristics of schools, we will use the Kruskal Wallis H test. The Kruskal 

Wallis H test is a nonparametric test that allows the comparison of mean ranks of more than 

two categorical groups. We have chosen to use this test because, in our study, the dependent 

variables of interest (our survey responses about the stages of implementation and about 

sufficiency of assistance) are ordinal and the independent variables of interest (geographic and 

demographic characteristics of schools) are nominal. We will conduct four series of Kruskal 

Wallis H tests:

1) Implementation by geographic characteristics of schools, (i.e., by locale codes 

representing the type of rural area a school is in, such as a rural remote area versus a 

distant town)

2) Sufficiency of assistance by geographic characteristics of schools

3) Implementation by demographic characteristics of schools (i.e., by whether the school 

serves primarily white, Latino, or Native American students)

4) Sufficiency of assistance by demographic characteristics of schools

We discuss each of these series of analyses in more detail below.

To examine variations in reported implementation by geographic characteristics, we will 

conduct the H tests for each item’s response category by each of the six locale codes (31, 32, 33, 

41, 42, and 43). However, any cells with fewer than five responses will be eliminated from the 

analysis or will be collapsed if possible, (e.g., strongly agree and agree could be combined and 

strongly disagree and disagree could be combined). In each analysis, the ordinal Likert scale 

responses for the survey item will serve as the dependent variable. The independent or 

“grouping” variable will include the six locale codes. If the H test is statistically significant, we 
Supporting Statement for OMB Clearance 13

will perform a post hoc Dunn’s test to determine which specific groups were different from one 

another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test 

statistics for each analysis. To determine statistical significance we will use the BenjaminiHochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by geographic characteristics, we will also 

conduct H tests for the items about assistance. For these items, we will consider assistance from 

each type of provider (district, state, university, and other technical assistance providers) 

separately. We expect that in many instances this may result in cells with fewer than five 

responses. Those items with small cell sizes will be eliminated from the analysis. In each 

analysis, the ordinal Likert scale responses for the survey item will serve as the dependent 

variable. The independent or “grouping” variable will include the six locale codes. If the H test 

is statistically significant, we will perform a post hoc Dunn’s test to determine which specific 

groups were different from one another. We will display mean ranks, degrees of freedom, H 

test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance 

we will use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & 

Hochberg, 1995).

To examine variations in reported implementation by demographic characteristics, we will 

classify the schools in four ways: those that are more than 50 percent Latino (28 schools), those 

that are more than 50 percent Native American (50 schools), those that are more than 50 percent 

White (68 schools), and those that do not have a single racial majority (64 schools).2 We will use 

Kruskal Wallis H tests to examine variations in the data on implementation by student 

demographic characteristics. We will conduct the H tests for each item’s response category by 

each of the four demographic categories, although any cells with fewer than five responses will 

be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the 

survey item will serve as the dependent variable. The independent or “grouping” variable will 

include the four demographic categories. If the H test is statistically significant, we will perform 

a post hoc Dunn’s test to determine which groups differed from one another. We will display 

mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a 

technical appendix. To determine statistical significance we will use the Benjamini-Hochberg 

correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995).

To examine variations in reported assistance by demographic characteristics, we will also use 

Kruskal Wallis H tests. As in the previously described analysis of implementation, we will 

consider assistance from each type of provider (district, state, university, and other technical 

assistance providers) separately. We expect that in many instances we will need to eliminate 

analysis for cells with fewer than five responses. In each analysis, the ordinal Likert scale 

responses for the survey item will serve as the dependent variable. The independent or 

“grouping” variable will include the four demographic categories. If the H test is statistically 

significant, we will perform a post hoc Dunn’s test to determine which groups differed from 

 

2 One school had missing data for student ethnicity in the SIG baseline database.
Supporting Statement for OMB Clearance 14

one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test 

statistics for each analysis in a technical appendix. To determine statistical significance we will 

use the Benjamini-Hochberg correction to adjust for multiple comparisons (Benjamini & 

Hochberg, 1995).

Publications

For this study, we will create two publications. A What’s Happening? report will be targeted to 

our national practitioner and policymaker audience, particularly state agencies, districts, 

universities, and other technical assistance providers involved in supporting school 

improvement efforts similar to those in SIG transformation schools. This report will contain 

graphic summaries of the study and its findings. 

A Stated Briefly report will summarize findings that may appeal to regional and state 

organizations that are interested in improving schools but that do not provide direct technical 

assistance to schools—particularly those organizations that set school improvement policies and 

guidance. The Stated Briefly report should help these organizations set policy and create 

guidance that supports aspects of transformation that fewer participants reported were fully 

implemented, as well as potentially reducing or remediating such reported challenges. 

Both What’s Happening and Stated Briefly reports will be available on the REL Northwest website 

and the IES website. As stated earlier, geographic and demographic analyses will not be 

conducted when fewer than five respondents are included. This will reduce the risk of 

deductive disclosure.

Timeline

The timeline for the study, including the publications, is presented in table 2.

Table 2. Timeline

Timeline Milestone Date for deliverable

January 2013–April 2013 Develop data collection protocol and 

instruments

NA

April 2, 2013 Submit draft study plan to TWG, 

which includes study methodology, 

data collection activities, and 

analysis plan

NA

April 30, 2013 Submit draft study plan to IES, 

revised in response to TWG 

feedback

Draft plan to IES April 30, 

2013

July 24, 2013 Submit final study plan to IES Final plan to IES June 14, 

2013

October 2013 OMB submission of 60-day Federal 

Register Notice

NA

March–May 2014 Data collection NA

June 2014 Data analysis NA

July 2014 Write reports NA

September 2014 Submit draft What’s Happening? Draft What’s Happening?
Supporting Statement for OMB Clearance 15

Timeline Milestone Date for deliverable

report to IES report, 12 months after 

approval

November 2014 Submit final What’s Happening

?report to IES

Final What’s Happening?

report, 13–14 months after 

approval

December 2015 Submit draft Stated Briefly to IES Draft Stated Briefly,

15 months after approval

January 2015 Submit final Stated Briefly to IES Final Stated Briefly, 

16 months after approval

A17. Approval to Not Display the Expiration Date for OMB Approval

This is not applicable. The instrument will display the expiration date for OMB approval. 

A18. Exception to the Certification Statement

We do not seek any exceptions to the certification statement.
Supporting Statement for OMB Clearance 16

References

Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and 

secondary school principals in the United States: Results from the 2007–08 Schools and Staffing 

Survey (First Look, NCES 2009-323). Retrieved from U.S. Department of Education, 

National Center for Education Statistics website: 

http://nces.ed.gov/pubs2009/2009323.pdf

Battle, D. (2010). Principal attrition and mobility: Results from the 2008–09 principal follow-up survey

(First Look, NCES 2010-337). Retrieved from U.S. Department of Education, National 

Center for Education Statistics website: http://nces.ed.gov/pubs2010/2010337.pdf

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful 

approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 

57(1), 289–300.

Corbett, J. (2011). Lead turnaround partners: How the emerging marketplace of lead turnaround

partners is changing school improvement. Lincoln, IL: Academic Development Institute, 

Center on Innovation and Improvement.

Dee, T. (2012). School turnarounds: Evidence from the 2009 stimulus (NBER Working Paper No. 

17990). Cambridge MA: National Bureau of Economic Research.

Fixsen, D.L., Blase, K.A., Naoom, S.F., & Wallace F. (2009). Core implementation components.

Research on Social Work Practice, 19(5), 531–540.

Hammer, P.C., Hughes, G., McClure, C., Reeves, C., & Salgado, D. (2005). Rural teacher 

recruitment and retention practices: A review of the research literature, national survey of rural 

superintendents, and case studies of programs in Virginia. Nashville, TN: Edvantia, 

Appalachia Educational Laboratory.

Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications 

and SIG-eligible and SIG-awarded schools (NCEE 2011-4019). Retrieved from U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance website: 

http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdf

Hurlburt, S., Therriault, S.B., & Le Floch, K.C. (2012). School Improvement Grants: Analyses of state 

applications and eligible and awarded schools (NCEE 2012-4060). Retrieved from U.S. 

Department of Education, Institute of Education Sciences, National Center for Education 

Evaluation and Regional Assistance website: 

http://ies.ed.gov/ncee/pubs/20124060/pdf/20124060.pdf
Supporting Statement for OMB Clearance 17

Kaplowitz, M.D., Hadlock, T.D., & Levine, R. (2004). A comparison of web and mail survey 

response rates. Public Opinion Quarterly, 68(1), 94–101.

Klein, A. (2010, May 21). School turnaround models draw bipartisan concern. Education Week, 

29(33). Retrieved from 

http://www.edweek.org/ew/articles/2010/05/21/33turnaround.h29.html

Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1(2). 

Retrieved from http://www.qualitativeresearch.net/index.php/fqs/article/view/1089/2386

Millar, M.M., & Dillman, D.A. (2011). Improving response to web and mixed-mode surveys. 

Public Opinion Quarterly, 75(2), 249–269. 

Monk, D.H. (2007). Recruiting and retaining high-quality teachers in rural areas. Future of 

Children, 17(1), 155–174.

No Child Left Behind Act of 2001. (2002). Pub. L. No. 107-110, 115 Stat. 1425.

Owens, J.S., Richerson, L., Murphy, C.E., Jageleweski, A., & Rossi, L. (2007). The parent 

perspective: Informing the cultural sensitivity of parenting programs in rural 

communities. Child & Youth Care Forum, 36(5/6), 179–194.

Rosenberg, S.J. (2011). The challenges in implementing school improvement grant models in rural high 

schools (Master’s thesis). Duke University, Durham, NC.

Sandel, K., & Bhat, S. (2008). Financing and sustaining out-of-school time programs in rural 

communities. Retrieved from Finance Project website: 

http://www.financeproject.org/publications/ostruralyouth.pdf

Scott, C., Davis, A., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011–

2012. Portland, OR: Education Northwest.

Scott, C., McMurrer, J., McIntosh, S., & Dibner, K. (2012). Opportunities and obstacles: 

Implementing stimulus-funded School Improvement Grants in Maryland, Michigan, and Idaho. 

Washington, DC: Center on Education Policy.

Scott, G. (with Sirois, E., Spicer, S., Arsenault, J., King, M., Sorbello, S., & Steel-Lowney, B.). 

(2012). School Improvement Grants: Education should take additional steps to enhance 

accountability for schools and contractors (GAO-12-373). Washington DC: U.S. Government 

Accountability Office.
Supporting Statement for OMB Clearance 18

U.S. Department of Education. (2012, June 7). ESEA flexibility (Updated ed.). Retrieved from 

http://www.ed.gov/esea/flexibility/documents/esea-flexibility.doc

U.S. Department of Education, Office of Elementary and Secondary Education. (2011). Guidance 

on fiscal year 2010 School Improvement Grants under section 1003(g) of the Elementary and 

Secondary Education Act of 1965. Retrieved from

http://www2.ed.gov/programs/sif/sigguidance02232011.pdf

Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. 

Retrieved from Center on Education Policy website: http://www.cepdc.org/displayDocument.cfm?DocumentID=308