CONTRACT: ED

- IES

- 12

- C

- 0003



## Supporting Statement for OMB Clearance of a Survey of **Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model** 

**Section A**

**August 16, 201 3**  **Revised September 6, 2013 Revised October 8, 2013**  **Revised Oct 16, 2013**ii



### Contents Section A: Justification
................................ ................................ ................................ ............................... 1 A1. Circumstances Necessitating Collection of Information
................................ ............................... 1 A2. How, by Whom, and for What Purpose Information Is to Be Used
................................ ............ 3 A3. Use of Automated, Electronic, Mechanical or Other Technological Collection Techniques
................................ ................................ ................................ ................................ ..... 5 A4. Efforts to Identify Duplication
................................ ................................ ................................ .......... 6 A5. Sensitivity to Burden on Sma ll Entities
................................ ................................ ............................ 6 A6. Consequence to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently
................................ ................................ ............ 7 A7. Special Circumstances
................................ ................................ ................................ ........................ 7 A8. Fe deral Register Announcement and Consultation
................................ ................................ ....... 7 A9. Payment or Gift to Respondents
................................ ................................ ................................ ....... 8 A10. Confidentiality of the Data
................................ ................................ ................................ .............. 8 A11. Additional Justification for Sensitive Questions
................................ ................................ .......... 9 A12. Estimates of Hour Burden
................................ ................................ ................................ ............... 9 A13. Estimate for the Total Annual Cost Burden to Respondents or Record Ke ep ers .................. 10 A14. Estimates of Annualized Costs to the Federal Government
................................ ..................... 10 A15. Reasons for Program Changes or Adjustments
................................ ................................ .......... 10 A16. Plans for Tabulation and Publication
................................ ................................ ........................... 10 A17. Approval to Not Display the Expiration Date for OMB Approval
................................ .......... 15 A18. Exception to the Certification Statement
................................ ................................ ..................... 15 References
................................ ................................ ................................ ................................ ................. 16iii



### List of Tables Table 1. Administration times
................................ ................................ ................................ ................ 10 Table 2 . Timeline
................................ ................................ ................................ ................................ ..... 1 5 The following attachments are stored in the “ NW S07” folder in the “ REL Northwest Memos and Correspondence ” library, with the exceptions of Attachments D and E which we will add later since the Federal Register notices cannot be submitted until this document is approved by IES: Attachment A : ESRA Attachment B : Copy of Survey and Online Survey Screen Shots Atta chment C : Intro Letter Invitation and Follow

-Up Emails Attachment D : Affidavits of Nondisclosure Attachment E : IRB Approval FormSupporting Statement for OMB Clearance 1



## Supporting **Statement for OMB Clearance of a Surv** **ey of**  **Principals of Rural Sch ools**  **Receiv ing School Improv** **ement Grants**  **and Using the Transformation Model**

**Section A : Justification**  **A1 .**  **Circumstances Necessitating Collection of Information** This data collection is authorized by the Educational Sciences Reform Act (ESRA) of 2002 (see a ttachment A) . Part D, Section 174(f)(2 ) of ESRA states that as part of its central mission and primary function, each regional educational laboratory “shall support applied research by ... developing and widely d isseminating, including through Internet

-based me ans, scientifically valid research, information, reports, and publications that are usable for improving academic achievement, closing achievement gaps, and encouraging and sustaining school improvement, to schools, districts, institutions of higher edu cation, educators (including early childhood educators and librarians), parents, policymakers, and other constituencies, as appropriate, within the region in which the regional educational laboratory is located.” REL Northwest and our partner Policy Studi es Associates (PSA) , the author s of this study, are interested in how rural schools implement School Improvement Grants (SIGs) . These competitive federal grants have provide d funds to two cohorts of low

- performing schools to increase student achievement. The grants require schools to implement one of four school improvement models : the tran sformation model, which involves a variety of reforms; the turnaround model, which involves primarily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school. This descriptive study collects and reports on data from ru ral school principals across the nation that received federal SIGs in Cohort 1 and used the transformation model
. This information colle ction is essential to a study by the Regional Educational Laboratory (REL) Northwest, which serves five Northwest states (Alaska, Idaho, Montana, Oregon, and Washington). M ost rural Northwest schools with SIG funding (96 pe rcent) use d the transformation mo del to improve. REL Northwest and its partner PSA are collecting survey data about the implementation of the transformation model to inform regional and national supports for future school improvement efforts. Data collection will occur once in spring 2014
. IES has approved the study plan
. **Statement of Need and Background Information** To date, there is little systematic evidence about the implementation of the SIG program. In the case of rural SIG settings, this lack of research evidence is particularly problematic . Rural schools often face steep challenges when trying to implement the kinds of staff replacement and on- site professional development practices promoted by the SIG program (Klein, 2010). This study addresses these challenges by examining the level of implementation of the SIG transformation model in rural regions, identifying the organizations

— states, districts,Supporting Statement for OMB Clearance 2

universities, and other entities

— that assist schools in these implementation efforts, and exploring perceived gaps in assistance. S IG funding mechanisms are not new. Congress put them in place as part of the No Child Left Behind Act (NCLB) of 2001 (2002), the reauthorization of the Elementary and Secondary Education Act (ESEA). However, in 2009 the American Re covery and Reinvestment A ct (ARRA) substantially increased funding for these grants
. With this increase, the U.S. Department of Education issued new guidance, dramatically changing SIG requirements. Under this guidance, federal funds pass to states as usual, but states are now charged with assisting schools in implementing one of four school improvement models (U.S. Department of Education, Office of Elementary and Secondary Education [OESE], 2011): (1) transformation, replacing the principal and imp lementing a number of strategies to improve the school; (2) turnaround, replacing the principal and 50 percent of staff members and providing operational flexibility needed to implement a variety of school improvement strategies ; (3) r estart, reopening as a school operated by a charter or education management organization; and (4) c losure, closing the school and ensuring that students transfer to a high

- performing school. B e cause the transformation model require s less disruption and staff me mber replacement than other models, some consider it the most flexible of the four (Klein, 2010). Perhaps as a result of this flexibility, most grantee schools

— and 96 percent of ru ral schools nationally

— have chose n this model ( Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch, 2012 ). Therefore, our study focuses on the transformation model in rural settings
. The transformation model requires the following 11 activities ( U.S. Department of Education, OESE, 2011): 1. Replace the principal who led the school prior to the transformation model 2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors 3. Identify and reward s chool leaders, teachers, and other staff me mbers who i mproved student outcomes, and identify and r emove those who did not 4. Provide staff with ongoing, high

- quality, job

- em bedded professional development 5. Implement strategies designed to recruit, place, and r etain staff 6. Use d ata to identify and implement a new instructional program 7. Promote continuous use of student data in order to inform and differentiate instruction 8. Establish schedules and strategies that provide increased learning time 9. Provide ongoing mechanisms for family and community engagement 10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes 11. Ensure that the school receives ongoing, intensive technical assistance and related supp ort from the LEA, the SEA, or an external organization **Study Design, D ata Collection, and Timing**  This descriptive study collects and analyze s principals’ reports of SIG transformation model implementation in their rural schools. It uses a one - time survey of the 211 Cohort 1 principals ofSupporting Statement for OMB Clearance 3

rural schools implementing the SIG transformation model (attachment B). The survey will occur in sprin g 2014
. It also incorporates existing data from the SIG baseline database available on the U.S. Department of Education website, including each school ’s National Center for Education Statistics ( NCES ) locale code ( i.e., the geographic characteristics of the school) , p ercentage of students eligible for free or reduced

- price lunch ( i.e., poverty levels among students) , and p ercentage s of non- White and Native American students
. The study uses the data to address the following research questions: 1) How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation mod el? 2) What do principals report were the challenges to implementation of each activity? 3) What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how d id the principals rate the sufficiency of this assistance? 4) Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics? 

### A**2. How, by** 

**Whom, an d for What Purpose Information I**  **s T**  **o Be Used** To make good decisions about ongoing school transformation efforts and future similar school improvement initiatives, the nation’s education leaders need more information about rural SIG implementation of the transformation model. More specifically, the data from this study will be used by REL Northwest and our state contacts to provide information to federal policy makers during ongo ing efforts to reauthorize ESEA. These policymakers need such information as they consider changing or retaining language around the requir ements for the continued use of SIGs under ESEA, which are likely to continue to provide funds to schools for school improvement, as well as language incorporating ele me nts of the ESEA waivers involving school improvement efforts (U.S. Department of Education, 2012)
. We will also use the s e data to provide information to s tate policymakers in states with similar legislation around school improvement. Furthermore, we will use the s e data to inform education practitioners, particularly those who provide support for school improvement . These practitioners need the data to identify gaps in technical assistance for rural schools seeking to improve and uncover opportunities to provide additional and/or different services to schools. Why is it crucial for these policymakers and practitioners to consider changes in policies and practices for transforming rural schools? De spite the fact that the transformation model requires less disruption and staffing changes, m any education leaders anticipate that rur al schools, in particular, will have difficulty meeting the requirements of the transformation model. For example, recruiting and retaining high

- quality staff members is known to be more difficult in rural areas ( Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Monk, 2 007 ). In addition, finding technical assistance providers who can work with rural schools can be challenging (Corbett, 2011); transforming school schedules to allow for extended learning time can be difficult due to the longer bus rides and higher transpor tation costs (Sandel & Bhat, 2008); and rural parents can be mistrustful of outside technical assistance providers

— such as stateSupporting Statement for OMB Clearance 4

employees or university professors

— who seem de termined to change their schools (Owens, Richerson, Murphy, Jageleweski, & Rossi, 2007). However, there is little empirical information about the implementation of the transformation model in rural schools. This study will provide education policymakers and practitioners with crucial information, including : specifying the extent to which the transformation activities were implemented and the challenges to implementation and identifying which activities were supported by technical assistance providers and how sufficient principals found this support. The study will do this primarily through a specially designed survey of the 211 rural school principals implementing SIG transformation models in Cohort 1 (fund ed from 2010 through 2013, with an option to carry over funds to 2014) . We have designed this survey ( attachment B ) based on our review of the literature on SIG and rural schools and on our staff members’ past work on SIG evaluation. The first section of the survey asks principals how long they have been a principal and when they became an administrator at their school. If the principal was not an administrator at the school during the SIG funding period, a follow

- up item will ask whether the principal knows how the school implemented the SIG. If the principal is new to the school and does not know about the SIG i mplementation, the survey will prompt the principal to provide contact information for the prior principal or for a designated proxy, wh o is identified as a staff member who was employed during the SIG funding period and served in one of the following role s:  A school

- level administrator, such as an assistant principal, who participated in the school’s SIG
 A teacher who was responsibl e for assisting with the implementation of the school’s SIG
 A district administrator responsible for school improvement The s econd section of the survey asks principals to rate their school’s implementation of the 11 required transformation model activities using a four

- point scale that reflects the stages of implementation developed by Fixsen, Blase, Naoom, and Wallace (2009). Transformation activities include : 1 1. Replace the principal who led the school prior to commenceme nt of the transformation model 2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors 3. Identify and reward school leaders, teachers, and other staff me mbers who i mproved student outcomes, and identify and r emove those who did not 4. Provide staff with ongoing, high

- quality, job

- em bedded professional development

1 In order to ensure that each survey item represents only one SIG transformation activity, we split several activities into multiple survey items. We also eliminated the activity, “Ensure that the school receives ongoing, intensive technical assistance and related sup port from the LEA, the SEA, or an external organization” because we ask principals about this assistance in a separate section of the survey.Supporting Statement for OMB Clearance 5

5. Implement strategies designed to recruit, place, and retain staff 6. Use d ata to identify and implement a new instructional program 7. Promote the continuous use of student data in order to inform and differentiate instruction 8. Establish schedules and strategies that provide increased learning time 9. Provide ongoing mechanisms for family and community engagement 10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes 11. Ensure that the school receives ongoing, intensive technical assistance and related supp ort from the LEA, the SEA, or an external organization We also want to gather information ab out challenges to implementation
. In the second section of the survey, the principals will be asked to select one or more challenges for each activity . We will allow principals to select multiple responses for this survey item, since some items may be related to one another. For example, “insufficient funding” may mean that staff members are overworked and, therefore, also have “insufficient staff time” to implement the model fully. Principals may select both responses if appropriate. They will also have the option of selecting “NA (no challenges).” This section of the survey also includes two open

- ended items. The firs t item gathers information about which activities were important to the school ’ s improvement efforts. The second will collect information about other school improvement activities that rural schools may engage in. The third and last section of the survey asks principals to identify the entities that assisted them as they implemented the required transformation model activities. We will allow principals to check all that apply. The choice of entities includes their district, their state education agency, a partner in higher education, or another technical assistance provider. This list is based on the entities that other studies have found assisted with SIG most often (Corbett, 2011; Scott , McMurrer, McIntosh, & Dibner, 2012). If a principal selects “another technical assistance provider,” the survey will prompt the principal to fill in the name of the organization that provided the technical assistance. Then, for each entity that the principal selects, the principal will indicate the degree to which he or she agrees that assistance from this entity was sufficient. Participants will rate the sufficiency on a four

- point agree/disagree scale. 

### A**3. Use of A**  **utomated, Electronic, Mechanical or Other Technological Collection Techniques**  Initially, we will invite principals to complete an online survey (with electronic submission of responses through Survey Gizmo ) . Those that do not respond to this initial invitation and three reminders will have the option of completing a paper version of the survey (with electronic, faxed, or handwritten submission of responses) that they may em ail, fax, or mail back to our partner PSA. Either version will take about the same amount of time to complete (20 minutes).Supporting Statement for OMB Clearance 6

PSA will host a website where respondents can access the link to the online survey. Attachment C contains the advance mail notification letter, the invitation to participate in the survey, and the follow

- up notification email/letter s . We have decided to firs t invite principals to respond to the survey online and then, after three reminders, to allow nonrespondents a paper option in case principals preferred this format
. Kaplowitz, Hadlock, and Levine (2004) found that a web survey application achieved similar response rates to a mailed hard copy questionnaire when both were preceded by an advance mail notification. In addition, Millar and Dillman (2011) found that a combined use of multiple response - inducing techniques (web and mail) produced the best response rates
. Study r esults will be available via an online link to the study on the IES and REL Northwest websites . Participating principals and their districts will be invited to participate in webinars to discuss the results of the study. In these webinars, participants will share ideas about implementing the SIG transformation model and similar school improvement efforts . They will also discuss the supports provided by technical assistance providers and how to leverage additional supports to address challenges to school improvement
. This participation will be entirely voluntary and is presented as a way of motivating principals to participate. 

### A**4. Efforts to Identify Duplication**  This one - time study collects unique information that is not available elsewhere and is not being collected by other studies. To date, most discussions of rural SIG have combined all rural schools regardless of their other ge ographic and demographic characteristics and h ave not delved deeply into the implementation of the required activities in the transformation model (e.g., Dee, 2012; Rosenberg, 2011; Scott, 2012; Scott et al. , 2012 ). While these past studies offered important information about SIG, the current study fi lls a need for information about rural schools’ implementation of the required activities in the transformation model. Ongoing n ational studies of SIG, such as Hurlburt and colleagues (2011 , 2012), focus on all types of schools and on the four school turna round models broadly, not on characteristics of rural schools and the required activities in the transformation model . The se studies do not provide information about how states and other technical assistance providers might support specific school transformation activities in rural settings
. The A merican Institutes for Research and Mathematica are currently conducting an in
- de pth descriptive study of 25 SIG school s, including 10 in rural areas . While this study will provide an in
- de pth look at some ru ral schools, it relies on less than 5 percent of rural SIG schools and will not gene ralize across rural SIG schools. 

### A5 **. Sensitivity** 

**to Burden on Small Entities** This project collects data from principals of schools, which by definition are small entities. We have minimized the burden for respondents by (1) requesting information that is generallySupporting Statement for OMB Clearance 7

available to school leadership, including principals, other administrators, and teacher leaders directly involved in SIG implementation , and (2) re questing the minimum amount of information required to answer the study questions. Answering these questions will provide policymakers and practitioners wi th information needed to make good decisions about the SIG transformation model and similar school im provement efforts. For example, i tems about the geographic and demographic characteristics of the schools are not included in the survey because this information is available through the SIG baseline database
. As shown in section “A12. Total Annual Cost Burden to Respondents,” we estimate a 20

- minute total time burden on each respondent
. In addition, we will allow p rincipals to designate a proxy, if it help s minimize the burden on the school
. 

### A6 **. Consequence to**  **Federal Program or Polic y**  **Ac tiv**  **ities if the Collection Is Not Conducted or I**  **s Conducted Less Frequently** This one - time data collection has the potential to inform federal, state, and local agency programs and policies related to turning around low

- perform ing schools both thr ough future SIG allocations and other efforts such as NCLB waivers
. Without this study, policymakers and their supporters would have little informa tion about the unprecedented $3 billion that went into SIG in 2009 through the ARRA

— about six times what Congress had allocated the previous year ( Scott et al., 2012 ). Of these funds, more than $300 million went to rural transformation in the first cohort of schools (Hurlburt et al., 2011). In addition, the types of efforts required by federal guidance to turn around low

- performing schools may be similar in the future. This study informs this future work on other school improvement e fforts that are similar to SIG. 

### A7 **. Special Circumstances**  There are no special circumstances . The request fully complies with the regulations. 

### A**8. Federal Register A**  **nnouncement and Consultation** Federal Register Announcement ED will publish Federal Register Notices to allow both a 60

- day and 30

- day public comment period. The REL will assist ED in addressing any public comments received. **Consultations Outside the A gency**  The research team will seek the expertise of persons outside the agency through REL Northwest’s Te chnical Working Group (TWG). The main responsibility of TWG members is to provide written feedback on study plans and/or reports and to meet via telephone as a group with the research team to clarify that feedback and provide additional recommendations . The TWG members are also available to advise the research alliances as questions arise , speak with REL staff about future research project ideas, and generally lend their expertise as needed. In some cases, TWG members have also provided input on technical assistance projects. This work is not expected to exceed six days in 2013, and each TWG member will receive a stipend ofSupporting Statement for OMB Clearance 8

$3,000, to be dispersed as deliverables are met. All TWG members are experts who are distinguished in their fields and extremely knowledgeable about research design and methods
. Their contracts have been approved by IES. Three TWG members will provide insight and guidance in support of this study of the rural SIG transformation model
. They suggested revisions to the study plan and the survey instrument prior to this request for OMB approval. The y include :  Tom Dee, Stanford University
 Hans Bos, American Institute s for Research 

### A**9. Pay**  **ment or Gift to R espondents**  We will not provide any payment or gifts to participants. Participants will, however, receive a link to the published study and invitations to several webinars to discuss the study. While this is not a payment or gift, it will allow participants to have a voice in the discussion of study results . D uring the webinar, participants will share information about implementing SIG and similar school improvement efforts. They will also discuss the supports provided by technical assistance providers and how to leverage additional supports for schools to address challenges to school improvement . We believe this opportunity will motivate people to complete the survey . 

### A **10. Confidentiality** 

**of the Data** REL Northwest and PSA will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 : “ All collection, maintenance, use, and wide dissemination of data by the Institute ” are required to “ conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards o f subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h). ” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amendment. Subsection (c) of section 183 referenced above requires the Director of IES to “ develop and enforce standards designed t o protect the confidentiality of persons in the collection, reporting, and publication of data
. ” Subsection (d) of section 183 prohibits the disclosure of individually identifiable information , and makes any publishing or communicating of individually identifiable information by employees or staff a felony. REL Northwest and PSA will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Unique identification numbers will be assigned to each participating school and used to identify all survey responses. The ID number/na me association files will be kept secure in a confidential file separate from the data analysis file. No information identifying respondents will be included in the study data files or reports. Data from the onlineSupporting Statement for OMB Clearance 9

survey software system will be downloaded and deleted from the online system within one week after the data collection ends . These data files will then be stripped of any identifying information. Information will be reported in aggregate so that individual responses are not identifiable. All identification lists will be destroyed at the end of the project. The research team is trained to follow strict guidelines for soliciting consent, administering data collection instruments, and preserving respondent confidentiality. All members of the resear ch team will have successfully completed the Collaborative Institutional Training Initiative (CITI) course in the Protection of Human Research Subjects through Liberty IRB. A copy of the affidavit of nondisclo sure is provided in attachment D for each resea rcher who will have access to the data. All study materials will include the following language: Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will b e used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony. 

### A **11. Ad**  **ditional Justification for Sensitive Questions**  No questions of a highly sensitive nature are included in the survey or interview. Attachment E contains copies of the approved Institutional Review B oard clearance forms. 

### A **12. Estimates of Hour B**  **urden** We estimate the total reporting burden associated wi th this data collection to be 57. 9 2 hours and $ 2,454.51 (table 1 ). This includes 63 principals designating a proxy and 158 survey respondents completing the survey (i.e. , 75 percent of 211). More details about these activities are included below in table 1. We estimated a principal ’ s hourly wage using May 2012 Bureau of Labor Statistics National Occupational Employment and Wage Estimates ( http://www.bls.gov/oes/current/oes_nat.htm ). **Designating a Proxy** We anticipate that a pproximately 63 principals (i.e., 3 0 percent of 211) may designate a proxy to fill out the survey because they are either new to the school or too busy to fill out the survey. We based this estimate on the national average of 20 percent principal attrition yearly (Battle, 2010) , plus an additional 10 percent of principals who may be too busy
. We anticipate that the designation of proxies will add about 5 minutes to the burden on sch ools. Designating a proxy adds 5. 25 hours to the burden and $222 .50.Supporting Statement for OMB Clearance 10

**Survey** We will invite all 211 principals or designated proxies of our target population (i.e., the Cohort 1 rural SIG transformation schools ) to participate in the survey. The target response rate for these individuals is 75 percent, and the approximate time required for each respondent to complete the online s ur vey is 20 minutes on average , based on a similar survey (Scott, Davis, & Krasnoff , 2012) . The total hours for the survey are 52. 6 7 , and the total cost is $ 2,232.0 1
. The most recent Schools and Staffing Survey of principals by the National Center of Education Statistics had a response rate of 79 percent for both public school and Bureau of Indian Education principals ( Battle, 2009). This report did not disaggregate the response rates for rural school principals. But, w e suspect that response rates would be somewhat lower for rural school principals. For example, the response rate of the rural district administrators ’ survey was 74 percent ( Zhang, 2008 ). Therefore, we have set our target at 75 percent
. **Table 1 : A** **dministration t**  **imes Reporting**  **Me thod**  **Number of**

**Respondents Av**  **erage**

**Time (hours)**  **Total Burden**

**(hours) Hourly Rate**  **Estimated Cost**  Designating a proxy 63 5 minutes 5. 25 $42.38 $ 222.50 Survey 158 20 minutes 52. 67 $42.38 $2,232.01 **Total 57.9**  **2 $2,454.51** 


### A **13. Estimate for the Total A**  **nnual Cost Burden to Respondents or Record Keepers**  There are no direct start

-up costs to respondents
. 

### A **14. Estimates of A**  **nnualized Costs to the Federal Government** The total estimated cost for this study is $373,250. Although the data collection occurs only once, these cost s cover a two
- ye ar study period; therefore the annual cost of the study is $186,625. We estimated these costs using the amounts budgeted in the REL Northwest Year 2 plan and approved by IES. These costs include planning the study, administering the survey, collecting and cleaning data, and analyzing and reporting results
. In addition to the evaluation costs, there are personnel costs of several Federal employees involved in the oversight and analysis of information collection that amount to an annualized cost of $5,000 for Federal labor. 

### A **15. Reasons for Program Changes or A** **djustments** This is a new study. 

### A **16 .**  **Plans for Tabulation and Publication**Supporting Statement for OMB Clearance 11

We will tabulate and analyze the data by the four research question s . We will address the first three questions by providing the frequencies of survey responses for ea ch transformational activity by level of implementation, challenges, and technical assistance provider. We will also explore the first three research questions using open
- ended items on the survey. The fourth research question requires inferential statistics. Plans for tabulation and analysis are described in more detail below. **Research question 1** : How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation model? We will address the first research question primarily by calculating frequencies of survey responses for each transformation activity by the extent of implementation, ( i.e. , N/A, p lanning/developin g, e arly or partial implementation, or full implementation ) . The categories of level s of implementation are defined more fully in the survey (attachment B). We will also provide descriptive statistics about the percentages of principals that rate these activities as essential to their core efforts to improve schools. We will use one open- ended item to add de tails about the implementation of activities related to school improvement. The item asks principals to list any additional school improvement activities their school implemented during the SIG funding period. To examine the item, we will develop codes inductively (Mayring, 2000), but expect these codes to include additional school improve ment activities and those strategies identified in our review of the literature about rural schools
. To develop codes, the two study authors will review approximately one third of the data independently and create codes. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a particular piece of data, a third researcher will determine the final code. **Research question 2** : What do prin cipals report were the challenges to implementation of each activity? As in research question one, we will address this question primarily through frequencies of su rvey responses about challenges. Because some activities may pose multiple challenges, prin cipals may select all challenges that apply (i.e., insufficient funding, insufficient staff expertise, insufficient staff time, insufficient technological capacity/equipment, lack of teacher support, lack of district support/guidance, other )
. Principals se lecting “other” as a challenge will have the opportunity to specify the other challenges their school experienced. To conduct an exploratory analysis of this qualitative data, we will develop codes inductively (Mayring, 2000). To create codes, the two stud y authors will review approximately one third of the data independently. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a par ticular piece of data, a third researcher will determine the final code.Supporting Statement for OMB Clearance 12

**Research question 3** : What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how d id the principals rate the sufficiency of this assistance? We will provide two types of descriptive statistics to explore this question. First, we will provide frequencies for the principals reporting that their school received assistance on each of the transformation activitie s for particular types of providers (i.e. , districts, states, universities, and other). Then, we will provide descriptive statistics representing the degree to which principals whose schools received this assistance agreed that the assistance was sufficient. To provide more detail about technical assistance providers, we will examine the open

- ended response that asks principals to provide the name(s) of their technical assistance organization(s) We will list the top 10 organizations in the main body of the report and the complete list of organizations in the technical appendix. **Research question 4** : Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics? To examine variation in stages of implementation and in assistance by the geographic and demographic characteristics of schools, we will use the Kruskal Wallis H test
. The Kruskal Wallis H test is a nonparametric test that allows the comparison of mean ranks of more than two categorical groups. We have chosen to use this test because, in our study, the dependent variables of interest (our survey responses about the stages of implementation and about sufficiency of assistance) are ordinal and the independent variables of interest (geographic and demographic characteristics of schools) are nominal. We will conduct four series of Kruskal Wallis H tests: 1) Implementation by geographic characteristics of schools, (i.e. , by locale codes representing the type of rural area a school is in, such as a rural remote area versus a distant town) 2) Sufficiency of assistance by geographic characteristics of schools 3) Implementation by demographic characteristics of schools (i.e. , by whether the school serves primarily white, Latin o, or Native American students) 4) Sufficiency of assistance by demographic characteristics of schools We discuss each of these series of analyses in more detail below. **To examine variations in reported implementation**  **by geographic characteristics,** we will conduct the H tests for each item’s response category by each of the six locale codes (31, 32, 33, 41, 42, and 43). However, any cells with fewer than five responses will be eliminated from the analysis or will be collapsed if possible, (e.g., strongly agr ee and agree could be combined and strongly disagree and disagree could be combined) . In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, weSupporting Statement for OMB Clearance 13

will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini

Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variations in reported assistance**  **by geographic characteristics,** w e will also conduct H tests for the items about assistance. For these items, we will consider assistance from each type of provider (district, state, university, and other technical assistance providers) separately. We expect that in many instances this may resu lt in cells with fewer than five responses. Those items with small cell sizes will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test stati stics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variations in reported implementation**  **by d emographic characteristics,**  we will classify the schools in four ways: those that are more than 50 percent Latino (28 schools), those that are more than 50 percent Native American (50 schools), those that are more than 50 percent White (68 schools), and th ose that do not have a single racial majority (64 schools). 2 We will use Kruskal Wallis H tests to examine variations in the data on implementation by student demographic characteristics. We will conduct the H tests for each item’s response category by each of the four demographic categories, although any cells with fewer than five re sponses will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the four demographic categories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendi x. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variations in reported assistance**  **by demographic characteristics,** we will also use Kruskal W allis H tests. As in the previously describe d analysis of implementation, we will consider assistance from each type of provider (district, state, university, and other technical assistance providers) separately. We expect that in many instances we will ne ed to eliminate analysis for cells with fewer than five responses
. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the four demographic categ ories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from

2 One school had missing data for student ethnicity in the SIG baseline database.Supporting Statement for OMB Clearance 14

one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendix. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **Publications** F or this study, we will create two publications. A What’s Happening ? r eport will be targeted to our national practitioner and policymaker audience, particularly state agencies, districts, universities, and other technical assistance providers involved in supporting school improvement efforts similar to those in SIG transfor mation schools. This report will contain graphic summaries of the study and its findings. A Stated Briefly report will summarize findings that may appeal to regional and state organizations that are interested in improving schools but that do not provide direct technical assistance to schools

— particularly those organizations that set school improvement policies and guidance. The Stated Briefly report should help these organizations set policy and create guidance that supports aspects of transformation tha t fewer participants reported were fully implemented, as well as potentially reducing or remediating such reported challenges. Both What’s Happening and Stated Briefly reports will be available on the REL Northwest website and the IES website. As stated earlier, geographic and demographic analyses will not be conducted when fewer than five respondents are included. This will reduce the risk of deductive disclosure. **Tim eline**  The timeline for the study, including the publications, is presented in table 2
. **Table 2**  **. Timeline**

**Timeline Milestone**  **Date for deliverable** January 2013

–April 2013 Develop data collection protocol and instruments NA April 2, 2013 Subm it draft s tudy p lan to TW G , which includes study methodology, data collection activities , and analysis plan NA
April 30, 2013 Subm it draft s tudy pl an to IES , revised in response to TW G feedback Draft plan to IES April 30, 2013 July 24, 2013 Subm it final study plan to IES Final plan to IES June 14, 2013 October 201 3 OMB submission of 60
- day Federal Register Notice NA March

– May 2014 Data collection NA June 2014 Data analysis NA July 2014 Wr ite report s NA September 2014 Subm it draft What’ s Happening ? Draft What’ s Happening ?Supporting Statement for OMB Clearance 15 **Timeline Milestone**  **Date for deliverable** report to IES report, 12 months after approval November 2014 Subm it final What’ s Happening ? report to IES Final What’ s Happening ? report, 13

– 14 months after approval December 2015 Subm it draft Stated Briefly to IES Draft Stated Briefly, 15 mo nths after approval January 2015 Subm it final Stated Briefly to IES Final Stated Briefly, 16 mo nths after approval 

### A **17. Ap**  **proval to Not Display the Expiration D**  **ate for OMB A**  **pprov al**  This is not applicable. The instrument will display the expiration date for OMB approval. 

### A **18. Exception to the Certification Statement**  We do not seek any exceptions to the certification statement.Supporting Statement for OMB Clearance 16



### References Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007

–08 Schools and Staffing Survey ( First Look, NCES 2009

- 323). Re trieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf Battle, D. (2010). Principal attrition and mobility: Results from the 2008

–09 principal follow

- up survey (First Look, NCES 20 10

- 337). Retrieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2010/2010337.pdf Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57 (1), 289

–

300. Corbett, J. (2011). Lead turnaround partners: How the emerging marketplace of lead turnaround partners is changing school improvement. Lincoln, IL: Academic Development Institute, Center on Innovation and Improvement. Dee, T. (2012). School turnarounds: Evidence from the 2009 stimulus (NBER Working Paper No. 17990)
. Cambridge MA: National Bureau of Economic Research. Fixsen, D.L., Blase, K.A., Naoom, S.F., & Wallace F. (2009). Core implementation components. Research on Social Work Practice , 19 (5), 531

–540. Hammer, P.C., Hughes, G., McClure, C., Reeves, C., & Salgado, D. (2005). Rural teacher re cruitment and retention practices: A review of the research literature, national survey of rural superintendents, and case studies of programs in Virginia. Nashville, TN: Edvantia, Appalachia Educational Laboratory. Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG

- e ligible and SIG

- a warded s chools (NCEE 2011

- 4019). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019 .pdf Hurlburt, S., Therriault, S.B., & Le Floch, K.C. (2012). School Improvement Grants: Analyses of state applications and eligible and awarded schools (NCEE 2012

- 4060). Retrieved from U.S. Department of Education, Institute of Education Sciences, Nation al Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20124060/pdf/20124060.pdfSupporting Statement for OMB Clearance 17

Kaplowitz, M.D., Hadlock, T. D., & Levine, R. (2004). A comparis on of w eb and mail survey response rates. Public Opinion Quarterly, 68 (1) , 94

– 101
. Klein, A. (2010, May 21). School turnaround models draw bipartisan concern. Education Week, 29 (33)
. Re trieved from http://www.edweek.org/ew/articles/2010/05/21/33turnaround.h29.html Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1 (2). Retrieved from http://www.qualitative

research.net/index.php/fqs/article/view/1089/2386 Millar, M. M. , & Dillman, D. A. (2011). Improving response to web and mixed

- mode surveys. Public Opinion Quarterly, 75 (2), 249

–269. Monk, D.H. (2007). Recruiting and retaining high

- quality teachers in rural areas. Future of Children, 17 (1), 155

– 174. No Child Left Behind Act of 2001. (2002). Pub. L. No. 107

- 110, 115 Stat. 1425. Owens, J.S., Richerson, L., Murphy, C.E., Jageleweski, A. , & Rossi, L. (2007). The parent perspective: Informing the cultural sensitivity of parenting programs in rural communities. Child & Youth Care Forum, 36 (5/6), 179

– 194. Rosenberg, S.J. (2011). The challenges in implementing school improvement grant models in rural high schools (Master’s thesis)
. Duke University, Durham, NC
. Sandel, K., & Bhat, S. (2008). Financing and sustaining out

- of

-school time programs in rural communities . Retrieved from Finance Project website: http://www.financeproject.org/publications/ostruralyouth.pdf Scott, C
. , Davis, A ., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011

– 2012. Portland, OR: E ducation Northwest. Scott, C., McMurrer, J., McIntosh, S
. , & Dibner, K. (2012). Opportunities and obstacles: Implementing stimulus

-

funded School Improvement Grants in Maryland, Michigan, and Idaho
. Washington, DC: Center on Education Policy. Scott, G. (with Sirois, E., Spicer, S., Arsenault, J., King, M., Sorbello, S., & Steel

- Lowney, B.). (2012). School Improvement Grants: Education should take additional steps to enhance accountability for schools and contractors (GAO

- 12

- 373). Washington DC: U.S. Go vernment Accountability Office.Supporting Statement for OMB Clearance 18

U.S. Department of Education
. (2012, June 7). ESEA flexibility (Updated ed.)
. Retrieved from http://www.ed.gov/esea/flexibility/documents/esea

- flexibility.doc U.S. Department of Education, Office of Elementary and Secondary Education. (2011 ). Guidance on fiscal year 2010 School Improvement Grants under section 1003(g) of the Elementary and Secondary Education Act of 1965 . Retrieved from http://www2.ed.gov/programs/sif/sigguidance02232011.pdf Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from Center on Education Policy website: http://www.cep

dc.org/displayDocument.cfm?DocumentID=308