CONTRACT: ED

- IES

- 12

- C

- 0003



## Supporting Statement for OMB Clearance of a Survey of **Principals of Rural Schools Receiving School Improvement Grants and Using the Transformation Model** 

**Section A August 16, 201**  **3 Revised September 6, 2013**  **Revised October 8, 2013 Revised Oct 16, 2013**  **Revised December 1 3**  **, 2013 Revised December 30, 2013** ii 

### Contents Section A: Justification

................................ ................................ ................................ ............................... 1 A1. Circumstances Necessitating Collection of Information

................................ ............................... 1 A2. How, by Whom, and for What Purpose Information Is to Be Used

................................ ............ 3 A3. Use of Automated, Electronic, Mechanical or Other Technological Collection Techniques

................................ ................................ ................................ ................................ ..... 5 A4 . Efforts to Identify Duplication

................................ ................................ ................................ .......... 6 A5. Sensitivity to Burden on Small Entities

................................ ................................ ............................ 7 A6. Consequence to Federal Program or Policy Activities if the Collection Is Not Conducted or Is Conducted Less Frequently

................................ ................................ ............ 7 A7. Special Circumstances

................................ ................................ ................................ ........................ 7 A8. Fe de ral Register Announcement and Consultation

................................ ................................ ....... 7 A9. Payment or Gift to Respondents

................................ ................................ ................................ ....... 8 A10. Confidentiality of the Data

................................ ................................ ................................ .............. 9 A11. Additional Justification for Sensitive Questions

................................ ................................ ........ 10 A12. Estimates of Hour Burden

................................ ................................ ................................ ............. 10 A13. Estimate for the Total Annual Cost Burden to Respondents or Record Keepers

.................. 11 A14. Estimates of Annualized Costs to the Federal Government

................................ ..................... 11 A15. Reasons for Program Changes or Adjustments

................................ ................................ .......... 11 A16. Plans for Tabulation and Publication

................................ ................................ ........................... 11 A17. Approval to Not Display the Expiration Date for OMB Approval

................................ .......... 15 A18. Exception to the Certification Statement

................................ ................................ ..................... 16 References

................................ ................................ ................................ ................................ ................. 17iii 

### List of **t ables**  Table 1. Administration times

................................ ................................ ................................ ................ 1 1 Table 2 . Timeline

................................ ................................ ................................ ................................ ..... 1 5 The following attachments are stored in the “ NW S07” folder in the “ REL Northwest Memos and Correspondence ” library, with the exceptions of Attachments D and E, which we will add later since the Federal Register notices cannot be submitted until this document is approved by IES: Attachment A : ESRA Attachment B : Copy of Survey and Online Survey Screen Shots Attachment C : Intro Letter Invitation and Follow

-Up Emails Attachment D : 60

- Day Federal Register Notice (to be added after the notice is submitted) Attachment E : 30

- Day Federal Register Notice (to be added after the notice is submitted) Attachment F: Pilot Survey Report Attachment G : Affidavits of NondisclosureSupporting statement for OMB clearance 1



## Supporting **Statement for OMB Clearance of a Surv** **ey of**  **Principals of Rural Schools Receiv**  **ing School Improv ement**  **Grants and Using the Transformation Model** 

**Section A : Justification**  A1. Circumstances n ecessitating c ollection of i nformation This data collection is autho rized by the Educational Sciences Reform Act (ESRA) of 2002 (see a ttachment A) . Part D, Section 174(f)(2 ) of ESRA states that as part of its central mission and primary function, each regional educational laboratory “shall support applied research by ... developing and widely d isseminating, including through Internet

-based means, scientifically valid research, information, reports, and publications that are usable for improving academic achievement, closing achievement gaps, and encouraging and sustaining school improvement, to schools, districts, institutions of higher education, educators (including early childhood educators and librarians), parents, policymakers, and other constituencies, as appropriate, within the region in which the regional educationa l laboratory is located.” REL Northwest and our partner Policy Studies Associates (PSA) , the author s of this study, are interested in how rural schools implement School Improvement Grants (SIGs) . These competitive federal grants have provide d funds to two cohorts of low

- performing schools to increase student achievement. The grants require schools to implement one of four school improvement models : the tran sformation model, which involves a variety of reforms; the turnaround model, which involves primar ily replacing staff; the restart model, which involves becoming a charter school; and the closure model, which closes the school. This descriptive study collects and reports on data from ru ral school principals across the nation that received federal SIG s in Cohort 1 and used the transformation model

. This information collection is essential to a study by the Regional Educational Laboratory (REL) Northwest, which serves five Northwest states (Alaska, Idaho, Montana, Oregon, and Washington). M ost rural Northwest schools with SIG funding (96 pe rcent) use d the transformation model to improve. REL Northwest and its partner PSA are collecting survey data about the implementation of the transformation model to inform regional and national supports for future school improvement efforts. Data collection will occur once in spring 2014. IES has approved the study plan

. **Statement of n**  **eed and b**  **ackground i**  **nformation** To date, there is little systematic evidence about the implementation of the SIG program. In the case of rural SIG settings, this lack of research evidence is particularly problematic . Rural schools often face steep challenges when trying to implement the kinds of staff replacement and on- site professional development practices promoted by the SIG program (Klein, 2010). This study addresses these challenges by examining the level of implementation of the SIG transformation model in rural regions, identifying the organizations

— states, districts,Supporting statement for OMB clearance 2

universities, and other entities

— that assist schools in these implementation efforts, and exploring perceived gaps in assistance. SIG funding mechanisms are not new. Congress put them in place as part of the No Child Left Behind Act (NCLB) of 2001 (2002) , the reauthorization of the Elementary and Secondary Education Act (ESEA). However, in 2009 the American Re covery and Reinvestment Act (ARRA) substantially increased funding for these grants

. With this increase, the U.S. Department of Education issued new guidance, dramatically changing SIG requirements ( U.S. Department of Education, Office of Elementary and Secondary Education, 2011 )

. Under this guidance, federal funds pass to states as usual, but states are now charged with assisting schools in implementing one of four school improvement models (U.S. Department of Education, Office of Elementary and Secondary Education [OESE], 2011): (1) transformation, replacing the principal and implementing a number of strategies to improve the school; (2) turnaround, replacing the principal and 50 percent of staff members and providing operational flexibility needed to implement a variety of school improvement strategies ; (3) r estart, reopening as a school operated by a charter or education management organization; and (4) c losure, closing the school and ensuring that students transfer to a high

- performing school. B e cause the transformation model require s less disruption and staff me mber replacement than other models, some consider it the most flexible of the four (Klein, 2010). Perhaps as a result of this flexibility, most grantee schools

— and 96 percent of ru ral schools nationally

— have chose n this model ( Hurlburt, Le Floch, Therriault, & Cole, 2011; Hurlburt, Therriault, & Le Floch , 2012 ). Therefore, our study focuses on the transformation model in rural settings

. The transformation model requires the following 11 activities ( U.S. Department of Education, OESE, 2011): 1. Replace the principal who led the school prior to the transformation model 2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors 3. Identify and reward school leaders, teachers, and other staff me mbers who i mproved student outcomes, and identify and r emove those who did not 4. Provide staff with ongoing, high

- quality, job

- em bedded professional development 5. Implement strategies designed to recruit, place, and retain staff 6. Use d ata to identify and implement a new instructional program 7. Promote continuous use of student data in order to inform and differentiate instruction 8. Establish schedules and strategies that provide increased learning time 9. Provide ongoing mechanisms for family and community engagement 10. Use operati onal flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes 11. Ensure that the school receives ongoing, intensive technical assistance and related supp ort from the LEA, the SEA, or an external organizationSupporting statement for OMB clearance 3

**Study d**  **esign, d**  **at a**  **c ollection, and**  **t iming**  This descriptive study collects and analyze s principals’ reports of SIG transformation model implementation in their rural schools. It uses a one - time survey of the 211 Cohort 1 principals of rural schools implementing the SIG transformation model (attachment B). The survey will occur in sprin g 2014

. It also incorporates existing data from the SIG baseline database available on the U.S. Department of Education website, including each school ’s National Center for Education Statis tics ( NCES ) locale code ( i.e., the geographic characteristics of the school) , p ercentage of students eligible for free or reduced

- price lunch ( i.e., poverty levels among students) , and p ercentage s of non- White and Native American students

. The study uses the data to address the following research questions: 1) How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation model? 2) What do principals report were the challenges to implementation of each activity? 3) What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how d id the principals rate the sufficiency of this assistance? 4) Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics? A2. How, by w hom, an d for w hat p urpose i nformation i s t o b e u sed To make good decisions about ongoing school transformation efforts and future similar school improvement initiatives, the nation’s education leaders need more information about rural SIG implementation of the transformation model. More specifically, the data from this study will be used by REL Northwest and our state cont acts to provide information to federal policymakers during ongo ing efforts to reauthorize ESEA. These policymakers need such information as they consider changing or retaining language around the requirements for the continued use of SIGs under ESEA, which are likely to continue to provide funds to schools for school improvement, as well as language incorporating ele me nts of the ESEA waivers involving school improvement efforts (U.S. Department of Education, 2012)

. We will also use the s e data to provide information to s tate policymakers in states with similar legislation around school improvement. Furthermore, we will use the s e data to inform education practitioners, particularly those who provide support for school improvement . These practitioners need the data to identify gaps in technical assistance for rural schools seeking to improve and uncover opportunities to provide additional and/or different services to schools. Why is it crucial for these policymakers and practitioners to consider changes in pol icies and practices for transforming rural schools? De spite the fact that the transformation model requires less disruption and staffing changes, m any education leaders anticipate that rur al schools, in particular, will have difficulty meeting the requirem ents of the transformation model. For example, recruiting and retaining high

- quality staff members is known to be more difficult in rural areas ( Hammer, Hughes, McClure, Reeves, & Salgado, 2005; Monk, 2007 ). In addition, finding technical assistance provid ers who can work with rural schools can be challenging (Corbett, 2011); transforming school schedules to allow for extended learning time can beSupporting statement for OMB clearance 4

difficult due to the longer bus rides and higher transportation costs (Sandel & Bhat, 2008); and rural parents can be mistrustful of outside technical assistance providers

— such as state employees or university professors

— who seem de termined to change their schools (Owens, Richerson, Murphy, Jageleweski, & Rossi, 2007). However, there is little empirical information about the implementation of the transformation model in rural schools. This study will provide education policymakers and practitioners with crucial information, including : specifying the extent to which the transformation activities were implemented and the challenges to implementation and identifying which activities were supported by technical assistance providers and how sufficient principals found this support. The study will do this primarily through a specially designed survey of the 211 rural school principals implementing SIG transformation models in Cohort 1 (fund ed from 2010 through 2013, with an option to carry over funds to 2014) . We have designed this survey ( attachment B ) based on our review of the literature on SIG and rural schools and on our staff members’ past work on SIG evaluation. The first section of the survey asks principals when they became an administrator at their school and if they are familiar with SIG implementation at their sch ool . If the principal is new to the school and does not know about the SIG implementation, the survey will prompt the principal to provide contact information for the prior principal or for a designated proxy, wh o is identified as a staff member who was em ployed during the SIG funding period and served in one of the following roles:
 A school

- level administrator, such as an assistant principal, who participated in the school’s SIG
 A teacher who was responsibl e for assisting with the implementation of the sch ool’s SIG
 A district administrator responsible for school improvement The second section of the survey asks principals to rate their school’s implementation of the 11 required transformation model activities using a four

- point scale that reflects the stages of implementation developed by Fixsen, Blase, Naoom, and Wallace (2009). Transformation activities include : 1 1. Replace the principal who led the school prior to commenceme nt of the transformation model 2. Use rigorous, transparent, and equitable evaluation systems for teachers and principals that take into account data on student growth, as well as other factors

1 In order to ensure that each survey item represents only one SIG transformation activity, we split several activities into multiple survey items. To ensure that the principal is knowledgeable about all activities, we eliminated those activities that are district rather than principal responsibilities, such as principal replac ement and evaluation. We also eliminated the activity, “Ensure that the school receives ongoing, intensive technical assistance and related support from the LEA, the SEA, or an external organization” because we ask principals about this assistance in a sep arate section of the survey.Supporting statement for OMB clearance 5

3. Identify and reward school leaders, teachers, and other staff me mbers who i mproved student outcomes, and identify and r emove those who did not 4. Provide staff with ongoing, high

- quality, job

- em bedded professional development 5. Implement strategies designed to recruit, place, and retain staff 6. Use d ata to identify and implement a new instructional program 7. Promote the continuous use of student data in o rder to inform and differentiate instruction 8. Establish schedules and strategies that provide increased learning time 9. Provide ongoing mechanisms for family and community engagement 10. Use operational flexibility (such as staffing, calendars/time, and budgeting) to improve student outcomes 11. Ensure that the school receives ongoing, intensive technical assistance and related supp ort from the LEA, the SEA, or an external organization We ask about other school improvement activities in which ru ral schools may engage (through an open- ended item), and we also ask principals to indicate which activities they believe were essential to core efforts to improve the school. Next, the principals will be asked to select one or more challenges for each activity . We will allow principals to select multiple responses for this survey item, since some items may be related to one another. For example, “insufficient funding to implement activities ” may mean that staff members are overworked and, therefore, also have “insufficient st aff time to implement activities

. ” Principals may select both responses if appropriate. They will also have the option of selecting “NA (no challenges) ” or “NA (did not implement this activity).” Finally, in an open

- ended item, principals may list other ch allenges to implementing the activities. The next section of the survey asks principals the extent to which they agree that technical assistance from other entities was sufficient to help the school implement the required transformation activities

. Participants will rate the sufficiency on a four

- point agree/disagree scale. The choice of entities includes their district, their state education agency, a partner in higher education, or another technical assistance provider. This list is based on the entities that other studies have found assisted with SIG most often (Corbett, 2011; Scott , McMurrer, McIntosh, & Dibner, 2012). If a principal selects “another technical assistance provider,” the survey will prompt the principal to fill in the name of the or ganization that provided the technical assistance. The principal s will also be able to elaborate on their response s regarding the sufficiency of the assistance from each entity. A3. Use of a utomated, e lectronic, m echanical , or o ther t echnological c ollection t echniques Initially, we will invite principals to complete an online survey (with electronic submission of responses through Survey Gizmo )

. I f return rates are less than 50 percent after the second reminder to complete the survey, we will offer an electronic gift card to increase the response rate. We will do this for three reasons. First, our pilot showed that offering an incentive (an Amazon electronic gift card) after several reminders increase d the response rate from 44 percentSupporting statement for OMB clearance 6

to 77 percent. Se veral research studies support the use of incentives (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992 ), particularly for online surveys of principals (Jacob & Jacob, 2012). Principals who completed the survey before the second reminder will be sent the incentive retroactively. Those that do not respond after three reminders will have the option of completing a paper version of the survey that they may email, fax, or mail back to our partner, PSA. Either version will take about the same amount of time to complete (20 minutes). PSA will host a website where respondents can access the link to the online survey. Attachment C contains the advance mail notification letter, the invitation to participate in the survey, and the follow

- up notification email/letter s . We have decided to first invite principals to respond to the survey online and then, after three reminders, to allow nonrespondents a paper option in case principals preferred this format

. Kaplowitz, Hadlock, and Levine (2004) found that a web survey application achieved similar response rates to a mailed hard copy questionnaire when both were preceded by an advance ma il notification. In addition, Millar and Dillman (2011) found that a combined use of multiple response - inducing techniques (web and mail) produced the best response rates

. Study r esults will be available via an online link to the study on the IES and REL Northwest websites . Participating principals and their districts will be invited to participate in webinars to discuss the results of the study. In these webinars, participants will share ideas about implementing the SIG transformation model and similar school improvement efforts . They will also discuss the supports provided by technical assistance providers and how to leverage additional supports to address challenges to school improvement

. This participation will be entirely voluntary and is presented as a way of motivating principals to participate. A4. Efforts to i dentify d uplication This one - time study collects unique information that is not available elsewhere and is not being collected by other studies. To date, most discussions of rural SIG have combined all rural schools regardless of their other ge ographic and demographic characteristics and have not delved deeply into the implementation of the required activities in the transformation model (e.g., Dee, 2012; Rosenberg, 2011; Scott, 2012; Scott et al. , 2012 ). While these past studies offered important information about SIG, the current study fills a need for information about rural schools’ implementation of the required activities in the transformation model. Ongoing n ational studies of SIG, such as Hurlburt and colleagues (2011 , 2012), focus on all types of schools and on the four school turnaround models broadly, not on characteristics of rural schools and the required activities in the transformation model . The se studies do not provide information about how states and other technical assistance providers might support specific school transformation activities in rural settings

. The A merican Institutes for Research and Mathematica are currently conducting an in

- de pth descriptive study of 25 SI G school s, including 10 in rural areas . While this study will provide anSupporting statement for OMB clearance 7

in

- de pth look at some ru ral schools, it relies on less than 5 percent of rural SIG schools and will not gene ralize across rural SIG schools. A5. Sensitivity to b urden on s mall e ntities This project collects data from principals of schools, which by definition are small entities. We have minimized the burden for respondents by (1) requesting information that is generally available to school leadership, including principals, other administrators, and teacher leaders directly involved in SIG implementation , and (2) requesting the minimum amount of information required to answer the study questions. Answering these questions will provide policymakers and practitioners wi th information needed to make good decisions about the SIG transformation model and similar school improvement efforts. For example, i tems about the geographic and demographic characteristics of the schools are not included in the survey because this information is available through the SIG baseline database

. As shown in section “A12. Total Annual Cost Burden to Respondents,” we estimate a 20

- minute total time burden on each respondent

. In addition, we will allow p rincipals to designate a proxy, if it help s minimize the burden on the school

. A6. Consequence to F ederal p rogram or p olicy a ctivities if the c ollection i s n ot c onducted or i s c onducted l ess frequently This one - time data collection has the potential to inform federal, state, and local agency prog rams and policies related to turning around low

- performing schools both thr ough future SIG allocations and other efforts such as NCLB waivers

. Without this study, policymakers and their supporters would have little informa tion about the unprecedented $3 billion that went into SIG in 2009 through the ARRA

— about six times what Congress had allocated the previous year ( Scott et al., 2012 ). Of these funds, more than $300 million went to rural transformation in the first cohort of schools (Hurlburt et al., 201 1). In addition, the types of efforts required by federal guidance to turn around low

- performing schools may be similar in the future. This study informs this future work on other school improvement e fforts that are similar to SIG. A7

. Special c ircumstances There are no special circumstances . The request fully complies with the regulations. A8. Federal Register a nnouncement and c onsultation **Federal Register a**  **nnouncement** ED will publish Federal Register Notices to allow both a 60

- day and 30

- day public comment period. The REL will assist ED in addressing any public comments received. There has been no comments received under the 60 day comment period notice.Supporting statement for OMB clearance 8

**Consultations o**  **utside the a**  **gency** Prior to this request for OMB approval, t he research team received expert feedback on this survey and study from researchers outside the agency through REL Northwest’s Te chnical Working Group (TWG). The two TWG members who contributed to this study were distinguished experts in their fields and extremely knowledgeable about research design and methods : Tom Dee of Stanford University and Hans Bos of American Institutes for Research. The se TWG members provide d written feedback on study plans and/or reports and met via telephone with the research team to clarify their feedback and provide additional recommendations

. A9. Payment or g if

t to respondents As described in section A3, i f return rates are less than 50 percent after the second reminder to complete the survey, w e will offer an electronic gift card to increase the response rate. We will do this for three reasons. First, during the pilot, response rate s were low until we offered a gift card from Amazon. Upon the first request, only two principals (22 percent ) completed the survey. After two reminders, four principals (44 percent ) had completed the survey and one had partially completed it. PSA staff decided to provide an Amazon gift card as an incentive. Several principals responded positively to the incentive. The principal, who had only partially completed the survey, finished the survey. Two others also completed the pilot survey for a response rate of 77 percent
. More detail s about the pilot are contained in attachment F. Second, we reviewed the literature and found several reviews of survey research that showed incentives increased response rates (Armstrong, 1975; Church, 1993; James, & Bolstein, 1992 ). In addition, a recent experimental study of the impact of incentives on the return rate of a principal survey (Jacob & Jacob, 2012) influenced our thinking. In this experimental survey of 1,177 principals, offering an incentive had a statistically significant positive impact on the return rate: 48 pe rcent of principals offered an incentive with the online survey returned the survey compared to 22 percent who were not offered the incentive. Third, REL guidance recommends the use of incentives for groups that are surveyed frequently ( Sloan, Ingels, & Burghardt, 2012 ). Be cause our participants are principals, they are likely to be surveyed frequently. Research shows that 50 percent of principals, in general, receive four or more survey requests per year and 25 percent receive more than seven pe r year (Jacob, Scott, & Bowers, 2008)

. Therefore, ba sed on the results of our pilot, the research literature, and REL guidance on survey incentives, we will o ffer an incentive to principals. In order to avoid offering the incentive unnecessarily, we will first ask principals to complete the survey without offering an incentive. Then, if at the second reminder the return rate is less than 50 percent (as i t was in the pilot), then we will offer a $20 Amazon gift card. Principals, who alre ady completed the survey without an incentive, will receive the $20 Amazon gift card retroactively as a thank you. We chose $20 as the amount of the gift card because this is the amount recommended in REL guidance ( Sloan, Ingels, & Burghardt, 2012 ).Supporting statement for OMB clearance 9

In ad dition to the gift card, p articipants will receive a link to the published study and invitations to several webinars to discuss the study. While this is not a payment or gift, it will allow participants to have a voice in the discussion of study results . During the webinar, participants will share information about implementing SIG and similar school improvement efforts. They will also discuss the supports provided by technical assistance providers and how to leverage additional supports for schools to a ddress challenges to school improvement . We believe this opportunity will motivate people to complete the survey

. A 10. Conf identiality of the d ata REL Northwest and PSA will be following the new policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183 : “ All collection, maintenance, use, and wide dissemination of data by the Institute ” are required to “ conform with the requirements of section 552 of title 5, United States Code, the confidentiality standards of subsection (c) of this section, and sections 444 and 445 of the General Education Provision Act (20 U.S.C. 1232g, 1232h). ” These citations refer to the Privacy Act, the Family Educational Rights and Privacy Act, and the Protection of Pupil Rights Amend ment. Subsection (c) of section 183 referenced above requires the Director of IES to “ develop and enforce standards designed to protect the confidentiality of persons in the collection, reporting, and publication of data

. ” Subsection (d) of section 183 prohibits the disclosure of individually identifiable information , and makes any publishing or communicating of individually identifiable information by employees or staff a felony. REL Northwest and PSA will protect the confidentiality of all information collected for the study and will use it for research purposes only. No information that identifies any study participant will be released. Information from participating institutions and respondents will be presented at aggregate levels in reports. Unique identification numbers will be assigned to each participating school and used to identify all survey responses. The ID number/name association files will be kept secure in a confidential file separate from the data analysis file. No information identifyin g respondents will be included in the study data files or reports. Data from the online survey software system will be downloaded and deleted from the online system within one week after the data collection ends . These data files will then be stripped of any identifying information. Information will be reported in aggregate so that individual responses are not identifiable. All identification lists will be destroyed at the end of the project. The research team is trained to follow strict guidelines for soliciting consent, administering data collection instruments, and preserving respondent confidentiality. All members of the research team will have successfully completed the Collaborative Institutional Training Initiative (CITI) course in the Protection of Human Research Subjects through Liberty IRB. A copy of the affidavit of nondisclo sure is provided in attachment G and will be signed by each researcher who will have access to the data. All study materials will include the following language:Supporting statement for OMB clearance 10

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across th e sample and will not associate responses with a specific district or individual. We will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony. A 11. Additional j ustification for s ensitive q uestions No questions of a highly sensitive nature are included in the survey or interview, and we have received Institutional Review B oard approval for this study

. A 12. Estimates of h our b urden We estimate the total reporting burden associated wi th this data collection to be 61.58 hours and $ 2,609.90 (table 1 ). This includes 63 principals designating a proxy and 1 69 survey respondents completing the survey (i.e. , 80 percent of 211). More details about these activities are included below in table 1. We estimated a principal ’ s hourly wage using May 2012 Bureau of Labor Statistics National Occupational Employment and Wage Estimates ( http://www.bls.gov/oes/current/oes_nat.htm ). **Designating a p**  **roxy** We anticipate that a pproximately 63 principals (i.e., 3 0 percent of 211) may designate a proxy to fill out the survey because they are either new to the school or too busy to fill out the survey. We based this estimate on the national average of 20 percent principal attrition yearly (Battle, 2010) , plus an additional 10 percent of principals who may be too busy

. We anticipate that the designation of proxies will add about 5 minutes to the burden on sch ools. Designating a proxy adds 5. 25 hours to the burden and $222 .50. **Survey** We will invite all 211 principals or designated proxies of our target population (i.e., the Cohort 1 rural SIG transformation schools ) to participate in the survey. The target response rate for these individuals is 80 percent, and the approximate time required for each respondent to complete the online s ur vey is 20 minutes on average , based on the pilot and on a similar survey (Scott, Davis, & Krasnoff

, 2012) . The tota l hours for the survey are 56.33, and the total cost is $ 2, 387 .41

. The most recent Schools and Staffing Survey of principals by the National Center of Education Statistics had a re sponse rate of 79 percent for both public school and Bureau of Indian Education principals ( Battle, 2009). This report did not disaggregate the response rates for rural school principals. But, w e suspect that response rates would be somewhat lower for rura l school principals. For example, the response rate of the rural district administrators ’ survey was 74 percent ( Zhang, 2008 ). Therefore, because we set our target at 80 percent, we will use several methods to maximize response rates and address nonresponse, such as offering an incentive as described in A9 and the use of online data collection described in A3

.Supporting statement for OMB clearance 11

**Table 1: A dministration times**  **Reporting m**  **ethod Number of** 

**r espondents**  **Av erage** 

**t ime**  **(hours) Total**  **b urden** 

**(hours) Hourly**  **r ate**  **Estimated c**  **ost** Designating a proxy 63 5 minutes 5. 25 $42.38 $ 222.50 Survey 1 69 20 minutes 56.33 $42.38 $2, 387.41 **Total 61.58**  **$2, 609.90**  A 13. Estimate for the t otal a nnual c ost b urden to respondents or record k eepers There are no direct start

-up costs to respondents

. A 14. Estimates of a nnualized c osts to the Federal Government The total estimated cost for this study is $373,250. Although the data collection occurs only once, these cost s cover a two
- ye ar study period; therefore, the annual cost of the stud y is $186,625. We estimated these costs using the amounts budgeted in the REL Northwest Year 2 plan and approved by IES. These costs include planning the study, administering the survey, collecting and cleaning data ( including the use of an incentive ) , and analyzing and reporting results

. In addition to the evaluation costs, there are personnel costs of several Federal employees involved in the oversight and analysis of information collection that amount to an annualized cost of $ 5,000 for Federal labor. A 15. Reasons for p rogram c hanges or a djustments This is a new study. A 16

. Plans for t abulation and p ublication We will tabulate and analyze the data by the four research question s . We will address the first three questions by providing the frequencies of survey responses for ea ch transformational activity by level of implementation, challenges, and technical assistance provider. We will also explore the first three research questions using open

- ended items on the survey. The fourth research question requires inferential statistics. Plans for tabulation and analysis are described in more detail below. **Research question 1** : How do principals of rural SIG transformation schools rate the extent to which their school implemented the required activities in the transformation model? We will address the first research question primarily by calculating frequencies of survey responses for each transformation activity by the extent of implementation, ( i.e. , N/A, p lanning/developing, e arly or partial implementati on, or full implementation ) . The categories of level s of implementation are defined more fully in the survey (attachment B). We will also provide descriptive statistics about the percentages of principals that rate these activities as essential to their co re efforts to improve schools.Supporting statement for OMB clearance 12

We will use one open- ended item to add de tails about the implementation of activities related to school improvement. The item asks principals to elaborate on their responses regarding transformation model activities that are fully implemented in their school and essential to core efforts to improve the school

. To examine the item, we will develop codes inductively (Mayring, 2000)

. To develop codes, the two study authors will review approximately a third of the data independently and create codes. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a particular piece of data, a third researcher will determine the final code. **Research question 2** : What do principals report were the challenges to implementation of each activity? As in research question one, we will address this question primarily through frequencies of su rvey responses about chal lenges. Because some activities may pose multiple challenges, principals may select all challenges that apply (i.e., insufficient funding, insufficient staff expertise, insufficient staff time, insufficient technological capacity/equipment, lack of teacher support, lack of district support/guidance, other )

. Principals will also be given the opportunity to elaborate on their responses regarding challenges . To conduct an exploratory analysis of this qualitative data, we will develop codes inductively (Mayrin g, 2000). To create codes, the two study authors will review approximately one third of the data independently. They will then compare these codes and agree on final codes. Next, they will both independently code the remaining data and compare results. If they cannot agree on the code for a particular piece of data, a third researcher will determine the final code. **Research question 3** : What types of technical assistance providers assisted schools, which particular transformation activities did they assist with, and how d id the principals rate the sufficiency of this assistance? We will provide two types of descriptive statistics to explore this question. First, we will provide frequencies for the principals reporting that their school received assistance on each of the transformation activities for particular types of providers (i.e. , districts, states, universities, and other). Then, we will provide descriptive statistics representing the degree to which principals whose schools received this assistance agreed that the assistance was sufficient. To provide more detail about technical assistance providers, we will examine the open

- ended response that asks principals to provide the name(s) of their technical assistance organization(s) We will list the top 10 organizations in the main body of the report and the complete list of organizations in the technical appendix. Finally, open

- ended items about each type of technical assistance provider will allow principals to elaborate on their responses regarding the sufficiency of the technical assistance from eachSupporting statement for OMB clearance 13

type of provider. We will analyze this qualitative data using deductive coding (Mayring, 2000). A s described previously, two study authors will review the data independently, then compare and agree on final codes. Disagreements, if any, will be settled by a third researcher. **Research question 4** : Do principal ratings of implementation, challenges, and technical assistance providers vary by school geographic or demographic characteristics? To examine variation in stages of implementation and in assistance by the geographic and demographic characteristics of schools, we will use the Kruskal Wallis H test

. The Kruskal Wallis H test is a nonparametric test that allows the comparison of mean ranks of more than two categorical groups. We have chosen to use this test because, in our study, the dependent variables of interest (our survey responses about the stages of implementation and about sufficiency of assistance) are ordinal and the independent variables of interest (geographic and demographic characteristics of schools) are nominal. We will conduct four series of Kruskal Wallis H tests: 1) Implementation by geographic characteristics of schools, (i.e. , by locale codes representing the type of rural area a school is in, such as a rural remote area versus a distant town) 2) Sufficiency of assistance by geographic characteristics of schools 3) Implementation by demographic characteristics of schools (i.e. , by whether the school serves primarily white, Latin o, or Native American students) 4) Sufficiency of assistance by demographic characteristics of schools We discuss each of these series of analyses in more detail below. **To examine variations in reported implementation**  **by geographic characteristics,** we will conduct the H tests for each item’s response category by each of the six locale codes (31, 32, 33, 41, 42, and 43). However, any cells with fewer than five responses will be eliminated from the analysis or will be collapsed if possible, (e.g., strongly agr ee and agree could be combined and strongly disagree and disagree could be combined) . In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini

Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variations in reported assistance**  **by geographic characteristics,** w e will also conduct H tests for the items about assistance. For these items, we will consider assistance from each type of provider (district, state, university, and other technical assistance providers) separately. We expect that in many instances this may resu lt in cells with fewer than five responses. Those items with small cell sizes will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependentSupporting statement for OMB clearance 14

variable. The independent or “grouping” variable will include the six locale codes. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which specific groups were different from one another. We will display mean ranks, degrees of freedom, H test stati stics, and Dunn’s test statistics for each analysis. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variations in reported implementation**  **by demographic characteristics,** we will classify the schools in four ways: those that are more than 50 percent Latino (28 schools), those that are more than 50 percent Native American (50 schools), those that are more than 50 percent White (68 schools), and those that do not have a single racial majority (64 schools). 2 We will use Kruskal Wallis H tests to examine variations in the data on implementation by student demographic characteristics. We will conduct the H tests for each item’s response category b y each of the four demographic categories, although any cells with fewer than five responses will be eliminated from the analysis. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent variable. The independent or “grouping” variable will include the four demographic categories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendix. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995). **To examine variati ons in reported**  **assistance by demographic characteristics,**  we will also use Kruskal Wallis H tests. As in the previously describe d analysis of implementation, we will consider assistance from each type of provider (district, state, university, and other te chnical assistance providers) separately. We expect that in many instances we will need to eliminate analysis for cells with fewer than five responses

. In each analysis, the ordinal Likert scale responses for the survey item will serve as the dependent var iable. The independent or “grouping” variable will include the four demographic categories. If the H test is statistically significant, we will perform a post hoc Dunn’s test to determine which groups differed from one another. We will display mean ranks, degrees of freedom, H test statistics, and Dunn’s test statistics for each analysis in a technical appendix. To determine statistical significance we will use the Benjamini

- Hochberg correction to adjust for multiple comparisons (Benjamini & Hochberg, 1995)

. **Publications** F or this study, we will create two publications. A What’s Happening ? r eport will be targeted to our national practitioner and policymaker audience, particularly state agencies, districts, universities, and other technical assistance provide rs involved in supporting school improvement efforts similar to those in SIG transformation schools. This report will contain graphic summaries of the study and its findings.

2 One school had missing data for student ethnicity in the SIG baseline database.Supporting statement for OMB clearance 15

A Stated Briefly report will summarize findings that may appeal to regional and state organizations that are interested in improving schools but that do not provide direct technical assistance to schools

— particularly those organizations that set school improvement policies and guidance. The Stated Briefly report should help these organizations set policy and create guidance that supports aspects of transformation that fewer participants reported were fully implemented, as well as potentially reducing or remediating such reported challenges. Both What’s Happening and Stated Briefly reports will be available on the REL Northwest website and the IES website. As stated earlier, geographic and demographic analyses will not be conducted when fewer than five respondents are included. This will reduce the risk of deducti ve disclosure. Timeline The timeline for the study, including the publications, is presented in table 2

. **Table 2**  **. Timeline**

**Timeline Milestone**  **Date for deliverable** January 2013

–April 2013 Develop data collection protocol and instruments NA April 2, 2013 Subm it draft s tudy p lan to TW G , which includes study methodology, data collection activities , and analysis plan NA

April 30, 2013 Subm it draft s tudy pl an to IES , revised in response to TW G feedback Draft plan to IES April 30, 2013 July 24, 2013 Subm it final study plan to IES Final plan to IES June 14, 2013 October 201 3 OMB submission of 60
- day Federal Register Notice NA March

– May 2014 Data collection NA June 2014 Data analysis NA July 2014 Wr ite report s NA September 2014 Subm it draft What’ s Happening ? report to IES Draft What’ s Happening ? report, 12 months after approval November 2014 Subm it final What’ s Happening? report to IES Final What’ s Happening ? report, 13

– 14 months after approval December 2015 Subm it draft Stated Briefly to IES Draft Stated Briefly, 15 mo nths after approval January 2015 Subm it final Stated Briefly to IES Final Stated Briefly, 16 mo nths after approval A 17. Approval to n ot d isplay the e xpiration d ate for OM B a pproval This is not applicable. The instrument will display the expiration date for OMB approval.Supporting statement for OMB clearance 16

A 18. Exception to the c ertification s tatement We do not seek any exceptions to the certification statement.Supporting statement for OMB clearance 17

References Armstrong, S.J. (1975). Monetary incentive s in mail surveys. Public Opinion Quarterly, 39 (1), 111

– 116. Battle, D. (2009). Characteristics of public, private, and Bureau of Indian Education elementary and secondary school principals in the United States: Results from the 2007

–08 Schools and Staffi ng Survey ( First Look, NCES 2009

- 323). Re trieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2009/2009323.pdf Battle, D. (2010). Principal attrition and mobility: Results from the 2008

–09 principal follow

- up survey (First Look, NCES 2010

- 337). Retrieved from U.S. Department of Education, National Center for Education Statistics website: http://nces.ed.gov/pubs2010/2010337.pdf Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society. Series B (Methodological), 57 (1), 289

–

300. Church, A.H. (1993). Estimating the effect of incentives on mail survey response rates: A meta

analysis. Public Opinion Quarterly, 57(1), 62

–79. Corbett, J. (2011). Lead turnaround partners: How the emerging marketplace of lead turnaround partners is changing school improvement. Lincoln, IL: Academic Development Institute, Center on Innovation and Improvement. Dee, T. (2012). School turnarounds: Evidence from the 200 9 stimulus (NBER Working Paper No. 17990)

. Cambridge MA: National Bureau of Economic Research. Fixsen, D.L., Blase, K.A., Naoom, S.F., & Wallace F. (2009). Core implementation components. Research on Social Work Practice , 19 (5), 531

–540. Hammer, P.C., Hughes, G., McClure, C., Reeves, C., & Salgado, D. (2005). Rural teacher recruitment and retention practices: A review of the research literature, national survey of rural superintendents, and case studies of programs in Virginia. Nashville, TN: Edvantia, Appalachia Educational Laboratory. Hurlburt, S., Le Floch, K.C., Therriault, S.B., & Cole, S. (2011). Baseline analyses of SIG applications and SIG

- e ligible and SIG

- a warded s chools (NCEE 2011

- 4019). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20114019/pdf/20114019.pdfSupporting statement for OMB clearance 18

Hurlburt, S., Therriault, S.B., & Le Floch, K.C. (2012). School Improvement Grants: Analyses of state applications and eligible and awarded schools (NCEE 2012

- 4060). Retrieved from U.S. Department of Education, Institute of Education Sciences, National Center for Education Eva luation and Regional Assistance website: http://ies.ed.gov/ncee/pubs/20124060/pdf/20124060.pdf Jacob, R.T., & Jacob, B. (2012). Pre

-notification, incentives, and survey modality: An exp erimental test of methods to increase survey response rates of school principals. Journal of Research on Educational Effectiveness, 5 (4) , 401

–418. Jacob, R.T., Scott, L., & Bowers, A. (2008, April). Evaluation and implementation of web

- surveys in school

- b ased settings . Paper presented at the annual meeting of the American Educational Research Association, New York, NY. James, J.M., & Bolstein, J. (1992). The effect of monetary incentives and follow

- up mailings on the response rate and response quality in mail surveys. Public Opinion Quarterly, 54 (3) , 346

–361. Kaplowitz, M.D., Hadlock, T. D., & Levine, R. (2004). A comparison of w eb an d mail survey response rates. Public Opinion Quarterly, 68 (1) , 94

– 101

. Klein, A. (2010, May 21). School turnaround models draw bipartisan concern. Education Week, 29 (33)

. Re trieved from http://www.edweek.org/ew/articles/2010/05/21/33turnaround.h29.html Mayring, P. (2000). Qualitative content analysis. Forum: Qualitative Social Research, 1 (2). Retrieved from http://www.qualitative

research.net/index.php/fqs/article/view/1089/2386 Millar, M. M. , & Dillman, D. A. (2011). Improving response to web and mixed

- mode surveys. Public Opinion Quarterly, 75 (2), 249

–269. Monk, D.H. (2007). Recruiting and retaining high

- quality teachers in rural areas. Future of Children, 17 (1), 155

– 174. No Child Left Behind Act of 2001. (2002). Pub. L. No. 107

- 110, 115 Stat. 1425. Owens, J.S., Richerson, L., Murphy, C.E., Jageleweski, A. , & Rossi, L. (2007). The parent perspective: Informing the cultural sensitivity of parenting programs in rural communities. Child & Youth Care Forum, 36 (5/6), 179

– 194. Rosenberg, S.J. (2011). The challenges in implementing school improvement grant models in rural high schools (Master’s thesis)

. Duke University, Durham, NC

.Supporting statement for OMB clearance 19

Sandel, K., & Bhat, S. (2008). Financing and sustaining out

- of

-school time programs in rural communities . Retrieved from Finance Project website: http://www.financeproject.org/publications/ostruralyouth.pdf Scott, C

. , Davis, A ., & Krasnoff, B. (2012). Oregon School Improvement Grants: Status report 2011

– 2012. Portland, OR: E ducation Northwest. Scott, C., McMurrer, J., McIntosh, S

. , & Dibner, K. (2012). Opportunities and obstacles: Implementing stimulus

-

funded School Improvement Grants in Maryland, Michigan, and Idaho

. Washington, DC: Center on Education Policy. Scott, G. (with Sirois, E., Spicer, S., Arsenault, J., King, M., Sorbello, S., & Steel

- Lowney, B.). (2012). School Improvement Grants: Education should take additional steps to enhance accountability for schools and contractors (GAO

- 12

- 373). Washington DC: U.S. Go vernment Accountability Office. Sloan, M., Ingels, J., & Burghardt, J. (2012). Analytical and technical support for advancing education evaluation: How to put together an OMB supporting statement . Princeton, NJ: Mathematica Policy Research. U.S. De partment of Education

. (2012, June 7). ESEA flexibility (Updated ed.)

. Retrieved from http://www.ed.gov/esea/flexibility/documents/esea

- flexibility.doc U.S. Department of Education, Office of Elementary and Secondary Education. (2011 ). Guidance on fiscal year 2010 School Improvement Grants under section 1003(g) of the Elementary and Secondary Education Act of 1965 . Retrieved from http://www2.ed.gov/programs/sif/sigguidance02232011.pdf Zhang, Y. (2008). Some perspectives from rural school districts on the No Child Left Behind Act. Retrieved from Center on Education Policy website: http://www.cep

dc.org/displayDocument.cfm?DocumentID=308