# Supporting Justification for OMB Clearance of an

**Evaluation of the Early Warning and Intervention Monitoring System Under the Regional Educational** 

**Laboratory Program (REL) OMB Clearance**  **Request, Part B**

**July 2013**  **Submitted to: Submitted by:**  U.S. Department of Education Am erican Institutes for Research Institute of Education Sciences 1000 Thomas Jefferson Street NW 555 New Jersey

Ave. NW, Rm. 308 Suite 200 Washington, DC 20208 Washington , DC 20007 - 3835

1120 East Diehl Road, Suite 200 Naperville , IL 60563 - 1486 866 - 730 - 6735 www.relmidwest.org This publication was prepared for the Institute of Education Sciences (IES) under contract ED - IES - 12 - C - 0004 by Regional Educational Laboratory Midwest, administered by American Institutes for Research. The content of the publication does not necessarily re flect the views or policies of IES or the U.S. Department of Education, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. The publication is in the public domain. Authorization to reproduce in whole or in part for educational purposes is granted. 0 7 /1 6REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—2 **CONTENTS Page**  Study Background ................................ ................................ ................................ ....................... 4 1. Respondent Universe and Sampling Methods ................................ ................................ ........ 6 2. Description of Procedures for the Collection of Information ................................ ............... 10 3. Description of Procedures for Maximizing Response Rates ................................ ................. 29 4. Description of Tests, Procedures, and Methods ................................ ................................ .... 30 5. Contact Information for Statistical Consultants ................................ ................................ .... 31 References ................................ ................................ ................................ ................................ ..... 34 Attachment A. Recruitment Materials ................................ ................................ .......................... 36 A - 1. Advance Letter (E - Mail Text) ................................ ................................ ........................... 37 A - 2. Study

Overview for Schools and Districts ................................ ................................ ........ 38 A - 3. Letter From IES Endorsing the Study ................................ ................................ ............... 40 A - 4. Nonresponse Follow - Up (E - Mail Text) ................................ ................................ ............ 41 A - 5. Telephone School Screening Protocol/Interview ................................ .............................. 42 A - 6. Telephone District Screening Protocol/Interview ................................ ............................. 50 A - 7. Final Agreement Form (School MOU) ................................ ................................ ............. 53 A - 8. Final Agreement Form (District MOU) ................................ ................................ ............ 65 A - 9. Final Agreement Form (State MOU) ................................ ................................ ................ 73 Attachment B. Data Collection Instruments ................................ ................................ ................. 77 B - 1. Draft Protocol

— School - Level Survey ( Treatment Contrast and Data - In formed Allocation of Dropout Prevention Interventions for Students) ................................ ................. 78 B - 2. Draft Protocol

— School D ata Culture Survey ................................ ................................ ... 94 B - 3. Archival Records Protocol ................................ ................................ .............................. 104 B - 4. Protocol for Collecting Data Stored in the EWS tool in Treatment Schools ................. 117REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 3 **CONTENTS ( continued**  **) TABLES AND FIGURES**  **Page Tables**  Table 1. Potential Respondent Universe of Public Schools In Ohio ................................ ............................. 7 Table 2. Power Analyses: Minimal Detectable Effect Sizes for Each Binary Student Outcome ................ 8 Table 3. Recruitment Process and Respondent Universe for the EWIMS Impact Study ........................... 12 Table 4. Mapping of Recruitment Screening Protocol Items to Topics ................................ ...................... 14 Table 5. Data Sources, Data Elements, and Timeline for Collection ................................ .......................... 17 Table 6. Mapping Web - Based School Survey Items to Constructs ................................ ............................ 22 Table 7. Mapping Web - Based School Survey Items to Constructs ................................ ............................ 23 Table 8. Mapping Treatment Contrast Survey Items to Constructs ................................ ............................ 25 Table 9. Schedule of Activities ................................ ................................ ................................ ................... 28 Table A - 1. Major Tasks of the Project ................................ ................................ ................................ ........ 60 Table A - 2. Schedule and Key Activities for Early Warning Intervention and Monitoring System Implementation ................................ ................................ ................................ ................................ ........... 61 Table A - 3. Major Tasks of the Project ................................ ................................ ................................ ........ 71 Table B - 1. Reliability Estimates of Principals’ Key Dimensions of Data Us e Scales and Subscales ....... 95 Table B - 2. Ohio and REL Midwest Data Sharing ................................ ................................ .................... 105 **Figure s**  Figure 1. Minimal Detectable Effect Size for Continuous Student - Level Outcomes ................................ . 10 Figure A - 1. Early Warning Intervention Monitoring System Implementation Process .............................. 54 Figure A - 2. Early Warning Intervention Monitoring System Implementation Process .............................. 66REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—4 

### SUPPORTING JUSTIFICA **TION FOR OMB CLEARAN CE OF AN**  **EVALUATION OF THE EA RLY WARNING AND INTE**  **RVENTION MONITORING SYSTEM**  **UNDER THE REGIONAL E DUCATIONAL** 

**LABORATORY PROGRAM ( REL)**  **OMB CLEARANCE REQUES T, PART B**  **Study Back ground**  The U.S. Department of Education (ED) requests clearance for the recruitment materials and data collection protocols under the OMB generic clearance agreement (OMB Number [IES to complete]) for activities related to the Regional Educational Laboratory Program (REL). ED, in consultation with American Institutes for Research (AIR), is planning a two - part evaluation of the Earl y Warning and Intervention Mon itoring S ystem (EWIMS), consisting of an impact study and an implementation study . OMB approval is being requested for a multimode data collection and anal ys is of a group of schools, students, and staff members in Ohio public schools. The impact study

cons ists of data collection from the state of Ohio, districts, and schools. The implementation component consists of data collection from participating schools. Specifically , in this OMB clearance package, ED is requesting clearance for the following data coll ection approaches:  Recruitment materials for all participating districts and schools  Extant administrative records data collections from the state of Ohio, school districts within Ohio, and treatment and control schools within Ohio  The transfer of data fro m treatment schools to the evaluation team via populated Earl y Warning S ys tem ( EWS ) tools  A Web - based survey

of school leaders in treatment and control schools The implementation study will include additional forms of data collection ( satisfaction survey , interviews with a EWIMS data team member, monthly logs of EWIMS data team meetings) however, the ED is not seeking approval for these measures of implementation because they are part of the typical EWIMS intervention and present no burden to participants . As detailed more full y in the project description that follows, this impact study (designed as a cluster randomized controlled trial) will focus on student outcomes spanning multiple domains of school success (student risk status, scores on graduation tests, persistence and progress in school, and being on track at the end of ninth grade) and will examine whether the EWIMS model has an impact on intermediate outcomes in schools, including the schools’ data culture 1 and data -

1 Data culture is a school’s general approach toward using data to inform educational decision making and includes contextual factors (e.g., the assessment and instructional context), supports for data use (e.g., professional development or structured time to review data), working with data (e.g. , frequency and depth of data use), and responses to data (e.g., assignment of interventions to students).REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 5 informed allocation of dropout prevention interventions. The implementation study will focus on schools’ experience with implementation, the extent to which schools faith fully implement the EWIMS model and the interventions provided to students identified as at risk by the EWS tool . The purpose of the project is to assess the implementation and impact of EWIMS, a data tool and process for implementing a sy stem of data - driven decision making. Developed by the National High School Center, EWIMS provides a means of sy stematically and rel iably

identifying students at risk for dropping out of high school. The proposed study is a two - ye ar school - level randomized controlled trial (RCT) to examine the impact of implementing EWIMS on school processes and student outcomes. **Impact Study** The focu s of the impact study will be to assess the effectiveness of EWIMS on student and school outcomes. For that reason, the study will address the following research questions: 1. What is the impact of the EWIMS model on outcomes for students in schools, includi ng: a. indicators of student risk? b. scores on graduation tests? c. persistence and progress in school? d. predicted probability

of on - time graduation ? In addition, RELMidwest will conduct exploratory research anal ys es for both students and schools. 2. What is the imp act of EWIMS on outcomes for subgroups of students, including: a. students who receive free or reduced - price lunch (FRPL)? b. students who are English language learners (ELLs)? c. students with Individualized Education Plans (IEPs)? d. students with baseline indicato rs of risk for attendance, course failures, and/or behavior? 3. What is the impact of EWIMS on other school outcomes, including: a. data - informed allocation of dropout prevention interventions for students? b. school data culture (including the context, supports fo r data use, wor king with data, and responses to data)? The exploratory

student subgroup anal ys es will address research questions 1a

– 1d that will assess whether the impact of EWIMS on student outcomes differs for key subgroups of students: students who rece ive free or reduced - price lunch (FRPL), students who are English language learners (ELLs), students with Individualized Education Plans (IEPs), and students with initial risk (with one or multiple risk factors) in the fall 2013 semester, preceding random a ssignment and implementation. We acknowledge that some students may be included in more than one of these anal ys es because they may be members of more than one subgroup.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 6 The exploratory

school - level impact questions are intended to understand how earl y ad option of EWIMS may

cause initial changes in how schools use data to identify at - risk students and respond with interventions. To minimize burden on participating districts and schools, this study draws heavily

on extant data to address the study’ s research questions outlined earlier in this statement. Student - and school - level baseline and outcome data for the impact study will be obtained from multiple sources, including the Ohio Department of Education (ODE), the EWS tool, and school and district administrative data. **Implementation Study** RELMidwest also will conduct an implementation study to describe treatment schools’ experiences with adoption and earl y implementation of EWIMS, and the extent to which there is a difference between the intervention and business as usual in control schools. The implementation study

will address the following research questions: 1. To what extent do treatment schools faithfully implement the EWIMS model? 2. To what extent does business - as - usual practice in control schoo ls include the use of data for identifying at - risk students (treatment contrast)? 3. What are the specific interventions provided to students identified as “at risk” by the EWIMS model in treatment schools? T he remainder of Part B address es the following: respondent universe and sampling methods (though no sampling methods will be used for this randomized controlled trial); description of procedures for maximizing response rates; description of tests, procedures and methods; and contact info rmation for statistical consultants and key staff . 

### 1. Respondent Universe **and Sampling Methods** This section describe s the respondent universe and sampling methods for recruitment . To test the impact of EWIMS o n student outcomes , ED ’s contractor will recru it and randoml y assign high schools in Ohio to use or not use the EWIMS model. The recruitment strategy will focus on both districts and schools . ED’s contractor will conduct recruitment both in a top

–down approach (district then school) and in a bottom

– up approach (school then district). The recruitment strategy

is designed to target schools that are relativel y large (e.g., more than 150 Grade 9 students) and that demonstrate a need for improved graduation rates and student outcomes (e.g., school s with graduation rate s between 25 and 95 percent , with a particular emphasis on those schools with graduation rates between 50 and 90 percent ) . The cutoff of 150 Grade 9 s tudents was selected to establish sufficient statistical power to achieve a minimum detectable effect size of 0.25 or smaller. The sample will consist of Ohio regular public schools serving Grades 9

– 12 that indicate their commitment to implement the EWIMS model and their willingness to adhere to all study requirements (i.e., random assignment and the completion of data collection activities). Although EWIMS can be implemented in both middle schools and high schools, this study

will focus on high schools bec ause some key intended outcomes of the EWIMS, such as whether students are on track for graduation in terms of course credits andREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 7 grades , are relevant onl y for high school students. Moreover, recruit ing and studying both middle school and high school outcomes would exceed available resources . ED’s contractor will target schools with at least 150 students in Grade 9 with estimated graduation rates between 25 and 95 percent to ensure that the study can feasibly

detect an impact on student outcomes . Taking these two inclusion criteria together (more than 150 Grade 9 students and graduation rates between 25 and 95 percent), there is an estimated pool of 321 potential schools for recruitment in the EWIMS Impact Study , as s hown in Table 1. Based on a power analy sis, the ED expect s to need 72 schools to provide adequate po we r for detecting the effects on outcomes .

**Table 1**  **. Potential Respondent Universe of Public Schools In Ohio**  **Total n**  **Treatment Control**  All school dis tricts in Ohio* 609

-- -- All p ublic s chools in Ohio s erving Grades 9

– 12** 858

-- -- Schools with Grade 9 enrollment greater than 150* 369

-- -- Schools with graduation rates between 25 and 95 percent* 601

-- -- Schools with both Grade 9 enrollment greater than 150 and graduation rates between 25 and 95 percent 321 Schools in study sample 72 36 36 Source: *O hio Department of Education (2010

– 11) ; * *Common Core of Data (2010

– 11) . **Degree of Accuracy Needed** Because this study

is focused on examining policy- relevant impacts for at - risk students , some continuous outcomes (e.g., number of course failures) will be examined as binary indicators (e.g., 1=failed one or more courses, 0=did not fail any courses) for impact analy ses. This decision is based on e xtensive research documenting the predictive power of these binary “flags” for on - time graduation ( Allensworth & Easton, 2005, 2007; Balfanz, Herzog, & Mac Iver, 2007; Neild & Balfanz, 2006; Silver, Saunders, & Zarate, 2008) . Further, some of these binary indicators are used explicitly within the EWS tool to flag at - risk students (e.g., attendance flags for student s who miss more than 10 percent of school day s). A comprehensive list of the binary variables that will be used in the impact models to examine the impact of EWIMS on student outcomes is provided in Table 2 , along with information about how these variables will be coded and our corresponding assumptions for the power anal ys es. Power anal ys es for binary student - level outcomes were conducted using equations derived from Optimal Design Plus Empirical Evidence (Version 3.0) ( Raudenbush, Spybrook, Congdon, Liu, Martinez, Bloom, & Hill, 2011 ). To estimate the number of schools needed to have adequate statistical power (0.80) for detecting differences betw een students in treatment and control schools for the impact analy ses, the power anal ys es assumed an alpha of 0.05 (two - tailed). ED’s contractor estimated power for a constant effects blocked cluster - random assignment design with the treatment occurring at Lev el 2. Following Bloom, Hill, Black, and Lipsey (2008), the study

is powered to be able to detect a minimum detectable effect size (MDES) roughl y equivalent toREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 8 0.25 for student outcomes; this effect size is generally considered necessary for an intervention to have educational significance. To date there are no studies that document the existence or relative size of the impact of implementing an early warning sy stem on student outcomes, thus the suggested MDES of 0.25 is a conservative estimate for this study . Specifically , the study is powered with enough schools and students within schools to be able to obtain effects of 0.25 or below for all student outcomes. Give n the expected low incidence of students flagged as no longer in school 2 , the enrollment/persistence outcome requires a minimum of 72 schools and an average of 150 students per school to obtain a difference between treatment and control schools that is les s than or equal to 0.25, using a standardized Cox Index for binary measures. Including 72 schools and 150 students per school provides sufficient power to detect effects ranging from 0.09 to 0.17 for the other binary outcomes of interest. It is important t o note that the estimated MDES for each outcome is sensitive to the assumptions about the 95 percent plausible interval of school means (in proportions). Assumed 95 percent plausible intervals of school means for each outcome are detailed in Table 2 . Based on these analy ses, the study requires approximately 72 schools in the study , with half assigned to each condition, to detect an MDES of 0.25 (or smaller) for each outcome variable of interest.

**Table 2**  **. Power Analyses: Minimal**  **D etectable**  **E ffect**  **S izes for** 

**E ach**  **B inary**  **S tudent**  **O utcome** 

**Student Outcome**  **Minim um**  **Detectable Effect**  **Size(MDES) (as Cox Index)**  **Expected Mean**

**Proportion by Condition**  **3 95% Plausible Interval**  **s**

**of School Mean**  **s (**  **in Proportions**  **) Control**  **Treatment Low**  **er Bound**  **Upper**

**Bound Attendance**  (1=missed more than 10% of school days; 0=did not miss more than 10%

) - 0.128 0.150 0.125 0.100 0.200 **Fail Any Course** (1=failed one or more courses ; 0=did not fail any courses) - 0.098 0.250 0.221 0.200 0.300 **Fail Any Core Course** (1=failed one or more core course ; 0=did not fail any core courses) - 0.108 0.200 0.173 0.150 0.250 **Behavior** (1=suspended one or more times; 0= not suspended) - 0.174 0.100 0.077 0.050 0.150

2 We are basing our school selection on graduation rates, and we know that they do not correspond directly to dropout rates that often underestimate the number of students who actually dropout. According to NCES, the Ohio graduation rate in 2008 - 09 was 79.6, while the dropout rate was 4.2. Documented dropout is a low incident event. 3 The MDES s as well as the E xpected Mean proportion by condition are based on the contractor’s prior research on dropout prevention including the IES funded Check &Connect Impact Evaluation , as well as the REL Midwest Dropout Prevention Alliance suite of studies, previously mentioned.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 9 **Student Outcome**  **Minim um**  **Detectable Effect**  **Size(MDES) (as Cox Index)**  **Expected Mean**

**Proportion by Condition**  **3 95% Plausible Interval**  **s**

**of School Mean**  **s (**  **in Proportions**  **) Control**  **Treatment Low**  **er Bound**  **Upper**

**Bound Predicted Probability of**  **Graduation** (1=predicted probability less than 0.80; 0= predicted probability greater than or equal to 0.80) - 0.108 0.200 0.173 0.150 0.250 **GPA** (1=GPA less than or equal to 2.0; 0=GPA greater than 2.0) - 0.089 0.300 0.270 0.250 0.350 **Grade Retention/Promotion** (1=not promoted to next grade ; 0=promoted to next grade) - 0.128 0.150 0.125 0.100 0.200 **Enrollment/Persistence** (1=no longer enrolled in school; 0=still enrolled in school) - 0.244 0.050 0.034 0.020 0.100 **Credits Earned** (1=insufficient credits earned ; 0=sufficient credits earned) - 0.128 0.150 0.125 0.100 0.200 **“On Track” at the End of Ninth**  **Grade** (1=off track at the end of ninth grade; 0= on track at the end of ninth grade) - 0.108 0.200 0.173 0.150 0.250 The treatment group mean proportions presented in Table 2 represent the smallest possible departure from the control group mean proportion ED’s contractor expects to observe with 72 schools, 150 students per school , and the assumed 95 percent plausible interval of school mean s in proportions. ED’s contractor conducted extensive sensitivity anal ys es examining how expanding the 95 percent plausible interval affects the number of schools needed to conduct the study . These anal ys es demonstrate that including 72 schools provides sufficient power for enrollment/per sistence and more than adequate power for other binary outcomes with minor deviations to the plausible interval. To determine the MDES for continuous outcomes (e.g., standardized achievement scores), ED’s contractor conducted a power analy sis using Optimal Design Plus Empirical Evidence (Version 3.0) ( Raudenbush et al. , 2011). Specificall y , ED’s contractor examined the MDES for a cluster randomized trial examining the impact on person - level outcomes (Level 1) with treatment at Level 2. These anal ys es conser vativel y assumed (a) 72 schools (36 treatment, 36 control); (b) 150 students per grade; (c) an intraclass correlation of 0.15 (based on prior research conducted by the research team examining these continuous outcomes) ; (d) a Level 1 covariate explainingREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 10 7 0 percent of the variation in the L evel 1 outcomes; 4 (e) an alpha of 0.05; and (f) power of 0.80. Based on these analy ses, this study is powered to detect a minimal detectable effect (MDE) of 0.15 for continuous outcomes as illustrated in Figure 1 . **Figure 1**  **. Minimal D**  **etectable E**  **ffect S**  **ize for C**  **ontinuous S**  **tudent -**  **L evel**  **O utcomes** 

ED’s contractor will also will examine two school - level outcomes, including data - informed allocation of dropout prevention interventions for students and school data culture. These anal ys es will be tested using intent - to - treat, linear regression models at the school level. Th e study is not sufficientl y powered to detect school - level outcomes with effect sizes less than or equal to 0.25. For example, using G*Power (Buchner, Erdfelder, & Faul, 1997) and assuming an alpha of 0.05, 0.80 power, and 72 schools, a linear regression m odel with four predictors (three blocking strata indicators and one treatment indicator) at the school level is powered to detect an effect size 0.85 or larger. Given this large MDES, school level anal ys es will be considered exploratory

anal ys es. 

### 2 **. Descr**  **iption of Procedures for**  **the Collection**  **of Information** ED’s contractor, REL Midwest , will manage data collection and ensure quality and timeliness. The draft materials and instruments for recruiting, screening and negotiating final memoranda of understanding (MOUs) for participation in the study are included in Attachments A - 1 through A 9. In addition, the draft instruments for data collection for the impact and implementations studies are included in Attachments B - 1 through B - 6 . The most common source of data for this study (including all estimates of the student - level impact) will involve extant data collected b y the state of Ohio, school districts, and schools. 4 Analyses of continuous outcomes include standardized achievement scores and GPA. These analyses will control for prior standardized achievement and/or GPA, which typically account for more than 70 percent of the variance in these respective outcomes. To ta l number of cl us te rs

E f f e c t S i z e

24 43 62 81 100

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 = 0.050 n = 150 P = 0 . 8 0 ,  = 0 . 1 5 , R2

L2 = 0 . 7 0REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 11 Primary data collection for the impact and implementation studies will only include a W eb - based school - level survey

to measure data - informed use of dropout prevention interventions for students and school data culture, and the treatment contrast. **Data Collection for School and District Recruitment** We will contact all schools that meet t he two inclusion criteria

— 150 or more Grade 9 students and graduation rates between 25 and 95 percent

— to determine whether they are currently

implementing or planning to implement any new programming that might conflict with or too closely mirror the EWIMS model. Only

schools that are not currently using or planning to use an early warning system (EWS) tool and process similar to EW IMS will ultimately be invited to participate in the study . We acknowledge that many

schools report that they

use some form of earl y warning system data (e.g., a competing EWS tool made available commercially or publically

or a student information system (SIS) used to identify

students at risk). The two main components of the early warning system process that exclude schools are f lagging and assigning students to interventions on the basis of sy stematic data review in more than one domain (attendance, discipline, and academics/failures) eithe r with or without a data team. Schools that have a sophisticated student information sy st em (SIS) but do not actually use the tool to flag or assign at - risk students to interventions will still be eligible. Also, schools that only flag students (generate student - level reports of risk) but do not assign students to interventions or monitor progress will still be eligible. Finally , schools that have teams that review data that are not specific to dropout prevention, (e.g., teams that meet about benchmark assessment scores or school improvement teams) will still be eligible to participate in the s tudy . Also, participation in the EWIMS impact study does not preclude schools from adopting new EWS - like programs during the course of the 1.5 - year implementation. The study will not impose any restrictions on the dropout prevention practices that make up the “b usiness as usual” control group. However, as an incentive for control schools to participate, the implementation team will provide them with the same implementation support in Year 3 as provided to treatment schools in Years 1 and 2. This delayed treatment will also help mitigate the risk that control schools will adopt an intervention similar to that being tested in this study . Expert panels convened in the technical working group (TWG) agree that this approach will limit the risk of control schools adopti ng similar EWS during the 1.5 - year implementation period because they are unlikel y to invest in a new tool during this time period when they are promised a complete tool and accompan yi ng technical assistance and professional development beginning in the 20 15

–2016 school year. The members of the REL Midwest Dropout Prevention Research Alliance will be activel y involved in recruitment. Our recruitment strategy will focus on both school districts and the high schools within districts. Recruitment will be faci litated by clear materials and communication about the nature of the study and the requirements for participation, including compl yi ng with random assignment. Based on prior experience recruiting schools for school - level random assignment studies (e.g., Ac cess to Algebra I: The Effects of Online Mathematics for Grade 8 Students, Teacher and Leader Evaluation Systems Study), the ED’s contractor does not anticipate random assignment to be a barrier to participation. To incentivize participation for controlREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 12 sc hools, the implementation team will provide them with the same implementation support in Year 3 as provided to treatment schools in Year 1. ED’s contractor will involve the state, districts, and alliance members in recruitment to garner local support for the study . In addition, to facilitate the transfer of extant data from school districts when necessary

(see Table 2), REL Midwest proposes to reimburse school districts for the time involved in compiling the data through a stipend of $720 per data pull per district. Compensating districts will ensure that all districts have ample time allotted for data compilation and transmission. 5 Based on the willingness of all invited schools and school districts to participate in similar projects currently underway

via RELMidwest 6 and continued interest expressed by

additional districts and schools, ED’s contractor anticipates that a wide variety of schools will be interested in receiving the EWIMS model as part of the study. The process for re cruitment is outlined in Table 3 . District and school recruitment will occur simultaneously . ED acknowledges that districts can act as a gateway

for access to interested and eligible schools; however, some schools (ty pically in smaller districts) may

also have the autonom y to agre e to participation without district approval. Therefore, in larger districts recruitment will begin at the district level and in smaller districts ED’s contractor may

first recruit schools and then follow up with districts when necessary . This process includes contacting districts to garner district buy- in for the study and prescreening interviews at the school level. The following sections describe more fully each of these steps in the recruitment process: (1) identifying the pool of schools to be screened , (2) contacting districts to gain district support for the study , (3) conducting school - level screening interviews, (4) prioritizing schools for recruitment, (5) recruiting eligible schools, and (6) negotiating final agreements. **Table 3**  **. Recruitment Proc ess and Respondent Universe for the EWIMS Impact Study**  **Steps in the Recruitment Process Respondent Universe** 

Identify pool of schools eligible to participate in the study 321 eligible schools in Ohio District first contact (e - mail and/or phone) 250 districts in Ohio Follow up for nonresponders (districts) 100 districts

5 ED’s contractor anticipates that it will take district staff approximately 25 hours each year to compile and transmit all required student - level administrative d ata, as described in the Data Sources section. The average salary of the responsible individuals is $40/hour, resulting in a stipend of $1,000. 6 ED’s contractor is currently conducting a series of interrelated projects for the Regional Educational Laboratory (REL) Midwest’s Dropout Prevention Research Alliance that examine different aspects of implementing an early warning system to achieve the Alliance’s shared goals of improving graduation outcomes and reducing persistent disparities in graduation and dropout rates among student subgroups. This work is primarily focused on the state of Ohio and includes projects focused on (1) validating early warning indicators; (2) providing technical assistance to support the implementation of early warning systems; and (3) the current study - evaluating the impact of using early warning systems on school and student outcomes.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 13 District screening 80 districts with eligible schools First school contact (e - mail and/or phone) 321 eligible schools in Ohio Follow -up for nonresponders 128 eligible schools in Ohio (~ 60% of those eligible at first contact) School screening/interview 160 most qualified schools (~ 50% of those eligible at first contact) School visit (face - to - face or virtual) 100 of the 160 highest qualifying schools that are interested in study participation Negotiate final agreements (district and school me moranda of understanding [MOUs]) 72 study schools Conduct random assignment 72 study schools Identify survey respondents within schools 72 school leaders Note. If there are issues with identifying and successfully recruiting schools that meet all of the specified qualifications, the ED’s contractor will consider relaxing the criteria for inclusion in negotiation with the Institute of Education Sciences (IES). **(1) Identifying the pool of schools to be screened.** To identify the pool of eligible and interested schools, the contractor will use the Common Core of Data to pinpoint schools within Ohio that serve Grades 9 through 12, have more than 150 Grade 9 students , and have graduation rates between 25 and 95 percent. ED’s contractor will then reach out to the estimated pool of 321 schools to identify

interest and the presence or absence of an earl y warning s ys tem similar to EWIMS. After the pool of schools has been selected based on the previously mentioned criteria, ED’s contractor will send an informational e - mail to each school (see Attachment A - 1 for the text of this e - mail). This e - mail will include notification that the school meets the initial eligibility requirements for participation in the study and will contain documents that briefly describe the study and a letter from IES endorsing the study (see Attachment A - 2 for these documents). After ED’s contractor sends the e - mail, a recruitment team member will c all each school to inform it about the study and ask the school to participate in a short telephone interview. The recruitment team comprises REL staff who will also subsequently serve on the study or implementation team following random assignment. **(2) C ontacting districts to gain district support for the study.**  ED’s contractor will identify the districts in which the 321 eligible schools are located, and begin a wide outreach plan that involves e - mails and follow - up phone calls to district - level staff. T his “first touch” of recruitment at the district level will facilitate our communications with interested and eligible schools (see Attachment A - 3 for the text of this e - mail). After the pool of districts that represent eligible schools has been identified , ED’s contractor will send an informational e - mail to each district (see Attachment A - 1 for the text of this e - mail). This e - mail will include notification that the district meets the initial eligibility requirements for participation in the study and wil l contain documents that briefl y describe the study and ED’s contractor and a letter from IES endorsing the study (see Attachment A - 3 for these documents). After the e - mail is sent, a recruitment team member will call each district to inform the district a bout the study

and ask the district to participate in a telephone interview. The district - level screening protocol that will be used to guide this interview is presented in Attachment A - 6.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 14 **(3) C onducting district** 

**- and school** 

**- level screening interviews.**  The district - and school - level screening protocols are designed to verify school size, graduation rate, and use of an earl y warning s ys tem. The interviews also allow for early termination of the interview if a school or all schools within a district do not me et these eligibility criteria . ED’s contractor will prepopulate the screeners for schools based on the Common Core of Data but will also seek to confirm these data are correct at the beginning of the screening interviews. ED’s contractor estimates that there are 250 districts with eligible high schools and 321 eligible high schools. Of those, ED’s contractor estimates that 68 districts and 128 schools will complete the screener (40 percent of contacted schools

— a reasonable response rate to the initial conta ct given ED’s contractor’s prior experience recruiting for randomized control trials). The items on the screening protocols are mapped to the constructs they measure (Table 4 ). The district and school screening protocols will be administered b y project sta ff b y telephone interviews and drafts can be found in Attachment A - 5 and A - 6.

**Table 4**  **. Mapping of Recruitment Screening Protocol Items to Topics School Screening Protocol**  **Construct Items**  School Characteristics 1

–4 Data - Driven Dropout Prevention Efforts 5

– 14 Data - Use Practices 15

–22 **District Screening Protocol Construct**  **Items** District and School Characteristics 1

– 3 Data - Driven Dropout Prevention Efforts 4

– 10 **(4) Prioritizing schools for recruitment.** Among the 128 schools that ED’s contractor expects to complete the screener, it is anticipated that about 110 will pass the initial screening based on extant data; some will be more appropriate candidates for the study than others. ED’s contractor will pri oritize its recruitment efforts in schools with:  Moderate graduation rates. Although man y high schools in Ohio have graduation rates above 90 percent, ED’s contractor will focus the recruitment efforts on the 173 schools (out of the entire pool of 321) th at have at least 150 Grade 9 students and graduation rates between 50 and 90 percent. Schools with lower graduation rates (i.e., 25 to 49 percent) may not have enough available interventions for students identified through the EWIMS process, and schools wi th higher graduation rates (i.e., 91 to 95 percent) may not have enough at - risk students to be included in the evaluation. For this reason, ED’s contractor will focus first on recruiting these schools which are most likel y to benefit from the EWIMS intervention andREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 15 secondaril y focus on those schools (with graduation rates between 25 - 49 percent or between 91 and 95 percent) that could benefit from EWIMS but that face additional challenges or have a more limited need for dropout prevention.  Greater interest in the study. Schools that signal to alliance members or ED’s contractor a greater interest to participate will be given higher priority . For example, some schools will make a greater effort to include multiple administrators in initial meetings to discuss the opportunity

and/or may

be more responsive to e - mail and phone communication from ED’s contractor than other schools. Whenever possible, ED’s contractor will also incorporate its prior work and knowledge of specific districts to gauge the interest leve l of districts.  Feasibility of implementation . Districts that have fewer competing initiatives for dropout prevention programming, and specifically the use of an earl y warning s ys tem, will be given higher priority . ED’s contractor will also determine wheth er schools have the capacity to assemble an EWIMS team to upload and anal yz e student - level data and risk indicators. Considering these criteria while canvassing the total 321 eligible schools will allow REL staff to prioritize the schools during recruitment and help ED’s contractor efficiently use resources to conduct site visits at eligible and interested schools. **(5) Recruiting eligible schools.**  Once ED’s contractor has finalized the pool of potentially eligible schools, face - to - face recruitment site visi ts with key staff will begin. ED’s contractor will begin recruitment b y targeting schools that are considered most eligible given the three priority areas listed above. The recruitment team members, with input from district personnel, will either organize meetings for clusters of schools or will visit each eligible and potentially interested campus separately . A school’s eligibility

to participate in the study will be reconfirmed in a meeting with the principal by

using the criteria listed in the previous s ection (size, graduation rates, use of an EWS tool and sy stem, and interest). The implementation team’s experience with implementing EWIMS has shown that an important benefit of the study to schools is not just the EWS tool but also the supportive technical assistance to implement the EWIMS process. Although treatment schools will receive these supports in 2014, control schools too will receive these supports beginning in fall 2015. Therefore, each school that agrees to participate will receive the full EWIMS implementation (tool, technical assistance) at no cost to the school. ED’s contractor believes this delayed treatment model will facilitate school recruitment. **(6) Negotiating final agreements.**  After successful recruitment meetings with the 72 study sc hools, ED’s contractor will then enter into final agreements through memoranda of understanding (MOUs) that detail the roles and responsibilities of study participation. Drafts of these final agreements are included in Attachments A - 5 and A - 6. If necessary , project staff will make additional phone calls or visits to build consensus and obtain commitment from principals. The principals’ signatures are expected to be gathered shortl y after the districts’ MOUs are obtained in order to allow random assignment o f schools immediately following OMB clearance. The timeline for recruiting schools is ambitious; however, ED’s contractor has substantial experience conducting large - scale successful recruitment efforts on extremely tight timelines.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 16 **(7) Conducting random a ssignment.**  Once the exclusion criteria are applied, schools will be assigned to treatment and control conditions within blocks based on graduation rates and school size. To conduct blocked random assignment ED’s contractor will create clusters of schools u sing matching techniques (e.g., Mahalanobis distance), potentially based on their graduation rates, size, district, and baseline data culture and will randomly assign half of the schools in each block to treatment status and half of the schools within each block to control status. Again the key variables to consider during blocking are:  graduation rates,  school size,  district,  and potentially

baseline data culture and/or the sophistication of schools’ baseline data sy stems. ED and its contractor will finalize the blocking approach post - recruitment and before random assignment. The matching techniques that will be used to conduct blocked random assignment may also take into account information about schools’ home district and baseline data capacity , using data collected during the screening process. **(8) Identifying respondents for the school surveys .**  After random assignment and during data collection, ED’s contractor will also strategicall y identify school leaders/administrat ors to complete the Web - based school survey s. In each school ED’s contractor will identify the most relevant individual to complete the school survey on data - informed use of dropout prevention interventions for students and school data culture. In treatment schools ED’s contractor will sample the lead individual from the EWIMS data team. In control schools ED’s contractor will recruit the principal, assistant principal, counselor, or other school leader, depending on their level of involvement with data use and dropout prevention initiatives in their school. **Data Sources for Impact and Implementation Studies**  To understand the effects of the EWIMS model and schools’ experiences with implementation, this project will consist of an impact study and an implementation study . The following sections describe the data collection for each of these separatel y . Table 5 provides a summary of each data element collected in the schools and associated data sources .REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 17 **Table 5**  . **Data Sources, Data**  **Elements, and**  **Timeline for Collection Data Source**  **Data Element(s) Timeline for Data Collection**  ODE **School Administrative Data** Baseline

–Jan. 201 4

**Student Administrative Data** Jan. 2014, June 2014 , Jan. 2015, June 2015 District **Student Administrative Data** Jan. 2014, June 2014 , Jan. 2015, June 2015 School Administrators **Annual Web -**  **Based Survey** May 2014, May 2015

**Implementation Case Study:**  **Principal Interview 1**  May 2015 School - L evel EWIMS Data Teams **Populated EWS tool**  **s** Jan. 2014, March 2014, June 2014, Aug. 2014, Jan. 2015, March 2015, June 2015 Participation in EWIMS trainings or convenings Jan. 2014, March 2014, June 2014, Aug. 2014, Jan. 2015, March 2015, June 2015 Satisfaction with EWIMS trainings or convenings Jan. 2014, March 2014, June 2014, Aug. 2014, Jan. 2015, March 2015, June 2015 EWIMS data team monthly logs Monthly from Jan. 2014

–June 2014; Aug. 2014

–June 2015 Interview of one EWIMS Team M ember **1** May 2014, May 2015 Note . Bolded data elements are data collection instruments involving burden of study participants. 1 . Items adapted from the California Comprehensive Assistance Center (CCAC). All student - level data will be collected through extant data. There is no student - level primary data collection burden on schools. As detailed in Table 5, stu dent - level extant data will be obtained from the Ohio Department of Education (ODE), districts, and schools. Protocols for transferring the data are outlined in the protocol in Attachment B - 3 . When feasible, student administrative data will be collected from ODE to reduce the burden on participating schools. In addition, ODE provides detailed guidance on the collection and reporting of many of the data elements through the Education Management Information Sy stem (EMIS). ODE’s EMIS Manual includes data defin itions, requirements, and procedures for reporting and transferring data to the department. For more information, see http://www.ode.state.oh.us/GD/Templates/Pages/ODE/ODEDetail.aspx?page=3&TopicRelationI D=367&ContentID=100905&Content=135642 . Because some of the outcome data (e.g., attendance, course performance) would be available only from the participating sc hool districts and schools (not the state), REL Midwest proposes to reimburse districts and schools for the time involved in compiling the data through a stipend of $ 720 per school, per data pull (with three scheduled data pulls

—June 2014, January 2015, an d June 2015). It is expected to take district staff approximatel y 16 hours in year 2 and 32 hours in year 3 to compile and transmit all required student - level administrative data, as described in Table 4. The average salary

of the responsible individuals i s $45 per hour, 7 thus resulting in a $720

7 The hourly wage rates used to calculate cost are based on previous OMB packages and are consistent with ED’s contractor’s knowledge of wage rat es at Ohio Department of Education.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 18 stipend in year 2 and a $1,440 stipend in year 3. It is anticipated that compensating schools will ensure that all schools have ample time allotted to data compilation and transmission. **Impact Study Measures** The impact study

will assess the effects of the EWIMS model on student outcomes, including indicators of student risk (e.g., attendance, number of course failures, credits earned, behavior). Other outcome measures include measures of student success that are not in the EWS tool, including performance on graduation tests, persistence and progress in school, and predicted probability

of on - time graduation. Data collection activities will focus on students in Grades 9 and 10 during Year 1 (2013 - 14) and Grades 9, 10, and 11 during Year 2 (2014 - 15). All variables for the impact study will come from school and district data sources. The impact study will obtain the same extant data for all participating schools and use these data to construct the outcome measures. **Base line Measures.**  In January 201 4 , at the beginning of implementation, RELMidwest will obtain student - level covariate data from ODE for students in treatment and control schools. Data will include demographic characteristics (e.g., race/ethnic i ty , gender, fr ee or reduced - price lunch [FRPL ], individualized education program [IEP] and English language learner [ELL] status, and parents’ education) ; prior academic achievement (e.g., GPA, course failures from the previous ye ar, state mathematics and reading scores ) ; and previous - year attendance rates. School - level baseline data from ODE will include high school graduation rates, average state achievement scores in mathematics and reading, and the percentage of FRPL students. **Outcome Measures.** The impact evaluation is designed to determine if the EWIMS model has an impact on student outcomes over time, in four domains: (1) student risk status , (2) scores on graduation tests, (3) persistence and progress in school, (4) predicted probability of o n - time graduation. The study will also examine if the EWIMS model has an impact on intermediate outcomes in schools , including the way s in which they allocate dropout prevention programming and their school data culture. The student - level outcome measures include the following :  Student Risk Status indicators are binary variables reflecting the presence or absence of Ohio validated risk indicators, including attendance (e.g., missing 10 percent or more of instructional time), course performance (e.g., one or more course Fs), GPA (e.g., 2.0 or lower), behavior 8 , and “ o n - track” at the end of 9th grade ( a composite measure that classifies students as on - track or off - track to graduate based on Grade 9 course credits earned and core course failures). The underlyin g cutoffs will be finaliz ed on the basis of the results of the REL Midwest Year 1 validation study ( 4.1.01 ) 9 in three districts that examines middle school

8 Although it is acknowledged that schools can vary in coding and recording behavior referrals and suspensions, these variables are often predictive of student graduation and/or dropout ( e.g., Balfanz, Herzog, & Mac Iver, 2007; Goldschmidt, & Wang, 1999; Mac Iver, & Mac Iver, 2010; Neild, Stoner - Eby , & Furstenberg, 2008). If the results of the Year 1 REL Midwest 4.1.01 study , Local Validation of Early Warning System Indicator Study , demonstrate inconsistency in r ecording and coding of behavior data, the evaluation team will consider dropping this as an outcome measure. 9 The cut offs for indicator flags for use in the EWIM S tool in the impact study will be based on locally validated indicators that are based on multiple districts in the state of Ohio. REL Midwest is currently conducting a validation study using data from four districts in Ohio. The cutoffs in the tool to flag a student as at - risk or not at - risk will incorporate findings from this initial study c onducted by REL Midwest in Ohio that validates early warningREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 19 and high school data from two recent graduating classes to identify indicators and indicator thresho lds that most accuratel y predict student graduation 10

. Treatment schools will be expected to work directly with these same data for implementation purposes on a monthly basis. These risk indicators will be available for all treatment students within the EWS tool . The evaluation contractor will calculate comparable indicator flags for students in control schools based on extant data from the state, district, and schools that are used to populated the EWS tool in treatment schools .  Scores on graduation tests include standardized test scores on the Ohio Graduation Test (OGT; available only

in Grade 10 in Ohio) b y content area.  Persistence and progress in school includes a measure of persistence reflecting whether students are enrolled or have withdrawn for reas ons other than transfer to another district, including dropping out. ED’s contractor will examine persistence on an annual basis. ED’s contractor will use state - level administrative data from Ohio to identify the pool of all possible codes for transfer and dropout. Because ED’s contractor will use state - level data that will be consistent across all districts and schools, this will reduce the variability in possible codes that identify

transfer and dropout . The second measure

—of progress

— reflects whether stu dents have been promoted from grade to grade each ye ar and how man y credits they have earned , and will be collected by school or district administrative data. This domain also includes students’ credit accumulation b y semester.  Predicted probability of on - time graduation will be a composite variable that incorporates data from multiple indicators of risk for each student in both control and treatment schools. This value is a model - based estimate of each individual student ’ s probability of on - time graduatio n that will be derived by appl yi ng the coefficient estimates as weights for each risk factor 11

. First, each student will be assigned a binary indicator of having or not having each indicator of risk. Next, the model - based estimate of the intercept, or avera ge logit, will be used as a starting point for each student. Next, the weight of each indicator 12 will be added for students with risk factors, yielding individual student logits for on - time graduation that are based on their numbers of risk factors. Finall y, logits will be transformed into a predicted probability

of on - time graduation for each student. For example, the equation for calculating the predicted probability

of on - time graduation for an individual student with the following indicators based on local Ohio data. The initial study for the Dropout Prevention Alliance was designed to validate indicators of student risk for academic failure. These analyses use historical student - level data from four Ohio districts to identify middle and high school indicators and indicator thresholds that most accurately predict student graduation outcomes (e.g., the number of 8th or 9th grade course failures that most accurately distingui shes graduates from non - graduates). 10 If the results of the REL Midwest Year 1 validation study ( 4.1.01) are not available in ample time to be used for these purposes, indicator flags and respective thresholds will be then based on the recommendations of the National High School Center. The National High School Center has synthesized the existing research on early warning system indicators in order to provide guidance and technical assistance on developing and implementing early warning systems in states, districts, and schools. 11 The coefficients will be derived from analyses conducted under the previously mentioned study conducted by REL Midwest in Ohio that validates early warning indicators based on local Ohio data. We will use the findings from the va lidation study to inform the weights for each indicator in the RCT. 12 If district - specific analyses are available for any schools that both participated in the 4.1.01 study and the randomized controlled trial , the contractor will utilize the most specific weights (coefficients) in the equations that predict each student ’ s probability of on - time graduation .REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—20 risk factors could be: Student Predicted L ogit = (4.1.01 Estimated Intercept) + (Binary Absence Flag * Absence Coefficient from 4.1.01) + (Binary Course Failure Flag * Course Failure Coefficient F rom 4.1.01) + (Binary

Behavior Flag * Behavior Coefficient F rom 4.1.01) + (Binary Algebra Failure Flag * Algebra Failure Coefficient F rom 4.1.01) The logits can then be transformed to predicted probabilities using the following equation: Predicted Pro bability

= (1 / (1+(ln( - logit)) Thus, each student would have an individual predicted probability of graduation that is based on multiple indicator flags and the relative weight of each flag derived from the anal ys es in 4.1.01. This predicted probability reflects the likelihood of graduation within four years based on continuous and dichoto mous measures of the indicator variables , and will be derived from the validation anal ys es conducted in 4.1.01. ED’s contractor will calculate predicted probabilities at the end of Grades 9 and 10 after Year 1, and at the end of Grades 9, 10, and 11 after Year 2. The predicted probability

will then be the outcome in a model that tests the impact of EWIMS on students’ lik elihood of on - time graduation. In addition to the student - level outcomes outlined above, the study will conduct impact anal ys es on the fol lowing school - level outcomes:  Data - informed allocation of dropout prevention programs and interventions will measure how schools use dropout prevention programs in both treatment and control schools. The goal of this measure is to tap the theorized interme diate school - level outcome of the EWIMS model, where data are used to inform which students need interventions and how to assign them to specific programs . ED’s contractor will collect these school - level data to inform the construction of this measure thro ugh annual Web - based survey s from a school administrator at each site identified b y the evaluation team during recruitment to be knowledgeable about the school’s dropout prevention and intervention services. ED’s contractor plan s to collect this informatio n at the school level rather than the individual student level because student level information about the ty pes of services and microinterventions in which students participate is not routinely recorded sy stematicall y in most schools . Of course, if the EW IM S model is implemented with fidelity in treatment schools, ED’s contractor will have these detailed student - level data for students in treatment schools and will leverage these data to answer implementation research question 3 (What are the specific inte rventions provided to students identified as “at risk” by the EWIMS model in treatment schools?) . However, assuming these data are otherwise not available for students in control schools, ED’s contractor will not attempt to collect them at the student leve l because this could potentially dilute the service contrast between the treatment and control conditions. 13 Therefore, ED’s contractor will rely

on school - level survey s to gauge the effects of the EWIMS model on whether and how both treatment and control schools provide dropout prevention supports for students. School staff who participate in the annual Web - based survey will receive a $3 0 stipend in the form of a gift card, which is aligned with Analytical and Technical Support for Advancing

13 That is, collecting this information for all students in control schools could constitute an intervention in itself.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—21 Education Evaluation: How to Put Together an OMB Supporting Statement , Appendix E (Sloan, Ingels, & Burghardt, 2012) . This $30 incentive applies to r esponses on one survey that measures three key

constructs, two of which are outcomes for the impact study (data informed allocation of dropout prevention programs and interventions and school data culture), and one of which is for the implementation study , described in the next section ( treatment contrast). The incentives will be administered twice throughout the study , accompan yi ng each administration of the survey ( May 2014, May

2015) . Attachment B - 1 includes a draft survey that will be developed to colle ct this information. A draft of the survey

will be included and ED will submit revisi o n s to OMB before the 30 - day review. The items are designed to solicit information about the ty pe and frequency of dropout prevention programs and the strategies to which students may be exposed within and outside of school. These items were based on the constructs presented in Dropout Prevention: A Practice Guide ( Dy narski et al., 2008) and Approaches to Dropout Prevention: Heeding Early W arning Signs w ith Appropriate Interventions (Kennelly & Monrad, 2007). These items inquired about students ’ access to online credit recovery , mentorship programs (including Check & Connect), tutoring, internship or school - related work prep programs (e.g., Job Corp s , MAA C Project, c areer and t echnical e ducation c lass es ), college prep programs (e.g., AVID, African American and Latino Student Summit, GEAR UP ), and school - and community- based extracurricular activities (e.g., sports, band, club s or organization s , student gov ernment). In this survey , the latter questions (items 2

– 6) inquire about schools ’ implementation of microinterventions , including the names of specific programs, participation rate s , grade level availability , and the student selection process. ED’s contra ctor will operationalize the data collected from the online survey into one summary variable that captures schools’ proclivity to engage in data - informed allocation of dropout prevention programs and interventions. First , ED’s contractor will creat e a scor e that summarizes the extent to which sc h ool s use data to assign students to different types of dropout prevention interventions and programs (as measured by the school survey), including:  T argeted academic interventions  T argeted behavior al interventions  Attendance/truancy

interventions  O nline content recovery programs  M entoring programs  Internship or school - related work - prep programs  C ollege prep ED’s contractor will test the impact of EWIMS on the proportion of students in a school th at participate in one of the above dropout prevention interventions or programs, as well as how often that school uses data to inform the allocation of dropout prevention interventions or programs. Table 6 provides a map of the items included in the draft protocol and the constructs measured in the survey found in Attachment B - 1.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—22 **Table 6**  **. Mapping Web -**  **B ased**  **S chool**  **S urvey Items to Constructs** 

**Data -**  **Informed Allocation of Dropout Prevention Constructs**  **Items** Academic interventions 1 Behavioral interventions 2 Attendance/truancy interventions 3 Online credit or content recovery 4 Student mentoring programs 5 Internship or school - related work preparation program 6 College preparation programs 7

 School d

ata c ulture is another school - level outcome hy pothesized to change as a function of implementing EWIMS. School d ata c ulture encompasses the following four key dimensions of school data use: 1. Context includes district and school goals for data use and the professional climate around data use in schools. 2. Supports for data use include the presence or absence of structured time to review data, training and professional development around data use, collaboration around data, data co aching, and principal leadership. 3. Working with data includes the frequency, duration, mode, and ty pe of data review. 4. Responses to data include how teachers and schools respond to student data , including the assignment of general and targeted interventions, determining teacher professional development needs, and informing school improvement plans. ED’s contractor will use the aforementioned annual Web - based survey to measure school data culture in both treatment and control schools. School administrators from treatment and control schools will be invited to respond to this school - level survey . This surv e y i ncludes items adapted from the Bill & Melinda Gates Foundation

– funded Urban Data Study (Faria et al., 2012). While the Faria et al. (2012) survey measured specific data use practices around benchmark assessments , ED’s contractor adapted these items for use with early warning sy stem data structures. Attachment B - 2 includes a draft protocol with specific items to be used in the EWIMS impact study . Table 7 provides a map of the items included in the draft protocol and the constructs measured in the survey found in Attachment B - 2 .REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—23 **Table 7**  **. Mapping Web -**  **B ased**  **S chool**  **S urvey Items to Constructs**  **School Data Culture Constructs**  **Items** Context 1 , 2 Supports for Data Use 3 - 11 Working with Data 12 - 15 Instructional Responses to Data 16 - 19 **Implementation Study Measures** The Dropout Prevention Research Alliance and practitioners more broadl y will benefit from a clear understanding of what it takes to implement the EWIMS model and, whether or not it shows an impact, from clear documentation about the implementation process. The implementation study

will focus on the following questions: (1) To what extent do treatment schools faithfully

implement the EWIMS model? ; (2) To what extent does business - as - usual practice in control schools include the use of data for identifying at - risk students (treatment contrast)? **;** (3) What are the specific interventions provided to students identified as “at risk” b y the EWIMS model in treatment schools (i.e., Step 5 of the EWIMS process)? ; and (4) What are the key components of high quality implementation? This information is of critical interest to alliance members who may use this information to inform the design or implementation of EWIMS models. **Measures of Fidelity .**  RELMidwest will collect EWIMS data tools from treatment schools twice ye arl y (January and June) as a primary means for assessing the degree to which schools are faithfully implementing the EWIMS model. The data to be collected to measure fidelity of implementation include 14 :  The extent to which schools upload data into the EWS tool (percentage of student demographic and administrative data records uploaded , measured by the EWS tool )  The extent to which identified stude nts have documented interventions recorded in the tool (measured by the EWS tool )  Participation in EWIMS trainings/convenings (measured by attendance sheets from trainings collected b y the implementation team and submitted to the evaluation team)  Satisfact ion with EWIMS trainings/convenings (measured by satisfaction survey s)  Way s in which each treatment school implements the intervention (measured by interviews with an EWIMS team member)

14 REL Midwest and REL Appalachia are working together to coordinate data collection across two studies that will document the implementation of EWS in two states (V irginia and O hio ) ; thus , the contractor has incorporated some of the REL Appalachia items into its survey of treatment contrast and the interviews protocols to use with EWIMS teams to document fidelity of implementation.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—24 To document implementation, the evaluation team will have access to p opulated tools and will determine whether student data are regularly loaded, and whether students identified as at risk within the EWS tool also have documented interventions recorded in the tool. ED’s contractor will calculate the percent age of students with complete data in the tool in comparison with the extant data available for those students collected from the state, district, and school. ED’s contractor will also collect information about participation in EWIMS - related trainings and convenings that will be used to address the first research question of the implementation study (T o what extent do treatment schools faithfully implement the EWIMS model? ). First , the implementation team will share attendance and sign - in sheets for all EWIMS trainings an d convenings. Second , the implementation team will facilitate the data collection of satisfaction survey s similar to the process b y which college professors receive course satisfaction survey s. One person attending the training will volunteer to collect al l satisfaction survey s ; place them in a self - addressed, stamped, and sealed envelope ; sign the back of the envelope ; and mail the satisfaction survey s to the evaluation team. The satisfaction survey s will be adapted from those developed b y the California C omprehensive Assistance Center (CACC), which documented implementation of an EWIMS pilot project conducted as a collaboration between the National High School Center and the California Department of Education . The satisfaction survey s will be administered after each technical assistance activity (i.e., technical and implementation trainings, community

of practice, on - site support, and responsive technical assistance), and will gauge participants’ satisfaction with the quality , relevance , and utility of the service provided. The satisfaction survey s are part of the ty pical implementation of the EWIMS intervention and therefore are not included in our estimates of burden for treatment schools. In addition, the evaluation team will collect qualitative informati on via interviews about the way s in which each treatment school implements the EWIMS model . T he interviews are part of the ty pical implementation of the EWIMS intervention and therefore are not included in our estimates o f burden for treatment schools . **Me asures of Treatment Contrast**  . To understand the extent to which business - as - usual practices in control schools include the use of data for identifying students at risk of dropping out , ED’s contractor will use the aforementioned Web - based survey to collect information about the presence or absence of an earl y warning data - based sy stem in control schools. In this survey , items will inquire about current usage of an EWS tool or other s ys tematic use of data to identify students who are at risk of academic failure, disengagement, and/or dropping out of high school. If schools are using data for these purposes, follow - up questions explore the ty pes of indicators used , including thresholds and time frames applied, ability to customize indicators, and who has access to the data. Attachment B - 1 includes a draft protocol of the items measuring the treatment contrast. Table 8 provides a map of the items included in the draft protocol and the constructs m easured in the survey found in Attachment B - 1 .REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—25 **Table 8**  **. Mapping Treatment Contrast**  **S urvey Items to Constructs** 

**Treatment Contrast Survey Constructs**  **Items** Use of an early warning system 1 , 2 Use of an EWS tool 3 - 5 Types of EWS indicators 6, 7 Application of EWS data 8, 9 Presence of an EWS or data use team 10 - 11 **Measures of Specific Intervention Information.** The EWIMS model is a school - level macrointervention, through which schools can use data to efficiently allocate dropout prevention and academic support interventions to targeted students. Its overall impact will be driven in part by the effectiveness of the specific interventions assigned to students and the appropriateness of the “fit” between student needs and assigned interventions. Al though ED’s contractor will not directly assess the “fit” between students and microinterventions, nor the impact of microinterventions on student outcomes, ED’s contractor will collect information on the specific microinterventions allocated to students i n treatment schools. To collect these data, ED’s contractor will use the populated EWS data tools from each treatment school to carefull y record the microinterventions assigned to students. Information on specific interventions assigned to students will on ly

be collected in treatment schools, as noted above in the description of school level measures of allocation of dropout prevention programming. However, ED’s contractor will collect a school - level measure of allocation of dropout prevention programming i n each control school to attempt to understand whether control schools offer students interventions . 15 **Estimation Procedures Student**  **- Level Impact Analyses.**  ED’s contractor will conduct student - level impact analy ses at the end of Year 2 16 that assess the effects of EWIMS on measures in the four student outcome domains

—indicator flags, academic achievement, persistence and progression, and predicted probability

of on - time graduation

— and ED’s contractor will conduct exploratory analy ses that examine the impact of EWIMS on subgroups of students and two school - level outcomes. These anal ys es align to the research questions about the impact of EWIMS as stated above. The anal yt ic strategy

for questions about impacts on student - level outcomes will compare students in schools randomly assigned to receive the EWIMS intervention with students in 15 Because asking school staff to document the student - level service contrast information for control schools is likely too much of a burden, and too similarly resembles the actual implementation of EWIMS, the contractor will only collect these data in control schools at the school level (through Web - based survey s of school administrators). 16 We will conduct the main impact analyses o n student outcomes at the end of the 2014 - 2015 school year. However, we also plan to conduct these same models as exploratory analyses at the end of Year 1 (summer 2014). These are not confirmatory analyses because we do not hypothesize an impact of EWIMS on student outcomes after only six months of implementation.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—26 schools in the control group. The data for this study are hierarchical (students are nested within schools); therefore, units at the same level are not statis tically independent. To allow the effects of student - and school - level factors to be modeled and adjustments made for the non independence of observations within clusters, this study will use hierarchical linear modeling to estimate the treatment effect on the student - level outcomes of interest. In all anal ys es of student level outcomes, ED’s contractor will estimate intent - to - treat two - level models, where students are the Level 1 unit and schools are the Level 2 unit. ED’s contractor will define the intent to treat student sample as “all students included enrolled on the 20th day of school.” Some students within the at - risk populations may not enroll during the first week of school, but do enroll within the first 20 school day s. This cut - off date is also po licy relevant because schools must report their total enrollment to the state on the 20th day of school to inform state and annual funding. For all of the binary

student - level outcomes, ED’s contractor will use a hierarchical generalized linear model with a Bernoulli sampling distribution and logit link function. For all continuous student - level outcomes, ED’s contractor will use hierarchical linear models. ED’s contractor will run models separatel y b y year and cohort. Most of our student level outcome variables are dichotomous. For models with binary outcomes ED’s contractor will use a logit link function to transform the dependent variable into the odds of achieving a particular outcome (i.e., being on track or off tra ck, persisting). ED’s contractor will assume a constant treatment effect across blocks and include dummy blocking variables at Level 2 (school graduation rate and school size dummies). 17 At the student level, ED’s contractor will include baseline student de mographic and prior achievement characteristics, including race/ethnicity , gender, FRPL , ELL, IEP status, and prior standardized reading and mathematics test scores. 18 The treatment indicator will be included at Level 2 to indicate that a school was randoml y assigned to the treatment group. These student - and school - level variables are expected to be correlated with the outcomes of interest, and will be used as covariates in all analy tic models for this study

to improve the precision of the estimates of impact. These models will estimate both probability and odds of showing a particular characteristic or indicator of risk for students in the treatment and control groups. **Exploratory Student Subgroup Analyses.** To test for a differential impact of EWIMS for subgroups of students, ED’s contractor will use the same impact models outlined above for the student outcomes, but additionally include a cross - level interaction between the treatment status of participating schools and an indicator for students in each subgroup tested. ED’s contractor will conduct these anal ys es for student impact research questions 1a through 1d. ED’s contractor will address four key

subgroups of students: students who are English language l earners (ELLs), students with Individualized Education Programs (IEPs), students who receive free or reduced price lunch (FRPL), and students who are flagged as at risk with one or multiple risk factors as calculated using administrative data from the participating schools during the semester preceding implementation (fall 2013). ED’s contractor will therefore categorize students as “initially at risk” (or not) based on data from the fall 2013 marking period preceding implementation and use this subgroup categorization throughout the remainder of the study for each cohort. ED’s

18 Because we will not impute missing data, student test scores will only be used as a covariate in impact analyses if missing data is less than five percent.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—27 contractor will not redefine this subgroup later, because changes in flagged status in subsequent semesters may

be a consequence of the impact of the intervention (e.g., students who are flagged receive high - quality micro - interventions that may decrease their likelihood of being flagged in later semesters). Because the theory

of action does not include a priori hy potheses about the differential impact of EWIMS on the four subgroups to be tested with four separate exploratory analy ses, ED’s contractor will adjust for multiple comparisons given the potential for alpha inflation. Specificall y, the p - value associated with the cross - level interaction coefficient in each of the four subgroup anal ys es must be less than .0125 to be considered statistically significant (adjusting for four post hoc exploratory tests). In addition, it is possible that students in one subgroup are also included in other subgroups tested under separate exploratory analyses. For this reason, ED’s contractor will report the number and percentage of students in each subgroup that are also included in the other three subgroups. Understanding the overlap across subgroups will be important for interpreting a differential i mpact of EWIMS on one or more subgroups of students. **Exploratory School**

**- Level Impact Analyses.**  To analyze the impact of EWIMS on school - level outcomes and answer research questions 3a and 3b, ED’s contractor will conduct intent - to - treat, linear regression models on the school outcomes of interest. These anal ys es will be treated as exploratory

because they are underpowered to detect effects and are not the primary focus of the EWIMS model. These models will include strata blocking variables and the indicator for whether a school was randomly

assigned to the treatment or control group. **Sensitivity Analyses.** ED’s contractor also will conduct a series of sensitivity analy ses to test the robustness of the observed estimates from the benchmark impact models. These anal ys es include fixed effects models with school dummy variables, models with no covariates, and models that use the continuous or ordinal data on which the binary indicators used in the primary anal ys es are based (e.g., using all values for GPA rather than the binary indicator of above 2.0 vs. 2.0 or lower). **Study Milestone Timeline** This array

of data will permit ED’s contractor to gather a deep understanding of program implementation and how the intervention differs from business as usual in control sch ools . The above data collections are specified in the following study timeline:REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—28 **Table 9**  **. Schedule of**  **Activities Activity**  **Expected Date** Draft Office of Management and Budget (OMB) package July 2013 Documentation of institutional review board approval April 2013 Submit 60 day FRN July 2013 Submit 30 day FRN September 2013 Draft proposal accepted by ED March 2013 Final proposal approved by ED July 2013 Expected OMB clearance data January 2014 Complete school recruitment January 2014 Obtain signed district/school memoranda of understanding from all participating schools January 2014 Complete random assignment of participating schools January 2014 Academic year (AY) 1, 2013 - 14 Collect baseline data from participating districts January /February 2014 Pilot the Web -based surv ey and interview protocols Ju ne 2014 Treatment schools implement EWIMS with Grades 9 and 10 January 2014

–June 2014 Collect EWS tool data from treatment schools (AY1, quarter 3) March 2014 Collect EWS tool data from treatment schools (AY1, quarter 4) Ju ne 2014 Collect end - of

- year student - level data from participating districts Ju ne 2014 Conduct interviews with EWIMS teams at treatment schools May 2014 Administer survey measure of treatment contrast at particip ating schools May 2014 Academic year (AY) 2, 2014 - 15 Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014

–June 2015 Collect EWS tool data from treatment schools (AY2, quarter 1) November 2014 Collect midyear student - level data from participating districts January 2015 Collect EWS tool data from treatment schools (AY2, quarter 2) January 2015 Collect EWS tool data from treatment schools (AY2, quarter 3) March 2015 Collect EWS tool data from treatment schools (AY2, quarter 4) Ju ne 2015 Collect end - of

- year student - level data from participating districts Ju ne 2015 Conduct interviews with EWIMS teams at treatment schools May 2015 Administer survey measure of treatment contrast at participating schools May 2015 Academic year 3, 2015 - 16 August 2015 Control schools implement EWIMS August 2015

–June 2016 Submit Making an Impact report (first draft) December 2015 Making an Impact report accepted by ED TBDREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

—29 

### 3 **. Description of Procedures for**  **Maximiz ing**  **Response Rates** ED is committed to obtaining complete data for this evaluation. Because the evaluation relies heavily on administrative data, the ED’s contractor does not expect response rates to be an issue and anticipate s a 95 - percent response rate for district/school and state archival record requests. A key to achieving complete administrative data is tracking the data components from ODE and schools/districts with e - mail and telephone contact to the appropriate parties to resolve issues of missing or delayed data file s ( s ee the protocol for collecting extant data in Attachment B - 3 ) . Also, b ecause ED’s contractor will have direct access to EWS tool s in treatment schools , response rates will be 100 percent for all applicable data. The completeness of those data will vary , however, by the degree to which treatment schools use and implement EWIMS. All administrative data files and direct instrument responses received as part of this evaluation will be reviewed for consistency and completeness . I f a data file has too many mi ssing values or if an instrument in the implementation study has too few items completed to be counted as a response, AIR staff will seek to obtain more complete responses by e - mail or phone. Based on its extensive experience with administering survey s in a variety of schools, districts, and states, ED’s contractor expect s the response rate for the survey to be at least 85 percent . Although ED’s contractor expect s high response rates for the unique data collections related to this evaluation (as schools volunteer for this study in order to receive EWIMS for free and instruments are developed with sensitivity to respondent burden), nonresponse follow - up will be performed to ensure adequate response rates. Follow - up reminders and phone calls will be placed to individual respondents in the event that responses are not obtained for W eb - based survey s, and AIR staff will follow up directly with respondents in the event that face - to - face interviews or reflection interviews are not cond ucted . Additional steps will be taken to maximize response rates for the data collected for this study . For example, s ampled respondents will receive advance communications explaining their role and an assurance of confidentiality . Respondents also will b e given a contact number to reach ED’s contractor with questions and will be informed of the incentive that they will receive (on top of the incentive received by

all control schools) . For example, the study plan includes the following incentives:  $2, 880 stipend ( $1,440 in year 1 and $1,440 in year 2) per district for time involved in compiling outcome data across four extant data collections ( January 2014, June 2014, January 2015, June 2015) 19  $30 gift card for school administrators who participate in the annual Web - based survey (May 2014, May

2015)  $30 gift card for interview participants from EWIMS data teams at each of the 36 treatment schools (May 2014, May 2015)

19 It is expected to take district staff approximately 32 hours in year 2 and 32 hours in year 3 to compile and transmit all required student - level administrative data. Estimated stipends assume that the average salary of the responsible individuals is $45 per hour.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 30 Finally , ED’s contractor will work with IE S to determine the best anal yt ic approach for ad dressing missing data on key outcomes and covariates in impact models estimating the effect of EWIMS on student and school outcomes. In the proposed anal yt ic approach, ED’s contractor will drop any

covariates from impact models that have greater than 5 per cent missing data and will list wise delete cases with missing data on outcome measures. Upon collecting the data, ED’s contractor will conduct anal ys es examining the percent age of missing cases on key outcomes and covariates and work with the IES to deter mine whether list wise deletion is the best anal yt ic approach or whether ED’s contractor should impute missing data (using mean imputation or multiple imputation) to minimize bias associate d with attrition or nonresponse. 

### 4 **. Description of**  **Tests ,**  **Procedures ,**  **and Methods** Because all student - level outcome data used to estimate the impact of EWIMS on students will come from state - , district - , or school - level administrative records and /or directly

from the EWS tool , there will be no need to test pro cedures or instruments. However, ED’s contractor will invite internal staff at AIR with expertise in leveraging partnerships with states and districts to collect extant data to confirm that the data collection protocols are appropriate and do not pose any unnecessary burden on state or district staff. To measure school - level outcomes, including data - informed use of dropout prevention interventions for students, and school data culture , ED’s contractor will use a W eb - based survey . The survey

items will be r eviewed b y AI R colleagues who were formerl y employed as teachers and/or school administrators or colleagues with content expertise. These “critical colleagues” will look for three things during their review: (1) whether the questions asked are clear, under standable/ free of research jargon, and answerable; (2) whether the questions actually assess the intended constructs (e.g., the degree to which schools are implementing EWIMS as intended or implementing an EWIMS - like intervention); and (3) whether the number and type of questions are suitable for a 60 - minute survey (e.g., not redundant, focused enough to solicit clear answers). Ne x t , ED’s contractor will pretest the W eb - based survey with five school leaders who are currentl y implementing EWIMS in their schools. ED’s contractor will recruit these five participants f

rom the network of schools currently implementing EWIMS. ED’s contractor will ask participants to complete the W eb - based survey using a cognitive think - aloud protocol to pilot the items and correct any issues with item sets before administering the first wave of survey s in spring 2014. School - level staff who participate in the pilot testing will receive a one time $30 stipend in the form of a gift card , which is aligned with Analytical and Technical Support for Advancing Education Evaluation: How to Put Together an OMB Supporting Statement , Attachment E. The implementation also involve s face - to - face data collection through interviews with the data team . ED’s contractor will again utilize its network of experts within AIR , and the interview protocols will be reviewed by these critical colleagues for clarity , face validity of questions, and brevity . ED’s contractor will also pilot the interview protocol s with five teachers/school leaders who are currently

implementing EWIMS in their schools. Again, ED’s contractor will recruit five participants form the network of schools currently implementing EWIM S . ED’s contractor will ask participants to complete the i nterview and provide feedback on the length andREL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 31 appropriateness of the items to correct an y issues with item sets before conducting the first interviews . School - level staff who participate in the pilot testing will receive a one - time $30 stipend in the for

m of a gift card, which is aligned with Analytical and Technical Support for Advancing Education Evaluation: How to Put Together an OMB Supporting Statement , Attachment E . 

### 5 **. Contact Information for**  **Statistical Consultants**  ED’s contractor consulted the following experts internally within AIR concerning the methodology , study

de sign, and data collection approa ch/burden :  Dr. Hans Bos, Vice President, AIR (650 - 843 - 8110)  Dr. Chris Brandt, Senior Researcher, AIR ( 630 - 649 - 6649 )  Dr. Dean Gerdeman, Principal Res earcher, AIR (202 - 403 - 6223)  Dr. Mengli Song, Principal Researcher, AIR (202 - 403 - 5267) The following individuals were consulted on the statistical, data collection, and anal yt ic aspects of the EWIMS evaluation study through REL Midwest’s Technical Working Group (TWG) . **Margaret Burchinal, Ph.D.** Senior Scientist Frank Porter Graham Child Development Institute Research Professor, Department of Psychology Adjunct Professor, Department of Biostatistics University

of North Carolina 515 Oakcrest Drive Chapel Hill, NC 27516 - 9638 Ph: 919 - 966 - 5059 Fax: 919 - 962 - 5771 E - M ail: burchinal@unc.edu **Thomas Cook, Ph.D.** Joan and Sarepta Harrison Chair in Ethics and Justice Professor of Sociology , Psychology , Education and Social Policy Faculty Fellow, Institute for Policy Research Northwestern University 2040 Sheridan Road Evanston, IL 60208 Ph: 847 - 491 - 3776 E - M ail: t - cook@northwestern.edu

**Sara Goldrick -**  **Rab, Ph.D.**REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 32 Associate Professor of Educational Policy Studies and Sociology Senior Scholar, Wisconsin Center for the Advancement of Postsecondary Education University

of Wisconsin 211 Education Bldg. 1000 Bascom Mall Madison, WI 53706 Ph: 608 - 265 - 2141 E - M ail: srab@education.wisc.edu **Larry Hedges, Ph.D.** Board of Trustees Professor of Statistics and Social Policy Faculty Fellow, Institute for Policy Research Northwestern University 2006 Sheridan Road, EV 4070 Evanston, IL 60208 Ph: 847 - 491 - 8899 E - M ail: l - hedges@northwestern.edu **James J. Kemple, Ed.D.** Executive Director Research Alliance for New York City Schools Research Professor, Steinhardt School of Culture, Education, and Human Development New York Universit y 726 Broadway , 756 New York, NY 10003 Ph: 212 - 998 - 5463 Fax: 212 - 995 - 4049 E - M ail: james.kemple@ny u.edu **Brian Rowan, Ph.D.** Burke A. Hinsdale Collegiate Professor and School of Education Research Professor Institute for Social Research University

of Michigan 610 E. University

Ave. Room 4112 Ann Arbor, MI 48109 - 1259 Ph: 734 - 615 - 0286 E - M ail: brow@umich.edu **Barbara Schneider, Ph.D.** John A. Hannah Chair and University Distinguished Professor College of Education and Department of Sociology Michigan State University Erickson Hall 620 Farm Lane, Room 516REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 33 East Lansing, M I 48824 Ph: 517 - 432 - 0188 E - M ail: bschneid@msu.edu ED’s contractor also consulted with the RELMidwest Dropout Prevention Research Alliance members listed below to gather feedback on the design and measures to be used in the study .  Cherie Mourlam, Assistant Superintendent, Washington Local Schools (419 - 473 - 8222)  Sue Zake, Exe cutive Director, Ohio Department of Education (419 - 72 0 - 8999)  Amy Szy manksi, Consultant, State Support Team Region 1 (419 - 833 - 6771)  Melissa Ramirez, Assistant Principal, Findlay City Schools (419 - 425 - 8257)  Jay Wollenburg, Principal, Ohio Virtual Academ y (86 6 - 339 - 9071)  Mike O’Shea, Springfield High School (419 - 867 - 5633)  Laurie Krusz ynski , Data Coordinator, Scott High School, Toledo Public Schools (419 - 671 - 4000 )  Jeremy

Herr, Principal, McComb High School, McComb L ocal Schools (419 - 293 - 3286) Key s taff that will be directly involved with the data collection and anal ys es include:  Dr. Jessica Heppen, Principal Researcher, AIR (202 - 403 - 5488)  Dr. Mindee O’Cummings, Principal Researcher, AIR (202 - 403 - 5254)  Dr. Ann - Marie Faria, Senior Researcher, AIR (202 - 403 - 5356)  Dr. Nicholas Sorensen, Senior Researcher, AIR (312 - 283 - 2318)REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 34 

## References Allensworth, E., & Easton, J. (2005). The on - track indicator as a predictor of high school graduation. Chicago: Consortium on Chicago School Research. Retrieved from track http://ccsr.uchicago.edu/publications/track - indicator - predictor - high - school - graduation . Allensworth, E., & Easton, J. Q. (2007). Wh at matters for staying on - track and graduating in Chicago public high schools: A close look at course grades, failures and attendance in the freshman year. Chicago: Consortium on Chicago School Research. Retrieved from http://ccsr.uchicago.edu/publications/what - matters - staying - track - and - graduating - chicago public - schools Balfanz, R., Herzog, L., & Mac Iver, D. J. (2007). Preventing student disengagement a nd keeping students on the graduation path in urban middle - grades schools: Early identification and effective interventions. Educational Psychologist, 42 (4), 223

–235. Bloom, H. S., Hill, C., Black, A., & Lipsey , M. W. (2008). Performance trajectories and p erformance gaps as achievement effect-size benchmarks for educational interventions. New York: MDRC. Dynarski, M., Clarke, L., Cobb, B., Finn, J., Rumberger, R., & Smink, J. (2008). Dropout prevention: A practice guide (NCEE 2008 - 4025) . Washington, DC: U.S . Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. Retrieved from http://ies.ed.gov/ncee/wwc/pdf/practiceguides/dp_pg_090308.pdf Faria, A.M., Heppen, J., Li, Y., Stachel, S., Jones, W., Sawyer, K., et al . (2012). Charting success: Data use and student achievement in urban schools. Washington, D.C.: Council of the great c ity

schools. Goldschmidt, P, & Wang, J. (1999). When can schools affect dropout behavior? A longitudinal multilevel analy sis. American Educational Research Journal , 36, 715 - 738. Kennell y, L., & Monrad, M. (2007). Approaches to dropout prevention: Heeding e arly warning signs with appropriate interventions. Washington, DC: American Institutes for Research , National High School Center. Retrieved from http://www.betterh ighschools.org/docs/NHSC_ApproachestoDropoutPrevention.pdf Mac Iver, M. A., & Mac Iver, D. J. (2010). Keeping on track in ninth grade and beyond: Baltimore’s ninth graders in 2007

– 08 . Baltimore: Baltimore Education Research Consortium. Retrieved from http://baltimore - berc.org/pdfs/3A%20Final%20report_06 - 15 - 10.pdf Neild, R. R., & Balfanz, R. (2006). An extreme degree of difficulty: The educational demographics of the urban nei ghborhood high school. Journal of Education for Students Placed at Risk 11 (2), 123

– 141.REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program

— 35 Neild, R. C., Stoner - Eby , S., & Furstenberg, F. (2008). Connecting entrance and departure: The transition to ninth grade and high school dropout. Education and Urban Society , 40 (5), 543

– 569. Silver, D., Saunders, M., & Zarate, E. (2008). What factors predict high school graduation in the Los Angeles Unified School District? ( California Dropout Research Project Report 14). Santa Barbara , CA : University of California. R etrieved from http://www.hewlett.org/uploads/files/WhatFactorsPredict.pdf Raudenbush, S.W., Spy

brook, J., Congdon, R., L iu, X., Martinez, A., Bloom, H., & Hill, C. ( 2011). Optimal Design Plus Empirical Evidence (Version 3.0) . Available at http://www.wtgrantfoundation.org/resources/consultation - service - and - optimal - design . Sloan, M., In gels, J., & Burghardt, J. (2012). Analy tical and Technical Support for Advancing Education Evaluation: How to Put Together an OMB Supporting Statement . Available at http://relpacific.wikispaces.com/file/view/OMB%20guidance%20compiled%205 - 30 - 12.pdf/350956206/OMB%20guidance%2 0compiled%205 - 30 - 12.pdf .REL Midwest Recruitment Materials

— 36 

## Attachment **A. Recruitment Materials** Attachments A - 1 through A - 9 include recruitment materials for distri cts and schools to be used in the EWIMS impact study . A - 1 includes the advance letter email text. A - 2 includes a 2 page study overview. A - 3 includes a letter from IES endorsing the study . A - 4 includes follow up text for schools that are non responsive during recruitment. The advance email letter, the 2 - page study overview and a letter from IES endorsing the study will be sent to principals at each school and superintendents at each district. A - 5 and A - 6 include screening protocols for school s and district s . These protocols will be administered to a relevant stakeholder within each school or district identified by an initial point of contact. For example, the study team will first contact principals at each school and superintendents at each district though these high - level administrators might request that another staff member (e.g., dropout prevention special ist) complete the screening interviews. A - 7 and A - 8 include memorandum of understanding that will be signed between districts and REL Midwest (at AIR) and schools and RELMidwest (at AIR) to formalize study participation and commitment. A - 9 includes a memo randum of understanding that will be signed between Ohio Department of Education and RELMidwest (at AIR)REL Midwest Recruitment Materials

— 37 

## A **- 1. Advance Letter (E**  **- Mail Text)**  Dear < principal or superintendent name> , I am contacting you on behalf of the U.S. Department of Education to di scuss an opportunity for **<district or school name> to**  **participate in a study that would provide access to**  **the National High School Center’s Early Warning and Intervention Monitoring System**  **(EWIMS) including training and technical support at no cost.** This project, conducted b y the Midwest Regional Educational L aboratory at American Institutes for Research (AIR), is aimed at preventing high school dropout by using data to identify

at - risk students early

in high school and assign struggling students to interv entions that get them back on track for eventual graduation. The study will examine the impact of implementing EWI MS on a host of critical student outcomes (validated indicators of dropout, course performance, persistence and progress in school and likelih ood of on - time graduation). In addition, the study will examine the impact of EWIMS on how schools make use of data to allocate limited dropout prevention resources. Attached please find a document that briefl y describes the study. Our initial records indi cate that yo ur district/school may

be eligible to participate in this Ohio - based study funded by

the U.S. Department of Education’s Institutes for Educational Sciences. I would like to schedule a brief phone call with you to tell you more about the study , ascertain your interest in taking part, and to discuss whether this might be a great opportunity for <district or school name> . **Would you be able to touch base briefly by phone on <day>, <month> <date>?** Please feel free to reach me b y email or at 312 - 283 - 2318. Thank you and I look forward to speaking with you! <name of project outreach recruiter> <name of project outreach recruiter> Project Outreach Earl y Warning and Intervention Monitoring System Study Midwest Regional Educational L aboratory American Institutes for Research 20 North Wacker Drive, Suite 1231, Chicago, IL 60606 t: 312.283.2318 | www.air.org

Per the policies and procedures required by the Education Sci ences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings across the sample and will not associate responses with a spec ific district or individual. ED’s contractor will not provide information that identifies you or your district to anyone outside the study team, except as required by la w. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.REL Midwest Recruitment Materials

— 38



## A **- 2. Study Overview**  **for Schools and Districts Purpose of the Study**  One in four students in the United States fails to graduate from high school and graduation rates are significantly lower for students who are racial or ethnic minorities, economically disadvantaged, migrant and/or limited English proficient, and receiving special education services. However, strong foundational research hig hlights critical indicators in ninth and tenth grade that powerfully predict whether students are “on track” for high school graduation. These indicators, which typically focus on student engagement measures (attendance) and course performance (grades, cre dits earned), can be used as part of an early warning system to flag at - risk students early, assign appropriate interventions, and get students back on track for graduation. As part of Ohio’s Comprehensive Continuous Im provement Plan, the state has committ ed, through adoption of the Ohio Improvement Process, to promoting student success through data - driven decision making, targeted programming, ongoing student monitoring, and evaluation of improvement process effectiveness. Use of an early warning system is one strategy that schools can use to address the Ohio Improvement Process. One such early warning system is the early warning and intervention monitoring system (EWIMS) developed by the National High School Center. The EWIMS model encourages educators to use data to consider and provide appropriate interventions to students, and provides a means for monitoring student progress over time. At the core of EWIMS is the use of simple tools that encourage educators to routinely examine indicators that identify s tudents as “off track” and take action. Despite increasingly widespread implementation of early warning systems by states, districts, and schools, there have been no rigorous studies testing the impact of using an early warning system to improve student o utcomes such as staying in school, progressing in school, and probability of graduating. There is also little research examining how using an early warning system can shape a school’s culture for data use including how schools allocate their limited dropout prevention resources. To address this gap, this study will provide the first rigorous test of the impact of an early warning system. This study, funded by the U.S. Department of Education’s Institutes of Educational Sciences, is being conducted by the Mi dw est Regional Educational Laboratory at the American Institutes for Research. The planned study will examine the impact of EWIMS on (1) student outcomes including student risk status for dropout, scores on graduation tests , persistence and progress in sc hool and likelihood of on - time graduation; and (2) school outcomes including how schools allocate dropout prevention interventions for students and their data - use culture. Participating in this high - profile, large scale study will give your school and district an opportunity to access these innovative resources at no cost and help inform educational policy in Ohio and at the federal level. 

## Study Design The study will take place in approximately 70 high schools in Ohio during the 2013

– 14 and 2014

– 15 school years. All schools that participate in the study will receive the early warning system at no cost. Participating schools will be assigned by lottery to receive access to the EWIMS model (including the tool and technical support for implementation) in the 2 0 1 3 - 201 4 school year or the fall of 2015 (following the completion of the study). Schools assigned to receive the EWIMS model in the fall of 2015 will continue “business as usual” practices to identify at - risk students and allocate dropout prevention resources through 2014 - 15. The study will examine student outcomes for all students in grades 9 and 10 during the 2013

– 14 school year and all students in grades 9 through 11 during the 2014

– 15 school year for schools that implement EWIMS in 2013 - 2014 and those that do not implement EWIMS during the study period. All student outcome data will be collected from school or district administrative data, the EWS tool or the Ohio Department of Education; therefore there will be no primary student data collection for this study. To underst and how EWIMS may impact schools, all participating schools will be asked to complete an annual Web - based survey about data use practices and schools im plementing EWIMS during the two - year study period may be asked to participate in interviews about their experiences using the tool.REL Midwest Recruitment Materials

— 39 

## EWIMS Intervention The EWIMS process is designed to i dentify students who are at risk of dropping out of school and s upport and mo nitor at - risk students through school - wide strategies and targeted interventions. EWIMS is curr ently in use in 67 districts in six states, and the tool has been dow nloaded more than 20,000 times from the National High School Center’s website. **EWS tool .**  At the heart of the EWIMS model is an early warning data tool used to flag students as “at risk” based on attendance, course performance (grades, credits, GPA), and behavior indicators. The tool enables users to identify students who are at risk of dropping out of school, record assignments to available interventions, and monitor students’ response to those interventions. The Excel based tool provides a number of reports accessible to users, including:  **School -**  **Level Reports:** Graphical reports that show trends in student risk status across the school  **Student -**  **Level Reports:** Lists of students, grouped by indicators of risk and/or assigned intervention programs, that include summary information  **Detailed Student Reports:** Simple reports that can be generated for each individual student and show student information, indicators of risk status, and any assigned interventions  **Student -**  **Level Intervention Summary Reports:** Lists of students, their indicators of risk (flagged or not flagged), and the number and types of interventions to which each student was assigned **EWIMS Implementation Process.** In addition to the tool, the National High School Center has devised a seven - step EWIMS implementation process to support im plementation. The process guides users to make informed decisions about how to use data to support at risk students and how

to continue to monitor thei r progress over time. In addition to focusing on individual students, the process guides users to examine the success of specific supports or interventions and to examine possible systemic issues (school climate) that may relate to dropout trends. **EWIMS Seven**  **- Step Implementation Cycle** 



## Initial Criteria for Participation The study will include approximately 70 high schools in Ohio. To qualify, schools must (1) have at least 150 ninth grade students; (2) a graduation rate between 25 and 95 percent and (3) not already be implementing an early warning system tool for using data to flag at - risk students. 

## Study Timeline Through January 2014 , the study team will discuss participation with districts and schools that meet the initial criteria and conduct on - site or virtual meetings with school principals, guidance counselors and dropout prevention coordinators. Participating schools will sign memoranda of understanding in January 2014 and be assigned by lottery to receive access to the EWIMS model in January 2014 (treatment group) or the 2015 - 2016 school year (“business as usual” control group). Training for EWIMS implementation in treatment schools will begin in January 2014 . Data collection activities will continue through but not beyond the 2014

– 15school year. 

## F **or Additional Inform ation**  For more information about the study, or to begin a conversation about partnering on this project, contact **Dr. Nicholas Sorensen (nsorensen@air.org or 312**  **- 283**  **- 2318)**  or **Dr. Mindee O’Cummings (m**  **ocummings@air.org or 202 -**  **403 -**  **5254). STEP 1**  **— Establish**  **Roles & Process**

**STEP 2 —**  **Use the EWS tool** 

**STEP 3 —**  **Analyze EWS data** 

**STEP 4 —**  **Interpret EWS data**  **STEP 5 —**  **Assign & Provide**  **Interventions STEP 6**  **— Monitor**  **Students STEP**  **7 —**  **Evaluate & Refine**  **EWIMS**REL Midwest Recruitment Materials

—40 

## A **- 3. Letter From IES Endorsing the Study** 

**UNITED ST AT**  **ES DEPA RTMENT OF EDUCA**  **TION** INSTITUTE OF EDUCATION SCIENCES National Center for Education Evaluation and Regional Assistance 

## Month **XX , 2013**  Dear District/Principal, I am writing to introduce you to a new project funded by the U.S. Department of Education (ED) through the Regional Educational Laboratory (REL) program called the Early Warning System and Intervention Monitoring System (EWIMS) study. This project is aimed at preventing high school dropout by using data to identify at - risk students early in high school and assigning students to interventions that help get them back on track for on - time graduation. The study will be conducted by REL Midwest at American Insti tutes for Research (AIR). For this study, ED is seeking more than 70 high schools in Ohio to participate. In addition to being part of a high -profile, innovative study, participating districts will receive free access to the EWIMS model, developed by the N ational High School Center at AIR, including the tool and training and technical support for implementation. Improving the state’s current graduation rate is a focal point in Ohio’s Comprehensive Continuous Improvement Plan and the state has committed, thr ough adoption of the Ohio Improvement Process, to promoting student success through data - driven decision making, targeted programming, ongoing student monitoring, and evaluation of improvement process effectiveness. In that vein, Ohio districts and schools are increasingly interested in using an early warning system to identify students who are off track for graduation as early as possible. For this reason, the REL Midwest formed the Dropout Prevention Research Alliance, focused on improving graduation outc omes and reducing persistent disparities in graduation and dropout rates among student subgroups in Ohio. The goal of the alliance is to expand the implementation of early warning systems in Ohio districts and to conduct an efficacy study testing their imp act on student outcomes and school processes. The proposed study is a two - ye ar randomized controlled trial (RCT) to examine the impact of implementing an early warning system on school processes and student outcomes. The project responds to a need expresse d by members of the REL Midwest’s Ohio Dropout Prevention Alliance for clear information about the efficacy of early warning systems. On behalf of the U.S. Department of Education, I encourage your participation, as many districts and schools are needed t o generate clear and usable results from this evaluation. Thank you in advance for your consideration. If you have any questions about the study, please feel free to contact AIR’s project directors, Dr. Ann - Maria Faria ( afaria@air.org , 202 - 403 - 5356) or Dr. Nicholas Sorensen ( nsorensen@air.org , 312 - 283 - 2318). In addition, if I can provide any assistance to you in this matter, please feel free to contact me by phone at 202 - 219 - 1674 or email at Chris.Boccanfuso@ed.gov . Thank you for your time, and we look forward to speaking with you. Sin cerely, Chris Boccanfuso, Ph.D. Contracting Officer’s Representative, REL MidwestREL Midwest Recruitment Materials

—41



## A **- 4. Nonresponse Follow**  **- Up (E**  **- Mail Text)**  Dear < principal or superintendent name> , I am trying to reach you on behalf of the U.S. Department of Education to discuss an opportunity for **<district or school name> to**  **participate in a study that would provide access to**  **the National High School Center’s Early Warning and Intervention Monitorin**  **g System (EWIMS) including training and technical support at no cost.**  This project, conducted by the Midwest Regional Educational Laboratory at American Institutes for Research (AIR) and funded by the U.S. Department of Education’s Institutes of Educatio nal Sciences, is aimed at preventing high school dropout by using data to identify at - risk students early in high school and assign struggling students to interventions that get them back on track for eventual graduation. **The timeline for recruiting dist ricts and school to participate in this opportunity is extremely tight.**  **Would you be able to touch base briefly by phone on <day>, <month> <date> to discuss whether**  **this might be a good opportunity for <district or school name>?**  Please feel free to reach me by email or at 312 - 283 - 2318. Thank you and I look forward to speaking with you! <name of project outreach recruiter> <name of project outreach recruiter> Project Outreach Early Warning and Intervention Monitoring System S tudy Midwest Regional Educational Laboratory American Institutes for Research 20 North Wacker Drive, Suite 1231, Chicago, IL 60606 t: 312.283.2318 | www.air.org

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings a cross the sample and will not associate responses with a specific district or individual. ED’s contractor will not provide information that identifies you or your district to anyone outside the study team, except as required by la w. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.REL Midwest Recruitment Materials

—42



## A **- 5.**  **Telephone School Screening Protocol/Interview** 

**Paperwork Burden Statement** According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is **XXXX -** **XXXX** . The time required to complete this information collection is estimated to average 30 minutes per response . This information collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202

– 4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Christopher Boccanfuso, U.S. D epartment of Education, Institute of Education Sciences, Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208 - 5500. 

## Draft School Screening Protocol **School Screening Protocol**  Intro ductory script: Thank you for the opportunity to meet with you today and discuss the Impact of an Early Warning and Intervention Monitoring System (EWIMS) on Student and School Outcomes Study funded by the Institutes of Educational Science within the U.S. Department of Education. We are very excited about your school’s interest in participating in the study and are looking forward to talking more about whether the study might be a good opportunity for your school. Participation in this study is voluntary for schools. Today we will first ask you questions about how your school wor ks with students who are at - risk for not graduating on time and then we have some questions about your school’s data use practices. By data use practices we are referring to things like the professional climate around using data, supports for data such as professional development and/or structured time to review data, frequency of data use and using data to track progress. School Name: Principal Name: Email: Telephone: **School Characteristics** 1. How many 9 th grade students do you anticipate having enrolled in fall 2013? 2. What was your school’s graduation rate for the 2012 - 13 academic year? (note - pre populated with ODE data if available) 3. What was your school’s graduation rate for the 2011 - 12 academic year? (note - pre populated with ODE data if available) 4. What methodology was used to calculate the graduation rate?REL Midwest Recruitment Materials

—43

**Data -**  **Driven Dropout Preventions Efforts** For each of the follow ing questions, interviewer will allow the sch ool interviewee to respond to the question and the interviewer will select most appropriate response option. 5. Does your high school have an early warning system tool that identifies students who may be at - risk of not graduating ? o Yes, we have an early warni ng system tool used to identify students who may be at - risk of not graduating in my high school. If yes , what is the name of the system or tool ?

o No, my high school does not currently have an early warning system tool to identify students who may be at - risk of not graduating in my high school. o I'm not sure. 6. I s your high school using the student information system (SI S) or another other data tool to identify students who may be at - risk of not graduating from high school on-time? o Yes, we have a tool to identify students who may be at - risk of not graduating in my high school. o No, my high school does not currently have a tool to identify students who may be at - risk of not graduating in my high school. o I'm not sure. If yes, please specify: If the school uses an early warning system , SIS or another other data tool to identify students who may be at- risk of not graduating 7. What types of data are used? For each indicator used, what is the level of evidence?

N/A Evidence -based (e.g.,

based on research conducted using data not related to the school) Locally - validated

(e.g., based on research using state, district or school data) Based on other criteria (specify) Attendance REL Midwest Recruitment Materials

—44

Behavior referrals  Behavior suspensions  Course performance  Credits earned  Grade Point Average (GPA)  State assessment results  Other:  Other:  Other:  Other:  8. How do you assign at - risk students to supports and/or interventions? (check all that apply) o Student - level indicator data (i.e., evidence or locally validated indicators) o Other student level data (e.g., demographics, student performance data ) Specify:

o Other types of data Specify:

o No data -based strategy to assign at - risk students to supports and/or interventions 9. How often does your school review data to identify students at risk of not graduating? o 4 times per year (once per quarter) o 3 times per year o 2 times per year o 1 time p er year o OtherREL Midwest Recruitment Materials

—45 Specify:

o I'm not sure. 10. Does your school have a team or group of individuals that focuses on students who are identified as at risk of not graduating from high school? o Yes, we have a dedicated school -based team. o Yes, we have a school -based team that focuses on students who are identified as at risk of not graduating from high school, but it is part of another team. o Yes, the district has a team that focuses on students who are identified as at risk of not graduating from high sc hool. o No, there is no team at the district or high school levels. o Other (please specify) 11. Do teachers in your school have access to student data that identifies students at risk of not graduating? o Yes, and all teachers access it regularly o Yes, and some teachers access it regularly o Yes, and some teachers access it occasionally o Yes, but very few teachers actually access it o Yes, but no teachers actually access it o No, teachers do not have access to this type of data o Other (please specify)

12. Please describe the nature of collaboration around student data in your school. Probes: o Are these collaborations generally formal (scheduled meeting times) or informal?

o Who participates in these meetings?REL Midwest Recruitment Materials

—46

o What is the focus of these meetings? (data analysis, sharing/discussing student work, lesson planning, ways to improve or modify instruction, focus on individual students, focus on problem content areas etc...)

o Wh at are the goals of these meetings (identify students who need additional support? identify school or class - level instructional issues? Create instructional plans)

o Do teachers review other data (e.g., interim assessment data, curriculum -based assessments, teacher - created assessments) along with data that identifies at students at risk of not graduating?

13. Does your school have a system to monitor students' progress in interventions/supports to wh ich they are assigned? o Yes (please specify)

o No o Unsure 14. Is there anything else you would like to share with us about how your school approaches dropout prevention interventions and strategies?

**Data -**  **Use Practic es**  Thanks for telling me about how your school identifies students who are at - risk and allocates dropout prevention services at your school. We would now like to talk more broadly about data use in your school. 15. Has your district or school set clear and co nsistent goals or expectations for data use?REL Midwest Recruitment Materials

—47

If yes: o What are these goals? (get a general idea of their understanding of district priorities)

o How have these goals been communicated to you? (Probe: see if they can share any sample documents to show communication?)

o Are there specific areas in which you would like additional guidance? If no: o What type of guidance would you like to receive from the district?

16. Has your school received any professional development on data use? (In - service, pre - service, ongoing coaching). If yes: o Who delivered the PD (Is this delivered by district and/or school staff)?

o Is the training mandatory?

o What has been the focus (content) of the professional development sessions?

o What was the duration of the PD? (one - time training? is ongoing training provided or available?)

o What is the mode of deli very of the PD? (Is it face to face? Online? video?)REL Midwest Recruitment Materials

—48

o What materials have been provided to support your/your staffs’ learning? (Are there any sample materials that we can take a look at?) If no: o What type of professional de velopment would be most useful?

17. Aside from professional development, has your district or school provided any resources/tools designed to facilitate the use of data at your school for activities such as accessing data, analyzing/interpretin g data, planning instructional responses, implementing instructional responses and/or monitoring data use? (tangible resources/products, staff [e.g. coaches, technical support staff etc...] , tools [e.g., templates, protocols, books etc...] funding) o Yes (please describe)

o No 18. Are there structures in place to assess how well your teachers are analyzing or interpreting student data? o Yes (please describe)

o No

19. D o you feel that the use of stud ent data has led to changes in teacher instructional practices? o Yes (please describe)

o NoREL Midwest Recruitment Materials

—49

20. Is there a formal feedback system in place to determine whether planned instructional responses are effective? o Yes (please describe)

o No

21. Have student data been used for any other school - wide data initiatives or activities (e.g., to inform school improvement, evaluate teacher performance, evaluate program/curriculum, make between or within school comparis ons)?

o Yes (please describe)

o No 22. Is there anything else you would like to shar e with us about your school’s data use practices?REL Midwest Recruitment Materials

— 50



## A **- 6.**  **Telephone District Screening Protocol/Interview** 

**Paperwork Burden Statement**  According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is **XXXX -** **XXXX** . The time required to complete this information collection is estimated to average 30 minutes per response . This information collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202

– 4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208 - 5500.



## Draft District Screening Protocol Intro du ctory script: Thank you for the opportunity to meet with you today and discuss the Impact of an Early Wa rning and Intervention Monitoring System (EWIMS) on Student and School Outcomes Study funded by the Institutes of Educational Science within the U.S. Department of Education. We are very excited about your district’s interest in the study and are looking forward to talking to you about potential participation of high schools within yo ur district. Participation in this study is voluntary for schools. Today we will ask you questions about how your district supports schools in working with students who are at - risk for not graduating on time. District Name: District Contact: Email: Telephone: **School Characteristics** 1. How many high school (serving grades 9 - 12) do you have in your district? What are the names of those high schools and their graduation rates for the 2012 - 13 academic year? (note pre populated with names, if available)

_____________________ 2. What methodology was used to calculate the graduation rate?

**Data -**  **Driven Dropout Preventions Efforts** 3. Does your district promote the use of a student information system, an early warning system (EWS), or another other data tool to identify students who may be at-risk of not graduating from high school on-time? [intervie wer will allow

the interviewee to describe their process and will select most appropriate response] o Yes, we have a district - wide tool to identify students who may be at-risk of not graduating and we use the tool in all schools.REL Midwest Recruitment Materials

— 51 o Yes, we have a district - wide tool to identify students who may be at-risk of not graduating and we use the tool in some schools. If yes, please specify which schools:

o No , the district does not promote and/or high schools do not use a tool to identify students who may be at-risk of not graduating. o I'm not sure. 4. Can you please describe the tool:

5. Does your district have an early warning system team or group of individuals that focuses on students who are identified as at risk of not graduating from high school? [interviewer will allow the interviewee to describe their process and will select most appropriate response] o Yes, the district has a team that focuses o n students who are identified as at risk of not graduating from high school. o No, there is no team at the district or high school levels. o Other If yes, specify who serves on this team:

6. Does your district have a policy or practice on how schools should assign at-risk students to supports and/or interventions?

Specifically, how is data used within this process (e.g. how often and to what extent?) Does this vary by school? If so, how ?REL Midwest Recruitment Materials

— 52

7. Does your district have a system for schools to monitor individual high school students' progress in interventions/supports to which they are assigned? o Yes o No o Unsure If "yes," please specify which high schools use it: 8. If your district monitors students' progress in interventions/supports to w hich they are assigned, what data are used in this process? [interviewer will allow the intervie wee to describe the ir process and will check all that apply] o Attendance o State assessment results o Course performance o Data from students' classroom teachers o Core course performance o Data from other adults who interact with the student (e.g., counselors, coaches, etc.) o Credits e arned o Behavior referrals o Behavior suspensions o Information from the at-risk (flagged) student o Information from at-risk (flagged) students' parent/guardian o Other If "other," please specify: 9. Is there anything else you would like to share with us about how your district approaches dropout prevention interventions and strategies?REL Midwest Recruitment Materials

— 53 

### A **- 7. Final Agreement Form (**  **School MOU)**  

## School Roles and Responsibilities: **Early Warning and Intervention Monitoring System Study** Dear <principal name>, On behalf of American Institutes for Research (AIR) (www.air.org ) and its partners, we welcome yo u to the Early Warning and Intervention Monitoring System (EWIMS) study . We are excited about this project. Its purpose is to examine the impact of EWIMS on (1) student outcomes including student risk status for dropout, scores on graduation tests , persistence and progress in school and likelihood of on - time graduation; and (2) school outcomes including h ow schools allocate dropout prevention interventions for students and their data - use culture. This document contains an overview of the study and a brief description of the intervention, followed by a description of the roles and responsibilities for your school and for the study team, including the benefits of participation, and the project timeline. Please review the contents of this document and sign the last page to indicate your agreement to participate. Return the signed last page to Dr. Nicholas Sore nsen (nsorensen@air.org , fax 312 - 288 - 7601). 

### Overview The primary

goal of the EWIMS study is to evaluate whether implementing an earl y warning and intervention monitoring sy stem for identifying students at - risk of dropping out and using this sy stem to assign students to dropout prevention interventions will improve student outcomes including student risk status for dropout, scores on graduation tests , persistence and progress in school and likelihood of on - time graduation. In addition, this study will also examine the impact of implementing EWIMS on school outcomes including how schools allocate dropout prevention resources for students and their data - use culture. Study results will yield valuable information for the state of Ohio and for districts and schools across the country about the viability

and benefits of using an early

warning sy stem to prevent high school dropout and help struggling students get back on track for eventual graduation. The study is being funded b y the U.S. Department of Education, Institute of Education Sciences, and will January 2014 through the spring of 2016. We look f

orward to working with <High School> as a partner in this project! 

### The EWIMS Model The EWIMS model, developed by the National High School Center at AIR, is a multistep process intended to encourage s ys tematic and comprehensive implementation within school s. The process is based on a combination of research on data use in schools and National High School Center’s experience working with states, districts, and schools implementing earl y warning s ys tems. At the heart of the EWIMS process is an early warning data tool used to flag students as “at risk” based on attendance, course performance (grades, credits, grade point average [GPA]), and behavior indicators. The tool enables schools to identify students who are at risk of dropping outREL Midwest Recruitment Materials

— 54

of school, record assi gnments to available interventions, and monitor students’ response to those interventions. Be yond the development of the data tools, the National High School Center has devised a seven step EWIMS implementation process to support implementation. The proc ess guides users to make informed decisions about how to use data to support at - risk students and how to continue to monitor their progress over time. In addition to focusing on individual students, the process guides users to examine the success of specif ic supports or interventions and to examine possible sy stemic issues (e.g., school climate) that may relate to dropout trends. **Figure A**  **- 1. Early Warning Intervention Monitoring System Implementation Process** 

As shown in Figure 1, the steps are intended to be cyclical. At the core of this data - driven decision - making process, the steps focus users on key indicators that identify

which students are showing signs of risk of dropping out of high school and guide us ers to go beyond the indicator data and other relevant information to connect at - risk students to dropout prevention or academic support interventions. **The EWIMS model does not prescribe specific interventions for schools to implement**  . Instead, the model i s designed allow schools flexibility to decide which interventions they

believe are most effective for their students’ needs. Ideally , the EWIMS model allows users to identify

students with accuracy

and provide supports and intervention of your school’s ch oosing to at - risk students, resulting in improved outcomes for students, including higher attendance rates and improvement in academic performance leading toward successful graduation. 

### This Study Despite increasingl y widespread implementation of earl y warning s ys tems by states, districts, and schools, there have been no rigorous studies testing the impact of using an earl y warning sy stem to improve student outcomes such as staying in school, progressing in school, and graduating. There is also little re search examining how using an earl y warning s ys tem can shape a school’s culture for data use

—increased data - driven decision making (assigning interventions **STEP 1 —**  **Establish Roles & Process** 

**STEP 2 —**  **Use the EWS tool** 

**STEP 3 —**  **Analyze EWS data** 

**STEP 4 —**  **Interpret EWS data**  **STEP 5 —**  **Assign & Provide**  **Interventions STEP 6**  **— Monitor**  **Students STEP**  **7 —**  **Evaluate & Refine EWIMS** REL Midwest Recruitment Materials

— 55

to students) and processes and professional development to support using data to improve teaching an d learning. This study will address these gaps and provide the first rigorous test of the impact of an earl y warning s ys tem. The study will:  **Identify a sample of eligible and interested schools in Ohio.** The study team will conduct outreach to schools that meet initial eligibility criteria to confirm eligibility and discuss interest in participating in the EWIMS study . The study will include approximately

70 high schools in Ohio. To qualify, schools must (1) have at least 150 ninth - grade students; (2) a grad uation rate between 25 and 95 percent and (3) not already be implementing an earl y warning s ys tem tool for using data to flag at - risk students.  **Use a lottery to randomly assign half of the participating schools to implement EWIMS in**  **January 2014**  **and the o ther half to implement in fall of 2015.**  Half of participating schools will be randoml y assigned to receive access to the EWIMS model in January 201 4 (including the tool and technical support for implementation) and the other half of participating schools w ill be randomly assigned to conduct “business as usual” for identifying at - risk students until the fall of 2015 when they will receive the same resources and supports.  **Implement EWIMS.** All schools will implement the seven - step EWIMS process either in January 2014 or the fall of 2015 (depending on random assignment by lottery). The seven steps for implementation are as follows: o Step 1

—Establish Roles and Process. In this step, the composition of the EWIMS team is established; 20 team members then determine fre quency and duration of meetings and develop a shared vision or focus of the team’s work. o Step 2

—

Use the EWS tool . In this step, the school - based EWIMS teams are trained on the use and purpose of the tool itself. This step also includes first customizing th e tool settings and importing the student demographic and initial administrative data and also ongoing refreshing of the administrative data in the tool and the running of automated and custom lists and reports available within the tool. o Step 3

—

Analyze EW S data . In Step 3, EWIMS teams focus their attention on student - and school - level data, based on the indicators available in the tool. This data review process is intended to identify areas of focus and further investigation. o Step 4

—Interpret EWS data . St ep 4 guides teams to bring in additional data (external to the tool) to provide more context and a fuller picture to inform the EWIMS team’s consideration of specific needs of individuals or groups of flagged students. Unlike Step 3, which is focused on th e indicator flags themselves (i.e., the data in the tool), this step addresses root causes of wh y students might be identified as at risk for one or more indicators. The implementation team will provide training to identify root causes that focus on acquiring additional formal (e.g., administrative records) and informal (e.g., from

20 The goal is not to create yet another “team” with functions that may or may not overlap with functions of other already existing teams. Rather, integration with existing team structures ( if functionally operational) is optimal.REL Midwest Recruitment Materials

— 56 teacher, famil y, and student) input. This training will occur face to face and will last for one hour. o Step 5

—

Assign and Provide Interventions. In this step, EWIMS team members make informed decisions about the allocation of available resources and strategies to support students identified as at risk of dropping out of high school. The EWIMS team matches individual students to specific interventions after having gathered informat ion about (1) potential root causes for individual flagged students (Step 4) and (2) the available dropout prevention and academic and behavioral support programs in the school, district, and community , which are locally determined. o Step 6

—Monitor Student s. In this step, EWIMS teams continue to examine student indicators at regular intervals to continually identify students who show signs of being at risk. The teams will use the same indicators to closely monitor already- identified students who were assign ed to interventions for progress in school and risk status. This step provides critical ongoing feedback about additional student - and school - level needs and apparent successes. o Step 7

—Evaluate and Refine the EWIMS. Through active and structured reflectio n, EWIMS team members assess whether students are responding to assigned interventions, revise their specific strategies or general approach as needed, and determine how resources are allocated to improve support for students. This step encourages EWIMS te ams to make course corrections to all parts of the EWIMS implementation. As implied by the cyclical depiction of the seven - step process, this step (as well as the other six) reflects an ongoing process of continual improvement.  **Evaluate the effects of EWI MS on student and school outcomes.**  The study will examine student outcomes for all students in grades 9 and 10 during the 2013

– 14 school ye ar and all students in grades 9 through 11 during the 2014

– 15 school year. All student outcome data will be collected from school or district administrative data, the EWS tool or the Ohio Department of Education. In addition, all participating schools will be asked to complete an annual Web - based survey about data use practices and schools randomly assigned to implement EWIMS in January 2014 may

be asked to participate in interviews about their experiences using the tool. 

### School Roles and Responsibilities We look forward to partnering with <High School Name> for this exciting project! More detailed information on the responsibilities of participating schools follows. Final determination of school eligibility

requires willingness to adhere to the study guidelines and responsibilities.  **Maintain a sustained commitment to participate in the study.** To evaluate the impact of EWIMS, it is critical that schools agree to adhere to the study guidelines and timelines associated with random assignment b y lottery to implement EWIMS in January 2014 or fall of 2015. All schools are expected to implement the E WIMS model as designed (including all seven steps of the implementation cycle).REL Midwest Recruitment Materials

— 57

 **Adhere to the results of the lottery/random assignment process.** It is essential that the groupings that result from the lottery remain intact over the course of the study . Sch ools assigned b y lottery to implement the EWIMS model in January 2014 will serve as the treatment group. Their counterparts assigned to implement EWIMS in fall of 2015 will serve as a control group between January 2014 through the spring of 2015. Schools randomly assigned to the control group must continue with “business as usual” practices for identifying at - risk students and assigning dropout prevention interventions until EWIMS implementation begins in the fall of 2015. Students in control schools should continue to receive an y s ervices that would be offered to them in the absence of the study . As for students in the “treatment” group, no “ty pical” services should be withheld. Please refer an y questions or concerns from parents or school staff about this to the study team.  **Participate in all data collection activities.** Participating schools should provide school level information including high school graduation rates, average state achievement scores in reading and mathematics and demographics (e.g., perc entage of students receiving free or reduced - priced lunch). Administrative student - level data collection in participating schools will focus on all students in grades 9 and 10 during the 2013

– 14 school year and all students in grades 9, 10 and 11 during the 2014

– 15 school year. The study team will obtain as much administrative data as possible from the Ohio Department of Education (ODE) and the school district. However, all participating schools should provide the study team access to the following adminis trative data for students should this data be unavailable through other sources: o Demographic information ( e.g., race/ethnicity, gender, free or reduced - price lunch [FRPL ], individualized education program [IEP], and English language learner [ELL ] status, and parents’ education) o Grade point average (GPA) o State test scores o Attendance rates o Course grades in core academic courses b y semester o Credits earned b y s emester o Disciplinary

information (e.g., suspensions) o Enrollment information (e.g. whether students are enrolled or have left school for reasons other than transfer to another district, including dropping out) o Grade promotion In addition to administrative records, one administrator at all schools should complete an annual Web - based survey assessing how sch ools use data to allocate dropout prevention resources to students. Finally , all schools assigned by lottery to the treatment group (implementing EWIMS in January 2014 ) should participate in data collection efforts focused on implementation. Specificall y, the project team will collect data on attendance and satisfaction withREL Midwest Recruitment Materials

— 58

EWIMS training sessions and meetings. EWIMS teams at schools assigned to the treatment group will be required to submit their EWS tool securely to the study team and may be asked to participate in interviews about their implementation experience. **EWIMS implementation** (either in January 2014 or fall of 2015) will require the following:  **Develop an EWIMS team within your school.** A diverse, well - informed, EWIMS team within your school is ess ential to the success of this process. The EWIMS team may be established as a new team or may build on or be integrated into existing teams (school improvement team, response to intervention team, student support team). It is not necessary to create an ent irely

new team for EWIMS work, but an existing team that takes on the responsibility to use the tool for dropout prevention efforts should include a broad representation of staff within the school (e.g., principals, representatives from feeder elementary

/m iddle schools, guidance counselors, teachers, specialists). The EWIMS team is responsible for identifying students who are at risk and ensuring that their individual needs are met through school - based interventions. In most cases, this team is not directly responsible for appl yi ng interventions for students but their focus should be on helping students navigate the school sy stems to access appropriate and needed services.  **Participate in EWIMS professional development.** The EWIMS team will receive profession al development on the EWIMS process and tool capabilities, and subsequently be given adequate time to implement the EWIMS process.  **Import student data into the EWS tool**  **.** A member of the EWIMS team is responsible for entering, or importing, data into the EWS tool , which facilitates the EWIMS team’s use of the tool to use the data to initially flag students as at risk.  **Produce reports of at -**  **risk students and assign students to appropriate interventions or services.**  The tool houses information about interventions assigned to each student and documents students’ transition in and out of each intervention and their ultimate response to the intervention(s) (i.e., for each student flagged, did the assigned intervention(s) have a positive influence on the number o f flags as calculated during subsequent grading periods?). The EWS tool does not prescribe specific interventions for students based on the ty pe or number of indicators, but rather relies on EWIMS teams to make data - driven decisions within their own local context of potential interventions available, to match students with interventions.  **Conduct EWIMS team meetings that are organized and documented.** An agenda for each meeting should be prepared at the end of the prior meeting, and at least some agenda item s should be routine, such as a review of the data from the tool, actions taken for individual or groups of students, a review of previous meetings’ action items (ongoing or completed), new action items, and communication with staff and leadership. Notes sh ould be taken at each meeting and include action items assigned to specified individuals to accomplish. Agenda and meeting notes should be kept on file to provide a record of the team’s work.  **Communicate with individuals and groups outside of the EWIMS tea m**  . In formation on flagged students, intervention effectiveness, and team - identified needs to supportREL Midwest Recruitment Materials

— 59

students should be routinely reported to and discussed with school and district leadership. Teachers should receive regular updates about students in their classes who are displaying indicators of risk, as well as input about supports available to them to use with these students. Last, students and their parents should be engaged in the conversation about their risk status and the plans to ensure that they are able to get back on track for graduation. Although the EWIMS team may not be directl y responsible for meetings with individual students and their parents (i.e., delivering the individual interventions), the team should be in a position to prompt such me etings or to share information routinely about student progress and the earl y warning signs of risk. Of critical note, the team should share the knowledge of students’ risk with sensitivity , ensuring that identification is used to prompt action and support , not to assign labels that carry stigma.  **Solicit feedback from stakeholders** . Feedback from administrators, teachers, staff, students, and parents can help the EWIMS team uncover underl yi ng causes for students displaying indicators of risk. This informatio n may help the EWIMS team match students to appropriate interventions and supports.  **Monitor progress.** The EWIMS team should monitor progress as it strives to improve educational outcomes for students during a single school year and over the course of mult iple school years. The team should be responsible for presenting progress reports to key stakeholders, including principals, staff, district leadership, the local board of education, and parents. 

### Study Team Role and Responsibilities The project team is c omposed of researchers from the Midwest Regional Educational Laboratory at AIR. The major responsibilities for the study team are as follows:  Obtain necessary

approvals from review boards (federal, district, and organizational Internal Review Boards) and comply with the research protocols in place.  Provide access to the EWS tool and training and technical support for implementation.  Collect data for the study. The majority of the data for this study will be administrative records transmitted from the district, thus minimizing the data collection burden on participating schools. The study team also will conduct an annual Web - based survey of all schools and collect all implementation data from schools assigned to the treatment group ( EWS tool data, interviews with EWIMS team members).  Assure confidentiality . The study team will collect data only for the purposes of this study and will not use or allow the use of the data for evaluating individual participants, schools, or districts.  Each participant will be as signed a study- specific identification number, in place of their names. A data file that links each participant with their identification number will be kept in a password - protected file that only the study team can access.  The published anal ys is of the re sults will aggregate results across all schools and will not include results that have been disaggregated by school or district.REL Midwest Recruitment Materials

— 60

 All members of the study team are required to complete a comprehensive training course that addresses current federal government standards and sign federal data confidentiality

agreements.  Analyze data and produce reports. The study team will be responsible for aggregating information about the effectiveness of EWIMS on student and school outcomes. The study team expects that the final report will be released in 2016, pending the federal review process, and will ensure that participating schools receive this report. 

### Timeline Table A - 1 presents the major tasks of the project as they were described previously . **Table A**  **- 1. Major**  **Tasks of the Project Tasks**  **Dates EWIMS Implementation**  Treatment schools implement EWIMS with Grades 9 and 10 Jan 2014

–June 2014 Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014

–June 2015 Control schools implement EWIMS August 2015

–June 2016 **Data Collection** Collect administrative records from the state and district Jan 2014

–June 2015 Collect EWS tool data from treatment schools Jan 2014

–June 2015 Conduct annual Web - based survey May 2014 and May 2015 Conduct interviews with EWIMS team members in treatment schools June 2014 and June 2015 **Analysis and Reporting** Draft and submit final report. December 2015 EWIMS implementation is aligned with the academic calendar. The school - based EWIMS teams meet monthly , with other critical activities occurring prior to school beginning, after the first 20 or 30 day s of school, shortly after the end of each grading period , and at the end of the academic ye ar. Table A - 2 details expected key activities of EWIMS implementation over the course of an academic year.REL Midwest Recruitment Materials

— 61

**Table A**  **- 2. Schedule and Key Activities for Early Warning Intervention and** 

**Monitoring System Implementation Year 1**  **Activities Schedule**  **Key Activities (aligned to the Early Warning Intervention and Monitoring System**

**[EWIMS] implementation steps)** January 2014  Forming/designating an EWIMS team (Step 1)

 Setting up the early warning system (EWS) Tool (Step 2)  Begin convening monthly EWIMS team meetings (Step 1) After the midyear grading period (~January 2014)  Importing or entering students’ absences, course failures, and behavior information

(e.g., referrals and suspensions, by grading period) (Step 2)  Reviewing an d interpreting student - and school - level reports (Steps 3 and 4)  Identifying and implementing student interventions (Step 5)  Monitoring students’ responses to existing interventions in which they are participating (Step 6)  Revising students’ intervention assignments, as needed (Steps 5 and 6)

At the end of the school year (~June 2014)  Updating student roster to reflect new enrollees, transfers in and out, and so forth

(Step 2)  Importing or entering students’ absences, course failures, and behavior information (e.g., referrals and suspensions), by grading period, if applicable (Step 2)  Reviewing and interpreting student - and school - level data (Steps 3 and 4)  Identifying and implementing new student interventions (Step 5)  Monitoring students’ responses to existing interventions in which they are participating (Step 6)  Revising students’ intervention assignments for summer and for the next academic year, if needed (Steps 5 and 6)  Evaluating the EWIMS process, using student - and school - level reports, and revi se as necessary (Step 7)  Exporting student data to (1) prepare the EWS tool for the next school year and/or (2) for those students who are transitioning to high school, share data with students’ high school(s). **Year 2 Activities Schedule**  **Key Activities (aligned to the Early Warning Intervention and Monitoring System**

**[EWIMS] implementation steps)** At the beginning of the school year  Reconvening the EWIMS team meetings (Step 1)

 Importing or entering student information and, if available, incoming risk indicator data into the EWS Tool (Step 2)REL Midwest Recruitment Materials

— 62

(~August 2014)  Reviewing and interpreting student needs based on data from the previous year (e.g.,

review the Overage Student Report) (Steps 3 and 4)  Verifying student information, especially enrollme nt status, and updating student roster to reflect new enrollees, transfers in and out, and so forth (Step 2)  Reviewing incoming risk indicators or previous year data, including any additional information (e.g., bridge program participation, summer school p articipation, prior course performance), to review and interpret student needs (Steps 3 and 4)  Identifying and implementing student interventions or supports based on incoming risk indicator information, if available (Step 5) After the first 20 or 30 days of the school year (~October 2014)  Updating student roster to reflect new enrollees, transfers in and out, and so forth

(Step 2)  Importing students’ absences (Step 2)  Reviewing and interpreting student - and school - level reports (Steps 3 and 4)  Identifying and implementing student interventions (Step 5)  Monitoring students’ initial response to interventions (Step 6)  Revising students’ intervention assignments, as needed (Steps 5 and 6) After the midyear grading period (~January 2015)  Updating student roster to reflect new enrollees, transfers in and out, and so forth

(Step 2)  Importing or entering students’ absences, course failures, and behavior information (e.g., referrals and suspensions, by grading period) (Step 2)  Reviewing and interpretin g student - and school - level reports (Steps 3 and 4)  Identifying and implementing student interventions (Step 5)  Monitoring students’ responses to existing interventions in which they are participating (Step 6)  Revising students’ intervention assignments, a s needed (Steps 5 and 6)

At the end of the school year (~June 2015)  Updating student roster to reflect new enrollees, transfers in and out, and so forth

(Step 2)  Importing or entering students’ absences, course failures, and behavior information (e.g., referrals and suspensions), by grading period, if applicable (Step 2)  Reviewing and interpreting student - and school - level data (Steps 3 and 4)  Identifying and implementing new student interventions (Step 5)  Monitoring students’ responses to existing interventions in which they are participating (Step 6)  Revising students’ intervention assignments for summer and for the next academic year, if needed (Steps 5 and 6)  Evaluating the EWIMS process, using student - and school - level reports, and revise as necessary (Step 7)  Exporting student data to (1) prepare the EWS tool for the next school year and/or (2) for those students who are transitioning to high school, share data with students’ high school(s).REL Midwest Recruitment Materials

— 63



### Benefits to Participation There are man y benefits of participation for high schools in Ohio. More than one out of five students in Ohio fails to graduate from high school and graduation rates are significantly lower for students who are racial or ethnic minorities, economically disadvantaged, receiving speci al education services, migrant and/or limited - English proficient. Critical indicators in ninth and tenth grade that powerfully predict whether students are “on track” for high school graduation can be used as part of an earl y warning s ys tem to flag at - risk students early , assign appropriate interventions, and get students back on track. Participating in this high - profile, large - scale study will give your school and district an opportunity to access the Earl y Warning and Intervention Monitoring Sy stem (devel oped by the National High School Center at AIR) at no cost. The EWIMS model, currently in use in 67 districts in six states, includes both an excel - based tool and training and technical support for implementation. Your participation in this study will also play an important role in informing educational policy focused on dropout prevention in Ohio and at the federal level. 

### Questions or Comments If you have an y questions or comments about the study or the opportunity

it provides for your school, please feel free to contact Dr. Nicholas Sorensen (nsorensen@air.org or 312 - 283 - 2318) or Dr. Mindee O’Cummings (mocummings@air.org or 202 - 403 - 5254).REL Midwest Recruitment Materials

— 64 

### Signatures of **Commitment** Return via fax (312 - 288 - 7601) or e - mail ( nsorensen @air.org

) The following people have read this document detailing the study and agree to the roles, responsibilities, and conditions of participation on behalf of <High School Name> and the study team. <insert name> <insert title> District Representative Signature Printed Name Title Date Principal Principal Signature Printed Name Title Date Jessica Heppen Co - Principal Investigator Principal - Investigator Printed Name Title Date Mindee O’Cummings Co - Principal Investigator Principal - Investigator Printed Name Title Date Ann - Marie Faria Project Director Project Director Printed Name Title Date Nicholas Sorensen Deputy Project Director Project Director Printed Name Title Date

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, Part E, Section 183, responses to this data collection will be used only for statistical purposes. The reports prepared for this study will summarize findings a cross the sample and will not associate responses with a specific district or individual. ED’s contractor will not provide information that identifies you or your district to anyone outside the study team, except as required by law. Any willful disclosure of such information for nonstatistical purposes, without the informed consent of the respondent, is a class E felony.REL Midwest Recruitment Materials

— 65 

### A **- 8. Final Agreement**  

## Form **(District MOU)** 

## District Roles and Responsibilities: **Early Warning and Intervention Monitoring System Study** Dear <superintendent>, On behalf of American Institutes for Research (AIR) (www.air.org ) and its partners, we welcome yo u to the Early Warning and Intervention Monitoring System (EWIMS) study . We are excited about this project. Its purpose is to examine the impact of EWIMS on (1) student outcomes including student risk status for dropout, scores on graduation tests , persistence and progress in school and likelihood of on - time graduation; and (2) school outcomes including h ow schools allocate dropout prevention interventions for students and their data - use culture. This document contains an overview of the study and a brief description of the intervention, followed by a description of the roles and responsibilities for your district and for the study team, including the benefits of participation, and the project timeline. Please review the contents of this document and sign the last page to indicate your agreement to participate. Return the signed last page to Dr. Nicholas So rensen (nsorensen@air.org , fax 312 - 288 - 7601). 

### Overview The primary

goal of the EWIMS study is to evaluate whether implementing an earl y warning and intervention monitoring sy stem for identifying students at - risk of dropping out and using this sy stem to assign students to dropout prevention interventions will improve student outcomes including student risk status for dropout, scores on graduation tests , persistence and progress in school and likelihood of on - time grad uation. In addition, this study will also examine the impact of implementing EWIMS on school outcomes including how schools allocate dropout prevention resources for students and their data - use culture. Study results will yield valuable information for the state of Ohio and for districts and schools across the country about the viability

and benefits of using an early

warning sy stem to prevent high school dropout and help struggling students get back on track for eventual graduation. The study is being fund ed b y the U.S. Department of Education, Institute of Education Sciences, and will run from January 2014 through the spring of 2016. We look forward to working with <district name> as a partner in this project!REL Midwest Recruitment Materials

— 66 **STEP 1 –**  **Establish Roles & Process** 

**STEP 2 –**  **Use the EWS tool** 

**STEP 3 –**  **Analyze EWS data** 

**STEP 4 –**  **Interpret EWS data**  **STEP 5 –**  **Assign & Provide**  **Interventions STEP 6**  **– Monitor**  **Students STEP**  **7 –**  **Evaluate & Refine EWIMS**  

### The EWIMS Model The EWIMS model, developed by the National High School Center at AIR, is a multistep process intended to encourage s ys tematic and comprehensive implementation within schools. The process is based on a combination of research on data use in schools and National High School Center’s experience working with states, districts, and schools implementing earl y warning s ys tems. At the heart of the EWIMS process is an early warning data tool used to flag students as “at risk” based on attendance, course performance (grades, credits, grade point average [GPA]), and behavior indicators. The tool enables schools to identify students who are at risk of dropping out of school, record assignments to available interventions, and monitor students’ response to those interventions. Be yond the devel opment of the data tools, the National High School Center has devised a seven step EWIMS implementation process to support implementation. The process guides users to make informed decisions about how to use data to support at - risk students and how to cont inue to monitor their progress over time. In addition to focusing on individual students, the process guides users to examine the success of specific supports or interventions and to examine possible sy stemic issues (e.g., school climate) that may relate t o dropout trends. **F igure**  **A -**  **2 . Early Warning Intervention Monitoring System Implementation Process** 

As shown in Figure 1, the steps are intended to be cyclical. At the core of this data - driven decision - making process, the steps focus users on key indicator s that identify

which students are showing signs of risk of dropping out of high school and guide users to go beyond the indicator data and other relevant information to connect at - risk students to dropout prevention or academic support interventions. The EWIMS model does not prescribe specific interventions for schools to implement. Instead, the model is designed allow schools flexibility to decide which interventions they believe are most effective for their students’ needs. Ideally , the EWIMS model allow s users to identify

students with accuracy

and provide supports and intervention of schools’ choosing toREL Midwest Recruitment Materials

— 67

at - risk students, resulting in improved outcomes for students, including higher attendance rates and improvement in academic performance leading toward successful graduation. 

### This Study Despite increasingl y widespread implementation of earl y warning s ys tems by states, districts, and schools, there have been no rigorous studies testing the impact of using an earl y warning sy stem to improve student outcomes such as staying in school, progressing in school, and graduating. There is also little research examining how using an earl y warning s ys tem can shape a school’s culture for data use

—increased data - driven decision making (assigning interventions to students) and processes and professional development to support using data to improve teaching and learning. This study will address these gaps and provide the first rigorous test of the impact of an earl y warning s ys tem. The study will:  **Identify a sample of eligible and interested schools in Ohio.**  The study team will conduct outreach to schools that meet initial eligibility criteria to confirm eligibility and discuss interest in participating in the EWIMS study . The study will include approximately

70 high schools in Ohio. To qualify, schools must (1) have at least 150 ninth - grade students; (2) a graduation rate between 25 and 95 percent and (3) not already be implementing an earl y warning s ys tem tool for using data to flag at - risk students.  **Use a lottery to randomly assign half of the participating schools to implement**  **EWIMS in January 2014**  **and the other half to implement in fall of 2015.** Half of participating schools will be randoml y assigned to receive access to the EWIMS model in January 2014 (including the tool and technical support for implementation) and the other half of participating schools will be randomly assigned to conduct “business as usual” for identifying at - risk students until the fall of 2015 when they will receive the same reso urces and supports for implementing EWIMS.  **Implement EWIMS.** All schools will implement the seven - step EWIMS process either in January

2014 or the fall of 2015 (depending on random assignment by lottery). The seven steps for implementation are as follows: o S tep 1

—Establish Roles and Process. In this step, the composition of the EWIMS team is established; 21 team members then determine frequency and duration of meetings and develop a shared vision or focus of the team’s work. o Step 2

—

Use the EWS tool . In this step, the school - based EWIMS teams are trained on the use and purpose of the tool itself. This step also includes first customizing the tool settings and importing the student demographic and initial administrative data and also ongoing refreshing of the administrative data in the tool and the running of automated and custom lists and reports available within the tool. o Step 3

—

Analyze EWS data . In Step 3, EWIMS teams focus their attention on student - and school - level data, based on the indicators available in the tool. This

21 The goal is not to create yet anot her “team” with functions that may or may not overlap with functions of other already existing teams. Rather, integration with existing team structures ( if functionally operational) is optimal.REL Midwest Recruitment Materials

— 68 data review process is intended to identify areas of focus and further investigation. o Step 4

—Interpret EWS data . Step 4 guides teams to bring in additional data (external to the tool) to provide more context and a fuller picture to inform the EWIMS team’s consideration of specific needs of individuals or groups of flagged students. Unlike Step 3, which is focused on the indicator flags themselves (i.e., the data in the tool), this step addresses root causes of wh y students might be identifie d as at risk for one or more indicators. The implementation team will provide training to identify root causes that focus on acquiring additional formal (e.g., administrative records) and informal (e.g., from teacher, famil y, and student) input. This train ing will occur face to face and will last for one hour. o Step 5

—

Assign and Provide Interventions. In this step, EWIMS team members make informed decisions about the allocation of available resources and strategies to support students identified as at risk of dropping out of high school. The EWIMS team matches individual students to specific interventions after having gathered information about (1) potential root causes for individual flagged students (Step 4) and (2) the available dropout prevention and aca demic and behavioral support programs in the school, district, and community , which are locally determined. o Step 6

—Monitor Students. In this step, EWIMS teams continue to examine student indicators at regular intervals to continually identify students who show signs of being at risk. The teams will use the same indicators to closely monitor already- identified students who were assigned to interventions for progress in school and risk status. This step provides critical ongoing feedback about additional stu dent - and school - level needs and apparent successes. o Step 7

—Evaluate and Refine the EWIMS. Through active and structured reflection, EWIMS team members assess whether students are responding to assigned interventions, revise their specific strategies or g eneral approach as needed, and determine how resources are allocated to improve support for students. This step encourages EWIMS teams to make course corrections to all parts of the EWIMS implementation. As implied by the cyclical depiction of the seven - st ep process, this step (as well as the other six) reflects an ongoing process of continual improvement.  **Evaluate the effects of EWIMS on student and school outcomes.** The study will examine student outcomes for all students in grades 9 and 10 during the 201 3

– 14 school ye ar and all students in grades 9 through 11 during the 2014

– 15 school year. All student outcome data will be collected from school or district administrative data, the EWS tool or the Ohio Department of Education. In addition, all participatin g schools will be asked to complete an annual Web - based survey about data use practices and schools randomly assigned to implement EWIMS in January 2014 may

be asked to participate in interviews about their experiences using the tool.REL Midwest Recruitment Materials

— 69



### District Roles and R**esponsibilities** We look forward to partnering with <District Name> for this exciting project! More detailed information on the responsibilities of participating schools follows.  **Maintain a sustained commitment to participate in the study.** To evaluate the impact of EWIMS, it is critical that participating districts agree to adhere to the study guidelines and timelines associated with random assignment by lottery to implement EWIMS in January 2014 fall of 2015.  **Support school recruitment.** Participating districts should actively support the study team’s efforts to recruit high schools b y communicating with school leadership and endorsing district participation and encourage school - level participation.  **Support the use of random assignment.** It is essential that the district support the use of random assignment of schools to receive access to and implement EWIMS in January 2014 or the fall of 2015. All schools will receive access to EWIMS including the tool and training and technical support for implementation; random assignment will only determine whether schools receive the “treatment” in January 2014 or a “delayed treatment” in the fall of 2015. Schools assigned b y lottery to implement the EWIMS model in January

2014 will serve as the treatment group. Their counterparts assigned to implement EWIMS in fall of 2015 will serve as a control group between January 2014 through the spring of 2015. Schools randomly assigned to the control group must continue with “business as usual” practices for id entifying at - risk students and assigning dropout prevention interventions until EWIMS implementation begins in the fall of 2015. Students in control schools should continue to receive any services that would be offered to them in the absence of the study . As for students in the “treatment” group, no “ty pical” services should be withheld.  **Provide access to student -**  **level administrative records.** Administrative student - level data collection in participating schools will focus on all students in grades 9 and 10 during the 2013

– 14 school year and all students in grades 9, 10 and 11 during the 2014

– 15 school year. The study team will obtain as much administrative data as possible from the Ohio Department of Education (ODE). However, all participating schools shoul d provide the study

team access to the following student - level administrative data should this data be unavailable from ODE: o Demographic information ( e.g., race/ethnicity, gender, free or reduced - price lunch [FRPL ], individualized education program [IEP], and English language learner [ELL ] status, and parents’ education) o Grade point average (GPA) o State test scores o Attendance rates o Course grades in core academic courses b y semester o Credits earned b y s emester o Disciplinary

information (e.g., suspensions)REL Midwest Recruitment Materials

— 70 o Enro llment information (e.g. whether students are enrolled or have left school for reasons other than transfer to another district, including dropping out) o Grade promotion 

### Study Team Role and Responsibilities The project team is composed of researchers from the Midwest Regional Educational Laboratory at AIR. The major responsibilities for the study team are as follows:  Obtain necessary

approvals from review boards (federal, district, and organizational Internal Review Boards) and comply with the research prot ocols in place.  Provide access to the EWS tool and training and technical support for implementation to participating schools.  Collect data for the study. The majority of the data for this study will be administrative records transmitted from the district, thus minimizing the data collection burden on participating schools. The study team will provide a detailed data request to participating districts in a pre - determined format that will minimize burden on district staff.  The study

team will conduct an annual Web - based survey of all participating schools and collect all implementation data from schools assigned to the treatment group ( EWS tool data, interviews with EWIMS team members).  Assure confidentiality . The study team will collect data only for the purposes of this study and will not use or allow the use of the data for evaluating individual participants, schools, or districts.  Each participant will be assigned a study- specific identification number, in place of their names. A data file that links each participant with their identification number will be kept in a password - protected file that only the study team can access.  The published anal ys is of the results will aggregate results across all schools and will not include re sults that have been disaggregated by school.  All members of the study team are required to complete a comprehensive training course that addresses current federal government standards and sign federal data confidentiality

agreements.  Analyze data and prod uce reports. The study

team will be responsible for aggregating information about the effectiveness of EWIMS on student and school outcomes. The study team expects that the final report will be released in 2016, pending the federal review process, and will ensure that participating schools receive this report.REL Midwest Recruitment Materials

— 71



### Timeline Table A - 3 presents the major tasks of the project as they were described previously . **Table A**  **- 3**  **. Major Tasks of the Project Tasks**  **Dates EWIMS Implementation**  Treatment schools implement EWIMS with Grades 9 and 10 Jan 2014

–June 2014 Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014

–June 2015 Control schools implement EWIMS August 2015

–June 2016 **Data Collection** Collect administrative records from the state and district Jan 2014

–June 2015 Collect EWS tool data from treatment schools Jan 2014

–June 2015 Conduct annual Web - based survey May 2014 and May 2015 Conduct interviews with EWIMS team members in treatment schools June 2014 and June 2015 **Analysis and Reporting** Draft and submit final report. December 2015 

### Benefits to Participation There are many

benefits of participation for districts in Ohio. More than one out of five students in Ohio fails to graduate from high school and graduation rates are significantl y lower for students who are racial or ethnic minorities, economicall y disadvantaged, receiving special education services, migrant and/or limited- English proficient. Critical indicators in ninth and tenth grade that powerfull y predic t whether students are “on track” for high school graduation can be used as part of an earl y warning system to flag at - risk students earl y, assign appropriate interventions, and get students back on track. Participating in this high -profile, large - scale st udy will give your district an opportunity to access the Earl y W arning and Intervention Monitoring S ys tem (developed by the National High School Center at AIR) at no cost. The EWIMS model, currently in use in 67 districts in six states, includes both an ex cel - based tool and training and technical support for implementation. Your participation in this study will also play

an important role in informing educational policy

focused on dropout prevention in Ohio and at the federal level. 

### Questions or Comments If you have an y questions or comments about the study or the opportunity

it provides for your district, please feel free to contact Dr. Nicholas Sorensen (nsorensen@air.org or 312 - 283 - 2318) or Dr. Mindee O’Cummings (mocummings@air.org or 202 - 403 - 5254).REL Midwest Recruitment Materials

— 72 

### Signa **tures of Commitment**

Return via fax (312 - 288 - 7601) or e - mail ( nsorensen@air.org

) The following people have read this document detailing the study and agree to the roles, responsibilities, and conditions of participation on behalf of <District Name> and the study team. <insert name> <insert title> District Representative Signature Printed Name Title Date Jessica Heppen Co - Principal Investigator Principal - Investigator Printed Name Title Date Mindee O’Cummings Co - Principal Investigator Principal - Investigator Printed Name Title Date Ann - Marie Faria Project Director Project Director Printed Name Title Date Nicholas Sorensen Deputy Project Director Project Di rector Printed Name Title Date The below <District Name> staff person has agreed to serve as the primary point of contact for school recruitment and data collection during the course of the study. Name: Position: Telephone Number: E - M ail:REL Midwest Recruitment Materials

— 73 

### A **- 9. Final Agreement Form (**  

## State **MOU) Memorandum of Understanding**  **By and Between the Ohio Department of Education and __________** This agreement is entered into by the Ohio Department of Education (“ODE”) and _____________ (“Researcher”) for the purpose of sharing information between the parties in a manner consistent with the Famil y Education Records Privacy A ct of 1974 (“FERPA”). The information will be used by researchers at _____________ to conduct evaluative studies designed to improve instruction for children in the state of Ohio. Topics of these studies will include: ___________. In order to complete these studies and in order to have a positive impact on the instruction of children, the Researcher re quires the use of student data from the ODE. The Family

E ducational Rights and Privacy A cts Statute (FERPA) describes circumstances under which State Educational Agencies (SEAs) are authorized to release data from an education record. This information can be disclosed to organizations conducting studies on behalf of SEAs, provided that Federal, State or local law authorizes the evaluation in question. Ohio Revised Code § 3301.12 grants to the ODE State Superintendent of Public Instruction the authority t o conduct studies of education programs, but the provision prohibits access to a student’s name, address or social security number. 1. PARTIES. The ODE is an SEA that is authorized to receive information from local educational agencies (“LEAs”) subject to F ERPA, as authorized by 3 4 CFR Section 99.31. Researcher desires to conduct studies on behalf of ODE for the purpose of improving instruction in Ohio public schools in accordance with the Scope of Work Agreement attached hereto as Appendix A. The parties wi sh to share data collected by the ODE regarding education in Ohio, none of which will allow the identification of individual students. 2. COMPLIANCE WI TH FERPA. To effect the transfer of data subject to FERPA, Researcher agrees to: 1. In all respects compl y with the provisions of FERPA. For purposes of this agreement, “FERPA” includes an y amendments or other relevant provisions of federal law, as well as all requirements of Chapter 99 of Title 34 of the Code of Federal Regulations. Nothing in this agreement may

b e construed to allow either party t o maintain, use, disclose or share student information in a manner not allowed by federal law or regulation. 2. Use the data shared under this agreement for no purpose other than research authorized under Section 99.3 1(a)(3)(iv) or 99.31(a)(6) of Title 34 of the Code of Federal Regulations. Researcher further agrees not to share data received under this MOU with any o ther entity without the ODE’s approval. Researcher agrees to allow the Office of the State Auditor, sub ject to FERPA restrictions, access to data shared under this agreement and any r elevant records of Researcher for purposes of completing authorized audits of the parties. Researcher shall be liable for an y audit exception that results solely f rom its acts or omissions in the performance of this agreement. ODE shall be liable for an y audit exception that results solely f rom its acts orREL Midwest Recruitment Materials

— 74

omissions in the performance of this agreement. In the event that the audit exception results from the act or omissions of b oth parties, the financial liability f or the audit exception shall be shared by the parties in proportion to their relative fault. 3. Require all employees, contractors and agents of any kind to comply

w ith all applicable provisions of FERPA and other federal laws with respect to the data shared under this agreement. Researcher agrees to require and maintain an appropriate confidentiality a greement from each employee, contractor or agent with access to data pursuant to this agreement. Nothing in this paragra ph authorizes sharing data provided under this Agreement with any other entity f or an y purpose other than completing Researcher’s work authorized under this Agreement. 4. Maintain all data obtained pursuant to this agreement in a secure computer environment and not copy , reproduce or transmit data obtained pursuant to this agreement except as necessary t o fulfill the purpose of the original request. All copies of data of an y ty pe, including an y modifications or additions to data from an y source that contains information regarding students, are subject to the provisions of this agreement in the same manner as the original data. The ability to access or maintain data under this agreement shall not under any c ircumstances transfer from Researcher to an y other in stitution or entity . 5. Not to disclose any

d ata obtained under this agreement in a manner that could identify an individual student to any other entity i n published results of studies as authorized by this agreement, nor attempt to infer or deduce the ide ntity o f an y student or teacher based on data provided by

O DE, nor claim to have identified or deduced the identity o f any s tudent based on data provided b y ODE. 6. Not to provide any

data obtained under this agreement to any party ineligible to receive data protected b y FERPA or prohibited from receiving data from any entity . 7. Provide to the ODE a list of specific research studies, updated annuall y, for which the data are being used, and to notify the ODE in advance of an y new project or research question researcher proposes to address. This list of research studies will identify linkages of all data possessed by

r esearcher under this agreement and covered by F ERPA to specific research studies. Further, it will include the fixed ending date for use of all data linked to each project. 8. Provide to the ODE any materials designed for public dissemination, based in whole or in part on data obtained under this agreement, at least ten day s prior to public release. 9. Destro y all data obtained under this agreement, within the time frame established in Appendix A, Section II, when it is no longer needed for the purpose for which it was obtained. Nothing in this agreement authorizes either party t o maintain data beyond the time period reasonabl y needed to co mplete the purpose of the request. All data no longer needed shall be destroyed or returned to the ODE in compliance with 34 CFR Section 99.35(b)(2). Researcher agrees to require all employees, contractors, or agents of any kind to comply

with this provisi on.REL Midwest Recruitment Materials

— 75

3. DATA REQUESTS. 1. The ODE may

decline to comply

w ith a request if it determines that providing the data in the manner requested would violate FERPA and/or would not be in the best interest of current or former students in Ohio public schools. All requests shall include a statement of the purpose for which it is requested and an estimation of the time needed to complete the project for which the data is requested. Data requests may b e submitted by

p ost, electronic mail or facsimile. 2. Researcher agrees that ODE makes no warranty concerning the accuracy o f the student data provided. 4. AUTHORIZED REPRESENTATIVE. Researcher shall designate in writing a single authorized representative able to request data under this agreement. The authorized representative s hall be responsible for transmitting all data requests and maintaining a log or other record of all data requested and received pursuant to this agreement, including confirmation of the completion of any p rojects and the return or destruction data as requi red by this agreement. The ODE or its agents may

upon request review the records required to be kept under this section. RELATED PARTIES. Researcher represents that it is authorized to bind to the terms of this contract, including confidentiality and destruction or return of student data, all related or associated institutions, individuals, employees or contractors who may have access to the data or may own, lease or control equipment or facilities of any kind where the data is stored, maintained or used i n an y way b y R esearcher. This Agreement takes effect onl y upon acceptance b y an authorized representative of _____________, by which that institution agrees to abide by its terms and return or destro y all student data upon completion of the research for wh ich it was intended or upon the termination of its current relationship with Researcher. TERM. This agreement takes effect upon signature by the authorized representative of each party and will remain in effect until _________________. The parties further understand that the ODE may cancel this agreement at any time for reasonable cause, upon thirty- day written notice. Notice of such cancellation shall be sent or otherwise delivered to the persons signing this agreement. The ODE specificall y reserves the right to immediately cancel this agreement upon discovery of non - compliance with an y applicable federal or state laws, rules or regulations. Further, the ODE specificall y reserves the right to immediately cancel this agreement should the ODE, in its sole di scretion, determine that student information has been released in a manner inconsistent with this agreement, has not been maintained in a secure manner, or that substantially similar data access has become generally

available for research purposes through an y other mechanism approved b y the ODE. In the event of immediate cancellation, a notice specifying the reasons for cancellation shall be sent as soon as possible after the cancellation to the persons signing the agreement. 5. BREACH AND DEFAUL T. Upon brea ch or default of an y of the provisions, obligations, or duties embodied in this agreement, the parties may e xercise an y administrative, contractual, equitable, or legal remedies available, without limitation. The waiver of an y occurrence of breach or defau lt is not a waiver of such subsequent occurrences, and the parities retain the right to exercise all remedies mentioned herein.REL Midwest Recruitment Materials

— 76

6. AMENDMENT. This agreement may b e modified or amended provided that an y such modification or amendment is in writing and is signe d by the parties to this agreement. It is agreed, however, that any amendments to laws, rules, or regulations cited herein will result in the correlative modification of this agreement, without the necessity f or executing written amendment. 7. ASSI GNMENT OF R IGHTS. Neither this agreement, nor an y rights, duties, or obligations described herein shall be assigned by R esearcher without the prior express written consent of ODE. 8. ENTIRETY OF AGREEMENT. All terms and conditions of this agreement are embodied herein and in the Scope of Work Agreement attached hereto as Appendix A. No other terms and conditions will be considered a part of this agreement unless expressly a greed upon in writing and signed b y both parties. Entered into b y

____________________________ _____ Michael L. Sawyers Date Acting State Superintendent of Public Instruction

_____ Matthew Cohen Date Chief Research Officer

_____________________________ _____ [Signature] Date [Printed Name and Title] [Name of Organization]REL Midwest Data Collection Instruments

— 77



## Attachment **B. Data Collection Instruments** Attachments B - 1 through B - 6 include data collection instruments that will be used in the EWIMS impa ct study . B - 1 includes a draft protocol of the school - level survey that measures the treatment contrast and data - informed allocation of dropout prevention interventions for students. B - 2 includes a draft protocol of the school data culture survey . B - 3 inc ludes archival records protocol. B - 4 includes protocol for collecting data stored in the EWS tool in treatment schools.REL Midwest Data Collection Instruments

— 78 

### B **- 1. Draft Protocol** 

**— School**  **- Level Survey**  **(Treatment Contrast and Data -**  **Informed Allocation of Dropout Prevention Interventions for St**  **udents)** 

## Paperwork Burden Statement According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is **XXXX -** **XXXX** . The time required to complete this information collection is estimated to average 60 minutes per response . This information collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202

– 4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208 - 5500. ** This survey

will be administered to both treatment and control schools to estimate the impact of EWIMS on school - level outcomes The purpose of this draft school level survey is twofold: the first section documents current usage of an earl y warning s ys tem tool or the use of data to identify students who are at risk of academic failure, disengagement, and/or dropping out of high school, and will be used to measure the treatment contrast in the EWIMS impact study . We will use this information to document the treatment contrast. Measures are adapted from the REL Appalachia survey in an attempt to coordinate data collection across REL s on si milar EWS implementation studies. These second section collect s consistent information from treatment and control schools about the ty pes of dropout interventions they are implementing and whether and how they use student data to allocate those dropout int erventions. This section will be used to answer school - level research questio n 2a : Wh at is the impact of EW IMS on data - informed allocation of dropout prevention interventions for students? It ems are based on up - to - date input from the Dropout Prevention Res earch Alliance members ’ input, as well as a collaboration between REL Appalachia and REL Midwest to develop similar tools for EWS implementation studies. The study team plans to continue to review and populate these response options prior to administration and in collaboration with alliance members to reflect the most up - to - date local. These items are still in draft form and will be finalized with input from REL Midwest’s Dropout Prevention Alliance. **Section I:**  **Treatment Contrast —** **Use of an Early Warning System**  **1.** Is your high school using an earl y warning s ys tem ( **EWS** ) to identify students who may

be at risk of not graduating from high school on time? o Yes, we are implementing EWS in my high school o No, my

high school is not currentl y imp l ementing EWS in any high schools o I ’ m not sure.REL Midwest Data Collection Instruments

— 79

T he following questions are designed to obtain information about EWS implementation and tools in your high school. **2.** Does your high school have an earl y warning s ys tem tool?

a. Yes, we have an earl y warning s y stem tool used to identify students who may

be at risk of not graduating in my high school. If so, what is the name of the sy stem? b. No, my

high school does not currentl y have an early warning s ys tem tool to identify students who may

be at risk of not graduating in my high school. c. I ’ m not sure. **3.** Is your high school using the student information system (SIS) or another data tool to identify students who may

be at risk of not graduating from high school on time? a. Yes, we have a tool to identify students who may

be at risk of not graduating in my high school. b. No, my

high school does not currentl y have a tool to identify students who may be at risk of not graduating in m y high school. c. I ’ m not sure. If yes, please specify : **4.** How do you use the EW S Tool? ( Please c heck all that apply .) o To identify which students are at risk for dropping out of school o To target resources to support off - track students before they drop out o To monitor student progress or students’ response to intervention o To assign students to intervention(s) o To document causes of student risk on the basis of EWS reports and in formation sources o To examine patterns and identify school climate issues o To transfer students at risk to alternative schools o Other (Please specify.) **5.** If your school uses an early warning s ys tem, SI S , or another data tool to identify students who may be at risk of not graduating , what types of data are used? For each indicator used, what is the level of evidence? **6.**REL Midwest Data Collection Instruments

— 80 N/A Evidence -based (e.g.,

based on research conducted using data not related to the school) Locally validated (e.g.,

based on research using state, district , or school data) Based on other

criteria (specify) Attendance  Behavior referrals  Behavior suspensions  Course performance  Credits earned  Grade Point Average (GPA)  State assessment results  Other:  Other:  Other:  Other: 

**7.** How do you assign at - risk students to supports and/or interventions? ( Please check all that apply . ) o Student - level indicator data (i.e., evidence or locally validated indicators)REL Midwest Data Collection Instruments

— 81

o Other student - level data (e.g., demographics, student performance data ) o Specify :

o Other ty

pes of data o Specify :

o No data - based strategy to assign at - risk students to supports and/or interventions **8.** For which grade levels are you examining EWS indicators of risk? (Please c heck all that apply .) o 9th Grade o 10 th Grade o 11 th Grade o 12 th Grade o Other If “ other, ” please s pecify __________________________________ **9.** Can you provide information on when you began implementing EWS in your school, as well as the degree of implementation in each school ye ar? o Before the 2010 – 11 school y ear o 2010 – 11 school year o 2011 – 12 school year o 2012 – 13 school year o 2013 – 14 school year o Other . If “ other, ” please specify :

______________________________________________________________ **10.** How often does your school review data to identify students at risk of not graduating? o 4 times per year (once per quarter) o 3 times per year o 2 times per yearREL Midwest Data Collection Instruments

— 82

o 1 time per year o Other o Specify :

o I ’ m not sure Th e followin g question s relat e to th e grou p o f individual s wh o focu s o n using earl y warnin g system data o n student s who ar e flagge d a s a t ris k o f no t graduatin g fro m high school . Thes e individual s may b e part o f a distinc t EWIMS team o r part o f anothe r tea m that i s responsibl e fo r ex aminin g thes e data. **11.** Do you have a group of individuals who focus on implementation of EWS as its own EWIMS team or as part of another existing team (e.g. , school improvement team, district improvement team) at the school or district level? o Yes, we have a dedicated EWIMS team . o Yes, we have a team that focuses on EWS, but it is part of another team. o No, we do not hav e an existing EWIMS team . o Other If “ other, ” please specify :

______________________________________________________________ **12.** D escribe the members of the EWIMS team at the high school level (if applicable)? (Please check all that apply .) o School principal o Assistant principal o Representative from feeder middle schools o Guidance counselors o Content area teachers o Special education teachers o English language learner instructors o District office representative o Other If “ other, ” please specify

________________________ __________________________________________REL Midwest Data Collection Instruments

— 83

**S ection**  **II: D**  **ata -**  **I nformed**  **A llocation of**  **D ropout**  **P revention**  **I nterventions for**  **S tudents**  1. Does your school offer targeted academic interventions (e.g., tutoring, reading remediation, study skills) for students? o Yes o No (skip to 3 ) 2. If yes, please name the targeted academic interventions that the school offer s. In the table below, indicate which programs are available, the number of students who participat ed during the school year , and the grade level of students who participate d .

**Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are use**  **d to select students for**  **participation in targeted**  **academic interventions?**  **What data do you**

**use to determine whether an**  **intervention/support is working?** 

Write in Drop down

menu (Reading, Mathematics, Science, Tutoring, Remediation, Study skills, Other please specify) Drop down menu

(from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other please specify) Drop down menu

allowing checking all that apply (Course failure [ e.g., failing algebra ] , Grade point average [ GPA ] , Number of credits, Student attendance [ e.g., missing a number of days of school Drop down menu

allowing checking all that apply (attendance, behavior, course performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)REL Midwest Data Collection Instruments

— 84

**Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are use**  **d to select students for**  **participation in targeted**  **academic interventions?**  **What data do you**

**use to determine whether an**  **intervention/support is working?**  or class ] , Student behavior [ e.g., office referrals, suspensions ] , No criteria, Oth er please specify)

3. Does your school offer targeted behavior interventions (e.g., social skill training, character education) for students ? o Yes o No (skip to 4 ) a. Please name the targeted behavior interventions at the school . In the table below, indicate which programs are available, the number of students who participat ed during the school year , and the grade level of students who participate d .REL Midwest Data Collection Instruments

— 85

**Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to select**  **students for participation in**  **targeted behavior**  **interventions? What data**  **d o you** 

**use to determine whether an**  **intervention/support is working?** 

Write in Drop down

menu ( Social Skill Training, Character Education, Other please specify) Drop down menu

(from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) Drop down menu

allowing checking all that apply ( Course failure [e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance [ e.g., missing a num

ber of days of school or class], Student behavior [ e.g., office referrals, suspensions], No criteria, Other Please specify) Drop down menu

allowing checking all that apply (attendance, behavior, course performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)REL Midwest Data Collection Instruments

— 86

4. Does your school offer targeted attendance/truancy interventions (e.g., attendance monitor) for students ? o Yes o No (skip to 5 ) b. P lease name the targeted attendance/truancy interventions that the school offers . In the table below, indicate which programs are available, the number of students who participated during the school year, and the grade level of students who participated.

**Program Name**  **Focus of th e** 

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to select**  **students for participation in**  **targeted attendance**  **interventions? What data do you use** 

**to determine whether**  **an intervention/support**  **is working?**

Write in Drop down

menu ( Attendance Monitoring , Contacting Truancy Officers Other please specify) Drop down menu

(from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) Drop down menu

allowing checking all that apply ( Course failure [e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance [ e.g., missing a num

ber of days of school or class], Student behavior [ e.g., office referrals, suspensions], No criteria, Other Please specify) Drop down menu

allowing checking all that apply (attendance, behavior, course performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)REL Midwest Data Collection Instruments

— 87

**Program Name**  **Focus of th e** 

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to select**  **students for participation in**  **targeted attendance**  **interventions? What data do you use** 

**to determine whether**  **an intervention/support**  **is working?**

5. Does your school offer o nline credit or c ontent r ecovery (e.g., Apex, K12, Plato , district program )? o Yes o No ( s kip to 6 ) c. Which of the following o nline c ontent r ecovery programs does the school? In the table below, indicate which programs are available, the number of student who participated during the school year , and the grade level of students who participated . **Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion** 

**is/criteria are used to sele**  **ct students for**  **participation in online credit**  **recovery ?**  **What data do you use**

**to determine whether the program is**  **working?**

Write in Drop down

menu allowing to check all that Drop down menu

(from 1 to 999) Drop

down menu (from Drop down

menu allowing to check all Drop down menu

allowing to check all that apply Drop down menu

allowing to check all that apply (attendance,REL Midwest Data Collection Instruments

— 88

**Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion** 

**is/criteria are used to sele**  **ct students for**  **participation in online credit**  **recovery ?**  **What data do you use**

**to determine whether the program is**  **working?** apply (Reading, Mathematics, Science, Social Studies, Other please specify) grade 9 to grade 12) that apply

(Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) ( Course failure

[e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance [ e.g., missing a num

ber of days of school or class], Student behavior [ e.g., office referrals, suspens ions], No criteria, Other Please specify) behavior, course

performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)REL Midwest Data Collection Instruments

— 89

6. Does your school offer student mentoring programs (e.g., Check & Connect, Check In/Check Out)? o Yes o No (skip to 7 ) d. Which of the following mentoring programs does the school offer or do students have access to in the community? In the table below, indicate which programs are available, the number of students who participated during the school year, and the grade level of students who participated. **Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to sele**  **ct students for**  **participation in mentoring**  **? What data do you use** 

**to determine whether the program is**  **working?**

Write in Drop down menu

( Academic mentoring , School adjustment mentoring [ e.g., 9th grade transition program, reentry from adjudication], Career mentoring , Project based and community -based mentoring , Group specific mentoring, Peer mentoring, Other please specify) Drop down menu (from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) Drop down menu

allowing checking all that apply ( Course failure [e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance (e.g., missing a num

ber of days of school or class), Student behavior [ e.g., office referrals, suspensions], No criteria, Other Please spec ify) Drop down menu

allowing checking all that apply (attendance, behavior, course performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)REL Midwest Data Collection Instruments

— 90

**Program Name**  **Focus of the**

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to sele**  **ct students for**  **participation in mentoring**  **? What data do you use** 

**to determine whether the program is**  **working?** Check & Connect Check In Check Out Gear Up

7. Does your school offer a student internship or school - related work - prep aration program (such as Job Corp, MACC Project, or career and technical education class es or programs )? o Yes o No (skip to 8 ) a. Which of the followi ng student internship or school - related work - preparation programs does the school offer? In the table below, indicate which programs are available, number of student who participated during the school year, and the grade level of students who participated.REL Midwest Data Collection Instruments

— 91

**Program Name Focus of the** 

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to sele**  **ct students for**  **participation in internsh**  **ips/work****** **prep programs ?**  **What data do you use**

**to determine whether the program is**  **working?**

Write in Drop down

menu (internship, work -prep, Other pleas specify) Drop down

menu (from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) Drop down menu

allowing checking all that apply ( Course failure [e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance [ e.g., missing a num

ber of days of school or class], Student behavior [ e.g., office referrals, suspensions], No criteria, Other Please specify) Drop down menu

allowing checking all that apply (attendance, behavior, course per formance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)

Job Corps AmeriCorps Post - Secondary Enrollment Options program at Kent State University Career and technical education program(s) (e.g.,REL Midwest Data Collection Instruments

— 92

**Program Name Focus of the** 

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to sele**  **ct students for**  **participation in internsh**  **ips/work****** **prep programs ?**  **What data do you use**

**to determine whether the program is**  **working?** culinary arts) Career and technical education class(es) that are not part of a program Other

— please describe:

8. Does your school offer student college prep aration programs (such as AVID or Gear Up)? o Yes o No (skip to 9 ) a. Which of the following college prep aration programs does the school offer ? In the table below, indicate which programs are available, the number of students who participated during the school year, and the grade level of students who participated .REL Midwest Data Collection Instruments

— 93

**Program Name Focus of the** 

**Intervention #**  **of**

**Participating Students**  **Grade**

**Level(s) of**  **Students Who refers** 

**students for this program?**  **What criterion**

**is/criteria are used to sele**  **ct students for**  **participation in**  **college -**  **prep programs**  **? What data do you use** 

**to determine whether the program is**  **working?**

Write in Drop down

menu (general college prep, financial aid assistance, application process assistance, support for college entrance exams, academics, Other please specify) Drop down

menu (from 1 to 999) Drop

down menu (from grade 9 to grade 12) Drop down

menu allowing checking all that apply (Administrator, Counselor, Parent, Teacher, Data team, Student, [ self ] , Other Please specify) Drop down menu

allowing checking all that apply ( Course failure [e.g., failing algebra], Grade point average [ GPA ], Number of credits, Student attendance [ e.g., missing a num

ber of days of school or class], Student behavior [ e.g., office referrals, suspensions], No criteria, Oth er Please specify) Drop down menu

allowing checking all that apply (attendance, behavior, course performance, core course performance, GPA, credits earned, behavior referrals, behavior suspensions, no criteria, Other please specify)

Early College High Sch ool AVID Gear Up Talent DevelopmentREL Midwest Data Collection Instruments

— 94 

### B **- 2. Draft Protocol** 

**— School Data Culture Survey**  This survey

will be administered to both treatment and control schools to estimate the impact of EWIMS on school level outcomes . This attachment provides additional information on the draft protocols to measure School D ata Culture. Table B - 1 presents the reliability of the subscales and Key Dimensions of School Data Culture for principals and the reliability of the subsca les and scales measuring principals’ report of School Data Culture. Cronbach’s alpha was calculated for each subscale and scale. It is generall y accepted that Cronbach’s alphas above 0.70 indicate adequate internal consistency among the items in a scale. The reliability

statistics of each scale and the number of survey items in each scale are shown below. The alpha statistic for all scales and subscales measuring School Data Culture (Assessment/Instructional Context, Supports for Data Use, Working with Dat a, and Instructional Responses to Data) is above 0.70, suggesting adequate reliability . Although ED’s contractor will revise items to fit the needs of the EWIMS evaluation, these Cronbach’s alphas are promising for the internal consistency

of the revised S chool Data Culture Scale.REL Midwest Data Collection Instruments

— 95 **Table B -**  **1. Reliability Estimates of Principals’ Key Dimensions of**

**Data Use Scales and Subscales Cronbach’s Alpha**  **Key Dimension**  **Scale Subscales within each Key** 

**Dimension scale S** **u r** **v e** **y I** **t e** **m N**  **General Principal**  **Responses Reading** 

**Instruction Specific**  **Math**

**Instruction Specific**  **Context 32**  **0.87 0.87** 

Assessment/Instructional Context 23 0.82 0.81 State, District, and School Data Culture 9 0.84 **Supports for Data Use**  **51 0.92** 

Data Infrastructure 8 0.82 Organizational Supports 31 0.90 Staffing/Human Resources 12 0.93 **Working with Data**  **42 0.97**  **0.97**

Individual Principal Attention to Data 17 0.93 0.93 Principal Collaboration Around Data 3 0.79 Principal - Teacher Collaboration 9 0.86 Principal - Coach Collaboration 4 0.83 Principal - Parent Collaboration 3 0.75 0.75 Principal - Student Collaboration 6 0.87 **Instructional Responses**  **24 0.96**  **0.97**

Establish/Adjust Groupings 4 0.80 0.80 Adjust Lesson Plans 6 0.88 0.88 Provide Supplemental Resources to Targeted Students 3 0.81 0.81 School - Level Instructional Response 11 0.92 0.93 **Barriers to Data Use 13**  **0.76** Sam ple size: N = 212 Principals. Note: The Supports for Data Use scale and Barriers to Data Use scale did not contain survey items that were reading or mathematics specific. School - Level Instructional Response is a subscale specific to the Principal Survey.REL Midwest Data Collection Instruments

— 96 

## Paperwork Burden Statement According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid Office of Management and Budget (OMB) control number. The valid OMB control number for this information collection is **XXXX -** **XXXX** . The time required to complete this information coll ection is estimated to average 60 minutes per response . This information collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202

– 4651. If you have comments or concerns regarding the status of your individual submission of this form, write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208 - 5500. 

### Data Use Survey Items 1. How much do you agree or disagree with the following statements about your district’s priorities about using data? **Strongly Disagree**  **Disagree Agree**  **Strongly Agree**  a. The district sets clear, consistent goals for schools to use data for school improvement. ⃝ ⃝ ⃝ ⃝ b. District staff provide information and expertise that support the data use efforts at my school. ⃝ ⃝ ⃝ ⃝ c. The district’s data use policies help us address student needs at our school. ⃝ ⃝ ⃝ ⃝ d. The district has designated adequate resources (e.g. time, staff, money) to facilitate teachers’ use of data. ⃝ ⃝ ⃝ ⃝ 2. Now consider the professional climate in your school. To what extent do you agree or disagree with each of the following statements? **Strongly Disagree**  **Disagree Agree**  **Strongly Agree**  a. Educators in this school are continually learning and seeking new ideas. ⃝ ⃝ ⃝ ⃝ b. Educators are engaged in systematic analysis of student performance data. ⃝ ⃝ ⃝ ⃝ c. Educators in this school approach their work with inquiry and reflection. ⃝ ⃝ ⃝ ⃝ d. Assessment of student performance leads to changes in the curriculum. ⃝ ⃝ ⃝ ⃝ e. Educators in this school regularly examine school performance on assessments. ⃝ ⃝ ⃝ ⃝ 3. To what extent do you agree or disagree with the following statements about your campus leaders/ administrators (e.g. principal, assistant principal)?REL Midwest Data Collection Instruments

— 97

The **campus leaders/administrators** ... **Strongly Disagree**  **Disagree Agree**  **Strongly Agree**  a. Encourage teachers to make decisions based on data. ⃝ ⃝ ⃝ ⃝ b. facilitate conversations about using student data ⃝ ⃝ ⃝ ⃝ c. Commit resources to help teachers interpret and use the student data ⃝ ⃝ ⃝ ⃝ d. Place too much emphasis on using the student data ⃝ ⃝ ⃝ ⃝ 4. Please read the following response options and pick the one that best reflects the degree of student data support provided by the campus leaders/administrators in your building: o **Not at all supportive:** Does not make the student data a priority. There is limited discussion of the data with s taff. o **Not very supportive:** Occasional support for the student data use in faculty and staff discussions, but administrators do not see using the student data as central to the school’s mission. o **Supportive:** Administration is supportive of teacher’s effort s, speaks positively about the student data with staff, problem - solves obstacles to using the student data, uses the sy stem themselves. o **Very supportive:** Administration is a “cheerleader” for the student data, effectivel y supports staff use of the system, and sees it as central to school mission. 5. Over the last 6 weeks, about how often did your school have scheduled meeting time to **As Departments**  **As Grade**  

** Level Teams** **As a Whole**

**Staff** a. Review student data (e.g., state test scores, student work, curriculum -based unit tests) Dropdown menu a Dropdown

menu a Dropdown

menu a b. Discuss student achievement by subgroup (e.g. students with disabilities, ELL/LEP, gender, race/ethnicity) Dropdown menu a Dropdown

menu a Dropdown

menu a c. Discuss individual student achievement Dropdown menu a Dropdown

menu a Dropdown

menu a d. Discuss and share instructional strategies Dropdown menu a Dropdown

menu a Dropdown

menu a e. Meet with an instructional coach Dropdown menu a Dropdown

menu a Dropdown

menu a f. Meet with a data coach Dropdown menu a Dropdown

menu a Dropdown

menu a a Dropdow

n menu choices are About once a week, 1

–2 times per month, 1

–2 times per quarter, My school does provide time for this, and NA)REL Midwest Data Collection Instruments

— 98

6. In the last 12 months, how much professional development have you attended that focused on identifying students at risk for dropping out? **No Emphasis**  **Minor**

**Emphasis Moderate** 

**Emphasis Major** 

**Emphasis** a. Linking student data to classroom practice ⃝ ⃝ ⃝ ⃝ b. Analyzing student data ⃝ ⃝ ⃝ ⃝ c. Identifying strengths or weaknesses of your teaching ⃝ ⃝ ⃝ ⃝ d. Incorporating student data into lesson planning ⃝ ⃝ ⃝ ⃝ e. Using data to target interventions for low performing students ⃝ ⃝ ⃝ ⃝ f. Using data to target interventions for high performing students ⃝ ⃝ ⃝ ⃝ g. Understanding how data can be used to guide instruction ⃝ ⃝ ⃝ ⃝ h. Understanding of how data can help you build on your students’ strengths ⃝ ⃝ ⃝ ⃝ 7. In the past 6 weeks, approximately how man y hours did you engage in analyzing or reviewing assessment data in the following way s? **None 1 Hour** 

**or Less 2 to 4** 

**Hours 5 to 7** 

**Hours 11 to 15** 

**Hours More** 

**Than 15 Hours**  a. Independently ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ b. With teachers ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ c. With other administrators (principal, vice principal) ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ d. With a content - area coach (e.g., mathematics or reading coach) ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ e. With a data coach (e.g., RDA accountability coordinators) ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ f. With students ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ g. With parents ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ h. In a multidisciplinary team ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ i. In a team that is made up of teachers, a dministrators, and other roles ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ j. Other ⃝ ⃝ ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 99

**8.** Does your school have a person who acts as a coordinator, facilitator, or coach to support data use? o No, we do not have a person in this role o Yes, this person is also a teacher at our school o Yes, this person is a school administrator o Yes, this person is from the district’s central office o Yes, other if “other,” please specify :

______________________________________________________________ 9. In your opinion, how much expertise in assessment data does your data coach have? o Non e at all o A slight amount of expertise o A moderate amount of expertise o A great amount of expertise 10. Please indicate how much assistance the data coach provides to your school in the following activities: **No assistance**  **Limited**

**assistance Moderate** 

**assistance Substantial** 

**assistance Don’t** 

**know** a. Organizing existing data, including assessment data ⃝ ⃝ ⃝ ⃝ ⃝ b. Analyzing and interpreting data ⃝ ⃝ ⃝ ⃝ ⃝ c. Helping staff develop their own capacity to analyze data ⃝ ⃝ ⃝ ⃝ ⃝ d. Writing school improvement plans ⃝ ⃝ ⃝ ⃝ ⃝ e. Suggesting curricular programs that could help our school ⃝ ⃝ ⃝ ⃝ ⃝ f. Suggesting specific strategies to address our needs (e.g., strategies for parent involvement, for English learners) ⃝ ⃝ ⃝ ⃝ ⃝ g. Acting as a facilitator during discussions among school staff about school improvement strategies ⃝ ⃝ ⃝ ⃝ ⃝ h. Working with the community or parents to explain the assessment process or results ⃝ ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 100

11. How man y teachers at your school have the following characteristics? **None Some**  **About Half**  **Most All**  a. The ability to use data from student assessments ⃝ ⃝ ⃝ ⃝ ⃝ b. The ability to analyze trends in individual student performance over time ⃝ ⃝ ⃝ ⃝ ⃝ c. The ability to analyze trends in classroom - level performance over time ⃝ ⃝ ⃝ ⃝ ⃝ d. The ability to translate data into knowledge about student strengths and weaknesses ⃝ ⃝ ⃝ ⃝ ⃝ e. The ability to make instructional changes based on data ⃝ ⃝ ⃝ ⃝ ⃝ 12. To what extent does your school use the following student data? **Do not use at all**  **Use**

**minimally Use** 

**moderately Use** 

**extensively Don’t** 

**know** a. Number or percent of students flagged for attendance problems in your school or class ⃝ ⃝ ⃝ ⃝ ⃝ b. Behavior/conduct/discipline referrals for individual students ⃝ ⃝ ⃝ ⃝ ⃝ c. Scale scores on achievement tests across your class ⃝ ⃝ ⃝ ⃝ ⃝ d. Individual student’s scores on achievement tests ⃝ ⃝ ⃝ ⃝ ⃝ e. Scores for subgroups of students (e.g., students with disabilities, ELL/LEP, gender, race/ethnicity) in your school or class ⃝ ⃝ ⃝ ⃝ ⃝ f. Student transcripts/course grades ⃝ ⃝ ⃝ ⃝ ⃝ g. Student enrollment in advanced courses ⃝ ⃝ ⃝ ⃝ ⃝ h. College entrance exam scores (PSAT, SAT, ACT) ⃝ ⃝ ⃝ ⃝ ⃝ i. Student information, including parent contact information and student program enrollment ⃝ ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 101

13. To what extent does your school use the following student data? **Not made available**  **this way Used** 

**minimally Used** 

**moderately Use** 

**extensively** a. Percentage of students scoring at or above the proficiency level on state tests ⃝ ⃝ ⃝ ⃝ b. Percentage of students scoring at or above the proficiency level on local tests ⃝ ⃝ ⃝ ⃝ c. Results for subgroups of students (e.g., students with disabilities, ELL/LEP, gender, race/ethnicity) ⃝ ⃝ ⃝ ⃝ d. Results for the school as a whole. ⃝ ⃝ ⃝ ⃝ e. Results for each grade level ⃝ ⃝ ⃝ ⃝ f. Results for each classroom ⃝ ⃝ ⃝ ⃝ g. Results on specific topics or skills (e.g., computation, applications, word recognition, grammar) ⃝ ⃝ ⃝ ⃝ h. Changes or trends in the school’s results across years ⃝ ⃝ ⃝ ⃝ i. Student information, including parent contact information and student program enrollment ⃝ ⃝ ⃝ ⃝ 14. Please tell us approximately how much time you spent reviewing or anal yz ing student data in the last 6 weeks. o None o 1 hour or less o 1

–2 hours o 2

–4 hours o 4

– 5 hours o 5 hours or more **15.** In the past 6 weeks, approximately how man y hours did you engage in reviewing the following types of data

— not including training time?

**None 1 Hour** 

**or Less 1 to 2** 

**Hours 2 to 4** 

**Hours 4 to 5** 

**Hours 5 or** 

**More Hours**  a. Any type of data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ b. School - level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ c. Classroom - level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝ d. Student - level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 102

**16.** How much do you use student data to do the following things? **Do Not Use at All**  **Use**

**Minimally Use** 

**Moderately Use** 

**Extensively** a. Diagnose learning problems ⃝ ⃝ ⃝ ⃝ b. Identify individual students who need remedial assistance ⃝ ⃝ ⃝ ⃝ c. Identify individual students who need acceleration ⃝ ⃝ ⃝ ⃝ d. Identify areas in which you want to strengthen your own content knowledge or teaching skills ⃝ ⃝ ⃝ ⃝ e. Identify individual students with attendance problems ⃝ ⃝ ⃝ ⃝ f. Identify individual students with behavior problems ⃝ ⃝ ⃝ ⃝ g. Identify individual students who are off track for graduation ⃝ ⃝ ⃝ ⃝ h. Assist students in setting individual performance goals ⃝ ⃝ ⃝ ⃝ **17.** How much influence do student data have on the following? **No Influence**  **Minor**

**Influence Moderate** 

**Influence Major** 

**Influence** a. Determining a student’s grouping for instruction ⃝ ⃝ ⃝ ⃝ b. Determining a student’s class placements ⃝ ⃝ ⃝ ⃝ c. Adjusting goals for student learning ⃝ ⃝ ⃝ ⃝ d. Adjusting pacing in areas where students encountered problems ⃝ ⃝ ⃝ ⃝ e. Adjusting use of textbooks and instructional materials ⃝ ⃝ ⃝ ⃝ f. Evaluating school initiatives/programs ⃝ ⃝ ⃝ ⃝ g. Identifying students to be retained at the same grade level ⃝ ⃝ ⃝ ⃝ h. Identifying students for tutoring or other supplemental instruction ⃝ ⃝ ⃝ ⃝ i. Identifying professional development needs ⃝ ⃝ ⃝ ⃝ j. Evaluating teachers ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 103

18. On the basis of student data, how much have the teachers in your school changed the following aspects of their teaching? **No change Minor**  **change Moderate** 

**change Major** 

**change** a. The curriculum content they teach ⃝ ⃝ ⃝ ⃝ b. Their expectations for student performance ⃝ ⃝ ⃝ ⃝ c. The instructional strategies they employ ⃝ ⃝ ⃝ ⃝ d. The types or mix of assessments they use to evaluate students ⃝ ⃝ ⃝ ⃝ 19. To what extent do you use student data as the basis for the following actions? **Not at All To a** 

**Minimal Extent**  **To a**

**Moderate Extent**  **To a Great**

**Extent** a. Recommending attendance monitoring for students ⃝ ⃝ ⃝ ⃝ b. Recommending behavior monitoring for students ⃝ ⃝ ⃝ ⃝ c. Recommending reading interventions for students ⃝ ⃝ ⃝ ⃝ d. Recommending mathematics interventions for students ⃝ ⃝ ⃝ ⃝ e. Recommending social - emotional/behavior interventions for students ⃝ ⃝ ⃝ ⃝ f. Contacting truancy officers ⃝ ⃝ ⃝ ⃝REL Midwest Data Collection Instruments

— 104 

### B **- 3**  **. Archival Records Protocol**  The following section describes our approach to collect extant data primarily from the state of Ohio. The following data sharing processes and requests will also apply to data collected from districts and schools. **Context in Ohio** The Ohio Department of Ed ucation (ODE) has a process established for researchers to use when requesting student level data. The following text is pulled directly from the Ohio department of Education website 22 and describes the state’s approach to sharing student level data: **Reques ting Student Level Data**  To protect student privacy, the Family Education Records Privacy Act of 1974 (FERPA) restricts the public release of student- level data by state education agencies. FERPA does include circumstances under which the Ohio Department o f Education (ODE) is authorized to release data from an education record. Ohio Revised Code § 3301.12 grants to the State Superintendent of Public Instruction the authority to approve the release of this information to research organizations conducting stu dies on behalf of ODE. Research institutions wishing to conduct evaluative studies using student- level data must file a Memorandum of Understanding (MOU) with ODE (see below for template). The MOU will, among other things, detail the expectation that the research organization will assume responsibility for the ongoing protection of the education records upon receipt of student level data. To learn more about the commitments required of researchers in ODE’s MOU, please review the “Understanding ODE’s MOU” d ocument below. ODE’s MOU requires researchers to provide a detailed list of data elements requested for the purpose of the research study. ODE collects student data via the Education Management Information System (EMIS). Prior to submitting an MOU, pleas e review the EMIS Manual to compile an initial list of requested data elements to complete the MOU Appendix A. The EMIS Manual is available to the public on ODE’s Education Management Information System website. Contact Information Matthew Cohen, Chief Research Officer Office of Polic

y & Research Ohio Department of Education 25 S. Front Street, 7th Floor Columbus, OH 43215 614 - 752 - 8729 Matt.Cohen@education.ohio.gov

22 http://education.ohio.gov/GD/Templates/Pages/ODE/ODEDetail.aspx?page=3&TopicRelationID=3&ContentID=109 164&Content=133034REL Midwest Data Collection Instruments

— 105

Our process will take into account Ohio’s approach to sharing data w ith researchers. Table B - 2 describes the steps in the data sharing process between Ohio and REL Midwest.

**Table B -**  **2 .**  **Ohio and REL Midwest Data Sharing Action**  **Method/Mode Key Ohio Staff**  **Key REL Midwest Staff**  **Date** First contact with ODE Half an hour

phone conversation Matthew Cohen,

Chief Research Officer Ann - Marie Faria,

Nicholas Sorensen, Mindee O’Cummings, Jessica Heppen April 2013

AIR will draft data sharing MOU and share with ODE E - mail

communication Matthe w Cohen,

Chief Research Officer Ann - Marie Faria May 2013 ODE will provide feedback on data sharing MOU E - mail

communication Matthe w Cohen,

Chief Research Officer Ann - Marie Faria May 2013 AIR will finalize and sign MOU E - mail

communication Matthe w Cohen,

Chief Research Officer Ann - Marie Faria January 201 4 ODE will sign MOU E - mail

communication with hard copies mailed to AIR Matthew Cohen,

Chief Research Officer, Michael L. Sawyers, Acting State Superintendent of Public Instruction Ann - Marie Faria January 201 4

AIR will establish secure FTP site for data sharing Create the

Secure FTP N/A Ann - Marie Faria January 201 4 AIR will submit second data request E - mail

communication Matthew Cohen,

Chief Research Officer Ann - Marie Faria January 2014 ODE will provide second round of student - level data Secure FTP

established by AIR for use in the EWIMS study Matthew Cohen,

Chief Research Officer Ann - Marie Faria January 2014

AIR will submit third data request E - mail

communication Matthew Cohen,

Chief Research Officer Ann - Marie Faria Ju ne 2014 ODE will provide third round of student - level data Secure FTP

established by AIR for use in the EWIMS study Matthew Cohen,

Chief Research Officer Ann - Marie Faria Ju ne 2014REL Midwest Data Collection Instruments

— 106

**Action Method/Mode**  **Key Ohio Staff Key REL Midwest**  **Staff Date**  AIR will draft annual report to ODE summarizing research conducted with the data Email

communication with attachment Ann - Marie Faria, Nicholas Sorensen, Jessica Heppen December 2014

AIR will submit fourth data request E - mail

communication Matthew Cohen,

Chief Research Officer Ann - Marie Faria January 2015 ODE will provide fourth round of student - level data Secure FTP

established by AIR for use in the EWIMS study Matthew Cohen,

Chief Research Officer Ann - Marie Faria January 2015

AIR will submit fifth and final data request E - mail

communication Matthew Cohen,

Chief Research Officer Ann - Marie Faria Ju ne 2015 ODE will provide fifth round of student - level data Secure FTP

established by AIR for use in the EWIMS study Matthew Cohen,

Chief Research Officer Ann - Marie Faria Ju ne 2015

AIR will draft annual report to ODE summarizing research conducted with the data Email

communication with attachment Ann - Marie Faria, Nicholas Sorensen, Jessica Heppen December 2015

Provide ODE with any material designed for public dissemination, (copy of final report) Email

communication with attachment Matthew Cohen,

Chief Research Officer, Michael L. Sawyers, Acting State Superintendent of Public Instruction Ann - Marie Faria,

Nicholas Sorensen, Mindee O’Cummings, Jessica Heppen December 2015

Destroy all data upon completion of project (after creating RUF files for the U.S. Department of Education) Matthew Cohen, Chief Research Officer Ann - Marie Faria,

Nicholas Sorensen December 2016

**Establishing a secure system to share data (SFTP).** One key aspect of using extant data is the secure transfer of data between agencies. AIR will create a secure FTP and will share the log - in information with ODE. A draft of the directions that will be shared with ODE from AIR are included below.REL Midwest Data Collection Instruments

— 107 **WEB HOSTING SERVICES "SFTP Client Guide" Public Information**  **DESCRIPTION:** This SOP outlines the procedures to use Web Services’ SFTP service. **PROCEDURE:** A. Open the email received from WS Engineer with access credentials to the SFTP site.REL Midwest Data Collection Instruments

— 108

B. Follow Steps 1 - 7 in the email: 1. Open your internet browser and type http://sftp.airws.org or http://sftp.astprojects.org in the address field, depending on which website y ou were given in the email.

A. Verify whether Java is installed/enabled in your browser: B. If Java is not installed/ enabled in y our browser, click “Install”.REL Midwest Data Collection Instruments

— 109

C. Accept license agreement, and choose all defaults:REL Midwest Data Collection Instruments

— 110

D. Reboot your computer, and repeat step #1.

2. Click the “Connect Button” on the webpage:REL Midwest Data Collection Instruments

— 111

3. A dialog box will pop up, asking for a username and password. Refer to the email for these credentials. T yp e the given credentials EXACTLY as they appear in the email. Click OK. **EMAIL**

**DIALOG BOX**REL Midwest Data Collection Instruments

— 112

4. Upon a successful connection y ou should see the following on the webpage:

The figure above shows a message that you have successfully connected to the SFTP server. On the right, it also shows a list of virtual directories located on the remote sys tem, and a field to type the directory path (encircled). On the left is the directory listing for your local machine.REL Midwest Data Collection Instruments

— 113

5. Refer to step #5 in the email for the name of the specific directory created for y our project. T ype this name EXACTLY as it appears in the instructions, or find it in the directory listing and navigate manually using your mouse.REL Midwest Data Collection Instruments

— 114

6. Double click the subfo lder contained in the virtual directory . In this subfolder, you can upload/download files or create more folders, depending on the read/write permissions granted for yo ur SFTP account.

7. Use the arrows between the two windows to move files back and fort h between your local machine and the SFTP folder in the remote server. Press the “right” arrow button to upload a highlighted file from your machine to the SFTP folder, and the “left” arrow button to download a file from the SFTP folder to your machine.

8. Click the “Disconnect” button when finished.REL Midwest Data Collection Instruments

— 115

**Data Sharing Agreements/MOUs.** A second key component of leveraging extant data for research purposes are the MOUs and data requests between the two organizations. We will request data at 5 time points: baseline data (September 2014), end of Semester 1 Year 1 (January 2014) end of Seme ster 2 Year 1 (June 2014), end of Semester 1 Year 2 (January 2015) end of Semester 2 Year 2 (June 2015). The data requests to be shared between AIR and ODE will follow the structure of the following structure: Dear Matthew Cohen, Chief Research Officer, My name is Dr. Ann - Marie Faria and I am writing to submit our request for student data for all the students in the EWIMS Impact Study . This data request includes students in 72 schools in Ohio. This is the first of five data requests that we will submit ov er the next two years. Today we are requesting all data for the fall of the 2013 - 2014 school year. Below is a list of the ty pes of information that we need, with specific data formats and time periods indicated: 1. **Attendance data:** a. Attendance status (present/ absent by day and first week enrollment status ) for the first 5 days of the school year for 2013 - 14 school year). b. Attendance data (day s enrolled, day s present, day s absent, and percent of day s present) for the first 30 day s of the school year for 2013 - 14 c. Cumulative number of credits attempted and earned by each student that will count toward graduation, to date as of January 2014 . **2. School enrollment data:**  a. School enrolled in ( school name and school ID number) at the beginning and end of the semester for e ach semester of 2013 - 14 b. Student grade level in 2013 - 14 c. Enrollment status data ( school name, exit reason/ still enrolled, and exit date and receiving school as applicable) for 2013 - 14 **3. Disciplinary incidents:**  a. All disciplinary

incidents ( suspensions and other school disciplinary reports) for 2013 - 14 4. **Student special education and ELL identification** : a. Special education data ( special education status, identification date, and disability categories if applicable ) for 2013 - 14 b. English language learner data ( ELL statu s, identification date, and level if applicable ) for 2013 - 14REL Midwest Data Collection Instruments

— 116

5. **Student and family contact information:**  a. Contact info ( address, phone number, primary contact person) as of September 2013 Thanks so much for all your help. And, as alway s, let us know if you have any questions. Ann - Marie Faria, Ph.D. 1000 Thomas Jefferson St., NW Washington DC 20009 202 403 5356 afaria@air.orgREL Midwest Data Collection Instruments

— 117



### B **- 4**  **. Protocol for Collecting Data Stored in the EWS tool** 

**in Treatment Schools** To facilitate the transfer of EWS tool s from treatment schools to the EWIMS evaluation team, we will use the same FTP procedure as outlined in Attachment B - 3 . The table below describes the timeline of activities between treatment schools and the evaluation team during the two ye ars of the study . **Action Method/Mode**  **Key Ohio Staff Key REL**  **Midwest Staff Date**  AIR will establish secure FTP site for data sharing Create the

Secure FTP N/A Ann - Marie Faria January 2014 AIR will email FTP secure log in information to Treatment schools E - mail

communication January 2014, Ju ne 2014, January 2015, Ju ne 2015 Treatment schools will upload populated EWS tool s to the secure FTP E - mail

communication Data Teams in

Treatment Schools Ann - Marie Faria January 2014, Ju ne 2014, January 2015, Ju ne 2015REL Midwest Data Collection Instruments

— 118

**Establishing a secure system to share data (SFTP).** One key aspect of transferring populated EWS tool s between schools and the evaluation team is the use of a secure transfer of data between agencies. AIR will create a secure FTP and will share the log - in information with each treatment school. A draft of the directions that will be shared with each treat ment school from AIR are included below.

**WEB HOSTING SERVICES "SFTP Client Guide" Public Information**  **DESCRIPTION:** This SOP outlines the procedures to use Web Services’ SFTP service. **PROCEDURE:** A. Open the email received from WS Engineer with access credentials to the SFTP site.REL Midwest Data Collection Instruments

— 119

B. Follow Steps 1 - 7 in the email: 1. Open your internet browser and type http://sftp.airws.org or http://sftp.astprojects.org in the address field, depending on whi ch website you were given in the email.

a. Verify whether Java is installed/enabled in y our browser: b. If Java is not installed/ enabled in your browser, click “Install”.REL Midwest Data Collection Instruments

— 120

c. Accept license agreement, and choose all defaults: **Web Services Page 2 of 6**

d. Reboot your computer, and repeat step #1.REL Midwest Data Collection Instruments

— 121

2. Click the “Connect Button” on the webpage:REL Midwest Data Collection Instruments

— 122

3. A dialog box will pop up, asking for a username and password. Refer to the email for these credentials. T yp e the given credentials EXACTLY as they appear in the email. Click OK. **EMAIL**

**DIALOG BOX**REL Midwest Data Collection Instruments

— 123

4. Upon a successful connection y ou should see the following on the webpage:

The figure above shows a message that you have successfully connected to the SFTP server. On the right, it also shows a list of virtual directories located on the remote sys tem, and a field to type the directory path (encircled). On the left is the directory listing for your local machine.REL Midwest Data Collection Instruments

— 124

5. Refer to step #5 in the email for the name of the specific directory created for y our project. Ty pe this name EXACTLY as it appears in the instructions, or find it in the directory listing and navigate manually

using your mouse.REL Midwest Data Collection Instruments

— 125

6. Double click the subfolder contained in the virtual directory. In this subfolder, you can upload/download files or create more folders, depending on the read/write permissions granted for yo ur SFTP account. **Web Services Page 5 of 6**

7. Use the arrows between the two windows to move files back and forth between y our local machine and the SFTP folder in the remote server. Press the “right” arrow button to upload a highlighted file from your machine to the SFTP folder, and the “left” arrow button to downl oad a file from the SFTP folder to y our machine.

Click the “Disconnect” button when finished.