Supporting Justification for OMB Clearance of an 

Evaluation of the Early Warning and Intervention 

Monitoring System Under the Regional Educational 

Laboratory Program (REL)

OMB Clearance Request, Part B

July 2013

Submitted to: Submitted by:

U.S. Department of Education American Institutes for Research

Institute of Education Sciences 1000 Thomas Jefferson Street NW

555 New Jersey Ave. NW, Rm. 308 Suite 200

Washington, DC 20208 Washington, DC 20007-3835

1120 East Diehl Road, Suite 200

Naperville, IL 60563-1486

866-730-6735

www.relmidwest.org

This publication was prepared for the Institute of Education Sciences (IES) under contract ED-IES-12-C-0004 

by Regional Educational Laboratory Midwest, administered by American Institutes for Research. The content 

of the publication does not necessarily reflect the views or policies of IES or the U.S. Department of Education,

nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. 

government. The publication is in the public domain. Authorization to reproduce in whole or in part for 

educational purposes is granted.

07/16
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—2

CONTENTS

Page

Study Background....................................................................................................................... 4

1. Respondent Universe and Sampling Methods ........................................................................ 6

2. Description of Procedures for the Collection of Information ............................................... 10

3. Description of Procedures for Maximizing Response Rates................................................. 29

4. Description of Tests, Procedures, and Methods.................................................................... 30

5. Contact Information for Statistical Consultants.................................................................... 31

References..................................................................................................................................... 34

Attachment A. Recruitment Materials.......................................................................................... 36

A-1. Advance Letter (E-Mail Text)........................................................................................... 37

A-2. Study Overview for Schools and Districts........................................................................ 38

A-3. Letter From IES Endorsing the Study............................................................................... 40

A-4. Nonresponse Follow-Up (E-Mail Text)............................................................................ 41

A-5. Telephone School Screening Protocol/Interview.............................................................. 42

A-6. Telephone District Screening Protocol/Interview............................................................. 50

A-7. Final Agreement Form (School MOU)............................................................................. 53

A-8. Final Agreement Form (District MOU)............................................................................ 65

A-9. Final Agreement Form (State MOU)................................................................................ 73

Attachment B. Data Collection Instruments................................................................................. 77

B-1. Draft Protocol—School-Level Survey (Treatment Contrast and Data-Informed 

Allocation of Dropout Prevention Interventions for Students)................................................. 78

B-2. Draft Protocol—School Data Culture Survey................................................................... 94

B-3. Archival Records Protocol .............................................................................................. 104

B-4. Protocol for Collecting Data Stored in the EWS tool in Treatment Schools................. 117
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—3

CONTENTS (continued)

TABLES AND FIGURES

Page

Tables

Table 1. Potential Respondent Universe of Public Schools In Ohio.............................................................7

Table 2. Power Analyses: Minimal Detectable Effect Sizes for Each Binary Student Outcome ................8

Table 3. Recruitment Process and Respondent Universe for the EWIMS Impact Study ...........................12

Table 4. Mapping of Recruitment Screening Protocol Items to Topics......................................................14

Table 5. Data Sources, Data Elements, and Timeline for Collection..........................................................17

Table 6. Mapping Web-Based School Survey Items to Constructs............................................................22

Table 7. Mapping Web-Based School Survey Items to Constructs............................................................23

Table 8. Mapping Treatment Contrast Survey Items to Constructs............................................................25

Table 9. Schedule of Activities...................................................................................................................28

Table A-1. Major Tasks of the Project........................................................................................................60

Table A-2. Schedule and Key Activities for Early Warning Intervention and Monitoring System 

Implementation ...........................................................................................................................................61

Table A-3. Major Tasks of the Project........................................................................................................71

Table B-1. Reliability Estimates of Principals’ Key Dimensions of Data Use Scales and Subscales.......95

Table B-2. Ohio and REL Midwest Data Sharing ....................................................................................105

Figures

Figure 1. Minimal Detectable Effect Size for Continuous Student-Level Outcomes.................................10

Figure A-1. Early Warning Intervention Monitoring System Implementation Process..............................54

Figure A-2. Early Warning Intervention Monitoring System Implementation Process..............................66
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—4

SUPPORTING JUSTIFICATION FOR OMB CLEARANCE OF AN 

EVALUATION OF THE EARLY WARNING AND INTERVENTION 

MONITORING SYSTEM UNDER THE REGIONAL EDUCATIONAL 

LABORATORY PROGRAM (REL)

OMB CLEARANCE REQUEST, PART B

Study Background

The U.S. Department of Education (ED) requests clearance for the recruitment materials and 

data collection protocols under the OMB generic clearance agreement (OMB Number [IES to 

complete]) for activities related to the Regional Educational Laboratory Program (REL). ED, in 

consultation with American Institutes for Research (AIR), is planning a two-part evaluation of 

the Early Warning and Intervention Monitoring System (EWIMS), consisting of an impact study

and an implementation study. OMB approval is being requested for a multimode data collection 

and analysis of a group of schools, students, and staff members in Ohio public schools. The 

impact study consists of data collection from the state of Ohio, districts, and schools. The 

implementation component consists of data collection from participating schools. Specifically, in 

this OMB clearance package, ED is requesting clearance for the following data collection 

approaches:

 Recruitment materials for all participating districts and schools

 Extant administrative records data collections from the state of Ohio, school districts 

within Ohio, and treatment and control schools within Ohio

 The transfer of data from treatment schools to the evaluation team via populated Early 

Warning System (EWS) tools

 A Web-based survey of school leaders in treatment and control schools

The implementation study will include additional forms of data collection (satisfaction survey, 

interviews with a EWIMS data team member, monthly logs of EWIMS data team meetings)

however, the ED is not seeking approval for these measures of implementation because they are 

part of the typical EWIMS intervention and present no burden to participants. 

As detailed more fully in the project description that follows, this impact study (designed as a 

cluster randomized controlled trial) will focus on student outcomes spanning multiple domains of 

school success (student risk status, scores on graduation tests, persistence and progress in school, 

and being on track at the end of ninth grade) and will examine whether the EWIMS model has an 

impact on intermediate outcomes in schools, including the schools’ data culture1and data-

 

1 Data culture is a school’s general approach toward using data to inform educational decision making and includes 

contextual factors (e.g., the assessment and instructional context), supports for data use (e.g., professional 

development or structured time to review data), working with data (e.g., frequency and depth of data use), and 

responses to data (e.g., assignment of interventions to students). 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—5

informed allocation of dropout prevention interventions. The implementation study will focus on 

schools’ experience with implementation, the extent to which schools faithfully implement the 

EWIMS model and the interventions provided to students identified as at risk by the EWS tool.

The purpose of the project is to assess the implementation and impact of EWIMS, a data tool and 

process for implementing a system of data-driven decision making. Developed by the National 

High School Center, EWIMS provides a means of systematically and reliably identifying 

students at risk for dropping out of high school. The proposed study is a two-year school-level 

randomized controlled trial (RCT) to examine the impact of implementing EWIMS on school 

processes and student outcomes. 

Impact Study

The focus of the impact study will be to assess the effectiveness of EWIMS on student and 

school outcomes. For that reason, the study will address the following research questions: 

1. What is the impact of the EWIMS model on outcomes for students in schools, including:

a. indicators of student risk?

b. scores on graduation tests?

c. persistence and progress in school?

d. predicted probability of on-time graduation ?

In addition, REL Midwest will conduct exploratory research analyses for both students and 

schools.

2. What is the impact of EWIMS on outcomes for subgroups of students, including:

a. students who receive free or reduced-price lunch (FRPL)? 

b. students who are English language learners (ELLs)?

c. students with Individualized Education Plans (IEPs)?

d. students with baseline indicators of risk for attendance, course failures, and/or 

behavior?

3. What is the impact of EWIMS on other school outcomes, including:

a. data-informed allocation of dropout prevention interventions for students?

b. school data culture (including the context, supports for data use, working with data, 

and responses to data)?

The exploratory student subgroup analyses will address research questions 1a–1d that will assess 

whether the impact of EWIMS on student outcomes differs for key subgroups of students: 

students who receive free or reduced-price lunch (FRPL), students who are English language 

learners (ELLs), students with Individualized Education Plans (IEPs), and students with initial 

risk (with one or multiple risk factors) in the fall 2013 semester, preceding random assignment 

and implementation. We acknowledge that some students may be included in more than one of 

these analyses because they may be members of more than one subgroup. 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—6

The exploratory school-level impact questions are intended to understand how early adoption of 

EWIMS may cause initial changes in how schools use data to identify at-risk students and 

respond with interventions.

To minimize burden on participating districts and schools, this study draws heavily on extant 

data to address the study’s research questions outlined earlier in this statement. Student- and 

school-level baseline and outcome data for the impact study will be obtained from multiple 

sources, including the Ohio Department of Education (ODE), the EWS tool, and school and district 

administrative data. 

Implementation Study

REL Midwest also will conduct an implementation study to describe treatment schools’ 

experiences with adoption and early implementation of EWIMS, and the extent to which there is 

a difference between the intervention and business as usual in control schools. The 

implementation study will address the following research questions:

1. To what extent do treatment schools faithfully implement the EWIMS model?

2. To what extent does business-as-usual practice in control schools include the use of data 

for identifying at-risk students (treatment contrast)?

3. What are the specific interventions provided to students identified as “at risk” by the 

EWIMS model in treatment schools? 

The remainder of Part B addresses the following: respondent universe and sampling methods 

(though no sampling methods will be used for this randomized controlled trial); description of 

procedures for maximizing response rates; description of tests, procedures and methods; and 

contact information for statistical consultants and key staff.

1. Respondent Universe and Sampling Methods

This section describes the respondent universe and sampling methods for recruitment.

To test the impact of EWIMS on student outcomes, ED’s contractor will recruit and randomly 

assign high schools in Ohio to use or not use the EWIMS model. The recruitment strategy will 

focus on both districts and schools. ED’s contractor will conduct recruitment both in a top–down 

approach (district then school) and in a bottom–up approach (school then district). The

recruitment strategy is designed to target schools that are relatively large (e.g., more than 150 

Grade 9 students) and that demonstrate a need for improved graduation rates and student 

outcomes (e.g., schools with graduation rates between 25 and 95 percent, with a particular 

emphasis on those schools with graduation rates between 50 and 90 percent). The cutoff of 150 

Grade 9 students was selected to establish sufficient statistical power to achieve a minimum 

detectable effect size of 0.25 or smaller. The sample will consist of Ohio regular public schools 

serving Grades 9–12 that indicate their commitment to implement the EWIMS model and their 

willingness to adhere to all study requirements (i.e., random assignment and the completion of 

data collection activities). Although EWIMS can be implemented in both middle schools and 

high schools, this study will focus on high schools because some key intended outcomes of the 

EWIMS, such as whether students are on track for graduation in terms of course credits and 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—7

grades, are relevant only for high school students. Moreover, recruiting and studying both middle 

school and high school outcomes would exceed available resources.

ED’s contractor will target schools with at least 150 students in Grade 9 with estimated 

graduation rates between 25 and 95 percent to ensure that the study can feasibly detect an impact 

on student outcomes. Taking these two inclusion criteria together (more than 150 Grade 9 

students and graduation rates between 25 and 95 percent), there is an estimated pool of 321 

potential schools for recruitment in the EWIMS Impact Study, as shown in Table 1. Based on a 

power analysis, the ED expects to need 72 schools to provide adequate power for detecting the 

effects on outcomes.

Table 1. Potential Respondent Universe of Public Schools In Ohio

Total n Treatment Control 

All school districts in Ohio* 609 -- --

All public schools in Ohio serving Grades 9– 12** 858 -- --

Schools with Grade 9 enrollment greater than 150* 369 -- --

Schools with graduation rates between 25 and 95 percent* 601 -- --

Schools with both Grade 9 enrollment greater than 150 and 

graduation rates between 25 and 95 percent

321

Schools in study sample 72 36 36

Source: *Ohio Department of Education (2010–11); **Common Core of Data (2010–11).

Degree of Accuracy Needed

Because this study is focused on examining policy-relevant impacts for at-risk students, some 

continuous outcomes (e.g., number of course failures) will be examined as binary indicators 

(e.g., 1=failed one or more courses, 0=did not fail any courses) for impact analyses. This 

decision is based on extensive research documenting the predictive power of these binary “flags” 

for on-time graduation (Allensworth & Easton, 2005, 2007; Balfanz, Herzog, & Mac Iver, 2007; 

Neild & Balfanz, 2006; Silver, Saunders, & Zarate, 2008). Further, some of these binary

indicators are used explicitly within the EWS tool to flag at-risk students (e.g., attendance flags 

for students who miss more than 10 percent of school days). A comprehensive list of the binary

variables that will be used in the impact models to examine the impact of EWIMS on student 

outcomes is provided in Table 2, along with information about how these variables will be coded 

and our corresponding assumptions for the power analyses.

Power analyses for binary student-level outcomes were conducted using equations derived from 

Optimal Design Plus Empirical Evidence (Version 3.0) (Raudenbush, Spybrook, Congdon, Liu, 

Martinez, Bloom, & Hill, 2011). To estimate the number of schools needed to have adequate 

statistical power (0.80) for detecting differences between students in treatment and control 

schools for the impact analyses, the power analyses assumed an alpha of 0.05 (two-tailed). ED’s 

contractor estimated power for a constant effects blocked cluster-random assignment design with 

the treatment occurring at Level 2. Following Bloom, Hill, Black, and Lipsey (2008), the study is 

powered to be able to detect a minimum detectable effect size (MDES) roughly equivalent to 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—8

0.25 for student outcomes; this effect size is generally considered necessary for an intervention 

to have educational significance. To date there are no studies that document the existence or 

relative size of the impact of implementing an early warning system on student outcomes, thus 

the suggested MDES of 0.25 is a conservative estimate for this study. Specifically, the study is 

powered with enough schools and students within schools to be able to obtain effects of 0.25 or 

below for all student outcomes. Given the expected low incidence of students flagged as no 

longer in school2, the enrollment/persistence outcome requires a minimum of 72 schools and an 

average of 150 students per school to obtain a difference between treatment and control schools 

that is less than or equal to 0.25, using a standardized Cox Index for binary measures. Including 

72 schools and 150 students per school provides sufficient power to detect effects ranging from 

0.09 to 0.17 for the other binary outcomes of interest. It is important to note that the estimated 

MDES for each outcome is sensitive to the assumptions about the 95 percent plausible interval of 

school means (in proportions). Assumed 95 percent plausible intervals of school means for each 

outcome are detailed in Table 2. Based on these analyses, the study requires approximately 72

schools in the study, with half assigned to each condition, to detect an MDES of 0.25 (or smaller) 

for each outcome variable of interest. 

Table 2. Power Analyses: Minimal Detectable Effect Sizes for 

Each Binary Student Outcome

Student Outcome

Minimum

Detectable 

Effect 

Size(MDES) 

(as Cox Index)

Expected Mean 

Proportion by 

Condition3

95% Plausible Intervals

of School Means (in 

Proportions)

Control Treatment Lower 

Bound

Upper 

Bound

Attendance

(1=missed more than 10% of 

school days; 0=did not miss 

more than 10%)

-0.128 0.150 0.125 0.100 0.200

Fail Any Course

(1=failed one or more courses;

0=did not fail any courses)

-0.098 0.250 0.221 0.200 0.300

Fail Any Core Course

(1=failed one or more core

course; 0=did not fail any core 

courses)

-0.108 0.200 0.173 0.150 0.250

Behavior

(1=suspended one or more 

times; 0= not suspended)

-0.174 0.100 0.077 0.050 0.150

 

2 We are basing our school selection on graduation rates, and we know that they do not correspond directly to 

dropout rates that often underestimate the number of students who actually dropout. According to NCES, the Ohio 

graduation rate in 2008-09 was 79.6, while the dropout rate was 4.2. Documented dropout is a low incident event.

3 The MDESs as well as the Expected Mean proportion by condition are based on the contractor’s prior research on 

dropout prevention including the IES funded Check & Connect Impact Evaluation, as well as the REL Midwest 

Dropout Prevention Alliance suite of studies, previously mentioned. 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—9

Student Outcome

Minimum

Detectable 

Effect 

Size(MDES) 

(as Cox Index)

Expected Mean 

Proportion by 

Condition3

95% Plausible Intervals

of School Means (in 

Proportions)

Control Treatment Lower 

Bound

Upper 

Bound

Predicted Probability of 

Graduation

(1=predicted probability less 

than 0.80; 0= predicted 

probability greater than or 

equal to 0.80)

-0.108 0.200 0.173 0.150 0.250

GPA

(1=GPA less than or equal to 

2.0; 0=GPA greater than 2.0)

-0.089 0.300 0.270 0.250 0.350

Grade Retention/Promotion

(1=not promoted to next grade;

0=promoted to next grade)

-0.128 0.150 0.125 0.100 0.200

Enrollment/Persistence

(1=no longer enrolled in 

school; 0=still enrolled in 

school)

-0.244 0.050 0.034 0.020 0.100

Credits Earned

(1=insufficient credits earned;

0=sufficient credits earned)

-0.128 0.150 0.125 0.100 0.200

“On Track” at the End of 

Ninth Grade

(1=off track at the end of ninth 

grade; 0= on track at the end of 

ninth grade)

-0.108 0.200 0.173 0.150 0.250

The treatment group mean proportions presented in Table 2 represent the smallest possible 

departure from the control group mean proportion ED’s contractor expects to observe with 72 

schools, 150 students per school, and the assumed 95 percent plausible interval of school means

in proportions. ED’s contractor conducted extensive sensitivity analyses examining how 

expanding the 95 percent plausible interval affects the number of schools needed to conduct the 

study. These analyses demonstrate that including 72 schools provides sufficient power for 

enrollment/persistence and more than adequate power for other binary outcomes with minor 

deviations to the plausible interval.

To determine the MDES for continuous outcomes (e.g., standardized achievement scores), ED’s 

contractor conducted a power analysis using Optimal Design Plus Empirical Evidence (Version 

3.0) (Raudenbush et al., 2011). Specifically, ED’s contractor examined the MDES for a clusterrandomized trial examining the impact on person-level outcomes (Level 1) with treatment at 

Level 2. These analyses conservatively assumed (a) 72 schools (36 treatment, 36 control); (b) 

150 students per grade; (c) an intraclass correlation of 0.15 (based on prior research conducted 

by the research team examining these continuous outcomes); (d) a Level 1 covariate explaining 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—10

70 percent of the variation in the Level 1 outcomes;4(e) an alpha of 0.05; and (f) power of 0.80. 

Based on these analyses, this study is powered to detect a minimal detectable effect (MDE) of 

0.15 for continuous outcomes as illustrated in Figure 1.

Figure 1. Minimal Detectable Effect Size for Continuous Student-Level Outcomes

ED’s contractor will also will examine two school-level outcomes, including data-informed 

allocation of dropout prevention interventions for students and school data culture. These 

analyses will be tested using intent-to-treat, linear regression models at the school level. The 

study is not sufficiently powered to detect school-level outcomes with effect sizes less than or 

equal to 0.25. For example, using G*Power (Buchner, Erdfelder, & Faul, 1997) and assuming an 

alpha of 0.05, 0.80 power, and 72 schools, a linear regression model with four predictors (three 

blocking strata indicators and one treatment indicator) at the school level is powered to detect an 

effect size 0.85 or larger. Given this large MDES, school level analyses will be considered 

exploratory analyses. 

2. Description of Procedures for the Collection of Information

ED’s contractor, REL Midwest, will manage data collection and ensure quality and timeliness. 

The draft materials and instruments for recruiting, screening and negotiating final memoranda of 

understanding (MOUs) for participation in the study are included in Attachments A-1 through A9. In addition, the draft instruments for data collection for the impact and implementations 

studies are included in Attachments B-1 through B-6. 

The most common source of data for this study (including all estimates of the student-level 

impact) will involve extant data collected by the state of Ohio, school districts, and schools. 

 

4Analyses of continuous outcomes include standardized achievement scores and GPA. These analyses will control 

for prior standardized achievement and/or GPA, which typically account for more than 70 percent of the variance in 

these respective outcomes.

Total number of clusters

E

f

f

e

c

t

S

i

z

e

24 43 62 81 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

 = 0.050 

n = 150 

P= 0.80,= 0.15,R

2

L2= 0.70
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—11

Primary data collection for the impact and implementation studies will only include a Web-based 

school-level survey to measure data-informed use of dropout prevention interventions for 

students and school data culture, and the treatment contrast. 

Data Collection for School and District Recruitment

We will contact all schools that meet the two inclusion criteria—150 or more Grade 9 students and 

graduation rates between 25 and 95 percent—to determine whether they are currently implementing 

or planning to implement any new programming that might conflict with or too closely mirror the 

EWIMS model. Only schools that are not currently using or planning to use an early warning system 

(EWS) tool and process similar to EWIMS will ultimately be invited to participate in the study. We 

acknowledge that many schools report that they use some form of early warning system data (e.g., a 

competing EWS tool made available commercially or publically or a student information system 

(SIS) used to identify students at risk). The two main components of the early warning system 

process that exclude schools are flagging and assigning students to interventions on the basis of 

systematic data review in more than one domain (attendance, discipline, and academics/failures) 

either with or without a data team. 

Schools that have a sophisticated student information system (SIS) but do not actually use the 

tool to flag or assign at-risk students to interventions will still be eligible. Also, schools that only

flag students (generate student-level reports of risk) but do not assign students to interventions or 

monitor progress will still be eligible. Finally, schools that have teams that review data that are 

not specific to dropout prevention, (e.g., teams that meet about benchmark assessment scores or 

school improvement teams) will still be eligible to participate in the study. 

Also, participation in the EWIMS impact study does not preclude schools from adopting new 

EWS-like programs during the course of the 1.5-year implementation. The study will not impose 

any restrictions on the dropout prevention practices that make up the “business as usual” control 

group. However, as an incentive for control schools to participate, the implementation team will 

provide them with the same implementation support in Year 3 as provided to treatment schools 

in Years 1 and 2. This delayed treatment will also help mitigate the risk that control schools will 

adopt an intervention similar to that being tested in this study. Expert panels convened in the 

technical working group (TWG) agree that this approach will limit the risk of control schools 

adopting similar EWS during the 1.5-year implementation period because they are unlikely to 

invest in a new tool during this time period when they are promised a complete tool and 

accompanying technical assistance and professional development beginning in the 2015–2016 

school year. 

The members of the REL Midwest Dropout Prevention Research Alliance will be actively 

involved in recruitment. Our recruitment strategy will focus on both school districts and the high 

schools within districts. Recruitment will be facilitated by clear materials and communication 

about the nature of the study and the requirements for participation, including complying with 

random assignment. Based on prior experience recruiting schools for school-level random 

assignment studies (e.g., Access to Algebra I: The Effects of Online Mathematics for Grade 8 

Students, Teacher and Leader Evaluation Systems Study), the ED’s contractor does not anticipate 

random assignment to be a barrier to participation. To incentivize participation for control 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—12

schools, the implementation team will provide them with the same implementation support in 

Year 3 as provided to treatment schools in Year 1. 

ED’s contractor will involve the state, districts, and alliance members in recruitment to garner 

local support for the study. In addition, to facilitate the transfer of extant data from school 

districts when necessary (see Table 2), REL Midwest proposes to reimburse school districts for 

the time involved in compiling the data through a stipend of $720 per data pull per district. 

Compensating districts will ensure that all districts have ample time allotted for data compilation 

and transmission.5 Based on the willingness of all invited schools and school districts to 

participate in similar projects currently underway via REL Midwest6and continued interest 

expressed by additional districts and schools, ED’s contractor anticipates that a wide variety of 

schools will be interested in receiving the EWIMS model as part of the study. The process for 

recruitment is outlined in Table 3.

District and school recruitment will occur simultaneously. ED acknowledges that districts can act 

as a gateway for access to interested and eligible schools; however, some schools (typically in 

smaller districts) may also have the autonomy to agree to participation without district approval. 

Therefore, in larger districts recruitment will begin at the district level and in smaller districts 

ED’s contractor may first recruit schools and then follow up with districts when necessary. This 

process includes contacting districts to garner district buy-in for the study and prescreening 

interviews at the school level. The following sections describe more fully each of these steps in 

the recruitment process: (1) identifying the pool of schools to be screened, (2) contacting districts 

to gain district support for the study, (3) conducting school-level screening interviews, 

(4) prioritizing schools for recruitment, (5) recruiting eligible schools, and (6) negotiating final 

agreements.

Table 3. Recruitment Process and Respondent Universe for the EWIMS Impact Study

Steps in the Recruitment Process Respondent Universe

Identify pool of schools 

eligible to participate 

in the study

321 eligible schools in Ohio

District first contact (e-mail and/or phone) 250 districts in Ohio

Follow up for nonresponders (districts) 100 districts

 

5ED’s contractor anticipates that it will take district staff approximately 25 hours each year to compile and transmit 

all required student-level administrative data, as described in the Data Sources section. The average salary of the 

responsible individuals is $40/hour, resulting in a stipend of $1,000. 

6 ED’s contractor is currently conducting a series of interrelated projects for the Regional Educational Laboratory 

(REL) Midwest’s Dropout Prevention Research Alliance that examine different aspects of implementing an early 

warning system to achieve the Alliance’s shared goals of improving graduation outcomes and reducing persistent 

disparities in graduation and dropout rates among student subgroups. This work is primarily focused on the state of 

Ohio and includes projects focused on (1) validating early warning indicators; (2) providing technical assistance to 

support the implementation of early warning systems; and (3) the current study- evaluating the impact of using early 

warning systems on school and student outcomes.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—13

District screening 80 districts with eligible schools 

First school contact (e-mail and/or phone) 321 eligible schools in Ohio

Follow-up for nonresponders 128 eligible schools in Ohio (~ 60% of those eligible at 

first contact)

School screening/interview 160 most qualified schools (~ 50% of those eligible at 

first contact)

School visit (face-to-face or virtual) 100 of the 160 highest qualifying schools that are 

interested in study participation

Negotiate final agreements (district and school 

memoranda of understanding [MOUs]) 

72 study schools 

Conduct random assignment 72 study schools 

Identify survey respondents within schools 72 school leaders

Note. If there are issues with identifying and successfully recruiting schools that meet all of the specified 

qualifications, the ED’s contractor will consider relaxing the criteria for inclusion in negotiation with the Institute of 

Education Sciences (IES). 

(1) Identifying the pool of schools to be screened. To identify the pool of eligible and interested 

schools, the contractor will use the Common Core of Data to pinpoint schools within Ohio that 

serve Grades 9 through 12, have more than 150 Grade 9 students, and have graduation rates 

between 25 and 95 percent. ED’s contractor will then reach out to the estimated pool of 321 

schools to identify interest and the presence or absence of an early warning system similar to 

EWIMS. After the pool of schools has been selected based on the previously mentioned criteria, 

ED’s contractor will send an informational e-mail to each school (see Attachment A-1 for the 

text of this e-mail). This e-mail will include notification that the school meets the initial 

eligibility requirements for participation in the study and will contain documents that briefly

describe the study and a letter from IES endorsing the study (see Attachment A-2 for these 

documents). After ED’s contractor sends the e-mail, a recruitment team member will call each 

school to inform it about the study and ask the school to participate in a short telephone 

interview. The recruitment team comprises REL staff who will also subsequently serve on the 

study or implementation team following random assignment. 

(2) Contacting districts to gain district support for the study. ED’s contractor will identify the 

districts in which the 321 eligible schools are located, and begin a wide outreach plan that 

involves e-mails and follow-up phone calls to district-level staff. This “first touch” of 

recruitment at the district level will facilitate our communications with interested and eligible 

schools (see Attachment A-3 for the text of this e-mail). After the pool of districts that represent 

eligible schools has been identified, ED’s contractor will send an informational e-mail to each 

district (see Attachment A-1 for the text of this e-mail). This e-mail will include notification that 

the district meets the initial eligibility requirements for participation in the study and will contain 

documents that briefly describe the study and ED’s contractor and a letter from IES endorsing 

the study (see Attachment A-3 for these documents). After the e-mail is sent, a recruitment team 

member will call each district to inform the district about the study and ask the district to 

participate in a telephone interview. The district-level screening protocol that will be used to 

guide this interview is presented in Attachment A-6.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—14

(3) Conducting district- and school-level screening interviews. The district- and school-level 

screening protocols are designed to verify school size, graduation rate, and use of an early 

warning system. The interviews also allow for early termination of the interview if a school or all 

schools within a district do not meet these eligibility criteria. ED’s contractor will prepopulate 

the screeners for schools based on the Common Core of Data but will also seek to confirm these 

data are correct at the beginning of the screening interviews. ED’s contractor estimates that there 

are 250 districts with eligible high schools and 321 eligible high schools. Of those, ED’s 

contractor estimates that 68 districts and 128 schools will complete the screener (40 percent of 

contacted schools—a reasonable response rate to the initial contact given ED’s contractor’s prior 

experience recruiting for randomized control trials). The items on the screening protocols are 

mapped to the constructs they measure (Table 4). The district and school screening protocols will 

be administered by project staff by telephone interviews and drafts can be found in Attachment

A-5 and A-6.

Table 4. Mapping of Recruitment Screening Protocol Items to Topics

School Screening Protocol

Construct Items

School Characteristics 1–4

Data-Driven Dropout Prevention Efforts 5–14

Data-Use Practices 15–22

District Screening Protocol

Construct Items

District and School Characteristics 1–3

Data-Driven Dropout Prevention Efforts 4–10

(4) Prioritizing schools for recruitment. Among the 128 schools that ED’s contractor expects to 

complete the screener, it is anticipated that about 110 will pass the initial screening based on 

extant data; some will be more appropriate candidates for the study than others. ED’s contractor 

will prioritize its recruitment efforts in schools with: 

 Moderate graduation rates. Although many high schools in Ohio have graduation rates 

above 90 percent, ED’s contractor will focus the recruitment efforts on the 173 schools (out 

of the entire pool of 321) that have at least 150 Grade 9 students and graduation rates 

between 50 and 90 percent. Schools with lower graduation rates (i.e., 25 to 49 percent) may

not have enough available interventions for students identified through the EWIMS process, 

and schools with higher graduation rates (i.e., 91 to 95 percent) may not have enough at-risk 

students to be included in the evaluation. For this reason, ED’s contractor will focus first on 

recruiting these schools which are most likely to benefit from the EWIMS intervention and 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—15

secondarily focus on those schools (with graduation rates between 25-49 percent or between 

91 and 95 percent) that could benefit from EWIMS but that face additional challenges or 

have a more limited need for dropout prevention. 

 Greater interest in the study. Schools that signal to alliance members or ED’s contractor a 

greater interest to participate will be given higher priority. For example, some schools will 

make a greater effort to include multiple administrators in initial meetings to discuss the 

opportunity and/or may be more responsive to e-mail and phone communication from ED’s 

contractor than other schools. Whenever possible, ED’s contractor will also incorporate its 

prior work and knowledge of specific districts to gauge the interest level of districts.

 Feasibility of implementation. Districts that have fewer competing initiatives for dropout 

prevention programming, and specifically the use of an early warning system, will be given 

higher priority. ED’s contractor will also determine whether schools have the capacity to 

assemble an EWIMS team to upload and analyze student-level data and risk indicators.

Considering these criteria while canvassing the total 321 eligible schools will allow REL staff to 

prioritize the schools during recruitment and help ED’s contractor efficiently use resources to 

conduct site visits at eligible and interested schools.

(5) Recruiting eligible schools. Once ED’s contractor has finalized the pool of potentially

eligible schools, face-to-face recruitment site visits with key staff will begin. ED’s contractor 

will begin recruitment by targeting schools that are considered most eligible given the three 

priority areas listed above. The recruitment team members, with input from district personnel, 

will either organize meetings for clusters of schools or will visit each eligible and potentially

interested campus separately.

A school’s eligibility to participate in the study will be reconfirmed in a meeting with the 

principal by using the criteria listed in the previous section (size, graduation rates, use of an EWS 

tool and system, and interest). The implementation team’s experience with implementing 

EWIMS has shown that an important benefit of the study to schools is not just the EWS tool but 

also the supportive technical assistance to implement the EWIMS process. Although treatment 

schools will receive these supports in 2014, control schools too will receive these supports 

beginning in fall 2015. Therefore, each school that agrees to participate will receive the full 

EWIMS implementation (tool, technical assistance) at no cost to the school. ED’s contractor 

believes this delayed treatment model will facilitate school recruitment. 

(6) Negotiating final agreements. After successful recruitment meetings with the 72 study

schools, ED’s contractor will then enter into final agreements through memoranda of 

understanding (MOUs) that detail the roles and responsibilities of study participation. Drafts of 

these final agreements are included in Attachments A-5 and A-6. If necessary, project staff will 

make additional phone calls or visits to build consensus and obtain commitment from principals. 

The principals’ signatures are expected to be gathered shortly after the districts’ MOUs are 

obtained in order to allow random assignment of schools immediately following OMB clearance. 

The timeline for recruiting schools is ambitious; however, ED’s contractor has substantial 

experience conducting large-scale successful recruitment efforts on extremely tight timelines.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—16

(7) Conducting random assignment. Once the exclusion criteria are applied, schools will be 

assigned to treatment and control conditions within blocks based on graduation rates and school 

size. To conduct blocked random assignment ED’s contractor will create clusters of schools

using matching techniques (e.g., Mahalanobis distance), potentially based on their graduation 

rates, size, district, and baseline data culture and will randomly assign half of the schools in each 

block to treatment status and half of the schools within each block to control status. Again the 

key variables to consider during blocking are:

 graduation rates,

 school size,

 district,

 and potentially baseline data culture and/or the sophistication of schools’ baseline data 

systems.

ED and its contractor will finalize the blocking approach post-recruitment and before random 

assignment. The matching techniques that will be used to conduct blocked random assignment 

may also take into account information about schools’ home district and baseline data capacity, 

using data collected during the screening process. 

(8) Identifying respondents for the school surveys. After random assignment and during data 

collection, ED’s contractor will also strategically identify school leaders/administrators to 

complete the Web-based school surveys. In each school ED’s contractor will identify the most 

relevant individual to complete the school survey on data-informed use of dropout prevention 

interventions for students and school data culture. In treatment schools ED’s contractor will 

sample the lead individual from the EWIMS data team. In control schools ED’s contractor will 

recruit the principal, assistant principal, counselor, or other school leader, depending on their 

level of involvement with data use and dropout prevention initiatives in their school.

Data Sources for Impact and Implementation Studies

To understand the effects of the EWIMS model and schools’ experiences with implementation, 

this project will consist of an impact study and an implementation study. The following sections 

describe the data collection for each of these separately. Table 5 provides a summary of each 

data element collected in the schools and associated data sources. 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—17

Table 5. Data Sources, Data Elements, and Timeline for Collection

Data Source Data Element(s) Timeline for Data Collection

ODE School Administrative Data Baseline–Jan. 2014

Student Administrative Data Jan. 2014, June 2014, Jan. 2015, June 2015

District Student Administrative Data Jan. 2014, June 2014, Jan. 2015, June 2015

School 

Administrators

Annual Web-Based Survey May 2014, May 2015

Implementation Case Study: 

Principal Interview1

May 2015

School-Level 

EWIMS Data 

Teams

Populated EWS tools Jan. 2014, March 2014, June 2014, Aug. 2014,

Jan. 2015, March 2015, June 2015

Participation in EWIMS trainings or 

convenings

Jan. 2014, March 2014, June 2014, Aug. 2014, 

Jan. 2015, March 2015, June 2015

Satisfaction with EWIMS trainings 

or convenings

Jan. 2014, March 2014, June 2014, Aug. 2014, 

Jan. 2015, March 2015, June 2015

EWIMS data team monthly logs Monthly from Jan. 2014 –June 2014; Aug. 

2014–June 2015

Interview of one EWIMS Team 

Member1

May 2014, May 2015

Note. Bolded data elements are data collection instruments involving burden of study participants.

1. Items adapted from the California Comprehensive Assistance Center (CCAC). 

All student-level data will be collected through extant data. There is no student-level primary

data collection burden on schools. As detailed in Table 5, student-level extant data will be 

obtained from the Ohio Department of Education (ODE), districts, and schools. Protocols for 

transferring the data are outlined in the protocol in Attachment B-3. When feasible, student 

administrative data will be collected from ODE to reduce the burden on participating schools. In 

addition, ODE provides detailed guidance on the collection and reporting of many of the data 

elements through the Education Management Information System (EMIS). ODE’s EMIS Manual 

includes data definitions, requirements, and procedures for reporting and transferring data to the 

department. For more information, see 

http://www.ode.state.oh.us/GD/Templates/Pages/ODE/ODEDetail.aspx?page=3&TopicRelationI

D=367&ContentID=100905&Content=135642.

Because some of the outcome data (e.g., attendance, course performance) would be available 

only from the participating school districts and schools (not the state), REL Midwest proposes to 

reimburse districts and schools for the time involved in compiling the data through a stipend of 

$720 per school, per data pull (with three scheduled data pulls—June 2014, January 2015, and 

June 2015). It is expected to take district staff approximately 16 hours in year 2 and 32 hours in 

year 3 to compile and transmit all required student-level administrative data, as described in Table

4. The average salary of the responsible individuals is $45 per hour,7thus resulting in a $720 

 

7 The hourly wage rates used to calculate cost are based on previous OMB packages and are consistent with ED’s 

contractor’s knowledge of wage rates at Ohio Department of Education.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—18

stipend in year 2 and a $1,440 stipend in year 3. It is anticipated that compensating schools will 

ensure that all schools have ample time allotted to data compilation and transmission.

Impact Study Measures

The impact study will assess the effects of the EWIMS model on student outcomes, including 

indicators of student risk (e.g., attendance, number of course failures, credits earned, behavior). 

Other outcome measures include measures of student success that are not in the EWS tool, 

including performance on graduation tests, persistence and progress in school, and predicted 

probability of on-time graduation. Data collection activities will focus on students in Grades 9 

and 10 during Year 1 (2013-14) and Grades 9, 10, and 11 during Year 2 (2014-15). All variables 

for the impact study will come from school and district data sources. The impact study will 

obtain the same extant data for all participating schools and use these data to construct the 

outcome measures. 

Baseline Measures. In January 2014, at the beginning of implementation, REL Midwest will 

obtain student-level covariate data from ODE for students in treatment and control schools. Data 

will include demographic characteristics (e.g., race/ethnicity, gender, free or reduced-price lunch 

[FRPL], individualized education program [IEP] and English language learner [ELL] status, and 

parents’ education); prior academic achievement (e.g., GPA, course failures from the previous 

year, state mathematics and reading scores); and previous-year attendance rates. School-level 

baseline data from ODE will include high school graduation rates, average state achievement 

scores in mathematics and reading, and the percentage of FRPL students. 

Outcome Measures. The impact evaluation is designed to determine if the EWIMS model has an 

impact on student outcomes over time, in four domains: (1) student risk status, (2) scores on 

graduation tests, (3) persistence and progress in school, (4) predicted probability of on-time 

graduation. The study will also examine if the EWIMS model has an impact on intermediate 

outcomes in schools, including the ways in which they allocate dropout prevention programming 

and their school data culture. The student-level outcome measures include the following:

 Student Risk Status indicators are binary variables reflecting the presence or absence of Ohio 

validated risk indicators, including attendance (e.g., missing 10 percent or more of 

instructional time), course performance (e.g., one or more course Fs), GPA (e.g., 2.0 or 

lower), behavior8, and “on-track” at the end of 9th grade (a composite measure that classifies 

students as on-track or off-track to graduate based on Grade 9 course credits earned and core 

course failures). The underlying cutoffs will be finalized on the basis of the results of the 

REL Midwest Year 1 validation study (4.1.01)

9

in three districts that examines middle school 

 

8Although it is acknowledged that schools can vary in coding and recording behavior referrals and suspensions, 

these variables are often predictive of student graduation and/or dropout (e.g., Balfanz, Herzog, & Mac Iver, 2007; 

Goldschmidt, & Wang, 1999; Mac Iver, & Mac Iver, 2010; Neild, Stoner-Eby, & Furstenberg, 2008). If the results 

of the Year 1 REL Midwest 4.1.01 study, Local Validation of Early Warning System Indicator Study, demonstrate 

inconsistency in recording and coding of behavior data, the evaluation team will consider dropping this as an 

outcome measure. 

9

The cut offs for indicator flags for use in the EWIM S tool in the impact study will be based on locally validated 

indicators that are based on multiple districts in the state of Ohio. REL Midwest is currently conducting a validation 

study using data from four districts in Ohio. The cutoffs in the tool to flag a student as at-risk or not at-risk will 

incorporate findings from this initial study conducted by REL Midwest in Ohio that validates early warning 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—19

and high school data from two recent graduating classes to identify indicators and indicator 

thresholds that most accurately predict student graduation10. Treatment schools will be 

expected to work directly with these same data for implementation purposes on a monthly

basis. These risk indicators will be available for all treatment students within the EWS tool. 

The evaluation contractor will calculate comparable indicator flags for students in control 

schools based on extant data from the state, district, and schools that are used to populated 

the EWS tool in treatment schools. 

 Scores on graduation tests include standardized test scores on the Ohio Graduation Test 

(OGT; available only in Grade 10 in Ohio) by content area.

 Persistence and progress in school includes a measure of persistence reflecting whether 

students are enrolled or have withdrawn for reasons other than transfer to another district, 

including dropping out. ED’s contractor will examine persistence on an annual basis. ED’s 

contractor will use state-level administrative data from Ohio to identify the pool of all 

possible codes for transfer and dropout. Because ED’s contractor will use state-level data that 

will be consistent across all districts and schools, this will reduce the variability in possible 

codes that identify transfer and dropout. The second measure—of progress—reflects whether 

students have been promoted from grade to grade each year and how many credits they have 

earned, and will be collected by school or district administrative data. This domain also 

includes students’ credit accumulation by semester. 

 Predicted probability of on-time graduation will be a composite variable that incorporates 

data from multiple indicators of risk for each student in both control and treatment schools. 

This value is a model-based estimate of each individual student’s probability of on-time 

graduation that will be derived by applying the coefficient estimates as weights for each risk 

factor11. First, each student will be assigned a binary indicator of having or not having each 

indicator of risk. Next, the model-based estimate of the intercept, or average logit, will be 

used as a starting point for each student. Next, the weight of each indicator12 will be added 

for students with risk factors, yielding individual student logits for on-time graduation that 

are based on their numbers of risk factors. Finally, logits will be transformed into a predicted 

probability of on-time graduation for each student. For example, the equation for calculating 

the predicted probability of on-time graduation for an individual student with the following 

 

indicators based on local Ohio data. The initial study for the Dropout Prevention Alliance was designed to validate 

indicators of student risk for academic failure. These analyses use historical student-level data from four Ohio 

districts to identify middle and high school indicators and indicator thresholds that most accurately predict student 

graduation outcomes (e.g., the number of 8th or 9th grade course failures that most accurately distinguishes 

graduates from non-graduates). 

10 If the results of the REL Midwest Year 1 validation study (4.1.01) are not available in ample time to be used for 

these purposes, indicator flags and respective thresholds will be then based on the recommendations of the National 

High School Center. The National High School Center has synthesized the existing research on early warning 

system indicators in order to provide guidance and technical assistance on developing and implementing early 

warning systems in states, districts, and schools.

11 The coefficients will be derived from analyses conducted under the previously mentioned study conducted by 

REL Midwest in Ohio that validates early warning indicators based on local Ohio data. We will use the findings 

from the validation study to inform the weights for each indicator in the RCT.

12If district-specific analyses are available for any schools that both participated in the 4.1.01 study and the 

randomized controlled trial, the contractor will utilize the most specific weights (coefficients) in the equations that 

predict each student’s probability of on-time graduation. 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—20

risk factors could be:

Student Predicted Logit = (4.1.01 Estimated Intercept) + (Binary Absence Flag * Absence 

Coefficient from 4.1.01) + (Binary Course Failure Flag * Course Failure Coefficient From 

4.1.01) + (Binary Behavior Flag * Behavior Coefficient From 4.1.01) + (Binary Algebra 

Failure Flag * Algebra Failure Coefficient From 4.1.01)

The logits can then be transformed to predicted probabilities using the following equation:

Predicted Probability = (1 / (1+(ln(-logit))

Thus, each student would have an individual predicted probability of graduation that is based on 

multiple indicator flags and the relative weight of each flag derived from the analyses in 4.1.01. 

This predicted probability reflects the likelihood of graduation within four years based on 

continuous and dichotomous measures of the indicator variables, and will be derived from the 

validation analyses conducted in 4.1.01. ED’s contractor will calculate predicted probabilities at 

the end of Grades 9 and 10 after Year 1, and at the end of Grades 9, 10, and 11 after Year 2. The 

predicted probability will then be the outcome in a model that tests the impact of EWIMS on 

students’ likelihood of on-time graduation. 

In addition to the student-level outcomes outlined above, the study will conduct impact analyses 

on the following school-level outcomes:

 Data-informed allocation of dropout prevention programs and interventions will measure 

how schools use dropout prevention programs in both treatment and control schools. The 

goal of this measure is to tap the theorized intermediate school-level outcome of the EWIMS 

model, where data are used to inform which students need interventions and how to assign 

them to specific programs. ED’s contractor will collect these school-level data to inform the 

construction of this measure through annual Web-based surveys from a school administrator 

at each site identified by the evaluation team during recruitment to be knowledgeable about 

the school’s dropout prevention and intervention services. ED’s contractor plans to collect 

this information at the school level rather than the individual student level because studentlevel information about the types of services and microinterventions in which students 

participate is not routinely recorded systematically in most schools. Of course, if the EWIMS 

model is implemented with fidelity in treatment schools, ED’s contractor will have these 

detailed student-level data for students in treatment schools and will leverage these data to 

answer implementation research question 3 (What are the specific interventions provided to 

students identified as “at risk” by the EWIMS model in treatment schools?). However, 

assuming these data are otherwise not available for students in control schools, ED’s 

contractor will not attempt to collect them at the student level because this could potentially

dilute the service contrast between the treatment and control conditions.13 Therefore, ED’s 

contractor will rely on school-level surveys to gauge the effects of the EWIMS model on 

whether and how both treatment and control schools provide dropout prevention supports for 

students. 

School staff who participate in the annual Web-based survey will receive a $30 stipend in the 

form of a gift card, which is aligned with Analytical and Technical Support for Advancing 

 

13That is, collecting this information for all students in control schools could constitute an intervention in itself.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—21

Education Evaluation: How to Put Together an OMB Supporting Statement, Appendix E

(Sloan, Ingels, & Burghardt, 2012). This $30 incentive applies to responses on one survey

that measures three key constructs, two of which are outcomes for the impact study (datainformed allocation of dropout prevention programs and interventions and school data 

culture), and one of which is for the implementation study, described in the next section 

(treatment contrast). The incentives will be administered twice throughout the study, 

accompanying each administration of the survey (May 2014, May 2015).

Attachment B-1 includes a draft survey that will be developed to collect this information. A draft 

of the survey will be included and ED will submit revisions to OMB before the 30-day review. 

The items are designed to solicit information about the type and frequency of dropout prevention 

programs and the strategies to which students may be exposed within and outside of school. 

These items were based on the constructs presented in Dropout Prevention: A Practice Guide

(Dynarski et al., 2008) and Approaches to Dropout Prevention: Heeding Early Warning Signs 

with Appropriate Interventions (Kennelly & Monrad, 2007). These items inquired about 

students’ access to online credit recovery, mentorship programs (including Check & Connect), 

tutoring, internship or school-related work prep programs (e.g., Job Corps, MAAC Project, 

career and technical education classes), college prep programs (e.g., AVID, African American 

and Latino Student Summit, GEAR UP), and school- and community-based extracurricular 

activities (e.g., sports, band, clubs or organizations, student government). In this survey, the 

latter questions (items 2–6) inquire about schools’ implementation of microinterventions,

including the names of specific programs, participation rates, grade level availability, and the 

student selection process. 

ED’s contractor will operationalize the data collected from the online survey into one summary

variable that captures schools’ proclivity to engage in data-informed allocation of dropout 

prevention programs and interventions. First, ED’s contractor will create a score that summarizes 

the extent to which schools use data to assign students to different types of dropout prevention 

interventions and programs (as measured by the school survey), including: 

 Targeted academic interventions 

 Targeted behavioral interventions 

 Attendance/truancy interventions 

 Online content recovery programs 

 Mentoring programs 

 Internship or school-related work-prep programs 

 College prep

ED’s contractor will test the impact of EWIMS on the proportion of students in a school that 

participate in one of the above dropout prevention interventions or programs, as well as how 

often that school uses data to inform the allocation of dropout prevention interventions or 

programs. Table 6 provides a map of the items included in the draft protocol and the constructs 

measured in the survey found in Attachment B-1.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—22

Table 6. Mapping Web-Based School Survey Items to Constructs

Data-Informed Allocation of Dropout Prevention

Constructs Items

Academic interventions 1

Behavioral interventions 2

Attendance/truancy 

interventions 3

Online credit or content 

recovery

4

Student mentoring programs 5

Internship or school-related 

work preparation program 6

College preparation programs 7

 School data culture is another school-level outcome hypothesized to change as a function of 

implementing EWIMS. School data culture encompasses the following four key dimensions 

of school data use:

1. Context includes district and school goals for data use and the professional climate 

around data use in schools. 

2. Supports for data use include the presence or absence of structured time to review data, 

training and professional development around data use, collaboration around data, data 

coaching, and principal leadership.

3. Working with data includes the frequency, duration, mode, and type of data review.

4. Responses to data include how teachers and schools respond to student data, including 

the assignment of general and targeted interventions, determining teacher professional 

development needs, and informing school improvement plans. 

ED’s contractor will use the aforementioned annual Web-based survey to measure school data 

culture in both treatment and control schools. School administrators from treatment and control 

schools will be invited to respond to this school-level survey. This survey includes items adapted

from the Bill & Melinda Gates Foundation–funded Urban Data Study (Faria et al., 2012). While 

the Faria et al. (2012) survey measured specific data use practices around benchmark 

assessments, ED’s contractor adapted these items for use with early warning system data 

structures. Attachment B-2 includes a draft protocol with specific items to be used in the EWIMS 

impact study. Table 7 provides a map of the items included in the draft protocol and the 

constructs measured in the survey found in Attachment B-2. 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—23

Table 7. Mapping Web-Based School Survey Items to Constructs

School Data Culture

Constructs Items

Context 1, 2

Supports for Data Use 3-11 

Working with Data 12-15

Instructional Responses to Data 16-19

Implementation Study Measures

The Dropout Prevention Research Alliance and practitioners more broadly will benefit from a 

clear understanding of what it takes to implement the EWIMS model and, whether or not it 

shows an impact, from clear documentation about the implementation process. The 

implementation study will focus on the following questions: (1) To what extent do treatment 

schools faithfully implement the EWIMS model?; (2) To what extent does business-as-usual 

practice in control schools include the use of data for identifying at-risk students (treatment 

contrast)?; (3) What are the specific interventions provided to students identified as “at risk” by 

the EWIMS model in treatment schools (i.e., Step 5 of the EWIMS process)?; and (4) What are 

the key components of high quality implementation? This information is of critical interest to 

alliance members who may use this information to inform the design or implementation of 

EWIMS models.

Measures of Fidelity. REL Midwest will collect EWIMS data tools from treatment schools twice 

yearly (January and June) as a primary means for assessing the degree to which schools are 

faithfully implementing the EWIMS model. The data to be collected to measure fidelity of 

implementation include14:

 The extent to which schools upload data into the EWS tool (percentage of student 

demographic and administrative data records uploaded, measured by the EWS tool)

 The extent to which identified students have documented interventions recorded in the 

tool (measured by the EWS tool)

 Participation in EWIMS trainings/convenings (measured by attendance sheets from 

trainings collected by the implementation team and submitted to the evaluation team)

 Satisfaction with EWIMS trainings/convenings (measured by satisfaction surveys)

 Ways in which each treatment school implements the intervention (measured by

interviews with an EWIMS team member)

 

14REL Midwest and REL Appalachia are working together to coordinate data collection across two studies that will 

document the implementation of EWS in two states (Virginia and Ohio); thus, the contractor has incorporated some 

of the REL Appalachia items into its survey of treatment contrast and the interviews protocols to use with EWIMS 

teams to document fidelity of implementation.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—24

To document implementation, the evaluation team will have access to populated tools and will 

determine whether student data are regularly loaded, and whether students identified as at risk 

within the EWS tool also have documented interventions recorded in the tool. ED’s contractor

will calculate the percentage of students with complete data in the tool in comparison with the 

extant data available for those students collected from the state, district, and school. 

ED’s contractor will also collect information about participation in EWIMS-related trainings and 

convenings that will be used to address the first research question of the implementation study

(To what extent do treatment schools faithfully implement the EWIMS model?). First, the 

implementation team will share attendance and sign-in sheets for all EWIMS trainings and 

convenings. Second, the implementation team will facilitate the data collection of satisfaction 

surveys similar to the process by which college professors receive course satisfaction surveys. 

One person attending the training will volunteer to collect all satisfaction surveys; place them in 

a self-addressed, stamped, and sealed envelope; sign the back of the envelope; and mail the 

satisfaction surveys to the evaluation team. The satisfaction surveys will be adapted from those 

developed by the California Comprehensive Assistance Center (CACC), which documented 

implementation of an EWIMS pilot project conducted as a collaboration between the National 

High School Center and the California Department of Education. The satisfaction surveys will be 

administered after each technical assistance activity (i.e., technical and implementation trainings, 

community of practice, on-site support, and responsive technical assistance), and will gauge 

participants’ satisfaction with the quality, relevance, and utility of the service provided. The 

satisfaction surveys are part of the typical implementation of the EWIMS intervention and 

therefore are not included in our estimates of burden for treatment schools.

In addition, the evaluation team will collect qualitative information via interviews about the ways 

in which each treatment school implements the EWIMS model. The interviews are part of the 

typical implementation of the EWIMS intervention and therefore are not included in our 

estimates of burden for treatment schools. 

Measures of Treatment Contrast. To understand the extent to which business-as-usual practices 

in control schools include the use of data for identifying students at risk of dropping out, ED’s 

contractor will use the aforementioned Web-based survey to collect information about the 

presence or absence of an early warning data-based system in control schools. In this survey, 

items will inquire about current usage of an EWS tool or other systematic use of data to identify

students who are at risk of academic failure, disengagement, and/or dropping out of high school. 

If schools are using data for these purposes, follow-up questions explore the types of indicators 

used, including thresholds and time frames applied, ability to customize indicators, and who has 

access to the data. Attachment B-1 includes a draft protocol of the items measuring the treatment 

contrast. Table 8 provides a map of the items included in the draft protocol and the constructs 

measured in the survey found in Attachment B-1.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—25

Table 8. Mapping Treatment Contrast Survey Items to Constructs

Treatment Contrast Survey

Constructs Items

Use of an early warning system 1, 2

Use of an EWS tool 3-5

Types of EWS indicators 6, 7

Application of EWS data 8, 9

Presence of an EWS or data use team 10-11

Measures of Specific Intervention Information. The EWIMS model is a school-level 

macrointervention, through which schools can use data to efficiently allocate dropout prevention 

and academic support interventions to targeted students. Its overall impact will be driven in part 

by the effectiveness of the specific interventions assigned to students and the appropriateness of 

the “fit” between student needs and assigned interventions. Although ED’s contractor will not 

directly assess the “fit” between students and microinterventions, nor the impact of 

microinterventions on student outcomes, ED’s contractor will collect information on the specific 

microinterventions allocated to students in treatment schools. To collect these data, ED’s 

contractor will use the populated EWS data tools from each treatment school to carefully record 

the microinterventions assigned to students. Information on specific interventions assigned to 

students will only be collected in treatment schools, as noted above in the description of schoollevel measures of allocation of dropout prevention programming. However, ED’s contractor will 

collect a school-level measure of allocation of dropout prevention programming in each control 

school to attempt to understand whether control schools offer students interventions.

15

Estimation Procedures

Student-Level Impact Analyses. ED’s contractor will conduct student-level impact analyses at 

the end of Year 216 that assess the effects of EWIMS on measures in the four student outcome 

domains—indicator flags, academic achievement, persistence and progression, and predicted 

probability of on-time graduation— and ED’s contractor will conduct exploratory analyses that 

examine the impact of EWIMS on subgroups of students and two school-level outcomes. These 

analyses align to the research questions about the impact of EWIMS as stated above.

The analytic strategy for questions about impacts on student-level outcomes will compare 

students in schools randomly assigned to receive the EWIMS intervention with students in 

 

15Because asking school staff to document the student-level service contrast information for control schools is likely 

too much of a burden, and too similarly resembles the actual implementation of EWIMS, the contractor will only 

collect these data in control schools at the school level (through Web-based surveys of school administrators).

16 We will conduct the main impact analyses on student outcomes at the end of the 2014-2015 school year. 

However, we also plan to conduct these same models as exploratory analyses at the end of Year 1 (summer 2014). 

These are not confirmatory analyses because we do not hypothesize an impact of EWIMS on student outcomes after 

only six months of implementation.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—26

schools in the control group. The data for this study are hierarchical (students are nested within 

schools); therefore, units at the same level are not statistically independent. To allow the effects 

of student- and school-level factors to be modeled and adjustments made for the nonindependence of observations within clusters, this study will use hierarchical linear modeling to 

estimate the treatment effect on the student-level outcomes of interest. In all analyses of studentlevel outcomes, ED’s contractor will estimate intent-to-treat two-level models, where students 

are the Level 1 unit and schools are the Level 2 unit. ED’s contractor will define the intent to 

treat student sample as “all students included enrolled on the 20th day of school.” Some students 

within the at-risk populations may not enroll during the first week of school, but do enroll within 

the first 20 school days. This cut-off date is also policy relevant because schools must report 

their total enrollment to the state on the 20th day of school to inform state and annual funding. 

For all of the binary student-level outcomes, ED’s contractor will use a hierarchical generalized 

linear model with a Bernoulli sampling distribution and logit link function. For all continuous 

student-level outcomes, ED’s contractor will use hierarchical linear models. ED’s contractor will 

run models separately by year and cohort. 

Most of our student level outcome variables are dichotomous. For models with binary outcomes 

ED’s contractor will use a logit link function to transform the dependent variable into the odds of

achieving a particular outcome (i.e., being on track or off track, persisting). ED’s contractor will 

assume a constant treatment effect across blocks and include dummy blocking variables at Level 

2 (school graduation rate and school size dummies).17 At the student level, ED’s contractor will 

include baseline student demographic and prior achievement characteristics, including 

race/ethnicity, gender, FRPL, ELL, IEP status, and prior standardized reading and mathematics 

test scores.18 The treatment indicator will be included at Level 2 to indicate that a school was 

randomly assigned to the treatment group. These student- and school-level variables are expected 

to be correlated with the outcomes of interest, and will be used as covariates in all analytic 

models for this study to improve the precision of the estimates of impact. These models will 

estimate both probability and odds of showing a particular characteristic or indicator of risk for 

students in the treatment and control groups. 

Exploratory Student Subgroup Analyses. To test for a differential impact of EWIMS for 

subgroups of students, ED’s contractor will use the same impact models outlined above for the 

student outcomes, but additionally include a cross-level interaction between the treatment status 

of participating schools and an indicator for students in each subgroup tested. ED’s contractor

will conduct these analyses for student impact research questions 1a through 1d. ED’s contractor

will address four key subgroups of students: students who are English language learners (ELLs), 

students with Individualized Education Programs (IEPs), students who receive free or reducedprice lunch (FRPL), and students who are flagged as at risk with one or multiple risk factors as 

calculated using administrative data from the participating schools during the semester preceding 

implementation (fall 2013). ED’s contractor will therefore categorize students as “initially at 

risk” (or not) based on data from the fall 2013 marking period preceding implementation and use 

this subgroup categorization throughout the remainder of the study for each cohort. ED’s 

 

18 Because we will not impute missing data, student test scores will only be used as a covariate in impact analyses if 

missing data is less than five percent.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—27

contractor will not redefine this subgroup later, because changes in flagged status in subsequent 

semesters may be a consequence of the impact of the intervention (e.g., students who are flagged 

receive high-quality micro-interventions that may decrease their likelihood of being flagged in 

later semesters). 

Because the theory of action does not include a priori hypotheses about the differential impact of 

EWIMS on the four subgroups to be tested with four separate exploratory analyses, ED’s 

contractor will adjust for multiple comparisons given the potential for alpha inflation. 

Specifically, the p-value associated with the cross-level interaction coefficient in each of the four 

subgroup analyses must be less than .0125 to be considered statistically significant (adjusting for 

four post hoc exploratory tests). In addition, it is possible that students in one subgroup are also 

included in other subgroups tested under separate exploratory analyses. For this reason, ED’s 

contractor will report the number and percentage of students in each subgroup that are also 

included in the other three subgroups. Understanding the overlap across subgroups will be 

important for interpreting a differential impact of EWIMS on one or more subgroups of students.

Exploratory School-Level Impact Analyses. To analyze the impact of EWIMS on school-level 

outcomes and answer research questions 3a and 3b, ED’s contractor will conduct intent-to-treat, 

linear regression models on the school outcomes of interest. These analyses will be treated as 

exploratory because they are underpowered to detect effects and are not the primary focus of the 

EWIMS model. These models will include strata blocking variables and the indicator for whether 

a school was randomly assigned to the treatment or control group.

Sensitivity Analyses. ED’s contractor also will conduct a series of sensitivity analyses to test the 

robustness of the observed estimates from the benchmark impact models. These analyses include 

fixed effects models with school dummy variables, models with no covariates, and models that 

use the continuous or ordinal data on which the binary indicators used in the primary analyses 

are based (e.g., using all values for GPA rather than the binary indicator of above 2.0 vs. 2.0 or 

lower).

Study Milestone Timeline

This array of data will permit ED’s contractor to gather a deep understanding of program 

implementation and how the intervention differs from business as usual in control schools. The 

above data collections are specified in the following study timeline:
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—28

Table 9. Schedule of Activities

Activity Expected Date

Draft Office of Management and Budget (OMB) package July 2013

Documentation of institutional review board approval April 2013

Submit 60 day FRN July 2013

Submit 30 day FRN September 2013

Draft proposal accepted by ED March 2013

Final proposal approved by ED July 2013

Expected OMB clearance data January 2014

Complete school recruitment January 2014

Obtain signed district/school memoranda of understanding from all 

participating schools

January 2014

Complete random assignment of participating schools January 2014

Academic year (AY) 1, 2013-14 

Collect baseline data from participating districts January/February 2014

Pilot the Web-based survey and interview protocols June 2014

Treatment schools implement EWIMS with Grades 9 and 10 January 2014–June 2014

Collect EWS tool data from treatment schools (AY1, quarter 3) March 2014

Collect EWS tool data from treatment schools (AY1, quarter 4) June 2014

Collect end-of-year student-level data from participating districts June 2014

Conduct interviews with EWIMS teams at treatment schools May 2014

Administer survey measure of treatment contrast at participating schools May 2014

Academic year (AY) 2, 2014-15 

Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014–June 2015

Collect EWS tool data from treatment schools (AY2, quarter 1) November 2014

Collect midyear student-level data from participating districts January 2015

Collect EWS tool data from treatment schools (AY2, quarter 2) January 2015

Collect EWS tool data from treatment schools (AY2, quarter 3) March 2015

Collect EWS tool data from treatment schools (AY2, quarter 4) June 2015

Collect end-of-year student-level data from participating districts June 2015

Conduct interviews with EWIMS teams at treatment schools May 2015

Administer survey measure of treatment contrast at participating schools May 2015

Academic year 3, 2015-16 August 2015

Control schools implement EWIMS August 2015–June 2016

Submit Making an Impact report (first draft) December 2015

Making an Impact report accepted by ED TBD
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—29

3. Description of Procedures for Maximizing Response Rates 

ED is committed to obtaining complete data for this evaluation. Because the evaluation relies 

heavily on administrative data, the ED’s contractor does not expect response rates to be an issue 

and anticipates a 95-percent response rate for district/school and state archival record requests. A 

key to achieving complete administrative data is tracking the data components from ODE and 

schools/districts with e-mail and telephone contact to the appropriate parties to resolve issues of 

missing or delayed data files (see the protocol for collecting extant data in Attachment B-3).

Also, because ED’s contractor will have direct access to EWS tools in treatment schools, 

response rates will be 100 percent for all applicable data. The completeness of those data will 

vary, however, by the degree to which treatment schools use and implement EWIMS. All 

administrative data files and direct instrument responses received as part of this evaluation will 

be reviewed for consistency and completeness. If a data file has too many missing values or if an 

instrument in the implementation study has too few items completed to be counted as a response, 

AIR staff will seek to obtain more complete responses by e-mail or phone.

Based on its extensive experience with administering surveys in a variety of schools, districts, 

and states, ED’s contractor expects the response rate for the survey to be at least 85 percent.

Although ED’s contractor expects high response rates for the unique data collections related to 

this evaluation (as schools volunteer for this study in order to receive EWIMS for free and 

instruments are developed with sensitivity to respondent burden), nonresponse follow-up will be 

performed to ensure adequate response rates. Follow-up reminders and phone calls will be placed 

to individual respondents in the event that responses are not obtained for Web-based surveys, 

and AIR staff will follow up directly with respondents in the event that face-to-face interviews or 

reflection interviews are not conducted. 

Additional steps will be taken to maximize response rates for the data collected for this study. For 

example, sampled respondents will receive advance communications explaining their role and an 

assurance of confidentiality. Respondents also will be given a contact number to reach ED’s 

contractor with questions and will be informed of the incentive that they will receive (on top of the 

incentive received by all control schools). For example, the study plan includes the following 

incentives:

 $2,880 stipend ($1,440 in year 1 and $1,440 in year 2) per district for time involved in 

compiling outcome data across four extant data collections (January 2014, June 2014, 

January 2015, June 2015)19

 $30 gift card for school administrators who participate in the annual Web-based survey

(May 2014, May 2015)

 $30 gift card for interview participants from EWIMS data teams at each of the 36 

treatment schools (May 2014, May 2015)

 

19 It is expected to take district staff approximately 32 hours in year 2 and 32 hours in year 3 to compile and transmit 

all required student-level administrative data. Estimated stipends assume that the average salary of the responsible 

individuals is $45 per hour.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—30

Finally, ED’s contractor will work with IES to determine the best analytic approach for 

addressing missing data on key outcomes and covariates in impact models estimating the effect 

of EWIMS on student and school outcomes. In the proposed analytic approach, ED’s contractor

will drop any covariates from impact models that have greater than 5 percent missing data and 

will list wise delete cases with missing data on outcome measures. Upon collecting the data, 

ED’s contractor will conduct analyses examining the percentage of missing cases on key

outcomes and covariates and work with the IES to determine whether list wise deletion is the 

best analytic approach or whether ED’s contractor should impute missing data (using mean 

imputation or multiple imputation) to minimize bias associated with attrition or nonresponse.

4. Description of Tests, Procedures, and Methods

Because all student-level outcome data used to estimate the impact of EWIMS on students will 

come from state- , district- , or school-level administrative records and/or directly from the EWS 

tool, there will be no need to test procedures or instruments. However, ED’s contractor will 

invite internal staff at AIR with expertise in leveraging partnerships with states and districts to 

collect extant data to confirm that the data collection protocols are appropriate and do not pose 

any unnecessary burden on state or district staff. 

To measure school-level outcomes, including data-informed use of dropout prevention 

interventions for students, and school data culture, ED’s contractor will use a Web-based survey. 

The survey items will be reviewed by AIR colleagues who were formerly employed as teachers 

and/or school administrators or colleagues with content expertise. These “critical colleagues” 

will look for three things during their review: (1) whether the questions asked are clear, 

understandable/ free of research jargon, and answerable; (2) whether the questions actually

assess the intended constructs (e.g., the degree to which schools are implementing EWIMS as 

intended or implementing an EWIMS-like intervention); and (3) whether the number and type of 

questions are suitable for a 60-minute survey (e.g., not redundant, focused enough to solicit clear 

answers). 

Next, ED’s contractor will pretest the Web-based survey with five school leaders who are 

currently implementing EWIMS in their schools. ED’s contractor will recruit these five

participants from the network of schools currently implementing EWIMS. ED’s contractor will 

ask participants to complete the Web-based survey using a cognitive think- aloud protocol to 

pilot the items and correct any issues with item sets before administering the first wave of 

surveys in spring 2014. School-level staff who participate in the pilot testing will receive a onetime $30 stipend in the form of a gift card, which is aligned with Analytical and Technical 

Support for Advancing Education Evaluation: How to Put Together an OMB Supporting 

Statement, Attachment E. 

The implementation also involves face-to-face data collection through interviews with the data 

team. ED’s contractor will again utilize its network of experts within AIR, and the interview 

protocols will be reviewed by these critical colleagues for clarity, face validity of questions, and 

brevity. ED’s contractor will also pilot the interview protocols with five teachers/school leaders 

who are currently implementing EWIMS in their schools. Again, ED’s contractor will recruit 

five participants form the network of schools currently implementing EWIMS. ED’s contractor

will ask participants to complete the interview and provide feedback on the length and 
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—31

appropriateness of the items to correct any issues with item sets before conducting the first 

interviews. School-level staff who participate in the pilot testing will receive a one-time $30 

stipend in the form of a gift card, which is aligned with Analytical and Technical Support for 

Advancing Education Evaluation: How to Put Together an OMB Supporting Statement, 

Attachment E.

5. Contact Information for Statistical Consultants

ED’s contractor consulted the following experts internally within AIR concerning the 

methodology, study design, and data collection approach/burden:

 Dr. Hans Bos, Vice President, AIR (650-843-8110)

 Dr. Chris Brandt, Senior Researcher, AIR (630-649-6649)

 Dr. Dean Gerdeman, Principal Researcher, AIR (202-403-6223)

 Dr. Mengli Song, Principal Researcher, AIR (202-403-5267)

The following individuals were consulted on the statistical, data collection, and analytic aspects 

of the EWIMS evaluation study through REL Midwest’s Technical Working Group (TWG).

Margaret Burchinal, Ph.D. 

Senior Scientist 

Frank Porter Graham Child Development Institute

Research Professor, Department of Psychology

Adjunct Professor, Department of Biostatistics

University of North Carolina

515 Oakcrest Drive

Chapel Hill, NC 27516-9638

Ph: 919-966-5059

Fax: 919-962-5771

E-Mail: burchinal@unc.edu

Thomas Cook, Ph.D.

Joan and Sarepta Harrison Chair in Ethics and Justice

Professor of Sociology, Psychology, Education and Social Policy

Faculty Fellow, Institute for Policy Research

Northwestern University

2040 Sheridan Road

Evanston, IL 60208

Ph: 847-491-3776

E-Mail: t-cook@northwestern.edu

Sara Goldrick-Rab, Ph.D.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—32

Associate Professor of Educational Policy Studies and Sociology

Senior Scholar, Wisconsin Center for the Advancement of Postsecondary Education

University of Wisconsin

211 Education Bldg.

1000 Bascom Mall

Madison, WI 53706

Ph: 608-265-2141

E-Mail: srab@education.wisc.edu

Larry Hedges, Ph.D.

Board of Trustees Professor of Statistics and Social Policy

Faculty Fellow, Institute for Policy Research

Northwestern University

2006 Sheridan Road, EV 4070

Evanston, IL 60208

Ph: 847-491-8899

E-Mail: l-hedges@northwestern.edu

James J. Kemple, Ed.D.

Executive Director 

Research Alliance for New York City Schools

Research Professor, Steinhardt School of Culture, Education, and Human Development

New York University

726 Broadway, 756

New York, NY 10003

Ph: 212-998-5463

Fax: 212-995-4049

E-Mail: james.kemple@nyu.edu

Brian Rowan, Ph.D.

Burke A. Hinsdale Collegiate Professor and School of Education Research Professor

Institute for Social Research

University of Michigan

610 E. University Ave.

Room 4112

Ann Arbor, MI 48109-1259

Ph: 734-615-0286

E-Mail: brow@umich.edu

Barbara Schneider, Ph.D.

John A. Hannah Chair and University Distinguished Professor

College of Education and Department of Sociology

Michigan State University

Erickson Hall

620 Farm Lane, Room 516
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—33

East Lansing, MI 48824

Ph: 517-432-0188

E-Mail: bschneid@msu.edu 

ED’s contractor also consulted with the REL Midwest Dropout Prevention Research Alliance 

members listed below to gather feedback on the design and measures to be used in the study.

 Cherie Mourlam, Assistant Superintendent, Washington Local Schools (419-473-8222)

 Sue Zake, Executive Director, Ohio Department of Education (419-720-8999)

 Amy Szymanksi, Consultant, State Support Team Region 1 (419-833-6771)

 Melissa Ramirez, Assistant Principal, Findlay City Schools (419-425-8257)

 Jay Wollenburg, Principal, Ohio Virtual Academy (866-339-9071)

 Mike O’Shea, Springfield High School (419-867-5633)

 Laurie Kruszynski, Data Coordinator, Scott High School, Toledo Public Schools 

(419- 671-4000)

 Jeremy Herr, Principal, McComb High School, McComb Local Schools (419-293-3286)

Key staff that will be directly involved with the data collection and analyses include: 

 Dr. Jessica Heppen, Principal Researcher, AIR (202-403-5488)

 Dr. Mindee O’Cummings, Principal Researcher, AIR (202-403-5254)

 Dr. Ann-Marie Faria, Senior Researcher, AIR (202-403-5356)

 Dr. Nicholas Sorensen, Senior Researcher, AIR (312-283-2318)
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—34

References

Allensworth, E., & Easton, J. (2005). The on-track indicator as a predictor of high school 

graduation. Chicago: Consortium on Chicago School Research. Retrieved from trackhttp://ccsr.uchicago.edu/publications/track-indicator-predictor-high-school-graduation. 

Allensworth, E., & Easton, J. Q. (2007). What matters for staying on-track and graduating in 

Chicago public high schools: A close look at course grades, failures and attendance in 

the freshman year. Chicago: Consortium on Chicago School Research. Retrieved from 

http://ccsr.uchicago.edu/publications/what-matters-staying-track-and-graduating-chicagopublic-schools

Balfanz, R., Herzog, L., & Mac Iver, D. J. (2007). Preventing student disengagement and 

keeping students on the graduation path in urban middle-grades schools: Early

identification and effective interventions. Educational Psychologist, 42(4), 223–235.

Bloom, H. S., Hill, C., Black, A., & Lipsey, M. W. (2008). Performance trajectories and 

performance gaps as achievement effect-size benchmarks for educational interventions. 

New York: MDRC.

Dynarski, M., Clarke, L., Cobb, B., Finn, J., Rumberger, R., & Smink, J. (2008). Dropout prevention: A 

practice guide (NCEE 2008-4025). Washington, DC: U.S. Department of Education, 

Institute of Education Sciences, National Center for Education Evaluation and Regional 

Assistance. Retrieved from 

http://ies.ed.gov/ncee/wwc/pdf/practiceguides/dp_pg_090308.pdf

Faria, A.M., Heppen, J., Li, Y., Stachel, S., Jones, W., Sawyer, K., et al. (2012). Charting 

success: Data use and student achievement in urban schools. Washington, D.C.: Council 

of the great city schools.

Goldschmidt, P, & Wang, J. (1999). When can schools affect dropout behavior? A longitudinal

multilevel analysis. American Educational Research Journal, 36, 715-738.

Kennelly, L., & Monrad, M. (2007). Approaches to dropout prevention: Heeding early warning 

signs with appropriate interventions. Washington, DC: American Institutes for Research,

National High School Center. Retrieved from 

http://www.betterhighschools.org/docs/NHSC_ApproachestoDropoutPrevention.pdf

Mac Iver, M. A., & Mac Iver, D. J. (2010). Keeping on track in ninth grade and beyond: 

Baltimore’s ninth graders in 2007–08. Baltimore: Baltimore Education Research 

Consortium. Retrieved from http://baltimore-berc.org/pdfs/3A%20Final%20report_06-

15-10.pdf

Neild, R. R., & Balfanz, R. (2006). An extreme degree of difficulty: The educational 

demographics of the urban neighborhood high school. Journal of Education for Students 

Placed at Risk 11(2), 123–141.
REL Midwest Justification for OMB Clearance of an Evaluation of EWIMS Under the REL Program—35

Neild, R. C., Stoner-Eby, S., & Furstenberg, F. (2008). Connecting entrance and departure: The 

transition to ninth grade and high school dropout. Education and Urban Society, 40(5), 

543–569. 

Silver, D., Saunders, M., & Zarate, E. (2008). What factors predict high school graduation in the

Los Angeles Unified School District? (California Dropout Research Project Report 14). 

Santa Barbara, CA: University of California. Retrieved from 

http://www.hewlett.org/uploads/files/WhatFactorsPredict.pdf

Raudenbush, S.W., Spybrook, J., Congdon, R., Liu, X., Martinez, A., Bloom, H., & Hill, C. 

(2011). Optimal Design Plus Empirical Evidence (Version 3.0). Available at 

http://www.wtgrantfoundation.org/resources/consultation-service-and-optimal-design. 

Sloan, M., Ingels, J., & Burghardt, J. (2012). Analytical and Technical Support for Advancing 

Education Evaluation: How to Put Together an OMB Supporting Statement. Available at 

http://relpacific.wikispaces.com/file/view/OMB%20guidance%20compiled%205-30-

12.pdf/350956206/OMB%20guidance%20compiled%205-30-12.pdf. 
REL Midwest Recruitment Materials—36

Attachment A. Recruitment Materials

Attachments A-1 through A-9 include recruitment materials for districts and schools to be used 

in the EWIMS impact study. 

A-1 includes the advance letter email text. A-2 includes a 2 page study overview. A-3 includes a 

letter from IES endorsing the study. A-4 includes follow up text for schools that are nonresponsive during recruitment. The advance email letter, the 2-page study overview and a letter 

from IES endorsing the study will be sent to principals at each school and superintendents at 

each district. A-5 and A-6 include screening protocols for schools and districts. These protocols 

will be administered to a relevant stakeholder within each school or district identified by an 

initial point of contact. For example, the study team will first contact principals at each school 

and superintendents at each district though these high-level administrators might request that 

another staff member (e.g., dropout prevention specialist) complete the screening interviews. A-7 

and A-8 include memorandum of understanding that will be signed between districts and REL

Midwest (at AIR) and schools and REL Midwest (at AIR) to formalize study participation and 

commitment. A-9 includes a memorandum of understanding that will be signed between Ohio 

Department of Education and REL Midwest (at AIR)
REL Midwest Recruitment Materials—37

A-1. Advance Letter (E-Mail Text)

Dear < principal or superintendent name>, 

I am contacting you on behalf of the U.S. Department of Education to discuss an opportunity for 

<district or school name> to participate in a study that would provide access to the 

National High School Center’s Early Warning and Intervention Monitoring System 

(EWIMS) including training and technical support at no cost. This project, conducted by the 

Midwest Regional Educational Laboratory at American Institutes for Research (AIR), is aimed at 

preventing high school dropout by using data to identify at-risk students early in high school and 

assign struggling students to interventions that get them back on track for eventual graduation. 

The study will examine the impact of implementing EWIMS on a host of critical student 

outcomes (validated indicators of dropout, course performance, persistence and progress in 

school and likelihood of on-time graduation). In addition, the study will examine the impact of 

EWIMS on how schools make use of data to allocate limited dropout prevention resources.

Attached please find a document that briefly describes the study. Our initial records indicate that 

your district/school may be eligible to participate in this Ohio-based study funded by the U.S. 

Department of Education’s Institutes for Educational Sciences. I would like to schedule a brief 

phone call with you to tell you more about the study, ascertain your interest in taking part, and to 

discuss whether this might be a great opportunity for <district or school name>.

Would you be able to touch base briefly by phone on <day>, <month> <date>? Please feel 

free to reach me by email or at 312-283-2318. 

Thank you and I look forward to speaking with you!

<name of project outreach recruiter>

<name of project outreach recruiter>

Project Outreach

Early Warning and Intervention Monitoring System Study

Midwest Regional Educational Laboratory

American Institutes for Research

20 North Wacker Drive, Suite 1231, Chicago, IL 60606

t: 312.283.2318 | www.air.org

Per the policies and procedures required by the Education Sciences Reform Act of 2002, 

Title I, Part E, Section 183, responses to this data collection will be used only for statistical 

purposes. The reports prepared for this study will summarize findings across the sample and 

will not associate responses with a specific district or individual. ED’s contractor will not 

provide information that identifies you or your district to anyone outside the study team, 

except as required by law. Any willful disclosure of such information for nonstatistical 

purposes, without the informed consent of the respondent, is a class E felony. 
REL Midwest Recruitment Materials—38

A-2. Study Overview for Schools and Districts

Purpose of the Study

One in four students in the United States fails to graduate 

from high school and graduation rates are significantly 

lower for students who are racial or ethnic minorities, 

economically disadvantaged, migrant and/or limitedEnglish proficient, and receiving special education 

services. However, strong foundational research 

highlights critical indicators in ninth and tenth grade that 

powerfully predict whether students are “on track” for 

high school graduation. These indicators, which typically 

focus on student engagement measures (attendance) and 

course performance (grades, credits earned), can be used 

as part of an early warning system to flag at-risk students 

early, assign appropriate interventions, and get students 

back on track for graduation.

As part of Ohio’s Comprehensive Continuous 

Improvement Plan, the state has committed, through 

adoption of the Ohio Improvement Process, to promoting 

student success through data-driven decision making, 

targeted programming, ongoing student monitoring, and 

evaluation of improvement process effectiveness. Use of 

an early warning system is one strategy that schools can 

use to address the Ohio Improvement Process.

One such early warning system is the early warning and 

intervention monitoring system (EWIMS) developed by 

the National High School Center. The EWIMS model 

encourages educators to use data to consider and provide 

appropriate interventions to students, and provides a 

means for monitoring student progress over time. At the 

core of EWIMS is the use of simple tools that encourage 

educators to routinely examine indicators that identify 

students as “off track” and take action. 

Despite increasingly widespread implementation of early 

warning systems by states, districts, and schools, there 

have been no rigorous studies testing the impact of using 

an early warning system to improve student outcomes 

such as staying in school, progressing in school, and 

probability of graduating. There is also little research 

examining how using an early warning system can shape 

a school’s culture for data use including how schools 

allocate their limited dropout prevention resources.

To address this gap, this study will provide the first 

rigorous test of the impact of an early warning system. 

This study, funded by the U.S. Department of Education’s 

Institutes of Educational Sciences, is being conducted by 

the Midwest Regional Educational Laboratory at the 

American Institutes for Research. 

The planned study will examine the impact of EWIMS on 

(1) student outcomes including student risk status for 

dropout, scores on graduation tests, persistence and 

progress in school and likelihood of on-time graduation; 

and (2) school outcomes including how schools allocate 

dropout prevention interventions for students and their 

data-use culture. Participating in this high-profile, largescale study will give your school and district an 

opportunity to access these innovative resources at no 

cost and help inform educational policy in Ohio and at the 

federal level.

Study Design 

The study will take place in approximately 70 high 

schools in Ohio during the 2013–14 and 2014–15 school 

years. All schools that participate in the study will receive 

the early warning system at no cost. Participating schools 

will be assigned by lottery to receive access to the 

EWIMS model (including the tool and technical support 

for implementation) in the 2013-2014 school year or the 

fall of 2015 (following the completion of the study). 

Schools assigned to receive the EWIMS model in the fall 

of 2015 will continue “business as usual” practices to 

identify at-risk students and allocate dropout prevention 

resources through 2014-15.

The study will examine student outcomes for all students 

in grades 9 and 10 during the 2013–14 school year and all 

students in grades 9 through 11 during the 2014–15 

school year for schools that implement EWIMS in 2013-

2014 and those that do not implement EWIMS during the 

study period. All student outcome data will be collected 

from school or district administrative data, the EWS tool

or the Ohio Department of Education; therefore there will 

be no primary student data collection for this study. To 

understand how EWIMS may impact schools, all 

participating schools will be asked to complete an annual 

Web-based survey about data use practices and schools 

implementing EWIMS during the two-year study period 

may be asked to participate in interviews about their 

experiences using the tool.
REL Midwest Recruitment Materials—39

EWIMS Intervention

The EWIMS process is designed to identify students who 

are at risk of dropping out of school and support and 

monitor at-risk students through school-wide strategies 

and targeted interventions. EWIMS is currently in use in 

67 districts in six states, and the tool has been 

downloaded more than 20,000 times from the National 

High School Center’s website. 

EWS tool. At the heart of the EWIMS model is an early 

warning data tool used to flag students as “at risk” based 

on attendance, course performance (grades, credits, GPA), 

and behavior indicators. The tool enables users to identify 

students who are at risk of dropping out of school, record 

assignments to available interventions, and monitor 

students’ response to those interventions. The Excel based 

tool provides a number of reports accessible to users, 

including: 

 School-Level Reports: Graphical reports that show 

trends in student risk status across the school

 Student-Level Reports: Lists of students, grouped by 

indicators of risk and/or assigned intervention 

programs, that include summary information

 Detailed Student Reports: Simple reports that can be 

generated for each individual student and show student 

information, indicators of risk status, and any assigned 

interventions

 Student-Level Intervention Summary Reports: Lists 

of students, their indicators of risk (flagged or not 

flagged), and the number and types of interventions to 

which each student was assigned

EWIMS Implementation Process. In addition to the 

tool, the National High School Center has devised a 

seven-step EWIMS implementation process to support 

implementation. The process guides users to make 

informed decisions about how to use data to support atrisk students and how to continue to monitor their 

progress over time. In addition to focusing on individual 

students, the process guides users to examine the success 

of specific supports or interventions and to examine 

possible systemic issues (school climate) that may relate 

to dropout trends.

EWIMS Seven-Step Implementation Cycle

Initial Criteria for Participation

The study will include approximately 70 high schools in 

Ohio. To qualify, schools must (1) have at least 150 ninthgrade students; (2) a graduation rate between 25 and 95 

percent and (3) not already be implementing an early 

warning system tool for using data to flag at-risk students. 

Study Timeline

Through January 2014, the study team will discuss 

participation with districts and schools that meet the 

initial criteria and conduct on-site or virtual meetings with 

school principals, guidance counselors and dropout 

prevention coordinators. Participating schools will sign 

memoranda of understanding in January 2014 and be 

assigned by lottery to receive access to the EWIMS model 

in January 2014 (treatment group) or the 2015-2016 

school year (“business as usual” control group). Training 

for EWIMS implementation in treatment schools will 

begin in January 2014. Data collection activities will 

continue through but not beyond the 2014–15school year.

For Additional Information

For more information about the study, or to begin a 

conversation about partnering on this project, contact Dr. 

Nicholas Sorensen (nsorensen@air.org or 312-283-

2318) or Dr. Mindee O’Cummings 

(mocummings@air.org or 202-403-5254).

STEP 1—Establish 

Roles & Process

STEP 2—Use the 

EWS tool

STEP 3—Analyze 

EWS data

STEP 4—Interpret 

EWS data 

STEP 5—Assign & 

Provide 

Interventions

STEP 6—Monitor 

Students

STEP 7—

Evaluate & 

Refine EWIMS
REL Midwest Recruitment Materials—40

A-3. Letter From IES Endorsing the Study

UNITED STATES DEPARTMENT OF EDUCATION

INSTITUTE OF EDUCATION SCIENCES

National Center for Education Evaluation and Regional Assistance

Month XX, 2013

Dear District/Principal,

I am writing to introduce you to a new project funded by the U.S. Department of Education (ED) through 

the Regional Educational Laboratory (REL) program called the Early Warning System and Intervention 

Monitoring System (EWIMS) study. This project is aimed at preventing high school dropout by using 

data to identify at-risk students early in high school and assigning students to interventions that help get 

them back on track for on-time graduation. The study will be conducted by REL Midwest at American 

Institutes for Research (AIR). For this study, ED is seeking more than 70 high schools in Ohio to 

participate. In addition to being part of a high-profile, innovative study, participating districts will receive 

free access to the EWIMS model, developed by the National High School Center at AIR, including the 

tool and training and technical support for implementation.

Improving the state’s current graduation rate is a focal point in Ohio’s Comprehensive Continuous 

Improvement Plan and the state has committed, through adoption of the Ohio Improvement Process, to 

promoting student success through data-driven decision making, targeted programming, ongoing student 

monitoring, and evaluation of improvement process effectiveness. In that vein, Ohio districts and schools

are increasingly interested in using an early warning system to identify students who are off track for 

graduation as early as possible. For this reason, the REL Midwest formed the Dropout Prevention 

Research Alliance, focused on improving graduation outcomes and reducing persistent disparities in 

graduation and dropout rates among student subgroups in Ohio. The goal of the alliance is to expand the 

implementation of early warning systems in Ohio districts and to conduct an efficacy study testing their 

impact on student outcomes and school processes. The proposed study is a two-year randomized 

controlled trial (RCT) to examine the impact of implementing an early warning system on school 

processes and student outcomes. The project responds to a need expressed by members of the REL 

Midwest’s Ohio Dropout Prevention Alliance for clear information about the efficacy of early warning 

systems. 

On behalf of the U.S. Department of Education, I encourage your participation, as many districts and 

schools are needed to generate clear and usable results from this evaluation. Thank you in advance for 

your consideration. If you have any questions about the study, please feel free to contact AIR’s project 

directors, Dr. Ann-Maria Faria (afaria@air.org, 202-403-5356) or Dr. Nicholas Sorensen 

(nsorensen@air.org, 312-283-2318). In addition, if I can provide any assistance to you in this matter, 

please feel free to contact me by phone at 202-219-1674 or email at Chris.Boccanfuso@ed.gov. Thank 

you for your time, and we look forward to speaking with you.

Sincerely,

Chris Boccanfuso, Ph.D.

Contracting Officer’s Representative, REL Midwest
REL Midwest Recruitment Materials—41

A-4. Nonresponse Follow-Up (E-Mail Text)

Dear < principal or superintendent name>, 

I am trying to reach you on behalf of the U.S. Department of Education to discuss an opportunity for 

<district or school name> to participate in a study that would provide access to the National High 

School Center’s Early Warning and Intervention Monitoring System (EWIMS) including training 

and technical support at no cost. 

This project, conducted by the Midwest Regional Educational Laboratory at American Institutes for 

Research (AIR) and funded by the U.S. Department of Education’s Institutes of Educational Sciences, is 

aimed at preventing high school dropout by using data to identify at-risk students early in high school and 

assign struggling students to interventions that get them back on track for eventual graduation. 

The timeline for recruiting districts and school to participate in this opportunity is extremely tight. 

Would you be able to touch base briefly by phone on <day>, <month> <date> to discuss whether 

this might be a good opportunity for <district or school name>?

Please feel free to reach me by email or at 312-283-2318. 

Thank you and I look forward to speaking with you!

<name of project outreach recruiter>

<name of project outreach recruiter>

Project Outreach

Early Warning and Intervention Monitoring System Study

Midwest Regional Educational Laboratory

American Institutes for Research

20 North Wacker Drive, Suite 1231, Chicago, IL 60606

t: 312.283.2318 | www.air.org

Per the policies and procedures required by the Education Sciences Reform Act of 2002, 

Title I, Part E, Section 183, responses to this data collection will be used only for statistical 

purposes. The reports prepared for this study will summarize findings across the sample and 

will not associate responses with a specific district or individual. ED’s contractor will not 

provide information that identifies you or your district to anyone outside the study team, 

except as required by law. Any willful disclosure of such information for nonstatistical 

purposes, without the informed consent of the respondent, is a class E felony. 
REL Midwest Recruitment Materials—42

A-5. Telephone School Screening Protocol/Interview

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless such collection displays a valid Office of Management and Budget (OMB) control 

number. The valid OMB control number for this information collection is XXXX-XXXX. The time required 

to complete this information collection is estimated to average 30 minutes per response. This information 

collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or 

suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–

4651. If you have comments or concerns regarding the status of your individual submission of this form, 

write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, 

Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208-5500.

Draft School Screening Protocol

School Screening Protocol

Introductory script: Thank you for the opportunity to meet with you today and discuss the Impact of an 

Early Warning and Intervention Monitoring System (EWIMS) on Student and School Outcomes Study 

funded by the Institutes of Educational Science within the U.S. Department of Education. We are very 

excited about your school’s interest in participating in the study and are looking forward to talking more 

about whether the study might be a good opportunity for your school. Participation in this study is 

voluntary for schools. Today we will first ask you questions about how your school works with students 

who are at-risk for not graduating on time and then we have some questions about your school’s data use 

practices. By data use practices we are referring to things like the professional climate around using data, 

supports for data such as professional development and/or structured time to review data, frequency of 

data use and using data to track progress.

School Name: Principal Name: 

Email: Telephone: 

School Characteristics

1. How many 9th grade students do you anticipate having enrolled in fall 2013? 

2. What was your school’s graduation rate for the 2012-13 academic year? (note- pre populated 

with ODE data if available)

3. What was your school’s graduation rate for the 2011-12 academic year? (note- pre populated 

with ODE data if available)

4. What methodology was used to calculate the graduation rate? 
REL Midwest Recruitment Materials—43

Data-Driven Dropout Preventions Efforts 

For each of the following questions, interviewer will allow the school interviewee to respond to the 

question and the interviewer will select most appropriate response option.

5. Does your high school have an early warning system tool that identifies students who may be at-risk 

of not graduating? 

o Yes, we have an early warning system tool used to identify students who may be at-risk of 

not graduating in my high school.

If yes, what is the name of the system or tool? 

o No, my high school does not currently have an early warning system tool to identify students 

who may be at-risk of not graduating in my high school.

o I'm not sure. 

6. Is your high school using the student information system (SIS) or another other data tool to identify 

students who may be at-risk of not graduating from high school on-time? 

o Yes, we have a tool to identify students who may be at-risk of not graduating in my high 

school.

o No, my high school does not currently have a tool to identify students who may be at-risk of 

not graduating in my high school.

o I'm not sure.

If yes, please specify: 

If the school uses an early warning system, SIS or another other data tool to identify students who 

may be at-risk of not graduating

7. What types of data are used? For each indicator used, what is the level of evidence?

N/A

Evidence-based (e.g., 

based on research 

conducted using data 

not related to the 

school)

Locally-validated 

(e.g., based on 

research using state, 

district or school data)

Based on other 

criteria (specify)

Attendance


REL Midwest Recruitment Materials—44

Behavior 

referrals 

Behavior 

suspensions 

Course 

performance 

Credits earned



Grade Point 

Average (GPA) 

State 

assessment 

results



Other:



Other:



Other:



Other:



8. How do you assign at-risk students to supports and/or interventions? (check all that apply) 

o Student-level indicator data (i.e., evidence or locally validated indicators) 

o Other student level data (e.g., demographics, student performance data )

Specify: 

o Other types of data

Specify: 

o No data-based strategy to assign at-risk students to supports and/or interventions

9. How often does your school review data to identify students at risk of not graduating?

o 4 times per year (once per quarter)

o 3 times per year

o 2 times per year

o 1 time per year

o Other
REL Midwest Recruitment Materials—45

Specify: 

o I'm not sure.

10. Does your school have a team or group of individuals that focuses on students who are identified as at 

risk of not graduating from high school? 

o Yes, we have a dedicated school-based team.

o Yes, we have a school-based team that focuses on students who are identified as at risk of not 

graduating from high school, but it is part of another team.

o Yes, the district has a team that focuses on students who are identified as at risk of not 

graduating from high school.

o No, there is no team at the district or high school levels.

o Other (please specify)

11. Do teachers in your school have access to student data that identifies students at risk of not 

graduating?

o Yes, and all teachers access it regularly

o Yes, and some teachers access it regularly

o Yes, and some teachers access it occasionally

o Yes, but very few teachers actually access it

o Yes, but no teachers actually access it

o No, teachers do not have access to this type of data

o Other (please specify)

12. Please describe the nature of collaboration around student data in your school. 

Probes:

o Are these collaborations generally formal (scheduled meeting times) or informal?

o Who participates in these meetings?
REL Midwest Recruitment Materials—46

o What is the focus of these meetings? (data analysis, sharing/discussing student work, lesson 

planning, ways to improve or modify instruction, focus on individual students, focus on 

problem content areas etc...) 

o What are the goals of these meetings (identify students who need additional support? identify 

school or class-level instructional issues? Create instructional plans)

o Do teachers review other data (e.g., interim assessment data, curriculum-based assessments, 

teacher-created assessments) along with data that identifies at students at risk of not 

graduating?

13. Does your school have a system to monitor students' progress in interventions/supports to which they 

are assigned?

o Yes (please specify)

o No

o Unsure

14. Is there anything else you would like to share with us about how your school approaches dropout 

prevention interventions and strategies? 

Data-Use Practices

Thanks for telling me about how your school identifies students who are at-risk and allocates dropout 

prevention services at your school. We would now like to talk more broadly about data use in your 

school. 

15. Has your district or school set clear and consistent goals or expectations for data use?
REL Midwest Recruitment Materials—47

If yes:

o What are these goals? (get a general idea of their understanding of district priorities)

o How have these goals been communicated to you? (Probe: see if they can share any sample 

documents to show communication?)

o Are there specific areas in which you would like additional guidance?

If no:

o What type of guidance would you like to receive from the district?

16. Has your school received any professional development on data use? (In-service, pre-service, ongoing 

coaching). 

If yes:

o Who delivered the PD (Is this delivered by district and/or school staff)?

o Is the training mandatory?

o What has been the focus (content) of the professional development sessions?

o What was the duration of the PD? (one-time training? is ongoing training provided or 

available?) 

o What is the mode of delivery of the PD? (Is it face to face? Online? video?)
REL Midwest Recruitment Materials—48

o What materials have been provided to support your/your staffs’ learning? (Are there any 

sample materials that we can take a look at?)

If no:

o What type of professional development would be most useful?

17. Aside from professional development, has your district or school provided any resources/tools 

designed to facilitate the use of data at your school for activities such as accessing data, 

analyzing/interpreting data, planning instructional responses, implementing instructional responses 

and/or monitoring data use? (tangible resources/products, staff [e.g. coaches, technical support staff 

etc...], tools [e.g., templates, protocols, books etc...] funding)

o Yes (please describe)

o No

18. Are there structures in place to assess how well your teachers are analyzing or interpreting student 

data? 

o Yes (please describe)

o No

19. Do you feel that the use of student data has led to changes in teacher instructional practices?

o Yes (please describe)

o No
REL Midwest Recruitment Materials—49

20. Is there a formal feedback system in place to determine whether planned instructional responses are 

effective?

o Yes (please describe)

o No

21. Have student data been used for any other school-wide data initiatives or activities (e.g., to inform 

school improvement, evaluate teacher performance, evaluate program/curriculum, make between or 

within school comparisons)?

o Yes (please describe)

o No

22. Is there anything else you would like to share with us about your school’s data use practices?
REL Midwest Recruitment Materials—50

A-6. Telephone District Screening Protocol/Interview

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless such collection displays a valid Office of Management and Budget (OMB) control 

number. The valid OMB control number for this information collection is XXXX-XXXX. The time required 

to complete this information collection is estimated to average 30 minutes per response. This information 

collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or 

suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–

4651. If you have comments or concerns regarding the status of your individual submission of this form, 

write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, 

Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208-5500.

Draft District Screening Protocol

Introductory script: Thank you for the opportunity to meet with you today and discuss the Impact of an Early 

Warning and Intervention Monitoring System (EWIMS) on Student and School Outcomes Study funded by the 

Institutes of Educational Science within the U.S. Department of Education. We are very excited about your district’s 

interest in the study and are looking forward to talking to you about potential participation of high schools within 

your district. Participation in this study is voluntary for schools. Today we will ask you questions about how your 

district supports schools in working with students who are at-risk for not graduating on time. 

District Name: District Contact: 

Email: Telephone: 

School Characteristics

1. How many high school (serving grades 9-12) do you have in your district? 

What are the names of those high schools and their graduation rates for the 2012-13 academic year? (notepre populated with names, if available) _____________________

2. What methodology was used to calculate the graduation rate? 

Data-Driven Dropout Preventions Efforts 

3. Does your district promote the use of a student information system, an early warning system (EWS), or another 

other data tool to identify students who may be at-risk of not graduating from high school on-time? [interviewer 

will allow the interviewee to describe their process and will select most appropriate response]

o Yes, we have a district-wide tool to identify students who may be at-risk of not graduating and we use 

the tool in all schools.
REL Midwest Recruitment Materials—51

o Yes, we have a district-wide tool to identify students who may be at-risk of not graduating and we use 

the tool in some schools.

If yes, please specify which schools: 

o No, the district does not promote and/or high schools do not use a tool to identify students who may be 

at-risk of not graduating. 

o I'm not sure.

4. Can you please describe the tool:

5. Does your district have an early warning system team or group of individuals that focuses on students who are 

identified as at risk of not graduating from high school? [interviewer will allow the interviewee to describe their 

process and will select most appropriate response]

o Yes, the district has a team that focuses on students who are identified as at risk of not graduating from 

high school.

o No, there is no team at the district or high school levels.

o Other

If yes, specify who serves on this team: 

6. Does your district have a policy or practice on how schools should assign at-risk students to supports and/or 

interventions? 

Specifically, how is data used within this process (e.g. how often and to what extent?)

Does this vary by school? If so, how? 
REL Midwest Recruitment Materials—52

7. Does your district have a system for schools to monitor individual high school students' progress in 

interventions/supports to which they are assigned?

o Yes

o No

o Unsure

If "yes," please specify which high schools use it: 

8. If your district monitors students' progress in interventions/supports to which they are assigned, what data are 

used in this process? [interviewer will allow the interviewee to describe their process and will check all that 

apply]

o Attendance

o State assessment results

o Course performance

o Data from students' classroom teachers

o Core course performance

o Data from other adults who interact with the student (e.g., counselors, coaches, etc.)

o Credits earned

o Behavior referrals 

o Behavior suspensions

o Information from the at-risk (flagged) student

o Information from at-risk (flagged) students' parent/guardian 

o Other

If "other," please specify: 

9. Is there anything else you would like to share with us about how your district approaches dropout prevention 

interventions and strategies? 
REL Midwest Recruitment Materials—53

A-7. Final Agreement Form (School MOU)

School Roles and Responsibilities: 

Early Warning and Intervention Monitoring System Study

Dear <principal name>,

On behalf of American Institutes for Research (AIR) (www.air.org) and its partners, we welcome 

you to the Early Warning and Intervention Monitoring System (EWIMS) study. We are excited 

about this project. Its purpose is to examine the impact of EWIMS on (1) student outcomes 

including student risk status for dropout, scores on graduation tests, persistence and progress in 

school and likelihood of on-time graduation; and (2) school outcomes including how schools 

allocate dropout prevention interventions for students and their data-use culture. This document 

contains an overview of the study and a brief description of the intervention, followed by a 

description of the roles and responsibilities for your school and for the study team, including the 

benefits of participation, and the project timeline. Please review the contents of this document 

and sign the last page to indicate your agreement to participate. Return the signed last page to Dr. 

Nicholas Sorensen (nsorensen@air.org, fax 312-288-7601).

Overview

The primary goal of the EWIMS study is to evaluate whether implementing an early warning and 

intervention monitoring system for identifying students at-risk of dropping out and using this 

system to assign students to dropout prevention interventions will improve student outcomes 

including student risk status for dropout, scores on graduation tests, persistence and progress in 

school and likelihood of on-time graduation. In addition, this study will also examine the impact 

of implementing EWIMS on school outcomes including how schools allocate dropout prevention 

resources for students and their data-use culture. Study results will yield valuable information for 

the state of Ohio and for districts and schools across the country about the viability and benefits 

of using an early warning system to prevent high school dropout and help struggling students get 

back on track for eventual graduation. The study is being funded by the U.S. Department of 

Education, Institute of Education Sciences, and will January 2014 through the spring of 2016.

We look forward to working with <High School> as a partner in this project! 

The EWIMS Model

The EWIMS model, developed by the National High School Center at AIR, is a multistep 

process intended to encourage systematic and comprehensive implementation within schools. 

The process is based on a combination of research on data use in schools and National High 

School Center’s experience working with states, districts, and schools implementing early 

warning systems. 

At the heart of the EWIMS process is an early warning data tool used to flag students as “at risk” 

based on attendance, course performance (grades, credits, grade point average [GPA]), and 

behavior indicators. The tool enables schools to identify students who are at risk of dropping out 
REL Midwest Recruitment Materials—54

of school, record assignments to available interventions, and monitor students’ response to those 

interventions.

Beyond the development of the data tools, the National High School Center has devised a sevenstep EWIMS implementation process to support implementation. The process guides users to 

make informed decisions about how to use data to support at-risk students and how to continue 

to monitor their progress over time. In addition to focusing on individual students, the process 

guides users to examine the success of specific supports or interventions and to examine possible 

systemic issues (e.g., school climate) that may relate to dropout trends.

Figure A-1. Early Warning Intervention Monitoring System Implementation Process

As shown in Figure 1, the steps are intended to be cyclical. At the core of this data-driven 

decision-making process, the steps focus users on key indicators that identify which students are 

showing signs of risk of dropping out of high school and guide users to go beyond the indicator 

data and other relevant information to connect at-risk students to dropout prevention or academic 

support interventions. The EWIMS model does not prescribe specific interventions for 

schools to implement. Instead, the model is designed allow schools flexibility to decide which 

interventions they believe are most effective for their students’ needs. Ideally, the EWIMS model 

allows users to identify students with accuracy and provide supports and intervention of your 

school’s choosing to at-risk students, resulting in improved outcomes for students, including 

higher attendance rates and improvement in academic performance leading toward successful 

graduation. 

This Study

Despite increasingly widespread implementation of early warning systems by states, districts, 

and schools, there have been no rigorous studies testing the impact of using an early warning 

system to improve student outcomes such as staying in school, progressing in school, and 

graduating. There is also little research examining how using an early warning system can shape 

a school’s culture for data use—increased data-driven decision making (assigning interventions 

STEP 1—Establish 

Roles & Process

STEP 2—Use the 

EWS tool

STEP 3—Analyze 

EWS data

STEP 4—Interpret 

EWS data 

STEP 5—Assign & 

Provide 

Interventions

STEP 6—Monitor 

Students

STEP 7—Evaluate 

& Refine EWIMS
REL Midwest Recruitment Materials—55

to students) and processes and professional development to support using data to improve 

teaching and learning. This study will address these gaps and provide the first rigorous test of the 

impact of an early warning system. The study will:

 Identify a sample of eligible and interested schools in Ohio. The study team will 

conduct outreach to schools that meet initial eligibility criteria to confirm eligibility and 

discuss interest in participating in the EWIMS study. The study will include 

approximately 70 high schools in Ohio. To qualify, schools must (1) have at least 150 

ninth-grade students; (2) a graduation rate between 25 and 95 percent and (3) not already 

be implementing an early warning system tool for using data to flag at-risk students.

 Use a lottery to randomly assign half of the participating schools to implement 

EWIMS in January 2014 and the other half to implement in fall of 2015. Half of 

participating schools will be randomly assigned to receive access to the EWIMS model in 

January 2014 (including the tool and technical support for implementation) and the other 

half of participating schools will be randomly assigned to conduct “business as usual” for 

identifying at-risk students until the fall of 2015 when they will receive the same 

resources and supports.

 Implement EWIMS. All schools will implement the seven-step EWIMS process either 

in January 2014 or the fall of 2015 (depending on random assignment by lottery). The 

seven steps for implementation are as follows:

o Step 1—Establish Roles and Process. In this step, the composition of the EWIMS 

team is established;20 team members then determine frequency and duration of 

meetings and develop a shared vision or focus of the team’s work.

o Step 2—Use the EWS tool. In this step, the school-based EWIMS teams are 

trained on the use and purpose of the tool itself. This step also includes first 

customizing the tool settings and importing the student demographic and initial 

administrative data and also ongoing refreshing of the administrative data in the 

tool and the running of automated and custom lists and reports available within 

the tool. 

o Step 3—Analyze EWS data. In Step 3, EWIMS teams focus their attention on 

student- and school-level data, based on the indicators available in the tool. This 

data review process is intended to identify areas of focus and further 

investigation. 

o Step 4—Interpret EWS data. Step 4 guides teams to bring in additional data 

(external to the tool) to provide more context and a fuller picture to inform the 

EWIMS team’s consideration of specific needs of individuals or groups of 

flagged students. Unlike Step 3, which is focused on the indicator flags 

themselves (i.e., the data in the tool), this step addresses root causes of why 

students might be identified as at risk for one or more indicators. The 

implementation team will provide training to identify root causes that focus on 

acquiring additional formal (e.g., administrative records) and informal (e.g., from 

 

20 The goal is not to create yet another “team” with functions that may or may not overlap with functions of other 

already existing teams. Rather, integration with existing team structures (if functionally operational) is optimal.
REL Midwest Recruitment Materials—56

teacher, family, and student) input. This training will occur face to face and will 

last for one hour. 

o Step 5—Assign and Provide Interventions. In this step, EWIMS team members 

make informed decisions about the allocation of available resources and strategies 

to support students identified as at risk of dropping out of high school. The 

EWIMS team matches individual students to specific interventions after having 

gathered information about (1) potential root causes for individual flagged 

students (Step 4) and (2) the available dropout prevention and academic and 

behavioral support programs in the school, district, and community, which are 

locally determined. 

o Step 6—Monitor Students. In this step, EWIMS teams continue to examine 

student indicators at regular intervals to continually identify students who show 

signs of being at risk. The teams will use the same indicators to closely monitor 

already-identified students who were assigned to interventions for progress in 

school and risk status. This step provides critical ongoing feedback about 

additional student- and school-level needs and apparent successes. 

o Step 7—Evaluate and Refine the EWIMS. Through active and structured 

reflection, EWIMS team members assess whether students are responding to 

assigned interventions, revise their specific strategies or general approach as 

needed, and determine how resources are allocated to improve support for 

students. This step encourages EWIMS teams to make course corrections to all 

parts of the EWIMS implementation. As implied by the cyclical depiction of the 

seven-step process, this step (as well as the other six) reflects an ongoing process 

of continual improvement. 

 Evaluate the effects of EWIMS on student and school outcomes. The study will 

examine student outcomes for all students in grades 9 and 10 during the 2013–14 school 

year and all students in grades 9 through 11 during the 2014–15 school year. All student 

outcome data will be collected from school or district administrative data, the EWS tool

or the Ohio Department of Education. In addition, all participating schools will be asked 

to complete an annual Web-based survey about data use practices and schools randomly

assigned to implement EWIMS in January 2014 may be asked to participate in interviews 

about their experiences using the tool.

School Roles and Responsibilities

We look forward to partnering with <High School Name> for this exciting project! More 

detailed information on the responsibilities of participating schools follows. Final determination 

of school eligibility requires willingness to adhere to the study guidelines and responsibilities.

 Maintain a sustained commitment to participate in the study. To evaluate the impact 

of EWIMS, it is critical that schools agree to adhere to the study guidelines and timelines 

associated with random assignment by lottery to implement EWIMS in January 2014 or 

fall of 2015. All schools are expected to implement the EWIMS model as designed 

(including all seven steps of the implementation cycle). 
REL Midwest Recruitment Materials—57

 Adhere to the results of the lottery/random assignment process. It is essential that the 

groupings that result from the lottery remain intact over the course of the study. Schools 

assigned by lottery to implement the EWIMS model in January 2014 will serve as the 

treatment group. Their counterparts assigned to implement EWIMS in fall of 2015 will 

serve as a control group between January 2014 through the spring of 2015. Schools 

randomly assigned to the control group must continue with “business as usual” practices 

for identifying at-risk students and assigning dropout prevention interventions until 

EWIMS implementation begins in the fall of 2015. Students in control schools should

continue to receive any services that would be offered to them in the absence of the 

study. As for students in the “treatment” group, no “typical” services should be withheld. 

Please refer any questions or concerns from parents or school staff about this to the study

team.

 Participate in all data collection activities. Participating schools should provide schoollevel information including high school graduation rates, average state achievement 

scores in reading and mathematics and demographics (e.g., percentage of students 

receiving free or reduced-priced lunch). 

Administrative student-level data collection in participating schools will focus on all 

students in grades 9 and 10 during the 2013–14 school year and all students in grades 9, 

10 and 11 during the 2014–15 school year. The study team will obtain as much 

administrative data as possible from the Ohio Department of Education (ODE) and the 

school district. However, all participating schools should provide the study team access to 

the following administrative data for students should this data be unavailable through 

other sources:

o Demographic information (e.g., race/ethnicity, gender, free or reduced-price lunch 

[FRPL], individualized education program [IEP], and English language learner 

[ELL] status, and parents’ education)

o Grade point average (GPA)

o State test scores 

o Attendance rates

o Course grades in core academic courses by semester

o Credits earned by semester

o Disciplinary information (e.g., suspensions)

o Enrollment information (e.g. whether students are enrolled or have left school for 

reasons other than transfer to another district, including dropping out)

o Grade promotion

In addition to administrative records, one administrator at all schools should complete an 

annual Web-based survey assessing how schools use data to allocate dropout prevention 

resources to students.

Finally, all schools assigned by lottery to the treatment group (implementing EWIMS in 

January 2014) should participate in data collection efforts focused on implementation. 

Specifically, the project team will collect data on attendance and satisfaction with 
REL Midwest Recruitment Materials—58

EWIMS training sessions and meetings. EWIMS teams at schools assigned to the 

treatment group will be required to submit their EWS tool securely to the study team and 

may be asked to participate in interviews about their implementation experience.

EWIMS implementation (either in January 2014 or fall of 2015) will require the following:

 Develop an EWIMS team within your school. A diverse, well-informed, EWIMS team 

within your school is essential to the success of this process. The EWIMS team may be 

established as a new team or may build on or be integrated into existing teams (school 

improvement team, response to intervention team, student support team). It is not 

necessary to create an entirely new team for EWIMS work, but an existing team that 

takes on the responsibility to use the tool for dropout prevention efforts should include a 

broad representation of staff within the school (e.g., principals, representatives from 

feeder elementary/middle schools, guidance counselors, teachers, specialists). The 

EWIMS team is responsible for identifying students who are at risk and ensuring that 

their individual needs are met through school-based interventions. In most cases, this 

team is not directly responsible for applying interventions for students but their focus 

should be on helping students navigate the school systems to access appropriate and 

needed services. 

 Participate in EWIMS professional development. The EWIMS team will receive 

professional development on the EWIMS process and tool capabilities, and subsequently

be given adequate time to implement the EWIMS process. 

 Import student data into the EWS tool. A member of the EWIMS team is responsible 

for entering, or importing, data into the EWS tool, which facilitates the EWIMS team’s 

use of the tool to use the data to initially flag students as at risk.

 Produce reports of at-risk students and assign students to appropriate interventions 

or services. The tool houses information about interventions assigned to each student and 

documents students’ transition in and out of each intervention and their ultimate response 

to the intervention(s) (i.e., for each student flagged, did the assigned intervention(s) have 

a positive influence on the number of flags as calculated during subsequent grading 

periods?). The EWS tool does not prescribe specific interventions for students based on 

the type or number of indicators, but rather relies on EWIMS teams to make data-driven 

decisions within their own local context of potential interventions available, to match 

students with interventions. 

 Conduct EWIMS team meetings that are organized and documented. An agenda for 

each meeting should be prepared at the end of the prior meeting, and at least some agenda 

items should be routine, such as a review of the data from the tool, actions taken for 

individual or groups of students, a review of previous meetings’ action items (ongoing or 

completed), new action items, and communication with staff and leadership. Notes 

should be taken at each meeting and include action items assigned to specified 

individuals to accomplish. Agenda and meeting notes should be kept on file to provide a 

record of the team’s work.

 Communicate with individuals and groups outside of the EWIMS team. Information 

on flagged students, intervention effectiveness, and team-identified needs to support 
REL Midwest Recruitment Materials—59

students should be routinely reported to and discussed with school and district leadership. 

Teachers should receive regular updates about students in their classes who are 

displaying indicators of risk, as well as input about supports available to them to use with 

these students. Last, students and their parents should be engaged in the conversation 

about their risk status and the plans to ensure that they are able to get back on track for 

graduation. Although the EWIMS team may not be directly responsible for meetings with 

individual students and their parents (i.e., delivering the individual interventions), the 

team should be in a position to prompt such meetings or to share information routinely

about student progress and the early warning signs of risk. Of critical note, the team 

should share the knowledge of students’ risk with sensitivity, ensuring that identification 

is used to prompt action and support, not to assign labels that carry stigma.

 Solicit feedback from stakeholders. Feedback from administrators, teachers, staff, 

students, and parents can help the EWIMS team uncover underlying causes for students 

displaying indicators of risk. This information may help the EWIMS team match students 

to appropriate interventions and supports. 

 Monitor progress. The EWIMS team should monitor progress as it strives to improve 

educational outcomes for students during a single school year and over the course of 

multiple school years. The team should be responsible for presenting progress reports to 

key stakeholders, including principals, staff, district leadership, the local board of 

education, and parents.

Study Team Role and Responsibilities 

The project team is composed of researchers from the Midwest Regional Educational Laboratory

at AIR. The major responsibilities for the study team are as follows: 

 Obtain necessary approvals from review boards (federal, district, and organizational 

Internal Review Boards) and comply with the research protocols in place.

 Provide access to the EWS tool and training and technical support for implementation.

 Collect data for the study. The majority of the data for this study will be administrative 

records transmitted from the district, thus minimizing the data collection burden on 

participating schools. The study team also will conduct an annual Web-based survey of 

all schools and collect all implementation data from schools assigned to the treatment 

group (EWS tool data, interviews with EWIMS team members).

 Assure confidentiality. The study team will collect data only for the purposes of this 

study and will not use or allow the use of the data for evaluating individual participants, 

schools, or districts.

 Each participant will be assigned a study-specific identification number, in place of 

their names. A data file that links each participant with their identification number 

will be kept in a password-protected file that only the study team can access.

 The published analysis of the results will aggregate results across all schools and will 

not include results that have been disaggregated by school or district.
REL Midwest Recruitment Materials—60

 All members of the study team are required to complete a comprehensive training 

course that addresses current federal government standards and sign federal data 

confidentiality agreements.

 Analyze data and produce reports. The study team will be responsible for aggregating 

information about the effectiveness of EWIMS on student and school outcomes. The 

study team expects that the final report will be released in 2016, pending the federal 

review process, and will ensure that participating schools receive this report.

Timeline 

Table A-1 presents the major tasks of the project as they were described previously.

Table A-1. Major Tasks of the Project

Tasks Dates

EWIMS Implementation

Treatment schools implement EWIMS with Grades 9 and 10 Jan 2014–June 2014

Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014–June 2015

Control schools implement EWIMS August 2015–June 2016

Data Collection

Collect administrative records from the state and district Jan 2014–June 2015

Collect EWS tool data from treatment schools Jan 2014–June 2015

Conduct annual Web-based survey May 2014 and May 2015

Conduct interviews with EWIMS team members in treatment 

schools

June 2014 and June 2015

Analysis and Reporting 

Draft and submit final report. December 2015

EWIMS implementation is aligned with the academic calendar. The school-based EWIMS teams 

meet monthly, with other critical activities occurring prior to school beginning, after the first 20 

or 30 days of school, shortly after the end of each grading period, and at the end of the academic 

year. Table A-2 details expected key activities of EWIMS implementation over the course of an 

academic year.
REL Midwest Recruitment Materials—61

Table A-2. Schedule and Key Activities for Early Warning Intervention and 

Monitoring System Implementation

Year 1 Activities 

Schedule Key Activities (aligned to the Early Warning Intervention and Monitoring System 

[EWIMS] implementation steps) 

January 

2014

 Forming/designating an EWIMS team (Step 1)

 Setting up the early warning system (EWS) Tool (Step 2)

 Begin convening monthly EWIMS team meetings (Step 1)

After the 

midyear 

grading 

period 

(~January 

2014)

 Importing or entering students’ absences, course failures, and behavior information 

(e.g., referrals and suspensions, by grading period) (Step 2)

 Reviewing and interpreting student- and school-level reports (Steps 3 and 4)

 Identifying and implementing student interventions (Step 5)

 Monitoring students’ responses to existing interventions in which they are 

participating (Step 6)

 Revising students’ intervention assignments, as needed (Steps 5 and 6)

At the end 

of the school 

year (~June 

2014)

 Updating student roster to reflect new enrollees, transfers in and out, and so forth 

(Step 2)

 Importing or entering students’ absences, course failures, and behavior information 

(e.g., referrals and suspensions), by grading period, if applicable (Step 2)

 Reviewing and interpreting student- and school-level data (Steps 3 and 4)

 Identifying and implementing new student interventions (Step 5)

 Monitoring students’ responses to existing interventions in which they are 

participating (Step 6)

 Revising students’ intervention assignments for summer and for the next academic 

year, if needed (Steps 5 and 6)

 Evaluating the EWIMS process, using student- and school-level reports, and revise as 

necessary (Step 7)

 Exporting student data to (1) prepare the EWS tool for the next school year and/or 

(2) for those students who are transitioning to high school, share data with students’ 

high school(s).

Year 2 Activities 

Schedule Key Activities (aligned to the Early Warning Intervention and Monitoring System 

[EWIMS] implementation steps) 

At the 

beginning of 

the school 

year

 Reconvening the EWIMS team meetings (Step 1)

 Importing or entering student information and, if available, incoming risk indicator 

data into the EWS Tool (Step 2)
REL Midwest Recruitment Materials—62

(~August 

2014)

 Reviewing and interpreting student needs based on data from the previous year (e.g., 

review the Overage Student Report) (Steps 3 and 4)

 Verifying student information, especially enrollment status, and updating student 

roster to reflect new enrollees, transfers in and out, and so forth (Step 2)

 Reviewing incoming risk indicators or previous year data, including any additional 

information (e.g., bridge program participation, summer school participation, prior 

course performance), to review and interpret student needs (Steps 3 and 4)

 Identifying and implementing student interventions or supports based on incoming 

risk indicator information, if available (Step 5)

After the 

first 20 or 30 

days of the 

school year

(~October 

2014)

 Updating student roster to reflect new enrollees, transfers in and out, and so forth

(Step 2)

 Importing students’ absences (Step 2)

 Reviewing and interpreting student- and school-level reports (Steps 3 and 4)

 Identifying and implementing student interventions (Step 5)

 Monitoring students’ initial response to interventions (Step 6)

 Revising students’ intervention assignments, as needed (Steps 5 and 6)

After the 

midyear 

grading 

period

(~January 

2015)

 Updating student roster to reflect new enrollees, transfers in and out, and so forth 

(Step 2)

 Importing or entering students’ absences, course failures, and behavior information 

(e.g., referrals and suspensions, by grading period) (Step 2)

 Reviewing and interpreting student- and school-level reports (Steps 3 and 4)

 Identifying and implementing student interventions (Step 5)

 Monitoring students’ responses to existing interventions in which they are 

participating (Step 6)

 Revising students’ intervention assignments, as needed (Steps 5 and 6)

At the end of 

the school 

year (~June 

2015)

 Updating student roster to reflect new enrollees, transfers in and out, and so forth 

(Step 2)

 Importing or entering students’ absences, course failures, and behavior information 

(e.g., referrals and suspensions), by grading period, if applicable (Step 2)

 Reviewing and interpreting student- and school-level data (Steps 3 and 4)

 Identifying and implementing new student interventions (Step 5)

 Monitoring students’ responses to existing interventions in which they are 

participating (Step 6)

 Revising students’ intervention assignments for summer and for the next academic 

year, if needed (Steps 5 and 6)

 Evaluating the EWIMS process, using student- and school-level reports, and revise as 

necessary (Step 7)

 Exporting student data to (1) prepare the EWS tool for the next school year and/or 

(2) for those students who are transitioning to high school, share data with students’ 

high school(s).
REL Midwest Recruitment Materials—63

Benefits to Participation

There are many benefits of participation for high schools in Ohio. More than one out of five 

students in Ohio fails to graduate from high school and graduation rates are significantly lower 

for students who are racial or ethnic minorities, economically disadvantaged, receiving special 

education services, migrant and/or limited-English proficient. Critical indicators in ninth and 

tenth grade that powerfully predict whether students are “on track” for high school graduation 

can be used as part of an early warning system to flag at-risk students early, assign appropriate 

interventions, and get students back on track. Participating in this high-profile, large-scale study

will give your school and district an opportunity to access the Early Warning and Intervention 

Monitoring System (developed by the National High School Center at AIR) at no cost. The 

EWIMS model, currently in use in 67 districts in six states, includes both an excel-based tool and 

training and technical support for implementation. Your participation in this study will also play

an important role in informing educational policy focused on dropout prevention in Ohio and at 

the federal level.

Questions or Comments

If you have any questions or comments about the study or the opportunity it provides for your 

school, please feel free to contact Dr. Nicholas Sorensen (nsorensen@air.org or 312-283-2318) 

or Dr. Mindee O’Cummings (mocummings@air.org or 202-403-5254).
REL Midwest Recruitment Materials—64

Signatures of Commitment

Return via fax (312-288-7601) or e-mail (nsorensen@air.org)

The following people have read this document detailing the study and agree to the roles, 

responsibilities, and conditions of participation on behalf of <High School Name> and the study team.

<insert name> <insert title>

District Representative Signature Printed Name Title Date

Principal

Principal Signature Printed Name Title Date

Jessica Heppen Co-Principal Investigator

Principal-Investigator Printed Name Title Date

Mindee O’Cummings Co-Principal Investigator

Principal-Investigator Printed Name Title Date

Ann-Marie Faria Project Director 

Project Director Printed Name Title Date

Nicholas Sorensen Deputy Project Director 

Project Director Printed Name Title Date

Per the policies and procedures required by the Education Sciences Reform Act of 2002, Title I, 

Part E, Section 183, responses to this data collection will be used only for statistical purposes. 

The reports prepared for this study will summarize findings across the sample and will not 

associate responses with a specific district or individual. ED’s contractor will not provide 

information that identifies you or your district to anyone outside the study team, except as 

required by law. Any willful disclosure of such information for nonstatistical purposes, without 

the informed consent of the respondent, is a class E felony. 
REL Midwest Recruitment Materials—65

A-8. Final Agreement Form (District MOU)

District Roles and Responsibilities: 

Early Warning and Intervention Monitoring System Study

Dear <superintendent>,

On behalf of American Institutes for Research (AIR) (www.air.org) and its partners, we welcome 

you to the Early Warning and Intervention Monitoring System (EWIMS) study. We are excited 

about this project. Its purpose is to examine the impact of EWIMS on (1) student outcomes 

including student risk status for dropout, scores on graduation tests, persistence and progress in 

school and likelihood of on-time graduation; and (2) school outcomes including how schools 

allocate dropout prevention interventions for students and their data-use culture. This document 

contains an overview of the study and a brief description of the intervention, followed by a 

description of the roles and responsibilities for your district and for the study team, including the 

benefits of participation, and the project timeline. Please review the contents of this document 

and sign the last page to indicate your agreement to participate. Return the signed last page to Dr. 

Nicholas Sorensen (nsorensen@air.org, fax 312-288-7601).

Overview

The primary goal of the EWIMS study is to evaluate whether implementing an early warning and 

intervention monitoring system for identifying students at-risk of dropping out and using this 

system to assign students to dropout prevention interventions will improve student outcomes 

including student risk status for dropout, scores on graduation tests, persistence and progress in 

school and likelihood of on-time graduation. In addition, this study will also examine the impact 

of implementing EWIMS on school outcomes including how schools allocate dropout prevention 

resources for students and their data-use culture. Study results will yield valuable information for 

the state of Ohio and for districts and schools across the country about the viability and benefits 

of using an early warning system to prevent high school dropout and help struggling students get 

back on track for eventual graduation. The study is being funded by the U.S. Department of 

Education, Institute of Education Sciences, and will run from January 2014 through the spring of 

2016.

We look forward to working with <district name> as a partner in this project! 
REL Midwest Recruitment Materials—66

STEP 1 – Establish 

Roles & Process

STEP 2 – Use the 

EWS tool

STEP 3 – Analyze 

EWS data

STEP 4 – Interpret 

EWS data 

STEP 5 – Assign & 

Provide 

Interventions

STEP 6 – Monitor 

Students

STEP 7 – Evaluate 

& Refine EWIMS

The EWIMS Model

The EWIMS model, developed by the National High School Center at AIR, is a multistep 

process intended to encourage systematic and comprehensive implementation within schools. 

The process is based on a combination of research on data use in schools and National High 

School Center’s experience working with states, districts, and schools implementing early 

warning systems. 

At the heart of the EWIMS process is an early warning data tool used to flag students as “at risk” 

based on attendance, course performance (grades, credits, grade point average [GPA]), and 

behavior indicators. The tool enables schools to identify students who are at risk of dropping out 

of school, record assignments to available interventions, and monitor students’ response to those 

interventions.

Beyond the development of the data tools, the National High School Center has devised a sevenstep EWIMS implementation process to support implementation. The process guides users to 

make informed decisions about how to use data to support at-risk students and how to continue 

to monitor their progress over time. In addition to focusing on individual students, the process 

guides users to examine the success of specific supports or interventions and to examine possible 

systemic issues (e.g., school climate) that may relate to dropout trends.

Figure A-2. Early Warning Intervention Monitoring System Implementation Process

As shown in Figure 1, the steps are intended to be cyclical. At the core of this data-driven 

decision-making process, the steps focus users on key indicators that identify which students are 

showing signs of risk of dropping out of high school and guide users to go beyond the indicator 

data and other relevant information to connect at-risk students to dropout prevention or academic 

support interventions. The EWIMS model does not prescribe specific interventions for schools to 

implement. Instead, the model is designed allow schools flexibility to decide which interventions 

they believe are most effective for their students’ needs. Ideally, the EWIMS model allows users 

to identify students with accuracy and provide supports and intervention of schools’ choosing to 
REL Midwest Recruitment Materials—67

at-risk students, resulting in improved outcomes for students, including higher attendance rates 

and improvement in academic performance leading toward successful graduation. 

This Study

Despite increasingly widespread implementation of early warning systems by states, districts, 

and schools, there have been no rigorous studies testing the impact of using an early warning 

system to improve student outcomes such as staying in school, progressing in school, and 

graduating. There is also little research examining how using an early warning system can shape 

a school’s culture for data use—increased data-driven decision making (assigning interventions 

to students) and processes and professional development to support using data to improve 

teaching and learning. This study will address these gaps and provide the first rigorous test of the 

impact of an early warning system. The study will:

 Identify a sample of eligible and interested schools in Ohio. The study team will 

conduct outreach to schools that meet initial eligibility criteria to confirm eligibility and 

discuss interest in participating in the EWIMS study. The study will include 

approximately 70 high schools in Ohio. To qualify, schools must (1) have at least 150 

ninth-grade students; (2) a graduation rate between 25 and 95 percent and (3) not already 

be implementing an early warning system tool for using data to flag at-risk students.

 Use a lottery to randomly assign half of the participating schools to implement 

EWIMS in January 2014 and the other half to implement in fall of 2015. Half of 

participating schools will be randomly assigned to receive access to the EWIMS model in 

January 2014 (including the tool and technical support for implementation) and the other 

half of participating schools will be randomly assigned to conduct “business as usual” for 

identifying at-risk students until the fall of 2015 when they will receive the same 

resources and supports for implementing EWIMS.

 Implement EWIMS. All schools will implement the seven-step EWIMS process either 

in January 2014 or the fall of 2015 (depending on random assignment by lottery). The 

seven steps for implementation are as follows:

o Step 1—Establish Roles and Process. In this step, the composition of the EWIMS 

team is established;21 team members then determine frequency and duration of 

meetings and develop a shared vision or focus of the team’s work.

o Step 2—Use the EWS tool. In this step, the school-based EWIMS teams are 

trained on the use and purpose of the tool itself. This step also includes first 

customizing the tool settings and importing the student demographic and initial 

administrative data and also ongoing refreshing of the administrative data in the 

tool and the running of automated and custom lists and reports available within 

the tool. 

o Step 3—Analyze EWS data. In Step 3, EWIMS teams focus their attention on 

student- and school-level data, based on the indicators available in the tool. This 

 

21 The goal is not to create yet another “team” with functions that may or may not overlap with functions of other 

already existing teams. Rather, integration with existing team structures (if functionally operational) is optimal.
REL Midwest Recruitment Materials—68

data review process is intended to identify areas of focus and further 

investigation. 

o Step 4—Interpret EWS data. Step 4 guides teams to bring in additional data 

(external to the tool) to provide more context and a fuller picture to inform the 

EWIMS team’s consideration of specific needs of individuals or groups of 

flagged students. Unlike Step 3, which is focused on the indicator flags 

themselves (i.e., the data in the tool), this step addresses root causes of why 

students might be identified as at risk for one or more indicators. The 

implementation team will provide training to identify root causes that focus on 

acquiring additional formal (e.g., administrative records) and informal (e.g., from 

teacher, family, and student) input. This training will occur face to face and will 

last for one hour. 

o Step 5—Assign and Provide Interventions. In this step, EWIMS team members 

make informed decisions about the allocation of available resources and strategies 

to support students identified as at risk of dropping out of high school. The 

EWIMS team matches individual students to specific interventions after having 

gathered information about (1) potential root causes for individual flagged 

students (Step 4) and (2) the available dropout prevention and academic and 

behavioral support programs in the school, district, and community, which are 

locally determined.

o Step 6—Monitor Students. In this step, EWIMS teams continue to examine 

student indicators at regular intervals to continually identify students who show 

signs of being at risk. The teams will use the same indicators to closely monitor 

already-identified students who were assigned to interventions for progress in 

school and risk status. This step provides critical ongoing feedback about 

additional student- and school-level needs and apparent successes. 

o Step 7—Evaluate and Refine the EWIMS. Through active and structured 

reflection, EWIMS team members assess whether students are responding to 

assigned interventions, revise their specific strategies or general approach as 

needed, and determine how resources are allocated to improve support for 

students. This step encourages EWIMS teams to make course corrections to all 

parts of the EWIMS implementation. As implied by the cyclical depiction of the 

seven-step process, this step (as well as the other six) reflects an ongoing process 

of continual improvement. 

 Evaluate the effects of EWIMS on student and school outcomes. The study will 

examine student outcomes for all students in grades 9 and 10 during the 2013–14 school 

year and all students in grades 9 through 11 during the 2014–15 school year. All student 

outcome data will be collected from school or district administrative data, the EWS tool

or the Ohio Department of Education. In addition, all participating schools will be asked 

to complete an annual Web-based survey about data use practices and schools randomly

assigned to implement EWIMS in January 2014 may be asked to participate in interviews 

about their experiences using the tool.
REL Midwest Recruitment Materials—69

District Roles and Responsibilities

We look forward to partnering with <District Name> for this exciting project! More detailed 

information on the responsibilities of participating schools follows. 

 Maintain a sustained commitment to participate in the study. To evaluate the impact 

of EWIMS, it is critical that participating districts agree to adhere to the study guidelines 

and timelines associated with random assignment by lottery to implement EWIMS in 

January 2014 fall of 2015. 

 Support school recruitment. Participating districts should actively support the study

team’s efforts to recruit high schools by communicating with school leadership and 

endorsing district participation and encourage school-level participation. 

 Support the use of random assignment. It is essential that the district support the use of 

random assignment of schools to receive access to and implement EWIMS in January

2014 or the fall of 2015. All schools will receive access to EWIMS including the tool and 

training and technical support for implementation; random assignment will only

determine whether schools receive the “treatment” in January 2014 or a “delayed 

treatment” in the fall of 2015. Schools assigned by lottery to implement the EWIMS 

model in January 2014 will serve as the treatment group. Their counterparts assigned to 

implement EWIMS in fall of 2015 will serve as a control group between January 2014

through the spring of 2015. Schools randomly assigned to the control group must

continue with “business as usual” practices for identifying at-risk students and assigning 

dropout prevention interventions until EWIMS implementation begins in the fall of 2015. 

Students in control schools should continue to receive any services that would be offered 

to them in the absence of the study. As for students in the “treatment” group, no “typical” 

services should be withheld. 

 Provide access to student-level administrative records. Administrative student-level 

data collection in participating schools will focus on all students in grades 9 and 10

during the 2013–14 school year and all students in grades 9, 10 and 11 during the 2014–

15 school year. The study team will obtain as much administrative data as possible from 

the Ohio Department of Education (ODE). However, all participating schools should 

provide the study team access to the following student-level administrative data should 

this data be unavailable from ODE:

o Demographic information (e.g., race/ethnicity, gender, free or reduced-price lunch 

[FRPL], individualized education program [IEP], and English language learner 

[ELL] status, and parents’ education)

o Grade point average (GPA)

o State test scores 

o Attendance rates

o Course grades in core academic courses by semester

o Credits earned by semester

o Disciplinary information (e.g., suspensions)
REL Midwest Recruitment Materials—70

o Enrollment information (e.g. whether students are enrolled or have left school for 

reasons other than transfer to another district, including dropping out)

o Grade promotion

Study Team Role and Responsibilities 

The project team is composed of researchers from the Midwest Regional Educational Laboratory

at AIR. The major responsibilities for the study team are as follows: 

 Obtain necessary approvals from review boards (federal, district, and organizational 

Internal Review Boards) and comply with the research protocols in place.

 Provide access to the EWS tool and training and technical support for implementation to 

participating schools.

 Collect data for the study. The majority of the data for this study will be administrative 

records transmitted from the district, thus minimizing the data collection burden on 

participating schools. The study team will provide a detailed data request to participating 

districts in a pre-determined format that will minimize burden on district staff. 

 The study team will conduct an annual Web-based survey of all participating schools and 

collect all implementation data from schools assigned to the treatment group (EWS tool

data, interviews with EWIMS team members).

 Assure confidentiality. The study team will collect data only for the purposes of this 

study and will not use or allow the use of the data for evaluating individual participants, 

schools, or districts.

 Each participant will be assigned a study-specific identification number, in place of 

their names. A data file that links each participant with their identification number 

will be kept in a password-protected file that only the study team can access.

 The published analysis of the results will aggregate results across all schools and will 

not include results that have been disaggregated by school.

 All members of the study team are required to complete a comprehensive training 

course that addresses current federal government standards and sign federal data 

confidentiality agreements.

 Analyze data and produce reports. The study team will be responsible for aggregating 

information about the effectiveness of EWIMS on student and school outcomes. The 

study team expects that the final report will be released in 2016, pending the federal 

review process, and will ensure that participating schools receive this report.
REL Midwest Recruitment Materials—71

Timeline 

Table A-3 presents the major tasks of the project as they were described previously.

Table A-3. Major Tasks of the Project

Tasks Dates

EWIMS Implementation

Treatment schools implement EWIMS with Grades 9 and 10 Jan 2014–June 2014

Treatment schools implement EWIMS with Grades 9, 10, and 11 August 2014–June 2015

Control schools implement EWIMS August 2015–June 2016

Data Collection

Collect administrative records from the state and district Jan 2014–June 2015

Collect EWS tool data from treatment schools Jan 2014–June 2015

Conduct annual Web-based survey May 2014 and May 2015

Conduct interviews with EWIMS team members in treatment schools June 2014 and June 2015

Analysis and Reporting 

Draft and submit final report. December 2015

Benefits to Participation

There are many benefits of participation for districts in Ohio. More than one out of five students in 

Ohio fails to graduate from high school and graduation rates are significantly lower for students 

who are racial or ethnic minorities, economically disadvantaged, receiving special education 

services, migrant and/or limited-English proficient. Critical indicators in ninth and tenth grade that 

powerfully predict whether students are “on track” for high school graduation can be used as part 

of an early warning system to flag at-risk students early, assign appropriate interventions, and get 

students back on track. Participating in this high-profile, large-scale study will give your district an 

opportunity to access the Early Warning and Intervention Monitoring System (developed by the 

National High School Center at AIR) at no cost. The EWIMS model, currently in use in 67 districts 

in six states, includes both an excel-based tool and training and technical support for 

implementation. Your participation in this study will also play an important role in informing 

educational policy focused on dropout prevention in Ohio and at the federal level.

Questions or Comments

If you have any questions or comments about the study or the opportunity it provides for your 

district, please feel free to contact Dr. Nicholas Sorensen (nsorensen@air.org or 312-283-2318) 

or Dr. Mindee O’Cummings (mocummings@air.org or 202-403-5254).
REL Midwest Recruitment Materials—72

Signatures of Commitment

Return via fax (312-288-7601) or e-mail (nsorensen@air.org)

The following people have read this document detailing the study and agree to the roles, responsibilities, 

and conditions of participation on behalf of <District Name> and the study team.

<insert name> <insert title>

District Representative Signature Printed Name Title

Date

Jessica Heppen Co-Principal Investigator

Principal-Investigator Printed Name Title

Date

Mindee O’Cummings Co-Principal Investigator

Principal-Investigator Printed Name Title

Date

Ann-Marie Faria Project Director 

Project Director Printed Name Title

Date

Nicholas Sorensen Deputy Project Director 

Project Director Printed Name Title

Date

The below <District Name> staff person has agreed to serve as the primary point of contact for school 

recruitment and data collection during the course of the study.

Name:

Position:

Telephone Number:

E-Mail:
REL Midwest Recruitment Materials—73

A-9. Final Agreement Form (State MOU)

Memorandum of Understanding

By and Between the Ohio Department of Education and __________

This agreement is entered into by the Ohio Department of Education (“ODE”) and _____________ 

(“Researcher”) for the purpose of sharing information between the parties in a manner consistent with 

the Family Education Records Privacy Act of 1974 (“FERPA”). The information will be used by 

researchers at _____________ to conduct evaluative studies designed to improve instruction for 

children in the state of Ohio. Topics of these studies will include: ___________. In order to complete 

these studies and in order to have a positive impact on the instruction of children, the Researcher 

requires the use of student data from the ODE. 

The Family Educational Rights and Privacy Acts Statute (FERPA) describes circumstances under 

which State Educational Agencies (SEAs) are authorized to release data from an education record. 

This information can be disclosed to organizations conducting studies on behalf of SEAs, provided 

that Federal, State or local law authorizes the evaluation in question. Ohio Revised Code § 3301.12 

grants to the ODE State Superintendent of Public Instruction the authority to conduct studies of 

education programs, but the provision prohibits access to a student’s name, address or social security

number.

1. PARTIES. The ODE is an SEA that is authorized to receive information from local educational 

agencies (“LEAs”) subject to FERPA, as authorized by 34 CFR Section 99.31. Researcher desires 

to conduct studies on behalf of ODE for the purpose of improving instruction in Ohio public 

schools in accordance with the Scope of Work Agreement attached hereto as Appendix A. The 

parties wish to share data collected by the ODE regarding education in Ohio, none of which will 

allow the identification of individual students. 

2. COMPLIANCE WITH FERPA. To effect the transfer of data subject to FERPA, Researcher 

agrees to:

1. In all respects comply with the provisions of FERPA. For purposes of this agreement, 

“FERPA” includes any amendments or other relevant provisions of federal law, as well as all 

requirements of Chapter 99 of Title 34 of the Code of Federal Regulations. Nothing in this 

agreement may be construed to allow either party to maintain, use, disclose or share student 

information in a manner not allowed by federal law or regulation.

2. Use the data shared under this agreement for no purpose other than research authorized 

under Section 99.31(a)(3)(iv) or 99.31(a)(6) of Title 34 of the Code of Federal Regulations. 

Researcher further agrees not to share data received under this MOU with any other entity 

without the ODE’s approval. Researcher agrees to allow the Office of the State Auditor, subject 

to FERPA restrictions, access to data shared under this agreement and any relevant records of 

Researcher for purposes of completing authorized audits of the parties. Researcher shall be liable 

for any audit exception that results solely from its acts or omissions in the performance of this 

agreement. ODE shall be liable for any audit exception that results solely from its acts or 
REL Midwest Recruitment Materials—74

omissions in the performance of this agreement. In the event that the audit exception results from 

the act or omissions of both parties, the financial liability for the audit exception shall be shared 

by the parties in proportion to their relative fault.

3. Require all employees, contractors and agents of any kind to comply with all applicable 

provisions of FERPA and other federal laws with respect to the data shared under this agreement. 

Researcher agrees to require and maintain an appropriate confidentiality agreement from each 

employee, contractor or agent with access to data pursuant to this agreement. Nothing in this 

paragraph authorizes sharing data provided under this Agreement with any other entity for any 

purpose other than completing Researcher’s work authorized under this Agreement.

4. Maintain all data obtained pursuant to this agreement in a secure computer environment and 

not copy, reproduce or transmit data obtained pursuant to this agreement except as necessary to 

fulfill the purpose of the original request. All copies of data of any type, including any 

modifications or additions to data from any source that contains information regarding students, 

are subject to the provisions of this agreement in the same manner as the original data. The 

ability to access or maintain data under this agreement shall not under any circumstances transfer 

from Researcher to any other institution or entity. 

5. Not to disclose any data obtained under this agreement in a manner that could identify an 

individual student to any other entity in published results of studies as authorized by this 

agreement, nor attempt to infer or deduce the identity of any student or teacher based on data 

provided by ODE, nor claim to have identified or deduced the identity of any student based on 

data provided by ODE. 

6. Not to provide any data obtained under this agreement to any party ineligible to receive data 

protected by FERPA or prohibited from receiving data from any entity. 

7. Provide to the ODE a list of specific research studies, updated annually, for which the data 

are being used, and to notify the ODE in advance of any new project or research question 

researcher proposes to address. This list of research studies will identify linkages of all data 

possessed by researcher under this agreement and covered by FERPA to specific research 

studies. Further, it will include the fixed ending date for use of all data linked to each project.

8. Provide to the ODE any materials designed for public dissemination, based in whole or in part 

on data obtained under this agreement, at least ten days prior to public release.

9. Destroy all data obtained under this agreement, within the time frame established in 

Appendix A, Section II, when it is no longer needed for the purpose for which it was obtained. 

Nothing in this agreement authorizes either party to maintain data beyond the time period 

reasonably needed to complete the purpose of the request. All data no longer needed shall be 

destroyed or returned to the ODE in compliance with 34 CFR Section 99.35(b)(2). Researcher 

agrees to require all employees, contractors, or agents of any kind to comply with this provision. 
REL Midwest Recruitment Materials—75

3. DATA REQUESTS. 

1. The ODE may decline to comply with a request if it determines that providing the data in the 

manner requested would violate FERPA and/or would not be in the best interest of current 

or former students in Ohio public schools. All requests shall include a statement of the 

purpose for which it is requested and an estimation of the time needed to complete the 

project for which the data is requested. Data requests may be submitted by post, electronic 

mail or facsimile. 

2. Researcher agrees that ODE makes no warranty concerning the accuracy of the student data 

provided.

4. AUTHORIZED REPRESENTATIVE. Researcher shall designate in writing a single authorized 

representative able to request data under this agreement. The authorized representative shall be 

responsible for transmitting all data requests and maintaining a log or other record of all data 

requested and received pursuant to this agreement, including confirmation of the completion of 

any projects and the return or destruction data as required by this agreement. The ODE or its 

agents may upon request review the records required to be kept under this section.

RELATED PARTIES. Researcher represents that it is authorized to bind to the terms of this 

contract, including confidentiality and destruction or return of student data, all related or 

associated institutions, individuals, employees or contractors who may have access to the data or 

may own, lease or control equipment or facilities of any kind where the data is stored, maintained 

or used in any way by Researcher. This Agreement takes effect only upon acceptance by an 

authorized representative of _____________, by which that institution agrees to abide by its terms 

and return or destroy all student data upon completion of the research for which it was intended or 

upon the termination of its current relationship with Researcher.

TERM. This agreement takes effect upon signature by the authorized representative of each party

and will remain in effect until _________________. The parties further understand that the ODE 

may cancel this agreement at any time for reasonable cause, upon thirty-day written notice. Notice 

of such cancellation shall be sent or otherwise delivered to the persons signing this agreement. 

The ODE specifically reserves the right to immediately cancel this agreement upon discovery of 

non-compliance with any applicable federal or state laws, rules or regulations. Further, the ODE 

specifically reserves the right to immediately cancel this agreement should the ODE, in its sole 

discretion, determine that student information has been released in a manner inconsistent with this 

agreement, has not been maintained in a secure manner, or that substantially similar data access 

has become generally available for research purposes through any other mechanism approved by 

the ODE. In the event of immediate cancellation, a notice specifying the reasons for cancellation 

shall be sent as soon as possible after the cancellation to the persons signing the agreement. 

5. BREACH AND DEFAULT. Upon breach or default of any of the provisions, obligations, or 

duties embodied in this agreement, the parties may exercise any administrative, contractual, 

equitable, or legal remedies available, without limitation. The waiver of any occurrence of 

breach or default is not a waiver of such subsequent occurrences, and the parities retain the right 

to exercise all remedies mentioned herein.
REL Midwest Recruitment Materials—76

6. AMENDMENT. This agreement may be modified or amended provided that any such 

modification or amendment is in writing and is signed by the parties to this agreement. It is 

agreed, however, that any amendments to laws, rules, or regulations cited herein will result in the 

correlative modification of this agreement, without the necessity for executing written 

amendment.

7. ASSIGNMENT OF RIGHTS. Neither this agreement, nor any rights, duties, or obligations 

described herein shall be assigned by Researcher without the prior express written consent of 

ODE.

8. ENTIRETY OF AGREEMENT. All terms and conditions of this agreement are embodied herein 

and in the Scope of Work Agreement attached hereto as Appendix A. No other terms and 

conditions will be considered a part of this agreement unless expressly agreed upon in writing 

and signed by both parties.

Entered into by

____________________________ _____

Michael L. Sawyers Date

Acting State Superintendent of Public Instruction

_____

Matthew Cohen Date

Chief Research Officer

_____________________________ _____

[Signature] Date

[Printed Name and Title] 

[Name of Organization]
REL Midwest Data Collection Instruments—77

Attachment B. Data Collection Instruments

Attachments B-1 through B-6 include data collection instruments that will be used in the 

EWIMS impact study. 

B-1 includes a draft protocol of the school-level survey that measures the treatment contrast and 

data-informed allocation of dropout prevention interventions for students. B-2 includes a draft 

protocol of the school data culture survey. B-3 includes archival records protocol. B-4 includes 

protocol for collecting data stored in the EWS tool in treatment schools. 
REL Midwest Data Collection Instruments—78

B-1. Draft Protocol—School-Level Survey 

(Treatment Contrast and Data-Informed Allocation of Dropout Prevention 

Interventions for Students)

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless such collection displays a valid Office of Management and Budget (OMB) control 

number. The valid OMB control number for this information collection is XXXX-XXXX. The time required 

to complete this information collection is estimated to average 60 minutes per response. This information 

collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or 

suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–

4651. If you have comments or concerns regarding the status of your individual submission of this form, 

write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, 

Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208-5500.

**This survey will be administered to both treatment and control schools to estimate the impact 

of EWIMS on school-level outcomes 

The purpose of this draft school level survey is twofold: the first section documents current 

usage of an early warning system tool or the use of data to identify students who are at risk of 

academic failure, disengagement, and/or dropping out of high school, and will be used to 

measure the treatment contrast in the EWIMS impact study. We will use this information to 

document the treatment contrast. Measures are adapted from the REL Appalachia survey in an 

attempt to coordinate data collection across RELs on similar EWS implementation studies.

These second section collects consistent information from treatment and control schools about 

the types of dropout interventions they are implementing and whether and how they use student 

data to allocate those dropout interventions. This section will be used to answer school-level 

research question 2a: What is the impact of EWIMS on data-informed allocation of dropout 

prevention interventions for students? Items are based on up-to-date input from the Dropout 

Prevention Research Alliance members’ input, as well as a collaboration between REL 

Appalachia and REL Midwest to develop similar tools for EWS implementation studies. The 

study team plans to continue to review and populate these response options prior to 

administration and in collaboration with alliance members to reflect the most up-to-date local.

These items are still in draft form and will be finalized with input from REL Midwest’s Dropout 

Prevention Alliance. 

Section I: Treatment Contrast—Use of an Early Warning System

1. Is your high school using an early warning system (EWS) to identify students who may be at 

risk of not graduating from high school on time? 

o Yes, we are implementing EWS in my high school

o No, my high school is not currently implementing EWS in any high schools

o I’m not sure.
REL Midwest Data Collection Instruments—79

The following questions are designed to obtain information about EWS implementation and tools in your 

high school.

2. Does your high school have an early warning system tool? 

a. Yes, we have an early warning system tool used to identify students who may be 

at risk of not graduating in my high school.

If so, what is the name of the system? 

b. No, my high school does not currently have an early warning system tool to 

identify students who may be at risk of not graduating in my high school.

c. I’m not sure. 

3. Is your high school using the student information system (SIS) or another data tool to 

identify students who may be at risk of not graduating from high school on time? 

a. Yes, we have a tool to identify students who may be at risk of not graduating in 

my high school.

b. No, my high school does not currently have a tool to identify students who may

be at risk of not graduating in my high school.

c. I’m not sure.

If yes, please specify: 

4. How do you use the EWS Tool? (Please check all that apply.)

o To identify which students are at risk for dropping out of school

o To targetresourcesto support off-track students before they drop out

o Tomonitorstudent progress orstudents’response to intervention

o To assign students to intervention(s)

o To document causes of student risk on the basis of EWS reports and information sources

o To examine patterns and identify school climate issues

o To transfer students at risk to alternative schools

o Other (Please specify.)

5. If your school uses an early warning system, SIS, or another data tool to identify students 

who may be at risk of not graduating, what types of data are used? For each indicator used, 

what is the level of evidence?

6.
REL Midwest Data Collection Instruments—80

N/A

Evidence-based (e.g., 

based on research 

conducted using data 

not related to the 

school)

Locally validated (e.g., 

based on research 

using state, district, or 

school data)

Based on other 

criteria (specify)

Attendance 

Behavior 

referrals 

Behavior 

suspensions 

Course 

performance 

Credits earned 

Grade Point 

Average (GPA) 

State 

assessment 

results



Other:



Other:



Other:



Other:



7. How do you assign at-risk students to supports and/or interventions? (Please check all that 

apply.) 

o Student-level indicator data (i.e., evidence or locally validated indicators) 
REL Midwest Data Collection Instruments—81

o Other student-level data (e.g., demographics, student performance data )

o Specify: 

o Other types of data

o Specify: 

o No data-based strategy to assign at-risk students to supports and/or interventions

8. For which grade levels are you examining EWS indicators of risk? (Please check all that 

apply.)

o 9th Grade

o 10th Grade

o 11th Grade

o 12th Grade

o Other

If “other,” please specify __________________________________

9. Can you provide information on when you began implementing EWS in your school, as well 

as the degree of implementation in each school year?

o Before the 2010–11 school year

o 2010–11 school year

o 2011–12 school year

o 2012–13 school year

o 2013–14 school year

o Other.

If “other,” please specify: 

______________________________________________________________

10. How often does your school review data to identify students at risk of not graduating?

o 4 times per year (once per quarter)

o 3 times per year

o 2 times per year
REL Midwest Data Collection Instruments—82

o 1 time per year

o Other

o Specify: 

o I’m not sure

The following questions relate to the group of individuals who focus on using early warning

system data on students who are flagged as at risk of not graduating from high school. These

individuals may be part of a distinct EWIMS team or part of another team that is responsible

for examining these data.

11. Do you have a group of individuals who focus on implementation of EWS as its own 

EWIMS team or as part of another existing team (e.g., school improvement team, district 

improvement team) at the school or district level?

o Yes, we have a dedicated EWIMS team.

o Yes, we have a team that focuses on EWS, but it is part of another team.

o No, we do not have an existing EWIMS team.

o Other

If “other,” please specify: 

______________________________________________________________

12. Describe the members of the EWIMS team at the high school level (if applicable)? (Please 

check all that apply.)

o School principal

o Assistant principal

o Representative from feeder middle schools

o Guidance counselors

o Content area teachers

o Special education teachers

o English language learner instructors

o District office representative

o Other

If “other,” please specify 

__________________________________________________________________
REL Midwest Data Collection Instruments—83

Section II: Data-Informed Allocation of Dropout Prevention Interventions for Students

1. Does your school offer targeted academic interventions (e.g., tutoring, reading remediation, study skills) for students?

o Yes

o No (skip to 3) 

2. If yes, please name the targeted academic interventions that the school offers. In the table below, indicate which programs are 

available, the number of students who participated during the school year, and the grade level of students who participated.

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

targeted 

academic 

interventions?

What data do you 

use to determine 

whether an 

intervention/support 

is working?

Write in

Drop down 

menu 

(Reading, 

Mathematics, 

Science, 

Tutoring, 

Remediation, 

Study skills, 

Other please 

specify)

Drop down menu 

(from 1 to 999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other 

please specify)

Drop down menu 

allowing

checking all that 

apply (Course 

failure [e.g., 

failing algebra], 

Grade point 

average [GPA], 

Number of 

credits, Student 

attendance [e.g., 

missing a number 

of days of school 

Drop down menu 

allowing checking all 

that apply 

(attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other 

please specify)
REL Midwest Data Collection Instruments—84

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

targeted 

academic 

interventions?

What data do you 

use to determine 

whether an 

intervention/support 

is working?

or class], Student 

behavior [e.g., 

office referrals, 

suspensions], No 

criteria, Other 

please specify)

3. Does your school offer targeted behavior interventions (e.g., social skill training, character education) for students?

o Yes

o No (skip to 4) 

a. Please name the targeted behavior interventions at the school. In the table below, indicate which programs are available, the 

number of students who participated during the school year, and the grade level of students who participated.
REL Midwest Data Collection Instruments—85

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

targeted 

behavior

interventions?

What data do you 

use to determine 

whether an 

intervention/support 

is working?

Write in

Drop down 

menu (Social 

Skill 

Training, 

Character 

Education, 

Other please

specify)

Drop down menu 

(from 1 to 999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

Drop down menu 

allowing checking 

all that apply 

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number 

of credits, Student 

attendance [e.g., 

missing a number 

of days of school 

or class], Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

Drop down menu 

allowing checking all 

that apply (attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other please 

specify)
REL Midwest Data Collection Instruments—86

4. Does your school offer targeted attendance/truancy interventions (e.g., attendance monitor) for students?

o Yes

o No (skip to 5) 

b. Please name the targeted attendance/truancy interventions that the school offers. In the table below, indicate which programs are 

available, the number of students who participated during the school year, and the grade level of students who participated.

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

targeted 

attendance 

interventions?

What data do you use 

to determine whether 

an 

intervention/support 

is working?

Write in

Drop down 

menu 

(Attendance 

Monitoring,

Contacting 

Truancy 

Officers Other 

please 

specify)

Drop down menu 

(from 1 to 999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

Drop down menu 

allowing checking 

all that apply 

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number 

of credits, Student 

attendance [e.g., 

missing a number 

of days of school 

or class], Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

Drop down menu 

allowing checking all 

that apply (attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other please 

specify)
REL Midwest Data Collection Instruments—87

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

targeted 

attendance 

interventions?

What data do you use 

to determine whether 

an 

intervention/support 

is working?

5. Does your school offer online credit or content recovery (e.g., Apex, K12, Plato, district program)?

o Yes 

o No (skip to 6)

c. Which of the following online content recovery programs does the school? In the table below, indicate which programs are 

available, the number of student who participated during the school year, and the grade level of students who participated.

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

online credit 

recovery?

What data do you use 

to determine whether 

the program is 

working?

Write in

Drop down 

menu 

allowing to 

check all that 

Drop down menu 

(from 1 to 999)

Drop 

down 

menu 

(from 

Drop down 

menu allowing 

to check all 

Drop down menu 

allowing to check 

all that apply 

Drop down menu 

allowing to check all 

that apply (attendance, 
REL Midwest Data Collection Instruments—88

Program 

Name

Focus of the 

Intervention 

# of Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

online credit 

recovery?

What data do you use 

to determine whether 

the program is 

working?

apply

(Reading, 

Mathematics, 

Science, 

Social 

Studies, Other 

please 

specify)

grade 9 to 

grade 12)

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number 

of credits, Student 

attendance [e.g., 

missing a number 

of days of school 

or class], Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other please 

specify)
REL Midwest Data Collection Instruments—89

6. Does your school offer student mentoring programs (e.g., Check & Connect, Check In/Check Out)?

o Yes

o No (skip to 7)

d. Which of the following mentoring programs does the school offer or do students have access to in the community? In the table 

below, indicate which programs are available, the number of students who participated during the school year, and the grade 

level of students who participated. 

Program 

Name

Focus of the 

Intervention 

# of

Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

mentoring?

What data do you use 

to determine whether 

the program is 

working?

Write in

Drop down menu 

(Academic 

mentoring, School 

adjustment mentoring 

[e.g., 9th grade 

transition program, 

reentry from 

adjudication], Career 

mentoring, Projectbased and 

community-based 

mentoring , Groupspecific mentoring, 

Peer mentoring, 

Other please specify)

Drop down 

menu 

(from 1 to 

999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

Drop down menu 

allowing checking 

all that apply 

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number 

of credits, Student 

attendance (e.g., 

missing a number 

of days of school 

or class), Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

Drop down menu 

allowing checking all 

that apply (attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other 

please specify)
REL Midwest Data Collection Instruments—90

Program 

Name

Focus of the 

Intervention 

# of

Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

mentoring?

What data do you use 

to determine whether 

the program is 

working?

Check & 

Connect

Check In 

Check Out 

Gear Up

7. Does your school offer a student internship or school-related work-preparation program (such as Job Corp, MACC Project, or 

career and technical education classes or programs)?

o Yes

o No (skip to 8) 

a. Which of the following student internship or school-related work-preparation programs does the school offer? In the table 

below, indicate which programs are available, number of student who participated during the school year, and the grade level 

of students who participated. 
REL Midwest Data Collection Instruments—91

Program Name Focus of the 

Intervention 

# of

Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

internships/workprep programs?

What data do you use 

to determine whether 

the program is 

working?

Write in

Drop down 

menu 

(internship, 

work-prep, 

Other pleas 

specify)

Drop down 

menu 

(from 1 to 

999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

Drop down menu 

allowing checking 

all that apply 

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number of 

credits, Student 

attendance [e.g., 

missing a number 

of days of school 

or class], Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

Drop down menu 

allowing checking all 

that apply (attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other 

please specify)

Job Corps

AmeriCorps 

Post-Secondary 

Enrollment Options 

program at Kent 

State University

Career and 

technical education 

program(s) (e.g., 
REL Midwest Data Collection Instruments—92

Program Name Focus of the 

Intervention 

# of

Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

internships/workprep programs?

What data do you use 

to determine whether 

the program is 

working?

culinary arts)

Career and 

technical education 

class(es) that are 

not part of a 

program 

Other—please 

describe:

8. Does your school offer student college preparation programs (such as AVID or Gear Up)?

o Yes

o No (skip to 9) 

a. Which of the following college preparation programs does the school offer? In the table below, indicate which programs are 

available, the number of students who participated during the school year, and the grade level of students who participated.
REL Midwest Data Collection Instruments—93

Program Name Focus of the 

Intervention 

# of

Participating 

Students

Grade 

Level(s) 

of

Students

Who refers 

students for 

this program?

What criterion 

is/criteria are 

used to select 

students for 

participation in 

college-prep 

programs?

What data do you use 

to determine whether 

the program is 

working?

Write in

Drop down 

menu (general 

college prep, 

financial aid 

assistance, 

application 

process 

assistance, 

support for 

college 

entrance 

exams, 

academics, 

Other please 

specify)

Drop down 

menu 

(from 1 to 

999)

Drop 

down 

menu 

(from 

grade 9 to 

grade 12)

Drop down 

menu allowing 

checking all 

that apply 

(Administrator, 

Counselor, 

Parent, 

Teacher, Data 

team, Student, 

[self], Other

Please specify)

Drop down menu 

allowing checking 

all that apply 

(Course failure 

[e.g., failing 

algebra], Grade 

point average 

[GPA], Number of 

credits, Student 

attendance [e.g., 

missing a number 

of days of school 

or class], Student 

behavior [e.g.,

office referrals, 

suspensions], No 

criteria, Other

Please specify)

Drop down menu 

allowing checking all 

that apply (attendance, 

behavior, course 

performance, core 

course performance, 

GPA, credits earned, 

behavior referrals, 

behavior suspensions, 

no criteria, Other 

please specify)

Early College High 

School

AVID 

Gear Up

Talent 

Development 
REL Midwest Data Collection Instruments—94

B-2. Draft Protocol—School Data Culture Survey

This survey will be administered to both treatment and control schools to estimate the impact of 

EWIMS on school level outcomes.

This attachment provides additional information on the draft protocols to measure School Data 

Culture. Table B-1 presents the reliability of the subscales and Key Dimensions of School Data 

Culture for principals and the reliability of the subscales and scales measuring principals’ report 

of School Data Culture. Cronbach’s alpha was calculated for each subscale and scale. It is 

generally accepted that Cronbach’s alphas above 0.70 indicate adequate internal consistency 

among the items in a scale. 

The reliability statistics of each scale and the number of survey items in each scale are shown 

below. The alpha statistic for all scales and subscales measuring School Data Culture 

(Assessment/Instructional Context, Supports for Data Use, Working with Data, and Instructional 

Responses to Data) is above 0.70, suggesting adequate reliability. Although ED’s contractor will 

revise items to fit the needs of the EWIMS evaluation, these Cronbach’s alphas are promising for 

the internal consistency of the revised School Data Culture Scale.
REL Midwest Data Collection Instruments—95

Table B-1. Reliability Estimates of Principals’ Key Dimensions of 

Data Use Scales and Subscales 

Cronbach’s Alpha

Key 

Dimension 

Scale

Subscales within each Key 

Dimension scale S

urvey

Item

N

General 

Principal 

Responses

Reading 

Instruction 

Specific

Math 

Instruction 

Specific

Context 32 0.87 0.87

Assessment/Instructional 

Context 23 0.82 0.81

State, District, and School Data 

Culture 9 0.84

Supports for 

Data Use 51 0.92

Data Infrastructure 8 0.82

Organizational Supports 31 0.90

Staffing/Human Resources 12 0.93

Working 

with Data 42 0.97 0.97

Individual Principal Attention 

to Data 17 0.93 0.93

Principal Collaboration Around 

Data 3 0.79

Principal-Teacher Collaboration 9 0.86

Principal-Coach Collaboration 4 0.83

Principal-Parent Collaboration 3 0.75 0.75

Principal-Student Collaboration 6 0.87

Instructional 

Responses 24 0.96 0.97

Establish/Adjust Groupings 4 0.80 0.80

Adjust Lesson Plans 6 0.88 0.88

Provide Supplemental 

Resources to Targeted Students 3 0.81 0.81

School-Level Instructional 

Response 11 0.92 0.93

Barriers to Data Use 13 0.76

Sample size: N = 212 Principals.

Note: The Supports for Data Use scale and Barriers to Data Use scale did not contain survey items that were reading 

or mathematics specific.

School-Level Instructional Response is a subscale specific to the Principal Survey.
REL Midwest Data Collection Instruments—96

Paperwork Burden Statement

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of 

information unless such collection displays a valid Office of Management and Budget (OMB) control 

number. The valid OMB control number for this information collection is XXXX-XXXX. The time required 

to complete this information collection is estimated to average 60 minutes per response. This information 

collection is voluntary. If you have any comments concerning the accuracy of the time estimate(s) or 

suggestions for improving this form, please write to: U.S. Department of Education, Washington, DC 20202–

4651. If you have comments or concerns regarding the status of your individual submission of this form, 

write directly to: Christopher Boccanfuso, U.S. Department of Education, Institute of Education Sciences, 

Room 506D, 555 New Jersey Ave. NW, Washington, DC 20208-5500.

Data Use Survey Items

1. How much do you agree or disagree with the following statements about your district’s 

priorities about using data?

Strongly

Disagree Disagree Agree StronglyAgree

a. The district sets clear, consistent goals for 

schools to use data for school improvement. ⃝ ⃝ ⃝ ⃝

b. District staff provide information and 

expertise that support the data use efforts at 

my school.

⃝ ⃝ ⃝ ⃝

c. The district’s data use policies help us address 

student needs at our school. ⃝ ⃝ ⃝ ⃝

d. The district has designated adequate resources 

(e.g. time, staff, money) to facilitate teachers’ 

use of data.

⃝ ⃝ ⃝ ⃝

2. Now consider the professional climate in your school. To what extent do you agree or 

disagree with each of the following statements?

Strongly 

Disagree Disagree Agree Strongly Agree

a. Educators in this school are continually 

learning and seeking new ideas. ⃝ ⃝ ⃝ ⃝

b. Educators are engaged in systematic analysis 

of student performance data. ⃝ ⃝ ⃝ ⃝

c. Educators in this school approach their work 

with inquiry and reflection. ⃝ ⃝ ⃝ ⃝

d. Assessment of student performance leads to 

changes in the curriculum. ⃝ ⃝ ⃝ ⃝

e. Educators in this school regularly examine 

school performance on assessments. ⃝ ⃝ ⃝ ⃝

3. To what extent do you agree or disagree with the following statements about your 

campus leaders/ administrators (e.g. principal, assistant principal)?
REL Midwest Data Collection Instruments—97

The campus leaders/administrators...

Strongly 

Disagree Disagree Agree Strongly Agree

a. Encourage teachers to make decisions based 

on data. ⃝ ⃝ ⃝ ⃝

b. facilitate conversations about using student 

data ⃝ ⃝ ⃝ ⃝

c. Commit resources to help teachers interpret 

and use the student data ⃝ ⃝ ⃝ ⃝

d. Place too much emphasis on using the student 

data ⃝ ⃝ ⃝ ⃝

4. Please read the following response options and pick the one that best reflects the degree 

of student data support provided by the campus leaders/administrators in your building:

o Not at all supportive: Does not make the student data a priority. There is limited 

discussion of the data with staff.

o Not very supportive: Occasional support for the student data use in faculty and staff 

discussions, but administrators do not see using the student data as central to the school’s 

mission.

o Supportive: Administration is supportive of teacher’s efforts, speaks positively about 

the student data with staff, problem-solves obstacles to using the student data, uses the 

system themselves.

o Very supportive: Administration is a “cheerleader” for the student data, effectively 

supports staff use of the system, and sees it as central to school mission.

5. Over the last 6 weeks, about how often did your school have scheduled meeting time to

As 

Departments

As GradeLevel TeamsAs a Whole 

Staff

a. Review student data (e.g., state test scores, 

student work, curriculum-based unit tests)

Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

b. Discuss student achievement by subgroup (e.g. 

students with disabilities, ELL/LEP, gender, 

race/ethnicity)

Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

c. Discuss individual student achievement Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

d. Discuss and share instructional strategies Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

e. Meet with an instructional coach Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

f. Meet with a data coach Dropdown 

menua

Dropdown 

menua

Dropdown 

menua

a Dropdown menu choices are About once a week, 1–2 times per month, 1–2 times per quarter, My school does 

provide time for this, and NA)
REL Midwest Data Collection Instruments—98

6. In the last 12 months, how much professional development have you attended that 

focused on identifying students at risk for dropping out?

No 

Emphasis

Minor 

Emphasis

Moderate 

Emphasis

Major 

Emphasis

a. Linking student data to classroom practice ⃝ ⃝ ⃝ ⃝

b. Analyzing student data ⃝ ⃝ ⃝ ⃝

c. Identifying strengths or weaknesses of your 

teaching ⃝ ⃝ ⃝ ⃝

d. Incorporating student data into lesson 

planning ⃝ ⃝ ⃝ ⃝

e. Using data to target interventions for lowperforming students ⃝ ⃝ ⃝ ⃝

f. Using data to target interventions for highperforming students ⃝ ⃝ ⃝ ⃝

g. Understanding how data can be used to guide 

instruction ⃝ ⃝ ⃝ ⃝

h. Understanding of how data can help you 

build on your students’ strengths ⃝ ⃝ ⃝ ⃝

7. In the past 6 weeks, approximately how many hours did you engage in analyzing or 

reviewing assessment data in the following ways?

None 1 Hour 

or Less

2 to 4 

Hours

5 to 7 

Hours

11 to 15 

Hours

More 

Than 15 

Hours

a. Independently ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

b. With teachers ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

c. With other administrators 

(principal, vice principal) ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

d. With a content-area coach 

(e.g., mathematics or 

reading coach)

⃝ ⃝ ⃝ ⃝ ⃝ ⃝

e. With a data coach (e.g., 

RDA accountability 

coordinators)

⃝ ⃝ ⃝ ⃝ ⃝ ⃝

f. With students ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

g. With parents ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

h. In a multidisciplinary team ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

i. In a team that is made up 

of teachers, administrators, 

and other roles

⃝ ⃝ ⃝ ⃝ ⃝ ⃝

j. Other ⃝ ⃝ ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—99

8. Does your school have a person who acts as a coordinator, facilitator, or coach to support 

data use?

o No, we do not have a person in this role

o Yes, this person is also a teacher at our school

o Yes, this person is a school administrator

o Yes, this person is from the district’s central office

o Yes, other

if “other,” please specify: 

______________________________________________________________

9. In your opinion, how much expertise in assessment data does your data coach have?

o None at all

o A slight amount of expertise

o A moderate amount of expertise

o A great amount of expertise

10. Please indicate how much assistance the data coach provides to your school in the 

following activities:

No 

assistance

Limited 

assistance

Moderate 

assistance

Substantial 

assistance

Don’t 

know

a. Organizing existing data, including 

assessment data ⃝ ⃝ ⃝ ⃝ ⃝

b. Analyzing and interpreting data ⃝ ⃝ ⃝ ⃝ ⃝

c. Helping staff develop their own 

capacity to analyze data ⃝ ⃝ ⃝ ⃝ ⃝

d. Writing school improvement plans ⃝ ⃝ ⃝ ⃝ ⃝

e. Suggesting curricular programs that 

could help our school ⃝ ⃝ ⃝ ⃝ ⃝

f. Suggesting specific strategies to address 

our needs (e.g., strategies for parent 

involvement, for English learners)

⃝ ⃝ ⃝ ⃝ ⃝

g. Acting as a facilitator during 

discussions among school staff about 

school improvement strategies

⃝ ⃝ ⃝ ⃝ ⃝

h. Working with the community or parents 

to explain the assessment process or 

results

⃝ ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—100

11. How many teachers at your school have the following characteristics?

None Some About 

Half Most All

a. The ability to use data from student 

assessments ⃝ ⃝ ⃝ ⃝ ⃝

b. The ability to analyze trends in 

individual student performance over 

time

⃝ ⃝ ⃝ ⃝ ⃝

c. The ability to analyze trends in 

classroom-level performance over 

time

⃝ ⃝ ⃝ ⃝ ⃝

d. The ability to translate data into 

knowledge about student strengths and 

weaknesses

⃝ ⃝ ⃝ ⃝ ⃝

e. The ability to make instructional 

changes based on data ⃝ ⃝ ⃝ ⃝ ⃝

12. To what extent does your school use the following student data? 

Do not use 

at all

Use 

minimally

Use 

moderately

Use 

extensively

Don’t 

know

a. Number or percent of students flagged 

for attendance problems in your school 

or class

⃝ ⃝ ⃝ ⃝ ⃝

b. Behavior/conduct/discipline referrals 

for individual students ⃝ ⃝ ⃝ ⃝ ⃝

c. Scale scores on achievement tests 

across your class ⃝ ⃝ ⃝ ⃝ ⃝

d. Individual student’s scores on 

achievement tests ⃝ ⃝ ⃝ ⃝ ⃝

e. Scores for subgroups of students (e.g., 

students with disabilities, ELL/LEP, 

gender, race/ethnicity) in your school 

or class

⃝ ⃝ ⃝ ⃝ ⃝

f. Student transcripts/course grades ⃝ ⃝ ⃝ ⃝ ⃝

g. Student enrollment in advanced 

courses

⃝ ⃝ ⃝ ⃝ ⃝

h. College entrance exam scores (PSAT, 

SAT, ACT) ⃝ ⃝ ⃝ ⃝ ⃝

i. Student information, including parent 

contact information and student 

program enrollment

⃝ ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—101

13. To what extent does your school use the following student data? 

Not made 

available 

this way

Used 

minimally

Used 

moderately

Use 

extensively

a. Percentage of students scoring at or above 

the proficiency level on state tests ⃝ ⃝ ⃝ ⃝

b. Percentage of students scoring at or above 

the proficiency level on local tests ⃝ ⃝ ⃝ ⃝

c. Results for subgroups of students (e.g., 

students with disabilities, ELL/LEP, 

gender, race/ethnicity)

⃝ ⃝ ⃝ ⃝

d. Results for the school as a whole. ⃝ ⃝ ⃝ ⃝

e. Results for each grade level ⃝ ⃝ ⃝ ⃝

f. Results for each classroom ⃝ ⃝ ⃝ ⃝

g. Results on specific topics or skills (e.g., 

computation, applications, word 

recognition, grammar)

⃝ ⃝ ⃝ ⃝

h. Changes or trends in the school’s results 

across years

⃝ ⃝ ⃝ ⃝

i. Student information, including parent 

contact information and student program 

enrollment

⃝ ⃝ ⃝ ⃝

14. Please tell us approximately how much time you spent reviewing or analyzing student 

data in the last 6 weeks.

o None

o 1 hour or less

o 1–2 hours

o 2–4 hours

o 4–5 hours

o 5 hours or more

15. In the past 6 weeks, approximately how many hours did you engage in reviewing the 

following types of data—not including training time?

None 1 Hour 

or Less

1 to 2 

Hours

2 to 4 

Hours

4 to 5 

Hours

5 or 

More 

Hours

a. Any type of data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

b. School-level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

c. Classroom-level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝

d. Student-level data ⃝ ⃝ ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—102

16. How much do you use student data to do the following things?

Do Not 

Use at All

Use 

Minimally

Use 

Moderately

Use 

Extensively

a. Diagnose learning problems ⃝ ⃝ ⃝ ⃝

b. Identify individual students who need 

remedial assistance ⃝ ⃝ ⃝ ⃝

c. Identify individual students who need 

acceleration ⃝ ⃝ ⃝ ⃝

d. Identify areas in which you want to 

strengthen your own content knowledge or 

teaching skills

⃝ ⃝ ⃝ ⃝

e. Identify individual students with 

attendance problems ⃝ ⃝ ⃝ ⃝

f. Identify individual students with behavior 

problems ⃝ ⃝ ⃝ ⃝

g. Identify individual students who are off 

track for graduation ⃝ ⃝ ⃝ ⃝

h. Assist students in setting individual 

performance goals ⃝ ⃝ ⃝ ⃝

17. How much influence do student data have on the following?

No 

Influence

Minor 

Influence

Moderate 

Influence

Major 

Influence

a. Determining a student’s grouping for 

instruction ⃝ ⃝ ⃝ ⃝

b. Determining a student’s class placements ⃝ ⃝ ⃝ ⃝

c. Adjusting goals for student learning ⃝ ⃝ ⃝ ⃝

d. Adjusting pacing in areas where students 

encountered problems ⃝ ⃝ ⃝ ⃝

e. Adjusting use of textbooks and 

instructional materials ⃝ ⃝ ⃝ ⃝

f. Evaluating school initiatives/programs ⃝ ⃝ ⃝ ⃝

g. Identifying students to be retained at the 

same grade level ⃝ ⃝ ⃝ ⃝

h. Identifying students for tutoring or other 

supplemental instruction ⃝ ⃝ ⃝ ⃝

i. Identifying professional development 

needs ⃝ ⃝ ⃝ ⃝

j. Evaluating teachers ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—103

18. On the basis of student data, how much have the teachers in your school changed the 

following aspects of their teaching?

No change Minor 

change

Moderate 

change

Major 

change

a. The curriculum content they teach ⃝ ⃝ ⃝ ⃝

b. Their expectations for student 

performance ⃝ ⃝ ⃝ ⃝

c. The instructional strategies they employ ⃝ ⃝ ⃝ ⃝

d. The types or mix of assessments they use 

to evaluate students ⃝ ⃝ ⃝ ⃝

19. To what extent do you use student data as the basis for the following actions?

Not at All

To a 

Minimal 

Extent

To a 

Moderate 

Extent

To a Great 

Extent

a. Recommending attendance monitoring for 

students ⃝ ⃝ ⃝ ⃝

b. Recommending behavior monitoring for 

students ⃝ ⃝ ⃝ ⃝

c. Recommending reading interventions for 

students ⃝ ⃝ ⃝ ⃝

d. Recommending mathematics interventions 

for students ⃝ ⃝ ⃝ ⃝

e. Recommending social-emotional/behavior 

interventions for students ⃝ ⃝ ⃝ ⃝

f. Contacting truancy officers ⃝ ⃝ ⃝ ⃝
REL Midwest Data Collection Instruments—104

B-3. Archival Records Protocol

The following section describes our approach to collect extant data primarily from the state of 

Ohio. The following data sharing processes and requests will also apply to data collected from 

districts and schools. 

Context in Ohio

The Ohio Department of Education (ODE) has a process established for researchers to use when 

requesting student level data. The following text is pulled directly from the Ohio department of 

Education website22 and describes the state’s approach to sharing student level data:

Requesting Student Level Data 

To protect student privacy, the Family Education Records Privacy Act of 1974 (FERPA) 

restricts the public release of student-level data by state education agencies. FERPA does 

include circumstances under which the Ohio Department of Education (ODE) is authorized 

to release data from an education record. Ohio Revised Code § 3301.12 grants to the State 

Superintendent of Public Instruction the authority to approve the release of this 

information to research organizations conducting studies on behalf of ODE. 

Research institutions wishing to conduct evaluative studies using student-level data must 

file a Memorandum of Understanding (MOU) with ODE (see below for template). The 

MOU will, among other things, detail the expectation that the research organization will 

assume responsibility for the ongoing protection of the education records upon receipt of 

student level data. To learn more about the commitments required of researchers in 

ODE’s MOU, please review the “Understanding ODE’s MOU” document below. 

ODE’s MOU requires researchers to provide a detailed list of data elements requested for the 

purpose of the research study. ODE collects student data via the Education Management 

Information System (EMIS). Prior to submitting an MOU, please review the EMIS Manual to 

compile an initial list of requested data elements to complete the MOU Appendix A. The EMIS 

Manual is available to the public on ODE’s Education Management Information System website.

Contact Information

Matthew Cohen, Chief Research Officer

Office of Policy & Research

Ohio Department of Education

25 S. Front Street, 7th Floor

Columbus, OH 43215

614-752-8729

Matt.Cohen@education.ohio.gov

 

22

http://education.ohio.gov/GD/Templates/Pages/ODE/ODEDetail.aspx?page=3&TopicRelationID=3&ContentID=109

164&Content=133034
REL Midwest Data Collection Instruments—105

Our process will take into account Ohio’s approach to sharing data with researchers. Table B-2 

describes the steps in the data sharing process between Ohio and REL Midwest.

Table B-2. Ohio and REL Midwest Data Sharing

Action Method/Mode Key Ohio Staff Key REL Midwest 

Staff Date

First contact with 

ODE 

Half an hour 

phone 

conversation

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria, 

Nicholas Sorensen, 

Mindee 

O’Cummings, 

Jessica Heppen

April 2013

AIR will draft data 

sharing MOU and 

share with ODE

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria May 2013

ODE will provide 

feedback on data 

sharing MOU

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria May 2013

AIR will finalize 

and sign MOU

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria January 2014

ODE will sign 

MOU

E-mail 

communication 

with hard copies 

mailed to AIR

Matthew Cohen, 

Chief Research 

Officer, 

Michael L. Sawyers, 

Acting State 

Superintendent of 

Public Instruction

Ann-Marie Faria January 2014

AIR will establish 

secure FTP site for 

data sharing

Create the 

Secure FTP

N/A Ann-Marie Faria January 2014

AIR will submit 

second data request 

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria January 2014

ODE will provide 

second round of 

student-level data

Secure FTP 

established by 

AIR for use in 

the EWIMS 

study

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria January 2014

AIR will submit 

third data request 

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria June 2014

ODE will provide 

third round of 

student-level data

Secure FTP 

established by 

AIR for use in 

the EWIMS 

study

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria June 2014
REL Midwest Data Collection Instruments—106

Action Method/Mode Key Ohio Staff Key REL Midwest 

Staff Date

AIR will draft 

annual report to 

ODE summarizing 

research conducted 

with the data

Email 

communication 

with attachment

Ann-Marie Faria, 

Nicholas Sorensen, 

Jessica Heppen

December 2014

AIR will submit 

fourth data request 

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria January 2015 

ODE will provide 

fourth round of

student-level data

Secure FTP 

established by 

AIR for use in 

the EWIMS 

study

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria January 2015

AIR will submit 

fifth and final data 

request 

E-mail 

communication

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria June 2015

ODE will provide 

fifth round of 

student-level data

Secure FTP 

established by 

AIR for use in 

the EWIMS 

study

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria June 2015

AIR will draft 

annual report to 

ODE summarizing 

research conducted 

with the data

Email 

communication 

with attachment

Ann-Marie Faria, 

Nicholas Sorensen, 

Jessica Heppen

December 2015

Provide ODE with 

any material 

designed for public 

dissemination, 

(copy of final 

report)

Email 

communication 

with attachment

Matthew Cohen, 

Chief Research 

Officer, 

Michael L. Sawyers, 

Acting State 

Superintendent of 

Public Instruction 

Ann-Marie Faria, 

Nicholas Sorensen, 

Mindee 

O’Cummings, 

Jessica Heppen

December 2015

Destroy all data 

upon completion of 

project (after 

creating RUF files 

for the U.S. 

Department of 

Education)

Matthew Cohen, 

Chief Research 

Officer

Ann-Marie Faria, 

Nicholas Sorensen

December 2016

Establishing a secure system to share data (SFTP). One key aspect of using extant data is the 

secure transfer of data between agencies. AIR will create a secure FTP and will share the log-in 

information with ODE. A draft of the directions that will be shared with ODE from AIR are 

included below. 
REL Midwest Data Collection Instruments—107

WEB HOSTING SERVICES "SFTP Client Guide" Public Information

DESCRIPTION: 

This SOP outlines the procedures to use Web Services’ SFTP service. 

PROCEDURE: 

A. Open the email received from WS Engineer with access credentials to the SFTP site. 
REL Midwest Data Collection Instruments—108

B. Follow Steps 1-7 in the email: 

1. Open your internet browser and type http://sftp.airws.org or http://sftp.astprojects.org in 

the address field, depending on which website you were given in the email. 

A. Verify whether Java is installed/enabled in your browser: 

B. If Java is not installed/ enabled in your browser, click “Install”. 
REL Midwest Data Collection Instruments—109

C. Accept license agreement, and choose all defaults: 
REL Midwest Data Collection Instruments—110

D. Reboot your computer, and repeat step #1. 

2. Click the “Connect Button” on the webpage: 
REL Midwest Data Collection Instruments—111

3. A dialog box will pop up, asking for a username and password. Refer to the email for these 

credentials. Type the given credentials EXACTLY as they appear in the email. Click OK. 

EMAIL 

DIALOG BOX 
REL Midwest Data Collection Instruments—112

4. Upon a successful connection you should see the following on the webpage: 

The figure above shows a message that you have successfully connected to the SFTP server. On 

the right, it also shows a list of virtual directories located on the remote system, and a field to 

type the directory path (encircled). On the left is the directory listing for your local machine. 
REL Midwest Data Collection Instruments—113

5. Refer to step #5 in the email for the name of the specific directory created for your 

project. Type this name EXACTLY as it appears in the instructions, or find it in the 

directory listing and navigate manually using your mouse. 
REL Midwest Data Collection Instruments—114

6. Double click the subfolder contained in the virtual directory. In this subfolder, you can 

upload/download files or create more folders, depending on the read/write permissions 

granted for your SFTP account. 

7. Use the arrows between the two windows to move files back and forth between your 

local machine and the SFTP folder in the remote server. Press the “right” arrow button to 

upload a highlighted file from your machine to the SFTP folder, and the “left” arrow 

button to download a file from the SFTP folder to your machine. 

8. Click the “Disconnect” button when finished. 
REL Midwest Data Collection Instruments—115

Data Sharing Agreements/MOUs. A second key component of leveraging extant data for 

research purposes are the MOUs and data requests between the two organizations. We will 

request data at 5 time points: baseline data (September 2014), end of Semester 1 Year 1 (January

2014) end of Semester 2 Year 1 (June 2014), end of Semester 1 Year 2 (January 2015) end of 

Semester 2 Year 2 (June 2015). The data requests to be shared between AIR and ODE will 

follow the structure of the following structure:

Dear Matthew Cohen, Chief Research Officer,

My name is Dr. Ann-Marie Faria and I am writing to submit our request for student data for all 

the students in the EWIMS Impact Study. This data request includes students in 72 schools in 

Ohio. This is the first of five data requests that we will submit over the next two years. Today we 

are requesting all data for the fall of the 2013-2014 school year. Below is a list of the types of 

information that we need, with specific data formats and time periods indicated:

1. Attendance data:

a. Attendance status (present/ absent by day and first week enrollment status) for the 

first 5 days of the school year for 2013-14 school year).

b. Attendance data (days enrolled, days present, days absent, and percent of days 

present) for the first 30 days of the school year for 2013-14

c. Cumulative number of credits attempted and earned by each student that will 

count toward graduation, to date as of January 2014. 

2. School enrollment data:

a. School enrolled in (school name and school ID number) at the beginning and end 

of the semester for each semester of 2013-14

b. Student grade level in 2013-14

c. Enrollment status data (school name, exit reason/ still enrolled, and exit date and 

receiving school as applicable) for 2013-14

3. Disciplinary incidents:

a. All disciplinary incidents (suspensions and other school disciplinary reports) for 

2013-14

4. Student special education and ELL identification:

a. Special education data (special education status, identification date, and disability

categories if applicable) for 2013-14

b. English language learner data (ELL status, identification date, and level if 

applicable) for 2013-14
REL Midwest Data Collection Instruments—116

5. Student and family contact information:

a. Contact info (address, phone number, primary contact person) as of September 

2013

Thanks so much for all your help. And, as always, let us know if you have any questions. 

Ann-Marie Faria, Ph.D.

1000 Thomas Jefferson St., NW

Washington DC 20009

202 403 5356

afaria@air.org
REL Midwest Data Collection Instruments—117

B-4. Protocol for Collecting Data Stored in the EWS tool

in Treatment Schools

To facilitate the transfer of EWS tools from treatment schools to the EWIMS evaluation team, 

we will use the same FTP procedure as outlined in Attachment B-3. The table below describes 

the timeline of activities between treatment schools and the evaluation team during the two years 

of the study. 

Action Method/Mode Key Ohio Staff Key REL 

Midwest Staff Date

AIR will establish 

secure FTP site for 

data sharing

Create the 

Secure FTP

N/A Ann-Marie Faria January 2014

AIR will email FTP 

secure log in 

information to 

Treatment schools

E-mail 

communication

January 2014, 

June 2014, 

January 2015, 

June 2015

Treatment schools 

will upload populated 

EWS tools to the 

secure FTP

E-mail 

communication

Data Teams in 

Treatment Schools

Ann-Marie Faria January 2014, 

June 2014, 

January 2015, 

June 2015
REL Midwest Data Collection Instruments—118

Establishing a secure system to share data (SFTP). One key aspect of transferring populated 

EWS tools between schools and the evaluation team is the use of a secure transfer of data 

between agencies. AIR will create a secure FTP and will share the log-in information with each 

treatment school. A draft of the directions that will be shared with each treatment school from 

AIR are included below. 

WEB HOSTING SERVICES "SFTP Client Guide" Public Information

DESCRIPTION: 

This SOP outlines the procedures to use Web Services’ SFTP service. 

PROCEDURE: 

A. Open the email received from WS Engineer with access credentials to the SFTP site. 
REL Midwest Data Collection Instruments—119

B. Follow Steps 1-7 in the email:

1. Open your internet browser and type http://sftp.airws.org or http://sftp.astprojects.org in the 

address field, depending on which website you were given in the email. 

a. Verify whether Java is installed/enabled in your browser: 

b. If Java is not installed/ enabled in your browser, click “Install”. 
REL Midwest Data Collection Instruments—120

c. Accept license agreement, and choose all defaults: Web Services Page 2 of 6 

d. Reboot your computer, and repeat step #1. 
REL Midwest Data Collection Instruments—121

2. Click the “Connect Button” on the webpage: 
REL Midwest Data Collection Instruments—122

3. A dialog box will pop up, asking for a username and password. Refer to the email for these 

credentials. Type the given credentials EXACTLY as they appear in the email. Click OK. 

EMAIL 

DIALOG BOX 
REL Midwest Data Collection Instruments—123

4. Upon a successful connection you should see the following on the webpage: 

The figure above shows a message that you have successfully connected to the SFTP server. On 

the right, it also shows a list of virtual directories located on the remote system, and a field to 

type the directory path (encircled). On the left is the directory listing for your local machine. 
REL Midwest Data Collection Instruments—124

5. Refer to step #5 in the email for the name of the specific directory created for your project. 

Type this name EXACTLY as it appears in the instructions, or find it in the directory listing 

and navigate manually using your mouse. 
REL Midwest Data Collection Instruments—125

6. Double click the subfolder contained in the virtual directory. In this subfolder, you can 

upload/download files or create more folders, depending on the read/write permissions 

granted for your SFTP account. 

Web Services Page 5 of 6 

7. Use the arrows between the two windows to move files back and forth between your local 

machine and the SFTP folder in the remote server. Press the “right” arrow button to upload a 

highlighted file from your machine to the SFTP folder, and the “left” arrow button to 

download a file from the SFTP folder to your machine. 

Click the “Disconnect” button when finished.