Memorandum

REL Midwest Memorandum on RELMW Responses to OMB—1

To: Joelle Lastica, Institute of Education Sciences

From: R. Dean Gerdeman, Director, REL Midwest

Jim Lindsay, Senior Researcher, REL Midwest 

Date: April 8, 2014

Re: Response to questions from OMB regarding ICR for Ramp-Up Implementation Study.

This memo contains REL Midwest’s responses to OMB’s questions (see italicized colored text following 

each question).

1. Do the later-implementing schools know that they will be getting resources to implement the 

Ramp-Up program at any point during the study? If so, could there be possible bias in how 

these schools describe their existing college-readiness activities since they have self-selected 

to receive something else?

RELMW: The later-implementing schools know that implementation within their school is being 

delayed by one year. The choice of which schools would implement Ramp-Up earlier versus 

later was made by the program developer, through a random process. The developer—The 

College Readiness Consortium—framed the school selection process as either implement 

Ramp-Up early or continue their present college readiness activities for another year at which 

point they would have the chance to implement Ramp-Up. In sum, the random assignment of 

schools to conditions should mitigate any bias in responding among schools.

2. FRPL is not a particularly reliable indicator of socio-economic status in high school. Will there 

be any other comparisons in measures related to college readiness for other subgroups of 

students such as by race/ethnicity? If no, why not?

RELMW: We did not propose examining subgroups of students by race/ethnicity because 

Minnesota schools in general serve predominantly white populations of students (per the latest 

data on Minnesota Department of Education’s website, 91 percent of the students in the 

sampled schools are Caucasian). Students of different racial/ethnic groups tend to concentrate 

in specific schools or counties (i.e., Native American populations in North Central region of the 

state, African American and Hmong—or Asian—in the Twin Cities). Thus race/ethnicity is 

confounded with school, making such a comparison less informative. 

In addition, this project is being performed in collaboration with a group of stakeholders from 

different states who all have an interest in using research to improve the college and career 

success of students. Perhaps it was their knowledge of Minnesota that influenced their 

preference for the research question to focus on student poverty status, represented by FRPL—

as unreliable as it is—for subgroup analysis. 
REL Midwest Memorandum on RELMW Responses to OMB —2

We will acknowledge the potential reliability concerns surrounding FRPL in the “limitations” 

section of our report.

3. Is there any planned follow up for an impact evaluation of the Ramp-Up program?

RELMW: Much as we would love to track the students into their post-secondary lives, we do 

not plan any longer-term follow-up with the sample of 22 schools or the students from these 

schools who may or may not continue on into post-secondary education. There are multiple 

reasons for why no follow-up is planned with this sample: 

1. A sample of 22 schools is insufficient for detecting impact of this intervention; 

2. The REL contract is time-limited, with an expectation that reports be completed by 

December 31, 2016. Another wave of data collection would make that deadline difficult 

to meet;

3. At present in Minnesota, attempting to track students from secondary level to postsecondary means asking for social security number, which might influence more parents 

to opt their child out of the implementation study. We did not want to face the 

possibility of sacrificing students from the implementation study for the chance to obtain 

a poorly-powered impact estimate. 

However, REL Midwest has proposed another study of Ramp-Up’s impact, this one with a 

larger sample of schools which is completely independent of the implementation study sample.

4. A comment on language: throughout the Supporting Statement, the study team describes the 

results of this study as “evidence”. OMB is very sensitive to how “evidence” is used, and in 

OMB’s view, evidence is only produced by studies with rigorous methodological designs that 

attempt to measure causal impact. OMB would prefer that the study team not use “evidence” 

in reference to this study since OMB views it as a mischaracterization of the information 

coming out it.

RELMW: We had made changes in Supporting Statement A and Supporting Statement B so 

that the term “evidence” is used only in reference to findings produced from studies that meet 

What Works Clearinghouses group design standards or studies that meet What Works 

Clearinghouses group design standards, with reservations.