Part B: Supporting 

Statement for 

Paperwork Reduction 

Act Submission

Effectiveness of 

Promising Strategies in 

Federal College Access 

Programs: Study

Student Messaging in 

GEAR UP 

Prepared for:

Marsha Silverberg

U.S. Department of Education

555 New Jersey Ave, NW

Room 502I

Washington, DC 20208-5500

Submitted by:

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Table of Contents ▌pg. i

Part B: Supporting Statement for Paperwork Reduction Act 

Submission

Table of Contents

B. Collection of Information Employing Statistical Methods.................................................B-1

Introduction..............................................................................................................................B-1

B.1 Respondent Universe and Sampling Methods...............................................................B-6

B.1.1 Respondent Universe.......................................................................................B-6

B.1.2 Sampling Methods...........................................................................................B-7

B.1.3 Expected Response Rates................................................................................B-7

B.2 Statistical Methods for Sample Selection and Degree of Accuracy Needed.................B-7

B.2.1 Sample Selection .............................................................................................B-7

B.2.2 Estimation Procedures....................................... Error! Bookmark not defined.

B.2.3 Degree of Accuracy Needed............................................................................B-7

B.2.4 Unusual Problems Requiring Specialized Sampling Procedures ..................B-14

B.2.5 Use of Periodic Data Collection Cycles to Reduce Burden...........................B-14

B.3 Methods to Maximize Response Rates and Deal with Nonresponse...........................B-14

B.4 Test of Procedures and Methods to be Undertaken.....................................................B-15

B.5 Individuals Consulted on Statistical Aspects of the Design ........................................B-15

References..............................................................................................................................B-16
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-1

B. Collection of Information Employing Statistical Methods

Introduction

The U.S. Department of Education (ED) will conduct a demonstration to test the effectiveness of a 

promising strategy to improve college-related outcomes in the federal college access program 

Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP). The 

demonstration will use a randomized controlled trial (RCT) design to test the effectiveness of sending 

customized messaging to students, first during the summer after high school graduation, and then in 

the fall and spring of their expected first year of college. 

This study is being sponsored by ED’s Institute of Education Sciences (IES) , in collaboration with 

the Office of Postsecondary Education, and implemented by Abt Associates Inc. and its partners, 

Survey Research Management (SRM), Program and Policy Insight (PPI), Digital Design Group 

(DDG), and consultant Ben Castleman (together, the “study team”).

Overview of the GEAR UP Program 

Section 404H of the Higher Education Act of 1965, amended by the 2008 Higher Education 

Opportunities Act (HEOA) (20 USC 1070A-27) authorizes the GEAR UP program. The program 

provides funding – over $300 million in FY 2014 – to state agencies and local partnerships to support 

disadvantaged students in obtaining a high school diploma and to prepare them for success in 

postsecondary education. Grantees are expected to provide a comprehensive set of intervention 

services such as academic support, counseling, mentoring, and outreach to disadvantaged middle and 

high school students, as well as information about college financial aid options and the benefits of 

postsecondary education to students and their families. State grantees may, and partnership grantees 

must, focus on a cohort of students (typically all those in 7th or 8th grade) and provide services to them 

through high school, though funds can be used to serve other students as well. State GEAR UP 

grantees are required to, and local partnership can, also provide scholarships to help participating 

students pay for college attendance. 

Importantly, starting with the 2008 Higher Education Opportunity Act, GEAR UP grantees may now 

apply for a seventh year of funding to support students into their first year of college in order to 

improve college enrollment and persistence. The first group of grantees with this funding, those 

awarded grants in FY 2011, will begin providing extended services to 2016-2017 seniors transitioning 

to postsecondary education the following year.

Overview of the Student Messaging in GEAR UP Demonstration

This demonstration is designed to address a new need among GEAR UP grantees for strategies to 

serve students during the 7th year. According to recent focus groups conducted by Abt with GEAR 

UP grantees, grantees find it difficult to reach every GEAR UP student on a one-on-one basis while in 

high school, or to track and support students as they leave school and transition to college, 

particularly since students may be scattered across a variety of postsecondary institutions both in and 

out of state. Nationally, as many as 40 percent of first-generation, low-income high school seniors 

intending to go to college fail to enroll the following fall, with information gaps, logistical 

complexities, and concerns about in fitting identified as primary explanations of this “summer melt” 

(Castleman and Paige, 2014). Of those who matriculate, many students drop out during their first 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-2

year or fail to re-enroll the second year (NSC, 2014b), and persistence is lower among students from 

low-income and high minority high schools (NSC, 2014a).

The demonstration will investigate a particular promising approach emerging from recent rigorous 

research of approaches to improve college matriculation (Castleman & Page 2013) and college 

persistence (Castleman & Page, 2014) through low-cost student messaging, which may enhance the 

existing GEAR UP services. Customized messages will be sent to students identified as seniors, who 

intend to go to college, in high schools that are part of the GEAR UP program. The messages, 

delivered via a digital format (phone texting), will remind students about tasks related to enrolling in 

and persisting in college. All students will continue to be eligible for any regular GEAR UP services 

provided. Outcomes for this study will include college enrollment, FAFSA completion, and 

persistence, which will be drawn from administrative data.

Student messaging will proceed in two phases and comprise two independent treatments. Phase 1 

messages (Treatment 1) will be sent during the summer that follows students’ senior year in high 

school. Phase 2 messages (Treatment 2) will be sent during what would be students’ first year of 

college if they enroll in college directly after high school. 

Students randomly assigned to Treatment 1 will receive a minimum of 10 messages designed to relay 

information that students may not be aware of or may not have easy access to. The messages will 

inform students about key enrollment-related tasks they need to complete such as submitting a 

deposit, filling out the housing form, submitting their proof of health insurance form, registering for 

classes, etc. Students randomly assigned to Treatment 2 will receive a minimum of 8 messages 

designed to support students to complete their first semester and persist into the second semester. 

These messages will be focused on key tasks students must complete such as paying tuition and 

registering for the next semester as well as common challenges faced in the first semester such as 

academic unpreparedness and lack of knowledge about campus supports.

All messages will be grounded in both social psychological and behavioral economic research 

suggesting that the messages be aligned to key time points such as housing deadlines, registration 

deadlines, tuition payment due dates, the add/drop period, and mid-term exams and that the messages 

be action oriented such as submit your housing form and go to this webpage to register for classes. 

To the extent possible, messages will be customized for each student based on the information they 

provide on the baseline survey about college intentions. For example, deadlines for submitting 

particular forms and links to find more information will be specific to the college a student indicated 

as the planned college of enrollment on the baseline survey. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-3

Exhibit B-1 contains a preliminary outline of key prompts that are timed to key milestones or pressure 

points in enrolling and persisting in college. 

Exhibit B-1. Draft Outline of Student Messages

Topic Personalization

Enrollment Messages (Treatment 1)

Log into college's web portal to access important

paperwork and deadlines

Link to specific website

Register for orientation and placement tests Institution-specific registration deadlines

Complete housing forms Link to specific forms, include deadlines

Sign up or waive health insurance (if relevant) Link to specific forms, include deadlines

Submit deposit for fall enrollment Institution-specific deposit deadlines

Register for fall classes Institution-specific registration deadlines

Reminder of orientation and/or welcome 

activities dates

Institution-specific dates/activities

Persistence Messages (Treatment 2)

Provide links to campus resources for student 

support

Links to specific resources

Reminder about upcoming midterms Link to campus specific academic support center

Mindset booster on overcoming challenges in 

college

Link to campus counseling center

Check financial aid eligibility for second year Link to requirements

Register for spring classes Institution-specific registration deadlines

Overview of the Evaluation

The study will use a rigorous random assignment design to test the effectiveness of sending 

customized messages to GEAR UP students. GEAR UP projects that were awarded awards in Federal 

Fiscal Year 2011 will be invited to volunteer for the demonstration (see Appendix A for invitation), 

because they are the first to have funds to support a 7th year of services. To test the effectiveness of 

the customized messaging, the study will assign students within participating high schools selected by 

volunteer GEAR UP projects to either a control condition (eligible for all GEAR UP services), or a 

treatment condition (eligible for all GEAR UP services, plus receiving the customized messages). 

Students will be randomly assigned to conditions at two points in time (1) in the spring of their senior 

year of high school and (2) the end of summer after senior of high school. This two-phase design 

maximizes our ability to investigate both the effects of messaging during the summer before college 

and the effects of messaging during freshman year of college.

 At the first phase of random assignment (end of 2016-2017 school year), students in high

schools served by GEAR UP projects who declare their intention to enroll in college will be 

randomly assigned to receive messages in the summer after high school (Treatment 1) or not 

(Control 1). The comparison of Treatment 1 to Control 1 will test the impacts of messaging 

on college enrollment – i.e., to see if it reduces summer melt among GEAR UP students. 

 At the second phase of random assignment (end of summer 2017), students will be randomly 

assigned to receive messages (Treatment 2) or not (Control 2) in what would be their first 

year of college if they go to college right after high school. Random assignment in this 

second phase will occur within both the Treatment 1 and Control 1 conditions, so that within 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-4

the Treatment 1 and Control 1 groups, half of the students will be assigned to Treatment 2 

and half assigned to Control 2. Exhibit B-2 displays the phases of random assignment and 

their relationship to each other. Comparison of Treatment 2 to Control 2 will test the impacts 

of different messaging on college persistence. 

Exhibit B-2: Two-Phase Randomization Process and Treatment Received 

Timing of 

Randomization

College Intending Seniors

Phase 1 End of 2016-17 

school year

T 1 C1

(summer messaging) (no summer messaging)

Phase 2

End of summer 

2017

 T2 C2 T2 C2

(summer and frosh 

messaging)

(only summer 

messaging)

(only frosh 

messaging)

(no messaging)

Baseline data will be collected from all students via a survey, and college-related outcome data will 

be extracted from national datasets (National Student Clearinghouse Data (NSC) and the Federal 

Student Aid (FSA) database). 

The study will address the three research questions:

1. To what extent do the messages—above and beyond the services GEAR UP may already 

provide—improve student college-related outcomes? 

2. What types of college advising are typically received by GEAR UP students? 

3. Is there variation in the impact of the messages, and to what extent is the variation associated 

with student characteristics or features of the GEAR UP project or school? 

To answer these questions we will conduct both impact and descriptive analyses. The first report, 

which will address each research question with a focus on college enrollment-related outcomes will 

be published in 2018. The final report, which will also include college persistence, will be available in 

2020.

Exhibit B-3 presents the research questions along with the data sources for each question, the analytic 

approach and outcomes of interest. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-5

Exhibit B-3. Evaluation Questions, Data Sources, Analytic Approach, and Outcomes of Interest

Research Question Data Sources Analytic Approach

Outcomes of 

Interest

1) To what extent do the 

messages—above and 

beyond the services 

GEAR UP may already 

provide—improve 

student college-related 

outcomes?

 Student rosters

 Student baseline 

survey data

 National Student 

Clearinghouse Data

 Federal Student Aid 

(FSA) database

 Impact analysis of RCT 

(HLM)

 Baseline survey sample: 

16,000 students in 80 

schools.

 Analytic sample size: 

approximately 5,600 

students in 80 high 

schools that intend to 

enroll in college in 

Phase 1; approximately 

4,000 students across a 

large number of colleges 

in Phase 2, equally 

divided between 

treatment and control 

groups. 

 College 

enrollment 

(Phase I)

 College 

persistence 

into sophomore 

year (Phase II)

 FAFSA 

completion 

2) What types of college 

advising are typically 

received by GEAR UP 

students?

 Student baseline 

survey data

 Descriptive analysis

 Survey sample: 16,000 

students in 80 schools.

 Description of 

GEAR UP 

program 

components 

provided and 

received 

3) Is there variation in the 

impact of the 

messages, and to what 

extent is the variation 

associated with student 

characteristics or 

features of the GEAR 

UP project or school?

 Student baseline 

survey data

 National Student 

Clearinghouse Data

 Federal Student Aid 

(FSA) database

 Descriptive (subgroup) 

analysis

 Sample size: 5,600 

students in 80 high 

schools that intend to 

enroll in college in 

Phase 1; approximately 

4,000 students across a 

large number of colleges 

in Phase 2.

 College 

enrollment 

(Phase I)

 College 

persistence 

into sophomore 

year (Phase II)

 FAFSA 

completion

Altogether, we expect that 80 GEAR UP high schools will participate in the first phase of the research 

project, with an estimated total of 16,000 seniors.

1

 All seniors will be surveyed, with the consent 

process giving them (or the parents) an option to opt of completing the baseline survey or other data 

collection activities. The survey will gather information from students for participation in the student 

messaging intervention (see Appendix C for the consent letter). 

 

1 An estimate of the number of seniors in each school is 200, given that the average secondary school 

enrollment in 2011-2012 was 788 students (NCES, 2013).
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-6

It is expected that 70 percent of seniors will express intent to attend college2 and 50 percent of these 

seniors would provide assent and information for texting, resulting in 2,800 students in the Treatment 

1 group, and 2,800 students in the Control 1 group. Assuming a matriculation rate for seniors of 50 

percent3the phase 2 sample will include 4,000 students entering college, 2,000 students in Treatment 

2 group and 2,000 students in the Control 2 group.

B.1 Respondent Universe and Sampling Methods

B.1.1 Respondent Universe

The respondent universe includes the FY 2011 GEAR UP grantees and associated schools that 

volunteer for the demonstration. Up to 12 GEAR UP grantees will be recruited involving 

approximately 80 schools. To participate in the demonstration, schools will need to agree to allow 

students to be randomly assigned to the intervention. The seniors in these schools in the 2016-2017 

academic year who intend to go to college and who who do not opt-out in the consent process will be 

the target population. 

As noted above an estimate of the number of seniors in each school is 200, given that the average 

secondary school enrollment in 2011-2012 was 788 students (NCES, 2013).

4 Of these, we expect 70 

percent of these to express intent to attend college, and 50 percent of these to assent to receive text 

messages, resulting in a total of 5,600 students in Phase I.5 Assuming a matriculation rate of high 

school seniors of 50 percent6the Phase II sample will include 4,000 students entering college. Exhibit 

B-4 indicates the estimated population sizes for the evaluation. 

 

2 Among 2010 seniors, 82 percent of graduating students planned to attend a postsecondary institution and 

the percent among parents with no college education was 71 percent (Aud et al. 2012).

3 This is the immediate college enrollment rate in the first fall after high school graduation for students 

graduating from low income schools according to the national student clearinghouse research center

(National Student Clearinghouse 2014a). 

4 The estimate of the number of seniors in GEAR-UP schools is based on the average enrollment of 788 in 

regular secondary schools in 2011-2012, assuming four grades per school (NCES 2013). 

5 Among 2010 seniors, 82 percent of graduating students planned to attend a postsecondary institution and 

the percent among parents with no college education was 71 percent (Aud et al. 2012).

6 This is the immediate college enrollment rate in the first fall after high school graduation for students 

graduating from low income schools according to the national student clearinghouse research center

(National Student Clearinghouse 2014a). 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-7

Exhibit B-4. Population Size Estimates

Estimated number of GEAR UP high schools 80 schools

Estimated number of 12th graders in GEAR UP cohort per school 200 students

TOTALS

 High schools 80

 12

th graders in GEAR UP high schools 16,000

 12th grade GEAR UP Phase 1 demonstration participants 5,600

 Phase 2 demonstration participants 4,000

B.1.2 Sampling Methods

There will be no sampling of schools for the evaluation. The entire universe of GEAR UP schools 

within the demonstration grants who have agreed to random assignment will participate in the 

evaluation study. All 12th grade students in these schools will participate in the evaluation. Those who 

assent to participate in the intervention and intend to go to college will be involved in the impact 

study. 

B.1.3 Expected Response Rates

We expect to obtain a 100 percent response on the rosters from schools. We expect an 85 percent 

response rate on the baseline survey. Outcome data including college matriculation and persistence 

will be drawn from extant NSC data. The NSC includes more than 3,600 enrolling 98 percent of all 

students in public and private U.S. institutions, (NSC 2014c), and NSC the primary reason enrolled 

students are not found in NSC is because the student suppressed their data; for the academic years of 

2006-2007, 2008-2009, and 2010-2011 was this rate was 4.7 percent (NSC 2012).

B.2 Statistical Methods for Sample Selection and Degree of Accuracy

Needed

B.2.1 Sample Selection

Random assignment will be conducted twice, once prior to each phase. The first random assignment 

will take place at the end of the 2016-2017 school year; students in high schools served by GEAR UP 

projects who declare their intention to enroll in college will be randomly assigned to receive messages 

in the summer after high school (Treatment 1) or not (Control 1). The second random assignment will 

take place at the end of summer 2017 and students within each Phase 1 condition will be randomly 

assigned to receive messages in what would be sent during their first year of college if they go to 

college right after high school (Treatment 2) or not (Control 2). 

B.2.2 Estimation Procedures

Impact and descriptive analyses will be conducted to answer the study research questions, as 

described below. Part B of this Supporting Statement provides additional methodological detail.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-8

1. To what extent do the messages—above and beyond the services GEAR UP may already 

provide—improve student outcomes? 

Given that participating schools are volunteering to participate in the study (rather than being selected 

at random) and inferences will be focused on students in these or similar schools, the study will use a 

model with fixed treatment effects to assess the effect of messaging on student outcomes (Schochet, 

2008).7The study team will control for student demographic characteristics, including gender, 

race/ethnicity, and first generation college student status. Reflecting the randomization of students 

within schools, the study team will include a set of dummy variables for schools and a treatment 

indicator to provide an estimate of the average impact of the messaging demonstration on students 

across all schools.

The study team will conduct a test of the null hypothesis of no treatment impact for each outcome 

variable. The study team will interpret a positive and statistically significant impact at the 5-percent 

level (using a two-tailed test) as compelling scientific evidence that the intervention improves the 

targeted outcome (e.g., college enrollment, college persistence, FAFSA completion). For continuous 

outcome variables, the study team will estimate multilevel linear models; for dichotomous outcomes 

(e.g., college enrollment, FAFSA completion), the study team will estimate multilevel logistic 

regression models.

Multilevel Logistic Regression

For dichotomous outcomes (e.g., enrollment, persistence, and FAFSA completion), a logistic 

regression will be used. This model is:

(1) [ = 1] =





(  0 +  1 (Tij) +  2 (Y∗ij) + 





1

1

J

j

 2+j Schj + 

M

m 1

 2+(J−1)+m X mij )

Where  is the covariate-adjusted log-odds of the outcome occurring (e.g., enrolled in college, 

completed FAFSA) versus not occurring (e.g., did not enroll in college, did not complete FAFSA) for 

students in school j; is a regression coefficient indicating the difference in log-odds of the 

outcome between the treatment and control group, controlling for all other variables in the model; and 

through  parameter estimate for the average effect (in log-odds) of each student-level 

covariate on the outcome , controlling for all other variables in the model.

To convert estimates produced by logistic regression to an impact measured in percentage point 

increases in probability of the binary outcome, we calculate the individual level probability of the 

outcome given treatment implied by individual level covariates and model estimates, we repeat this 

calculation if the individual were assigned to control, and find the mean difference between these two 

probabilities.

Hierarchical Linear Model

The following model will be used to estimate program impacts on continuous outcome variables. 

Given that our core outcome measures (i.e., confirmatory contrasts) are all dichotomous (i.e., college 

enrollment, FAFSA completion, and persistence into the second year of college), this HLM describes 

our methodology for assessing exploratory contrasts.

 

7

If we suspect that treatment effects differ by school, we will interact schools with the treatment indicator. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-9

Yij =  0 +  1 (Tij) +  2 (Y*

ij) + 





1

1

J

j

 2+j Schj

+ 

M

m 1

 2+(J-1)+m X mij +

ij 

where Yij is the value of the outcome (e.g., amount of Federal financial aid received) for the i

th student 

in the j

th GEAR UP school; Tij is 1 if student i is randomized to the treatment condition in schools j

and 0 if assigned to the control condition in school j; Y*

ij is the baseline score for the outcome of 

interest for student i in school j; Schj = 1 if student is in school j, =0 else; X mij are additional 

covariates representing demographic characteristics of student i at school j (e.g., race/ethnicity, 

English language learner status);  0 is the conditional mean follow-up score for the outcome of 

interest for control students; 1is the treatment effect i.e. the mean difference of the outcome 

between intervention and control students. The overall treatment estimate (  1) and will be a 

precision-weighted estimate, where the weights are inversely proportional to the variance of the 

treatment effect in each school. ij  is the residual error term assumed to be normally distributed with 

mean 0 and variance 

2

.

2. What types of college advising are typically received by GEAR UP students? 

Descriptive analyses will be used to provide information on the types of college advising GEAR UP 

students typically receive. It is important to understand the existing college advising to which the 

messaging will be added. Further, this collection and analysis has the added benefit of filling a gap in 

program information about the intensity of and emphasis on college advising. The study team will 

describe the types and intensity of college advising regularly received by GEAR UP students at the 

high school level as reported on the student survey. One example of the type of college advising that 

might be received by students is one-on-one counseling or advising sessions about applying to

college; the study team would describe the percent of students that attending such sessions. These 

results will provide another important source of contextual information for interpreting impact 

findings. 

3. Is there variation in the impact of the messages, and to what extent is the variation 

associated with student characteristics or features of the GEAR UP project or school? 

To examine variation in program impacts by student characteristics or features of the GEAR UP 

project or school, the study team will include an interaction of the treatment indicator with the 

subgroup variable of interest in the impact models described above.8 The study team will include 

interaction terms in the model for our core dichotomous outcomes of college enrollment, FAFSA 

completion, and college persistence (i.e., equation 1 above), allowing the treatment impact to vary by 

student characteristics, such as knowledge of the college application process, understanding of 

financial aid options and college costs, or amount of college advising received (e.g.one-on-one 

counseling or advising sessions about applying to college ) – or by program characteristics such as 

whether the GEAR UP program was funded under a state grant or a partnerships grant. If significant 

interaction effects are found, the sample will be broken into subgroups, the treatment effect will be 

estimated for each subgroup, and the means and variances of the subgroup will be compared

 

8 This approach requires the assumption that gender is not correlated with the marginal effect of the other 

model covariates. We will evaluate whether this assumption is tenable during preliminary data exploration.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-10

B.2.3 Degree of Accuracy Needed

This section presents the power analyses conducted to estimate the minimum detectable effect sizes 

(MDES) for the treatment/control differences on student outcomes.

The power analysis conducted utilizes a slightly modified9version of the MDE formula provided by 

Schochet (2008) for an RCT design that entails random assignment of student within schools and the 

corresponding 2-level impact model:

 = (, , ) ∗







(1 − )

(1 − )



Where

: estimated minimum detectable effect size for the treatment impact;

(, , ): a constant that is a function of the significance level (α), statistical power (β), 

and the number of degrees of freedom (df);

s: number of schools;

m: average number of students in each school;

p: proportion of students assigned to the treatment condition; 





: Student-level variance in the outcome;





: Proportion of student-level variance in the outcome explained by covariates;

σ: standard deviation of the outcome measure.

Using the equation above we make the following assumptions:

(1) Randomization of students within schools to treatment or control conditions

(2) Student level outcome of interest is measured on a continuous scale

(3) 80 percent power

(4) Two-tailed testing at the 5 percent level (alpha level=0.05)

(5) The demonstration includes s=80 schools

(6) The demonstration includes m=70 students per school

(7) 35 students are assigned to each group (p=0.5)

(8) The variance of the outcome across students (σ2

students) = σ=1 (because of standardization)

(9) The proportion of variation explained by student level covariates (R



) is 0.1 

Because our key outcomes (i.e., college enrollment, college, FAFSA completion, and college 

persistence) are all dichotomous variables, the power calculations also present the minimum 

 

9The Schochet (2008) formula was modified to include the 



 term.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-11

detectable effect size (MDES) expressed as a difference in percentage points between successes in the 

treatment and control groups using the following formula:

Conversion of MDES to MDE.PP

Since the standard deviation of a proportion (P) is P *(1 P)and MDES are expressed in 

standard deviation units, we used the following formula to convert MDES to MDE.PP (the MDE 

expressed as a difference in percentage points between successes in the treatment and control groups):

MDE.PP  MDES* BaseRate*(1 BaseRate) ; where the BaseRate is the rate (percent) of 

success in absence of treatment i.e. success in the control condition.

Exhibit B-3 shows the minimum detectable effect size (MDES) we would be powered to detect for 

Phase 1 with 80 schools and 70 students per school (35 in each group) would be approximately 2-4 

percentage points depending on the base rate. 

Exhibit B-5 shows the minimum detectable effect size (MDES) we would be powered to detect for 

Phase 2 with 50 students per school (25 in each group) would be approximately 2 to 4 percentage 

points10 depending on the base rate. 

 

10 This difference is smaller than the 12 percentage points difference in persistence rates into the fall

of sophomore year between community college freshmen who received a text intervention 

and those that did not receive the intervention, Castleman, B.L., & Page, L.C. (2014).
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-12

Exhibit B-5: Power Calculations for Dichotomous Outcomes in Phase 1 of the Study

Number of 

schools 

Number of 

students

Number of school 

X Number of 

students

R

2

student Minimum 

Detectable Effect 

Sizes (MDEs) in 

standard 

deviations

Base Rate 

(success in 

absence of 

treatment i.e. 

success in the

control 

condition)

MDE expressed 

as a difference in 

percentage points 

between 

successes in the 

treatment and 

control groups

80 70 5600 0.00 0.075 10 2.2

80 70 5600 0.05 0.073 10 2.2

80 70 5600 0.10 0.071 10 2.1

80 70 5600 0.20 0.067 10 2.0

80 70 5600 0.00 0.075 25 3.2

80 70 5600 0.05 0.073 25 3.2

80 70 5600 0.10 0.071 25 3.1

80 70 5600 0.20 0.067 25 2.9

80 70 5600 0.00 0.075 50 3.7

80 70 5600 0.05 0.073 50 3.6

80 70 5600 0.10 0.071 50 3.6

80 70 5600 0.20 0.067 50 3.4

80 70 5600 0.00 0.075 60 3.7

80 70 5600 0.05 0.073 60 3.6

80 70 5600 0.10 0.071 60 3.5

80 70 5600 0.20 0.067 60 3.3

80 70 5600 0.00 0.075 75 3.2

80 70 5600 0.05 0.073 75 3.2

80 70 5600 0.10 0.071 75 3.1

80 70 5600 0.20 0.067 75 2.9

80 70 5600 0.00 0.075 86 2.6

80 70 5600 0.05 0.073 86 2.5

80 70 5600 0.10 0.071 86 2.5

80 70 5600 0.20 0.067 86 2.3
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-13

Exhibit B-5: Power Calculations for Dichotomous Outcomes in Phase 2 of the Study

Number of 

schools 

Number of 

students

Number of school 

X Number of 

students

R

2

student Minimum 

Detectable Effect 

Sizes (MDEs) in 

standard 

deviations

Base Rate 

(success in 

absence of 

treatment i.e. 

success in the 

control 

condition)

MDE expressed 

as a difference in 

percentage points 

between 

successes in the 

treatment and 

control groups

80 50 4000 0.00 0.106 10 2.7

80 50 4000 0.05 0.103 10 2.6

80 50 4000 0.10 0.101 10 2.5

80 50 4000 0.20 0.095 10 2.4

80 50 4000 0.00 0.106 25 3.8

80 50 4000 0.05 0.103 25 3.7

80 50 4000 0.10 0.101 25 3.6

80 50 4000 0.20 0.095 25 3.4

80 50 4000 0.00 0.106 50 4.4

80 50 4000 0.05 0.103 50 4.3

80 50 4000 0.10 0.101 50 4.2

80 50 4000 0.20 0.095 50 4.0

80 50 4000 0.00 0.106 60 4.3

80 50 4000 0.05 0.103 60 4.2

80 50 4000 0.10 0.101 60 4.1

80 50 4000 0.20 0.095 60 3.9

80 50 4000 0.00 0.106 75 3.8

80 50 4000 0.05 0.103 75 3.7

80 50 4000 0.10 0.101 75 3.6

80 50 4000 0.20 0.095 75 3.4

80 50 4000 0.00 0.106 86 3.1

80 50 4000 0.05 0.103 86 3.0

80 50 4000 0.10 0.101 86 2.9

80 50 4000 0.20 0.095 86 2.8
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-14

B.2.4 Unusual Problems Requiring Specialized Sampling Procedures

Unusual problems that require specialized sampling procedures are not anticipated.

B.2.5 Use of Periodic Data Collection Cycles to Reduce Burden

The data collection plan reflects sensitivity to issues of efficiency and respondent burden. In this 

study, the study team planned data collection at the fewest intervals possible to reduce burden and 

ensure the quality of the data. The student rosters will only be collected once. Student surveys will be 

fielded once during this study. Administrative records, rather than surveys, will be used to gather data 

on outcomes. 

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

In order to obtain responses from the study sample, the study team has developed strategies to

facilitate communication with respondents during data collection activities and to maximize response 

rates. These strategies have proven successful in the study team’s extensive experience conducting 

large-scale evaluation studies (e.g., Reading First Impact Study, Evaluation of the U.S. Department of 

Education’s Student Mentoring Program, Evaluation of the Massachusetts Expanded Learning Time

Initiative; Enhanced Reading Opportunities Study; Career Academies Evaluation; The Teacher 

Incentive Fund Evaluation, and Study of Enhanced College Advising in Upward Bound). 

The study team will follow several procedures to maximize response rates and handle nonresponse.

Student Rosters. During the August 2016 meeting of the association of GEAR UP grantees 

organized by the National Council for Community and Education Partnerships, the study team will 

provide an evaluation session for participating project directors and school coordinators that will 

clearly explain the importance of the study and the roles and responsibilities of project directors and 

school coordinators, including the process for submitting student rosters. During the 2016-2017 

school year, each school will be assigned a study liaison to answer questions and explain the 

demonstration requirements. Next, the liaison will send an email to each site coordinator explaining 

the request that will be made of site coordinators (see Appendix B). School coordinators will be asked 

to submit rosters of 12th grade students through a password protected secure file transfer portal 

(SFTP). One week after sending out the email request, a member of the study team will reach out to 

each coordinator via a telephone call during to answer any questions about the roster submission. The 

submission of student rosters via a password protected SFTP will allow coordinators to submit rosters 

at a time most convenient for them and will assure coordinators that student confidentially is 

protected as permitted by law.

Baseline student surveys. The baseline student surveys will be administered in electronic or paper

format. To assist data collection, the site liaison will work with each GEAR UP school to determine 

whether in-school administration is feasible and to: facilitate the logistics for the baseline student 

survey data collection; track response rates per project and communicate with project directors, as 

needed; remind students about the survey; and distribute individualized thank you or reminder letters 

to students near the end of the survey field period. The study team will make the survey available in 

multiple modes (electronic and paper) and individual follow-up will be conducted to reach students 

who were absent during in-person administration. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-15

B.4 Test of Procedures and Methods to be Undertaken

We do not believe pretesting is necessary for the collection of student rosters, as the study team will 

be following a procedure that has been successful in studies such as the Evaluation of Citizen 

Schools: School-Level Expanded Learning Time and the Study of Enhanced College Advising in 

Upward Bound. 

During the 60-day comment period, the baseline survey will be pilot tested. The pilot tests will ensure 

that the survey administration procedures are effective, the instructions are understandable, the 

content is comprehensible, and the length is reasonable. High school seniors from GEAR UP will be 

recruited for the pilot testing; no more than nine students will be selected to participate in the pilot 

test.

B.5 Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical aspects of the study:

Name Title/Affiliation Telephone

Mr. Cristofer Price Principal Scientist, Abt Associates 301-634-1852

Dr. Alina Martinez Principal Associate, Abt Associates 617-349-2312

Dr. Rob Olsen Principal Scientist, Abt Associates 301-634-1716

Dr. Allan Porowski Principal Associate, Abt Associates 301-634-1765

Dr. Anne Wolf Associate, Abt Associates 301-634-1738

Ms. Amanda Parsad Senior Scientist, Abt Associates 301-634-1791

The following individuals will be responsible for the Data Collection and Analysis:

Name Title/Affiliation Telephone

Dr. Alina Martinez Principal Associate, Abt Associates 617-349-2312

Mr. Cristofer Price Principal Scientist, Abt Associates 301-634-1852

Dr. Rob Olsen Principal Scientist, Abt Associates 301-634-1716

Dr. Amanda Parsad Senior Scientist, Abt Associates 301-634-1791

Dr. Tamara Linkow Associate, Abt Associates 617-520-2978

Ms. Linda Kuhn President, Survey Research Management 303-247-0140
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-16

References

Aud, S., Hussar, W., Johnson, F., Kena, G., Roth, E., Manning, E., Wang, X., & Zhang, J. (2012). 

The condition of education 2012 (NCES 2012-045). U.S. Department of Education, National 

Center for Education Statistics. Washington, DC. Retrieved June 27, 2012 from 

http://nces.ed.gov/pubsearch.

Bailey, M. J., & Dynarski, S. M. (2011). Inequality in postsecondary education. In G. J. Duncan and 

R. J. Murnane (Eds.). Whither opportunity? Rising inequality and the uncertain life chances of 

disadvantaged children. New York, NY: Russell Sage Foundation.

Bowen, W. G., Chingos, M.M., & McPherson, M.S. 2009. Crossing the Finish Line: Completing

College at America’s Public Universities. Princeton, NJ: Princeton University Press.

Castleman, B. L. & Page, L. C. (2013). The not-so-lazy days of summer: Experimental interventions 

to increase college entry among low-income high school students. New Directions for Youth 

Development, 140, 77-97.

Castleman, B.L., & Page, L.C. (2014). Freshman year financial aid nudges: An experiment to 

increase FAFSA renewal and college persistence. EdPolicy Works Working Paper. Available 

http://curry.virginia.edu/uploads/resourceLibrary/29_Freshman_Year_Financial_Aid_Nudges

.pdf

Castleman, B.L., Page, L.C., & Schooley, K. (2014). The forgotten summer: Mitigating summer 

attrition among college-intending low-income high school graduates. The Journal of Policy

Analysis and Management. DOI: 10.1002/pam.21743

Darling, R. & Smith, M. (2008). First-generation college students: First-year challenges. In M. S. 

Hunter, B McCalla-Wriggins, & E. White (Eds) Academic Advising: New Insights for Teaching 

and Learning in the First Year. National Resource Center for the First-Year Experience and 

Students in Transition, Columbia SC.

Goyette, K. A. (2008). College for some to college for all: Social background, occupational 

expectations, and educational expectations over time. Social Science Research, 37(2), 461-484.

Haskins, R. (2008). Education and economic mobility. In J. B. Isaacs, I. V. Sawhill, and R. Haskins 

(Eds.). Getting ahead or losing ground: Economic mobility in America. Washington, DC: The 

Brookings Institution. Retrieved June 27, 2012 from 

http://www.brookings.edu/research/reports/2008/02/economic-mobility-sawhill.

Kao, G., & Tienda, M. (1998). Educational aspirations of minority youth. American Journal of 

Education, 106(3), 349-384.

Liu, A., Sharkness, J., & Pryor, J. (2008). Findings from the 2007 Administration of Your First 

College Year (YFCY): National Aggregates. Higher Education Research Institute, University of 

California, Los Angeles. Available http://www.heri.ucla.edu/PDFS/YFCY_2007_Report05-07-

08.pdf

National Center for Education Statistics (2012). Digest of Education Statistics, 2011. Washington, 

DC: US Department of Education.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-17

National Center for Education Statistics (2013). Common Core of Data (CCD), "Public 

Elementary/Secondary School Universe Survey," 2011-12. 

http://nces.ed.gov/programs/digest/d13/tables/dt13_216.80.asp

National Student Clearinghouse (2012). Impact of Directory Information Blocks on StudentTracker 

Results. Available http://nscresearchcenter.org/wpcontent/uploads/NSC_Directory_Block_Rates.pdf

National Student Clearinghouse (2014a). Report: High School Benchmarks 2014, National College 

Progression Rates for high schools participating in the National Student Clearinghouse Student

Tracker Service. Available http://nscresearchcenter.org/hsbenchmarks2014

National Student Clearinghouse (2014b). Snapshot Report: First-Year Persistence and Retention 

Rates by Starting Enrollment Intensity: 2009-2012. Available: 

http://nscresearchcenter.org/snapshotreport-persistenceretention14/

National Student Clearinghouse (2014c). Clearinghouse Facts. Available 

http://www.studentclearinghouse.org/about/clearinghouse_facts.php

Roderick, M., Nagaoka, J., Coca, V., & Moeller, E. (2009). From High School to the Future: Making 

Hard Work Pay Off. Chicago: Consortium on Chicago School Research.

Schochet, P. (2008). Statistical Power for Random Assignment Evaluations of Education Programs. 

Journal of Educational and Behavioral Statistics, 33(1), 62-87.

U.S. Department of Education. (2014). Gaining Early Awareness and Readiness for Undergraduate 

Programs (GEAR UP) Funding Status. Retrieved on January 13, 2015 from: 

http://www2.ed.gov/programs/gearup/funding.html