Part B: Supporting 

Statement for 

Paperwork Reduction 

Act Submission

Evaluation of the GEAR

UP College Savings 

Account Research 

Demonstration

Prepared for:

Marsha Silverberg

U.S. Department of Education

555 New Jersey Ave, NW

Room 502I

Washington, DC 20208-5500

Submitted by:

Abt Associates Inc.

55 Wheeler Street

Cambridge, MA 02138
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Table of Contents ▌pg. i

Part B: Supporting Statement for Paperwork Reduction Act Submission

Table of Contents

B. Collection of Information Employing Statistical Methods.................................................B-1

Introduction..............................................................................................................................B-1

B.1 Respondent Universe and Sampling Methods...............................................................B-4

B.1.1 Respondent Universe.......................................................................................B-4

B.1.2 Sampling Methods...........................................................................................B-5

B.1.3 Expected Response Rates................................................................................B-6

B.2 Statistical Methods for Sample Selection and Degree of Accuracy Needed.................B-6

B.2.1 Sample Selection .............................................................................................B-6

B.2.2 Estimation Procedures.....................................................................................B-6

B.2.3 Degree of Accuracy Needed..........................................................................B-10

B.2.4 Unusual Problems Requiring Specialized Sampling Procedures ..................B-12

B.2.5 Use of Periodic Data Collection Cycles to Reduce Burden...........................B-12

B.3 Methods to Maximize Response Rates and Deal with Nonresponse...........................B-12

B.4 Test of Procedures and Methods to be Undertaken.....................................................B-15

B.5 Individuals Consulted on Statistical Aspects of the Design ........................................B-15

References..............................................................................................................................B-16
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-1

B. Collection of Information Employing Statistical Methods

Introduction

The U.S. Department of Education (ED) will conduct a demonstration to test the effectiveness of 

supporting college savings accounts in the federal college access program, Gaining Early Awareness 

and Readiness for Undergraduate Programs (GEAR UP). This first Information Collection Request 

(ICR) for the study presents an overview of its design, including the plan to randomly assign high 

schools in the demonstration project into treatment and control conditions, and seeks clearance to 

collect student rosters in these high schools. A later ICR will request approval for the remaining data 

collection activities.

The study is being sponsored by ED’s Institute of Education Sciences and implemented by Abt 

Associates Inc. and its partners, the Center for Social Development, Survey Research Management, 

Professional and Scientific Associates, and Jay Pfeiffer (together, the “Abt study team”).

Overview of the GEAR UP Program and the College Savings Account Research Demonstration

Section 404H of the Higher Education Act of 1965, amended by the 2008 Higher Education 

Opportunities Act (HEOA) (20 USC 1070A-27) authorizes the GEAR UP program. The program 

provides over $300 million a year to state agencies and local partnerships to support disadvantaged

students in obtaining a high school diploma and to prepare them for success in postsecondary 

education. Grantees are expected to provide a comprehensive set of intervention services such as 

financial assistance, academic support, counseling, mentoring, and outreach to disadvantaged middle 

and high school students as well as information about college financial aid options and the benefits of 

postsecondary education to students and their families. State GEAR UP grantees are required to, and 

local partnership can, also provide scholarships to help participating students pay for college 

attendance. In 2012, GEAR UP state grantees served 403,818 students at a per-student cost of $347

and partnership grantees served 243,954 students at a per-student cost of $649 (U.S. Department of 

Education, 2012).

While GEAR UP is designed to provide comprehensive services to low-income students, increasing 

concern about the cost of college prompted ED to experiment with other strategies to reduce barriers 

to postsecondary education transitions. In 2012, ED announced its intention to use GEAR UP funds 

for a supplemental grant competition to establish and evaluate college savings accounts for students 

in GEAR UP high schools.1States that received GEAR UP grants in FY2011 or FY2012 and that will 

have cohorts of 9th graders in their participating high schools in either fall 2013 or fall 2014 are 

eligible for the College Savings Account Research Demonstration. (Below, high schools with eligible 

students in fall 2013 will be referred to as cohort 1 schools and high schools with eligible students in 

fall 2014 are cohort 2 schools.) As a condition of receiving the supplemental grant funds, both state 

and local officials must agree to participate in an evaluation of the project, including the random 

assignment of GEAR UP high schools and the data collection activities. In addition, states must set 

aside grant resources so that state and local coordinators involved in the demonstration can attend 

special professional development (PD) sessions, which will be held at the twice-a-year meetings of 

 

1

See https://www.federalregister.gov/articles/2012/06/01/2012-13232/proposed-priorities-gaining-earlyawareness-and-readiness-for-undergraduate-programs-college-savings
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-2

the association for GEAR UP grantees (the National Council for Community and Education 

Partnerships, or NCCEP).

Student participants in treatment schools will have a college savings account set up for them with an 

initial deposit (“seed”) of $200, and the possibility of matching contributions of up to $25 per month 

for up to four years. Students receiving the maximum match would finish high school with $2,600—

the $200 Federal seed, $1,200 in personal contributions, and $1,200 in Federal matching funds. In 

addition, treatment students and their families will be provided with financial counseling designed to 

help them understand the benefits of saving for college, encourage them to save, increase their 

understanding of relevant financial concepts and prepare them to make financial decisions.

The competition for the College Savings Account Research Demonstration grants will be underway in 

early 2013. The number of states selected to receive awards will depend on the number of schools and 

students served by each state GEAR UP program in the demonstration project. ED intends to select 

states, one after another, beginning with the highest-ranked proposal until there is a total across the 

selected states of about 100 high schools that have GEAR UP students enrolled in the 9th grade in 

either fall 2013 or fall 2104. We expect there to be approximately 20,000 9th grade students in these 

100 GEAR Up high schools.

Overview of the Evaluation

The study will use a multi-site cluster random assignment design to evaluate the effectiveness of the 

college savings and financial counseling intervention implemented as part of the demonstration. In 

spring 2013, after the demonstration grants are awarded, the study team will randomly assign one-half 

of the participating GEAR UP high schools in each state to the treatment group and one-half to the 

control group.2In all participating schools, students will continue to receive services supported by 

GEAR UP (e.g. tutoring, counseling, and college scholarships). In schools assigned to the treatment 

group, their 9

th grade cohort also will receive the college savings accounts and financial counseling. 

Students in both groups will be tracked over time to collect administrative and survey data on key 

outcomes, including college savings, financial literacy, academic outcomes in high school, and 

educational attainment expectations.3

The study will use the data to assess not only whether the intervention is effective, but how well it 

was implemented and whether its effectiveness depends on key components of the GEAR UP 

program or features of the college savings account intervention as it was designed and implemented. 

In particular, the evaluation is designed to answer four main research questions: 

1) To what extent do the college savings accounts and counseling—above and beyond the services 

GEAR UP may already provide—improve college savings and other student and family behaviors 

and attitudes associated with college going? 

2) What is the fidelity of implementation of the college savings account model in the participating 

GEAR UP state grantees? 

 

2 We plan to conduct blocked random assignment of schools, using state and one or more other blocking factors 

that are associated with the key student outcomes in this study. 

3 The current 5-year contract allows only for tracking students into the 11th grade. ED may execute an option to 

extend the contract to allow for collection and analysis of data on college enrollment and persistence.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-3

3) What types of services—including college savings, financial education, and financial literacy 

education—are typically provided to GEAR UP high school participants? 

4) Is there variation across states in the impact of the college savings account program, and to what 

extent is the variation associated with features or characteristics of the GEAR UP project or the 

college savings account program? 

To answer these questions we will conduct both impact and descriptive analyses. The first report, 

which will address each research question after one year of demonstration implementation, will be 

published in 2016. The final report, after implementation over three years, will be available in 2018.

Exhibit B-1 below presents the research questions along with the data sources for each question, the 

analytic approach and outcomes of interest. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-4

Exhibit B-1. Evaluation Questions, Data Sources, Analytic Approach, and Outcomes of Interest

Research Question Data Sources Analytic Approach

Outcomes of 

Interest

1) To what extent do the college 

savings accounts and 

counseling— above and 

beyond the services GEAR 

UP may already provide—

improve college savings and 

other student and family 

behaviors and attitudes 

associated with college going?

 Educational 

administrative data

 Aggregate high school 

level data

 Savings account data

 Parent survey data

 Student survey data

 Impact analysis 

 Administrative 

data sample size: 

20,000 students 

in 100 schools

 Survey sample: 

8,000 students 

and 8,000 parents

 College 

Savings

 Financial 

literacy

 Educational 

expectations

 Academic 

performance 

 Course-taking 

2) What is the fidelity of 

implementation of the college 

savings account model in the 

participating GEAR UP state 

grantees? 

 Project Director and 

Site Coordinator survey 

data

 Parent survey data

 Student survey data

 PD attendance sheets

 Descriptive 

analysis

 Survey sample: 

8,000 students, 

8,000 parents, 

110 project 

directors and site 

coordinators

 Participation 

in PD 

trainings

 Description of 

GEAR UP 

program 

components 

provided and 

received 

3) What types of services—

including college savings, 

financial education, and 

financial literacy education—

are typically provided to GEAR 

UP high school participants? 

 Project Director and 

Site Coordinator survey 

data

 Parent survey data

 Student survey data

 Descriptive 

analysis

 Survey sample: 

8,000 students, 

8,000 parents, 

110 project 

directors and site 

coordinators

 Description of 

GEAR UP 

program 

components

4) Is there variation across states 

in the impact of the college 

savings account program, and 

to what extent is the variation 

associated with features or 

characteristics of the GEAR 

UP project or the college 

savings account program? 

 Educational 

administrative data

 Aggregate high school 

level data

 Savings account data

 Parent survey data

 Student survey data

 Project Director and 

Site Coordinator survey 

data

 Impact analysis / 

moderator 

analysis

 Administrative 

data sample size: 

20,000 students 

in 100 schools

 Survey sample: 

8,000 students, 

8,000 parents, 

110 project 

directors and site 

coordinators

 College 

Savings

 Financial 

literacy

 Educational 

expectations

 Academic 

performance 

 Course-taking

B.1 Respondent Universe and Sampling Methods

B.1.1 Respondent Universe

The respondent universe includes all states selected through the College Savings Account Research 

Demonstration competition. Successful grantees will not only meet the eligibility criteria but provide 

a strong plan for how they will carry out the activities of the demonstration. Although there are 26 

states that received GEAR UP awards in FY 2011 or FY 2012 it is unclear at this time how many 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-5

states will meet the other criteria and choose to apply to or be awarded a grant through the 

demonstration. Our best estimate is 5-8 states will receive demonstration grants. 

Among states with demonstration grants, we expect there to be approximately 100 participating 

GEAR UP high schools. High schools will fall into one of two cohorts; cohort 1 includes high schools 

with eligible 9th grade students in fall 2013 and cohort 2 includes high schools with eligible 9th grade 

students in fall 2014.Given that demonstration grants will only be awarded to states implementing 

GEAR UP with a cohort model, all 9th graders in the school will be considered GEAR UP 

participants. For cohort 1 schools, participants will include all students in 9th grade in fall 2013 and 

for cohort 2 schools participants will include all students in 9th grade in fall 2014. Our estimate of the 

number of students in the evaluation sample is based on the average number of 9th graders in high 

schools in the U.S. (200); therefore we expect the evaluation sample to include approximately 20,000 

students in cohorts 1 and 2, representing all of the GEAR UP high school participants in the 

demonstration states.

Exhibit B-2 indicates the estimated population sizes for the evaluation. 

Exhibit B-2. Population Size Estimates

Estimated number of demonstration state grantees 5 to 8 grantees

Estimated number of GEAR UP high schools per grantee 12 to 20 schools

Estimated number of 9th graders in GEAR UP cohort per grantee 2,400 to 4,000 students

TOTALS

 High schools

 9

th grade GEAR UP participants

100

20,000

B.1.2 Sampling Methods

There will be no sampling of schools for the evaluation. The entire universe of GEAR UP schools 

within the demonstration state grants will participate in the evaluation study. 

All 9th grade students in demonstration schools, in cohort 1 and cohort 2, will participate in the 

evaluation. Student rosters, the only data for which forms clearance is currently being sought, will be 

collected to identify all participating students. Later, administrative records will be collected for all 

9

th grade students in cohorts 1 and 2 to maximize the statistical power of the analysis without 

substantially affecting study costs (e.g., since, for example, electronic matching ensures that 

extracting data for all sample members from a given state’s database is no more burdensome than 

extracting data from a smaller sample from the same database).

Parent and student surveys will be administered to a randomly selected sample of 40% of students to 

limit study burden and costs. This will result in the random selection of approximately 8,000 students 

and their parents from the population of 20,000 students. The selected students and one of their 

parents will be asked to complete surveys twice during the study period. Section B.2 describes the 

random selection procedure for the survey sample in more detail.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-6

B.1.3 Expected Response Rates

We expect to obtain a 100% response on the registration lists from schools. We expect an 80% 

response rate for the student and parent surveys. Although we will randomly select approximately 

8,000 students and their parents, we anticipate obtaining completed surveys from 6,800 students and 

their parents at each round of data collection – spring of 9th grade and spring of 11th grade.

B.2 Statistical Methods for Sample Selection and Degree of Accuracy

Needed

B.2.1 Sample Selection

Random assignment will occur at the school level. Although there will be two cohorts of schools –

one cohort of high schools with eligible students in 9th grade in fall 2013 and a second cohort of high 

schools with eligible students in 9th grade in fall 2014, both cohorts of schools will be awarded the 

demonstration grant at the same time, in spring 2013. Therefore the Abt study team will randomly 

assign both cohorts of schools in spring 2013. 

Within each state, the Abt study team will randomly assign half of the participating high schools to 

the treatment group and half to the control group. Stratification by state and cohort will ensure that 

there are an equal number of treatment and control schools within each grantee and within each 

cohort. In addition, for districts with a large number of participating high schools, the Abt study team 

will stratify by districts to avoid situations where the share of treatment schools is much larger in 

some districts than others. We also plan to block high schools for random assignment using data from 

the National Center for Education Statistics’ Common Core Data (CCD), such as school size and 

percentage of students receiving free- or reduced-price lunches, as well as the number of students in 

each high school who have completed their Free Application for Federal Student Aid (FAFSA) form 

from data made available by the Federal Student Aid (FSA) office. These variables are potentially 

correlated with the study’s key outcomes and so blocking may provide greater precision in calculating 

the impact estimates.

For the study, the student population will include all 20,000 9th grade GEAR UP participants enrolled 

in treatment and control schools. Administrative data will be collected from the full population of 

students.

To limit burden and study costs, for parent and student surveys, the Abt study team will randomly

select a 40-percent matched sample of students and their parents. Selecting matched samples will 

ensure that responses by students and their parents can be linked. The selection of the survey sample 

will occur twice – once for cohort 1 and a year later for cohort 2. For each cohort, the sample will be

stratified by school—that is, the Abt study team will randomly select a 40 percent sample of students

and their parents within each of the 100 schools—to account for variation in outcomes across high 

schools in the sample. However, the Abt study team will not oversample certain types of students or 

schools because baseline data on students and parents will not be available when the selection of the 

survey samples occurs, and none of the analysis would benefit from oversampling.

B.2.2 Estimation Procedures

This section presents the study’s estimation approach for addressing the research questions:
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-7

1. To what extent do the college savings accounts and counseling—above and beyond the 

services GEAR UP may already provide—improve college savings and other student and 

family behaviors and attitudes associated with college going? 

To examine overall program impacts, the Abt study team will use multilevel modeling to account for 

the nested structure of the data. The Abt study team will estimate a two-level model, with students 

(level-1) nested within schools (level-2). At the student level, the Abt study team will control for 

student achievement prior to 9th grade and student demographic characteristics, including 

race/ethnicity, eligibility for free or reduced-price lunch, English language learner status, special 

education status. At the school level, the Abt study team will control for baseline differences between 

states, by including a set of state indicator variables, and for baseline school characteristics, including 

the percentage of seniors applying for federal financial aid, average school achievement, and 

percentage of students eligible for free or reduced-price lunch. Reflecting the randomization of 

schools, the Abt study team will include a treatment indicator at the school level to provide an 

estimate of the average impact of the college savings demonstration on students across all schools at a 

given time (e.g., at the end of 9th grade, end of 11th grade, or after the expected date of high school 

graduation). 

The Abt study team will conduct a test of the null hypothesis of no treatment impact for each outcome 

variable. The Abt study team will interpret a positive and statistically significant impact at the 5-

percent level (using a two-tailed test) as compelling scientific evidence that the intervention improves 

the targeted outcome (e.g., college savings, number of credits earned in college preparatory courses; 

financial literacy, completion of specific courses by a specified grade level). For continuous outcome 

variables (e.g., number of credits, amount contributed to savings account), the Abt study team will 

estimate multilevel linear models; for dichotomous outcomes (e.g., course completion, contributed to 

college savings account), the Abt study team will estimate multilevel logistic regression models. If 

the Abt study team examines impacts on ordinal outcomes (e.g., plans to attend 4-year college, plans 

to attend 2-year college, no plans to attend college), the Abt study team will estimate multilevel 

ordered logistic regression models.

Hierarchical Linear Model

The following hierarchical linear model will be used to estimate program impacts on continuous 

outcome variables.

The Level-1 equation (student level or family level, depending on the outcome) is:

(1)  =  + h − h  + ∑ 





( −   ) + 

Where  is the value of the outcome (e.g., amount contributed to savings account) for the i

th student 

in the j

th school; h is a measure of achievement for ith student in the jth school 

(centered at the grand mean, h );  is a vector of k variables 

measuring the demographic characteristics of the i

th student in the jth school (each centered at the 

grand mean,    ); is the covariate-adjusted mean value of the outcome for students 

in school j; through  are regression coefficients indicating the effects of each student-level 

covariate on the outcome variable ; and  is the student-level residual for the i

th student in the jth

school (the assumed distribution of the residuals is normal, with mean 0 and variance 



.

The Level-2, or school-level, equation is:
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-8

(2)  =

 +  + h + ∑ (hh12_13



 − h h 12_

13)+ 

Where  is the covariate-adjusted mean value of the outcome measure across control schools;  is 

the mean difference in the covariate-adjusted outcome between treatment and control schools (i.e., the 

treatment impact); is the treatment status dummy variable with a value of 1 for a GEAR 

UP school assigned to the treatment condition and a value of 0 for a GEAR UP school assigned to the 

GEAR UP control condition; his indicates whether schools implement the demonstration with 

students in 9th grade in 2013 or with students in 9th grade in 2014; hh12_13is a vector 

of k variables measuring the characteristics of the j

th school in school year 2012-2013, prior to random 

assignment (each centered at the grand mean, h h 12_

13);  is a vector of k regression 

coefficients indicating the effect of each baseline school covariate on the covariate-adjusted mean 

value of the outcome measure; and  is the error term for the j

th school (the distribution is assumed 

to be normal with mean 0 and variance 



).

The parameter  indicates the impact of the demonstration on the outcome, . A t-test will be 

conducted to test the null hypothesis of no treatment impact using a .05 level criterion to reject the 

null hypothesis. A positive and statistically significant estimate of  will indicate that there is 

compelling scientific evidence (at the 5 percent level) that the demonstration improves the targeted 

outcome. The parameter  indicates the magnitude of the impact -- such that the college savings 

demonstration in GEAR UP high schools is estimated to have, on average, a  point effect on the 

specified outcome.

A standardized effect size will be calculated by dividing the estimated impact () by the standard 

deviation of the outcome variable, , in the control group (). The standardized effect size is 





 . The control group standard deviation will be used, as recommended by Burghardt, Deke, 

Kisker, Puma, and Schochet (2009), rather than the pooled standard deviation, because the 

intervention might affect the standard deviation in the treatment group. 

Multilevel Logistic Regression

For dichotomous outcomes, a logistic regression equation will replace the level-1 linear equation 

specified for continuous outcomes as follows:

For a multilevel logistic regression model with a dichotomous outcome, the level-1 equation is:

(3) [ = 1] =





(∑ 





)

Where is the covariate-adjusted log-odds of the outcome occurring (e.g., course completion; plans 

to attend college) versus not occurring (e.g., course not completed; does not plan to attend college) for 

students in school j; and through  are regression coefficients indicating the log-odds of each 

student-level covariate on the outcome occurring. 

The level-2, or school-level, equation remains the same as equation (2); however,  is log-odds of 

the outcome occurring in the control group (e.g., course completion by control students; plans to 

attend college by control students), rather than the mean value of the outcome in the control group, 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-9

and  is the log-odds ratio quantifying the treatment impact (i.e., difference between treatment and 

control schools in the log-odds of the outcome occurring versus not occurring). Impacts estimated as 

log-odds ratios are measured in standardized effect size units

To convert estimates produced by logistic regression to an impact measured in percentage point 

increases in probability of the binary outcome, we calculate the individual level probability of the 

outcome given treatment implied by individual and school level covariates and model estimates, we 

repeat this calculation if the individual were assigned to control, and find the mean difference 

between these two probabilities.

2. What is the fidelity of implementation of the college savings account model in the 

participating GEAR UP state grantees?

Descriptive analyses will be used to provide information on the fidelity of implementation. To 

describe fidelity of implementation, the Abt study team will define what constitutes “implementation 

as intended” and construct a measure of the extent to which sites implement program components 

with fidelity (e.g., establish college savings accounts, provide seed funds, provide matching funds, 

attend financial counseling professional development, provide financial counseling for parents, 

provide financial counseling for students). The Abt study team will report descriptive data on the 

percentage of sites that implement each key program component with fidelity, providing information 

about which program components are widely implemented and which are implemented less 

consistently. Information on the overall level of and variation in implementation fidelity will provide 

important contextual information for interpreting impact findings.

3. What types of services—including college savings, financial education, and financial literacy 

education—are typically provided to GEAR UP participants? 

Descriptive analyses will be used to provide information on the types of services typically provided to 

GEAR UP participants. The Abt study team will describe the types and intensity of services regularly 

offered to GEAR UP participants at the high school level as reported on the project directory survey 

and on the parent and student surveys. The Abt study team will describe the services that constitute 

the counterfactual condition in order to understand the extent to which treatment services are distinct 

from the usual services available to GEAR UP participants, illuminating the treatment-control 

contrast. These results will provide another important source of contextual information for 

interpreting impact findings. 

4. Is there variation across states in the impact of the college savings account program, and to 

what extent is the variation associated with features or characteristics of the GEAR UP 

project or the college savings account program? 

To examine variation in program impacts, the Abt study team will use a two-level model, controlling 

for baseline covariates, as described above. The Abt study team will include interaction terms in the 

school level model (i.e., equation 2 above), allowing the treatment impact to vary by state or by 

treatment program characteristics, such as type of savings bank or mode of counseling services 

offered. For example, to test for state variations in the program impact, the Abt study team will 

include a set of interactions between the treatment indicator and each of the state indicators in 

Equation 2, as follows: 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-10

(4)  =  +  +∑ (hh12_13



 − hh12 )+

∑  

 + ∑ 



  ∗  +

where  is the covariate-adjusted mean value of the outcome measure (or log-odds of the outcome 

occurring) among control schools in the reference category state; STATE is a vector of s-1 state 

indictor variables—one for all but one of the s states in the sample;  is the difference between 

STATE s and the reference-category state in the covariate-adjusted mean value (or log-odds) of the 

outcome among control schools;  is the mean difference in the covariate-adjusted outcome 

between treatment and control schools (i.e., the treatment impact); STATE *Trtjis a vector of state by 

treatment condition interaction terms—one for each s-1 state indicator variable; and is the 

difference in the treatment impact between STATE s and the reference-category state.

Using a .05 level criterion, the Abt study team will conduct a joint test of the null hypothesis that the s

parameters for all of the interaction terms are zero, i.e., that there is no statistically significant 

difference in the program impact for any given state. The null hypothesis is expressed as follows:

:  ∗  =  ∗  = ⋯ =  ∗  = 0

The Abt study team will interpret a statistically significant difference as evidence that the program 

impact varies by state (or by features of the treatment program). If the Abt study team finds a 

statistically significant difference in program impacts, the Abt study team will then test whether the 

program impact within each state or subgroup is statistically significant. Standardized effect sizes will 

be calculated as described above.

B.2.3 Degree of Accuracy Needed

This section presents the power analyses conducted to estimate the minimum detectable effect sizes 

(MDES) for the treatment/control differences on student and family outcomes. 

The MDES estimates for binary outcomes are based on the following assumptions:

(1) the demonstration includes 100 schools

(2) 50 schools are assigned to each group

(3) the full sample includes 200 students per school

(4) the student and parent survey samples includes 40 percent of the full sample

(5) the surveys obtain 80 percent response rates 

(6) the intra-class correlation equals .05

(7) the school-level R-square equals .32 

(8) the outcome variable equals 1 for 32 percent of control group members

(9) the proportion of disadvantaged students in the sample is 50 percent at baseline.

In addition, the Abt study team made standard statistical assumptions (i.e., two-tailed testing at the 5 

percent level and 80 percent power). Because many of our key outcomes are binary variables –

whether participants are saving for college; completion of college preparatory courses (e.g., 

completion of Algebra I in 9th grade; completion of calculus; completion of an AP or IB course); and 

expectations about college going– the power calculations assume binary outcomes. The Abt study 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-11

team used the formulas in Schochet (2011) for binary outcomes to calculate MDES. These power 

calculations for binary outcomes provide conservative MDES estimates. Our analyses of continuous 

outcomes (e.g., amount of college savings; financial literacy scale score), will have greater statistical 

power to detect impacts (i.e., lower MDES) with the same sample size (see Appendix B for the power 

calculations for continuous variables).

The Schochet (2011) paper derives MDE equations and variance estimates that reflect the process of 

transforming logistic regression estimates into percentage point impact estimates. Because the 

percentage point impact estimates are the expected difference in probability between the treatment 

and the control groups, these impact estimates reflect the joint distribution of the outcome and 

covariates in absence of treatment, in addition to the estimated log-odds treatment effect produced by 

the logistic regression. To make this relationship tractable, Schochet (2011) assumes a single binary 

covariate.

Row 1 in Exhibit B-3 demonstrates that the MDES will be 5 percentage points for outcomes based on 

administrative data (e.g., whether students take calculus or attend a four-year university) and 6 

percentage points for outcomes based on survey data (e.g., whether families save for college). If ICCs 

prove to be larger than our assumed ICC of .05, the MDES may be larger. For example, if the ICC is 

.10, which is consistent with some of the estimates reported in Schochet (2011), the MDE would be 

approximately 7 percentage points for outcomes based on both administrative data and 8 percentage 

points for outcomes based on survey data (see row 2).

Exhibit B-3. MDES Estimates for Binary Outcomes from Administrative and Survey Data

Intraclass Correlation

Administrative Data Outcomes

(200 students/school)

Survey Data Outcomes

(68 students/school)1

.05 5.1 percentage points 5.7 percentage points

.10 7.0 percentage points 7.5 percentage points

1

The survey sample includes 40% of the students in the full sample (i.e., 80 students/school); with an 80% 

response rate, the Abt study team assumes 64 students/school in the analysis sample for survey data outcomes.

Power analyses were based on the following MDES formula, in which Schochet refers to minimum 

detectible effect sizes for binary outcomes as minimum detectible impacts (MDI) in percentage 

points:

 =  



(, { = 1}, { = 1}, R, , , )

− 



1 −



2



where: 

: significance level for a two-tailed test,

: statistical power;

(. ) the cumulative density function of the normal distribution;





(. ) the inverse of the cumulative density function of the normal distribution;

: minimum detectible impact in percentage point terms;
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-12

(. ): the variance of the estimated impact, which is a function of the MDI and the following 

terms;

{ = 1}: the probability that the outcome occurs for control group members (e.g. the student 

completes the course);

{ = 1}: the probability that the covariate is one (e.g. the probability that a student is 

disadvantaged at baseline);

R



: proportion of the student-level variance explained by the covariate;

: total number of schools;

: average number of students per classroom with a valid outcome measure after account for 

attrition and non-response; and

: the intra-class correlation for the outcome.

Unlike the standard power analysis equation, the equation above cannot be solved for the MDI 

analytically, as the MDI enters the equation for variance: iterative methods must be used. To find the 

MDI for a given sample size and power, the Abt study team fixed the number of schools and iterated 

over MDI until we obtained the desired power.

B.2.4 Unusual Problems Requiring Specialized Sampling Procedures

Unusual problems that require specialized sampling procedures are not anticipated.

B.2.5 Use of Periodic Data Collection Cycles to Reduce Burden

The data collection plan reflects sensitivity to issues of efficiency and respondent burden. Whenever 

possible the Abt study team will collect data at the fewest intervals allowable to maintaining data 

accuracy. The student rosters will only be collected once. Student, parent, and project director surveys 

will be fielded twice during this study, to provide timely results and minimize burden; fewer 

collection periods would prevent us from assessing how implementation and key behavioral outcomes 

for students and parents (financial literacy, understanding of college costs) change over time. During 

this study annual collection of state or district administrative educational records each fall is 

necessary to track students across schools and districts. To maintain data accuracy, college savings 

account data will be collected annually during Years 3, 4, and 5 of the study.

B.3 Methods to Maximize Response Rates and Deal with Nonresponse

This section describes the strategies and methods that will be used to maximize response rates and 

deal with non-response.

In order to obtain responses from the study sample, the study team has developed strategies to

facilitate communication with respondents during data collection activities and to maximize response 

rates. These strategies have proven successful in the study team’s extensive experience conducting 

large-scale evaluation studies (e.g., The Reading First Impact Study, Evaluation of the U.S. 

Department of Education’s Student Mentoring Program, Evaluation of the Massachusetts Expanded 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-13

Learning Time Initiative; The Enhanced Reading Opportunities Study; The Career Academies 

Evaluation; The Teacher Incentive Fund Evaluation). 

There are two phases to this study. This first ICR requests approval for Phase I – Random Assignment 

and Collection of Student Rosters will be used to identify GEAR UP student participants and to 

randomly assign GEAR UP high schools to treatment and control groups. Specifically, the student 

rosters will define the evaluation sample for all analyses and serve as a sampling frame for the student 

and parent surveys. An addendum ICR will request clearance for Phase II –Collection of Survey and 

Administrative Data.

The Abt study team will follow several procedures to maximize response rates and handle 

nonresponse during the Phase I – Random Assignment and Collection of Student Rosters activities 

covered under this ICR. During the July 2013 meeting of the association of GEAR UP grantees (the 

National Council for Community and Education Partnerships, or NCCEP), the Abt Study Team will 

provide an evaluation session for participating project directors and site coordinators. The session will 

clearly explain the importance of the evaluation and the roles and responsibilities of project directors 

and site coordinators, including the process for submitting student rosters in the fall. Prior to the start 

of the school year, the Abt team will send an introductory email to each site coordinator explaining 

the evaluation and the requests that will be made of site coordinators (see Appendix A). Next, the Abt 

study team will reach out to each site coordinator via a telephone call during the first 10 days of the 

2013-2014 academic year to request student rosters of 9th grade GEAR UP participants in both 

treatment and control schools and to answer any questions that the site coordinators may have. In the 

event that there are two cohorts of eligible schools, the student roster collection will occur in both fall 

2013 and 2014; 9th grade student rosters will be collected from cohort 1 schools in fall 2013 and 

rosters of 9

th graders in cohort 2 schools will be collected in the fall of 2014. The submission of 

student rosters via a password protected secure file transfer portal (SFTP) will allow site coordinators 

to submit rosters at a time most convenient for them and will assure coordinators that student 

confidentially is protected. 

During Phase II – Collection of Survey and Administrative Data of this study, the Abt study team will 

face attrition from the study and missing data for some parts of the sample. Disadvantaged families 

and, therefore, high poverty schools tend to have high rates of student mobility and dropping out . In 

addition, cooperation with surveys has dropped in the last twenty years, and the Abt study team 

expects that obtaining willing cooperation will be challenging for the student and parent surveys.4

These respondents may not want to take the time necessary to complete the study surveys. 

Alternatively, they may be willing to participate, but only if the study team makes it easy for them to 

do so, or offers them something in return.

 

4 We believe we will be able to achieve high rates of response from project directors and site coordinators 

because they are required to participate in data collection for a national evaluation under ED’s Education 

Department General Administrative Regulations (EDGAR) and in the demonstration’s data collection as a 

condition of participating in the grant.
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-14

To track the sample of students and parents over time, the Abt study team will use four strategies. 

1. Site coordinators involvement. The study team will ask site coordinators to track the students 

from their school as they move, change schools, or drop out of school, and to provide updated 

contact information for the study sample. It is likely that site coordinators in treatment schools 

will need to track students anyway, given their role in implementing the intervention.5

Furthermore, for site coordinators in control schools, tracking sample members can reasonably be 

considered part of their designated responsibility to support the evaluation’s data collection 

efforts. Before each round of surveys, the Abt study team will ask site coordinators for updated 

contact information for students who move residences or schools. This will help to ensure high 

response rates for telephone surveys to parents and to students and it will also facilitate obtaining 

administrative educational record each fall.

2. Updates from administrative records. State or district administrative educational records will 

be gathered each fall. These records should contain information on where students were enrolled 

in school; in some states, it may also include directory information on addresses and telephone 

numbers. Even information on the school a student was attending may be helpful. It may help to 

explain why the Abt study team was unable to obtain completed surveys in the spring (e.g., if 

state data suggests that a student transfers to a public school in another part of the state, it 

suggests the family moved); it may also help the study team in locating where the family now 

lives, which will help us obtain a completed survey during the next administration.

3. Alternate contact information. The first round of surveys will request more detailed contact 

information than the Abt study team would expect to obtain from student rosters. In particular, 

the Abt study team will ask parents for cell phone numbers, email addresses, and/or the telephone 

number of a family member who could help the study team locate the sample member. This more 

detailed information will help to locate sample members for the second round of survey and 

administrative data collection. 

4. State grantee involvement. Sate grantees will be asked to notify the study team when a site 

coordinator leaves or is replaced. This will help us to locate site coordinators for the project 

director survey; it will also help to stay in touch with the current site coordinators, which will 

allow the Abt study team to enlist their help in tracking the student and parent samples.

To obtain willing cooperation from students, parents, project directors, and state and district officials 

to the evaluation’s data collection efforts, the Abt study team will use three strategies. 

First, the Abt study team will use multiple modes to obtain completed surveys and administrative 

data. For example, the study team will:

 Offer students and parents the option of completing surveys by web survey or telephone; 

 Offer site coordinators and project directors the option of completing surveys by web survey 

or in paper and;

 

5 Treatment group students and families are eligible to continue making contributions to the college saving 

accounts and to receiving federal matching funds even if the student leaves a GEAR UP high school to 

attend a non-GEAR UP high school. 
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-15

Second, the Abt study team is considering the use of incentive payments for parents and students 

completing the surveys. The Addendum to this ICR, which will cover Phase II of the study, will 

provide further details on the potential use of incentive payments. 

Third, the Abt study team will leverage contact with sample members to market the evaluation and 

prepare them for upcoming surveys. For example, the Abt study will:

 Introduce the project director survey to project staff at the biannual grantee conferences (the 

conferences at which professional development training will be provided);

 Use regular contact with site coordinators to obtain updated contact information for sample 

members and to remind them to complete the upcoming project director survey; and

 Ask GEAR UP project directors to include information about upcoming surveys in materials 

that they mail to students and parents regarding their college savings account or upcoming 

financial counseling sessions.

B.4 Test of Procedures and Methods to be Undertaken

We do not believe pretesting is necessary for the collection of student rosters. The Abt Study team 

has collected rosters for other studies such as the Evaluation of Citizen Schools: School-Level 

Expanded Learning Time. The burden estimates were developed based on these previous experiences. 

B.5 Individuals Consulted on Statistical Aspects of the Design

The following individuals were consulted on the statistical aspects of the study:

Name Title/Affiliation Telephone

Mr. Cristofer Price Principal Scientist, Abt Associates 301-634-1852

Dr. Alina Martinez Principal Scientist, Abt Associates 617-349-2312

Dr. Rob Olsen Principal Scientist, Abt Associates 301-634-1716

Dr. Anne Wolf Associate, Abt Associates 301-634-1738

Dr. Amanda Parsad Senior Associate, Abt Associates 301-634-1791

Dr. Michal Grinstein-Weiss Associate Director at Center for Social Development 

at Washington University in St. Louis

314-935-8201

The following individuals will be responsible for the Data Collection and Analysis:

Name Title/Affiliation Telephone

Dr. Alina Martinez Principal Scientist, Abt Associates 617-349-2312

Dr. Rob Olsen Principal Scientist, Abt Associates 301-634-1716

Dr. Amanda Parsad Senior Associate, Abt Associates 301-634-1791

Dr. Tamara Linkow Associate, Abt Associates 617-520-2978

Ms. Linda Kuhn President, Survey Research Management 303-247-0140
Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs

Abt Associates Inc. Part B: Collection of Information Employing Statistical Methods ▌pg. B-16

References

Burghardt, J., Deke, J., Kisker, E., Puma, M., & Schochet, P. (2009). Regional educational

laboratory rigorous applied research studies: Frequently asked analysis questions. Institute of 

Education U.S. Department of Education. Princeton, NJ: Mathematics Policy Research.

Schochet, P.Z. (2011). Statistical Power for Binary Outcomes for Clustered RCTs of Education 

Interventions. Presented at the Society for Research in Educational Effectiveness Spring 

Conference, March 3-6, 2011.

U.S. Department of Education. (2012). Gaining Early Awareness and Readiness for Undergraduate 

Programs (GEAR UP) Funding Status. Retrieved on January 8, 2013 from: 

http://www2.ed.gov/programs/gearup/funding.html