## Part B: Supporting **Statement for Paperw**  **ork Reduction Ac**  **t Submission Ev** **aluation of the GE AR**  **UP College Sav ings**  **Ac count Research**  **Demonstration**

Prepared for: Marsha Silverberg **U.S. Department of Education** 555 New Jersey Ave, NW

Room 502I Wa shington, DC 20208 - 5500

Submitted by: **Ab t A**  **ssociates Inc.** 55 W heeler Street Cambridge, MA 02138**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc.**  **Table of Contents**  **▌ pg.**  **i** 

### Part B: Supporting Statement for Paperwork Reduction A**ct Submission**  **Table of Contents B.**  **Collection of Information Employing Statistical Methods ................................**  **................. B**  **- 1**  Introduction ................................ ................................ ................................ .............................. B - 1 B.1 Respondent Universe and Sampling Methods ................................ ............................... B - 4 B.1.1 Respondent Universe ................................ ................................ ....................... B - 4 B.1.2 Sampling Methods ................................ ................................ ........................... B - 5 B.1. 3 Expected Response Rates ................................ ................................ ................ B - 6 B.2 Statistical Methods for Sample Selection and Degree of Accuracy Needed ................. B - 6 B.2.1 Sample Selection ................................ ................................ ............................. B - 6 B.2.2 Estimation Procedures ................................ ................................ ..................... B - 6 B.2.3 Degree of Accuracy Needed ................................ ................................ .......... B - 10 B.2.4 Unusual Problems Requiring Specialized Sampling Procedures .................. B - 12 B.2.5 Use of Periodic Data Collection Cycles to Reduce Burden ........................... B - 12 B.3 Methods to Maximize Response Rates and Deal with Nonresponse ........................... B - 12 B.4 Test of Procedures and Methods to be Undertaken ................................ ..................... B - 15 B.5 Individuals Consulted on Statistical Aspects of the Design ................................ ........ B - 15 References ................................ ................................ ................................ .............................. B - 16**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **1** 

## B. **Collection of Information Employ ing Statistical Methods**  **Introduction** The U.S. Department of Education (ED) will conduct a demonstration to test the effectiveness of supporting college savings accounts in the federal college access program, Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP ). This first Information Collection Request (ICR) for the study presents an overview of its design, including the plan to randomly assign high schools in the demonstration project into treatment and control conditions, and seeks clearance to collect student rosters in these high schools. A later ICR will request approval for the remaining data collection activities. The study is being sponsored by ED’s Institute of Education Sciences and implemented by Abt Associates Inc. and its partners, the Center for Social Deve lopment, Survey Research Management, Professional and Scientific Associates, and Jay Pfeiffer (together, the “Abt study team”). **Overview of the GEAR UP Program and the College Savings Account Research**  **Dem onstration**  Section 404 H of the Higher Education Act of 1965, amended by the 2008 Higher Education Opportunities Act (HEOA) (20 USC 1070A - 27 ) authorizes the GEAR UP program . The program p rovides over $300 million a year to state agencies and local partnerships to support disadvantaged students in obtaining a high school diploma and to prepare them for success in postsecondary education. Grantees are expected to provide a comprehensive set of intervention services such as financial assistance, academic support, counseling, mentoring, and outreach to disadvantaged middle and high school students as well as information about college financial aid options and the benefits of postsecondary education to students and their families . State GEAR UP grantees are required to, and local partnership can, also provide schol arships to help participating students pay for college attendance. In 201 2, GEAR UP state grantees served 4 03,818 students at a per - student cost of $ 347 and partnership grantees served 2 43 , 954 students at a per - student cost of $649 (U.S. Department of Educ ation, 2012). While GEAR UP is designed to provide comprehensive services to low - income students, increasing concern about the cost of college prompted ED to experiment with other strategies to reduce barriers to postsecondary education transitions. In 201 2, ED announced its intention to use GEAR UP funds for a supplemental grant competition to establish and evaluate college savings accounts for students in GEAR UP high schools. 1 States that received GEAR UP grants in FY2011 or FY2012 and that will have coh orts of 9 th graders in their participating high schools in either fall 2013 or fall 2014 are eligible for the College Savings Account Research Demonstration. (Below, high schools with eligible students in fall 2013 will be referred to as cohort 1 schools a nd high schools with eligible students in fall 2014 are cohort 2 schools.) As a condition of receiving the supplemental grant funds, both state and local officials must agree to participate in an evaluation of the project, including the random assignment of GEAR UP high schools and the data collection activities. In addition, states must set aside grant resources so that state and local coordinators involved in the demonstration can attend special professional development (PD) sessions, which will be held at the twice - a - year meetings of

1 See https://www.federalregister.gov/articles/2012/06/01/2012 - 13232/proposed -priorities - gaining - early awareness - and - readiness - for - undergraduate -programs - college - savings**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -** **2** the association for GEAR UP grantees (the National Council for Community and Education Partnerships, or NCCEP). Student participants in treatment schools will have a college savings account set up for them with an initial depos it (“seed”) of $200, and the possibility of matching contributions of up to $25 per month for up to four years. Students receiving the maximum match would finish high school with $2,600

— the $200 Federal seed, $1,200 in personal contributions, and $1,200 in Federal matching funds. In addition, treatment students and their families will be provided with financial counseling designed to help them understand the benefits of saving for college, encourage them to save, increase their understanding of relevant fin ancial concepts and prepare them to make financial decisions. The competition for the College Savings Account Research Demonstration grants will be underway in early 2013. The number of s tates selected to receive awards will depend on the number of schools and students served by each s tate GEAR UP program in the demonstration project. ED intends to select s tates, one after another, beginning with the highest -ranked proposal until there is a total across the selected s tates of about 100 high schools that have GEAR UP students enrolled in the 9th grade in either fall 2013 or fall 2104 . We expect there to be approximately 20,000 9 th grade students in these 100 GEAR Up high schools. **Overview of the Evaluation** The study will use a multi - site cluster random assignment design to evaluate the effectiveness of the college savings and financial counseling intervention implemented as part of the demonstration. In spring 2013, after the demonstration grants are awarded, the study team will randomly assign one -half of the participating GEAR UP high schools in each state to the treatment group and one -half to the control group. 2 In all participating schools , students will continue to receive services supported by GEAR UP (e.g. tutoring, counseling, and college scholarships ) . In schools assigned to the treatment group, their 9 th grade cohort also will receive the college savings accounts and financial counseling. Students in both groups will be tracked over time to collect administrative and survey data on key outcomes, including college savings, financial literacy, academic outcomes in high school, and educational attainment expectations. 3 The study will use the data to assess not only whether the intervention is effective, but how well it was implemented and whether its eff ectiveness depends on key components of the GEAR UP program or features of the college savings account intervention as it was designed and implemented. In particular, the evaluation is designed to answer four main research questions: 1) To what extent do the college s avings accounts and counseling

—above and beyond the servic es GEAR UP may already provide

—improve college savings and other student and family behaviors and attitudes associated with college going? 2) What is the fidelity of implementation of the c ollege savings account model in the participating GEAR UP state grantees?

2 We plan to conduct blocked random assignment of schools, using state and one or more other blocking factors that are associated with the key student outcomes in this study. 3 The current 5 - year contract allows only for tracking students into the 11 th grade. ED may execute an option to extend the contract to allow for collection and analysis of data on college enrollment and persistence.**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **3** 3) What types of services

—including college savings, financial education, a nd financial literacy education

—are typically provided to GEAR UP high school participants? 4) Is there variati on across states in the impact of the college savings account program, and to what extent is the variation associated with features or characteristics of the GEAR UP project or the college savings account program? To answer these questions we will conduct both impact and descriptive analyses. The first report, which will address each research question after one year of demonstration implementation , will be published in 2016. The final report, after implementation over three years, will be available in 2018 . Exhibit B - 1 below presents the research questions along with the data sources for each question, the analytic approach and outcomes of interest .**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -** **4 Exhibit B**  **- 1.**  **Evaluation Questions, Data Sources, A naly** **tic A pproach, and Outcomes of Interest**  **Research Question**  **Data Sources Anal**  **yt ic Approach**  **Outcomes of Interest**  1) To what extent do the college savings accounts and counseling
— above and beyond the services GEAR UP may already provide
— improve college savings and other student and family behaviors and attitudes associated with college going?  Educational administrative data  Aggregate high school level data  Savings account data  Parent survey data  Student survey data  Impact analysis  Administrative data sample size: 20,000 students in 100 schools  Survey sample: 8,000 students and 8,000 parents  College Savings  Financial literacy  Educational expectations  Academic performance  Course - taking 2) Wh at is the fidelity of implementation of the college savings account model in the participating GEAR UP state grantees?  Project Director and Site Coordinator survey data  Parent survey data  Student survey data  PD attendance sheets  Descriptive analysis  Sur vey sample: 8,000 students, 8,000 parents, 110 project directors and site coordinators  Participation in PD trainings  Description of GEAR UP program components provided and received 3) Wh at types of services
— including college savings, financial education, and financial literacy education
— are typically provided to GEAR UP high school participants?  Project Director and Site Coordinator survey data  Parent survey data  Student survey data  D escriptive analysis  Survey sample: 8,000 students, 8,000 parents, 110 project directors and site coordinators  Description of GEAR UP program components

4) Is there variation across states in the impact of the college savings account program, and to what extent is the variation associated with features or characteristics of the GEAR UP project or the college savings account program?  Educational administrativ e data  Aggregate high school level data  Savings account data  Parent survey data  Student survey data  Project Director and Site Coordinator survey data  Impact analysis / moderator analysis  Administrative data sample size: 20,000 students in 100 schools  Survey sample: 8,000 students, 8,000 parents, 110 project directors and site coordinators  College Savings  Financial literacy  Educational expectations  Academic performance  Course - taking



### B.1 **Respondent Universe and Sampling Methods B.1.1**  **Respondent Universe** The respondent universe includes all states selected through the College Savings Account Research Demonstration competition. Successful grantees will not only meet the eligibility criteria but provide a strong plan for how they will carry out the activities o f the demonstration. Although there are 26 states that received GEAR UP awards in FY 2011 or FY 2012 it is unclear at this time how many**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **5** states will meet the other criteria and choose to apply to or be awarded a grant through the demonstration. Our best estimate is 5 - 8 states will receive demonstration grants. Among states with demonstration grants, we expect there to be approximately 100 participating GEAR UP high schools . High schools will fall into one of two cohorts; cohort 1 includes high schools with eligible 9 th grade students in fall 2013 and cohort 2 includes high schools with eligible 9 th grade students in fall 2014. Given that demonstration grants will only be awarded to states implementing GEAR UP with a cohort model, all 9 th graders in the school will be considered GEAR UP participants. For cohort 1 schools, participants will include all students in 9 th grade in fall 2013 and for cohort 2 schools participants will include all students in 9 th grade in fall 2014. Our estimate o f the number of student s in the evaluation sample is based on the average number of 9 th graders in high schools in the U.S. (200); therefore we expect the evaluation sample to include approximately 20,000 students in cohorts 1 and 2, representing all of th e GEAR UP high school participants in the demonstration states. Exhibit B - 2 indicates the estimated population sizes for the evaluation. **Exhibit B**  **- 2**  **. Population Size Estimates** Estimate d num ber of demonstration state grantees 5 to 8 grantees Estimated nu mb er of GEAR UP high schools per grantee 12 to 20 schools Estimated number of 9 th graders in GEAR UP cohort per grantee 2,400 to 4,000 students TOTALS High schools 9 th grade GEAR UP participants 100 20,000
**B.1.2 Sampling Methods**  There will be no sampling of schools for the evaluation. The entire universe of GEAR UP schools within the demonstration state grant s will participate in the evaluation study. All 9 th grade students in demonstration schools, in cohort 1 and cohort 2, will participate in the evaluation. S tudent rosters , the only data for which forms clearance is currently being sought, will be collected to identify all participating students. Later, administrative records will be collected for all 9 th grade students in cohorts 1 and 2 to maximize the statistical power of the analysis without substantially affecting study costs (e.g., since, for example, electronic matching ensures that extracting data for all sample members from a given state’s database is no more burdensome than extracting data from a smaller sample from the same database). Parent and student surveys will be administered to a randomly selected sample of 40% of students to limit study burden and costs . This will result in the random selection of approximately 8,000 students and their parents from the population of 20,000 students . The selected students and one of their parents will be asked to complete surveys twice during the study period . Section B.2 describes the random selection procedure for the survey s ample in more detail.**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **6 B.1.3**  **Expected Response Rates** We expect to obtain a 100% response on the registration lists from schools. We expect an 80 % response rate for the student and parent surveys. Although we will randomly select approximately 8,000 students and their parents, we anticipate obtaining completed surveys from 6,800 students and their parents at each round of data collection

– spring of 9 th grade and spring of 11 th grade. 

### B.2 **Statistical Methods for Sample Selection and Degree of A ccuracy**  **Needed B.2.1**  **Sample S election**  Random assignment will occur at the school level. Although there will be two cohorts of schools

– one cohort of high schools with eligible students in 9 th grade in fall 2013 and a second cohort of high schools with eligible students in 9 th grade in fall 2014, both cohorts of schools will be awarded the demonstration grant at the same time, in spring 2013. Therefore the Abt study team will randomly assign both cohorts of schools in spring 2013. Within each state, the Abt study team will randomly assign half of the participating high schools to the treatment group and half to the control group. Stratification by state and cohort will ensure that there are an equal number of treatment and control schools within each grantee and within each cohort. In addition, for districts with a large number of participating high schools, the Abt study team will stratify by districts to avoid situations where the share of treatment schools is much larger in some districts than others. We also plan to block high schools for random assignment using data from the National Center for Education Statistics’ Common Core Data (CCD), such as school size and percentage of students receiving free - or reduced -price lunches, as well as the number of students in eac h high school who have completed their Free Application for Federal Student Aid (FAFSA) form from data made available by the Federal Student Aid (FSA) office. These variables are potentially correlated with the study’s key outcomes and so blocking may pro vide greater precision in calculating the impact estimates. For the study, the student population will include all 20,000 9 th grade GEAR UP participants enrolled in treatment and control schools . Administrative data will be collected from the full populati on of students. To limit burden and study costs, for parent and student surveys, the Abt study team will randomly select a 40 -percent matched sample of students and their parents. Selecting matched samples will ensure that responses by students and their parents can be linked. The selection of the survey sample will occur twice

– once for cohort 1 and a year later for cohort 2. For each cohort, the sample will be stratified by school

—that is, the Abt study team will randomly select a 40 percent sample of students and their parents within each of the 100 schools

— to account for variation in outcomes across high schools in the sample. However, the Abt study team will not oversample certain types of students or schools because baseline data on students and parents will not be available when the selection of the survey samples occurs, and none of the analysis would benefit f rom oversampling. **B.2.2 Estimation Procedures**  This section presents the study’s estimation approach for addressing the research questions:**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **7 1.**  **To w hat extent do the college sa** **vings accounts and counseling**
**— above and beyond the**  **services GEAR UP may already provide**
**— imp**  **rove college savings and other student and family behaviors and attitudes associated with college going?**  To examine overall program impacts, the Abt study team will use multilevel modeling to account for the nested structure of the data. The Abt study te am will estimate a two - level model, with students (level - 1) nested within schools (level - 2). At the student level, the Abt study team will control for student achievement prior to 9 th grade and student demographic characteristics, including race/ethnicity, eligibility for free or reduced -price lunch, English language learner status, special education status. At the school level, the Abt study team will control for baseline differences between states, by including a set of state indicator variables, and for baseline school characteristics, including the percentage of seniors applying for federal financial aid, average school achievement, and percentage of students eligible for free or reduced -price lunch. Reflecting the randomization of schools, the Abt study team will include a treatment indicator at the school level to provide an estimate of the average impact of the college savings demonstration on students across all schools at a given time (e.g., at the end of 9 th grade, end of 11 th grade, or after the expected date of high school graduation). The Abt study team will conduct a test of the null hypothesis of no treatment impact for each outcome variable. The Abt study team will interpret a positive and statistically significant impact at the 5 - percent leve l (using a two -tailed test) as compelling scientific evidence that the intervention improves the targeted outcome (e.g., college savings, number of credits earned in college preparatory courses; financial literacy, completion of specific courses by a speci fied grade level). For continuous outcome variables (e.g., number of credits, amount contributed to savings account), the Abt study team will estimate multilevel linear models; for dichotomous outcomes (e.g., course completion, contributed to college savings account), the Abt study team will estimate multilevel logistic regression models. If the Abt study team examines impacts on ordinal outcomes (e.g., plans to attend 4 -year college, plans to attend 2 -year college, no plans to attend college), the Abt stud y team will estimate multilevel ordered logistic regression models. **Hierarchical Linear Model** The following hierarchical linear model will be used to estimate program impacts on continuous outcome variables. The Level - 1 equation (student level or family level, depending on the outcome) is: (1) = + h − h + ∑ (
− ) + Where is the value of the outcome (e.g., amount contributed to savings account) for the i th student in the j th school; h is a measure of achievement for i th student in the j th school (centered at the grand mean, h ); is a vector of k variables measuring the demographic characteristics of the i th student in the j th school (each centered at the grand mean, ); is the covariate - adjusted mean value of the outcome for students in school j ; through are regression coefficients indicating the effects of each student - level covariate on the outcome variable ; and is the student - level residual for the i th student in the j th sch ool (the assumed distribution of the residuals is normal, with mean 0 and variance . The Level - 2, or school - level, equation is:**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **8** (2) = + + h + ∑ ( h h 12

_ 13 − h h 12

_ 13 ) + Where is the covariate - adjusted mean value of the outcome measure across control schools; is the mean difference in the covariate - adjusted outcome between treatment and control schools (i.e., the treatment impact); is the treatment status dummy variable with a value of 1 for a GEAR UP school assigned to the treatment condition and a value of 0 for a GEAR UP school assigned to the GEAR UP control condition; h is indicates whether schools implement the demonstration with students in 9 th grade in 2013 or with students in 9 th grade in 2014; h h 12

_ 13 is a vector of k variables measuring the characteristics of the j th school in school year 2012 - 2013, prior to random assignment (each centere d at the grand mean, h h 12

_ 13 ); is a vector of k regression coefficients indicating the effect of each baseline school covariate on the covariate - adjusted mean value of the outcome measure; and is the error term for the j th school (the distribution is assumed to be normal with mean 0 and variance ). The parameter indicates the impact of the demonstration on the outcome, . A t -test will be conducted to test the null hypothesis of no treatment impact using a .05 level criterion to reject the null hypothesis. A positive and statistically significant estimate of will indicate that there is compelling scientific evidence (at the 5 percent level) that the demonstration improves the targeted outcome. The parameter indicates the magnitude of the impact -- such that the college savings demonstration in GEAR UP high schools is estimated to have, on average, a point effect on the specified outcome. A standardized effect size will be calculated by dividing the estimated impact ( ) by the standard deviation of the outcome variable, , in the control group ( ). The standardized effect size is . The control group standard deviation will be used, as recommended by Burghardt, Deke, Kisker, Puma, and Schochet (2009), rather than the pooled standard deviation, because the intervention might affect the standard deviation in the treatment group. **Mult ilevel Logistic Regression**  For dichotomous outcomes, a logistic regression equation will replace the level - 1 linear equation specified for continuous outcomes as follows: For a multilevel logistic regression model with a dichotomous outcome, the level - 1 equation is: (3) 

## [ = 1 ] = ( ∑ ) Where is the covariate - adjusted log
- odds of the outcome occurring (e.g., course completion; plans to attend college) versus not occurring (e.g., course not completed; does not plan to attend college) for students in school j ; and through are regression coefficients indicating the log - odds of each student - level covariate on the outcome occurring. The level - 2, or school - level, equation remains the same as equation (2); however, is log - odds of the outcome occurring in the control group (e.g., course completion by control students; plans to attend college by control students), rather than the mean value of the outcome in the control group,**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **9** and is the log - odds ratio quantifying the treatment impact (i.e., difference between treatment and control schools in the log - odds of the outcome occurring versus not occurring). Impacts estimated as log - odds ratios are measured in standardized effect size units To convert estimates produced by logistic regression to an impact measured in percentage point increases in probability of the binary outcome, we calculate the individual level probability of the out come given treatment implied by individual and school level covariates and model estimates, we repeat this calculation if the individual were assigned to control, and find the mean difference between these two probabilities. **2. What is the fidelity of impleme**  **ntation of the college savings account model in the participating GEAR UP state grantees?**  Descriptive analyses will be used to provide information on the fidelity of implementation. To describe fidelity of implementation, the Abt study team will define what constitutes “implementation as intended” and construct a measure of the extent to which sites implement program components with fidelity (e.g., establish college savings accounts, provide seed funds, provide matching funds, attend financial counseling professional development, provide financial counseling for parents, provide financial counseling for students). The Abt study team will report descriptive data on the percentage of sites that implement each key program component with fidelity, providing information about which program components are widely implemented and which are implemented less consistently. Information on the overall level of and variation in implementation fidelity will provide important contextual information for interpreting impact findings. **3. What types of services** 
**— including college savings, financial education, and financial literacy**  **education**
**— are typically provided to GEAR UP participants?**  Descriptive analyses will be used to provide information on the types of services typically pro vided to GEAR UP participants. The Abt study team will describe the types and intensity of services regularly offered to GEAR UP participants at the high school level as reported on the project directory survey and on the parent and student surveys. The Abt study team will describe the services that constitute the counterfactual condition in order to understand the extent to which treatment services are distinct from the usual services available to GEAR UP participants, illuminating the treatment - control co ntrast. These results will provide another important source of contextual information for interpreting impact findings. **4. Is there variation across states in the impact of the college savings account program, and to**  **what extent is the variation associated w ith features or characteristics of the GEAR UP**  **project or the college savings account program?** To examine variation in program impacts, the Abt study team will use a two - level model, controlling for baseline covariates, as described above. The Abt study team will include interaction terms in the school level model (i.e., equation 2 above), allowing the treatment impact to vary by state or by treatment program characteristics, such as type of savings bank or mode of counseling services offered. For example, to test for state variations in the program impact, the Abt study team will include a set of interactions between the treatment indicator and each of the state indicators in Equation 2, as follows:**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **10** (4) = + + ∑ ( h h 12

_ 13 − h h 12 ) +

∑ + ∑ ∗ + where is the covariate - adjusted mean value of the outcome measure (or log
- odds of the outcome occurring) among control schools in the reference category state ; STATE is a vector of s - 1 state indictor variables

—one for all but one of the s states in the sample; is the difference between STATE s and the reference - category state in the covariate - adjusted mean value (or log
- odds) of th e outcome among control schools; is the mean difference in the covariate - adjusted outcome between treatment and control schools (i.e., the treatment impact); STATE * Trt j is a vector of state by treatment condition interaction terms

—one for each s - 1 st ate indicator variable; and is the difference in the treatment impact between STATE s and the reference - category state. Using a .05 level criterion, the Abt study team will conduct a joint test of the null hypothesis that the s parameters for all of the interaction terms are zero, i.e., that there is no statistically significant difference in the program impact for any given state. The null hypothesis is expressed as follows:

: ∗ = ∗ = ⋯ = ∗ = 0 The Abt study team will interpret a statistically significant difference as evidence that the program impact varies by state (or by features of the treatment program). If the Abt study team finds a statistically significant difference in p rogram impacts, the Abt study team will then test whether the program impact within each state or subgroup is statistically significant. Standardized effect sizes will be calculated as described above. **B.2.3 Degree of A** **ccuracy Needed** This section presents the power analyses conducted to estimate the minimum detectable effect sizes (MDES) for the treatment/control differences on student and family outcomes. The MDES estimates for binary outcomes are based on the following assumptions : (1) the demonstration includes 100 schools (2) 50 schools are assigned to each group (3) the full sample includes 200 students per school (4) the student and parent survey samples includes 40 percent of the full sample (5) the surveys obtain 80 percent response rates (6) the intra - class correlation equals .05 (7) the school - level R - square equals .32 (8) the outcome variable equals 1 for 32 percent of control group members (9) the proportion of disadvantaged students in the sample is 50 percent at baseline. In addition, the Abt study team made standard statistical assumptions (i.e., two -tailed testing at the 5 percent level and 80 percent power). Because many of our key outcomes are binary variables

– whether participant s are saving for college ; completion of college preparatory courses (e.g., completion of Algebra I in 9 th grade; completion of calculus; completion of an AP or IB course); and expect ations about college going

– the power calculations assume binary outcomes. The Abt study**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **11** team used the formulas in Schochet (2011) for binary outcomes to calculate MDES. These power calculations for binary outcomes provide conservative MDES estimates. Our analyses of continuous outcomes (e.g., amount of college savings; financial literacy scale score), will have greater statistical power to detect impacts (i.e., lower MDES) with the same sample size (see Appendix B for the power calculations for continuous variables) . The Schochet (2011) paper derives MDE equations and variance estimates that reflect the process of transforming logistic regression estimates into percentage po int impact estimates. Because the percentage point impact estimates are the expected difference in probability between the treatment and the control groups, these impact estimates reflect the joint distribution of the outcome and covariates in absence of treatment, in addition to the estimated log - odds treatment effect produced by the logistic regression. To make this relationship tractable, Schochet (2011) assumes a single binary covariate. Row 1 in Exhibit B - 3 demonstrates that the MDES will be 5 percentage points for outcomes based on administrative data (e.g., whether students take calculus or attend a four-year university) and 6 percentage points for outcomes based on survey data (e.g., whether families save for college). If ICCs prove to be larger than our assumed ICC of .05, the MDES may be larger. For example, if the ICC is .10, which is consistent with some of the estimates reported in Schochet (2011), the MDE would be approximately 7 percentage points for outcomes based on both administrative data and 8 percentage points for outcomes based on survey data (see row 2). **Exhibit B -**  **3 .**  **MDES Estimates for Binary Outcomes from A dministrative and Survey Data**  **Intraclass Correlation Ad**  **ministrative Data Outcomes (200 students/school)**  **Survey Data Outcomes (68 st**  **udents/school) 1**  .05 5.1 percentage points 5.7 percentage points .10 7.0 percentage points 7.5 percentage points 1 The survey sample includes 40% of the stu dents in the full sample (i.e., 80 students/school); with a n 80 % response rate, the Abt study team assumes 64 students/school in the analysis sample for survey data outcomes. Power analyses were based on the following MDES formula, in which Schochet refers to minimum detectible effect sizes for binary outcomes as minimum detectible impacts (MDI) in perc entage points:

= ( , { = 1 } , { = 1 } , R , , , ) − 1 − 2 where: : significance level for a two -tailed test, : statistical power; ( . ) the cumulative density function of the normal distribution; ( . ) the inverse of th e cumulative density function of the normal distribution; : minimum detectible impact in percentage point terms;**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **12** ( . ) : the variance of the estimated impact, which is a function of the MDI and the following terms; { = 1 } : the probability that the outcome occurs for control group members (e.g. the student completes the course); { = 1 } : the probability that the covariate is one (e.g. the probability that a student is disadvantaged at baseline); R : proportion of the student - level variance explained by the covariate; : total number of schools; : average number of students per classroom with a valid outcome measure after account for attrition and non -response; and : the intra - class correlation for the outcome. Unlike the standa

rd power analysis equation, the equation above cannot be solved for the MDI analytically, as the MDI enters the equation for variance: iterative methods must be used. To find the MDI for a given sample size and power, the Abt study team fixed the number of schools and iterated over MDI until we obtained the desired power.
**B.2.4 Unusual Problems Requiring Specialized Sampling Procedures**  Unusual problems that require specialized sampling procedures are not anticipated. **B.2.5 Use of Periodic Data Collection Cycles to Red**  **uce Burden** The data collection plan reflects sensitivity to issues of efficiency and respondent burden. Whenever possible the Abt study team will collect data at the fewest intervals allowable to maintaining data accuracy. The student rosters will only be collected once. Student, parent, and project director surveys will be fielded twice during this study, to provide timely results and minimize burden; fewer collection periods would prevent us from assessing how implementation and key behavioral outcomes fo r students and parents (financial literacy, understanding of college costs) change over time. During this study annual collection of state or district administrative educational records each fall is necessary to track students across schools and districts. To maintain data accuracy, college savings account data will be collected annually during Years 3, 4, and 5 of the study. 

### B.3 **Methods to Maximize Response Rates and Deal with Nonresponse** This section describe s the strategies and methods that will be used to maximize response rates and deal with non -response . In order to obtain responses from the study sample, the study team has developed strategies to facilitat e communication with respondents during data collection activities and to maximiz e response rates . These strategies have proven successful in the study team’s extensive experience conducting large - scale evaluation studie s (e.g., The Reading First Impact Study, Evaluation of the U.S. Department of Education’s Student Mentoring Program, Evaluation of the Ma ssachusetts Expanded**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **13** Learning Time Initiative; The Enhanced Reading Opportunities Study; The Career Academies Evaluation; The Teacher Incentive Fund Evaluation). T here are two phases to this study. This first ICR requests approval for Phase I

– Random Assignment and Collection of Student Rosters will be used to identify GEAR UP student participants and to randomly assign GEAR UP high schools to treatment and control groups . Specifically, the student rosters will define the evaluation sample for all analyses and serve as a sampling frame for the student and parent surveys. An addendum ICR will request clearance for Phase II

– Collection of Survey and Administrative Data . The Abt study team will follow several procedures to maximize response rates and handle nonresponse during the Phase I

– Random Assignment and Collection of Student Rosters activities covered under this ICR. During the July 2013 meeting of the association of GEAR UP grantees (the National Council for Community and Education Partnerships , o r NCCEP) , the Abt Study Team will provide an evaluation session for participating project directors and site coordinators. The session will clearly explain the importance of the evaluation and the roles and responsibilities of project directors and site co ordinators, including the process for submitting student rosters in the fall. Prior to the start of the school year, the Abt team will send an introductory email to each site coordinator explaining the evaluation and the requests that will be made of site coordinators (see Appendix A). Next, the Abt study team will reach out to each site coordinator via a telephone call during the first 10 days of the 2013 - 2014 academic year to request st udent rosters of 9 th grade GEAR UP participants in both treatment and control schools and to answer any questions that the site coordinators may have. In the event that there are two cohorts of eligible schools, the student roster collection will occur in both fall 2013 and 2014; 9 th grade student rosters will be collected f rom cohort 1 schools in fall 2013 and rosters of 9 th graders in cohort 2 schools will be collected in the fall of 2014. The submission of student rosters via a password protected secure file transfer portal (SFTP) will allow site coordinators to submit rosters at a time most convenient for them and will assure coordinators that student confidentially is protected. During Phase II

– Collection of Survey and Administrative Data of this study, the Abt study team will face attrition from the study and missing data for some parts of the sample. Disadvantaged families and, therefore, high poverty schools tend to have high rates of student mobility and dropping out . In addition, cooperation with surveys has dropped in the last twenty years, and the Abt study team expects that obtaining willing cooperation will be challenging for the student and parent surveys. 4 These respondents may not want to take the time necess ary to complete the study surveys. Alternatively, they may be willing to participate, but only if the study team makes it easy for them to do so, or offers them something in return.

4 We believe we will be able to achieve high rates of response from project directors and site coordinators because they are required to participate in data collection for a national evaluation under ED’s Educ ation Department General Administrative Regulations (EDGAR) and in the demonstration’s data collection as a condition of participating in the grant.**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **14** To track the sample of students and parents over time, the Abt study team will use four strategies. 1. **Site coordinator s**  **involvement.** The study team will ask site coordinators to track the students from their school as they move, change schools, or drop out of school, and to provide updated contact information for the study sample . Itis likely that site coordinators in treatment schools will need to track students anyway, given their role in implementing the intervention. 5 Furthermore, for site coordinators in control schools, tracking sample members can reasonably be considered p art of their designated responsibility to support the evaluation’s data collection efforts. Before each round of surveys, the Abt study team will ask site coordinators for updated contact information for students who move residences or schools. This will h elp to ensure high response rates for telephone surveys to parents and to students and it will also facilitate obtaining administrative educational record each fall. 2. **Updates from a dministrative records.**  S tate or district administrative educational records will be gathered each fall . These records should contain information on where students were enrolled in school; in some states, it may also include directory information on addresses and telephone numbers. Even information on the school a student was attending may be helpful. It may help to explain why the Abt study team was unable to obtain completed surveys in the spring (e.g., if state data suggests that a student transfers to a public school in another part of the state, it suggests the family moved); it may also help the study team in locating where the family now lives, which will help us obtain a completed survey during the next administration. 3. **Alternate contact information.** The first round of surveys will request more detailed contact information than the Abt study team would expect to obtain from student rosters. In particular, the Abt study team will ask parents for cell phone numbers, email addresses, and/or the telephone number of a family member who could help the study team locate the sample mem ber. This more detailed information will help to locate sample members for the second round of survey and administrative data collection. 4. **State grantee involvement.** S ate grantees will be asked to notify the study team when a site coordinator leaves or is replaced. This will help us to locate site coordinators for the project director survey; it will also help to stay in touch with the current site coordinators, which will allow the Abt study team to enlist their help in tracking the student and parent samp les. To obtain willing cooperation from students, parents, project directors, and state and district officials to the evaluation’s data collection efforts, the Abt study team will use three strategies. First , the Abt study team will use multiple modes to obtain completed surveys and administrative data. For example, the study team will:

 Offer students and parents the option of completing surveys by web survey or telephone;
 Offer site coordinators and project directors the option of completing surveys by web survey or in paper and;

5 Treatment group students and families are eligible to continue making contributions to the college saving accounts and to receiving federal matching funds even if the student leaves a GEAR UP high school to attend a non - GEAR UP high school.**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **15** Second , the Abt study team is considering the use of incentive payments for parents and students completing the surveys. The Addendum to this ICR, which will cover Phase II of the study, will provide further details on the potential use of incentive payments. Third , the Abt study team will leverage contact with sample members to market the evaluation and prepare them for upcoming surveys. For example, the Abt study will:

 Introduce the project director survey to project staff at the biannual grantee conferences (the conferences at which professional development training will be provided);
 Use regular contact with site coordinators to obtain updated contact information for sample me mbers and to remind them to complete the upcoming project director survey; and
 Ask GEAR UP project directors to include information about upcoming surveys in materials that they mail to students and parents regarding their college savings account or upcoming financial counseling sess ions. 

### B.4 **Test of Procedures and Methods to be Undertaken** We do not believe pretesting is necessary for the collection of student rosters. The Abt Study team has collected rosters for other studies such as the Evaluation of Citizen Schools: School
- Level Expanded Learning Time . T he burden estimates were developed based on these previous experiences. 

### B.5 **Individuals Consulted on Statistical A spects of the Design**  The following individuals were consulted on the statistical aspects of the study: **Name Title/A**  **ffi liation**  **Telephone** Mr. Cristofer Price Principal Scientist, Abt Associates 301 - 634 - 1852 Dr. Alina Martinez Principal Scientist, Abt Associates 617 - 349 - 2312 Dr. Rob Olsen Principal Scientist, Abt Associates 301 - 634 - 1716 Dr. Anne W olf Associate, Abt Associates 301 - 634 - 1738 Dr. Amanda Parsad Senior Associate, Abt Associates 301 - 634 - 1791 Dr. Michal Grinstein -We iss Associate Director at Center for Social Development at W ashington University in St. Louis 314 - 935 - 8201 The following individuals will be responsible for the Data Collection and Analysis: **Name Title/A**  **ffiliation Telephone**  Dr. Alina Martinez Principal Scientist, Abt Associates 617 - 349 - 2312 Dr. Rob Olsen Principal Scientist, Abt Associates 301 - 634 - 1716 Dr. Amanda Parsad Senior Associate, Abt Associates 301 - 634 - 1791 Dr. Tamara Linkow Associate, Abt Associates 617 - 520 - 2978 Ms. Linda Kuhn President, Survey Research Management 303 - 247 - 0140**Revised OMB Supporting Statement Promising Strategies in Federal College Access Programs** 

**Abt Associates Inc. Part B:**  **Collection of Information Employ ing Statistical Methods**  **▌ pg.**  **B -**  **16** 

### References Burghardt, J., Deke, J., Kisker, E., Puma, M., & Schochet, P. (2009). Regional educational laboratory rigorous applied research studies: Frequently asked analysis questions. Institute of Education U.S. Department of Education. Princeton, NJ: Mathematics Policy Research. Schochet, P.Z. (2011). Statistical Power for Binary Outcomes for Clustered RCTs of Education Interventions . Presented at the Society for Research in Educational Effectiveness Spring Conference, March 3 - 6, 2011. U.S. Department of Education. (2012). Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) Funding Status. Retrieved on January 8, 2013 from: http://www2.ed.gov/programs/gearup/funding.html