The Impacts of Regular Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High School Graduation

**Final Report**

Prepared by:

Neil S. Seftor

Arif Mamun

Allen Schirm

Mathematica Policy Research, Inc.

Princeton, N.J.

Prepared for:

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2009

This report was prepared for the U.S. Department of Education under Contract No. EA97030001.  The project monitor was Margaret Cahalan in the Policy and Program Studies Service.  The views expressed herein are those of the contractor.  The content of this report does not necessarily reflect the views or policies of the U.S. Department of Education, nor does the mention of commercial products or organizations imply endorsement by the U.S. government.  The inclusion of such information is for the reader's convenience and is not intended to endorse any views expressed, or products, programs, models or services offered.

**U.S. Department of Education**

Margaret Spellings

*Secretary*

**Office of Planning, Evaluation and Policy Development**

Bill Evers

*Assistant Secretary*

**Policy and Program Studies Service**

Alan L. Ginsburg

*Director*

**Program and Analytic Studies Division**

David Goodwin

*Director*

**January 2009**

______________________________________________________________________________________

This report is in the public domain.  Authorization to produce the report in whole or in part is granted.  Although permission to reprint this publication is not necessary, the citation should be:  U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service, The Impacts of Regular Upward Bound on Postsecondary Outcomes Seven to Nine Years After Scheduled High School Graduation*,* Washington, D.C., 2009.

# ACKNOWLEDGMENTS

This report on the effects of Upward Bound on students’ postsecondary outcomes reflects the contributions of many individuals.  The authors would like to thank David Goodwin and Margaret Cahalan of the Policy and Program Studies Service at the U.S. Department of Education.  We are grateful to David for his continued support, substantive guidance, and encouragement throughout the study and to Maggie for her many very valuable comments and other guidance.  Marisol Cunnington and Jay Noell of the U.S. Department of Education also provided helpful suggestions.  In addition, the report has been improved by comments from three anonymous external reviewers selected by the Institute of Education Sciences.

David Myers, who directed the national evaluation of Upward Bound for many years, played a key role in shaping the study design and led previous impact analyses.  Mary Moore, a principal investigator for the national evaluation, was also instrumental in shaping the study design.  Mark Dynarski and Peter Schochet provided insightful suggestions throughout the analysis and drafts of the report. 

The authors would also like to thank the individuals who contributed to the data collection and analysis and the production of the report.  David DesRoches oversaw the data collection, and Zhanyun Zhao created the survey weights.  Karin Zeller constructed the analysis variables and computed the impact estimates, and Mary Grider provided technical assistance with data and programming issues.  Jennifer Baskwell produced the document.

# CONTENTS

**ChapterPage**

EXECUTIVE SUMMARYxiii

IINTRODUCTION1

A.Context and Purpose of Upward Bound1

B.Context for Interpreting Program Effects7

1.Value-added of Upward Bound8

2.Upward Bound Applicants11

3.Effects for Students Who Participated During the Mid-1990s12

4.Variation in Exposure to Upward Bound14

C.Previous Findings16

IIRESEARCH DESIGN AND ANALYTIC ISSUES19

A.Research Design19

1.Selection of Upward Bound Projects and Random Assignment19

2.Outcome Measures22

3.Data Sources24

4.Construction of the Outcome Measures26

B.analytic issues28

1.Estimation of Program Impacts28

2.Subgroup Analysis31

3.Use of Weights to Account for Sample Design and Survey Nonresponse35

4.Potential Threats to the Study Design37

IIITHE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES39

A.The Effect of the Opportunity to Participate in Upward Bound (ITT)40

1.The Effect of Upward Bound on Postsecondary Enrollment and College Selectivity43

2.The Effect of Upward Bound on Financial Aid Application and Pell Grant Receipt43

3.The Effect of Upward Bound on Postsecondary Completion44

**ChapterPage**

b.The Effect of Participation in Upward Bound (CACE)44

C.Summary of Sensitivity Analyses44

1.Sensitivity Analyses Pertaining to the Measurement of Outcomes47

2.Sensitivity Analyses Pertaining to Sample Weighting48

IVTHE EFFECT OF UPWARD BOUND ON POSTSECONDARY OUTCOMES FOR SELECTED SUBGROUPS55

A.The Effect of the Opportunity to Participate in Upward Bound for Selected Subgroups (ITT)56

1.Grade at Application to Upward Bound56

2.Applicants’ Educational Expectations at the Time of Application to Upward Bound57

3.Level of Ninth-Grade Mathematics Class60

4.Grade Point Average in Ninth Grade63

B.The Effect of Participation in Upward Bound on Selected Subgroups (CACE)65

VTHE EFFECT OF ADDITIONAL UPWARD BOUND PARTICIPATION ON POSTSECONDARY OUTCOMES67

A.Research Questions67

B.Research Methods68

C.The Effect of Additional Upward Bound Participation71

1.Postsecondary Enrollment71

2.Financial Aid73

3.Postsecondary Completion74

D.Interpretation of the Findings74

References77

APPENDIX A:  Sample Design, Unit Nonresponse, and WeightsA.1

APPENDIX B:  DATA COLLECTION and outcome measuresB.1

**ChapterPage**

APPENDIX C:  sensitivity analyses pertaining to the                    measurement of outcomesC.1

APPENDIX D:  SAMPLE SIZES AND WEIGHTED STANDARD                    DEVIATIONS FOR ALL OUTCOME VARIABLESD.1

APPENDIX E:  ESTIMATION OF IMPACTS AND STANDARD ERRORSE.1

APPENDIX F:  METHODS USED TO ESTIMATE THE EFFECTS                   OF ADDITIONAL UPWARD BOUND PARTICIPATIONF.1

APPENDIX G:  SENSITIVITY ANALYSES pertaining to sample                    weightingG.1

APPENDIX H:  OTHER SUPPLEMENTAL SERVICE PROGRAMSH.1

APPENDIX I:  ADDITIONAL SUBGROUP TABLESI.1

# TABLES

**TablePage**

I.1Receipt of Supplemental Services9

I.2Receipt of Upward Bound and Other Supplemental Services10

I.3Unweighted Rates of Participation in Upward Bound Among Treatment Group Members15

II.1Response Rates25

II.2Control Variables Included in the Regression Models31

II.3Sample Size in Applicant Characteristic Subgroups by Treatment Status32

II.4Sample Size in Project Characteristic Subgroups by Treatment Status34

III.1Impact of Upward Bound on Postsecondary Outcomes (ITT)41

III.2Impact of Upward Bound on Postsecondary Outcomes (CACE)45

III.3Illustrative Impact Estimates from Sensitivity Analyses52

IV.1Impact of Upward Bound on Postsecondary Outcomes by Students’ Grade at Application (ITT)58

IV.2Impact of Upward Bound on Postsecondary Outcomes by Students’ Educational Expectations (ITT)59

IV.3Impact of Upward Bound on Postsecondary Outcomes by Students’ Ninth-Grade Math Class (ITT)62

IV.4Impact of Upward Bound on Postsecondary Outcomes by Students’ Ninth-Grade GPA (ITT)64

V.1Duration of Upward Bound Participation and Completion Rates, Excluding No-Shows69

V.2Quasi-Experimental Impact of an Additional Year of Upward Bound on Postsecondary Outcomes72

V.3Quasi-Experimental Impact of Upward Bound Completion on Postsecondary Outcomes73

# FIGURES

**FigurePage**

I.1Conceptual Framework for the Evaluation7

# Executive Summary

Policymakers have long been concerned about the disparities in college attendance between more and less advantaged groups of high school students.  Data from the 1990s indicate that students from low-income families were less than half as likely to attend a four-year college or university as students from high-income families.  This difference is not surprising given disparities in financial resources and college preparation between high- and low-income high school students.  While the vast majority of high-income high school graduates are qualified to attend a four-year college—based on grades and test scores—only half of low-income students have adequate qualifications (U.S. Department of Education 1997), and low-income students face greater financial barriers to college attendance (Kane 1999).

Upward Bound is one of the largest and longest-running federal programs designed to help disadvantaged students prepare for, enter, and succeed in college.  Upward Bound is “designed to generate skills and motivation necessary for success in education beyond high school among young people from low-income backgrounds and inadequate secondary school preparation” (Public Law 90-222, Dec 23, 1967).  Including the grants funded under the *College Cost**Reduction and Access Act of 2007*, 971 grants were awarded for fiscal year 2007 to serve over 65,000 students in the regular Upward Bound program.  The majority of Upward Bound projects are hosted by colleges and universities.  According to the program’s regulations, at least two-thirds of each project’s participants must be both low-income and potential first-generation college students.  Students typically enter Upward Bound while in ninth or tenth grade or the summer prior to those grades.  Although students may participate in Upward Bound through the summer following twelfth grade (for three to four years total), participants typically remain in Upward Bound for about 20 months (Myers et al. 2004).  Projects provide students with a variety of services, including instruction, tutoring, and counseling.  In addition to regularly scheduled meetings throughout the school year, projects offer an intensive instructional program that meets daily for about six weeks during the summer.  

In 1991, the Department of Education launched the National Evaluation of Upward Bound.  Conducted by Mathematica Policy Research, Inc. (MPR), the evaluation has included an implementation study—to assess how the program is implemented—and a longitudinal impact study.  The impact study was based on a random assignment design implemented in a nationally representative sample of 67 Upward Bound projects hosted by two- and four-year colleges and universities.  From 1992 to 1994, eligible applicants to these projects were enrolled in the study.  About 1,500 students were randomly assigned to the evaluation’s treatment group and allowed to participate in Upward Bound, and about 1,300 students were randomly assigned to the control group.  Comparing the experiences of treatment group members with the experiences of control group members, the evaluation has assessed the effects of the opportunity to participate in regular Upward Bound on high school and postsecondary outcomes.

From 1992 to 1994, a baseline survey collected information on students who applied to Upward Bound projects in the study.  Follow-up surveys of all treatment and control group members were conducted in 1994–95, 1996– 97, 199–99, 2001–02, and 2003–04, and high school and postsecondary transcripts were collected after each survey.  Upward Bound project staff reported on the participation of students in the program.  In addition to the survey, transcript, and participation data that were collected specifically for the evaluation, data from two administrative sources—the National Student Clearinghouse (NSC) and the federal Student Aid (FSA) records—were used in the evaluation.

This report is the last in a series of study reports from the Upward Bound evaluation.  It analyzes data from the final round of data collection as well as administrative records, and provides the national evaluation’s first estimates of the effects of Upward Bound on postsecondary completion.  It also updates previous estimates of the program’s effects on other postsecondary outcomes.  The survey data were collected between 2003 and 2004, approximately seven to nine years after sample members were scheduled to graduate from high school.  Other sources of data from the evaluation—previous surveys, high school and postsecondary transcripts, and data on Upward Bound participation provided by program staff—have also informed the findings.

The research questions addressed in this report are:

· What effect does Upward Bound have on the likelihood of attending a postsecondary institution and on the highest level of postsecondary attendance?

· What is the effect of Upward Bound on the likelihood of attending a relatively selective four-year college or university?

· What is the effect of Upward Bound on the likelihood of receiving financial aid in college?

· What is the effect of Upward Bound on the likelihood of earning a postsecondary degree, certificate, or license?

· For which groups of eligible applicants are the effects of Upward Bound greatest? 

· What is the effect of Upward Bound participation length and completion on postsecondary outcomes? 

## Study results

By comparing the study’s treatment group to its control group, this evaluation estimates the value-added effect of the opportunity to participate in Upward Bound—an unusually intensive precollege program—for the students who seek that opportunity and are eligible to participate in the program.  The main findings are:

· ***Upward Bound had no detectable effect on the rate of overall postsecondary enrollment or the type or selectivity of postsecondary institution attended for the average eligible applicant.***About four-fifths of both treatment group members and control group members attended some type of postsecondary institution, including four-year institutions, two-year colleges, and vocational schools, and the estimated impact is an increase of less than 2 percentage points in the rate of enrollment (effect size = 4 percent).  For enrollment at four-year colleges and universities, the estimated impact is 1 percentage point (effect size = 3 percent).  These effects are not statistically significant.  

· ***Upward Bound had no detectable effect on the likelihood of applying for financial aid, or, the likelihood of receiving a Pell Grant.***  The 1 and 2 percentage point increases in the rates of financial aid application and Pell Grant receipt (effect sizes = 3 and 5 percent) are not statistically significant.

· ***Upward Bound increased the likelihood of earning a postsecondary certificate or license from a vocational school.  It had no detectable effect on the likelihood of earning a bachelor’s degree or the likelihood of earning an associate degree.***  While about 4 percent of control group members received a vocational certificate or license, nearly 9 percent of treatment group members did, implying an impact of 5 percentage points (effect size = 23 percent).  The impacts on receiving any postsecondary credential and receiving a bachelor’s degree are 2 and 0 percentage points (effect size = 5 and 0 percent), respectively, and are not statistically significant.

· ***Upward Bound increased postsecondary enrollment or completion rates for some subgroups of students.***For the subgroup of students with lower educational expectations at baseline—that is, the students who did not expect to complete a bachelor’s degree—Upward Bound increased the rate of postsecondary enrollment and the likelihood of receiving a degree, license, or certificate by 6 and 12 percentage points, respectively, raising the overall postsecondary completion rate to about the level observed for students with higher expectations.  Because targeting on the basis of lower educational expectations might be challenging if it creates an incentive for applicants to understate their expectations, further analyses were conducted to examine the effects of Upward Bound on subgroups that might be more readily targeted.  According to these exploratory analyses, Upward Bound increased postsecondary enrollment rates for students who were in tenth grade or above at the time of application, students who took a mathematics course below algebra in ninth grade, and students with a ninth grade GPA above 2.5.  The estimated impacts were 3, 7, and 3 percentage points, respectively.  Additional analyses suggest that Upward Bound also had positive impacts on postsecondary outcomes for some other subgroups defined by student- and project-level characteristics.

· ***Longer participation in Upward Bound was associated with higher rates of postsecondary enrollment and completion.***  An additional year of Upward Bound participation was associated with a 9 percentage point increase in the rate of enrollment at four-year institutions and a 5 percentage point increase in the likelihood of receiving a bachelor’s degree.  Completing the Upward Bound program was associated with increases of 27 and 21 percentage points, respectively.  These findings are based on nonexperimental methods, and the validity of causal inferences based on these estimates depends on the validity of strong assumptions.

In the context of a complex, longitudinal study like the national Upward Bound evaluation, many difficult evaluation design and implementation issues arise and need to be considered when interpreting the study findings.  Comprehensive sensitivity analyses were conducted to ensure a thorough assessment of the implications of the design and implementation issues and whether the impact estimates are robust under alternative methods and assumptions.  Three key issues in particular are important in the Upward Bound evaluation—survey nonresponse, a highly stratified sample design, and no-shows and cross-overs.

**Response Rates to the Upward Bound Surveys Were High But Declined Over the Period of the Study**

One important design choice pertained to the length of the follow-up period for the evaluation.  Considering the objective of Upward Bound to prepare students for entry into and success in postsecondary education, the Department of Education specified a long follow-up period that allowed sample members to be observed for many years beyond expected high school graduation.  Although response rates to the evaluation’s follow-up surveys remained high, administrative data from the NSC and federal FSA files were obtained to assess and address the potential effects of survey nonresponse.

One set of sensitivity analyses examined alternative ways of combining data from the available sources—surveys, NSC, and FSA—to measure postsecondary enrollment and completion.  While nonresponse is one potential limitation of survey data, measurement and coverage error are concerns with administrative data.  Measuring postsecondary outcomes in different ways can shed light on how the relative strengths and weaknesses of the data sources affect the findings of the evaluation.  For that reason, the sensitivity analyses examined 27 different measures of postsecondary enrollment.  Estimates of Upward Bound’s impact on postsecondary enrollment across these 27 measures ranged from a negative 2.4 percentage points to a positive 2.8 percentage points, none of which were statistically significant.

**The Upward Bound Sample Design Was Highly Stratified with Highly Variable Selection Probabilities**

In designing the requirements for the Upward Bound evaluation sample, the Department of Education specified that the evaluation sample had to be nationally representative.  It also required that the sample have substantial overrepresentation of some less common, but key types of projects, including, for example, projects serving predominantly Native American students.  Alternative sampling schemes were considered, and a design was chosen to balance the competing needs of the evaluation.  The chosen design had much higher selection probabilities for the relatively rare projects than for more common types of projects, leading to substantial undersampling and underrepresentation of the latter.  This led to very unequal weighting of projects in the evaluation sample.

One implication of the sample design was that some of the most common types of Upward Bound projects had low selection probabilities and were substantially undersampled.  This is true of one set of projects in particular—projects that were medium-sized, located in an urban setting, hosted by a four-year public institution, and not serving a group of students that is predominantly Asian, Native American, or Latino.  This stratum of projects ends up accounting for about 26 percent of all eligible Upward Bound applicants nationwide.  The final sample selected for the impact evaluation included only one project out of 56 projects in this stratum.  The analysis weights the sample accordingly, and the sample members from this one project account for approximately 26 percent of the total weight. 

Because one project and its students comprise such a large proportion of the weighted sample, two additional types of analyses were conducted.  The first examined whether this one sampled project—labeled Project 69—is an outlier or unusual in any way.  Data from a grantee survey sample on project-level characteristics found that Project 69 was similar to other projects in this stratum on a broad range of characteristics.  Similarly, data from student surveys and NSC and FSA records indicated that the students from Project 69 did not have unusual characteristics.

The second type of analysis reduced the relative weight given to Project 69—in some cases by dropping the project entirely—when estimating impacts.  The impact estimates were sensitive to substantial changes in weighting.  Because Project 69 had below average impacts, reducing its weight relative to other projects resulted in larger overall impacts for most outcomes compared with the findings from the main impact analysis, which weighted all sample members according to their actual selection probabilities.  Reducing the weight of Project 69 also underestimates the standard errors associated with the impact estimates.  With larger impact estimates and reduced standard errors, many impact estimates become statistically significant when the sample weight for Project 69 is substantially reduced.  When the standard errors more accurately reflect the precision of the sample design, many of these impact estimates are not statistically significant.  Furthermore, impact estimates become smaller and fewer are significant when other projects with relatively large weights are dropped from the analysis along with Project 69.  This illustrates an important consideration—the potential for influencing the findings through post hoc adjustments that deviate from the chosen design.

Another important consideration in interpreting results from analyses that omit Project 69 or otherwise change the weights of projects in any substantial way is that the resulting sample no longer represents the actual universe of Upward Bound projects.  In particular, the sample does not appropriately represent the most common stratum of Upward Bound projects.  Thus, such analyses do not answer the evaluation’s research questions about the impacts of the national Upward Bound program.  Moreover, the estimates from such analyses do not generalize to urban projects, large projects, or any other well-defined subset of projects for which the findings might have policy implications.

In contrast, the findings from the main impact analyses, which include all projects weighted based on their selection probabilities, are intended to generalize to the national Upward Bound program.  In assessing the implications of those findings, however, a statistical consideration is that as a consequence of selecting a single project from a large stratum—the stratum represented by Project 69—the estimates and inferences for that stratum and, therefore, the universe of projects will generally not be as robust as the estimates and inferences that would be obtained with an alternative design with much less variable project selection probabilities and with several projects selected from the large stratum.  The lower robustness of the chosen sample design and the results from the extensive sensitivity analyses can be taken into account in determining the implications of the main findings.

**Some Control Group Members Received Upward Bound or Upward Bound Math-Science Services and Some Treatment Group Members Did Not**

After random assignment, project directors at some projects allowed a few control group members to receive regular Upward Bound.  In addition, some control group members reported participating in the Upward Bound Math-Science program, which was not part of the random assignment evaluation.  In total, 13.5 percent of the control group participated in either regular Upward Bound or Upward Bound Math-Science.  In contrast, about 15 percent of students assigned to the treatment group did not participate in either program.

To account for this cross-over and no-show issue, the impact analysis estimated models of the effects of actual Upward Bound participation (as opposed to the opportunity to participate) on student outcomes, where Upward Bound participation is defined as receiving either regular Upward Bound or Upward Bound Math-Science.  The impacts of actually participating are generally larger than the impacts of having the opportunity to participate.  Most impact estimates are not statistically significant.

## discussion

The national evaluation of Upward Bound began in 1991.  Study enrollment occurred from 1992 to 1994 and follow-up surveys and administrative records tracked student progress through high school and seven to nine years after expected high school graduation.  Several previous reports document the operations of Upward Bound projects (Moore 1997), the short-term impacts on high school experiences (Myers and Schirm 1999), and final impacts on high school outcomes and short-term impacts on postsecondary experiences (Myers et al. 2004).  These previous reports, together with this final impact report on postsecondary outcomes, highlight several important considerations for understanding the evaluation study results.

**Upward Bound Attracts Mostly Students Who Are Sufficiently Able and Motivated to Pursue Postsecondary Education**

Nationwide, among all students in eighth grade in 1988, approximately 76 percent reported enrollment in postsecondary education within about eight years after high school (Ingels et al. 2002).  Among disadvantaged students, the reported national postsecondary enrollment rate was much lower—less than 60 percent for students who were in the lowest quartile of socioeconomic status or whose parents did not attend college.  In comparison, survey data from this evaluation reveal that 81 percent of Upward Bound applicants assigned to the control group enrolled in postsecondary education within seven to nine years after high school.  Thus, even without the opportunity to participate in Upward Bound, Upward Bound applicants—who are disadvantaged students—reported attending postsecondary institutions at a rate higher than the national average, and at a much higher rate than the average disadvantaged student.  These results suggest a limited opportunity for Upward Bound to dramatically increase enrollment rates.

**Participants in Upward Bound Receive an Intensive Set of Precollege Services and Have Positive Educational Outcomes**

Offering Upward Bound increases both the percentage of youth receiving services and the intensity of services received.  More than 80 percent of the treatment group members received Upward Bound services; almost 90 percent of treatment group members received a high school diploma; and roughly 80 percent enrolled in some type of postsecondary program, with over half attending a four-year college or university.

**Upward Bound Operates in an Environment in Which Other Precollege Services Are Also Available to Students**

Many Upward Bound programs operate in environments in which the type of students who are eligible and apply for Upward Bound may also participate in other college programs.  Survey data indicate that nearly half of control group members reported participating in some kind of supplemental services in high school.  In particular, 11 percent of control group members reported participating in an Upward Bound Math-Science program (not part of the random assignment evaluation) and nine 9 reported participating in Talent Search, a less intensive precollege program for disadvantaged high school students.  The availability of other precollege services might limit the effects of Upward Bound if such services affect educational outcomes.

## looking ahead

National statistics continue to show substantial disparities in the postsecondary enrollment and completion rates between more and less advantaged groups.  Longitudinal data from the eighth-grade cohort from the National Education Longitudinal Study show that only 52 percent of students in the lowest socioeconomic status (SES) quartile reported some postsecondary attendance by eight years after expected high school education, compared with 76 percent for the cohort as a whole (Ingels et al. 2002).  The percentage obtaining a B.A. degree eight years after scheduled high school graduation was 7 percent for the lowest SES quartile, compared with 26 percent for the cohort.  

The national Upward Bound evaluation highlights the challenges faced by programs aiming to reduce these disparities in postsecondary enrollment and completion between more and less advantaged groups of high school students.  Disadvantaged students who seek out intensive programs like Upward Bound represent a strongly motivated segment of the target population.  As a result, they are able to access needed services, graduate from high school, enroll in postsecondary institutions, and complete postsecondary education at rates consistent with the youth population as a whole.  To address the long-standing inequality in postsecondary enrollment and completion rates may therefore require program strategies that reach and impact those less motivated students who have not accessed the Upward Bound or other precollege services available in their communities.

# I.  Introduction

## A.Context and Purpose of Upward Bound

Enrolling in college and completing a degree are significant milestones for many young adults.  Moreover, the importance of completing a college degree for success in the labor market is well-documented.  For full-time workers ages 25 to 34, median earnings are 64 percent higher for men with bachelor’s degrees than for men with high school diplomas, and 68 percent higher for women with bachelor’s degrees than for women with high school diplomas (U.S. Department of Education 2007).  These differences increase with age, as earnings rise more rapidly with work experience among college-educated workers than among workers without a college degree.  In addition to the economic benefits of a college education, individuals who complete college tend to have a greater civic orientation and are more likely to vote and assume leadership roles in their communities (Astin 1993; Bowen and Bok 1998). 

Although completion of a college education is important from the perspective of both the individual and society, many potential college students lack the skills or resources needed to enter college or complete a college degree.  Often, those who face the greatest barriers to pursuing a college education are young adults from low-income families and families in which neither parent completed college; many of these students are members of racial and ethnic minorities (U.S. Department of Education 2001).  These students may also face barriers to enrolling in college and completing a degree due to limited high school academic preparation, which is frequently linked to family socioeconomic status, race, and ethnicity (see, for example, Coleman et al. 1966; Jencks et al. 1972; Mosteller and Moynihan 1972; Congressional Budget Office 1987; Jacobson et al. 2001; St. John et al. 2002; Avery and Kane 2004). 

Low-income students are less likely than middle- and upper-income students to earn high school diplomas and attend and complete college.  In 2004, only 50 percent of high school graduates who came from families in the bottom 20 percent of family incomes enrolled in college immediately after completing high school.  In contrast, 64 percent of students from middle-income families and 80 percent of students from families in the top 20 percent of family incomes enrolled immediately after high school (U.S. Department of Education 2006).  These differential outcomes are attributable to several factors present in low-income families and their communities.  

Low-income students are concentrated in communities with high-poverty high schools.  Studies have shown that schools with a high percentage of low-income children have lower quality teachers, which is associated with lower achievement on state assessment tests and tests of college readiness, controlling for high school course-taking (Peske and Haycock 2006).  Despite progress in closing the gap in achievement test scores between disadvantaged and more advantaged students, large differences remain.  For example, on achievement tests reported by the National Assessment of Educational Progress, about 37 percent of white eighth-grade students were classified as proficient in mathematics in 2005, as compared with 8 percent of African American eighth-graders (U.S. Department of Education 2005a, Table A-9).  A similar gap is seen in reading, with about 37 percent of white eighth-graders and 11 percent of African American eighth-graders classified as proficient (U.S. Department of Education 2005b, Table A-9).  Furthermore, data from the National Education Longitudinal Study (1988–94) suggest that only half of low-income high school graduates are academically prepared to attend four-year colleges or universities (U.S. Department of Education 1997, Table 15).

Students at high-poverty high schools also experience higher dropout rates than similar students at middle- and upper-income high schools, reducing the likelihood of high school completion and postsecondary enrollment.  Attending a high-poverty high school is particularly deleterious for high school completion among African Americans (Swanson 2004; Balfanz and Legters 2004).

Most low-income students do not have a parent who has a college degree, which presents an informational barrier for students in terms of taking the courses in high school that would prepare them for college, accessing financial aid, and navigating the college admissions process.  The importance of parents as models and information sources is suggested by the finding that 82 percent of students whose parents had a bachelor’s degree enrolled in college immediately after high school, compared with 54 percent of students whose parents had only a high school diploma (U.S. Department of Education 2001).  Other studies have shown that while low-income students may aspire to higher education, they find the college admissions process (test-taking, financial aid application, and college application) difficult to navigate, and they are more likely to report that a lack of resources and someone to advise them are barriers to enrolling in college (St. John et al. 2002; Avery and Kane 2004).

Finally, low-income students do not take full advantage of financial aid programs.  While differences in high school curricula, parents’ education, and test scores partly explain the gap in enrollment rates, differences in financial resources available to students continue to play an important role (Kane 1999).  A study by the American Council on Education showed that 20 to 30 percent of college-going students with a family income below $40,000 (thus likely to be eligible for federal financial aid) did not apply in 2000 (King 2004).

Since the War on Poverty started in the 1960s, many federal, state, community, and private initiatives have been undertaken to alleviate some of the barriers to attending college and completing a degree faced by low-income, first-generation college students and minority students (see, for example, Adelman 2000; Swail and Perna 2000; James, Jurich, and Estes 2001).  Programs range from Equity 2000, Advancement Via Individual Determination (AVID), and Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP)—which are integrated with the regular high school or middle school experiences—to programs that more often supplement school experiences, such as Upward Bound, Talent Search, and I Have a Dream.  In the years after this evaluation’s sample attended Upward Bound, additional programs began providing supplemental services, including College Opportunity and Career Help (COACH), Roads to Success, and the College Advising Corps.  Because few of these programs have been subjected to rigorous evaluation, the effectiveness of these approaches is generally unknown; however, evaluations of postsecondary transition programs such as Talent Search and COACH indicate that providing low-income students with information and inspiration at the right time can significantly increase college enrollment rates (Constantine et al. 2006; Avery and Kane 2004).

Upward Bound is one of the largest and longest-running federal precollege programs for economically disadvantaged students.  Within Upward Bound, three programs operate: regular Upward Bound, Veterans Upward Bound, and Upward Bound Math-Science.  In 2006, 761 regular Upward Bound projects served 56,430 students, 39 Veterans Upward Bound projects served 4,909 participants, and 125 Upward Bound Math-Science projects served 6,707 students.  This report pertains to the regular Upward Bound program.  

Upward Bound was “designed to generate skills and motivation necessary for success in education beyond high school among young people from low-income backgrounds and inadequate secondary school preparation” (Public Law 90-222, Dec. 23, 1967).  Federal policy requires that two-thirds of students in each Upward Bound project must be both low-income (family income under 150 percent of the poverty line) and potential first-generation college students (from families in which neither parent holds a bachelor’s degree).  The remaining one-third of students must qualify either as low-income or potential first-generation college students.  In FY 2006, with federal funds of more than $267 million, the average cost per student for the regular Upward Bound program totaled about $4,725 per year and covered a variety of services.  This may be equivalent to about half of the amount of money spent by an average school district on a student per year, based on national per pupil expenditures of $8,468 in 2002–03 (U.S. Department of Education 2006, Table 166).

Upward Bound is an intensive program: during the academic year, participants engage in activities on a regular basis, often weekly; during the summer, participants attend a full-day academic program that generally lasts for about six weeks.  Students typically enter Upward Bound early in high school and are encouraged to participate through the summer following graduation.  In the evaluation sample, 15 percent applied for Upward Bound before ninth grade, 39 percent applied during ninth grade or the summer before tenth grade, 35 percent applied during tenth grade or the summer before eleventh grade, and the remaining 11 percent applied later.  Most Upward Bound projects emphasize the academic preparation needed for attending and completing college.  They offer:

· ***Academic courses and activities.*** A major focus of program activities is to help students acquire academic proficiency in challenging college preparatory courses.  Projects often require students to take Upward Bound courses during both the summer and the school year.  In addition, almost all projects provide students with tutoring for high school course work and help participants prepare for college entrance examinations.

· ***Nonacademic services.***Projects complement their academic offerings with a wide range of activities.  For example, students may attend plays, visit museums, tour college campuses, and learn about and apply for financial aid.

In December 1991, the U.S. Department of Education (ED) initiated a rigorous, longitudinal evaluation conducted by Mathematica Policy Research, Inc. (MPR), and its subcontractors, Educational Testing Service, Westat, Decision Information Resources, and Branch Associates, to determine whether Upward Bound enables students to perform better in high school and subsequently to enter and complete college at higher rates than without the program.  This report presents findings about the effect of the regular Upward Bound program on eligible applicants’ postsecondary experiences approximately seven to nine years after the applicants’ scheduled high school graduation.  

Earlier reports from the evaluation have documented the operations of Upward Bound projects (Moore 1997), the characteristics of students served by Upward Bound projects (Myers and Schirm 1997), the program’s short-term effects on eligible applicants’ high school experiences (Myers and Schirm 1999), and the program’s ultimate effects on eligible applicants’ high school experiences and short-term effects on applicants’ postsecondary experiences (Myers et al. 2004).  Myers et al. (2004) used data collected approximately three years after most individuals in the evaluation were scheduled to graduate from high school.  The present report updates those findings on Upward Bound’s effects on postsecondary enrollment by using data collected approximately seven to nine years after most individuals in the study were scheduled to graduate from high school.  This report also presents the first estimates of Upward Bound’s effects on postsecondary completion.

The remainder of this chapter summarizes the context for interpreting program effects and the previous findings from the evaluation, and is followed by a description of the evaluation design in Chapter II.  Chapter III presents estimates of Upward Bound’s effects on postsecondary enrollment and completion for the average eligible applicant, followed by estimates for certain subgroups in Chapter IV.  Chapter V presents estimates indicating how the effects of Upward Bound vary with the amount of participation in the program.  Finally, several appendixes present details concerning the evaluation and the data analyses. 

## B.Context for Interpreting Program Effects

Figure I.1 outlines a conceptual framework that indicates how Upward Bound projects structure recruitment and services based upon participant characteristics and policy requirements, which contribute to effects on intermediate and long-term student outcomes.  The population under study (furthest-left box in Figure I.1) are high school students who are low-income (family income under 150 percent of the poverty line as defined by the Census Bureau) or potential first-generation college students (neither parent with a bachelor’s degree) and therefore eligible for Upward Bound.  Within this population, students were randomly assigned to a treatment group or a control group; all students were allowed to utilize other supplemental services available in their schools and communities.

Upward Bound is designed to help disadvantaged students complete high school and to enter and succeed in postsecondary education.  Outcomes related to both of these goals are presented in Figure I.1.  Previous reports have examined student outcomes through high school and focused on measures associated with progress on the path toward college completion.  Both Upward Bound and intermediate high school outcomes affect the long-term outcomes related to college, such as high school completion, postsecondary application, enrollment, persistence, and completion; postsecondary enrollment and completion are the focus of this report.

To interpret the estimated effects of Upward Bound presented in this report, it is important to understand that these effects (1) are indicative of the “value-added” of Upward Bound relative to other programs in which students participate, (2) are based on students who chose to apply to Upward Bound, (3) are based on students who participated in Upward Bound in the mid-1990s, and (4) are based on students who chose to participate in Upward Bound for various lengths of time.

### 1.Value-added of Upward Bound

This report provides estimates of the value-added of regular Upward Bound above and beyond other available precollege programs and services.  Because eligible applicants were randomly assigned to the treatment and control groups, and because—with very few exceptions—only treatment group members were offered the opportunity to participate in regular Upward Bound, the differences between the two groups provide valid estimates of the value of that opportunity relative to the opportunities for participation in other programs. 

Many Upward Bound projects operate in service-rich environments, and the kinds of students who are eligible for Upward Bound may also participate in other precollege programs (see Appendix H for other precollege supplemental service programs attended by sample members during the same time period, as reported in our surveys).  Many of the treatment and control group members participated in precollege services other than regular Upward Bound. In fact, it is critical to the scientific validity of the study that treatment and control group members had the same opportunity to pursue other services as the typical eligible applicant to regular Upward Bound.

After applying to Upward Bound, nearly half of the control group members reported participating in some kind of supplemental services in high school (Table I.1).  The most common type of supplemental service received by control group members was instructional and tutoring sessions (33 percent), followed closely by programs with a math or science emphasis (31 percent).  We also found that 11 percent of control group members reported participating in an Upward Bound Math-Science program and 9 percent reported participating in Talent Search. Control group members were more likely to participate in supplemental services during the academic year than the summer (46 percent versus 20 percent).  

Like students in the control group, some treatment group members received supplemental services beyond those offered by Upward Bound.  According to the treatment group members themselves, 25 percent attended instructional and tutoring sessions outside of Upward Bound; 20 percent participated in a program with a math or science emphasis; 11 percent participated in Upward Bound Math-Science; and 7 percent participated in Talent Search.  Due to participation in Upward Bound, treatment group members received substantially more precollege services than did control group members.  More than four out of five treatment group members received services from regular Upward Bound, while less than half of control group members received precollege services from other programs (Table I.2).  

While about half of treatment group members would have received precollege services if they had not been given the opportunity to participate in regular Upward Bound (based on the rate for control group members), most would not have participated in programs that are as intensive as regular Upward Bound.  For example, while Upward Bound spends more than $4,700 per participant annually, Talent Search and GEAR-UP, two other large Department of Education precollege programs aimed at low-income and disadvantaged youths, spend about $400 and $300 per participant annually.  

With much higher spending, Upward Bound is able to offer many academic and nonacademic activities during both the academic year and the summer.  Specifically, we found 

that on average, Upward Bound participants attended about 265 academic sessions over their entire Upward Bound career (see Myers et. al. 2004, Table II.2); 174 of the sessions occurred during the summer program and 91 sessions occurred during the academic year.  Sessions in English, math, and science courses constituted the bulk of participants’ Upward Bound academic course work.  In addition to the academic course work completed through the Upward Bound program, participants engaged in a variety of nonacademic activities.  The most common activities attended, as reported by Upward Bound projects, focused on counseling, followed by skills development and college preparation courses.  On average, participants attended 212 activity sessions while in Upward Bound, with nonacademic activities split nearly equally between the summer and the academic year.  Not only are the treatment group members more likely to receive services, it appears that these services are generally much more intensive (with the exception of control group members who participated in programs such as Upward Bound Math-Science).  

### 2.Upward Bound Applicants

A comparison of overall postsecondary enrollment rates of Upward Bound applicants with national enrollment rates indicates that Upward Bound attracts students who are much more likely to enroll in postsecondary education than are similarly disadvantaged students; in fact, Upward Bound applicants are just as likely to enroll in postsecondary education as the average eighth-grader in the United States, regardless of socioeconomic status.  Data from the National Education Longitudinal Study’s 1988 sample (NELS:88) reveal that of all students in eighth grade in 1988, 76 percent had attended at least one postsecondary institution by 2000, that is, by about eight years after scheduled graduation from high school (Ingels et al. 2002).  Of students in the lowest quartile of socioeconomic status, 52 percent had enrolled in some postsecondary education; of students whose parents did not attend college, 56 percent had enrolled in some postsecondary education.  These national rates of postsecondary enrollment for disadvantaged students are approximately 25 percentage points lower than the 81 percent of control group members in the Upward Bound evaluation who attended a postsecondary institution within approximately seven to nine years after scheduled high school graduation (based on fifth follow-up survey responses).

### 3.Effects for Students Who Participated During the Mid-1990s

Recognizing that the treatment and control group members applied to participate in regular Upward Bound during the mid-1990s, this report characterizes the effects of Upward Bound as it operated at that time.  To the extent that the types of services provided by regular Upward Bound and other precollege programs and the types of students served by these programs have not changed much since the mid-1990s, this report may provide a reasonable assessment of the effects of Upward Bound as it operates today.  However, there have been changes in the universe of Upward Bound grantees during  the past 15 years, both in terms of new grantees and grantees that no longer operate Upward Bound programs.  There have also been changes in student targeting.  These program changes suggest that findings based on students who participated in the mid-1990s may not be directly applicable to current participants.

Considering positive subgroup impacts in previous reports, the Department of Education set aside $18 million for a Participant Expansion Initiative as a way to encourage Upward Bound projects to serve more higher-risk students.  The 2003 initiative allowed grantees to apply for additional funds if they served at least one target school in which at least 50 percent of the students were eligible for a free lunch and wanted to increase the number of participants from such schools who had the “greatest need” for project services.  Three indicators of “greatest need” could be used: a participant’s (1) not meeting state standards for eighth-grade proficiency in reading or language arts, (2) not meeting state standards for eighth-grade proficiency in math, or (3) having a GPA of 2.5 or lower in the most recent academic year.  Projects could receive $100,000 for serving 20 additional qualified students, $75,000 for 15 students, and $50,000 for 10 students.  Priority was given to the approximately 180 projects that had received funds under a prior expansion initiative that began in 2000, but many more projects received the expansion funds as a result of this initiative—219 in 2003, 259 in 2004, and 256 in 2005.

In another effort to improve overall program effectiveness, the Office of Postsecondary Education issued rules in 2006 that could substantially modify the composition of new students admitted to Upward Bound.  In making the grant awards, priority would be given to applicants who agreed to select all new participants from otherwise eligible students who are in or about to enter ninth grade, and reserve at least 30 percent of these first-time participant slots for students at “high risk of academic failure.”  Students would qualify as high risk if they (1) do not meet the proficient level on state assessments for eighth-grade reading or language arts, (2) do not meet the proficient level on state assessments for eighth-grade math, or (3) have a GPA of 2.5 or lower in the most recent academic year.  

Although the evaluation findings indicated that Upward Bound may have the greatest impact on students with lower educational expectations, it was decided that expectations would be difficult to assess accurately for determining eligibility among entering ninth-grade students and that low grades and performance on state assessments may serve as a proxy for lower educational expectations.  Requiring students to begin during or before ninth grade is intended to make it possible for all newly admitted students to participate for four full years.  Previous findings from a nonexperimental analysis conducted as part of this evaluation suggested that the longer students stay in the program, the larger are some impacts.  Along with the revised targeting initiatives, a new evaluation to examine the effects of these initiatives was started in 2007, though the evaluation was terminated in 2008.  

### 4.Variation in Exposure to Upward Bound

In considering the effects of Upward Bound, it is important to recognize that students’ Upward Bound experiences vary in the length of participation and, as a result, in the amount of services received.  The experience of treatment group members can be summarized from previous evaluation reports (see Myers et. al. 2004); no additional data have been collected on the Upward Bound experiences of treatment group members since the previous report because that report was based on data collected after sample members would have finished high school.  Approximately 84 percent of treatment group members (unweighted) received some Upward Bound services and can be classified as “participants” (Table I.3).  The participation rate was similar across most subgroups, with two exceptions: the subgroups defined by Upward Bound eligibility (low-income and/or first-generation) and the subgroups defined by ninth-grade grade point average.  The lack of a statistical difference between the subgroups defined by likelihood of being selected to participate in Upward Bound suggests that expanding the applicant pool for the evaluation did not cause the no-show rate to be artificially higher.

Conditional on any participation, the median length of time participants remained in Upward Bound was 20 months, with 64 percent participating for at least one year, 35 percent participating for at least two years, and 14 percent participating for at least three years.  About 39 percent completed the program, which we defined as still being in the program in the spring of their senior year in high school.  The services participants received in Upward Bound included academic and nonacademic sessions during both the school year and the summer.  Although Upward Bound participants typically attended a substantial number of academic sessions, there was variability in the number of sessions attended, just as there was in the duration of participation.  For example, while one-quarter of participants attended 104 or fewer Upward 

Bound sessions, another quarter attended 405 or more sessions.  Like the academic sessions, we found variability in the number of nonacademic activities attended by Upward Bound participants:  about 25 percent of the participants attended 74 or fewer sessions and about 25 percent attended 293 or more sessions.  As might be expected, the amount of services received by students is directly related to the duration of participation.  The average student who completed the program participated for about 50 percent longer than the average participant and received about 50 percent more services.  Chapter V of this report provides estimates of the effects of longer participation and program completion.

## C.Previous Findings

Earlier evaluation reports have presented findings based on data collected through the first three rounds of surveys and transcript collection.  Those reports reveal how Upward Bound affected eligible applicants while still in high school—in terms of both precollege services received and academic achievement—and how Upward Bound affected their postsecondary experiences approximately three years after completing high school, including whether they had enrolled in a postsecondary institution, their highest level of postsecondary attendance (four-year, two-year, or vocational), and the number of postsecondary credits earned.  The key findings were as follows:

· ***For the average eligible applicant, Upward Bound had little effect on most key high school outcomes, including credits, grades, and graduation.***Myers et al. (2004) found Upward Bound had no effect on total credits and a small effect on credits earned in high school math.  The program increased the number of math credits earned by 0.2 credits; that is, about one in five students completed an additional high school math course because of exposure to Upward Bound.  Upward Bound had no effect on credits earned in science, English, social studies, or foreign language courses.  Also, the program had no effect on honors and Advanced Placement credits, grades earned in high school, or high school graduation.

· ***For the average eligible applicant, Upward Bound had few short-run effects on postsecondary outcomes, but may have increased enrollment at four-year colleges and universities.***  Upward Bound had little effect on enrollment and credits earned at two-year or vocational postsecondary institutions and on the receipt of college financial aid.  Myers et al. (2004) found some inconclusive evidence that Upward Bound may have increased the percentage of treatment group members attending a four-year college or university.  When all postsecondary enrollment information reported by sample members was included, the estimated effect was 6 percentage points and was statistically significant; however, when unverified enrollment information was excluded (as described in Appendix H of the report), the effect fell to 5 percentage points and was not statistically significant.

· ***Upward Bound had positive effects for eligible applicants with lower educational expectations.***  For eligible applicants with lower educational expectations—those who did not expect to earn a bachelor’s degree when they applied to Upward Bound—Upward Bound increased Advanced Placement or honors credits as well as credits earned in core academic subjects in high school.  It also had short-term effects on some postsecondary outcomes for this group, such as the likelihood of enrolling in a four-year college or university, total postsecondary credits, and credits earned at four-year colleges and universities.  For eligible applicants who expected to obtain a bachelor’s degree or more, Upward Bound had little short-term effect on any of these outcomes. 

# II.  Research Design and analytic issues

## A.Research Design

### 1.Selection of Upward Bound Projects and Random Assignment

At its inception, the national evaluation of Upward Bound was unusual within education evaluation studies because of two important design elements:  (1) a nationally representative sample of Upward Bound projects and (2) random assignment of eligible applicants to Upward Bound and a control group.  These two design elements provide for both external validity and internal validity—that is, the ability to generalize the results to the population of regular Upward Bound projects and to make inferences about the causal effects of Upward Bound on eligible applicants’ outcomes.  Although the use of random assignment has become more common in recent years, it is still rare for evaluations to include a nationally representative sample of program sites.

**a.Selection of Upward Bound Projects**

For the evaluation, we randomly selected 70 Upward Bound projects representative of all 395 regular Upward Bound projects operating in the 50 states and the District of Columbia that were hosted by a postsecondary institution, had operated for at least three years as of October 1992, and were not dedicated to serving only students with physical disabilities.  Many different designs for selecting the sample of projects were considered.  Several designs had relatively modest stratification and modest variability in sampling rates for different types of projects defined by potentially policy-relevant characteristics, including project size and type of host institution.  Such a design would have supported precise estimates for many key subgroups while sacrificing very little precision in the estimates for the full sample.  Other designs that were considered were much more highly stratified and had highly variable sampling rates to yield substantial overrepresentation—relative to the full universe of projects—of some types of projects with less common characteristics (e.g., serving predominately Native American students) and substantial underrepresentation of some types of projects with more common characteristics.  Objectives of such a design included assuring the Upward Bound community that some relatively rare types of projects were adequately represented and, if policy interest later emerged, allowing more precise estimates of Upward Bound’s effects on particular applicant and project subgroups, even though estimates for other subgroups and the full sample would be less precise as a result of variability in project selection probabilities and, therefore, sample weights.  The design that was chosen sought to balance the competing needs of the evaluation.  Under the chosen design, project selection probabilities varied substantially across strata that were defined by location (urban or rural), type and control of the host institution (two- or four-year, public or private), size, and racial or ethnic composition.

Of the 70 projects originally selected, 11 could not participate or had to be excluded for various reasons.  For example, some did not plan to recruit new students for the 1992–93 school year, some had too few applicants to accommodate random assignment, and some did not have their Upward Bound grants renewed.  We replaced eight of these 11 projects with similar, randomly selected projects, arriving at a total sample of 67 projects.  See Appendix A for a detailed description of the sample selection and weighting procedures.  

**b.Random Assignment of Eligible Applicants to Upward Bound and a Control Group**

During the 1992–93 and 1993–94 school years, we randomly assigned eligible applicants from each project to either a treatment group, which was invited to participate in Upward Bound, or a control group, which was not invited to participate.  Eligible applicants were defined as students who the projects had recruited and who met both the federal eligibility criteria (low-income or potential first-generation college student status) and any project-specific criteria for participation.  All of the projects received more applications than they had openings, and all served the same number of students they would have normally served under their usual selection procedures.  

We implemented random assignment over 14 months so that projects could use their standard recruiting procedures and enroll students in accordance with their usual enrollment schedules.  Nationwide, the random assignment process resulted in a treatment group of about 1,500 students and a control group of about 1,300 students for subsequent impact analyses.  Myers et al. (1993) presented a detailed description of the random assignment procedures.  

To accommodate project wishes concerning the composition of the participants served by the program, such as sex, racial, or ethnic group balance, we used stratified random sampling to select the treatment and control groups (and weighted sample members appropriately to account for different random assignment probabilities).   Nonetheless, random assignment may have led some Upward Bound projects to serve students they would not normally have served.  Before random assignment, we asked project directors to rate each applicant as either most likely, somewhat likely, or least likely to have been selected under normal selection procedures; in this report, we assessed whether the effects of Upward Bound vary across these three groups.  Appendix I provides little evidence that the effects on postsecondary enrollment and completion varied across groups; however, there is evidence of significant positive effects on attendance and completion at vocational schools for the somewhat likely to be selected group.  

With random assignment, the only systematic difference between the treatment and control groups in the present evaluation is that treatment group members were offered the opportunity to participate in Upward Bound; otherwise, the two groups are statistically equivalent (Myers and Schirm 1997).  On important demographic variables such as gender, race or ethnicity, and Upward Bound eligibility status, differences between treatment and control proportions are small.  Statistically significant differences between the two groups exist within two categories of background variables: a student’s own educational expectations and the educational expectations held by his or her mother.  Even in a randomized experiment, there will generally be a few differences between the groups purely due to chance; using a 10 percent level for statistical significance, we would expect to find significant differences for 10 percent of the comparisons.  To adjust for the small differences between the treatment and control groups, we computed regression-adjusted estimates of program effects in which we statistically controlled for these and other background characteristics.  We describe our estimation methods in more detail below.

### 2.Outcome Measures

The outcomes for which impact estimates are presented in this report can be grouped into three areas: postsecondary enrollment, financial aid application and receipt, and postsecondary completion.

**Postsecondary Enrollment.**We estimate the impacts of Upward Bound on enrollment at any type of postsecondary educational institution, along with the highest level of postsecondary institution attended, and the selectivity of four-year colleges and universities attended.  Highest level of enrollment was defined as *four-year* for sample members who attended a public or private, nonprofit, four-year college or university; *two-year* for sample members who attended a public or private, nonprofit, two-year college, but not a four-year college or university; and *vocational* for sample members who attended a for-profit institution but no two- or four-year institution. 

Selectivity of four-year colleges and universities attended was measured by using school ratings from *Barron’s Profiles of American Colleges* (2003).  If a school was rated as “most competitive,” “highly competitive,” or “very competitive,” we classified the school as *more selective*.  If a school was rated as “competitive,” “less competitive,” “noncompetitive,” “special,” or unrated, or was excluded from *Barron’s*, we classified the school as *less selective*.  According to the classification system, *more selective* colleges and universities generally accept less than 75 percent of applicants, and students at *more selective* institutions were generally in the top half of their high school class.  *Less selective* postsecondary institutions generally admit more than 75 percent of their applicants.  The values of the four-year college or university selectivity outcome variables are set to 0  for sample members who did not attend a four-year college or university, that is, such sample members are classified the same as sample members who attended less selective four-year institutions.

**Financial Aid Application and Receipt.**  We also estimate the impacts of Upward Bound on the likelihood of a sample member applying for and receiving any financial aid, as well as on the likelihood of receiving a Pell grant. 

**Postsecondary Completion.  W**e estimate the impacts of Upward Bound on completion of any postsecondary credential, as well as on the highest postsecondary credential (degree, certificate, or license) earned.  Highest credential was defined as a *four-year degree* for sample members who earned a bachelor’s degree or higher; a *two-year degree* for sample members who earned an associate degree but not a bachelor’s degree; and a *certificate or license* for sample members who earned a postsecondary certificate or license but no higher degree. 

To measure these postsecondary outcomes, we use data from the fifth follow-up survey, as well as from administrative records.  We describe these different data sources below, along with their strengths and weaknesses in providing valid information for measuring these outcomes of interest. 

### 3.Data Sources

The analyses described in this report are based on information provided by treatment and control group members during the follow-up interviews and by the postsecondary institutions that they reported attending, as well as by two administrative data sources. 

**Surveys and Transcripts.**Almost all sample members completed a baseline questionnaire when they applied to Upward Bound (see Table II.1).  We then conducted follow-up surveys in 1994–95, 1996–97, 1998–99, 2001–02, and 2003-04 and achieved high response rates for all surveys.  The estimates in this report rely substantially on data from the fifth follow-up survey, conducted in 2003–04, which yielded a 74 percent response rate; if sample members are weighted to account for unequal selection probabilities (see Appendix A), the response rate is 72 percent.  This survey focused on obtaining information from sample members about their postsecondary educational attainment.  

The response rate for the treatment group was 4 percentage points higher than for the control group.  Given this small difference in response rates, the differences between marginal treatment respondents—treatment group members who would not have responded if they had been assigned to the control group—and other treatment respondents would have to be very large to have any perceptible effect on the impact estimates.  Furthermore, we use the extensive baseline data available to incorporate an adjustment for nonresponse into the sample weights.

After each follow-up survey, we also collected transcripts from high schools and postsecondary educational institutions attended by sample members.  Following the fifth follow-up survey, we requested postsecondary transcripts for 2,079 sample members and received transcripts for 1,772 of them (85 percent); Appendix B describes the data collection procedures.

**Administrative Data.**  Survey respondents may differ from nonrespondents in ways that may affect outcomes (see Appendix Table A.3).  While we attempt to account for these differences in observables in our estimation methods and weights, there may be differences in unobservables that remain.  Therefore, we collected data from other sources that allow us to help mitigate any differences due to survey nonresponse.  These two administrative data sources, the National Student Clearinghouse (NSC) and the federal Student Financial Aid (FSA) records, use completely different reporting systems.  The NSC collects enrollment and degree information from the majority of colleges and universities in the United States, enabling it to provide verification of these activities by institution and semester.  The FSA records are based on the Free Application for Federal Student Aid (FAFSA) filled out by most college aspirants, and include information on aid application and Pell receipt.  Refer to Appendix B for full descriptions of these administrative data sources.

### 4.Construction of the Outcome Measures

The data available from the follow-up surveys, the NSC, and the FSA records are used to construct various outcome measures in three different ways: using only the fifth follow-up survey, using only administrative records, and blending data from the surveys and the administrative sources in different combinations.  As data from the NSC were available for a period of time after the fifth follow-up survey was completed, we construct two versions of an outcome when data from the NSC records are used: one using all the information available from the NSC records (NSC Full), and the other using information available from the NSC by the end of calendar year 2004, when the fifth follow-up survey was complete (NSC Truncated).  A more detailed discussion about the construction of various outcome measures using these different data sources is provided in Appendix B.  In the main body of this report, we focus on one measure of enrollment (5B) and one measure of completion (7B); these measures use the fifth follow-up survey, full NSC data, and FSA records in combination and, when the data are not definitive, treat a sample member as a nonenrollee or noncompleter only if there is also no application for financial aid.  In the appendixes, we present estimates for many measures to assess the robustness of our findings. 

We use the different data sources because they have different relative strengths and weaknesses.  In conducting the impact analysis for this report, our basic principle has been to utilize the maximum amount of information that is available on the sample members.  While the follow-up surveys provide data on a broad range of outcomes, we face the problem of not having

data for survey nonrespondents, and the nonrespondents might be systematically different from respondents, potentially leading to nonresponse bias in our estimates.  The NSC and the FSA data are two convenient resources to mitigate this problem, as we can get information on both survey respondents and nonrespondents from these administrative records.  

However, these administrative sources have their own limitations.  The NSC does not cover the entire universe of postsecondary schools, and does not cover all member schools for the entire relevant time period.  Nationally, current rates of coverage are 87 percent for students attending a two-year institution and 90 percent for students attending a four-year institution.  The coverage rates were lower in earlier years (the NSC data go back to 1993–94); in terms of total U.S. college enrollment, coverage by the NSC data rose from 57 percent in 1997 to 88 percent in 2002, with small increases in subsequent years.  Thus, the NSC might be missing data for a sample member who attended and potentially completed his or her education at a postsecondary institution because the institution was not covered by the NSC during the relevant years.  FSA records provide data on all sample members; however, they do not have information on postsecondary completion, and they provide information on enrollment for only some students (those who receive a Pell grant).

## B.analytic issues

### 1.Estimation of Program Impacts

Some policymakers and program operators may be most interested in learning about the effects of offering Upward Bound services to eligible applicants.  Others may be more interested in learning about the effects of actually participating in Upward Bound.  The evaluation literature refers to the first impact as the “intended to treat effect” (ITT) and the second impact as the “complier average causal effect” (CACE) (see Angrist, Imbens, and Rubin 1996).  Our estimates of the ITT are based on a comparison of students randomly assigned to the treatment group with students randomly assigned to the control group.  Our estimates of the CACE are based on a comparison of Upward Bound participants to nonparticipants, using the outcome of random assignment, treatment or control status, as an instrumental variable to predict program participation.  

Both sets of effects are estimated from the same analysis sample.  The difference between the CACE and ITT effects can be illustrated by thinking about two hypothetical programs that are equally effective for participants, but are not equally effective in their ability to induce students to participate.  Suppose two programs (A and B) provide the same set of program services for participants.  Program A is very effective at inducing participation and has a 100 percent participation rate, whereas only 50 percent of students participate in Program B.  The CACE for these two programs would be the same, but the ITT effect for Program A would be twice as large as that for Program B, reflecting the fact that it has a larger effect on the average applicant, because it encourages more of them to actually participate in program services.

**a.Estimating the Impacts of the Opportunity to Participate in Upward Bound**

The ITT effect addresses the policy question, “What is the effect of the program on its intended beneficiaries—in this case, eligible applicants to the Upward Bound program?”  The ITT effect is the more comprehensive measure of the program’s effect because it captures both the effect of the offer of program services and the effect of the services on those who receive them.  Because eligible applicants were randomly chosen to be given the opportunity to participate in Upward Bound, we can obtain an unbiased estimate of this effect by computing the difference between the average outcome for the treatment group (those offered the opportunity to participate in Upward Bound) and the average outcome for the control group (those not offered the opportunity).  In practice, we computed program effects by estimating a regression model with the outcome of interest as the dependent variable.  Further details on our estimation procedures are presented later in this section.

**b.Estimating the Impacts of Upward Bound Participation**

The CACE answers a slightly different policy question:  “What is the effect of the program on those most likely to be affected—that is, eligible applicants who actually participated in the program?”  Because individuals who choose to participate in Upward Bound are self-selected from those who are given the opportunity and may have different characteristics from those who are given the opportunity and choose not to participate, estimating the CACE requires assumptions about the relationships between student characteristics, participation, and outcomes.  Appendix E presents more details on those assumptions and estimation of the CACE.  The estimates of the CACE, which capture the effects of actual participation in the program, account for both treatment group no-shows and control group crossovers. 

ITT estimates are more interesting in evaluations of existing programs, while CACE estimates are more interesting in the context of demonstrations designed to test the effectiveness of a particular bundle of services.  We present both types of estimates in the text to facilitate easy comparison.  Overall, the two sets of estimates tell similar stories, with nearly identical patterns of significant effects.  Because the participation rate of the treatment group in Upward Bound or Upward Bound Math Science was 85 percent, and about 14 percent of the control group members received services from either program, the CACE estimates are approximately 41 percent higher than the ITT estimates.  

**c.Regression-adjusted Estimates**

To compute the effects of Upward Bound, we use a regression-adjusted approach in which we estimate a statistical model that predicts the outcome of interest as a function of treatment status and a number of baseline characteristics.  We identify a core set of baseline characteristics as control variables based on the criteria that they have predictive power in regression models for key outcomes.  The same set of variables is used to estimate impacts for all outcome measures.  Inclusion of the baseline characteristics as control variables in the regression model allows us to (1) increase the precision with which we estimated the effects and (2) adjust for chance differences between the treatment and control groups on these characteristics. Appendix E provides more details about our estimation procedures. 

Table II.2 lists the control variables that are included in the regression specifications, along with the reference groups.  In addition to the variables listed in the table, our regression model included an intercept and a binary variable indicating whether the sample member was in the treatment group.  Because all of the outcome measures presented in this report are binary 

variables, we estimated logistic regression models.  Our method for estimating the standard errors of impact estimates takes into account the stratification of projects and the clustering of students within projects.

### 2.Subgroup Analysis

To assess whether Upward Bound is more effective for some eligible applicants than for others, we estimated impacts for subgroups based on a set of applicant characteristics.  These subgroups are listed in Table II.3.

Chapter IV devotes special attention to characteristics of eligible applicants that may be of particular interest in identifying students likely to benefit from Upward Bound because the characteristics are indicative of students’ risk of poor academic outcomes.  In the main text, we discuss the effects on the subgroups defined by the first four characteristics in Table II.3; the 

results for the remaining subgroups are presented in Appendix I.  In general, subgroups are based on characteristics of applicants at the time they applied for Upward Bound.  However, three subgroup characteristics are defined based on ninth-grade transcripts: level of ninth-grade mathematics class, ninth-grade GPA, and the academic performance index.  For students who applied to Upward Bound in eighth and ninth grades, our measures of these characteristics could be affected by Upward Bound participation if Upward Bound has an immediate effect on high school course taking and grades.

The grade at which a student applies to participate in Upward Bound may give us some insight into the motivation and attitudes of a student; those who give consideration to the program and apply earlier may be more motivated to pursue higher education.  Similarly, to measure educational expectations directly, our baseline survey before random assignment asked students to indicate the highest level of education they expected to achieve.  For the evaluation, we classified sample members as having *lower educational expectations* if they did not expect to obtain at least a bachelor’s degree.  Previous reports have found relatively large effects of Upward Bound for eligible applicants with *lower educational expectations* as defined by this measure (Myers and Schirm 1999; Myers et al. 2004).  However, when recruiting for Upward Bound, projects may find it difficult to target students based on this measure of educational expectations. If applicants are aware that their likelihood of admission to Upward Bound is influenced by their response to a question asking about educational expectations, some applicants may modify their responses to increase their chances of acceptance. 

We therefore also consider measures of educational performance which are less likely to be affected by such influences.  The first of these is the level of mathematics class taken in ninth grade, and in particular whether the sample member took a course *below algebra* or *algebra or above*.  Approximately two-thirds of sample members took algebra or above and one-third took a course below algebra.  This measure may be interpreted as a more readily available assessment of educational expectations, as students who do not take algebra or above in ninth grade would generally not be “on track” to attend a four-year college or university.  The level of mathematics class taken in ninth grade reflects, however, not only expectations but also prior academic performance.  Among control group members, 60 percent of those who took algebra or above in ninth grade attended a four-year college or university within four to six years after high school graduation, compared with just 30 percent of those who took a course below algebra.  Our second measure of academic performance early in high school is grade point average (GPA) in ninth grade, separating sample members into *higher GPA* (greater than 2.5) and *lower GPA* (less than 2.5) groups, with approximately half of the sample members in each group.  We sought measures that would be highly correlated with educational expectations, and therefore potentially useful for targeting; that said, several of the variables were, *a priori*, potentially interesting for purposes of targeting even if they are not highly correlated with expectations.

In addition to the subgroups based on characteristics of Upward Bound applicants, we also analyze program effects by subgroups based on project-level characteristics.  They are listed in Table II.4.  Data on several of the project level characteristics are derived from the Survey of Upward Bound Grantees, conducted in 1993–94 as part of this evaluation (for details, see 

Fasciano and Jacobson, 1997).  We present the estimated effects for the subgroups defined above in Appendix I. 

### 3.Use of Weights to Account for Sample Design and Survey Nonresponse

The sample members included in our analyses were weighted using standard weighting procedures to ensure that they represent all eligible applicants to Upward Bound nationwide.  The weights account for three probabilities: (1) the probability that the sample member’s Upward Bound project was selected for the study, (2) the probability that the sample member was selected to be given the opportunity to participate in Upward Bound, and (3) the probability that the sample member responded to the fifth follow-up survey.  Appendix A describes the weighting procedures in more detail.

Use of the weights is necessary for four reasons.  First, without the weights, the projects selected for the evaluation will not be representative of the universe of Upward Bound projects that existed during the 1992–93 academic year.  Due to the highly variable selection probabilities, the unweighted sample of projects does not resemble the full population of projects from which the sample was selected; therefore, unweighted estimates do not measure the impacts of the national Upward Bound program and, more generally, have no readily apparent interpretation.  Second, the weights are needed for generalizing the findings to the national population of students targeted by the Upward Bound program.  Third, due to unequal selection probabilities of projects, we need to use the weights to appropriately account for the imprecision in impact estimates that results from the sampling design.  Without the weights, we would obtain not only biased point estimates of impacts but also underestimated standard errors.  Fourth, the use of weights reduces the possibility of missing data bias in the estimated impacts.

Given that the evaluation is based on a sample of eligible Upward Bound applicants, the estimated program effects presented in this report contain sampling error.  To determine whether the estimated program effects can be attributed to the true effects of Upward Bound rather than to statistical chance, we tested whether the estimates are statistically significant—significantly different from 0—at three levels commonly used in conducting such tests.  More specifically, we conducted two-tailed tests of the null hypothesis that “Upward Bound had no effect” against the alternative that “Upward Bound had an effect” at the 0.10 level, the 0.05 level, and the 0.01 level.  Throughout the report, the term “significant” is used to denote statistical significance, not substantive significance.  In the chapters that follow, we note that Upward Bound had an effect on a particular outcome only if the estimated effect on the outcome measure is statistically significant.  If the estimated effect of Upward Bound on an outcome variable is not statistically significant, the true effect of Upward Bound cannot necessarily be assumed to be 0.  

The power analysis conducted for this evaluation in Myers et. al. (1993) suggests that for any binary outcome measure, the study sample design allows for the detection of an impact of 0.10 to 0.15 standard deviations, depending on the effective sample size available for the analysis.  This means that the sample design allows for the detection of an impact of 4 to 6 percentage points for postsecondary enrollment (assuming a 75 percent enrollment rate) and 5 to 7 percentage points for completion of a postsecondary credential (assuming a 30 percent completion rate).  Under the assumptions of these power calculations, the study would identify an impact that is significant at the 5 percent level 80 percent of the time; however, we may find smaller impacts that are significant, as well as larger impacts that are not.  The education evaluation literature (see, for example, Cohen 1988; Lipsey and Wilson, 1993) suggests that impacts of these sizes can be considered small, and the Department of Education’s What Works Clearinghouse requires an effect size larger than 0.25 for an impact to be regarded as substantively important.  Therefore, even if different from 0, the true effect is likely to be small if the estimated effect is not statistically significant, because the sample was designed to detect program effects of a small size.  

### 4.Potential Threats to the Study Design

Evaluations based on random assignment face potential threats that can bias the resulting effect estimates.  For this study, the bias is likely to be small, as the random assignment protocol was implemented correctly (see Myers et. al. 1993).  Analysis of baseline data for the treatment and control groups found small differences in some baseline characteristics between the two groups that arose by chance (see Myers and Schirm 1997).

A potential threat to the validity of the study is crossover or control group contamination.  After randomly assigning students to the treatment and control groups, we discovered that project directors at some of the 67 projects conducting random assignment had allowed 29 control group members (2.2 percent) to participate in regular Upward Bound.  In the follow-up student surveys, an alternative source of information on Upward Bound participation, 14 additional control group members (1.1 percent) reported that they had participated in Upward Bound projects, including projects not in the evaluation and thus not conducting random assignment.  Upward Bound Math-Science participation was reported by 148 control group members, including 11 who had previously reported that they had participated in regular Upward Bound, and 137 (10.4 percent) who did not; see Appendix E for details on how we identify participation in Upward Bound and Upward Bound Math-Science programs.  All told, 43 control group members (3.3 percent) participated in regular Upward Bound, and 180 (13.6 percent) participated in either Upward Bound or Upward Bound Math Science.  In our ITT analysis, we maintained each control group member’s original status in order to preserve the comparability between the two groups due to random assignment; in our CACE analysis, all 180 were treated as noncompliers.

The control group may have been affected by their experience of applying for Upward Bound and being turned away in a way that is different under the evaluation than under normal circumstances.  Control group members could have been demoralized by not receiving the opportunity to enroll in the program; on the other hand, they may have been referred to other programs when they were notified that they would not be able to participate in Upward Bound.  In either case, the “intervention”—a demoralizing rejection or a helpful referral—is likely to be much less intense than the Upward Bound program itself.  Therefore, any potential bias in the effect estimates is likely to be small relative to the potential effects of the program.

# III.  The Effect of Upward Bound on Postsecondary outcomes

The ultimate question for this study is whether the national Upward Bound program helps disadvantaged high school students attend and graduate from institutions of higher education.  To address that question, we used information from the fifth follow-up survey, along with data from previous surveys, the National Student Clearinghouse (NSC), and Federal Student Aid (FSA) records.  Using these data, we compared the postsecondary experiences of eligible applicants offered the opportunity to participate in Upward Bound (the treatment group) with the experiences of eligible applicants not offered that opportunity (the control group), over a period extending to about seven to nine years after they were scheduled to graduate from high school.  These estimated effects are “intended to treat” (ITT) effects.  Some sample members end up not “complying” with their experimental assignment: some treatment group members do not actually participate in Upward Bound, and some control group members do participate in Upward Bound (or Upward Bound Math-Science).  Therefore, we also present estimates of the effects of actually participating in Upward Bound.  These are “complier average causal effect” (CACE) estimates.  

This chapter presents these two kinds of estimates to assess Upward Bound’s effects on three main sets of outcomes relating to postsecondary educational attainment:

1. ***Enrollment:***Whether the sample member enrolled at any type of postsecondary institution, along with the highest level of postsecondary educational institution attended and the selectivity of four-year colleges and universities attended 

2. ***Aid:***Whether the sample member applied for, and received, financial aid

3. ***Completion:***Whether the sample member completed any postsecondary credential (degree, certificate, or license), and the highest postsecondary credential earned.

Our analyses of impacts on these outcomes drew on multiple data sources.  We used information collected in follow-up surveys during 1998–99, 2001–02, and 2003–04, along with data from NSC and FSA administrative records.  There are many ways to combine data from these sources to measure postsecondary enrollment or completion, and many different assumptions that can be made about enrollment or completion status when the data do not provide definitive evidence.  Each approach reflects choices about how to resolve issues that arise from the nature of the data: whether to use data from previous surveys or only the latest,; which data sources to use, in which “order of preference” to combine the data when there are apparent contradictions; how to deal with cases for which the data do not provide definitive evidence of enrollment or completion; and whether to use all available data or only data covering a certain period.  A detailed discussion of these issues is presented in Appendix B.  

The impact estimates presented in detail in this chapter are for one measure for each outcome, but other estimates are presented in Appendix C.  The main estimates presented in this chapter are based on the fifth follow-up survey data along with NSC and FSA administrative records.  The alternative estimates emerged from sensitivity analyses examining different measures of the outcomes based on different methods and assumptions.  Additional sensitivity analyses pertaining to the weighting of the evaluation sample are presented in Appendix G.  Findings from all of the sensitivity analyses are summarized at the end of this chapter.

## A.The Effect of the Opportunity to Participate in Upward Bound (ITT)

As documented in detail in this section, Upward Bound had no detectable effect on overall postsecondary enrollment for the average eligible applicant, and did not affect the types of institutions eligible Upward Bound applicants attended, either in level or selectivity.  Similarly, Upward Bound did not have a detectable effect on financial aid application or receipt.  However, there is evidence that Upward Bound increased the completion of certificates or licenses.

Table III.1 presents the effect of the opportunity to participate in Upward Bound—the ITT impact—on a range of postsecondary outcomes.  Each row in the table presents the results for one outcome, with columns for the regression-adjusted treatment group mean, the control group mean, the impact of Upward Bound on the outcome, an indicator for statistical significance, and the p-value underlying the significance indicator.  Each analysis is conducted using post-stratification adjusted weights that account for sample selection probabilities and survey nonresponse.  In this chapter, we focus on one measure of enrollment (designated 5B in the appendixes) and one measure of completion (designated 7B in the appendixes). 

With Measure 5B for enrollment, we code a sample member as an enrollee if he or she is found to be an enrollee in the full NSC data *or* is a Pell recipient according to the FSA data *or* said in the survey that he or she was enrolled at some time.  The sample member is not an enrollee if he or she does not appear in the NSC data (and is therefore not an enrollee) *and* has not been a Pell recipient *and* said in the survey that he or she had never been enrolled.  This leaves uncoded the survey nonrespondents who are not in the NSC data and did not receive a Pell grant.  For them, we assume that they are not enrollees if they never applied for financial aid.  If they did apply for financial aid, we code their enrollment status as missing.  The sample members with missing enrollment status get dropped from the analyses of enrollment, and weights for the remaining sample members are adjusted to compensate, as described in Appendix A.  For measuring completion, the FSA data do not provide information on the actual receipt of degrees, certificates, or licenses.  Recognizing this limitation in constructing Measure 7B for completion, we code a sample member as a completer if he or she is a completer according to the full NSC data *or* said in the survey that he or she has completed a degree, certificate, or license.  The sample member is not a completer if he or she has no evidence of completion in the NSC data *and* said in the survey that he or she had not completed a degree, certificate or license.  This leaves uncoded the survey nonrespondents who have no evidence of completion in the NSC data.  For them, we assume that they are not completers if they never applied for financial aid.  If they did apply for financial aid, we code their completion status as missing.  The sample members with missing completion status get dropped from the analyses of completion, and weights for the remaining sample members are adjusted to compensate.  See Appendix A for more information on weighting, Appendix E for more information on estimation, Appendix B for a discussion of alternative measures of postsecondary enrollment and completion, Appendix C for analyses using the alternative measures to assess the robustness of our findings, and Appendix G for analyses pertaining to the weighting of the evaluation sample.

### 1.The Effect of Upward Bound on Postsecondary Enrollment and College Selectivity

Upward Bound attracts students who are much more likely to enroll in postsecondary education than are similarly disadvantaged students.  A comparison of overall postsecondary enrollment rates for Upward Bound applicants and students nationwide shows that Upward Bound applicants are just as likely to enroll in postsecondary education as the average eighth-grader in the United States, regardless of socioeconomic status.  Serving this applicant pool of motivated disadvantaged youths, Upward Bound did not have a detectable effect on enrollment in postsecondary institutions within approximately seven to nine years after scheduled high school graduation.  Approximately 81 percent of treatment group members and 79 percent of control group members attended some type of postsecondary institution (four-year, two-year, or vocational).  The difference, which is the effect of the opportunity to participate in Upward Bound, is not statistically significant (effect size = 4 percent).  Similarly, the opportunity to participate in Upward Bound did not significantly affect the type or selectivity of postsecondary institutions attended by eligible applicants.

### 2.The Effect of Upward Bound on Financial Aid Application and Pell Grant Receipt

Although academic improvement is the primary focus of most Upward Bound projects, addressing financial barriers is also important.  Nearly one-half of all Upward Bound project directors reported in a survey of grantees that inadequate financial resources are the most serious obstacle to college completion (Fasciano and Jacobson 1997), and many projects aimed to link students to financial aid resources.  Despite such efforts, Upward Bound did not have a detectable effect on the percentage of eligible Upward Bound applicants who at some point applied for financial aid or received a Pell Grant for postsecondary education.  

### 3.The Effect of Upward Bound on Postsecondary Completion

Upward Bound had no detectable effect on the likelihood of completing a postsecondary credential in the seven to nine years after high school (effect size = 5 percent).  The program did increase the percentage of sample members whose highest credential was a certificate or license, from 4 to 9 percent (effect size = 23 percent).

It is possible that sample members could obtain a credential later.  Approximately 7 percent of both treatment and control group members were still enrolled at the time of the survey.

## b.The Effect of Participation in Upward Bound (CACE)

For the average eligible applicant to Upward Bound, the pattern of statistically significant effects of actual participation (the CACE effects) is nearly identical to that for the effects of the opportunity to participate in Upward Bound (the ITT effects).  However, the CACE estimates in Table III.2 are generally larger than the ITT estimates in Table III.1, reflecting the fact that approximately 15 percent of treatment group members did not participate in Upward Bound or Upward Bound Math-Science and about 14 percent of control group members did participate in Upward Bound or Upward Bound Math-Science.

## C.Summary of Sensitivity Analyses

Several important choices were made in designing the evaluation.  One choice pertained to the length of the follow-up period.  Considering the objective of Upward Bound to prepare students for entry into and success in postsecondary education, the Department of Education specified a long follow-up period that allowed sample members to be observed for many years 

beyond expected high school graduation.  Although response rates to the evaluation’s follow-up surveys remained high, administrative data from the NSC and federal FSA files were obtained to assess and address the potential effects of survey nonresponse.  While nonresponse is one potential limitation of survey data, measurement and coverage error are concerns with administrative data.

Measuring postsecondary outcomes in different ways can shed light on how the relative strengths and weaknesses of the data sources affect the findings of the evaluation.  Therefore, we have conducted a set of sensitivity analyses to examine alternative ways of combining data from the available sources—surveys, NSC, and FSA—to measure postsecondary enrollment and completion.

Other important design choices pertained to the sample of projects that would be selected.  In addition to specifying that the sample had to be nationally representative, the Department of Education required that the sample have substantial overrepresentation of some less common, but key types of projects, including, for example, projects serving predominantly Native American students.  Attempting to balance the competing needs of the evaluation, the chosen design had much higher selection probabilities for these relatively rare projects than for more common types of projects.  This led to substantial undersampling and underrepresentation of the latter and to very unequal weighting of projects in the evaluation sample.

One implication of the sample design was that some of the most common types of Upward Bound projects had low selection probabilities and were substantially undersampled.  This is true of one set of projects in particular—projects that were medium-sized, located in an urban setting, hosted by a four-year public institution, and not serving a group of students that is predominantly Asian, Native American, or Latino.  This stratum of projects ends up accounting for about 26 percent of all eligible Upward Bound applicants nationwide.  The final sample selected for the impact evaluation included only one project out of 56 projects in this stratum.  The main impact analyses weight the sample accordingly, and the sample members from this one project account for approximately 26 percent of the total weight. 

Because one project and its students comprise such a large proportion of the weighted sample, two additional sets of analyses were conducted.  The first examined whether this one sampled project—labeled Project 69—is an outlier or unusual in any way.  The second reduced the relative weight given to Project 69 when estimating impacts.

### 1.Sensitivity Analyses Pertaining to the Measurement of Outcomes

There are many approaches to combining data from the available sources to measure postsecondary enrollment or completion, and many assumptions that can be made about enrollment or completion status when the data do not provide definitive evidence.  In Appendix B, we discuss these issues, and describe many different measures for the postsecondary outcomes examined in this report.  We present estimates for the measures in Appendix C.

Across a wide range of approaches to measuring postsecondary enrollment, the basic finding of no detectable impact holds up.  For 27 different measures of postsecondary enrollment, the distribution of estimated impacts ranges from –2.4 to 2.8, with a mean of 1.3.  None of the estimates are significant.  For the impact on attending a four-year institution, the 27 estimates range from 0 to 5.5, with a mean of 1.8, and one estimate is significant.

There are fewer ways to use the available data for measuring completion than there are for measuring enrollment, because FSA data do not provide any direct information about postsecondary completion.  We considered nine different measures of completion (see Appendix B).  Estimates for three of the nine indicate that the impact on completing any degree, certificate, or license is significant.  The nine estimates range from 0.5 to 13.0 (the second largest is 3.7), with a mean of 3.5.  When we examine the estimated impacts on receipt of a bachelor’s degree, we find that estimates for the nine measures range from –1.0 to 4.3, with a mean of 0.6, and one estimate is significant.  These estimates and the estimates pertaining to the receipt of certificates or licenses as the highest degree completed suggest that if Upward Bound affects postsecondary completion, it might do so by increasing the likelihood of earning a certificate or license, as suggested previously by Table III.1.

Exploring results obtained with different measures of the outcomes is important because judgments are required to decide which results are most likely to reflect the overall national effects of the program.  Although not all approaches are equally good, there are some reasonable alternatives to the methods underlying the main results reported here, and reporting the sensitivity of those findings to alternative approaches can help readers evaluate and interpret the results. 

### 2.Sensitivity Analyses Pertaining to Sample Weighting

The sample design adopted for the evaluation has important consequences for the weighting of sample data.  As noted above, the sample selection stratum composed of the most common type of project is represented by one project, Project 69, and that project had a selection probability much lower than the average selection probability.  It also had a large pool of eligible applicants.  As a consequence, the students in Project 69 represent 26 percent of eligible applicants nationwide.  In the main analyses, their data are weighted as such to account for the precision of the sample decision and measure the effect of the national Upward Bound program on the average eligible applicant.

Because Project 69 and its students comprise such a large proportion of the weighted sample, we performed analyses to address two broad questions (see Appendix G for details):

· Is this project an “outlier,” that is, unusual in some way?

· Does this project have a large amount of influence on our results?

By the available measures, Project 69 is not an outlier.  We find, for example, that it is similar in terms of project-level characteristics to the five projects from the same sample selection stratum that were selected for the grantee survey sample but not the impact study sample.  Further analyses find that there are some significant differences between treatment and control groups in Project 69, as there are for other projects.  Some such differences at the project level are expected to occur by chance.  We adjust for these differences using regression methods, and include in our models covariates measuring student baseline characteristics, as well as interactions that capture the effects of these covariates specific to Project 69.  

This examination of baseline differences between the treatment and control groups revealed that as shown in Appendix G, treatment group members in Project 69 were more likely to have applied to the program in ninth grade and less likely to have applied in tenth grade than control group members were.  These differences are not statistically significant.  Nevertheless, although the evaluation has had a very long follow-up period for observing postsecondary outcomes, the treatment group members in Project 69 had somewhat less time, on average, to begin and complete postsecondary education.  Therefore, to assess the potential effects of this, we conducted additional sensitivity analyses.  One analysis derived impacts using a regression model that controlled for not only grade at application (as in the main analysis) but also expected year of high school graduation, including indicators for different years and estimating effects specific to Project 69.  With one exception, the impacts obtained are numerically smaller than the impacts in Table III.1, and are not significant.  For another sensitivity analysis, we constructed a standardized outcome measure—postsecondary enrollment within six years of the year of expected high school graduation.  With this standardized measure, the impact on overall postsecondary enrollment is numerically larger than the impact from our main analysis—1.60 compared with 1.54.  With p-values of 0.60 and 0.58, respectively, neither impact is significant.    

In addition to the analyses of project-level characteristics from the grantee survey and baseline differences between the treatment and control groups, we examined the distribution across projects of average baseline characteristics of sample members, no-show and crossover rates by treatment and control group members, mean outcomes of control group members as of the fifth follow up, and impacts on postsecondary outcomes.  These analyses support the finding that Project 69 is not an outlier, although as would be expected for any project, it is sometimes in the lower or upper portion of the distribution and not right at the center.

We also conducted analyses to examine the influence of Project 69 on overall impacts and assess the robustness of the main findings.  Detailed results and a detailed description of the analyses can be found in Appendix G.

In one analysis, we determined how much larger Project 69’s impact on each outcome would have to be to make the overall impact of Upward Bound statistically significant when Project 69 gets its full weight and standard errors correctly reflect the precision of the sample design.  We find that Project 69’s impact would often have to move from the lower end of the distribution of project-level impacts to the upper end in order for the overall impact of Upward Bound to be significant.  This implies that Project 69 and the other 55 projects in Project 69’s selection stratum would have to have had larger impacts, on average, than all of the other Upward Bound projects.  Otherwise, the results would not be affected. 

In contrast to this analysis, most of the sensitivity analyses involved changing weights to reduce the relative weight given to Project 69’s sample members.  One such analysis adjusted the weights within each project to weight up to the number of funded slots rather than the number of applicants.  This addresses concerns about not only the effects of typical year-to-year fluctuations in the number of applicants, but also whether the implementation of random assignment might have inflated the number of applicants differentially across projects.  With this approach, Project 69 accounts for about 15 percent, rather than 26 percent, of the total weight—a much lower but still appropriately large fraction.  The estimated impacts are generally somewhat bigger than those obtained in our main analyses.  The pattern of significance levels is essentially the same.  (See the last column of Table III.3.)

Most of the other analyses that reduce the relative weight given to Project 69’s sample members changed weights even more substantially.  In these analyses, we examined impact estimates obtained by: combining project sampling strata in various ways; redistributing much of Project 69’s weight to various sets of projects that were most similar to Project 69 on a wide range of project- and student-level characteristics; and redistributing much of the weight of each Project 69 sample member to sample members in other projects with similar individual characteristics.  We also ran unweighted analyses, and derived weighted estimates without Project 69.  Results from a few of these analyses are shown in Table III.3.  To facilitate comparisons of results, the first column of Table III.3 repeats the estimates from the main analysis, which were presented in Table III.1 above.

Many of these sensitivity analyses that changed sample weights substantially produced larger impacts for most outcomes compared with the findings from the main impact analysis, 

which weighted all sample members according to their actual selection probabilities.  Many of the impacts from the analyses with large changes in weights are also significant.  This suggests that the results are sensitive to such large changes in the weight of Project 69.

Because Project 69 had below average impacts for most outcomes, reducing its weight relative to other projects results in larger overall impacts.  Reducing the weight of Project 69 also underestimates the standard errors associated with the impact estimates.  With larger impact estimates and reduced standard errors, many impact estimates become statistically significant when the sample weight for Project 69 is substantially reduced.  When the standard errors more accurately reflect the precision of the sample design, many of these impact estimates are not statistically significant.  Furthermore, as shown in Appendix G, they become smaller and fewer are significant when other projects with relatively large weights are dropped from the analysis along with Project 69.  This illustrates an important consideration—the potential for influencing the findings through post hoc adjustments that deviate from the chosen design.

Another important consideration in interpreting results from analyses that omit Project 69 or otherwise change the weights of projects in any substantial way is that the resulting sample no longer represents the actual universe of Upward Bound projects.  In particular, the sample does not appropriately represent the most common stratum of Upward Bound projects.  Thus, with the possible exception of the analysis that adjusts weights to the number of funded slots, such analyses do not answer the evaluation’s research questions about the impacts of the national Upward Bound program.  Moreover, the estimates from such analyses do not generalize to urban projects, large projects, or any other well-defined subset of projects for which the findings might have policy implications.

In contrast, the findings from the main impact analyses, which include all projects weighted based on their selection probabilities, are intended to generalize to the national Upward Bound program.  In assessing the implications of those findings, however, a statistical consideration is that as a consequence of selecting a single project from a large stratum—the stratum represented by Project 69—the estimates and inferences for that stratum and, therefore, the universe of projects will generally not be as robust as the estimates and inferences that would be obtained with an alternative design with much less variable project selection probabilities and with several projects selected from the large stratum.  The lower robustness of the chosen sample design and the results from the extensive sensitivity analyses can be taken into account in determining the implications of the main findings.

# IV.  The Effect of Upward Bound on Postsecondary outcomes for selected subgroups

Impact estimates for the full evaluation sample might conceal important differences in impacts across subgroups.  If Upward Bound has an effect on the average eligible applicant, it might be heavily concentrated in, or could be much larger for, certain subgroups of applicants.  Conversely, if Upward Bound has no effect on the average eligible applicant, the program might still have positive effects for some subgroups, counterbalanced by negative effects for other subgroups.  Thus, estimates of subgroup impacts can help policymakers identify the persons for whom a program is most effective and, in some cases, help policymakers better target a program or better tailor its services.  However, this targeting or tailoring must be done with caution, as it is unknown what the effects of Upward Bound would be if the composition of students enrolled in Upward Bound or the services provided were different.

The subgroups for which we present estimates here in the main text are based on the following characteristics:  (1) grade at application to Upward Bound; (2) educational expectations at the time of application to Upward Bound; (3) level of ninth-grade mathematics class; and (4) grade point average in ninth grade.  Appendix I also presents estimates of effects for subgroups defined by gender, race and ethnicity, an academic performance index, the Upward Bound eligibility criteria, and the likelihood of a student being served under normal selection procedures (rather than random assignment), as well as project and project host characteristics.  As before, when an impact estimate is statistically significant, that is, significantly different from 0, we highlight the estimate in the table with one or more asterisks (*).  In addition, when the difference in impact estimates between two subgroups is statistically significant, we highlight the estimate for one of the two subgroups with a pound sign (#).  Because a large number of outcomes and subgroups are considered, some effects will be significant by chance.  We thus focus on a few key subgroups, and interpret results with caution; however, we make no formal adjustments for multiple comparisons.  In light of the evolving approaches to program targeting that were discussed in Chapter I, it is probably most appropriate to interpret the subgroup analyses as exploratory rather than confirmatory, providing suggestive evidence for further investigation.

Smaller sample sizes make it more difficult to obtain precise impact estimates.  Therefore, for small subgroups, we are likely to detect significant impacts only if the effects are relatively large.  When we report that the effect for a small subgroup is not detectable (not statistically significant), it may mean that Upward Bound has no effect on that outcome for that subgroup or that there is an effect but that it was not large enough to be detected given the sample size.  The evaluation would have had to have substantially more projects and students to reliably detect small impacts for each subgroup and small differences between subgroups.

## A.The Effect of the Opportunity to Participate in Upward Bound for Selected Subgroups (ITT)

### 1.Grade at Application to Upward Bound

The grade at which a student applies to participate in Upward Bound may give us some insight into the motivation and attitudes of a student.  Those students who learn about the program and apply earlier may be more motivated to continue their pursuit of education beyond high school.  In some ways, this may be an indication of the educational expectations of a student, based on their actions rather than responses to questions.  Additionally, applying earlier provides the opportunity for greater exposure to Upward Bound, the benefits of which are discussed in more detail in the next chapter.

For the 54 percent of students who applied in ninth grade or earlier, Upward Bound had no detectable effects on postsecondary enrollment or completion, either overall or by level (Table IV.1).  Upward Bound increased the likelihood of receiving a Pell Grant, from 52 to 58 percent, for this subgroup of early applicants.

Relative to the effects for eligible applicants who applied early in high school, our findings suggest that Upward Bound had more detectable effects on postsecondary outcomes for the eligible applicants who applied in tenth grade or later.  For this subgroup of later applicants, there is a significant 3 percentage point increase in any postsecondary enrollment, from 82 to 85 percent.  There are also significant increases in financial aid applications (from 73 to 77 percent) and certificate or license completion (from 3 to 9 percent).  None of the differences between impacts for the early and late applicant groups is significant.

### 2.Applicants’ Educational Expectations at the Time of Application to Upward Bound

Earlier evaluation reports have found that Upward Bound has larger effects on the eligible applicants who did not expect to obtain a bachelor’s degree at the time they applied for Upward Bound than for those who did.  The findings in this report are less conclusive in terms of statistical significance of impacts—there are no statistically significant differences between the group impacts—but still suggest that the program may have some more favorable effects on eligible applicants with lower educational expectations (Table IV.2).

For the 20 percent of eligible applicants with lower educational expectations—those who did not expect to earn a bachelor’s degree—Upward Bound had a detectable effect on overall postsecondary enrollment, increasing it from 70 to 75 percent, similar to the 6 percentage point 

increase in enrollment in vocational institutions.  Unlike previous findings, treatment group members with lower expectations were not significantly more likely to attend a more selective four-year college or university than were control group members with lower expectations; although the estimated rate at which they attended highly selective institutions nearly doubled, this impact is not significant.  Findings for postsecondary completion differed from the pattern for enrollment: there were separate increases in four-year, two-year, and other degrees that were each not statistically significant, but the overall rate of completion for sample members with lower expectations rose by a significant 12 percentage points, and reached the rate for sample members with higher expectations. 

Our findings suggest that Upward Bound also had a few detectable effects on postsecondary outcomes for the 80 percent of eligible applicants with higher educational expectations—those who expected to earn a bachelor’s degree when they applied to Upward Bound.  For this subgroup, there is a significant increase in any postsecondary enrollment of 3 percentage points and a significant decrease in associate degrees of 4 percentage points.  There is also a significant increase in the likelihood of receiving a Pell Grant.

### 3.Level of Ninth-Grade Mathematics Class

Targeting Upward Bound services to students based on their self-expressed expectations at the time of application to Upward Bound presents potential challenges.  If Upward Bound projects began to screen applicants based on their educational expectations and it became common knowledge that applicants with lower expectations were more likely to be admitted than were applicants with higher expectations, applicants might face a strong incentive to understate their educational expectations in the application process.  

In light of these practical limitations, we also examine whether Upward Bound has effects on subgroups defined by an alternative measure of academic expectations—the level of ninth-grade mathematics class taken.  In particular, we define subgroups based on whether a sample member’s ninth-grade mathematics class was below the level of algebra or at the level of algebra or above.  Students who do not take algebra or a course above algebra in ninth grade would not generally be considered to be on a “college track,” so this can be used as an indication of whether the student expects and is preparing to attend a four-year college or university.  Among eligible Upward Bound applicants, approximately two-thirds took algebra or above in ninth grade and approximately one-third took a course below algebra.

We find that Upward Bound has significantly different effects on overall postsecondary enrollment and enrollment at two-year institutions for the two subgroups (Table IV.3).  Among sample members who took a course below algebra, treatment group members were nearly 7 percentage points more likely than control group members to enroll in any postsecondary institution.  For sample members who took algebra or higher, the impact on overall enrollment is an insignificant decrease of 1 percentage point.  The impact on attendance at two-year institutions is a significant 6 percentage point decrease for this subgroup, whereas it is an insignificant 2 percentage point increase for the sample members who took a course below algebra in ninth grade.  There were significant increases in the likelihood of receiving a Pell Grant for both subgroups.

### 4.Grade Point Average in Ninth Grade

This measure divides sample members into two roughly equal-sized subgroups: those with ninth-grade GPAs greater than or equal to 2.5 and those with ninth-grade GPAs less than 2.5.  The results reveal several significant differences in impacts between the subgroups, and indicate that Upward Bound has larger effects on enrollment and financial aid for eligible applicants with higher academic performance in ninth grade and larger effects on completion for eligible applicants with lower academic performance in ninth grade (Table IV.4).

For eligible applicants with higher ninth-grade GPAs (above 2.5), Upward Bound had significant effects on overall postsecondary attendance, attendance at two- and four-year institutions, and attendance at more selective four-year institutions.  In this higher GPA group, treatment group members were 3 percentage points more likely than control group members to enroll in any postsecondary institution.  There were also significant shifts by type of institution, with an increase in attendance at four-year institutions (8 percentage points)  and a decrease in attendance at two-year institutions (7 percentage points).  Treatment group members with higher GPAs were also 6 percentage points more likely to enroll in a more selective four-year institution.  Among the higher GPA group, treatment group members were more likely to apply for financial aid and receive a Pell Grant, both by 6 percentage points.  The impacts on 4-year college or university enrollment did not translate to higher rates of postsecondary completion, with overall completion falling insignificantly, driven by a significant four percentage point drop in associate degrees and no significant increase in the likelihood of earning a bachelor’s degree.

In contrast, Upward Bound had no detectable effects on postsecondary enrollment, overall and by type, on financial aid application and receipt, and on completion of bachelor’s or associate’s degrees, for eligible applicants with lower ninth-grade GPAs (below 2.5).  However, 

treatment group members with lower GPAs were 6 percentage points more likely than control group members with lower GPAs to earn a certificate or license but no higher degree, leading to a significant 7 percentage point increase in postsecondary completion overall. 

## B.The Effect of Participation in Upward Bound on Selected Subgroups (CACE)

In the subgroups defined by the grade at application and ninth-grade GPA, the pattern of statistically significant CACE effects is exactly the same as the pattern of statistically significant ITT effects.  Generally, most of the patterns of effects are similar for subgroups defined by ninth-grade math course.  The exception is that the CACE effect on the completion of a bachelor’s degree or higher is statistically significant for sample members who took a course lower than algebra in ninth grade.  For the subgroups of eligible applicants defined by self-reported educational expectations, the pattern of effects of participation in Upward Bound is again very similar to the pattern of effects of being offered the opportunity to participate in Upward Bound.

Although the point estimate of the CACE effect is generally larger than the ITT effect estimate, the standard error also increases, leading to similar results with regard to statistical significance.  The larger standard error (seen for many outcomes) of the CACE effect is attributable to the estimation of additional parameters, specifically those relating participation in Upward Bound to assignment to the treatment or control group.

# V.  The Effect of Additional Upward Bound Participation on Postsecondary Outcomes

It is possible that students would reap larger benefits from Upward Bound if they spent more time in the program.  The typical participant remains in Upward Bound for a little more than a year and a half, and most participants—61 percent—do not complete the program.  In this chapter, we describe the relationships between postsecondary outcomes and two measures of the extent to which students participated in Upward Bound—the duration of program participation and program completion.  In measuring these relationships, we attempt to estimate the effects of additional participation on postsecondary outcomes.  As we describe later, the estimated effects of additional participation may overstate the true effects due to selection bias.  With this caution in mind, our findings suggest that keeping students in Upward Bound for longer periods may substantially improve their postsecondary outcomes.

## A.Research Questions

To assess the potential for keeping students in Upward Bound for longer periods of time, we classified participants as *low-duration* (1 to 12 months of participation), *medium-duration* (13 to 24 months of participation) or *high-duration* (25 or more months of participation), and also as program *completers* (still participating in the spring of senior year) or *noncompleters*.  Among Upward Bound participants, 36 percent participate for 1 to 12 months, 29 percent participate for 13 to 24 months and 35 percent participate for 25 or more months; 39 percent complete the program (see Table V.1).  These figures suggest that there is considerable opportunity to increase the completion rate and the length of time that participants remain in the program.

To better understand the potential effects of Upward Bound retention on postsecondary outcomes, we address two research questions: 

1.For *low-duration* and *medium-duration* participants, how would their postsecondary outcomes change if they participated for an additional year?

2.For *noncompleters*, how would their postsecondary outcomes change if they completed Upward Bound?

## B.Research Methods

To answer these questions, we compared the outcomes for students with relatively low levels of participation to those for students with relatively high levels of participation.  A simple****comparison of students with different levels of Upward Bound participation, however, may fail to reveal the effects of additional participation.  While random assignment ensures that there will be no systematic differences between treatment and control students, it does not ensure there will be no differences between completers and noncompleters or among students who choose to participate in Upward Bound for different lengths of time.  The characteristics of students may influence how long they choose to participate in Upward Bound and whether they complete Upward Bound.  If so, the average characteristics of students will vary with the level of Upward Bound participation.  For example, we found that students who participate in Upward Bound for longer periods of time are more likely to be female and have higher educational expectations than students who participate for shorter periods; we find similar differences between completers and noncompleters.  Therefore, we cannot infer the effects of additional Upward Bound participation simply from differences in average outcomes between shorter- and longer-duration participants and between completers and noncompleters.

Because students with different levels of Upward Bound participation have different characteristics, we used statistical matching to select samples of shorter- and longer-duration participants with similar observed characteristics and samples of noncompleters and completers with similar characteristics.  Unlike random assignment, which ensures two statistically equivalent groups that are similar in terms of observed and unobserved characteristics, the matching procedures can only create groups that are similar in terms of observed characteristics.  

To measure the potential effects of participating in Upward Bound for an additional year, we matched low-duration participants to similar medium-duration participants, and we matched medium-duration participants to similar high-duration participants.  These matches allow us to simulate what the outcomes of low-duration participants would have been had they instead been medium-duration participants and what the outcomes of medium-duration participants would have been had they instead been high-duration participants.  Likewise, to measure the potential effects of program completion, we matched noncompleters to similar completers.  The matching process ensured that matched samples contained participants with similar observed characteristics; we used propensity score matching to select the matched samples (see Appendix F for additional details).

Once matching was completed, we estimated program effects using the same methods employed throughout this report.  For example, consider the comparison of noncompleters to similar completers.  The completer-noncompleter difference in mean outcomes provides an estimate of the potential benefit of additional Upward Bound participation.  Mean outcomes were regression-adjusted with the same logistic model used throughout the report to compute the impacts of the opportunity to participate in Upward Bound.  Although the matched samples were observationally similar, they may differ in ways not revealed by the data collected for the evaluation.  For example, our matched samples of completers and noncompleters may differ in their motivation to attend college.  Unobserved differences between matched samples may bias the estimates presented in the next section.

## C.The Effect of Additional Upward Bound Participation

Though subject to the caveats discussed in this chapter, findings from this analysis suggest that longer program participation or program completion may yield large, positive effects on several postsecondary outcomes.

### 1.Postsecondary Enrollment

Our findings suggest that Upward Bound would encourage postsecondary enrollment among more shorter-duration participants—students who participated for no more than 24 months—if it could keep them in the program longer.  An additional year of Upward Bound participation increases enrollment at four-year institutions, raising it by 9 percentage points (see Table V.2).  

Our findings also suggest that Upward Bound would have a larger effect on noncompleters if it retained them through high school graduation (see Table V.3).  If noncompleters remained in Upward Bound through program completion, we estimate that they would, on average, participate for an additional 18 months, as the average duration was just over 13 months for noncompleters and more than 31 months for completers.  The impact estimate of Upward Bound completion for any postsecondary enrollment is 19 percentage points, raising enrollment from 77 to 96 percent.  The effect of program completion on postsecondary enrollment operates primarily through increased enrollment in a four-year college or university, raising it by 27 percentage points.  These effects of program completion are much more pronounced than the effects from increased program duration.

We have also examined the effect of longer participation and completion on the selectivity of the four-year institutions attended by Upward Bound participants.  Longer Upward Bound participation increased the likelihood of attending a highly selective four-year college or university by 4 percentage points.  For noncompleters, Upward Bound completion would also raise the likelihood of attending a highly selective four-year institution as indicated by the 10 percentage point impact estimate. 

### 2.Financial Aid

Our estimates suggest that an additional year of Upward Bound participation would increase the likelihood of applying for financial aid by 6 percentage points.  The evidence also suggests that the impact is substantially larger for completers, as the estimated impacts of program completion are 21 percentage points for aid application and 20 points for Pell Grant receipt.

### 3.Postsecondary Completion

Longer participation in Upward Bound increases the likelihood of completing any postsecondary credentials, with a statistically significant 8 percentage point impact.  Parallel to the effect of longer participation on postsecondary enrollment, the positive effect of longer participation on overall postsecondary completion appears to be driven by an increase in the likelihood of completing a degree at a four-year institution (an increase of 5 percentage points).  Longer participation did not have a detectable effect on the likelihood of completing an associate degree or a certificate or license.  Our estimates show similar, though much larger, positive effects of Upward Bound completion on the likelihood of completing a postsecondary credential.  The impact estimate for any postsecondary credential is 21 percentage points, primarily attributable to an 18 percentage point increase in the likelihood of completing a bachelor’s degree.

## D.Interpretation of the Findings

The potential effects of retaining Upward Bound participants who would otherwise leave the program early may be large, with estimates suggesting that additional participation would raise the postsecondary enrollment and completion rates for shorter-duration participants and noncompleters.  However, we suspect the true effects of additional participation are probably smaller than the estimates presented in this chapter.  Although we used rigorous statistical methods in our analysis, we could not randomly assign students to different levels of Upward Bound participation.  Because participants decide how long to participate and whether to complete the program (unless they are expelled), the groups may differ along many dimensions, including unmeasured characteristics like the motivation to attend college.  If so, the estimated effects of additional participation, based on comparisons between these groups, may be partly attributable to differences in motivation that predated the Upward Bound participation of these students.

While this selection bias could be positive or negative, we suspect that it leads us to overestimate the effects of additional participation.  It seems likely that more motivated students participate longer in Upward Bound and complete Upward Bound at higher rates than less motivated students, leading to higher levels of motivation among longer-duration participants and completers.  If more motivated students tend to enroll in college at higher rates than less motivated students, longer-duration participants and completers would have higher college enrollment rates than shorter-duration participants and noncompleters.  While matching may reduce the motivational differences between the samples, we expect that remaining unobserved differences partially explain the large positive effects of additional participation and completion reported in this chapter.

# References

Adelman, Cliff.  “Participation in Outreach Programs Prior to High School Graduation:  Socioeconomic Status by Race.”  Paper presented at the ConnectED Conference, San Diego, CA, January 10, 2000.

Angrist, Joshua D., Guido W. Imbens, and Donald B. Rubin.  “Identification of Causal Effects Using Instrumental Variables.”  *Journal of the American Statistical Association,*vol. 91, no. 434, June 1996. 

Astin, Alexander W.  *What Matters in College?  Four Critical Years Revisited*.  San Francisco, CA:  Jossey-Bass, 1993.

Avery, Christopher, and Thomas J. Kane.  “Student Perceptions of College Opportunities:  The Boston COACH Program.”  In *College Choices*, edited by Caroline M. Hoxby.  Cambridge, MA:  National Bureau of Economic Research, 2004.

Balfanz, Robert, and Nettie Legters.  “Locating the Dropout Crisis:  Which High Schools Produce the Nation’s Dropouts?  Where Are They Located?  Who Attends Them?”  Baltimore, MD:  Johns Hopkins University, June 2004.

*Barron’s Profiles of American Colleges 2003*.  New York, NY:  Barron’s, 2002.

Bloom, Howard.  “Accounting for No-Shows in Experimental Evaluation Designs.”  *Evaluation Review*, vol. 8, 1984.

Bowen, William G., and Derek Bok.  *The Shape of the River:  Long-Term Consequences of Considering Race in College and University Admissions*.  Princeton, NJ:  Princeton University Press, 1998.

Brogan, D.  “Software for Sample Survey Data, Misuse of Standard Packages*.*”  In *Encyclopedia of**Biostatistics*, vol. 5, edited by P. Armitage and T. Colton.  New York, NY:  Wiley, 1998, pp. 4167-4174.

Cohen, Jacob.  *Statistical Power Analysis for the Behavioral Sciences.*  Second Edition.  Hillsdale, NJ:  Lawrence Erlbaum, 1988.

Coleman, James, E. Campbell, C. Hobson, J. McPartland, A. Mood, F. Weinfield, and R. York.  “Equality of Educational Opportunity.”  Washington, DC:  U.S. Department of Health, Education, and Welfare, 1966.  

Congressional Budget Office.  “Educational Achievement:  Explanations and Implications of Recent Trends.”  Washington, DC:  Congressional Budget Office, August 1987.

Constantine, Jill, Neil Seftor, Emily Sama Martin, Tim Silva, and David Myers.  “A Study of the Effect of the Talent Search Program on Secondary and Postsecondary Outcomes in Florida, Indiana, and Texas.”  Final report submitted to the U.S. Department of Education.  Princeton, NJ:  Mathematica Policy Research, Inc., June 2006.

Fasciano, Nancy, and Jonathan E. Jacobson.  “The National Evaluation of Upward Bound: Grantee Survey Report.”  Washington, DC:  U.S. Department of Education, 1997.

Ingels, Steven J., Thomas R. Curtin, Philip Kaufman, Martha Naomi Alt, and Xianglei Chen.  “Coming of Age in the 1990s:  The Eighth-Grade Class of 1988 12 Years Later.”  (NCES 2002-321).  Washington, DC:  U.S. Department of Education, National Center for Education Statistics, Office of Educational Research and Improvement, March 2002.

Jacobson, Jonathan, Cara Olsen, Jennifer King Rice, Stephen Sweetland, and John Ralph.  “Educational Achievement and Black-White Inequality.”  Washington, DC:  U.S. Department of Education, National Center for Education Statistics, Office of Educational Research and Improvement, 2001.

James, Donna Walker, Sonia Jurich, and Steve Estes.  “Raising Minority Academic Achievement:  A Compendium of Education Programs and Practices.”  Washington, DC:  American Youth Policy Forum, 2001.

Jencks, Christopher, Marshall Smith, Henry Acland, Mary Jo Bane, David Cohen, Herbert Gintis, Barbara Heyns, and Stephen Michelson.  *Inequality:  A Reassessment of the Effect of Family and Schooling in America.*New York, NY:  Basic Books, 1972.

Kane, Thomas J.  *The Price of Admission: Rethinking How Americans Pay for College*.  Washington, DC:  Brookings Institution Press, 1999.

King, Jennifer.  “Missed Opportunities:  Students Who Do Not Apply for Financial Aid.”  Washington, DC:  American Council on Education, 2004.

Lipsey, M.W., and D.B. Wilson.  “The Efficacy of Psychological, Education,, and Behavioral Treatment:  Confirmation from Meta-analysis.” *American Psychologist*, vol. 48, 1993, pp. 1181-1209.

Moore, Mary T.  “The National Evaluation of Upward Bound:  A 1990s View of Upward Bound Programs Offered, Students Served, and Operational Issues*.*”  Washington, DC:  U.S. Department of Education, 1997.

Mosteller, Frederick, and Daniel Moynihan*.  On Equality of Educational Opportunity.*  New York, NY:  Vintage Books, 1972.

Myers, David, Mary Moore, Allen Schirm, and Zev Waldman.  “The National Evaluation of Upward Bound:  Design Report.”  Report submitted to the U.S. Department of Education.  Washington, DC:  Mathematica Policy Research, Inc., November 1993. 

Myers, David, Robert Olsen, Neil Seftor, Julie Young, and Christina Tuttle.  “The Impacts of Regular Upward Bound:  Results from the Third Follow-Up Data Collection.”  Report submitted to the U.S. Department of Education.  Washington, DC:  Mathematica Policy Research, Inc., April 2004.

Myers, David, and Allen Schirm.  “The Impacts of Upward Bound: Final Report on Phase I of the National Evaluation.”  Report submitted to the U.S. Department of Education.  Washington, DC:  Mathematica Policy Research, Inc., April 1999.  Available at [http://www.ed.gov/ offices/OUS/PES/higher/upward.pdf].

Myers, David, and Allen Schirm.  “The Short-Term Impact of Upward Bound:  An Interim Report.”  Report submitted to the U.S. Department of Education.  Washington, DC:  Mathematica Policy Research, Inc., May 1997. 

Peske, Heather G., and Kati Haycock.  “Teaching Inequality:  How Poor and Minority Students are Shortchanged on Teacher Quality.”  Washington, DC:  The Education Trust, June 2006.

Schirm, Allen, Elizabeth Stuart, and Allison McKie.  “The Quantum Opportunity Program Demonstration:  Final Impacts.”  Washington, DC: Mathematica Policy Research, Inc., July 2006.

Schochet, Peter, John Burghardt, and Steven Glazerman.  “National Job Corps Study:  The Impacts of Job Corps on Participants’ Employment and Related Outcomes.”  Princeton, NJ:  Mathematica Policy Research, Inc., June 2001.

St. John, Edward P., Glenda Droogsma Musoba, Ada B. Simmons, and Choong-Geun Chung.  “Meeting the Access Challenge:  Indiana’s Twenty-First Century Scholar Program.”  Indianapolis, IN:  Lumina Foundation, 2002.

Swail, Scott, and Laura Perna.  “A View of the Landscape:  Results of the National Survey of Outreach Programs.”  *Outreach Program Handbook 2001.*New York, NY:  The College Board, 2000. 

Swanson, Christopher B.  “Who Graduates?  Who Doesn’t?  A Statistical Portrait of Public High School Graduation, Class of 2001.”  Washington, DC:  Urban Institute Education Policy Center, February 2004.

Trenholm, Christopher, Barbara Devaney, Ken Fortson, Lisa Quay, Justin Wheeler, and Melissa Clark.  “Impacts of Four Title V, Section 510 Abstinence Education Programs.”  Princeton, NJ:  Mathematica Policy Research, Inc., April 2007.

U.S. Department of Education, National Center for Education Statistics.  “Access to Postsecondary Education for the 1992 High School Graduates.”  NCES 98-105, by Lutz Berkner and Lisa Chavez (MPR Associates).  Project Officer: C. Dennis Carroll.  Washington, DC, 1997.  Available at [http://nces.ed.gov/pubs98/98105.pdf].

U.S. Department of Education, National Center for Education Statistics.  “The Condition of Education, 2001.”  Washington, DC:  U.S. Department of Education, 2001.

U.S. Department of Education, National Center for Education Statistics.  “The Condition of Education, 2006.”  NCES 2006-071, Washington, DC:  U.S. Department of Education, 2006.

U.S. Department of Education, National Center for Education Statistics.  “The Condition of Education,**2007.”  NCES 2007-064, Washington, DC:  U.S. Department of Education, 2007.

U.S. Department of Education, National Center for Education Statistics.  “Digest of Education Statistics, 2005.”  NCES 2006-030, Washington, DC:  U.S. Department of Education, 2006.

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.  “National Assessment of Educational Progress:  The Nation’s Report Card, Mathematics 2005.”  NCES 2006-453, Washington, DC:  U.S. Department of Education 2005a.  

U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics.  “National Assessment of Educational Progress:  The Nation’s Report Card, Reading 2005.”  NCES 2006-451, Washington, DC:  U.S. Department of Education 2005b.

Wilde, Elizabeth Ty, and Robinson Hollister.  “How Close Is Close Enough?  Evaluating Propensity Score Matching Using Data from a Class Size Reduction Experiment.”  *Journal of Policy Analysis and Management*, vol. 26, 2007, pp. 455-477.

� EMBED Excel.Sheet.8  ���

III

(continued)

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED Excel.Sheet.8  ���

� EMBED PowerPoint.Show.8  ���

� EMBED Excel.Sheet.8  ���

� Upward Bound includes three programs: regular Upward Bound, Veterans Upward Bound, and Upward Bound Math-Science.  The focus of this report is regular Upward Bound and we use the term “Upward Bound” to refer to that program.

� These rates compare favorably to other studies with similar populations and long follow-up periods.  For the Quantum Opportunity Program Demonstration Evaluation’s third telephone survey about nine to ten years after the demonstration started, the response rate was 76 percent overall, and the treatment group response rate exceeded the control group response rate by 3 percentage points (Schirm, Stuart, and McKie 2006).  For the National Job Corps Evaluation Study, the 48-month follow-up survey had an 80 percent overall response rate, and the treatment group response rate exceeded the control group response rate by about 4 percentage points (Schochet, Burghardt, and Glazerman 2001).  In the study of Impacts of Four Title V Section 510 Abstinence Education Programs, the third follow-up survey conducted about four to six years after random assignment had an overall response rate of 82 percent, and the treatment group response rate exceeded the control group response rate by 1 percentage point (Trenholm et. al. 2007).

� Looking at outcomes from the FSA data suggests that survey nonresponse bias may be small.  For the treatment group, 73.0 percent of survey respondents and 75.0 percent of survey nonrespondents applied for financial aid, while 70.6 and 68.5 percent of survey respondents and nonrespondents, respectively, applied for aid from the control group.  Similarly, the rates of Pell grant receipt were close for survey respondents and nonrespondents in both groups: 58.8 and 58.3 percent for the treatment group; 55.6 and 52.7 percent for the control group.

� Of the 1,656 institutions reported by sample members in the survey, 1,465 could be matched to the Integrated Postsecondary Education Data System (IPEDS), and 925 of those (63 percent) appear in the NSC’s list of participating institutions.  The vast majority of the remaining schools were vocational institutions, along with some two-year schools.  A higher proportion (just under 80 percent) of students reported attending a school that was also in the NSC list, with the difference in these rates likely due to attendance at multiple institutions.  Based on the differences in coverage by sector, sample members were less likely to be confirmed as enrollees by the NSC data if they attended only vocational institutions.

� For the CACE analysis presented in this report, we regard participation in regular Upward Bound or Upward Bound Math Science by control group members as forms of crossover.

� A CACE estimate is roughly 1 / (0.85-0.14) = 1.408 times the ITT estimate.

� Table II.2 refers to “Project 69.”  As discussed below, sample members from this project comprise a large proportion of the weighted evaluation sample.  Our regression models include an indicator variable and interaction variables for Project 69 to capture effects of the other control variables that are specific to this project.

� For the subgroups based on the academic performance index, the bottom 20 percent of ninth-grade academic achievement was labeled in previous reports (Myers and Schirm 1999, Myers et al. 2004) as “higher academic risk,” and the top 80 percent of ninth-grade academic achievement was “lower academic risk.”  We instead use the terms “lower performing” and “higher performing,” respectively, to make the labels more intuitive, simplify discussion, and facilitate comparison with the lower and higher expectations groups.  Construction of the index is described in Myers et al. (2004).   

� Due to item nonresponse, some subgroups are not defined for all sample members, resulting in a grand total that is smaller than the full sample size of 2,844.

� For outcomes that were measured using data from sources in addition to the fifth follow-up survey, the weights reflected the probability of having the data needed to measure the outcomes, which is not just the survey response probability.

� To ensure inclusion in the sample of substantial numbers of some of the less common types of projects, as discussed earlier in this chapter, the sample included only one of the 56 projects that were medium-sized, urban, hosted by four-year public universities, and not serving a group of students that was predominantly Asian, Native American, or Latino (the most common type of project).  Because this one project’s probability of selection was much lower than the average selection probability, the students in this one project represent 26 percent of the eligible applicants nationwide and are weighted accordingly.  This was a consequence of the study's requirement to over-sample relatively uncommon types of projects.  The weights that are used in our main analyses account for the study design.  Chapter III presents a summary of the sensitivity analyses pertaining to sample weighting, with full details in Appendix G.

� The counts and percentages in this paragraph are unweighted.

� See Appendix H for a list of other supplemental service programs attended by sample members.

� As discussed in this chapter and elsewhere in the report, each data source has different relative strengths and weaknesses.  A specific concern with using NSC data to measure enrollment pertains to its coverage of postsecondary institutions.  Specifically, when students in the evaluation first began enrolling in postsecondary institutions, the percentage of institutions that participated in the NSC was lower than it was in later years.  Also, coverage rates vary across different types of institutions.  In light of concerns about coverage of the NSC, the sensitivity analyses include enrollment measures that are based on survey and FSA data only—relevant NSC data are ignored by these measures.  As shown in the main analyses and presented in Table III.1 (and Appendix C), the impacts on overall postsecondary enrollment and enrollment at four-year institutions according to a measure (5B) using all three data sources—survey, FSA, and NSC—are 1.54 and 1.29, respectively.  As shown in the sensitivity analyses and presented in Appendix C, the corresponding impacts according to a measure (6B) using only survey and FSA data are 1.38 and 1.29.  None of these estimates is statistically significant.

� The impacts on overall enrollment and completion, for example, are 1.04 and 1.57 (p-values = 0.73 and 0.51), while the impacts from the main analysis are 1.54 and 2.26.  The one exception to the pattern is the impact on the receipt of certificates and licenses, which is 4.66 and significant (p-value = 0.09), compared with the impact of 4.54 estimated in the main analysis.  

� A standardized measure has potentially important limitations.  It requires data pertaining to the timing of events, which are likely subject to greater recall error than data about whether an event occurred.  It also ignores relevant data, specifically, the available longer-run data about postsecondary outcomes that occur after the chosen cut-off date.  In light of these limitations, the main analysis examines outcomes that are observed at any time during the period for which data are available from the surveys, the NSC, or the FSA records.

� In conducting the subgroup analyses, we assessed the sensitivity of the findings to alternative ways of measuring the outcomes, and present the results in Appendix I.  We did not, however, conduct sensitivity analyses pertaining to sample weighting.

� Looking across the various measures of enrollment (see Appendix I), the largest estimated effect on four-year college or university enrollment by sample members with lower educational expectations is 21.0 percentage points, based on the measure that uses only fifth follow-up survey data.  This is similar to the 20 percentage point survey-based effect reported in Myers et al. (2004).  

� For students who applied to Upward Bound in eighth or ninth grade, this measure, like ninth-grade GPA, is based on ninth-grade transcripts, and could be affected by participation in Upward Bound if the program has an immediate effect on high school courses taken and achievement in those courses.

� Among eligible Upward Bound applicants, 76 percent of those who took a course below algebra in ninth grade reported that they expected to earn a bachelor’s degree or above, as compared with 87 percent of those who took algebra or above.  Although the level of ninth-grade mathematics class is far from a perfect predictor of self-reported educational expectations, the percentage of applicants with high self-reported expectations is significantly different between the two groups defined by the level of ninth-grade mathematics class.  Furthermore, the subgroups defined by ninth-grade mathematics class are interesting in their own right.  

� The model used the same variables that were included as control variables in our regression analyses (see Table II.2), excluding the indicator for Project 69 and its interactions with the other variables.

# _1293536712.xls

# _1293541435.xls

# _1293621581.xls

# _1293621768.xls

# _1293542039.xls

# _1293537255.xls

# _1293537873.xls

# _1293537012.xls

# _1273387797.xls

# _1273388279.xls

# _1273389254.xls

# _1265108093.xls

# _1268051016.xls

# _1264854929.ppt