**Upward Bound Math-Science:**

**Program Description and Interim Impact Estimates**

Prepared by:

Robert Olsen

Neil Seftor

Tim Silva

David Myers

David DesRoches

Julie Young

Mathematica Policy Research, Inc.

For:

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

2007

This report was produced for the U.S. Department of Education under
Contract Number ED-04-CO-0152. The project monitor was Marisol
Cunnington in the Policy and Program Studies Service. The views
expressed herein do not necessarily represent the positions or policies
of the Department of Education. No official endorsement by the U.S.
Department of Education of any product, commodity, service or enterprise
mentioned in this publication is intended or should be inferred.

**U.S. Department of Education**

Margaret Spellings

*Secretary*

**Office of Planning, Evaluation and Policy Development**

Doug Mesecar

*Acting Assistant Secretary*

**Policy and Program Studies Service**

Alan L. Ginsburg

*Director*

**Program and Analytic Studies Division**

David Goodwin

*Director*

**April 2007**

This report is in the public domain. Authorization to produce it in
whole or in part is granted. Although permission to reprint this
publication is not necessary, the citation should be: U.S. Department of
Education, Office of Planning, Evaluation and Policy Development, Policy
and Program Studies Service, *Upward Bound Math-Science: Program*
*Description and Interim Impact Estimates*, Washington, D.C., 2007.

To order copies of this report, write:

ED Pubs

Education Publications Center

U.S. Department of Education

P.O. Box 1398

Jessup, MD 20794-1398;

Or via electronic mail, send your request to: edpubs@inet.ed.gov.

You may also call toll-free 1-877-433-7827 (1-877-4-ED-PUBS). If 877
service is not yet available in your area, call 1-800-872-5327
(1-800-USA-LEARN). Those who use a telecommunication device for the deaf
(TDD) or a teletypewriter (TTY) should call 1-800-437-0833.

To order online, point your Internet browser to: www.edpubs.org.

This report is available on the Department's Web site at:
http://www.ed.gov/about/offices/list/opepd/ppss/index.html.

On request, this publication is available in alternative formats, such
as Braille, large print, or computer diskette. For more information,
please contact the Department's Alternate Format Center at (202)
260-9895 or (202) 205-8113.

**CONTENTS**

**Chapter Page**

LIST OF EXHIBITS v

ACKNOWLEDGMENTS vii

EXECUTIVE SUMMARY ix

I INTRODUCTION 1

A. UPWARD BOUND MATH-SCIENCE PROGRAM 2

B. EVALUATION OF THE UPWARD BOUND MATH-SCIENCE

PROGRAM 3

C. OVERVIEW OF THIS REPORT 4

II THE OPERATION OF THE UPWARD BOUND MATH-SCIENCE PROGRAM 5

A. PROJECT HOSTS AND STAFF 6

1\. Host Institutions: Two- and Four-Year Colleges and Universities 6

2\. Summer Program Staff and Project Director 6

B. ELIGIBILITY, RECRUITMENT AND ENROLLMENT, AND

STUDENT CHARACTERISTICS 8

1\. Eligibility 8

2\. Recruitment and Enrollment 10

3\. Student Characteristics 12

C. PROGRAM DESCRIPTION 13

1\. Setting 14

2\. Goals of the Program 14

3\. Academic Orientation 14

4\. Summer Offerings 16

5\. Academic-Year Offerings 18

6\. Instructional Approaches 19

D. INTENSITY AND QUANTITY OF SERVICES 20

**Chapter Page**

III THE IMPACTS OF THE UPWARD BOUND MATH-SCIENCE PROGRAM 23

A. STUDY DESIGN 23

1\. Selecting the Samples 25

2\. Collecting Baseline Data 27

3\. Selecting a Matched Comparison Sample for the Impact Analysis 28

4\. Estimating the Impacts of UBMS Participation 31

B. ANALYSIS AND FINDINGS 31

1\. The Effect of UBMS on High School Outcomes 32

2\. The Effect of UBMS on Postsecondary Attendance, Persistence, and
Completion 33

3\. The Effect of UBMS on Postsecondary Field of Study 36

C. INTERPRETATION OF THE FINDINGS 37

REFERENCES 41

APPENDIX A: DATA COLLECTION A-1

APPENDIX B: PROGRAM IMPACTS B-1

APPENDIX C: SENSITIVITY OF IMPACTS TO AN ALTERNATIVE

> MEASURE OF POSTSECONDARY ATTENDANCE C-1

APPENDIX D: SAMPLE SIZES AND STANDARD ERRORS D-1

# EXHIBITS

**Exhibit Page**

II.1 Types of Institutions that Hosted UBMS Projects, 1995 7

II.2 Predominant Racial and Ethnic Group for UBMS Staff, Summer 1998 9

II.3 Racial and Ethnic Distribution for UBMS Staff and Project
Directors,

Summer 1998 9

II.4 Targets for Recruiting by UBMS Projects, 1994 11

II.5 Geographic Scope of Recruiting by UBMS Projects, 1994 11

II.6 Predominant Racial and Ethnic Group for Participants at UBMS
Projects,

Summer 1998 13

II.7 Most Important Goals of UBMS Projects, Summer 1994 15

II.8 Academic Orientation of UBMS Projects 15

II.9 Instruction Offered by UBMS Projects, by Subject Area 17

II.10 Noninstructional Services Offered by UBMS Projects 18

II.11 Academic Year Services Offered by UBMS Projects, 1994-95 19

II.12 Instructional Methods Used by UBMS Projects, 1994 21

II.13 Per-Capita Funding for UBMS Projects, FY 2004 22

III.1 List of Baseline Variables 28

III.2 Summary Statistics from the Baseline Variables 29

III.3 Math and Science Courses Taken in High School 33

III.4 Postsecondary Attendance, Highest Level Attended and College
Selectivity 34

III.5 Years of College and Degree Completion 36

III.6 Field of Study 38

# Acknowledgments

This report on Upward Bound Math-Science reflects the contributions of
many individuals. The authors would like to thank David Goodwin from the
Policy and Program Studies Service (PPSS) of the U.S. Department of
Education for his support and guidance throughout the study. The authors
would also like to thank Maggie Cahalan, who directed the initial
student survey of former participants, and Marisol Cunnington from PPSS;
Frances Bergeron from the office of Federal TRIO Programs; an anonymous
peer reviewer; and Mark Dynarski, Paul Decker, and John Deke from
Mathematica Policy Research, inc., for providing suggestions that helped
us to improve the report.

The authors are also grateful to the individuals who contributed to the
data collection, the data analysis, and the production of the report.
Amy Johnson led the effort to obtain rosters of participants from Upward
Bound Math and Science Centers, and Susan Mitchell directed the survey
of those centers. Melissa Thomas contributed to the follow-up survey
design, oversaw student survey operations, and prepared the survey data
file. Stephanie Naber managed the day-to-day operations of the follow-up
student survey; she also led our efforts to locate sample members for
the survey, collect student transcripts, and enter the transcript data.
Giuseppe Scaturro and Mary Grider processed the transcript data. Grider
and Heather Berry matched the colleges that students reported attending
to the integrated Postsecondary Education Data System (IPEDS). Miki
Satake computed the impact estimates. Carol Soble edited the report and
Micki Morris provided production support.

# Executive Summary

For many years, policymakers have been concerned by the relatively low
levels of academic achievement by economically disadvantaged K-12
students in math and science, by the underrepresentation of
disadvantaged college students in math and science majors, and by the
underrepresentation of people from disadvantaged groups in math and
science careers. While racial gaps in math and science test scores
narrowed somewhat in the 1970s and 1980s, substantial gaps persisted
through the 1990s to the present.

To help address these disparities, the U.S. Department of Education (ED)
established a math and science initiative in 1990 within Upward Bound, a
federal grant program designed to provide disadvantaged high school
students with skills and experiences that will prepare them for college
success. The initiative, referred to as Upward Bound Math-Science
(UBMS), awards grants to institutions---largely colleges and
universities---to operate UBMS projects. These projects were designed to
differ from "regular" Upward Bound projects in several respects. To
ensure that participants receive an intensive math and science
precollege experience, UBMS projects provide instruction that includes
hands-on experience in laboratories, computer facilities, and at field
sites. Opportunities are also provided to learn from mathematicians and
scientists employed at the host institution or engaged in research or
applied science in other institutions in the community. A six-week
summer program providing intensive instruction in laboratory science and
mathematics through precalculus is also offered.

Initially, ED funded 30 UBMS projects. by FY 2004, there were 127 UBMS
projects serving 6,845 students at a total cost of \$32.8 million.
Therefore, the annual cost per student---approximately \$4,800---is
comparable in cost to regular Upward Bound but much more expensive than
other federally funded precollege programs. More than 80 percent of UBMS
projects are hosted by four-year colleges and universities; most of the
rest are hosted by two-year colleges (Curtin and Cahalan 2004).

Participants in UBMS must meet the same eligibility requirements as
regular Upward Bound students: students must (1) belong to families
classified as low-income (taxable income of no greater than 150 percent
of the poverty line), or (2) be a potential first-generation college
student (neither parent has a bachelor's degree). Some students who
participate in UBMS summer programs are referred from regular Upward
Bound programs and then return to those programs during the academic
year. However, as would be expected, UBMS projects are more likely to
consider students' interests in math and science when reviewing
applications than are most regular Upward Bound projects (Moore 1997b).
While 25 percent of participants are white, most program participants
are from underrepresented minority groups: about 60 percent of
participants are African American or Hispanic (Curtin and Cahalan 2004).

## EVALUATION OF UPWARD BOUND MATH-SCIENCE

Since 1991, Mathematica Policy Research, Inc., (MPR) has been conducting
the National Evaluation of Upward Bound for ED. The centerpiece of this
evaluation has been a random assignment evaluation of regular Upward
Bound. In 1997, ED added a new component to the evaluation that is
focused on UBMS. In 1998, MPR selected a random sample of the students
who participated in UBMS between 1993 and 1995 at projects that were
still operating at that time. This report constitutes the first of two
evaluation reports on UBMS, and it is based on participant surveys and
student transcripts collected for this sample between 1998 and 1999 and
again between 2001 and 2002. The second report is scheduled for
completion in 2006 and will be based on data collected between 2003 and
2005.

The evaluation of UBMS has two components: a descriptive analysis and an
impact analysis. The descriptive analysis relies primarily on a survey
of project directors to describe the resources available to UBMS
projects; the types of institutions that host them; the credentials and
demographic characteristics of project staff; recruitment, eligibility,
and enrollment of students; student characteristics; and program
offerings. The impact analysis is designed to measure the effects of
UBMS on (1) performance in high school, especially in math and science
courses; (2) postsecondary attendance, persistence and completion; and
(3) the likelihood of completing a postsecondary degree in mathematics
or a scientific field.

The impact analysis is based on a comparison of UBMS participants with a
sample of students that (1) applied to enroll in regular Upward Bound
programs in the early 1990s, (2) never participated in UBMS and (3) have
been tracked by MPR as part of the national evaluation. This comparison
group was selected to ensure that it had similar characteristics to the
sample of UBMS participants, and we controlled statistically for the
small remaining differences in these characteristics between UBMS
participants and the comparison group.

If UBMS participants are more interested or skilled in math and science
than the students in the comparison group, the estimated effects of the
program may be subject to "selection bias" and may overstate the true
effects of participating in UBMS. However, the comparison group we
selected was probably the best available short of a randomized control
group because the students in the comparison group exhibited the
motivation to pursue Upward Bound services, and our analysis shows that
the participant and comparison groups are similar in other ways as well.
In addition, we implemented a data collection and analysis plan designed
to minimize selection bias (see Chapter III for more details). While a
control group from a randomized experiment would have prevented
selection bias, the comparison group we selected greatly reduced the
cost of the evaluation because we were already collecting data for this
group as part of the national evaluation.

Note that the descriptive findings and impact estimates presented in
this report describe the operations and effects of the Upward Bound
Math-Science Program as it operated in the mid-1990s. At that time, it
was a relatively new program, and some changes have occurred in how UBMS
projects operate. In Chapter II, we mention some of these changes as
they are reflected in information provided to us by UBMS project
directors in a survey of grantees. It is certainly possible that some of
the changes in the program since the mid-1990s have influenced the
effectiveness of UBMS projects, and the evaluation does not attempt to
measure any changes in effectiveness since that time. In this report, we
measure the effects of the program on people who participated between
1993 and 1995 and describe the operations of the program at that time.

## REPORT FINDINGS

From our descriptive analysis, we found that UBMS projects:

-   ***Provide a large quantity of academic instruction.*** in the
    summer, the average UBMS project provided a total of 240 hours of
    academic instruction, and participation in the program is roughly
    full-time for a six-week period.

-   ***Are most active during the summers.*** UBMS projects typically
    provide services, such as tutoring or study sessions, during the
    school year, but they provide most of their services during summer
    residential programs at the colleges or universities hosting the
    program.

-   ***Provide academic enrichment in math and science subjects.*** Many
    UBMS projects offer courses in algebra II, geometry, precalculus,
    biology, chemistry, physics and computer software; in contrast, few
    offer courses in Social Studies (though many offer English courses
    in addition to their math and science offerings). At most projects,
    the course work is designed to provide academic enrichment instead
    of academic remediation.

-   ***Provide instruction through a combination of single-subject
    courses and interdisciplinary instruction.*** While other
    instructional techniques were used, three out of four projects
    provided instruction primarily through single-subject academic
    courses or the combination of these courses with interdisciplinary
    instruction.

Given the academic services provided by UBMS, it is natural to ask
whether participating in UBMS affects the educational outcomes of the
students that participate. From our impact analysis, we found that UBMS:

-   ***Improved high school grades in math and science and overall.***
    UBMS had a positive effect on high school grades, increasing the
    average GPA in math courses from 2.7 to 2.8, the average GPA in
    science courses from 2.7 to 2.9 and the average GPA overall.

-   ***Increased the likelihood of taking chemistry and physics in high
    school.*** UBMS increased the likelihood that participants took
    upper-level science courses in high school, raising the percentage
    of students taking chemistry from 78 percent to 88 percent and
    raising the percentage of students taking physics from 43 percent to
    58 percent. in contrast, UBMS did not affect coursetaking in
    advanced math subjects (see Chapter III, Exhibit III.3).

-   ***Increased the likelihood of enrolling in more selective four-year
    institutions.*** UBMS increased the percentage of students that
    attended four-year colleges and universities from 71 percent to 82
    percent. The increase in four-year attendance is particularly
    pronounced for more selective schools (those rated as "most
    selective", "highly selective" or "very selective" by the Barron's
    Guide): UBMS increased the percentage of students that attended more
    selective four-year colleges from 23 percent to 33 percent (see
    Chapter III, Exhibit III.4).

-   ***Increased the likelihood of majoring in math and science.*** UBMS
    affected students' choice of major, increasing the percentage
    majoring (or planning to major) in math or science from 23 percent
    to 33 percent and decreasing the percentage majoring in a field
    outside of math or science and the social sciences from 51 to 42
    percent. UBMS also seems to increase the percentage of participants
    majoring in the social sciences (see Chapter III, Exhibit III.6).

-   ***Increased the likelihood of completing a four-year degree in math
    and science.*** UBMS increased the percentage of students that
    earned a bachelor's degree in a math and science field from 6
    percent to 12 percent and decreased the percentage that earned a
    bachelor's degree outside of math, science, and the social sciences
    from 20 to 14 percent (see Chapter III, Exhibit III.6). Because 47
    percent of participants in our sample were still in college when we
    interviewed them in 2002, findings related to degree completion
    should be treated as preliminary, and a final assessment will be
    presented in a subsequent report.

In addition, we computed separate impact estimates for subgroups defined
by sex, race and ethnicity, and prior participation in regular Upward
Bound. For some outcomes, we found differences in subgroup impacts that
were statistically significant. For example, the effect of UBMS on
four-year college attendance was larger for women than for men. However,
the number of significant differences between subgroups was relatively
small, and there was no obvious pattern to the findings suggesting that
particular groups benefited more from UBMS than other groups. Therefore,
it is not clear whether the significant subgroup differences are due to
chance or to systematic differences in the effects of UBMS on different
groups of participants.

To summarize the report's findings, UBMS provides intensive academic
instruction in math and science, and our impact estimates suggest that
it improves several student outcomes in high school and college. In
addition, and consistent with the objectives of the program, preliminary
estimates suggest that UBMS participation increases the odds of majoring
in math or science. In the next report, we will reexamine the effects on
college completion, examine the effects on labor market outcomes, such
as employment in the sciences, and weigh the benefits of the program
against the costs.

It is tempting to compare the estimated impacts of UBMS to the estimated
impacts of regular Upward Bound presented in earlier reports. However,
it is important to recognize that the two studies used different
methods: while the evaluation of regular Upward Bound is based on an
experimental design, the "gold standard" in evaluation research, the
evaluation of UBMS is based on nonexperimental methods that may suffer
from selection bias, as described earlier. If the estimated effects of
UBMS are inflated due to selection bias, then the impression based on
our findings that UBMS is more effective than regular Upward Bound might
be attributable to differences in the methods used to estimate the
impacts instead of differences in the effectiveness of the two programs.

# I. introduction

For many years, policymakers have been concerned by the relatively low
levels of academic achievement by economically disadvantaged K-12
students in math and science, by the underrepresentation of
disadvantaged college students in math and science majors, and by the
underrepresentation of people from disadvantaged groups in math and
science careers. National statistics show that while the gaps between
minorities'[^1] and whites' math and science test scores narrowed
somewhat in the 1970s and 1980s, gaps in test scores and other
educational outcomes persisted through the 1990s to the present.

-   ***Disadvantaged students take fewer math and science courses in
    high school.*** in the 1991--92 school year, 57 percent of seniors
    in the lowest socioeconomic status (SES) quartile took a math
    course, compared with 75 percent of seniors from the highest SES
    quartile; 37 percent of seniors from the lowest SES quartile took a
    science course, compared with 61 percent of seniors from the highest
    SES quartile (U.S. Department of Education 1996b). in 1994, only 58
    percent of black high school graduates had completed geometry while
    in high school, compared with 73 percent of white high school
    graduates. in the same year, only 13 percent of black and Hispanic
    graduates had completed the common triad of science
    courses---biology, chemistry, and physics---compared with 23 percent
    of white graduates (U.S. Department of Education 1996a).

-   ***Minority college students are less likely to take math and
    science courses or earn a degree in math or science. T***en percent
    of black college students and 14 percent of Hispanics received
    credit for calculus or advanced math courses in the late 1980s,
    compared with 22 percent of whites. Sixteen percent of blacks and 21
    percent of Hispanic college students earned course credits in
    chemistry, compared with 27 percent of whites, and 8 percent of
    blacks and 11 percent of Hispanics earned college credit for
    physics, compared with 18 percent of white students (U.S. Department
    of Education 1994). Because minorities earned fewer college credits
    in math and science than whites, it is not surprising that they were
    less likely to earn degrees in those subjects. Black students earned
    7 percent of all bachelor's degrees in 1995-96, but just 7 percent
    of all bachelor's degrees in math and science fields. in the same
    year, Hispanic students earned 5 percent of all bachelor's degrees,
    but just 4 percent of all bachelor's degrees in math and science
    (U.S. Department of Education 1999).[^2]

-   ***Minorities are less likely than whites to enter careers in math
    and science.*** Among people who were working in a scientific field
    in 1995 and had obtained their college degree in the previous five
    years, only 6 percent were black (National Science Foundation 1995).
    However, in 1990, around the time those individuals would have been
    in college, blacks accounted for 14 percent of the U.S. population
    aged 18-24 years old (Census Bureau 1990a, Census Bureau 1990b).

## A. Upward Bound Math-Science Program

To help address these disparities, the U.S. Department of Education (ED)
in 1990 established the Upward Bound Math-Science Program (UBMS) within
Upward Bound, a federal grant program designed to provide disadvantaged
high school students with skills and experiences that will prepare them
for college success. UBMS was designed to differ from "regular" Upward
Bound in a few key respects. To ensure that participants receive an
intensive math and science precollege experience, ED requires UBMS
projects to provide instruction that includes hands-on experience in
laboratories, computer facilities, and at field sites. Also provided are
the following: opportunities to learn from mathematicians and scientists
employed at the host institution or engaged in research or applied
science in other institutions in the community;[^3] involvement with
tutors and counselors who are graduate and undergraduate math and
science majors; and a six-week summer program consisting of daily course
work and activities, instruction in laboratory science and mathematics
through precalculus (in addition to foreign language, composition and
literature, which are also required offerings at regular Upward Bound
projects).

Initially, ED funded 30 UBMS projects. By FY 2004, there were 127 UBMS
projects serving 6,845 students at a total cost of \$32.8 million.
Therefore, the annual cost per student---approximately \$4,800---is
comparable in cost to regular Upward Bound but much more expensive than
other federally funded precollege programs. More than 80 percent of UBMS
projects are hosted by four-year colleges and universities; most of the
rest are hosted by two-year colleges (Curtin and Cahalan 2004).

UBMS participants must meet the same eligibility requirements as regular
Upward Bound participants: they must (1) come from families that are
classified as low-income (taxable income not over 150 percent of the
poverty line), or (2) be a potential first-generation college student
(neither parent has a bachelor's degree). Some students who participate
in UBMS are referred from regular Upward Bound programs and then return
to those programs during the academic year. However, as would be
expected, UBMS projects are more likely to consider students' interests
in math and science when reviewing applications than are most regular
Upward Bound projects (Moore 1997b). While 25 percent of participants
are white, most program participants are from underrepresented minority
groups: about 60 percent of participants are African American or
Hispanic (Curtin and Cahalan 2004).

Despite coming from low-income families, the evidence suggests that on
average, UBMS serves students who do well in high school and attend
college at higher rates than the average low-income student. Data
reported by Upward Bound projects suggest that prior to participating in
Upward Bound, UBMS participants earned higher grades on average than
regular Upward Bound participants (Curtin and Cahalan 2004). In
addition, the national evaluation has shown that regular Upward Bound
participants would have attended college at much higher rates than the
average low-income student even if they had not participated in Upward
Bound (Myers et al. 2004).[^4] Therefore, the evidence strongly suggests
that UBMS serves high school students who are much more likely to attend
college than the average low-income student.

## B. Evaluation of the Upward Bound Math-Science Program 

The legislation establishing Upward Bound authorizes ED to sponsor
studies of it, including examinations of program effectiveness. In 1991,
ED awarded a contract to Mathematica to conduct the National Evaluation
of Upward Bound. This evaluation has several components, but its
signature feature is an experiment to measure the effects of
participating in regular Upward Bound. We selected a random sample of
Upward Bound projects (excluding UBMS projects); for each of these
projects, we randomly assigned eligible applicants to a treatment group,
which was offered the chance to participate in the program, or a control
group, which was not. The evaluation is ongoing, and it was one of the
first to use experimental methods to measure the effects of a federally
funded education program.

This report presents the results of an evaluation of the Upward Bound
Math-Science Program. In 1997, Mathematica completed two reports on
UBMS. One provided a descriptive analysis of the program based primarily
on site visits to a representative sample of 14 UBMS projects (Moore
1997a). The other provided an assessment of the feasibility of
conducting a rigorous evaluation of the effects of UBMS on student
outcomes (Myers 1997). When ED awarded a contract to Mathematica in 1997
to extend its evaluation of the effects of regular Upward Bound, it also
specified an evaluation of the effects of UBMS. This evaluation consists
of two components: a descriptive analysis and an impact analysis. The
descriptive analysis relies primarily on a survey of UBMS project
directors conducted in the spring of 1998. The analysis is designed to
describe the resources available to UBMS projects; the types of
institutions that host them; the credentials and demographic
characteristics of project staff; recruitment, eligibility, and
enrollment of students; student characteristics; and a description of
the program, including its goals, academic orientation, instructional
methods and the intensity and quantity of the services provided.

The UBMS impact study is designed to measure the effects of
participating in UBMS on college enrollment, choice of major, and other
outcomes for students who participated during the summer of 1993, 1994,
or 1995.[^5] Conceptually, the study contrasts how participants fared
with how they would have fared if they had not participated in UBMS. We
compared UBMS participants with eligible applicants to the regular
Upward Bound projects participating in the national evaluation. From
this pool, we systematically selected a matched comparison group of
students who were as similar as possible to UBMS participants in terms
of characteristics and experiences that could potentially predict later
outcomes. These characteristics included demographics---such as sex,
race, and ethnicity---and prior academic achievement such as grade point
average and math and science courses taken in 9th grade. The key
difference was that the matched comparison students did not participate
in UBMS.

The selection of matched comparison students also took into account
experiences in other precollege programs, in particular regular Upward
Bound. Because regular Upward Bound is an intensive program that can
influence high school achievement and postsecondary outcomes (Myers and
Schirm 1999; Myers et al. 2003), it is important to account for exposure
to regular Upward Bound when estimating how UBMS participants would have
fared if they had not participated in UBMS. For UBMS participants who
had previously participated in a regular Upward Bound program---perhaps
during the academic year---we selected comparison students who had also
participated in the regular Upward Bound.[^6] For UBMS participants who
had not participated in regular Upward Bound, we selected comparison
students who did not participate in regular Upward Bound. [^7]

Several data sources play a key role in the impact analysis. Baseline
characteristics were collected for comparison group members through the
baseline survey for the evaluation of regular Upward Bound; baseline
information on many of the same characteristics was collected for UBMS
participants through a follow-up survey conducted in 1999.[^8] This
follow-up survey was used to collect information about educational
outcomes for UBMS participants, and a similar survey was used to collect
analogous information for comparison students. Finally, secondary and
postsecondary transcripts were collected for both types of students to
assess academic achievement.

## C. Overview of This Report

The remainder of this report is organized as follows. Chapter II
describes the operation of the UBMS program. Chapter III presents
findings from the impact analysis.

# II. The Operation of the Upward Bound Math-Science Program

To interpret information on the impacts of UBMS, it is necessary to
understand what the program entails. This chapter describes key features
of the operations of UBMS projects, including the characteristics of
host institutions and staff, projects' recruitment practices and
enrollment levels, participants' characteristics and projects' goals and
services. For context, this chapter presents comparable information on
the operations of regular Upward Bound when possible.

The primary data source for this chapter is a survey of UBMS projects
conducted in the spring of 1998. The survey sample consisted of all 81
projects operating at the time, and 74 of the 81 projects responded to
the survey.[^9]^,^[^10] the survey requested information about program
operations in two separate years---(1) 1994 (in the middle of the period
over which our sample was participating in UBMS) and (2) 1998 (the year
prior to the survey)---but some questions were specific to 1998. When
possible, we focus our analysis of program operations on 1994 to
facilitate comparisons with regular Upward Bound projects operating in
1993, as reported in Fasciano and Jacobson (1997) and to describe the
programs that served the same cohorts of participants for whom we
measured the impacts of the program (see Chapter III).[^11] To augment
the information provided by the survey of UBMS projects, we also use
information from case studies and annual performance reports (Moore
1997b).

The findings in this chapter indicate that UBMS projects provide
intensive academic enrichment to disadvantaged high school students in
math and science using staff with strong academic credentials in those
subjects. Some of the features that make UBMS projects distinctive, even
from regular Upward Bound projects, are: (1) high levels of annual
funding per student and low student-teacher ratios, (2) recruiting
strategies that attract students from wide geographic areas, (3) service
provision that is heavily concentrated in residential programs during
the summer, (4) course offerings that focus on math and science relative
to other subjects, (5) academic preparation over nonacademic college
preparatory activities, and (6) academic enrichment over remediation.
The remainder of the chapter provides a description of UBMS and an
assessment of its distinctive features.

## A. Project Hosts and Staff

The impacts of UBMS projects on student outcomes may depend on the types
of institutions that host them and the people they hire to serve as
instructors and other staff. In this section, we describe the types of
institutions that host UBMS projects and the staff that provide services
to program participants.

### 1. Host institutions: Two- and Four-Year Colleges and Universities

The types of institutions that host a UBMS project may influence where
students attend college. Most Upward Bound programs are hosted by either
two- or four-year postsecondary institutions. Evidence from the national
evaluation of regular Upward Bound suggests that participation at
projects hosted by four-year colleges raises the probability of
attending a four-year college, and participation at projects hosted by
two-year colleges raises the probability of attending a two-year college
(Myers et al. 2002b). Therefore, the types of institutions that host
UBMS projects may influence the types of postsecondary institutions that
program participants subsequently attend.

Nearly nine out of ten UBMS projects operating in the mid--1990s were
hosted by four-year colleges, a substantially higher proportion than
among regular Upward Bound projects (see Exhibit II.1). Four-year
colleges may find it easier than other potential host institutions to
meet some of ED's guidelines for UBMS, including offering hands-on
experience in laboratories and computer facilities, opportunities to
learn from mathematicians and scientists engaged in research or applied
science and involvement with tutors and counselors who are graduate and
undergraduate students in math and science.

### 2. Summer Program Staff and Project Director

UBMS projects are directed by highly educated individuals and staffed by
people with strong credentials in math and science. These staff have
responsibility for a relatively small number of students, which may
provide opportunities for individual instruction. At the typical
project, the project director and staff can provide same-race role
models for many of the students they serve. The sections below provide
more detail on our findings concerning staff size and composition by job
title, staff credentials, and the racial composition of project staff.

#### a. Staff Size and Composition by Job Title

In 1994, UBMS projects had an average of 24 staff members, comprising
roughly eight instructors, five resident counselors, four mentors, three
tutors, two administrators, one academic or guidance counselor and one
clerical staff member.[^12] Overall, the average student-staff ratio in
summer 1998 was 2:1, with a range from about 1:1 to 5:1.

Exhibit II.1

Types of Institutions that Hosted UBMS Projects, 1995

  ------------------------ -------------------------- --------------------
  Type of institution      Upward Bound Math-Science  Regular Upward Bound

  Four-year college or     88%                        68%
  university                                          

  Two-year college         11                         28

  Other institution        2                          4
  ------------------------ -------------------------- --------------------

[Note:]{.smallcaps} Percentages may not sum to 100 percent due to
rounding.

[Source]{.smallcaps}: Moore 1997b, Exhibit II.1, p. 15.

These findings, combined with findings from Moore (1997b), suggest that
student-staff ratios are typically lower in UBMS projects than in
regular Upward Bound projects. The survey of grantees did not collect
information on the number of full-time-equivalent (FTE) staff, but the
information available suggests that UBMS projects typically maintain
student-staff ratios that are substantially lower than in regular Upward
Bound. Moore (1997b) found that 14 randomly selected UBMS projects
visited in summer 1996 had an average of 2.6 students per FTE staff,
including administrators, and 8.2 students per FTE instructional staff
(Moore 1997b, Exhibit II.6). In contrast, tabulations from the data used
by Fasciano and Jacobson (1997) indicate that in summer 1992, regular
Upward Bound projects had more students per staff member---5.1 students
per FTE staff and 13.6 students per FTE instructional staff (Moore
1997b, Exhibit II.6).

#### b. Credentials

At the average UBMS project in 1998, most staff were highly educated and
had educational backgrounds in math and science. About one-quarter had
attended some college but not obtained a degree, another quarter had
obtained a bachelor's or associate's degree (mostly bachelor's degrees)
and the rest had done graduate work or obtained a graduate degree. Most
staff members without undergraduate degrees were undergraduate students
who served as UBMS mentors, tutors and resident counselors while working
toward bachelor's degrees in math, science, or education. Approximately
two out of five staff members had their highest degree in science or the
social sciences (31 percent) or math (10 percent); additionally, most
staff members without a degree were working toward a bachelor's degree
in math, science, or education.

Most instructors at the average UBMS project had experience teaching
math or science. During the school year, most instructors were either
high school teachers (41 percent) or postsecondary teachers (31
percent); one-fifth were graduate students (14 percent) or
undergraduates (6 percent). Moreover, at the typical project, two-thirds
of the high school teachers and three-quarters of the postsecondary
teachers taught in a math or science field.

The professional and educational backgrounds of UBMS project directors
provide insight on their credentials to direct projects. One-fifth of
the directors were faculty members at the host institution or another
college, roughly the same percentage as in regular Upward Bound. In
1998, two-thirds of the directors held a master's degree and one-fifth
held a doctorate. About half had their highest degree in education, and
less than one-fifth had their highest degree in engineering,
mathematics, or physical sciences.[^13] Although UBMS project directors
were less likely than program staff as a whole to have a background in
math or science, this may not be surprising since subject area expertise
is probably less important for administrators than for other staff.
Compared with regular Upward Bound project directors, UBMS project
directors were more highly educated. They were, for example, twice as
likely to have a doctorate.

#### c. Race and Ethnicity

At about 9 out of 10 UBMS projects operating in 1998, one racial or
ethnic group accounted for a majority of the staff (see Exhibit II.2).
In many cases this pattern may have reflected a conscious strategy, also
used in regular Upward Bound, to provide minority students with
same-race role models. For example, at 21 UBMS projects, a majority of
the staff members were black; at 18 of these, a majority of the students
were also black.

The racial and ethnic profile of UBMS project directors was similar to
that of UBMS project staff (see Exhibit II.3). Both staff and project
directors were nearly evenly split between white and nonwhite; project
directors were slightly more likely to be black and less likely to be
Asian than other staff. The race and ethnicity of the UBMS project
director often matched that of the predominant student racial and ethnic
group: more than three-fourths of the minority directors headed programs
where students from the same group constituted a plurality of
participants.

## B. Eligibility, Recruitment and Enrollment, and Student Characteristics

To shed light on the types of students that participate in UBMS, we
examine the eligibility criteria that students must meet and the
recruiting strategies that UBMS projects use to attract students. We
also examine the characteristics of participants as reported by project
directors.

### 1. Eligibility

UBMS projects have to meet the same federal rules as regular Upward
Bound projects concerning the composition of participants. At each
project, at least two-thirds of the participants must be both low-income
and potential first-generation college students; the remaining students
must meet either of these two criteria. At the average project in summer
1998, about 77 percent of students met both of these eligibility
criteria, about 14 percent were first-generation only and about 9
percent were low-income only, very similar to the distribution in
regular Upward Bound during 1992-93. In addition, UBMS and regular
Upward Bound projects are only allowed to serve students who have
completed 8th grade.

Exhibit II.2

Predominant Racial and Ethnic Group for UBMS Staff, Summer 1998

![](media/image1.wmf)

[Note:]{.smallcaps} Percentages may not sum to 100 percent due to
rounding. The 12 percent of projects with no majority racial and ethnic
group included 6 percent of projects with a plurality of blacks and 4
percent with a plurality of whites; the remainder had no plurality.

[Source]{.smallcaps}: 1998 survey of UBMS projects.

Exhibit II.3

Racial and Ethnic Distribution for UBMS Staff and Project Directors,
Summer 1998

  --------------------------- ---------------------- ---------------------
  Race and Ethnicity          Staff                  Director

  White                       49%                    49%

  Black                       33                     39

  Hispanic or Latino          7                      7

  Asian                       4                      1

  American indian or Alaskan  2                      4
  Native                                             
  --------------------------- ---------------------- ---------------------

[Note]{.smallcaps}: Percentages may not sum to 100 percent because (1)
people may fall into multiple categories, (2) the Pacific Islander
category was excluded from the exhibit because only nine staff members
nationwide fell into this category and (3) some staff members may not
have been classified by race or ethnicity.

[Source]{.smallcaps}: 1998 survey of UBMS projects.

Most UBMS projects also adopt additional student eligibility criteria
for enrollment in the program---for example, requirements about grade
level, school course work, or recommendations. In 1994, over
three-fourths of projects required students to have finished 9th grade;
a few projects required 10th grade completion. In addition, nearly all
UBMS projects required a teacher recommendation and completion of at
least one high school course in math or science, and applicants enrolled
in regular Upward Bound commonly needed a recommendation from the
director. Finally, about 30 percent of UBMS projects prohibited students
from returning from the previous summer's program, and almost half
prohibited students from returning unless they met certain
criteria.[^14]

### 2. Recruitment and Enrollment

To find a pool of potentially eligible applicants, UBMS projects focused
mainly on other precollege programs or secondary schools (see Exhibit
II.4). Among UBMS projects operating in summer 1994, nearly all
recruited from regular Upward Bound projects, while substantial
majorities also recruited from Talent Search projects and from middle or
high schools directly. However, it is important to note that while
almost all UBMS projects recruited from regular Upward Bound projects in
1994, data from the evaluation suggest that fewer than one in five UBMS
participants had previously participated in regular Upward Bound.

Historically, UBMS projects have cast a wide net in recruiting students
beyond the local areas of the host institutions. In 1994, only about one
in ten UBMS projects recruited exclusively from a specific and typically
local set of feeder schools or Upward Bound projects; the rest recruited
from state-wide or regional lists of schools and programs (see Exhibit
II.5). By 1998, however, the percentage of UBMS projects that recruited
exclusively from a specific set of feeder schools or Upward Bound
projects had tripled. Therefore, it appears that over time, more UBMS
projects are taking local recruitment strategies like regular Upward
Bound projects.

By design, UBMS projects are smaller than regular Upward Bound projects.
Through recruitment, UBMS projects received an average of 108
applications for the summer of 1994, ranging from a low of 50 to a high
of 300, and they enrolled between 40 and 53 students. In contrast,
regular Upward Bound programs enrolled an average of about 75 students
in the mid-1990s (Moore 1997b).

Exhibit II.4

Targets for Recruiting by UBMS Projects, 1994

![](media/image2.wmf)[Source]{.smallcaps}: 1998 survey of UBMS projects.

Exhibit II.5

Geographic Scope of Recruiting by UBMS Projects, 1994

![](media/image3.wmf)

[Note:]{.smallcaps} Percentages may not sum to 100 percent due to
rounding.

[Source]{.smallcaps}: 1998 survey of UBMS projects.

### 3. Student Characteristics

In our 1998 survey of UBMS projects, project directors provided
information on the distribution of students participating in their
projects by sex, race, grade level, and place of residence: [^15]

-   ***Sex.*** At the average UBMS project, like the average regular
    Upward Bound project, 60 percent of students were female.[^16]
    However, this varied considerably across UBMS projects from a low of
    25 percent to a high of 78 percent.

-   ***Race***. On average, UBMS projects served an ethnically diverse
    group of students: 42 percent black, 27 percent white, 15 percent
    Hispanic, 8 percent Asian, 5 percent American indian and 1 percent
    Native Hawaiian or other Pacific Islander. However, most UBMS
    projects serve participants where one racial and ethnic group
    constituted a majority (see Exhibit II.6).[^17] Furthermore, some
    UBMS projects exclusively served students from a single racial or
    ethnic group. For example, six UBMS projects reported that all of
    its participating students were black.

-   ***Grade level. T***he eligibility guidelines discussed above, along
    with other factors, can affect the distribution of students across
    different grade levels. At the average project, 29 percent of
    participants were entering 12th grade, 37 percent were entering 11th
    grade, 27 percent were entering 10th grade and 6 percent were
    entering 9th grade. These exhibits suggest that on average, UBMS
    projects serve students who are slightly closer to graduation than
    is the case at regular Upward Bound projects.[^18] However, there
    was substantial variation in the grade level distribution of
    participants across projects in 1998. for example, one UBMS project
    reported that all of its participants were entering 12th grade,
    while four reported that none were rising seniors.[^19]

-   ***Place of residence.*** Given that most projects recruited across
    the state or region, it is not surprising that many UBMS
    participants came from outside the grantee's local city or town. At
    the average project, only about 25 percent of the students were
    locals. As we would expect, pprojects that recruited only from a set
    of local schools or regular Upward Bound projects served
    considerably higher percentages of students from the local area than
    other UBMS projects.

Exhibit II.6

Predominant Racial and Ethnic Group for Participants at UBMS Projects,

Summer 1998

![](media/image4.wmf)[Note:]{.smallcaps} Percentages may not sum to 100
percent due to rounding. The 37 percent of UBMS projects

> with no majority racial and ethnic group of students included 17
> percent with a plurality of white students, 11 percent with a
> plurality of Hispanics, 6 percent with a plurality of blacks, 3
> percent with a plurality of American indians and 1 percent with a
> plurality of Asians.

[Source]{.smallcaps}: 1998 survey of UBMS projects.

## 

Moore (1997b) described UBMS participants as a more select group than
regular Upward Bound participants based on having earned somewhat higher
grades and having greater interest in math and science (pp. 23, 26)
prior to participating in Upward Bound. Discussions with UBMS and
regular Upward Bound staff revealed that UBMS participants were
typically considered "more serious about school" than regular Upward
Bound participants (Moore 1997b, p. 26).

## C. Program Description

In a college-like setting, UBMS projects offer academic enrichment in
math and science to improve student achievement in those subjects and
expose students to math and science careers. In this section, we
describe the following features of UBMS projects in more detail: the
setting in which these projects provide services; the goals, academic
orientation, academic offerings, and instructional approaches of these
projects; and the intensity and quantity of services the UBMS projects
provide.

### 1. Setting

As described earlier, UBMS projects are typically hosted by two- and
four-year postsecondary institutions (see Section A.1). Most UBMS
projects are hosted by four-year colleges and universities, and most of
these institutions have dormitories to house their students. These
dormitories are often available in the summers to house participants of
summer programs hosted by these institutions.

UBMS projects typically exposed participants to a college setting during
the summer program by housing them in the college dormitories. Virtually
all the UBMS projects we surveyed (100 percent in 1994, 97 percent in
1998) offered a residential component to their summer programs, compared
with 87 percent of regular Upward Bound programs in 1992 (Moore 1997a).
At almost all UBMS projects, students lived in the dormitories for the
entire summer program, which lasted about six weeks on average.[^20]
Therefore, for six weeks, participants lived on campus like many college
students do during the academic year.

### 2. Goals of the Program

As mentioned in Chapter I, the general objective of the Upward Bound
Math-Science program is to prepare participating students for
postsecondary programs leading to careers in math and science. Seven out
of ten UBMS projects operating in 1994 rated "academic performance in
math and science" as their most or second most important goal (see
Exhibit II.7). The focus on academic improvement was similar to the
focus of regular Upward Bound projects operating around the same
time.[^21]

However, two goals that regular Upward Bound projects considered
moderately important were not considered important by UBMS projects.
First, only 13 percent of UBMS projects reported that one of their top
two goals was fostering students' personal skills (e.g., goal
orientation, ability to adapt to new settings), compared with 31 percent
of regular Upward Bound programs. Second, none of the UBMS projects
cited improving students' access to financial aid as one of their top
two goals, compared with 35 percent of regular programs.

### 3. Academic Orientation

UBMS projects try to provide academic enrichment beyond what students
are exposed to in school (see Exhibit II.8). Very few projects
emphasized remedial instruction in 1994.[^22] While

Exhibit II.7

Most Important Goals of UBMS Projects, Summer 1994

![](media/image5.wmf)

[Source]{.smallcaps}: 1998 survey of UBMS projects.

### 

Exhibit II.8

Academic Orientation of UBMS Projects^a^

+---------------------------------+----------------+------------------+
| Major Emphasis                  | UBMS           | Regular Upward   |
|                                 |                | Bound            |
|                                 | Summer 1994    |                  |
|                                 |                | Summer 1992^b^   |
+---------------------------------+----------------+------------------+
| *Support*---instruction that    | 33%            | 55%              |
| parallels what students are     |                |                  |
| taught in regular school        |                |                  |
| courses                         |                |                  |
+---------------------------------+----------------+------------------+
| *Remediation*---instruction     | 6              | 23               |
| that concentrates on            |                |                  |
| fundamental concepts and skills |                |                  |
| that were taught in earlier     |                |                  |
| grades                          |                |                  |
+---------------------------------+----------------+------------------+
| *Enrichment*---instruction in   | 73             | 69               |
| concepts and material beyond    |                |                  |
| what students are exposed to in |                |                  |
| regular school classes          |                |                  |
+---------------------------------+----------------+------------------+

^a^ Percentages do not sum to 100 because grantees were allowed to rate
more than one approach as a major emphasis.

^b^ Excludes summer bridge programs for Upward Bound participants who
have just graduated from high school.

[Sources]{.smallcaps}: 1998 survey of UBMS projects, Fasciano and
Jacobson 1997.

about one in four regular Upward Bound programs emphasized the provision
of remedial instruction, fewer than one in ten UBMS projects reported
doing the same.

The focus of UBMS on academic enrichment over remediation is consistent
with the types of students served by the program. As described earlier,
findings in Moore (1997b) indicate that, on average, UBMS participants
probably had less need for remedial support than regular Upward Bound
participants.

### 4. Summer offerings

In accordance with program guidelines, UBMS projects offered instruction
in a diverse array of academic subject areas (see Exhibit II.9).
Seventy-five percent or more of these projects in 1994 offered
instruction in the following subjects: writing and composition, algebra
II, geometry, precalculus, computer applications and software use,
biology, chemistry and physics. The average total number of offerings in
1994 was about 14, with a range of 2 to 22. The average number of
offerings in math and science was about 7, with a range of 1 to 11.[^23]
Thus, on average, math and science courses accounted for roughly half of
UBMS projects' total offerings.

UBMS projects clearly differed from regular Upward Bound projects in
their relative emphasis on certain subjects. First, as expected, they
concentrated their offerings more on math and science. Although UBMS
projects were no more likely than regular Upward Bound projects to offer
certain math or science courses (for example, algebra II, geometry
precalculus, calculus, biology and chemistry), UBMS projects were much
less likely to offer instruction in areas outside of math and science,
such as social science or history courses or electives or non-academic
courses. Second, consistent with their greater emphasis on enrichment
than on remediation, UBMS projects were less likely than regular Upward
Bound programs to offer low-end courses such as reading comprehension
and vocabulary, pre-algebra, and earth science (see Exhibit II.9).

To help prepare students for a postsecondary education and
post-collegiate careers in math or science, UBMS projects also offered a
range of support services and activities (see Exhibit II.10). Among the
most common activities were field trips (e.g., to math or science
facilities) and assistance with college and financial aid applications.
The average number of these non-instructional offerings in 1994 was
about 10, with a range of 3 to 15.

UBMS projects were substantially less likely than regular Upward Bound
projects to offer services focused on preparing for college. Many
regular Upward Bound projects would have provided these services during
the academic year: both regular Upward Bound and Math Science programs
focus on academics during the summer. Because many UBMS participants
participated in other precollege programs during the academic year, UBMS
project staff could

Exhibit II.9

Instruction Offered by UBMS Projects, by Subject Area^a^

+---------------------------------+-----------------+-----------------+
|                                 | UBMS            | Regular Upward  |
|                                 |                 | Bound           |
|                                 | Summer 1994     |                 |
|                                 |                 | 1992^b^         |
+---------------------------------+-----------------+-----------------+
| **English/Language Arts**       |                 |                 |
+---------------------------------+-----------------+-----------------+
| > Writing/Composition           | 93%             | 100%            |
+---------------------------------+-----------------+-----------------+
| > Literature                    | 60              | 83              |
+---------------------------------+-----------------+-----------------+
| > Reading Comprehension and     | 65              | 98              |
| > Vocabulary                    |                 |                 |
+---------------------------------+-----------------+-----------------+
| > English as a Second Language  | 13              | 11              |
+---------------------------------+-----------------+-----------------+
| > foreign Language              | 54              | 35              |
+---------------------------------+-----------------+-----------------+
| > Other                         | 9               | 13              |
+---------------------------------+-----------------+-----------------+
| **Mathematics**                 |                 |                 |
+---------------------------------+-----------------+-----------------+
| > Pre-Algebra                   | 36              | 82              |
+---------------------------------+-----------------+-----------------+
| > Algebra I                     | 69              | 96              |
+---------------------------------+-----------------+-----------------+
| > Algebra II                    | 81              | 95              |
+---------------------------------+-----------------+-----------------+
| > Geometry                      | 80              | 95              |
+---------------------------------+-----------------+-----------------+
| > Precalculus                   | 80              | 80              |
+---------------------------------+-----------------+-----------------+
| > Calculus                      | 52              | 58              |
+---------------------------------+-----------------+-----------------+
| > Statistics^c^                 | 17              | ^c^             |
+---------------------------------+-----------------+-----------------+
| > Trigonometry^c^               | 7               | ^c^             |
+---------------------------------+-----------------+-----------------+
| > Other                         | 9               | 24              |
+---------------------------------+-----------------+-----------------+
| **Computers**                   |                 |                 |
+---------------------------------+-----------------+-----------------+
| > Programming                   | 43              | 47              |
+---------------------------------+-----------------+-----------------+
| > Applications/Software Use     | 85              | 79              |
+---------------------------------+-----------------+-----------------+
| > internet/Web Page Design^c^   | 7               | ^c^             |
+---------------------------------+-----------------+-----------------+
| > Other                         | 7               | 6               |
+---------------------------------+-----------------+-----------------+
| **Science**                     |                 |                 |
+---------------------------------+-----------------+-----------------+
| > Physics                       | 76              | 63              |
+---------------------------------+-----------------+-----------------+
| > Biology                       | 87              | 89              |
+---------------------------------+-----------------+-----------------+
| > Chemistry                     | 81              | 81              |
+---------------------------------+-----------------+-----------------+
| > Earth Science                 | 48              | 66              |
+---------------------------------+-----------------+-----------------+
| > Other                         | 15              | 19              |
+---------------------------------+-----------------+-----------------+
| **Social Science/History**      |                 |                 |
+---------------------------------+-----------------+-----------------+
| > History                       | 11              | 47              |
+---------------------------------+-----------------+-----------------+
| > Geography                     | 9               | 24              |
+---------------------------------+-----------------+-----------------+
| > Sociology                     | 4               | 17              |
+---------------------------------+-----------------+-----------------+
| > Psychology                    | 8               | 15              |
+---------------------------------+-----------------+-----------------+
| > Government/Civics             | 9               | 40              |
+---------------------------------+-----------------+-----------------+
| > Other                         | 8               | 13              |
+---------------------------------+-----------------+-----------------+
| **Electives/Non-Academic        |                 |                 |
| Courses**                       |                 |                 |
+---------------------------------+-----------------+-----------------+
| > Performing Arts               | 31              | 53              |
+---------------------------------+-----------------+-----------------+
| > Art                           | 26              | 53              |
+---------------------------------+-----------------+-----------------+
| > Journalism                    | 28              | 52              |
+---------------------------------+-----------------+-----------------+
| > Speech/Public Speaking        | 48              | 59              |
+---------------------------------+-----------------+-----------------+
| > Physical Fitness              | 56              | 69              |
+---------------------------------+-----------------+-----------------+
| > Other                         | 6               | 26              |
+---------------------------------+-----------------+-----------------+

^a^ UBMS projects offer instruction in many areas besides math and
science, either to meet regulatory requirements or simply to ensure that
their program will interest and benefit students in many ways.

^b^ 1992 non-bridge summer programs or 1992-93 academic year.

^c^ Neither survey listed statistics, trigonometry, or internet or Web
page design, but enough project directors specified them under "other
courses" that we present data on these courses separately.

[Sources]{.smallcaps}: 1998 survey of UBMS projects, Fasciano and
Jacobson (1997:39).

Exhibit II.10

Noninstructional Services Offered by UBMS Projects

+----------------------------------+-----------------+-----------------+
|                                  | UBMS            | Regular Upward  |
|                                  |                 | Bound           |
|                                  | Summer 1994     |                 |
|                                  |                 | 1992^a^         |
+----------------------------------+-----------------+-----------------+
| **College preparation/skills**   |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Campus visits                  | 74%             | 98%             |
+----------------------------------+-----------------+-----------------+
| > Adjusting to college living    | 98              | 92              |
+----------------------------------+-----------------+-----------------+
| > ACT/SAT preparation            | 65              | 97              |
+----------------------------------+-----------------+-----------------+
| > PSAT/PLAN or PACT preparation  | 24              | 73              |
+----------------------------------+-----------------+-----------------+
| > Help with financial aid or     | 80              | 100             |
| > scholarships^b^                |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Assistance with college        | 78              | 99              |
| > applications                   |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Assistance with financial aid  | 72              | 100             |
| > applications                   |                 |                 |
+----------------------------------+-----------------+-----------------+
| **Career/employment assistance** |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Site visits to employers^c^    | 65              | 59              |
+----------------------------------+-----------------+-----------------+
| > On-campus (employers or career | 63              | 78              |
| > representatives)               |                 |                 |
+----------------------------------+-----------------+-----------------+
| **Project-related work           |                 | 49^d^           |
| experience**                     |                 |                 |
+----------------------------------+-----------------+-----------------+
| > JTPA job                       | 0               | ^d^             |
+----------------------------------+-----------------+-----------------+
| > Work-study job                 | 4               | ^d^             |
+----------------------------------+-----------------+-----------------+
| > Math or science internships    | 24              | ^d^             |
+----------------------------------+-----------------+-----------------+
| > Job through other partnerships | 7               | ^d^             |
+----------------------------------+-----------------+-----------------+
| **Field trips to**               |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Academic science or math       | 98              | ^e^             |
| > facilities                     |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Non-academic science or math   | 94              | ^e^             |
| > facilities                     |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Conduct math- or               | 85              | ^e^             |
| > science-related field work     |                 |                 |
+----------------------------------+-----------------+-----------------+
| > Other                          | 93              | ^e^             |
+----------------------------------+-----------------+-----------------+

^a^1992 nonbridge summer programs or 1992-93 academic year.

^b^ In the regular Upward Bound grantee survey, this item was phrased,
"Identify sources of financial aid."

^c^ In the regular Upward Bound grantee survey, this item was phrased,
"Site visit to employers or job shadowing."

^d^Although the regular Upward Bound grantee survey had three separate
items about JTPA, work-study and other partnerships, the results were
reported only in the aggregate, and it did not ask about math or science
internships (Fasciano and Jacobson 1997:54).

^e^the regular Upward Bound grantee survey asked about field trips of
varying lengths, not the destinations.

[Sources]{.smallcaps}: 1998 survey of UBMS projects, Fasciano and
Jacobson (1997:54).

have reasonably expected that those other programs were assisting
students in preparing for college.

### 5. Academic-Year offerings

While UBMS projects also provided services to students during the
academic year in the mid-1990s, these services were minimal compared to
UBMS summer services and also typically far less numerous and less
intense than academic year services provided in regular Upward Bound.
During the 1994-95 academic year, about one-third of UBMS projects
provided tutoring or study sessions, and just over half provided
assistance with college applications (see Exhibit

Exhibit II.11

Academic Year Services Offered by UBMS Projects, 1994-95

![](media/image6.wmf)[Source]{.smallcaps}: 1998 survey of UBMS projects.

II.11). The average UBMS project provided about three types of these
services during 1994-95.[^24] Not surprisingly, geographic proximity to
their participating students influenced whether UBMS projects provided
certain services during the academic year. UBMS projects were
substantially more likely to provide tutoring and workshops during the
academic year if a relatively large percentage of their participants
lived in the same city or town as the program host.[^25]

### 6. Instructional Approaches

In 1994, the most common instructional approach taken by UBMS projects
was the provision of instruction through courses in separate subjects.
Four out of five UBMS projects offered courses in separate subjects (see
Exhibit II.12, Panel A). In three out of four UBMS projects, the primary
method of instruction was either the provision of these courses (37
percent) or the combination of these courses with interdisciplinary
courses (also 37 percent, see Exhibit II.12, Panel B).

However, UBMS projects frequently employed other instructional methods.
The majority of projects (63 percent) offered interdisciplinary courses,
and in a large minority (35 percent), at least some students worked on a
large project or experiment that spanned multiple academic subject areas
(see Exhibit II.12, Panel A).

UBMS projects vary considerably in how they sort students into classes
or groups. in the summer of 1994, about half of projects placed their
students in instructional groups based on proficiency level (37 percent)
or grade level (16 percent). About one-fourth placed students with
diverse proficiency levels in the same group to facilitate learning
(presumably the learning of less proficient students), and the remaining
projects grouped students by their interests or in some other way.

UBMS participants do not spend most of their time in traditional
lecture-style classes. At the average project during the summer of 1994,
only one-fourth of the time was spent in lecture-style classes like
those offered in most schools. The remaining time was spent in small
group, teacher-led instruction (32 percent), laboratories (29 percent),
computer-based instruction (12 percent), and other settings (4 percent).

## D. INTENSITY AND QUANTITY OF SERVICES

The services that UBMS projects offer and the length of their summer
residential summer programs suggest that these projects offer intensive
programs that provide students with a "large dose" of services, at least
for one summer. Furthermore, larger doses of effective services may
yield larger impacts than smaller doses, as we found for regular Upward
Bound (Myers et al. 2004). Summary measures of program intensity
presented in this section indicate that UBMS projects offer an intensive
program that might be expected to improve the math and science
preparation of program participants.

UBMS is a resource-intensive program. Program grants to UBMS projects
provided an average of approximately \$4,800 per student in FY 2004 (see
Exhibit II.13). This is comparable to funding for regular Upward
Bound---approximately \$4,500 per student---and much more expensive than
most precollege programs. UBMS funding supports an extensive package of
instruction and services, as described earlier in the chapter.

Participants devote a substantial amount of time to the program, and
most of this time is spent on academics. At the average UBMS project in
the summer of 1994, students spent about 29 hours per week receiving
instruction and almost 11 hours per week on tutoring and homework. Thus,
participating in UBMS over the summer is somewhat like having a
full-time job requiring 40 hours per week. Because the vast majority of
UBMS summer programs last six weeks,[^26] participants at the average
project spend 240 total hours on academics during the summer.

Exhibit II.12

Instructional Methods Used by UBMS Projects, 1994

![](media/image7.wmf)

![](media/image8.wmf)

[Note:]{.smallcaps} Percentages may not sum to 100 percent due to
rounding.

[Source]{.smallcaps}: 1998 survey of UBMS projects.

Exhibit II.13

![](media/image9.wmf)Per-Capita Funding for UBMS Projects, FY 2004

[Source]{.smallcaps}: U.S. Department of Education Web page
(www.ed.gov/programs/triomathsci/funding.html and
www.ed.gov/programs/trioupbound/funding.html accessed June 2006).

The amount of time devoted to core program activities varied across UBMS
projects. For example, four projects reported that students spent over
40 hours per week in instruction alone, while a handful of projects
reported estimates of less than 10 hours per week. Estimates of time
spent on tutoring or homework ranged from 0 to 20 hours per week. And
the estimates of average total time spent on academics during the summer
varied considerably, from under 100 hours at some projects to over 340
hours at some others.

Most UBMS participants seem to stay for the entire summer. At the
average project in the summer of 1994, 94 percent of participants
completed all the requirements of the program. A few projects reported
completion rates of 50 percent or less, but this does not necessarily
indicate a high dropout rate. Students might have attended the summer
program for its full length but be counted as failing to "complete all
the requirements of the program"---the wording in the
questionnaire---because they did not, for example, turn in all their
assignments.

Some students participate for multiple summers or receive services
during the academic year. As indicated earlier, about 70 percent of UBMS
projects operating in summer 1994 allowed students to return from the
previous summer. Among those projects, an average of 35 percent of 1994
participants had also participated during 1993.[^27] Furthermore, some
UBMS projects extended students' exposure to the program beyond the
summer by providing services during the academic year (see Section C.4
for more details).

# III. The Impacts of the Upward Bound Math-Science Program

UBMS was established to increase economically disadvantaged students'
achievement in high school math and science courses, to increase the
likelihood that they would major in math and science in college, and
ultimately to increase their representation in math and science careers.
Until now, no rigorous studies have measured the extent to which the
program achieves its goals. In this chapter, we assess the effects of
UBMS on important outcomes for program participants four to eight years
after graduating from high school.

This chapter presents our estimates of the effects of UBMS on (1) high
school preparation for college and for majoring in math and science; (2)
college enrollment, highest level of college attended, college
selectivity, years of college completed and highest degree earned; and
(3) field of study in college. These estimates are based on information
collected in 2002 or earlier, and another round of data collection is
currently underway. Given many students in our sample may have, for
example, selected a major or completed a degree between 2002 and 2004,
the findings in this report should be treated as an initial assessment
of the effects of the program. Our final assessment will be based on the
data collected in 2004.

The evidence suggests that in addition to improving students' grades in
high school, UBMS increases the likelihood of:

-   Taking chemistry and physics in high school

-   Enrolling in more selective four-year colleges and universities

-   Completing a postsecondary degree in a math and science field

The effects of UBMS are nontrivial and in the expected direction. For
example, UBMS increased the likelihood of earning a bachelor's degree in
a math or science field four to eight years after high school graduation
from 12 percent to 20 percent. in the next report on UBMS, we will weigh
the size of the program's benefits against its costs.

Because UBMS participants in our sample were not randomly selected to
participate in the program, the impact estimates presented in this
chapter may suffer from selection bias. However, in designing the study,
we identified several likely sources of bias and have addressed them
through a combination of data collection and statistical methods
described later in the chapter. Therefore, the research design is
stronger than in the typical nonexperimental study, and as a result, the
impact estimates should be more credible.

## A. Study Design

UBMS provides intensive academic enrichment in math and science. Like
regular Upward Bound projects, most UBMS projects also offer some
assistance in preparing for college, such as assistance with college
applications (see Chapter II for more details). The combination of
intensive academic enrichment in math and science with college
preparation assistance suggests that UBMS might have positive effects on
the following outcomes for program participants: (1) performance in high
school, especially in math and science courses; (2) postsecondary
attendance, persistence, and completion; and (3) the likelihood of
completing a postsecondary degree in a math and science field.
Therefore, we designed the analysis to answer the following three
research questions:

1.  What are the effects of UBMS participation on student performance in
    high school overall and in math and science courses in particular?

2.  What are the effects of UBMS participation on college attendance,
    attendance at different types of colleges and universities, years of
    college, and college completion?

3.  What are the effects of UBMS participation on the likelihood of
    completing a degree in math or science?

We have designed the analysis to measure the impacts of UBMS on two
important subgroups: students who had previously participated in regular
Upward Bound and students who had not participated. UBMS participants
who previously participated in regular Upward Bound may have received a
large dose of precollege services and academic preparation before
participating in UBMS. However, most of the other UBMS participants
entered UBMS without having received such intensive services. It is
reasonable to expect UBMS to have larger effects on the students who had
not previously received intensive services: students who have already
received them may have already received the boost they needed to
succeed. On the other hand, UBMS participants may be better prepared to
benefit from their participation if they have previously participated in
intensive services. in our analysis, we compute the effects separately
for students who had previously participated in regular Upward Bound and
students who had not participated to assess whether the effects of UBMS
depend on the amount of precollege services students have received to
that point. Regular Upward Bound is just one of the other programs in
which UBMS participants could have participated. However, in our
analysis, regular Upward Bound deserves special consideration because
many UBMS participants participate in regular Upward Bound, and because
few other programs are as intensive as UBMS.[^28]

It is important to clarify we have not attempted to measure the effects
of UBMS versus no precollege services. The analysis was designed to
measure the effects of participating in UBMS relative to what students
would have participated in otherwise---which might include regular
Upward Bound---*not* the effects of participating in UBMS relative to no
program participation. Most UBMS participants in our sample did not
participate in regular Upward Bound, and only a few other precollege
programs are as intensive as Upward Bound. Therefore, most UBMS
participants would have participated in less intensive precollege
services if they had not participated in UBMS.

In this section, we describe the design of the analysis used to measure
the impacts of UBMS. The impact analysis is based on a matched
comparison group that attempts to reduce two types of bias common to
many nonexperimental studies: selection bias and bias attributable to
different data collection protocols for the participant and comparison
groups. The strength of the impact analysis rests on three features of
its design:

1.  How we selected our initial samples, particularly the comparison
    sample

```{=html}
<!-- -->
```
4.  How we collected "baseline" (preprogram) information on the two
    samples

5.  How we used that information to select a matched comparison group

In the remainder of this section, we describe these three features of
the study and our approach to estimating the effects of UBMS on student
participants.

### 1. Selecting the Samples

For the impact analysis, we obtained our sample of program participants
from the projects themselves. In 1998, we contacted the 62 Upward Bound
Math and Science Centers (MSCs) that were operating at that time and
that had been operating between 1993 and 1995.[^29] From these MSCs, we
requested lists of the students who had participated in their program in
the summer of 1993, the summer of 1994 or the summer of 1995.[^30] To
reduce the costs of collecting the necessary data, we selected one out
of every four of the students from these lists for our analysis sample.

A primary feature of any nonexperimental evaluation is the choice of a
comparison group. Experiments yield the best comparison groups because
differences in outcomes between treated and untreated cases cannot be
attributed to selection bias. In the absence of an experiment, the
strength of an evaluation depends on the comparability between the
participant and comparison groups.

The most convenient comparison group for the impact analysis is also a
compelling one---and the one we used. For the comparison group, we
selected students from the evaluation of regular Upward Bound who
reported that they had not participated in an MSC.[^31] In doing, so, we
selected a comparison group with three desirable attributes:

1.  ***Like UBMS participants, comparison students applied to
    participate in Upward Bound.*** Therefore, both UBMS participants
    and comparison students revealed a high level of motivation to
    pursue precollege services. This provides some protection against a
    common source of selection bias in nonexperimental studies---bias
    from comparing more motivated participants to less motivated
    nonparticipants.

```{=html}
<!-- -->
```
6.  ***Like UBMS participants, comparison students met the federal
    eligibility requirements to participate in some type of Upward Bound
    program---regular or math and science.*** The federal eligibility
    requirements are the same for both regular Upward Bound and UBMS. To
    be included in the sample for the evaluation of regular Upward
    Bound---the sample from which we selected our comparison group---a
    student must have applied to a regular Upward Bound project and been
    determined eligible to participate. Therefore, both UBMS
    participants and comparison students in our sample must either have
    come from "low-income" families (income below 150 percent of the
    poverty line) or potential "first-generation" families (neither
    parent had earned a bachelor's degree).

7.  ***Like UBMS participants, many comparison students would have met
    project-specific eligibility requirements imposed by some MSCs.***
    Moore (1997) indicates that MSCs often apply additional admissions
    criteria in selecting applicants---criteria that include a minimum
    grade point average (GPA) in math and science. in this chapter, we
    show that many comparison students met the same criteria: many were
    successfully "matched" to UBMS participants who took similar courses
    and earned similar grades in math and science in ninth grade.

While the regular Upward Bound sample is a useful comparison group for
measuring the effects of UBMS, it is not a perfect one. Data from the
1990s suggest that MSCs typically had more stringent minimum GPA
requirements than regular Upward Bound projects; and MSC staff---who
typically had recent experience with regular Upward Bound
participants---reported that UBMS participants tended to be "more
serious about school" than regular Upward Bound participants (Moore
1997, p. 26). Therefore, UBMS participants might fare better than
regular Upward Bound participants even without any assistance from UBMS,
and simple differences in mean outcomes between the UBMS participant and
comparison groups may overstate the effects of UBMS due to selection
bias.

To reduce selection bias, we selected a matched comparison sample from
the regular Upward Bound sample. More specifically, we matched each UBMS
participant to one or more regular Upward Bound sample members with
similar characteristics based on data collected from student surveys and
transcripts.[^32] See Section 3 for more details.

### 2. Collecting Baseline Data

The baseline variables constructed for the impact analysis characterize
members of the two samples early in high school before UBMS participants
in our sample entered UBMS. These variables are critical because they
allow us to account---through a combination of matching and regression
adjustments---for many preexisting differences between the two groups
that might otherwise bias our impact estimates. To collect the
information necessary to create baseline variables, we conducted student
surveys and collected high school transcripts (see Appendix A).

An important strength of the study's design is that the data collection
strategy was similar for both samples. Heckman et. al. (1997) argue that
different survey questionnaires for the participant and comparison
samples can be an important source of bias in nonexperimental studies.
In this evaluation, we developed the initial survey questionnaire for
UBMS participants from the baseline survey questionnaire for the regular
Upward Bound sample. Therefore, the survey questions from which we
constructed baseline variables were often identical and always similar
for the two samples.[^33] Furthermore, the approach to collecting and
coding high school transcripts was the same for the two samples.
Therefore, it is unlikely that differences in the data between the two
samples have biased the impact estimates presented in this chapter.

The baseline variables for the impact analysis fall into the following
three categories: (1) demographic and family characteristics, (2)
participation in other precollege programs and (3) ninth-grade academic
achievement in math and sciences and more generally (see Exhibit III.1).
We believe that the measures of students' ninth-grade academic
achievement are critical to the strength of the study. Given the
findings in earlier reports, it seems entirely possible that even among
students with similar demographic and family characteristics, students
who participated in UBMS might have a higher academic aptitude and
interest in math and science than students who participated in regular
Upward Bound. Therefore, we use information on ninth-grade courses taken
and ninth-grade grade point average---overall and specifically math and
science---to control for differences between the two samples in early
high school achievement.

Exhibit III.1

Baseline Variables

+----------------+--------------------------------+--------------------+
| Category       | Variables                      | Source             |
+----------------+--------------------------------+--------------------+
| Demographic    | Sex                            | Initial surveys    |
| and family     |                                |                    |
|                | Race and ethnicity             |                    |
|                |                                |                    |
|                | Census region                  |                    |
|                |                                |                    |
|                | Native English speaker         |                    |
|                |                                |                    |
|                | Mother's time in the U.S.      |                    |
|                |                                |                    |
|                | Sibling participated in Upward |                    |
|                | Bound                          |                    |
|                |                                |                    |
|                | High school cohort             |                    |
+----------------+--------------------------------+--------------------+
| Prior program  | Sample member participated in  | Initial survey and |
| participation  | regular Upward Bound           | project records    |
+----------------+--------------------------------+--------------------+
| Ninth-grade    | GPA in math and science        | High school        |
| achievement    |                                | transcripts        |
|                | GPA in other subjects          |                    |
|                |                                |                    |
|                | Math course taken              |                    |
|                |                                |                    |
|                | Science course taken           |                    |
+----------------+--------------------------------+--------------------+

N[ote]{.smallcaps}: The initial surveys were conducted in 1999 for UBMS
participants and in 1992-94 for comparison students in the regular
Upward Bound sample. To identify UBMS participants who had previously
participated in regular Upward Bound, we used responses to the 1999
initial survey; to identify comparison students who participated in
regular Upward Bound, we used participation information provided by
projects. We collected high school transcripts in 2000 and 2003 for UBMS
participants and in 2000 and earlier years for comparison students.

### 3. Selecting a Matched Comparison Sample for the Impact Analysis

The data collected on UBMS participants and regular Upward Bound sample
members are useful in identifying many similarities between the two
groups---and some differences as well (see Exhibit III.2). In both
groups, 37 percent of the students were African American and 11 percent
had a sibling who had participated in Upward Bound. The two groups were
similar with respect to many other characteristics as well, including
the percentage taking algebra in ninth grade. However, the two groups
exhibit differences on several dimensions. For example, UBMS
participants were more likely to be male and tended to have higher
grades than regular Upward Bound sample members.

To address possible selection bias, we selected a matched comparison
group from the regular Upward Bound sample using propensity score
matching methods.[^34] The goal in

Exhibit III.2

Summary Statistics from the Baseline Variables

(Percentage unless otherwise noted)

  -------------------------------- -------------- ------------ ------- ------------ ----
  Characteristic                   UBMS           Regular              Matched      
                                   Participants   Upward Bound         Comparison   
                                                  Sample               Group        

                                                                                    

  **Participated in Regular Upward 18             55\*\*\*             18           
  Bound**                                                                           

                                                                                    

  **Female**                       59             72\*\*\*             59           

                                                                                    

  **Race and Ethnicity**                                                            

  African American                 37             37                   37           

  White                            25             34\*\*\*             30           

  Hispanic                         18             20                   16           

  Other race                       20             9\*\*\*              17           

                                                                                    

  **Region**                                                                        

  Northeast                        11             9                    12           

  Midwest                          23             19\*\*               28           

  South                            40             45\*\*               35           

  West                             25             26                   24           

                                                                                    

  **Entry to High School**                                                          

  1991-92                          32             25\*\*\*             28           

  1992-93                          39             49\*\*\*             37           

  1993-94                          29             26                   35           

                                                                                    

  **Other Characteristics**                                                         

  Native English speaker           80             87\*\*\*             86\*\*       

  Mother in U.S. most of her life  79             87\*\*\*             83\*         

  Siblings in Upward Bound         11             11                   12           

                                                                                    

  **Ninth-Grade Math Course**                                                       

  Lower than algebra               16             32\*\*\*             14           

  Algebra                          55             53                   59           

  More than algebra                29             14\*\*\*             27           

                                                                                    

  **Ninth-Grade Science Course**                                                    

  Biology, chemistry, or physics   37             26\*\*\*             37           

                                                                                    

  **Ninth-Grade GPA (mean)**                                                        

  Math and science                 2.69           1.65\*\*\*           2.71         

  Other subjects                   3.24           2.64\*\*\*           3.25         

                                                                                    
  -------------------------------- -------------- ------------ ------- ------------ ----

\*/\*\*/\*\*\* Significantly different from UBMS participants at the
0.10 / 0.05 / 0.01 level.

matching was to select a matched comparison sample from the regular
Upward Bound sample such that the distributions of the baseline
variables for the UBMS participant sample and the matched comparison
sample were similar.[^35] Matching was conducted separately for sample
members who had previously participated in regular Upward Bound and for
those that had not:

-   UBMS participants who had previously participated in regular Upward
    Bound were matched to members of the treatment group in the
    evaluation of regular Upward Bound.

-   UBMS participants who had not previously participated in regular
    Upward Bound were matched to regular Upward Bound control group
    members who did not participate in regular Upward Bound.

Regular Upward Bound sample members that were matched to at least one
UBMS participant were included in the matched comparison sample.
[^36]^,^ [^37]

Using matching procedures, we were able to select a matched comparison
group that is highly similar to the sample of UBMS participants on many
dimensions (see the last column of Exhibit III.2). Only two of the
differences between the groups are statistically significant---the
difference in the percentage of sample members who are native English
speakers and the percentage whose mother has lived in the United States
for all or almost all of her life. Furthermore, given the number of
baseline variables, two is a small number of significant differences: we
would expect about two significant differences even if the differences
between the two groups were purely random. While there is no guarantee
that matching removed all unmeasured differences between the two
samples, matching removed differences on a broad range of baseline
variables---differences that might otherwise bias the impact estimates.

### 4. Estimating the Impacts of UBMS Participation

To measure the effects of UBMS participation on participating students,
we used a regression-based approach that allows us to (1) adjust for the
small remaining differences between the UBMS participant sample and the
matched comparison group and (2) increase the precision of our impact
estimates. The regression models yield estimates of the effect of UBMS
on students who participated in the program.[^38]

We estimated the effects of UBMS for the entire sample and separately
for selected subgroups of students. The effects of UBMS may depend on
the amount of other precollege services received. Therefore, we present
separate impact estimates for those who participated in regular Upward
Bound and those who did not participate. Furthermore, we present
separate estimates by sex and race since achievement in math and science
often varies along these dimensions, and the effects of UBMS may vary
along the same dimensions.[^39]^,^[^40]

## B. Analysis and Findings

In this section, we present the estimated effects of UBMS on students'
outcomes. Consistent with program objectives, participation in UBMS
seems to promote postsecondary study in math and science. Specifically,
our estimates suggest that UBMS participation (1) raises grades in high
school and course-taking in chemistry and physics, (2) increases
enrollment in more selective four-year institutions and the number of
years of college completed, and (3) raises the likelihood of both
pursuing postsecondary studies and completing a four-year degree in math
or science. In contrast, UBMS does not affect course-taking in advanced
math in high school.

### 1. The Effect of UBMS on High School Outcomes

The UBMS program is designed to strengthen participants' math and
science skills. MPR conducted follow-up student surveys and collected
high school transcripts to determine whether UBMS participation
increased the amount of course work in math and science as well as total
high school credits, high school graduation and GPA in math and science
and overall.[^41]

UBMS raised the likelihood of taking upper-level high school courses in
science but not in math (see Exhibit III.3). Our estimates indicate that
UBMS increased the likelihood of taking upper-level science courses in
high school, raising the percentage of students taking chemistry from 78
to 88 percent and raising the percentage of students taking physics from
43 to 58 percent.[^42] The estimated effects of UBMS on the likelihood
of taking algebra II, trigonometry, precalculus and calculus are
statistically insignificant.

It is not clear why UBMS affected course-taking in the sciences but not
in math. One possibility involves differences between the type of
instruction that MSCs offer in math and the type of instruction they
offer in sciences. Almost two-thirds of MSCs reported offering
interdisciplinary classes, and about a third of them offered instruction
that revolved around a large project or experiment (see Chapter II).
However, findings from Moore (1997) suggest that in MSCs, laboratories
and hands-on instruction were much more common in science instruction
than in math instruction, and that MSC instructors used math to help
student prepare for careers in science and technology (see Moore 1997,
pp. 41-42). It is possible that the combination of hands-on instruction
offered only in science with the science orientation of MSC instructors
is responsible for UBMS's effects on science course-taking.

Additional estimates suggest UBMS raises overall achievement and
educational attainment in high school (see Appendix B). UBMS had a
positive effect on high school grades, increasing the average GPA in
math courses from 2.7 to 2.8, the average GPA in science courses from
2.7 to 2.9 and the average GPA overall. It also raised the graduation
rate from 96 to 99 percent.

UBMS appears to have had larger effects on grades and science course
work for Hispanics than for African Americans (see Appendix B). For
example, UBMS raised the average GPA in math courses from 2.4 to 2.5 for
African Americans and from 2.4 to 2.7 for Hispanics. In addition, UBMS
raised the likelihood of taking chemistry and physics by 7 percentage
points each for African Americans and by 17 and 27 percentage points,
respectively, for Hispanics.[^43] Comparisons of males to females and
students who had participated in regular Upward Bound to other students
revealed few significant differences.

Exhibit III.3

Math and Science Courses Taken in High School

(Percentage of students)

+---------------------+--------------+---------------------+-----+---+
|                     | UBMS         | Matched Comparison  | Imp |   |
|                     | Participants | Group               | act |   |
+---------------------+--------------+---------------------+-----+---+
|                     |              |                     |     |   |
+---------------------+--------------+---------------------+-----+---+
| **Math Courses**    |              |                     |     |   |
+---------------------+--------------+---------------------+-----+---+
| Algebra             | 97           | 97                  | 0   |   |
+---------------------+--------------+---------------------+-----+---+
| Geometry            | 86           | 86                  | 0   |   |
+---------------------+--------------+---------------------+-----+---+
| Algebra II          | 73           | 74                  | -1  |   |
+---------------------+--------------+---------------------+-----+---+
| Trigonometry        | 35           | 30                  | 5   |   |
+---------------------+--------------+---------------------+-----+---+
| Analysis or         | 36           | 31                  | 5   |   |
| Precalculus         |              |                     |     |   |
+---------------------+--------------+---------------------+-----+---+
| Calculus            | 26           | 24                  | 2   |   |
+---------------------+--------------+---------------------+-----+---+
|                     |              |                     |     |   |
+---------------------+--------------+---------------------+-----+---+
| **Science Courses** |              |                     |     |   |
+---------------------+--------------+---------------------+-----+---+
| Biology             | 97           | 98                  | -1  |   |
+---------------------+--------------+---------------------+-----+---+
| Chemistry           | 88           | 78                  | >   |   |
|                     |              |                     |  10 |   |
|                     |              |                     | \*\ |   |
|                     |              |                     | *\* |   |
+---------------------+--------------+---------------------+-----+---+
| Physics             | 58           | 43                  | 15  |   |
|                     |              |                     | \*\ |   |
|                     |              |                     | *\* |   |
+---------------------+--------------+---------------------+-----+---+

\*/\*\*/\*\*\* Statistically significant at the 0.1 / 0.05 / 0.01 level.

N[otes]{.smallcaps}: Impact may not exactly equal the difference between
UBMS participants and the matched comparison group due to rounding. See
Table B.1 in Appendix B for more details and additional high school
outcomes and Tables B.2 through B.4 for estimates for subgroups of UBMS
participants.

### 2. The Effect of UBMS on Postsecondary Attendance, Persistence, and Completion

While UBMS focuses on preparing students to major in math and science
and to complete a degree in a math and science field, a person must
enroll in college before choosing a major and must complete college to
earn a degree in a math and science field. Even if UBMS had little
effect on students' choice of major, UBMS might be a cost-effective
strategy to increase college enrollment and completion for disadvantaged
students. Therefore, we assess whether UBMS promotes postsecondary
attendance, persistence, and completion before examining its effects on
college major.

The evidence suggests that UBMS participation increases the likelihood
of attending a postsecondary institution. Estimates for the matched
comparison group suggest that 90 percent of UBMS participants would have
attended a postsecondary institution if they had not participated in
UBMS (see Exhibit III.4). This indicates that UBMS participants are much
more likely to attend college than the average low-income high school
student, and a 90-percent attendance rate leaves little room for
improvement. Despite this, the evidence suggests that UBMS participation
increases the likelihood of attending a postsecondary institution from
90 to 95 percent, although as shown in Appendix C, this result is
sensitive to how postsecondary

Exhibit III.4

Postsecondary Attendance, Highest Level Attended and College Selectivity

(Percentage of students)

  -------------------------- --------------- -------------------- ----------
                             UBMS            Matched Comparison   Impact
                             Participants    Group                

                                                                  

  **Postsecondary            95              90                   5\*\*\*
  Attendance**                                                    

                                                                  

  **Highest Level Attended**                                      

  Four-year college or       82              71                   11\*\*\*
  university                                                      

  Two-year college           11              16                   -5\*\*\*

  Vocational institution     2               4                    -2

                                                                  

  **Most Selective Four-Year                                      
  institution**                                                   

  More selective             33              23                   11

  Less selective             48              48                   0
  -------------------------- --------------- -------------------- ----------

\*/\*\*/\*\*\* Statistically significant at the 0.1 / 0.05 / 0.01 level.

N[otes]{.smallcaps}: Impact may not exactly equal the difference between
UBMS Participants and the Matched Comparison Group due to rounding.
"Most Selective College or University" refers to four-year colleges and
universities only, and some four-year institutions were not classified
in *Barron's Profiles of American Colleges* (2003). See Table B.5 in
Appendix B for more details and additional postsecondary outcomes and
Tables B.6 through B.8 for estimates for subgroups of UBMS participants.

attendance is measured and, specifically, to whether college transcripts
are used to verify attendance.[^44]

The evidence also suggests that UBMS participation increases the
likelihood of attending a four-year college or university from 71
percent to 82 percent. The increase in four-year attendance can be
attributed to increased attendance at more selective schools as UBMS
raises the likelihood of attending such a school from 23 to 33
percent.[^45] It appears that some of this increase in four-year college
enrollment can be attributed to students who would otherwise have
attended only two-year colleges: UBMS reduced the likelihood of
enrolling in a two-year college but not in a four-year college or
university from 16 to 11 percent.

UBMS increased the number of years that students spend in four-year
colleges and universities without increasing the completion rate at
these schools. Our estimates suggest UBMS increased the number of years
enrolled in a four-year institution from 2.4 to 2.9 (see Exhibit III.5).
While UBMS increased the amount of time participants spent in college,
it had no effect on the likelihood of their completing a bachelor's
degree as of the time we last interviewed them in 2002. Given that
almost half of UBMS participants in our sample (47 percent) were
attending four-year colleges and universities when we last interviewed
them, it is too early to reach firm conclusions about the effects of
UBMS on the number of years of college attended and on college
completion. Our final report on the Upward Bound Math-Science Program
will be based on data collected in 2004.

The average effects of UBMS mask some interesting differences between
men and women. for both men and women, UBMS increased the percentage
attending a selective four-year institution---from 23 to 34 percent for
men and from 21 to 32 percent for women. However, UBMS only affected
overall four-year enrollment for women. For women, UBMS increased the
likelihood of attending a four-year institution from 68 to 82 percent
and reduced the likelihood of attending a two-year institution (without
also attending a four-year institution) from 18 percent to 10 percent;
for men, both of these effects were statistically insignificant.

The effects on college completion also differ between men and women. For
women, the positive effect of UBMS on enrollment in four-year
institutions---along with positive effects on credits earned in these
institutions---translates into higher graduation rates from four-year
institutions. UBMS raised the percentage of women earning a bachelor's
degree from 32 to 40 percent. For men, the effect of UBMS on the
percentage completing a bachelor's degree is statistically
insignificant; however, UBMS increased the percentage completing an
associate's degree from 4 to 8 percent.

In the last report from the national evaluation (Myers et al. 2004), we
presented two sets of estimates for postsecondary attendance and highest
level attended---the first based entirely on the information reported by
the student, and the second that required verification of enrollment
from the institution. In computing the second set of estimates, we made
the strong assumption that a student did not attend the school they
reported to us unless we received verification from the school in the
form of a college transcript (or a reason for not providing one that
clearly indicated that the student had attended the school). However,
additional analysis of the data suggests that in most cases for which we
did not receive a transcript, there is no reason to doubt that the
students actually attended the schools they reported. Schools provided a
variety of reasons for

Exhibit III.5

Years of College and Degree Completion

  ------------------------ -------------- ------------------ -------------
                           UBMS           Matched Comparison Impact
                           Participants   Group              

                                                             

  **Years of College**                                       

  Four-year college or     2.9            2.4                0.4\*\*
  university                                                 

  Two-year college         0.4            0.4                0.0

                                                             

  **Degree Completion                                        
  (percent)**                                                

  Bachelor's degree        35             33                 2

  Associate's degree       7              7                  -1
  ------------------------ -------------- ------------------ -------------

\*/\*\*/\*\*\* Statistically significant at the 0.1 / 0.05 / 0.01 level.

N[otes]{.smallcaps}: The comparison group estimates and impact estimates
are regression-adjusted (see Chapter III, Section A.4). Impact may not
exactly equal the difference between UBMS Participants and the Matched
Comparison Group due to rounding. Years of Postsecondary Education were
computed by dividing the number of credits earned by 30. See Table B.5
in Appendix B for more details and additional postsecondary outcomes and
Tables B.6 through B.8 for estimates for different subgroups of UBMS
participants.

not providing transcripts, including confidentiality considerations and
money that the student owed to the school. Therefore, for all outcome
variables except years of college, which are based on information from
college transcripts, the estimates presented in this chapter are based
on the information reported by the students themselves. However, we
conducted a sensitivity analysis to see whether requiring verification
of enrollment affected our impact estimates, and in most cases, it did
not. See Appendix C for more details about the sensitivity analysis.

### 3. The Effect of UBMS on Postsecondary Field of Study

The primary objective of UBMS is to prepare students for postsecondary
studies in math and science. Using information reported by students in
2002, we conducted an analysis to examine whether UBMS participation
increases the likelihood of pursuing postsecondary studies in math and
science or the likelihood of earning a postsecondary degree in a math or
science fields. To determine whether a sample member had pursued
postsecondary studies in math and science, we asked sample members in
the 2002 survey for their "most recent or intended field of study," and
we classified their responses according to the same classification
system used in the National Science Foundation's Scientists and
Engineers Statistical Data System (SESTAT).[^46] Furthermore, for the
analysis, we separated the social sciences from other math and science
fields (which we refer to as "math or science") because the objectives
of the program are more closely tied to the latter than the former.

The evidence suggests that UBMS participation encourages students to
pursue postsecondary studies in math or science and also in the social
sciences. UBMS increased the likelihood of majoring or intending to
major in math or science from 23 to 33 percent overall and from 18 to 28
percent if we focus on majors at four-year colleges and universities
(see Exhibit III.6).[^47] The effects are also positive but smaller in
magnitude for the social sciences: UBMS increased the likelihood of
pursuing postsecondary studies in the social sciences from 7 to 11
percent overall and from 7 to 10 percent if we focus on majors at
four-year institutions.

The evidence also suggests that UBMS participation may encourage
students to complete postsecondary degrees in math or science. However,
with almost half of participants still in college, it is too early to
reach final conclusions about UBMS's effects on college completion
overall or in specific fields. As of the 2002 survey, the effect of UBMS
on the likelihood of earning a degree or certificate in math and science
was statistically insignificant. However, if we focus on four-year
institutions, UBMS altered the types of fields in which students earned
degrees. More specifically, it raised the likelihood of earning a
bachelor's degree in math or science from 6 to 12 percent and reduced
the likelihood of earning a bachelor's degree outside of math or science
or the social sciences from 20 to 14 percent.

The effect of UBMS on the likelihood of majoring in math and science was
larger for men than for women. While UBMS increased the likelihood of
pursuing postsecondary studies in math and science from 38 to 58 percent
for men, the estimated effect for women is much smaller and
statistically insignificant. In contrast, there is no evidence that the
effect of UBMS on the likelihood of completing a math or science degree
differs between the two groups.

## C. interpretation of the Findings

The estimated effects of UBMS paint a fairly consistent picture of UBMS
helping its participants onto an educational path that could lead to
careers in math and science. Our findings suggest that UBMS
participation improves student outcomes in high school and college,
and---consistent with the objectives of the program---increases the odds
of completing a college degree in math and science.

While we took several steps to reduce selection bias, it is certainly
possible that the true effects of participating in UBMS are smaller than
our estimates suggest. As reported earlier in the chapter, UBMS
participants may be more serious about school than regular Upward Bound
participants on average. This difference is reflected in our data: UBMS
participants in our sample had higher GPAs and took more advanced math
and science courses in ninth grade than members of the regular Upward
Bound sample. We accounted for these differences by selecting a matched
comparison group that resembled the UBMS participant sample in both
grades earned and course-taking in ninth grade. However, it is possible
that despite earning similar grades and taking similar courses early in
high school, the UBMS participant sample is somewhat more serious about
school, more serious about math and science, or is different from the
matched comparison group in some other way that would lead our analysis
to overstate the effects of UBMS.

Exhibit III.6

Field of Study

(Percentage of students)

  ---------------------------- -------------- ------------------- ----------
                               UBMS           Matched Comparison  Impact
                               Participants   Group               

  **All Postsecondary                                             
  institutions**                                                  

  Postsecondary Studies                                           

  Math or science              33             23                  10\*\*\*

  Social science               11             7                   4\*

  Other                        42             51                  -9\*\*\*

                                                                  

  Postsecondary Studies                                           
  Completed                                                       

  Math or science              15             12                  3

  Social science               8              4                   3\*

  Other                        21             28                  -7\*\*

  **Four-Year Colleges and                                        
  Universities**                                                  

  Postsecondary Studies                                           

  Math or science              28             18                  10\*\*\*

  Social science               10             7                   4\*

  Other                        36             39                  -3

                                                                  

  Postsecondary Studies                                           
  Completed                                                       

  Math or science              12             6                   6\*\*\*

  Social science               6              4                   3

  Other                        14             20                  -6\*\*
  ---------------------------- -------------- ------------------- ----------

\*/\*\*/\*\*\* Statistically significant at the 0.1 / 0.05 / 0.01 level.

N[otes]{.smallcaps}: Impact may not exactly equal the difference between
UBMS Participants and the Matched Comparison Group due to rounding. See
Table B.9 in Appendix B for more details and additional outcomes and
Tables B.10 through B.12 for estimates for subgroups of UBMS
participants.

While we cannot measure the extent of selection bias, some informed
speculation is helpful in interpreting the impact estimates. As we
indicated earlier, students in both groups---the UBMS participant sample
and the matched comparison group---exhibited some motivation to improve
academically by applying to participate in Upward Bound. Therefore,
motivational differences between the two groups are likely to be small
and unlikely to bias the impact estimates. However, the impact estimates
would overstate the true impacts if:

-   ***Members of UBMS participant sample were higher achievers than
    members of matched comparison group.*** by the end of high school
    and after students in the UBMS sample had participated in UBMS, the
    average GPA was slightly but significantly higher for UBMS
    participants (3.14) than for matched comparison students (3.06).
    Even if the entire difference were attributable to selection bias
    instead of the effects of the program, the average student in each
    group was a B student. Therefore, if the UBMS participant sample
    contains higher achieving students than the matched comparison
    group, the difference seems to be small.

-   ***Members of the UBMS participant sample were better at math and
    science than members of the matched comparison group.*** If students
    in the UBMS participant sample were better at math and science than
    students in the matched comparison group, it is not reflected in
    their math and science course-taking or grades during ninth grade:
    the two groups had similar grades in math and science, were equally
    likely to take algebra or geometry and were equally likely to take
    biology, chemistry or physics. While we did not explicitly examine
    whether UBMS participants were more likely to enroll in the advanced
    sections of courses before entering UBMS, other findings suggest
    that UBMS participants were no more likely to have taken Advanced
    Placement classes in math or science by the time they finished high
    school. Finally, focus groups conducted in 1996 suggest that UBMS
    participants do not view themselves as particular strong in math and
    science (Moore 1997, p. 28). Therefore, the information we have
    collected provides no reason to believe that when they entered UBMS,
    the participant sample was better at math and science than the
    matched comparison group.

-   ***Members of the UBMS participant group were more interested in
    math and science than members of the matched comparison group.***
    None of the information that we extracted from student transcripts
    would suggest that the UBMS participant sample had greater interest
    in math and science. It is possible UBMS participants had greater
    interest in careers in math and science that simply was not
    reflected in their high school course-taking or grades early in high
    school. in 1996 focus groups, many UBMS participants expressed
    interest in pursuing careers in scientific fields, such as
    engineering, medicine, and nursing (Moore 1997, p. 28). However,
    many of these students indicated that their career interests had
    developed just that summer, and the expression of those interests
    could have been influenced by the fact that the focus groups were
    conducted on site at projects that emphasized math and science.

We suspect that the impacts most vulnerable to selection bias are those
that are most closely related to a person's interest in pursuing careers
in a math or science field. To gauge students' interest in math and
science early in high school would probably require conducting
assessments or survey interviews at that time. However, we first
interviewed members of the UBMS participant sample after they had
completed high school. The possibility that UBMS participants might have
had greater interest in pursuing a career in science than matched
comparison students raises the question of whether our estimates
overstate the effects of UBMS on the outcomes that are most closely
related to one's career interests, such as majoring in math or science
in college.

Therefore, while the findings in this report are promising, a note of
caution is appropriate. We speculate that the selection bias is likely
to be largest for outcome variables most closely tied one's interest in
pursuing math and science careers, but it is not possible to measure the
selection bias. While we took several steps to reduce selection bias,
the estimated effects of UBMS may overstate the true effects of the
program.

# References

Cahalan, Maggie, Tim Silva, Justin Humphrey, Melissa Thomas and Kusuma
Cunningham. *Implementation of the Talent Search Program, Past and
Present. Final Report from Phase I of the National Evaluation*. Report
prepared for the U.S. Department of Education, Planning and Evaluation
Service. Washington, D.C.: Mathematica Policy Research, Inc., October
2002.

Curtin, Thomas R., and Margaret W. Cahalan. *A Profile of the Upward
Bound Math-Science Program: 2000-2001*. Washington, D.C.: U.S.
Department of Education, Office of Postsecondary Education, December
2004.

Fasciano, Nancy J., and Jon E. Jacobson. "Grantee Survey Report" in *A
1990s View of Upward Bound: Programs Offered, Students Served, and
Operational Issues*, a monograph prepared by Mathematica Policy
Research, Inc., for the U.S. Department of Education, Office of the
Under Secretary, as part of the National Evaluation of Upward Bound.
Washington, D.C.: May 1997.

Moore, Mary T. *A 1990s View of Upward Bound: Programs Offered, Students
Served, and Operational Issues*. In the monograph of the same name,
prepared by Mathematica Policy Research, Inc., for the U.S. Department
of Education, Office of the Under Secretary, as part of the National
Evaluation of Upward Bound. Washington, D.C., May 1997.

Moore, Mary T. *Developing Math and Science Skills Among Disadvantaged
Youth: A Review of the Upward Bound Precollege Math/Science Centers*.
Report prepared for the U.S. Department of Education, Planning and
Evaluation Service. Washington, D.C.: Mathematica Policy Research, Inc.,
September 1997.

Myers, David. *Measuring the Impact of the Upward Bound Math/Science
Initiative: Feasibility of an Evaluation*. Report for the Department of
Education. Washington, D.C.: Mathematica Policy Research, Inc., May 7,
1997.

Myers, David, Rob Olsen, and Neil Seftor. The Impacts of Upward Bound:
High School Experiences and College Access. Presentation at Annual
Conference, Council for Opportunity in Education, Washington, D.C.,
September 2002.

Myers, David, Robert Olsen, Neil Seftor, Julie Young, and Christina
Tuttle*. The Impacts of Regular Upward Bound: Results from the Third
Follow-up Data Collection*. Report prepared for the U.S. Department of
Education, Policy and Program Studies Service. Washington, D.C.:
Mathematica Policy Research, Inc., April 2004.

Myers, David, and Allen Schirm. *The Impacts of Upward Bound: Final
Report for Phase I of the National Evaluation*. Prepared for the U.S.
Department of Education, Planning and Evaluation Service. Washington,
D.C.: Mathematica Policy Research, Inc., April 1999.

National Science Foundation. "Table C-11. Employed U.S. Scientists and
Engineers, by Level and Broad Field of Highest Degree Attained,
Race/Ethnicity, and Years Since Degree: 1995." Available at
http://srsstats.sbe.nsf.gov/preformatted-tables/1995/tables/tbC11.pdf,
accessed June 2006.

Rubin, Donald B. "Using Propensity Scores to Help Design Observational
Studies: Application to the Tobacco Litigation." *Health Services and
Outcomes Research Methodology*, 2, pp. 169-188, 2002.

U.S. Census Bureau. *Statistical Abstract of the United States: 2001*.
Washington, D.C., November 2001.

U.S. Census Bureau. "DP-1. General Population and Housing
Characteristics: 1990." Online data table from 1990 Census.
(http://factfinder.census.gov/servlet, accessed June 2006).

U.S. Census Bureau. "QT-P1D. Age and Sex of the Black Population: 1990."
Online data table from 1990 Census
(http://factfinder.census.gov/servlet, accessed June 2006).

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics. *NAEP 1999 Trends
in Academic Progress: Three Decades of Student Performance.* NCES
2000-469, by J.R. Campbell, C.M. Hombo, and J. Mazzeo. Washington, D.C.,
2000.

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics. *Digest of
Education Statistics 1998*. NCES 1999-036. Washington, D.C., 1999.

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics. *Confronting the
Odds: Students At Risk and the Pipeline to Higher Education*. NCES
98-094. Washington, D.C., 1997.

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics*. The Condition of
Education 1996*. NCES 96-304. Washington, D.C., 1996a.

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics*. The Condition of
Education 1994*. NCES 94-149. Washington, D.C., 1994.

U.S. Department of Education, Office of Educational Research and
Improvement, National Center for Education Statistics. *National
Education Longitudinal Study of 1988. High School Seniors' instructional
Experiences in Science and Mathematics*. NCES 95-278. Washington, D.C.,
1996b.

# APPENDIX A

# DATA COLLECTION

#  {#section-2 .unnumbered}

This appendix describes and assesses the procedures for collecting the
data that we used to construct student outcome measures for the impact
analysis presented in Chapter III. These data come from two different
sources:

1\. The fourth follow-up survey of students

2\. Secondary and postsecondary transcripts

This appendix focuses on procedures for obtaining completed interviews
in the fourth follow-up survey and for collecting academic transcripts.

## A. Fourth Follow-Up Survey of Students

The fourth follow-up survey was conducted between April 2001 and
December 2002. It was designed to collect information on secondary and
postsecondary educational outcomes approximately five to seven years
after scheduled completion of high school.

### 1. Data Collection Modes

One week before we began interviewing, we sent a letter to all study
participants. The letter indicated that we would call them to complete
an interview for an important study, and it encouraged them to
participate. In addition, the letter indicated that we would pay them
\$10 for completing the interview.

Most interviews were administered using computer-assisted telephone
interviewing (CATI). CATI interviews took about 30 minutes to complete.
When a CATI interview was not possible, we attempted to obtain a
completed questionnaire through the mail. Study participants were also
offered the option of completing the survey on the Web. In June 2001,
questionnaires were mailed to study participants that could not be
reached by telephone. Three additional follow up mailings were conducted
after the first mailing, with the last set of questionnaires being sent
out in January 2002.

### 2. Locating

Throughout the data collection period, locating staff used services such
as LexisNexis and Internet databases to obtain updated addresses and
phone numbers for study participants that were difficult to reach.

### 3. Incentives

Financial incentives for survey completion were used to obtain a high
response rate. Study participants were offered a \$10 incentive for
participating in the survey. Incentive checks were mailed after the
sample member completed the interview.

### 4. Response Rates

The eligible sample consisted of 1,759 UBMS participants and 2,830
sample members from the evaluation of regular Upward Bound. (See Chapter
III for more details on the samples.) We obtained completed interviews
for 1,425 UBMS participants and 2,146 regular Upward Bound sample
members for response rates of 81 percent and 76 percent, respectively
(see Table A.1).

## B. Transcript Data Collection

Secondary and postsecondary transcripts were collected between July 2002
and March 2003. Academic transcripts provided the primary source of
information on postsecondary achievement. Transcript requests were made
from institutions that were reported by sample members in the fourth
follow-up survey of students and in earlier surveys.

### 1. Preparation for Requesting Transcripts

Information about students' secondary and postsecondary enrollment was
primarily obtained from follow-up interviews. Students reported the
secondary and postsecondary institutions that they had attended.
Secondary transcripts were only requested from UBMS sample members
selected for the impact analysis; postsecondary transcripts were
requested from all sample members---both UBMS and regular Upward
Bound---who reported or confirmed having attended a particular
postsecondary institution.[^48]

To obtain mailing addresses for the schools that were attended by sample
members, we matched schools that were reported by survey respondents to
directories of secondary and postsecondary schools maintained by the
U.S. Department of Education. Secondary schools were matched to the
Common Core of Data (CCD); postsecondary schools were matched to the
integrated Postsecondary Education Data System (IPEDS).[^49]

### 

### ![](media/image10.wmf)

### 2. Procedures for Requesting Transcripts

Each school was sent a transcript request packet that included:

-   A letter, printed on Department of Education letterhead, which
    explained the purpose of the study and the reason we were requesting
    transcripts.

-   A statement of Authorization and Confidentiality, which cited the
    *Family Educational Rights and Privacy Act* and included questions
    and answers regarding consent and confidentiality.

-   A transcript checklist of all the materials that we requested from
    the school, including student transcripts, a course catalog, grade
    descriptions and a transcript reimbursement form, which would
    indicate the reimbursement that the school required for providing
    the requested transcripts.

-   A postage-paid business reply envelope for sending the transcripts.

-   A disclosure notice to be placed in each student's file, indicating
    that a copy of his or her transcript was released to Mathematica
    Policy Research as an agent to the U.S. Department of Education.

### 3. Follow-Up Procedures 

For schools that did not respond to our initial request for transcripts,
we mailed another request for student transcripts. These mailings were
done periodically as we tracked the schools that had not yet sent the
requested transcripts and corrected requests that contained errors.

As the targeted end date for collecting transcripts approached,
interviewers started calling schools directly to inquire about the
status of our requests. Many schools responded to these calls by faxing
us the requested transcripts. When the school indicated that they could
not provide one or more of the requested transcripts, the interviewer
completed a problem sheet indicating the reason. The reason generally
fell into one of the following categories:

-   ***The student was never enrolled at the school according to the
    school's records.*** When this occurred, our first response was to
    call the school and provide more information on the student (e.g.,
    provide or verify date of birth and dates of attendance) to see if a
    transcript could be located with additional information. In many
    cases, the school was able to locate and provide transcripts once
    additional information was provided. In other cases, the school
    provided some information that helped us determine where we might
    obtain the needed transcripts.[^50] If the school had no record of
    the student having ever attended and we were unable to obtain
    additional information, we marked the case as an invalid request.

-   ***Transcripts were held by the school district.*** Some schools
    only held the transcripts of currently enrolled students and all
    other transcripts were sent to the school district. in this
    situation, the school would sometimes forward the request packet to
    the district. Other times, the school returned the materials to us,
    and we sent them to the school district.

-   ***The student transferred to another school.*** When the student
    had transferred to another school, a transcript was requested from
    the school to which the student had transferred. In some cases, the
    registrar or school secretary forwarded the request materials to the
    transfer school. in other cases, the request materials were sent
    back to us and we sent a new request to the transfer school.

-   ***The school would not release any transcript without student's
    written consent.*** A few schools returned the transcript request
    materials with no transcripts, indicating that they required written
    consent from each student whose transcript we were requesting. A
    problem sheet was completed for these cases, and they were forwarded
    to the survey manager for follow-up. As a first step, the survey
    manager called the school to explain that, as an agent of the
    Department of Education, Mathematica Policy Research was authorized
    to collect student transcripts for the purposes of this study and
    that, according to the laws of *FERPA*, schools are permitted to
    release student transcripts to the Department of Education without
    the written consent of students participating in the study. It was
    also explained that students had given verbal consent over the
    telephone or written consent when they completed the mail survey,
    and that we did not request transcripts for any students who refused
    consent. Some schools agreed to send the requested transcripts upon
    hearing this explanation. Others reiterated that signed consent was
    required by school policy. in this case, we sent written consent
    forms to the students for them to sign and return to Mathematica so
    that we could obtain their student transcript for the impact study.
    A postage-paid return envelope was included with the consent form. A
    small number of students did sign and return the consent form, but
    most of the letters came back unopened because we no longer had a
    valid address for the student.

-   ***The school would not release transcripts without advance
    payment.*** In these cases, we sent a check to cover the cost of
    each transcript, along with a list of the students whose transcripts
    we were requesting.

-   ***The school would not release a transcript until the student paid
    an outstanding debt.*** In some cases we were eventually able to
    obtain these transcripts as students paid whatever bills they owed
    the school. When the debt remained unpaid, however, there was no way
    we could get the transcript. These cases were marked as unfilled
    requests.

### 4. Response Rates

From the samples used in the impact analysis described in Chapter III,
1,365 students reported having attended at least one postsecondary
institution that could be matched to IPEDS. for each of these 1,365
students, we requested a transcript from each of the postsecondary
institutions that he or she reported attending. In total, we requested
2,029 transcripts. We received 1,821 of the 2,029 transcripts requested
(90 percent), and we obtained a complete transcript record---that is,
transcripts for all postsecondary institutions attended---for 1,109
students (see Table A.2).

![](media/image11.wmf)

# 

# APPENDIX B

# PROGRAM IMPACTS

![](media/image12.wmf)

![](media/image13.wmf)

![](media/image14.wmf)

![](media/image15.wmf)

![](media/image16.wmf)

![](media/image17.wmf)

![](media/image18.wmf)

![](media/image19.wmf)

![](media/image20.wmf)

![](media/image21.wmf){width="8.916666666666666in"
height="5.548611111111111in"}

![](media/image22.wmf)

![](media/image23.wmf)

# 

# APPENDIX C

# SENSITIVITY OF IMPACTS TO AN ALTERNATIVE MEASURE OF POSTSECONDARY ATTENDANCE

###   {#section-7 .unnumbered}

### A. Verification of Students' Postsecondary Attendance

In Chapter III, we presented estimates of the effects of UBMS on
postsecondary attendance based on self-reported
attendance---specifically, the schools that sample members reported
attending after high school. However, in some instances, sample members
may not have actually attended the schools they reported. For example,
some may have reported schools that they planned to attend but never
attended, and others may have reported schools at which they
participated in a noncredit program but were not enrolled as students.
In this appendix, we conduct an analysis to determine if the impact
estimates presented in Chapter III are sensitive to an alternative
measure of postsecondary attendance, one that requires verification by
the postsecondary institution that students reported attending.

Myers et al. (2004) present the results of this analysis in the text of
the report along with the analysis based on self-reported data, which
effectively provided two impact estimates for each outcome. When one of
the two impact estimates was statistically insignificant, the results
were characterized as inconclusive because at that time, we were unable
to persuade ourselves that one method was more reliable than the other.
However, we have since conducted an investigation of the verification
process, and the results of this analysis indicate that on average, the
self-reported measures of postsecondary attendance are more accurate
than the measures requiring verification by the institution. While this
may seem counterintuitive, in constructing postsecondary attendance
measures that required verification, we assumed that students did not
attend the schools they reported unless attendance was verified by the
school, typically through the provision of a transcript. However, the
transcript collection process was not originally designed to verify
attendance.[^51] And in most instances when a school was unable or
unwilling to provide a transcript, a careful examination of the data
suggests that the student probably did attend the school---or at the
very least, the school gave us no reason to doubt the student had
attended when explaining why they would not provide a transcript.

The verification process worked as follows. In the third and fourth
follow-up surveys, students reported all postsecondary institutions that
they had attended since high school. We then attempted to match all
reported schools to the 1997-98 Integrated Postsecondary Education Data
System (IPEDS) maintained by the National Center for Education
Statistics (NCES) to determine whether they met NCES's definition of a
postsecondary institution and to obtain school contact information.
Transcript requests were then sent to all schools a student reported
attending that we were able to match to IPEDS and schools that did not
initially respond were followed up. If a school provided a transcript
that we requested for a student in the sample, then the student clearly
attended that school. Furthermore, some of the reasons given by school
staff for not providing transcripts can be treated as verification of
attendance, for example, a college indicating that it could not provide
a transcript for a student because he or she owed money to the school.
In many instances, however, the reason given for not providing
transcripts does not clearly indicate whether the student attended the
school. For example, some schools required written consent from the
students themselves even though the law does not require it, and we
typically obtained only oral consent (see Appendix B); in these cases,
lack of verification casts no doubt on the student's self-reported
attendance.

The information obtained while collecting transcripts is therefore
useful in verifying attendance in some but not all cases. In the vast
majority of cases considered (approximately 80 percent) we were able to
verify the student's attendance. However, for approximately 220
students, the fact that we were unable to verify attendance for at least
one of their reported postsecondary institutions changed at least one
outcome of interest.[^52] In investigating these cases, we found little
evidence that contradicted a student's report of their postsecondary
schools attended. One concern was that some Upward Bound participants
would report the school at which they attended an Upward Bound summer
bridge program but did not actually enroll; we found no evidence that
this happened.[^53] Similarly, there was little evidence that many
students reported attending institutions that are not classified by
IPEDS as postsecondary institutions, and, in fact, non-IPEDS
institutions such as Job Corps programs were excluded through the IPEDS
match. The lack of verification more often appeared to be due to schools
that would not release transcripts without written consent, ambiguities
in the exact campus attended by the student within a large state system,
or insufficient information about the school reported by the student to
know which school to contact for transcript information. More often than
not, lack of verification seemed to mainly reflect limitations of the
verification process rather than inaccuracies in the information
provided by students.

B. Differences in Impacts

This section presents estimates of the impacts of Upward Bound that use
information only on schools that we were able to verify the student
attended. These estimates make the strong assumption that lack of
verification of attendance implies that the student did not attend the
school in question or any school like it.

Because most attendance reported by students was verified through
transcript receipt, most of the impacts presented here are very similar
to those estimated using information on all schools that students
reported attending. There are a few differences between the estimates
presented here and the estimates presented in Chapter III, but there is
probably only one worth noting: the finding that UBMS has a positive
effect on the likelihood of attending some type of postsecondary
institution is sensitive to the verification process. If we require
verification of postsecondary attendance, the estimated effect becomes
statistically insignificant (compare Table C.1 to Table B.5). For the
full set of estimates from the sensitivity analysis, see Tables C.1 -
C.4.

![](media/image24.wmf)

![](media/image25.wmf)

![](media/image26.wmf)

![](media/image27.wmf)

# 

# 

# 

# APPENDIX D

# SAMPLE SIZES AND STANDARD ERRORS

![](media/image28.wmf)

![](media/image29.wmf)

![](media/image30.wmf)

![](media/image31.wmf)

![](media/image32.wmf)

![](media/image33.wmf)

![](media/image34.wmf)

![](media/image35.wmf)

![](media/image36.wmf)

![](media/image37.wmf)

![](media/image38.wmf)

![](media/image39.wmf)

![](media/image40.wmf)

![](media/image41.wmf)

![](media/image42.wmf)

![](media/image43.wmf)

[^1]: Ideally, socioeconomic measures such as income would be used to
    define groups, rather than race or ethnicity. For most education
    outcomes of interest, however, data are not presented on different
    income groups. Because racial and ethnic minorities are
    disproportionately lower-income (U.S. Census Bureau 2001:40), data
    based on race and ethnicity offer a reasonable, albeit imperfect,
    estimate of economically disadvantaged students' educational
    experiences.

[^2]: The following subjects were classified as math or science:
    biological sciences and life sciences, computer and information
    sciences, engineering, engineering-related technologies,
    mathematics, and physical sciences and science technologies.

[^3]: This requirement may have stemmed from concern that too much math
    and science instruction in high school is provided by teachers
    teaching out of their own field.

[^4]: While the last report from the national evaluation shows that
    about 70 percent of regular Upward Bound participants would have
    attended a postsecondary institution even if they had not
    participated in Upward Bound (Myers et al. 2004), only 53 percent of
    1992 high school graduates from the lowest SES quartile attended a
    postsecondary institution by 1994 (U.S. Department of Education
    1997).

[^5]: Because the sample was not selected until 1998, we restricted the
    sample to participants at UBMS projects that were still operating
    that year: obtaining lists of participants from programs that were
    no longer operating in 1998 would have been nearly impossible.

[^6]: These comparison students were selected from the treatment group
    for the evaluation of regular Upward Bound. For a more thorough
    discussion of how the treatment group was selected, see Myers et al.
    (2004), Appendix A.

[^7]: These comparison students were selected from the control group for
    the evaluation of regular Upward Bound. For a more thorough
    discussion of how the control group was selected, see Myers et al.
    (2004), Appendix A.

[^8]: While the 1999 survey was conducted four to six years after our
    sample had participated in the program, most of the baseline
    information collected---including sex, race, and ethnicity---is
    time-invariant.

[^9]: We did not adjust (weight) for survey nonresponse, reasoning that
    the number of nonrespondents was low enough to eliminate any serious
    concerns about data representativeness. Also, rarely did more than
    three UBMS projects fail to respond to any particular item on the
    questionnaire.

[^10]: We excluded the one project that reported serving only veterans
    in 1998. Note that veterans' projects were also excluded from the
    survey of regular Upward Bound grantees, so the comparisons that are
    made in this chapter between the two types of Upward Bound programs
    are based on Upward Bound projects that did not exclusively serve
    veterans.

[^11]: Unless noted otherwise, the results for 1998 were generally
    similar to those for 1994.

[^12]: By 1998 the average MSC had almost 26 staff, including 9
    instructors.

[^13]: An additional 12 percent had their highest degree in the social
    sciences.

[^14]: These practices changed dramatically by 1998, when only 7 percent
    of UBMS projects prohibited all students from returning from the
    prior summer, and 71 percent prohibited students from returning
    unless they met certain criteria. The questionnaire did not address
    the specific types of criteria that projects imposed.

[^15]: Most of these questions were focused on 1998 participants, so the
    analysis in this section is focused on 1998 instead of 1994.

[^16]: Data based on all UBMS projects operating in 1998 and all regular
    Upward Bound projects operating in 1992.

[^17]: In comparison, at 87 percent of regular Upward Bound projects
    operating in 1992--93, one racial and ethnic group accounted for a
    majority of participants.

[^18]: In regular Upward Bound during the summer of 1992, 20 percent of
    participants were entering 12th grade, 32 percent were entering 11th
    grade, 31 percent were entering 10th grade, and 16 percent were
    entering 9th grade.

[^19]: In addition, the proportion of rising juniors ranged from 0 to 72
    percent, and the proportion of rising sophomores ranged from 0 to 60
    percent.

[^20]: Only 4 percent of UBMS projects in 1994 had a residential
    component shorter than the summer program, but by 1998, the rate had
    increased to 11 percent.

[^21]: Eighty-seven percent of regular Upward Bound projects rated
    "academic improvement" as their most or second most important goal
    in 1993. If the regular Upward Bound grantee survey had also listed
    "academic improvement in math and science," it is possible that some
    respondents would have cited that as one of their top two goals:
    Fasciano and Jacobson (1997) characterized 37 percent of regular
    Upward Bound projects as having a strong emphasis on math and
    science.

[^22]: By 1998, the relative focus on enrichment was even greater, with
    83 percent of UBMS projects citing enrichment as a major emphasis
    and only 1 percent citing remediation.

[^23]: It is not clear why some UBMS projects apparently had such a
    small number of instructional offerings. One possibility is that
    their instructional approach centered on interdisciplinary courses
    or large projects that covered multiple subject areas. However, only
    3 of the 12 projects that reported five or fewer math and science
    offerings in 1994 also reported that their primary instructional
    method was interdisciplinary courses or large projects and
    experiments. A second possibility is underreporting of instructional
    offerings by project directors.

[^24]: By 1998-99, the percentage of UBMS projects regularly providing
    these services had increased substantially. For example, the
    percentage providing tutoring or study sessions rose from 81 percent
    to 93 percent, and the percentage providing assistance with college
    applications rose from 52 percent to 78 percent. This probably
    reflects the establishment of more locally oriented UBMS projects.

[^25]: For example, more than three-quarters of UBMS projects with a
    relatively large percentage of participants from the local area
    (above the median) offered tutoring during the academic year, versus
    less than half of UBMS projects with a relatively small percentage
    (below the median).

[^26]: Only five UBMS projects reported a program of a different
    length---three had a five-week session, one had a seven-week
    session, and one had an eight-week session.

[^27]: In summer 1998, not only did more UBMS projects (over 90 percent)
    allow students to return, but at those projects an average of 52
    percent of students had also participated in summer 1997.

[^28]: Moore (1997), Appendix A includes a list of 28 alternative math
    and science precollege programs. However, some other precollege
    programs without a specific precollege focus, such as regular Upward
    Bound, may share some common features with UBMS. Annual per student
    costs were available for 17 of the 28 programs; of these 17, only
    two were more expensive per student-year than UBMS.

[^29]: While we would have been interested in obtaining lists of
    students from projects that were no longer operating in 1998, we
    believed that it would be very difficult to obtain such lists. If
    MSCs that closed before 1998 operated less effective programs than
    those that remained open, then the results presented in this chapter
    may overstate the effectiveness of MSCs operating between 1993 and
    1995.

[^30]: We received participant lists from all but one of these MSCs.

[^31]: If UBMS projects admitted eligible applicants on a fairly random
    basis, rejected applicants would probably constitute the best
    comparison group. However, MSCs are not required to select randomly
    from eligible applicants, and the evidence suggests that they do
    not. Case studies of 14 MSCs in the mid-1990s suggest that many MSCs
    impose additional eligibility requirements beyond the federal
    requirements, including an interest in math and science (Moore
    1997). Therefore, it is unlikely that rejected applicants would have
    the virtues of a randomly selected control group for the purposes of
    this evaluation. Furthermore, the difficulty in obtaining lists of
    program participants from MSCs several years after they participated
    in UBMS suggests that many MSCs would not have been able to provide
    information on rejected eligible applicants.

[^32]: Very few of students in the regular Upward Bound sample
    participated in UBMS, and those who reported participating in UBMS
    were excluded from the comparison group.

[^33]: Two differences in the two surveys were addressed in constructing
    baseline variables for the MSC impact analysis: (1) the two survey
    questionnaires were different, and (2) the surveys were conducted at
    different times---after high school for the participant group and
    early in high school for the comparison group. To address the
    differences in the survey questionnaires, we used data from survey
    questions that are either identical or almost identical. To address
    the difference in the timing of the surveys, we used data from
    survey questions only when the timing of the survey was unlikely to
    affect the answer to the question.

[^34]: Many studies adjust for baseline differences of these types using
    standard covariance adjustments---that is, by controlling for these
    variables in a regression analysis. However, the differences between
    the UBMS participant sample and the regular Upward Bound sample are
    too large to expect covariance adjustment to be reliable. Regression
    adjustments are likely to be unreliable if the means of the
    propensity scores are more than half a standard deviation apart
    (Rubin 2002). For the participant sample, the mean and standard
    deviation of the propensity score are 0.58 and 0.22, respectively;
    for the comparison sample, the mean and standard deviation of the
    propensity score are 0.25 and 0.23, respectively. Therefore, the
    difference in mean propensity scores is more than one standard
    deviation, and regression adjustments alone are likely to be
    unreliable.

[^35]: More specifically, the goal is to ensure that the distributions
    of the baseline variables for the UBMS participant sample and the
    matched comparison sample are similar enough such that covariance
    adjustments will produce reliable impact estimates

[^36]: The propensity score matching and the impact analysis are
    restricted to sample members who entered high school between 1991
    and 1993. While some UBMS participant sample members entered high
    school before 1991 and after 1993, relatively few comparison sample
    members did so. Furthermore, high school cohort is related to the
    likelihood of participating in the program and to the outcomes of
    interest because earlier cohorts had more time to enter college and
    select a field of study by the time they were interviewed for the
    evaluation. Therefore, to protect the internal validity of the
    study, we focused the analysis on students who entered high school
    between 1991 and 1993.

[^37]: UBMS participants could be matched to more than one regular
    Upward Bound sample member, and regular Upward Bound sample members
    could be matched to more than one UBMS participant. To be matched, a
    pair of students must satisfy the following condition: the
    difference between matched students in the log odds of the
    propensity scores was less than 0.20 times the standard deviation of
    the log odds. Smith and Todd (2003) refer to this type of matching
    as "radius matching."

[^38]: We regressed each outcome on a set of control variables and an
    indicator of whether the student participated in UBMS. The control
    variables included the variables used in selecting the matched
    comparison group: prior participation in regular Upward Bound,
    siblings in Upward Bound, sex, race, ethnicity, mother's native
    language and immigrant status, high school cohort, region of the
    country, and several variables describing academic achievement in
    ninth grade, including GPA---separately for math and science courses
    and for other courses---math course taken, and science course taken.
    For continuous variables, such as number of college credits, we
    estimated linear regression models; for categorical outcomes, such
    as whether the sample member pursued postsecondary studies in math
    or science, we estimated logistic regression models or "logit"
    models. In estimating standard errors, we accounted for clustering
    by project and used the Taylor series linearization methods employed
    by the SUDAAN statistical analysis software.

[^39]: Other reports from the National Evaluation of Upward Bound have
    shown that the effects of regular Upward Bound vary with the
    educational expectations of student applicants (see Myers et al.
    2004 and Myers and Schirm 1999). In particular, regular Upward Bound
    greatly increased the likelihood of attending a four-year college or
    university for students who did not expect to earn a bachelor's
    degree when they applied to participate in regular Upward Bound.
    Because we first interviewed UBMS participants after they finished
    participating in UBMS, we lack information on their educational
    expectations when they applied to participate in UBMS. Therefore, we
    cannot assess whether the effects of UBMS vary with students'
    educational expectations before participating.

[^40]: In the national evaluation, we have estimated the effects of
    regular Upward Bound separately by eligibility category (low-income,
    first-generation, or both). However, student data was provided by
    project directors when they submitted students to Mathematica for
    random assignment. Because we did not conduct random assignment for
    the evaluation of UBMS, we lack data on eligibility for UBMS
    participants. While we did obtain rosters of prior participants from
    UBMS projects in 1998, we did not request data on eligibility
    because we did not believe that projects could easily provide them
    for students who participated three or more years ago.

[^41]: With the exception of the outcome variables describing high
    school completion, which were constructed from student survey
    responses, the high school outcomes were constructed using data from
    high school transcripts.

[^42]: These increases in part reflect increases in the total number of
    courses taken in high school: UBMS raised the average number of high
    school credits earned from 25 to 26.

[^43]: For African Americans, the estimated effect on the likelihood of
    taking physics was not statistically significant.

[^44]: This 95-percent figure suggests that almost all UBMS participants
    attend a postsecondary institution in the first few years after high
    school. In contrast, findings from a recent analysis of performance
    data submitted by UBMS projects seem to suggest a much lower rate of
    postsecondary attendance. Analyzing the performance data, Curtin and
    Cahalan (2004) find that only 68 percent of UBMS participants who
    graduated from high school in 1999-2000 were reported as attending a
    postsecondary institution in 2000-01. The difference between this
    figure and the enrollment rate presented in this report reflects
    substantial differences in how the two estimates were derived.
    According to information provided to us by ED, the estimate in
    Curtin and Cahalan (2004) indicates the percentage of former program
    participants for whom UBMS project directors could confirm
    postsecondary attendance. The estimate was not intended to measure
    the postsecondary enrollment rate per se, and it understates the
    true rate for UBMS participants for two related reasons. First, the
    estimate includes in the denominator but not the numerator UBMS
    participants from projects that were no longer funded and,
    therefore, did not submit performance data. Second, it excludes from
    the numerator about one-fifth of the participants who are included
    in the denominator because UBMS project staff were unable to confirm
    college attendance for these former participants. For our estimate,
    in contrast, we exclude from both the numerator and denominator the
    UBMS participants in our sample for whom we are missing data due to
    survey nonresponse and, then, use standard weighting techniques to
    adjust for the missing observations.

[^45]: If a school was rated as "most competitive," "highly
    competitive," or "very competitive," then we classified the school
    as "more selective." If a school was rated as "competitive," "less
    competitive," "noncompetitive," or "special," or the school was
    either excluded from Barron's or was included but not rated, then we
    classified the school as "less selective." Students were then
    classified as "more selective" school if they attended one or more
    of the "more selective" institutions, and "less selective" if they
    attended one or more of the "less selective" institutions and did
    not attend a "more selective" institution.

[^46]: The fields classified as science and engineering were biological
    sciences, computer science, engineering, mathematics, physical
    sciences, and technical fields. The fields classified as non-science
    and engineering were agriculture, arts, business, education,
    clerical or legal assistance, communications, health-related fields,
    humanities, trade and industry, protective services and consumer or
    personal services. A small number of fields reported by sample
    members could not be classified as either science and engineering or
    nonscience and engineering. (For more details about NSF's
    classification of fields of study, see
    www.nsf.gov/sbe/srs/nsf99337/pdf/appa.pdf.)

[^47]: For convenience, we use the terms "major" and "field of study"
    interchangeably.

[^48]: We did not collect secondary transcripts for regular Upward Bound
    sample members because we had already collected these transcripts
    for a large percentage of the sample in previous waves of data
    collection.

[^49]: Students were asked to provide the name and state of each
    secondary and postsecondary school they attended, but sometimes
    misspellings or incomplete information resulted in some invalid
    requests for student transcripts as schools were matched with an
    incorrect address and transcripts were requested from the wrong
    school. When a school indicated that they could not fill a request
    because they had no record of the student whose transcript we
    requested, it was sometimes due to such mismatches. In these cases,
    we attempted to learn the correct name and address of the school
    where the student was enrolled and make a new transcript request.

[^50]: For example, some school principals and registrars indicated that
    their school was often confused with another school having the same
    or a similar name and suggested that we direct our request to the
    other school. In this case, we would call the alternate school to
    find out if the student was ever enrolled there. If so, we made a
    correction to the database and sent a request to the newly
    identified school.

[^51]: For the final report of this evaluation, which will use
    postsecondary transcript data collected in 2004-05, the transcript
    data collection process has been modified to specifically verify
    attendance if possible.

[^52]: There may be cases in which we were not able to verify a
    student's attendance at all schools but for which that did not
    affect the outcome variables of interest. For example, if we
    verified a student's attendance at a four-year college or university
    but could not verify his or her attendance at a vocational school
    they reported attending, he or she will still be classified as
    having their highest level of schooling as a four-year college or
    university.

[^53]: Students participate in Upward Bound summer bridge programs the
    summer before enrolling in college. Because the survey question
    simply asks students to list all postsecondary institutions attended
    since high school graduation, there was concern that some students
    may list the school at which they participated in a summer bridge
    program.