**The Impacts of Upward Bound Math-Science on Postsecondary Outcomes 7–9 Years After Scheduled High School Graduation**

***Final Report***

****

Prepared by

Neil S. Seftor

Juan Carlos Calcagno

Mathematica Policy Research, Inc.

For

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2010

# This report was prepared for the U.S. Department of Education under Contract Number EA-97-0300001 with Mathematica Policy Research, Inc. The project monitor was Margaret Cahalan in the Policy and Program Studies Service. The views expressed herein do not necessarily represent the positions or policies of the Department of Education. No official endorsement by the U.S. Department of Education is intended or should be inferred.

**U.S. Department of Education**

Arne Duncan

*Secretary*

**Office of the Planning, Evaluation and Policy Development**

Carmel Martin

*Assistant Secretary*

**Policy and Program Studies Service**

Alan Ginsburg

*Director*

2010 

This report is in the public domain. Although permission to reprint this publication is not necessary, the suggested citation is: U.S. Department of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service*, The Impacts of Upward Bound Math-Science on Postsecondary Outcomes 7–9 Years After Scheduled High School Graduation*, Washington, D.C., 2010.

To order copies of this report:

**Write to** ED Pubs, Education Publications Center, U.S. Department of Education, NTIS, P.O. Box 22207, Alexandria, VA 22304. 

Or **fax** your request to 703-605-6794.

Or **e-mail** your request to edpubs@inet.ed.gov.  

Or **call** in your request toll-free to 1-877-433-7827 (1-877-4-ED-PUBS). Those who use a telecommunications device for the deaf (TDD) or a teletypewriter (TTY) should call 1-877-576-7734. If 877 service is not yet available in your area, call 1-800-872-5327 (1-800-USA-LEARN).  

Or **order online** at http://edpubs.ed.gov. 

This report is also available on the Department’s Web site at http://www.ed.gov/about/offices/list/opepd/ppss/index.html.

On request, this publication is available in alternate formats such as Braille, large print, or computer diskette. For more information, please contact the Department’s Alternate Format Center at 202-260-0852 or 202-260-0818.

# AcknowledgmentS

This report on the effects of Upward Bound Math-Science on students’ postsecondary outcomes reflects the contributions of many individuals. The authors would like to thank David Goodwin of the Policy and Program Studies Service at the U.S. Department of Education for his continued support, substantive guidance, and encouragement throughout the study. The authors would also like to thank Maggie Cahalan for her very valuable comments and other guidance. 

David Myers, who directed the national evaluation of Upward Bound for many years, played a key role in shaping the study design, led previous impact analyses. Mary Moore, a principal investigator for the national evaluation, was also instrumental in shaping the study design. Allen Schirm and Peter Schochet provided insightful suggestions throughout the analysis and drafts of the report. 

The authors would also like to thank the individuals who contributed to the data collection and analysis as well as to the production of the report. David DesRoches oversaw the data collection, and Zhanyun Zhao created the survey weights. Karin Zeller constructed the analysis variables and computed the impact estimates, and Mary Grider provided technical assistance with data and programming issues. Jennifer Baskwell produced the document.

# CONTENTS

**Chapter	Page**

		LIST OF EXHIBITS ......................................................................	vii

		EXECUTIVE SUMMARY	x

	I	INTRODUCTION	1

A.	UPWARD BOUND MATH-SCIENCE PROGRAM	2

B.	EVALUATION OF THE UPWARD BOUND MATH-SCIENCE PROGRAM	3

	II	THE OPERATION OF THE UPWARD BOUND MATH-SCIENCE PROGRAM	5

A.	PROJECT HOSTS AND STAFF	6

1.	Host Institutions: Two- and Four-Year Colleges and Universities	6

2.	Summer Program Staff and Project Director	6

B.	ELIGIBILITY, RECRUITMENT AND ENROLLMENT, AND STUDENT CHARACTERISTICS	8

1.	Eligibility	8

2.	Recruitment and Enrollment	10

3.	Student Characteristics	12

C.	PROGRAM DESCRIPTION	14

1.	Setting	15

2.	Goals of the Program	15

3.	Academic Orientation	16

4.	Summer Offerings	17

5.	Academic-Year Offerings	20

6.	Instructional Approaches	22

7.	Intensity and Quantity of Services	24

	III	THE IMPACTS OF THE UPWARD BOUND MATH-SCIENCE PROGRAM	26

A.	STUDY DESIGN	26

1.	Selecting the Samples	27

2.	Collecting Baseline Data	29

3.	Selecting a Matched Comparison Sample for the Impact Analysis	31

**CONTENTS***(continued)*****

**Chapter 	Page**

4.	Estimating the Impacts of UBMS Participation	34

                        B.  DATA SOURCES AND OUTCOMES MEASURES	35

1.	Outcome Measures	35

2.	Data Sources	36

3.	Construction of the Outcome Measures	37

C.	ANALYSIS AND FINDINGS	37

1.	The Effect of UBMS on Postsecondary Attendance, Persistence and Completion	38

2.	The Effect of UBMS on Postsecondary Field of Study	39

	IV	INTERPRETATION OF FINDINGS	49

		REFERENCES	52

		APPENDIX A: DATA COLLECTION AND OUTCOMES MEASURES	A-1

		APPENDIX B: PROGRAM IMPACTS BY SUBGROUPS	B-1

		APPENDIX C: SAMPLE SIZES AND STANDARD ERRORS	C-1

ix

# EXHIBITS

**Exhibit	Page**

	II.1	Types of Institutions That Hosted Upward Bound Math-Science Projects, 1995	7

	II.2	Predominant Racial and Ethnic Group for Upward Bound Math-Science Staff, Summer 1998	9

	II.3	Racial and Ethnic Distribution for Upward Bound Math-Science Staff and Project Directors, Summer 1998	10

	II.4	Targets for Recruiting by Upward Bound Math-Science Projects, 1994	11

	II.5	Geographic Scope of Recruiting by Upward Bound Math-Science Projects, 1994	12

	II.6	Predominant Racial and Ethnic Group for Participants at Upward Bound Math-Science Projects, Summer 1998	14

	II.7	Most Important Goals of Upward Bound Math-Science Projects, Summer 1994	16

	II.8	Academic Orientation of Upward Bound Math-Science Projects	17

	II.9	Instruction Offered by Upward Bound Math-Science Projects, by Subject Area	19

	II.10	Noninstructional Services Offered by Upward Bound Math-Science Projects	20

	II.11	Academic Year Services Offered by Upward Bound Math-Science Projects, 1994–95	21

	II.12	Instructional Methods Used by Upward Bound Math-Science Projects, 1994	23

	II.13	Per-Capita Funding for Upward Bound Math-Science Projects, FY 2004	25

	III.1	Baseline Variables	31

	III.2	Summary Statistics From the Baseline Variables	33

	III.3	Impact of Upward Bound Math-Science on Any Postsecondary Enrollment, Percentage of Students	41

	III.4	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment: Four-Year Institution, Percentage of Students	42

	III.5	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment: Two-Year Institution, Percentage of Students	43

**EXHIBITS***(continued)*

**Exhibit	Page**

	III.6	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment: Other Institution, Percentage of Students	44

	III.7	Impact of Upward Bound Math-Science on Attendance at a Highly-Selective Four-Year Postsecondary Institution, Percentage of Students	45

	III.8	Impact of Upward Bound Math-Science on Total Postsecondary Credits Earned, Percentage of Students	46

	III.9	Impact of Upward Bound Math-Science on Completion of Any Credential and Highest Credential Completed (Intent To Treat), Percentage of Students	47

	III.10	Impact of Upward Bound Math-Science on Postsecondary Field of Study (Intent To Treat), Percentage of Students	48

	B.1	Impact of Upward Bound Math-Science on Any Postsecondary Enrollment by Prior Participation in Regular Upward Bound (ITT)	B-2

	B.2	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment by Prior Participation in Regular Upward Bound: Four-Year Institution (ITT)	B-3

	B.3	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment by Prior Participation in Regular Upward Bound: Two-Year Institution (ITT)	B-4

	B.4	Impact of Upward Bound Math-Science on Highest Level of Postsecondary Enrollment by Prior Participation in Regular Upward Bound: Other Institution (ITT)	B-5

	B.5	Impact of Upward Bound Math-Science on Attendance at a Highly Selective Four-Year Postsecondary Institution by Prior Participation in Regular Upward Bound (ITT)	B-6

	B.6	Impact of Upward Bound Math-Science on Total Postsecondary Credits Earned by Prior Participation in Regular Upward Bound (ITT)	B-7

	B.7	Impact of Upward Bound Math-Science on Completion of Any Credential and Highest Credential Completed by Prior Participation in Regular Upward Bound (ITT)	B-8

	B.8	Impact of Upward Bound Math-Science on Postsecondary Field of Study by Prior Participation in Regular Upward Bound (ITT)	B-9

**EXHIBITS***(continued)*

**Exhibit	Page**

	C.1	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.3	C-2

	C.2	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.4	C-3

	C.3	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.5	C-4

	C.4	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.6	C-5

	C.5	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.7	C-6

	C.6	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.8	C-7

	C.7	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.9	C-8

	C.8	Sample Sizes and Standard Errors for Reported Impact Estimates: Exhibit III.10	C-9

	C.9	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.1	C-10

	C.10	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.2	C-11

	C.11	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.3	C-12

	C.12	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.4	C-13

	C.13	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.5	C-14

	C.14	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.6	C-15

	C.15	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.7	C-16

	C.16	Sample Sizes and Standard Errors for Reported Impact Estimates in Appendix B, Exhibit B.8	C-17

# Executive Summary

This final report updates the report *Upward Bound Math-Science: Program Description and Interim Impact Estimates*published in 2007 (Olsen et al. 2007). The 2007 interim report contained descriptive findings from a survey of Upward Bound Math-Science (UBMS) grantees from the late 1990s at the time of the study’s initiation and impact estimates through the period four to six years after expected high school graduation of the study sample. The current report presents impact estimates for the period seven to nine years after scheduled high school graduation. For context purposes we include descriptive information from the initial reports that gives a picture of the UBMS program as it was operating shortly after the time when the study sample members were participating in UBMS (1993–1995). It should be noted that the study sample and results represent the UBMS program as it was operating in the early years of its initiation.

For many years, policy-makers have been concerned by the relatively low levels of academic achievement by economically disadvantaged K–12 students in math and science, by the underrepresentation of disadvantaged college students in math and science majors, and by the underrepresentation of people from disadvantaged groups in math and science careers. Although racial gaps in math and science test scores narrowed somewhat in the 1970s and 1980s, substantial gaps persisted through the 1990s to the present (U.S. Department of Education 1996a).  

To help address these disparities, the U.S. Department of Education (ED) established a math and science initiative in 1990 within Upward Bound, a federal grant program designed to provide disadvantaged high school students with skills and experiences that will prepare them for college success. The initiative, referred to as Upward Bound Math-Science (UBMS), awards grants to institutions—largely colleges and universities—to operate UBMS projects. These projects were designed to differ from “regular” Upward Bound projects in several respects. To ensure that participants receive an intensive math and science precollege experience, UBMS projects provide instruction that includes hands-on experience in laboratories and computer facilities and at field sites. Opportunities are also provided to learn from mathematicians and scientists who are employed at the host institution or engaged in research or applied science in other institutions in the community. In addition to year-round services, a six-week summer program providing intensive instruction in laboratory science as well as mathematics through pre-calculus is also offered.

Initially, ED funded 30 UBMS projects. By FY 2006, there were 126 UBMS projects serving 6,707 students at a total cost of $34 million. Therefore, the annual cost per student was approximately $4,990 in FY06. More than 80 percent of UBMS projects are hosted by four-year colleges and universities; most of the rest are hosted by two-year colleges (Curtin and Cahalan 2004).  

Participants in UBMS must meet the same eligibility requirements as regular Upward Bound students: students must (a) belong to families classified as low-income (taxable income of no greater than 150 percent of the poverty line) or (b) be a potential first-generation college student (neither parent has a bachelor’s degree) or (c) have a need for academic support, as determined by the grantee. Some students who participate in UBMS summer programs are referred from regular Upward Bound programs and then return to those programs during the academic year, but the vast majority of UBMS students do not participate at all in regular Upward Bound. However, as would be expected, UBMS projects are more likely to consider students’ interests in math and science when reviewing applications than are most regular Upward Bound projects (Moore 1997). While 25 percent of participants are white, most program participants are from underrepresented minority groups: about 60 percent of participants are African American or Hispanic (Curtin and Cahalan 2004).  

## EVALUATION OF UPWARD BOUND MATH-SCIENCE

Since 1991, Mathematica Policy Research, Inc. (Mathematica) has been conducting the National Evaluation of Upward Bound for ED. The centerpiece of this evaluation has been a random assignment evaluation of regular Upward Bound. In 1997, ED added a new component to the evaluation that is focused on UBMS. In 1998, Mathematica selected a random sample of the students who participated in UBMS between 1993 and 1995 at projects that were still operating at that time. This report constitutes the second of two evaluation reports on UBMS, and it is based on participant surveys and student transcripts collected for this sample between 2003 and 2004. The first report was based on data collected between 2001 and 2002, focusing on high school outcomes and preliminary postsecondary outcomes. 

The evaluation of UBMS has two components: a descriptive analysis and an impact analysis. The descriptive analysis relies primarily on a survey of project directors to describe the resources available to UBMS projects; the types of institutions that host them; the credentials and demographic characteristics of project staff members; recruitment, eligibility and enrollment of students; student characteristics; and program offerings. The impact analysis is designed to measure the effects of UBMS on (a) performance in high school, especially in math and science courses; (b) postsecondary attendance, persistence and completion; and (c) the likelihood of completing a postsecondary degree in mathematics or a scientific field. The interim report (Olsen et al. 2007) presented detailed findings on the effects of UBMS on high school outcomes. By the 1999–2000 data collection on which that report was based, nearly all sample members had progressed out of high school and into college. Therefore, this report focuses only on postsecondary outcomes.

The impact analysis is based on a comparison of UBMS participants with a sample of students that (a) applied to enroll in regular Upward Bound programs in the early 1990s, (b) never participated in UBMS, and (c) have been tracked by Mathematica as part of the national evaluation. This comparison group was selected using propensity modeling to ensure that it had similar characteristics to the sample of UBMS participants, and we controlled statistically for the small remaining differences in these characteristics between UBMS participants and the comparison group. Because some of the matched comparison group sample participated in regular Upward Bound, the report presents separate impact estimates for those who participated in Upward Bound and those who did not participate. 

If UBMS participants are more interested or skilled in math and science than the students in the comparison group, the estimated effects of the program may be subject to “selection bias” and may overstate the true effects of participating in UBMS. Moore (1997) described UBMS participants as a more select group than regular Upward Bound participants based on UBMS students having earned somewhat higher grades and having greater interest in math and science (23, 26) before participating in Upward Bound. Discussions with UBMS and regular Upward Bound staff members revealed that UBMS participants were typically considered “more serious about school” than regular Upward Bound participants (Moore 1997, 26).

However, we believe that the comparison group we selected was the best choice available because random assignment was not possible. Our analysis shows that the participant and comparison groups are similar in other ways such as academic course-taking and achievement before program participation. In addition, we implemented a data collection and analysis plan designed to minimize selection bias (see Chapter III for more details).  

Note that the descriptive findings and impact estimates presented in this report describe the operations and effects of the Upward Bound Math-Science Program as it operated in the mid-1990s. At that time, it was a relatively new program, and some changes have occurred since that time in how UBMS projects operate. In Chapter II, we mention some of these changes as they are reflected in information provided to us by UBMS project directors in a survey of grantees. It is certainly possible that some of the changes in the program since the mid-1990s have influenced the effectiveness of UBMS projects, and the evaluation does not attempt to measure any changes in effectiveness since that time. In this report, we measure the effects of the program on people who participated between 1993 and 1995 and describe the operations of the program at that time. 

## REPORT FINDINGS 

Given the academic services provided by UBMS, it is natural to ask whether participating in UBMS affects the educational outcomes of the students who participate. From our impact analysis, we found that the UBMS program achieved the following:

· ***Increased enrollment at four-year institutions.*** The impact estimates were all positive (ranging from 8.6 to 18.2 percentage points, with an average of 12.0) and significant for attending a four-year institution.

· ***Shifted enrollment from two-year to four-year institutions for Upward Bound participants.*** Almost all impacts on attending a four-year institution are larger for UMBS participants without previous participation in regular Upward Bound. Moreover, the negative effect on enrollment in a two-year college is driven almost exclusively by the impact for students who did not participate in regular Upward Bound. Combined, these findings suggest that UBMS increases four-year enrollment for both regular Upward Bound participants and nonparticipants, but leads to an additional shift from two-year to four-year enrollment for those who did not have prior regular Upward Bound participation. 

· ***Increased enrollment at more selective institutions.***A substantial portion of UBMS participants who attended four-year institutions attended more-selective institutions, with positive (ranging from 12.2 to 22.9 percentage points, with an average of 18.6) and statistically significant differences for all of the measures.

· ***Increased math and science course taking.***Consistent with the primary objective of UBMS, the program increased the number of postsecondary credits earned in math and science from 21.6 to 29.5.

· ***Increased postsecondary degree completion overall and at four-year institutions.***UBMS increased the likelihood of completing any postsecondary credential (ranging from 9.0 to 12.1 percentage points, with an average of 11.0) (Exhibit III.9). This increase in overall postsecondary completion is primarily attributable to an increase in the percentage of sample members whose highest credential was a bachelor’s degree (ranging from 11.8 to 16.8 percentage points, with an average of 15.7, and statistically significant for all of the measures). 

· ***Increased likelihood of  earning a degree in a social science field of study.*** UBMS students showed statistically significant effects on the likelihood of earning a degree or certificate in the social sciences field and in earning a degree in a social science field from a four-year college or university. UBMS students showed positive effects in the direction of increasing the likelihood of majoring in math and physical science fields. However, the effects were not statistically significant.

It is tempting to compare the estimated impacts of UBMS with the estimated impacts of regular Upward Bound presented in earlier reports. However, it is important to recognize that the two studies used different methods: although the evaluation of regular Upward Bound is based on an experimental design, the “gold standard” in evaluation research, the evaluation of UBMS is based on nonexperimental methods that may suffer from selection bias, as described earlier. If the estimated effects of UBMS are inflated because of selection bias, then the impression based on our findings that UBMS is more effective than regular Upward Bound might be attributable to differences in the methods used to estimate the impacts instead of differences in the effectiveness of the two programs.

# 

22

# I. introduction

For many years, policy-makers have been concerned by the relatively low levels of academic achievement by economically disadvantaged K–12 students in math and science, by the underrepresentation of disadvantaged college students in math and science majors, and by the underrepresentation of people from disadvantaged groups in math and science careers. National statistics show that although the math and science test scores gaps between minority students and white students narrowed somewhat in the 1970s and 1980s, gaps in test scores and other educational outcomes persisted through the 1990s to the present:[footnoteRef:2] [2: . Ideally, socioeconomic measures such as income would be used to define groups rather than race or ethnicity. For most education outcomes of interest, however, data are not presented on different income groups. Because racial and ethnic minorities are disproportionately lower income (U.S. Census Bureau 2001, 40), data based on race and ethnicity offer a reasonable, albeit imperfect, estimate of economically disadvantaged students’ educational experiences.] 

***When the Upward Bound initiative started, disadvantaged students took fewer math and science courses in high school.*** In the 1991–92 school year, 57 percent of seniors in the lowest socioeconomic status (SES) quartile took a math course compared with 75 percent of seniors from the highest SES quartile; 37 percent of seniors from the lowest SES quartile took a science course compared with 61 percent of seniors from the highest SES quartile (U.S. Department of Education 1996b). In 1994, only 58 percent of black high school graduates had completed geometry while in high school compared with 73 percent of white high school graduates. In the same year, only 13 percent of black and Hispanic graduates had completed the common triad of science courses—biology, chemistry, and physics—compared with 23 percent of white graduates (U.S. Department of Education 1996a).

***Minority college students were less likely to take math and science courses or earn a degree in math or science.***Ten percent of black college students and 14 percent of Hispanic college students received credit for calculus or advanced math courses in the late 1980s compared with 22 percent of white college students. Sixteen percent of black college students and 21 percent of Hispanic college students earned course credits in chemistry compared with 27 percent of white college students, and 8 percent of black students and 11 percent of Hispanic students earned college credit for physics compared with 18 percent of white students (U.S. Department of Education 1994). Because minority students earned fewer college credits in math and science (biological sciences and life sciences, computer and information sciences, engineering, engineering-related technologies, mathematics, and physical sciences and science technologies) than white students, it is not surprising that they were less likely to earn degrees in those subjects. Black students earned 7 percent of all bachelor’s degrees in 1995–96, but just 7 percent of all bachelor’s degrees in math and science fields. In the same year, Hispanic students earned 5 percent of all bachelor’s degrees, but just 4 percent of all bachelor’s degrees in math and science (U.S. Department of Education 1999).

***Minority students are underrepresented in math and science fields.*** For example, about 8 percent of bachelor’s degrees in 1998 were awarded to blacks, and about 7 percent were awarded to Hispanics; these two groups made up about 12 percent and 9 percent, respectively, of the United States population in the same period (National Science Foundation n.d.; Census Bureau 2001).

## A.	Upward Bound Math-Science Program

To help address these disparities, the U.S. Department of Education (ED) in 1990 established the Upward Bound Math-Science Program (UBMS) within Upward Bound, a federal grant program designed to provide disadvantaged high school students with skills and experiences that would prepare them for college success. UBMS was designed to differ from “regular” Upward Bound in a few key respects. To ensure that participants receive an intensive math and science precollege experience, ED requires UBMS projects to provide instruction that includes hands-on experience in laboratories and computer facilities as well as at field sites. Also provided are the following: opportunities to learn from mathematicians and scientists employed at the host institution or engaged in research or applied science in other institutions in the community; involvement with tutors and counselors who are graduate and undergraduate math and science majors; and a six-week summer program consisting of daily course work and activities, instruction in laboratory science as well as mathematics through pre-calculus (in addition to foreign language, composition and literature, which are also required offerings at regular Upward Bound projects).

Initially, ED funded 30 UBMS projects. By FY 2006, there were 125 UBMS projects serving 6,707 students at a total cost of $32.3 million. Therefore, the annual cost per student—approximately $4,800—is comparable in cost with regular Upward Bound, but much more expensive than other federally funded precollege programs. More than 80 percent of UBMS projects are hosted by four-year colleges and universities; most of the rest are hosted by two-year colleges (Curtin and Cahalan 2004).    

UBMS participants must meet the same eligibility requirements as regular Upward Bound participants: they must (a) come from families that are classified as low income (taxable income not more than 150 percent of the poverty line) or (b) be a potential first-generation college student (neither parent has a bachelor’s degree). Some students who participate in UBMS are referred from regular Upward Bound programs and then return to those programs during the academic year, but the vast majority of UBMS students do not participate at all in regular Upward Bound. However, as would be expected, UBMS projects are more likely to consider students’ interests in math and science when reviewing applications than are most regular Upward Bound projects (Moore 1997). While 25 percent of participants are white, most program participants are from underrepresented minority groups: about 60 percent of participants are African American or Hispanic (Curtin and Cahalan 2004).  

	Despite coming from low-income families, the evidence suggests that on average, UBMS serves students do well in high school and attend college at higher rates than the average low-income student. Data reported by Upward Bound projects suggest that before participating in Upward Bound, UBMS participants earned higher grades on average than regular Upward Bound participants (Curtin and Cahalan 2004). In addition, the national evaluation has shown that regular Upward Bound participants would have attended college at much higher rates than the average low-income student even if they had not participated in Upward Bound (Seftor, Mamun, and Schirm 2009). Nationwide, among all students in eighth grade in 1988, approximately 76 percent enrolled in postsecondary education within about eight years after high school (Ingels et al. 2002). Among disadvantaged students, the national postsecondary enrollment rate was much lower—less than 60 percent for students who were in the lowest SES quartile or whose parents did not attend college. In comparison, 81 percent of Upward Bound applicants assigned to the control group enrolled in postsecondary education within seven to nine years after high school. Therefore, the evidence strongly suggests that UBMS serves high school students who are much more likely to attend college than the average low-income student.

## B.	Evaluation of the Upward Bound Math-Science Program

The legislation establishing Upward Bound authorizes ED to sponsor studies of it, including examinations of program effectiveness. In 1991, ED awarded a contract to Mathematica Policy Research, Inc. (Mathematica) to conduct the national evaluation of Upward Bound. This evaluation has several components, but its signature feature is an experiment to measure the effects of participating in regular Upward Bound. We selected a random sample of Upward Bound projects (excluding UBMS projects). For each of these projects, we randomly assigned eligible applicants to a treatment group, which was offered the chance to participate in the program, or a control group, which was not. The evaluation, which only recently ended, was one of the first to use experimental methods to measure the effects of a federally funded education program.

This report presents the results of an evaluation of the Upward Bound Math-Science Program. In 1997, Mathematica completed two reports on UBMS. One provided a descriptive analysis of the program based primarily on site visits to a representative sample of 14 UBMS projects (Moore 1997). The other provided an assessment of the feasibility of conducting a rigorous evaluation of the effects of UBMS on student outcomes (Myers 1997). When ED awarded a contract to Mathematica in 1997 to extend its evaluation of the effects of regular Upward Bound, it also specified an evaluation of the effects of UBMS. This evaluation consists of two components: a descriptive analysis and an impact analysis. The descriptive analysis relies primarily on a survey of UBMS project directors conducted in the spring of 1998. The analysis is designed to describe the resources available to UBMS projects; the types of institutions that host them; the credentials and demographic characteristics of project staff members; recruitment, eligibility and enrollment of students; student characteristics; and a description of the program, including its goals, academic orientation and instructional methods as well as the intensity and quantity of the services provided.

The UBMS impact study is designed to measure the effects of participating in UBMS on college enrollment, choice of major, and other outcomes for students who participated during the summer of 1993, 1994 or 1995. Because the sample was not selected until 1998, we restricted the sample to participants at UBMS projects that were still operating that year: obtaining lists of participants from programs that were no longer operating in 1998 would have been nearly impossible. Conceptually, the study contrasts how participants fared with how they would have fared if they had not participated in UBMS. We compared UBMS participants with eligible applicants to the regular Upward Bound projects participating in the national evaluation. From this pool, we systematically selected a matched comparison group of students who were as similar as possible to UBMS participants in terms of characteristics and experiences that could potentially predict later outcomes. These characteristics included demographics such as sex, race and ethnicity as well as prior academic achievement such as grade point average (GPA) and math and science courses taken in ninth grade. The key difference was that the matched comparison students did not participate in UBMS.

The selection of matched comparison students also took into account experiences in other precollege programs, in particular, regular Upward Bound. Because regular Upward Bound is an intensive program that can influence high school achievement and postsecondary outcomes (Myers and Schirm 1999; Myers et al. 2004; Seftor, Mamun, and Schirm 2009), it is important to account for exposure to regular Upward Bound when estimating how UBMS participants would have fared if they had not participated in UBMS. For UBMS participants who had previously participated in a regular Upward Bound program—perhaps during the academic year—we selected comparison students who had also participated in the regular Upward Bound; these comparison students were selected from the treatment group for the evaluation of regular Upward Bound. For UBMS participants who had not participated in regular Upward Bound, we selected comparison students who did not participate in regular Upward Bound; these comparison students were selected from the control group for the evaluation of regular Upward Bound. 

The treatment and comparison groups were enrolled in high school at the same period of time. However, because of the later addition of the UBMS analysis to the original Upward Bound evaluation, some pieces of data were collected at different points in time. As a result, several data sources play a key role in the impact analysis. Baseline characteristics were collected for comparison group members through the baseline survey for the evaluation of regular Upward Bound. Baseline information on many of the same characteristics was collected for UBMS participants through a follow-up survey conducted in 1999. Although the 1999 survey was conducted four to six years after our sample had participated in the program, most of the baseline information collected—including sex, race and ethnicity—is time invariant. Follow-up surveys of treatment and comparison group members were also conducted in 2001–2002 and 2003–2004 to collect information about educational outcomes. Secondary and postsecondary transcripts were collected for both types of students to assess academic achievement. Additionally, data were collected from two administrative data sources: the National Student Clearinghouse (NSC) and the federal Student Financial Aid (SFA) records.

At the time of the follow-up survey in 2003–2004, the majority of sample members had been out of high school for about seven to nine years, and more than half had received postsecondary credentials. This report updates previous estimates of the effect of UBMS on postsecondary outcomes, including enrollment, persistence and completion. Chapter II of this report describes the operation of the UBMS program, followed by the findings from the impact analysis in Chapter III. Appendices A–C provide additional information on the study and study findings. Appendix A presents a focused look at data sources and outcomes measures; Appendix B provides tables detailing program impacts by subgroups; and Appendix C presents tables describing sample sizes and standard errors.

# II. The Operation of the Upward Bound Math-Science Program

It is necessary to understand what the UBMS program entails****to interpret information on its impacts. This chapter describes key features of the operations of UBMS projects, including the characteristics of host institutions and staff members, projects’ recruitment practices and enrollment levels, participants’ characteristics, and projects’ goals and services. For context, this chapter presents comparable information on the operations of regular Upward Bound when possible.

The primary data source for this chapter is a survey of UBMS projects conducted in spring 1998. The survey sample consisted of all 81 projects operating at the time, and 74 of the 81 projects responded to the survey.[footnoteRef:3] The survey requested information about program operations in two separate years—1994 (in the middle of the period over which our sample was participating in UBMS) and 1998 (the year before the survey)—but some questions were specific to 1998. When possible, we focus our analysis of program operations on 1994 to facilitate comparisons with regular Upward Bound projects operating in 1993, as reported in Fasciano and Jacobson (1997), and to describe the programs that served the same cohorts of participants for whom we measured the impacts of the program (see Chapter III); unless noted otherwise, the results for 1998 were generally similar to those for 1994. To augment the information provided by the survey of UBMS projects, we also use information from case studies and annual performance reports (Moore 1997). [3: . We did not adjust (weight) for survey nonresponse, reasoning that the number of nonrespondents was low enough to eliminate any serious concerns about whether the data represented the sample. Also, rarely did more than three UBMS projects fail to respond to any particular item on the questionnaire. We excluded the one project that reported serving only veterans in 1998. Note that veterans’ projects were also excluded from the survey of regular Upward Bound grantees, so the comparisons that are made in this chapter between the two types of Upward Bound programs are based on Upward Bound projects that did not exclusively serve veterans.] 

The findings in this chapter indicate that UBMS projects provide intensive academic enrichment to disadvantaged high school students in math and science by using staff members with strong academic credentials in those subjects. Some of the features that make UBMS projects distinctive, even from regular Upward Bound projects, are (a) high levels of annual funding per student and low student-teacher ratios, (b) recruiting strategies that attract students from wide geographic areas, (c) the providing of services that are heavily concentrated in residential programs during the summer, (d) course offerings that focus on math and science relative to other subjects, (e) academic preparation over nonacademic college preparatory activities, and (f) academic enrichment over remediation. The remainder of the chapter provides a description of UBMS and an assessment of its distinctive features. 

## A.	Project Hosts and Staff

The impacts of UBMS projects on student outcomes may depend on the types of institutions that host them and the people they hire to serve as instructors and other staff members. In this section, we describe the types of institutions that host UBMS projects and the staff members who provide services to program participants.  

### 1.	Host Institutions: Two- and Four-Year Colleges and Universities

The types of institutions that host a UBMS project may influence where students attend college. Most Upward Bound programs are hosted by either two- or four-year postsecondary institutions. Evidence from the national evaluation of regular Upward Bound suggests that participation at projects hosted by four-year colleges raises the probability of attending a four-year college, and participation at projects hosted by two-year colleges raises the probability of attending a two-year college (Myers, Olsen, and Seftor 2002). Therefore, the types of institutions that host UBMS projects may influence the types of postsecondary institutions that program participants subsequently attend.

Nearly nine out of ten UBMS projects operating in the mid–1990s were hosted by four-year colleges, a substantially higher proportion than among regular Upward Bound projects (see Exhibit II.1). Four-year colleges may find it easier than other potential host institutions to meet some of ED’s guidelines for UBMS, including offering hands-on experience in laboratories and computer facilities, providing opportunities to learn from mathematicians and scientists engaged in research or applied science, and supporting involvement with tutors and counselors who are graduate and undergraduate students in math and science.  

### 2.	Summer Program Staff and Project Director

UBMS projects are directed by highly educated individuals and are staffed by people with strong credentials in math and science. These staff members have responsibility for a relatively small number of students, which may provide opportunities for individual instruction. At the typical project, the project director and staff members can provide same-race role models for many of the students they serve. The sections below provide more detail on our findings concerning staff size and composition by job title, staff credentials, and the racial composition of the project staff.

Exhibit II.1Types of Institutions that Hosted Upward Bound Math-Science Projects, 1995

Source: Reprinted from Mary T. Moore, *Developing Math and Science Skills Among Disadvantaged Youth: A Review of the Upward Bound Precollege Math/Science Centers*(Washington, D.C.: Department of Education, Planning and Evaluation Service, September 1997), Exhibit II.1, 15.

Note: Percentages may not total 100 percent due to rounding.

**a.	Staff Size and Composition by Job Title**

In 1994, UBMS projects had an average of 24 staff members, comprising roughly eight instructors, five resident counselors, four mentors, three tutors, two administrators, one academic or guidance counselor, and one clerical staff member; by 1998, the average Math and Science Center (MSC) had almost 26 staff members, including 9 instructors. Overall, the average student-staff ratio in summer 1998 was 2:1, with a range from about 1:1 to 5:1.  

These findings, combined with findings from Moore (1997), suggest that student-staff ratios are typically lower in UBMS projects than in regular Upward Bound projects. The survey of grantees did not collect information on the number of full-time equivalent (FTE) staff members, but the information available suggests that UBMS projects typically maintain student-staff ratios that are substantially lower than in regular Upward Bound. Moore (1997) found that 14 randomly selected UBMS projects visited in summer 1996 had an average of 2.6 students per FTE staff member, including administrators, and 8.2 students per FTE instructional staff member. In contrast, tabulations from the data used by Fasciano and Jacobson (1997) indicate that in summer 1992, regular Upward Bound projects had more students per staff member—5.1 students per FTE staff member and 13.6 students per FTE instructional staff member (see Exhibit II.6 in U.S. Department of Education, Office of the Under Secretary 1997).

**b.	Credentials**

At the average UBMS project in 1998, most staff members were highly educated and had educational backgrounds in math and science. About one-quarter had attended some college but had not obtained a degree, another quarter had obtained a bachelor’s or associate’s degree (mostly bachelor’s degrees), and the rest had done graduate work or had obtained a graduate degree. Most staff members without undergraduate degrees were undergraduate students who served as UBMS mentors, tutors and resident counselors while working toward bachelor’s degrees in math, science or education. Approximately two out of five staff members had their highest degree in science or the social sciences (31 percent) or math (10 percent).

Most instructors at the average UBMS project had experience teaching math or science. During the school year, most instructors were either high school teachers (41 percent) or postsecondary teachers (31 percent); one-fifth were graduate students (14 percent) or undergraduates (6 percent). Moreover, at the typical project, two-thirds of the high school teachers and three-quarters of the postsecondary teachers taught in a math or science field.

The professional and educational backgrounds of UBMS project directors provide insight on their credentials to direct projects. One-fifth of the directors were faculty members at the host institution or another college, roughly the same percentage as in regular Upward Bound. In 1998, two-thirds of the directors held a master’s degree and one-fifth held a doctorate. About half had their highest degree in education, less than one-fifth had their highest degree in engineering, mathematics, or physical sciences, and 12 percent had their highest degree in the social sciences. UBMS project directors were less likely than the program staff as a whole to have a background in math or science. Compared with regular Upward Bound project directors, UBMS project directors were twice as likely to have a doctorate.  

**c.	Race and Ethnicity**

At about 9 out of 10 UBMS projects operating in 1998, one racial or ethnic group accounted for a more than half of the staff members (see Exhibit II.2). In many cases this pattern may have reflected a conscious strategy, also used in regular Upward Bound, to provide minority students with same-race role models. For example, at 21 UBMS projects, a majority of the staff members were black; at 18 of those, a majority of the students were also black.   

The racial and ethnic profile of UBMS project directors was similar to that of the UBMS project staff (see Exhibit II.3). Both staff members and project directors were nearly evenly split between white and nonwhite; project directors were slightly more likely than other staff members to be black and were less likely to be Asian than other staff members. The race and ethnicity of the UBMS project director often matched that of the predominant student racial and ethnic group: more than three-fourths of the minority directors headed programs where students from the same group constituted a plurality of participants.

## B.	Eligibility, Recruitment and Enrollment, and Student Characteristics

To shed light on the types of students who participate in UBMS, we examine the eligibility criteria that students must meet and the recruiting strategies that UBMS projects use to attract students. We also examine the characteristics of participants as reported by project directors.

### 1.	Eligibility

UBMS projects have to meet the same federal rules as regular Upward Bound projects concerning the composition of participants. At each project, at least two-thirds of the participants must be both low-income and potential first-generation college students; the remaining students must meet either of these two criteria. At the average project in summer 1998, about 77 percent of students met both of these eligibility criteria, about 14 percent were first-generation only, and about 9 percent were low-income only, very similar to the distribution in regular Upward Bound during 1992–93. In addition, UBMS and regular Upward Bound projects are allowed only to serve students who have completed eighth grade.

Exhibit II.2

Predominant Racial and Ethnic Group for Upward Bound Math-Science Staff, Summer 1998

Source: 1998 survey of Upward Bound Math-Science projects.

Note: Percentages may not total 100 percent due to rounding. The 12 percent of projects with no majority racial and ethnic group included 6 percent of projects with a plurality of black students and 4 percent with a plurality of white students.

Exhibit II.3

Racial and Ethnic Distribution for Upward Bound Math-Science Staff and Project Directors, Summer 1998

Source: 1998 survey of Upward Bound Math-Science projects.

Note: Percentages may not total 100 percent because (a) people may fall into multiple categories, (b) the Pacific Islander category was excluded from the exhibit because only nine staff members nationwide fell into this category, and (c) some staff members may not have been classified by race or ethnicity.

Most UBMS projects also adopt additional student eligibility criteria for enrollment in the program—for example, requirements for completion of additional grades beyond grade 8, school course work or recommendations. In 1994, more than three-fourths of projects required students to have finished ninth grade; a few projects required 10th grade completion. In addition, nearly all UBMS projects required participants to have a teacher recommendation and to have completed at least one high school course in math or science, and those applicants enrolled in regular Upward Bound commonly needed a recommendation from the director. Finally, about 30 percent of UBMS projects in 1994 prohibited students from attending the project again after the previous summer’s program, and almost half prohibited students from attending the project again unless they met certain criteria. These practices changed dramatically by 1998 when only 7 percent of UBMS projects prohibited any participating students from attending the project again after the prior summer, and 71 percent prohibited students from returning unless they met certain criteria. The questionnaire did not address the specific types of criteria that projects imposed.

### 2.	Recruitment and Enrollment

To find a pool of potentially eligible applicants, UBMS projects focused mainly on other precollege programs or secondary schools (see Exhibit II.4). Among UBMS projects operating in summer 1994, nearly all recruited from regular Upward Bound projects while substantial majorities also recruited directly from Talent Search projects and from middle or high schools. However, it is important to note that although almost all UBMS projects made efforts to recruit from regular Upward Bound projects in 1994, data from the evaluation suggest that fewer than one in five UBMS participants had actually previously participated in regular Upward Bound.  

Historically, UBMS projects have cast a wide net in recruiting students beyond the local areas of the host institutions. In 1994, only about one in ten UBMS projects recruited exclusively from a specific and typically local set of feeder schools or Upward Bound projects; the rest recruited from statewide or regional lists of schools and programs (see Exhibit II.5). By 1998, however, the percentage of UBMS projects that recruited exclusively from a specific set of feeder schools or Upward Bound projects had tripled. Therefore, it appears that, over time, more UBMS projects are using strategies that focus on recruiting from local projects like regular Upward Bound projects.  

By design, UBMS projects are smaller than regular Upward Bound projects. Through recruitment, UBMS projects received an average of 108 applications for the summer of 1994, ranging from a low of 50 to a high of 300, and they enrolled between 40 and 53 students. In contrast, regular Upward Bound programs enrolled an average of about 75 students in the mid-1990s (U.S. Department of Education, Office of the Under Secretary 1997). 

Exhibit II.4

Targets for Recruiting by Upward Bound math-Science Projects, 1994

Source: 1998 survey of Upward Bound Math-Science projects.

Exhibit II.5

Geographic Scope of Recruiting by Upward Bound math-Science Projects, 1994

Source: 1998 survey of Upward Bound Math-Science projects.

Note: Percentages may not total 100 percent due to rounding.

### 

### 3.	Student Characteristics

In our 1998 survey of UBMS projects, project directors provided information on the distribution of students participating in their projects by sex, race, grade level and place of residence: 

***Sex.*** At the average UBMS project, like the average regular Upward Bound project, 60 percent of students were female (based on data for all UBMS projects operating in 1998 and all regular Upward Bound projects operating in 1992). However, this percentage varied considerably across UBMS projects, from a low of 25 percent to a high of 78 percent.  

***Race***. On average, UBMS projects served an ethnically diverse group of students: 42 percent black, 27 percent white, 15 percent Hispanic, 8 percent Asian, 5 percent American Indian, and 1 percent Native Hawaiian or other Pacific Islander. However, reflecting the demographics of the region served, many UBMS projects serve participants where one racial and ethnic group constituted more than half (see Exhibit II.6). In comparison, at 87 percent of regular Upward Bound projects operating in 1992–93, one racial and ethnic group accounted for a majority of participants. Furthermore, some UBMS projects exclusively served students from a single racial or ethnic group. For example, six UBMS projects reported that all of its participating students were black.  

***Grade level.***The eligibility guidelines discussed above, along with other factors, can affect the distribution of students across different grade levels. At the average project, 29 percent of participants were entering 12th grade, 37 percent were entering 11th grade, 27 percent were entering 10th grade, and 6 percent were entering 9th grade. In regular Upward Bound during the summer of 1992, 20 percent of participants were entering 12th grade, 32 percent were entering 11th grade, 31 percent were entering 10th grade, and 16 percent were entering 9th grade. These exhibits suggest that, on average, UBMS projects serve students who are slightly closer to graduation than is the case at regular Upward Bound projects. However, there was substantial variation in the grade-level distribution of participants across projects in 1998. For example, one UBMS project reported that all of its participants were entering 12th grade while four reported that none were rising seniors. In addition, the proportion of rising juniors ranged from 0 to 72 percent, and the proportion of rising sophomores ranged from 0 to 60 percent.  

***Place of residence.*** Given that most projects recruited across the state or region, it is not surprising that many UBMS participants came from outside the grantee’s local city or town. At the average project, only about 25 percent of the students were locals. As we would expect, projects that recruited only from a set of local schools or regular Upward Bound projects served considerably higher percentages of students from the local area than other UBMS projects.

Exhibit II.6

Predominant Racial and Ethnic Group for Participants at Upward Bound math-Science Projects, 

Summer 1998

Source: 1998 survey of Upward Bound Math-Science projects. 

Note: Percentages may not total 100 percent due to rounding. The 37 percent of Upward Bound Math-Science projects with no majority racial and ethnic group of students included 17 percent with a plurality of white students, 11 percent with a plurality of Hispanic students, 6 percent with a plurality of black students, 3 percent with a plurality of American Indian students, and 1 percent with a plurality of Asian students.

## 

Moore (1997) described UBMS participants as a more select group than regular Upward Bound participants based on their having earned somewhat higher grades and having expressed greater interest in math and science (23, 26) before participating in Upward Bound. Discussions with UBMS and regular Upward Bound staff members revealed that UBMS participants were typically considered “more serious about school” than regular Upward Bound participants (Moore 1997, 26).

## C.	Program Description

In a college-like setting, UBMS projects offer academic enrichment in math and science to improve student achievement in those subjects and to expose students to math and science careers. In this section, we describe the following features of UBMS projects in more detail: the setting in which these projects provide services; the goals, academic orientation, academic offerings (summer and academic year) and instructional approaches of these projects; and the intensity and quantity of services the UBMS projects provide.

### 1.	Setting

As described earlier, UBMS projects are typically hosted by two- and four-year postsecondary institutions (see Section A.1). Most UBMS projects are hosted by four-year colleges and universities, and most of these institutions have dormitories to house their students. These dormitories are often available in the summers to house participants of summer programs hosted by these institutions.

UBMS projects typically exposed participants to a college setting during the summer program by housing them in the college dormitories. Virtually all the UBMS projects we surveyed (100 percent in 1994, 97 percent in 1998) offered a residential component to their summer programs compared with 87 percent of regular Upward Bound programs in 1992 (Moore 1997). At almost all UBMS projects, students lived in the dormitories for the entire summer program, which lasted about six weeks on average. Only four percent of UBMS projects in 1994 had a residential component shorter than the summer program, but by 1998, the rate had increased to 11 percent. Therefore, for six weeks, most participants lived on campus like many college students do during the academic year.

### 2.	Goals of the Program

As mentioned in Chapter I, the general objective of the UBMS program is to prepare participating students for postsecondary programs leading to careers in math and science. Seven out of 10 UBMS projects operating in 1994 rated “academic performance in math and science” as their foremost or second most important goal (see Exhibit II.7). The focus on academic improvement was similar to the focus of regular Upward Bound projects operating at about the same time. Eighty-seven percent of regular Upward Bound projects rated “academic improvement” as their foremost or second most important goal in 1993. If the regular Upward Bound grantee survey had also listed “academic improvement in math and science,” it is possible that some respondents would have cited that as one of their top two goals: Fasciano and Jacobson (1997) characterized 37 percent of regular Upward Bound projects as having a strong emphasis on math and science.  

However, two goals that regular Upward Bound projects considered moderately important were not considered important by UBMS projects. First, only 13 percent of UBMS projects reported that one of their top two goals was fostering students’ personal skills (e.g., goal orientation, ability to adapt to new settings), compared with 31 percent of regular Upward Bound programs. Second, none of the UBMS projects cited improving students’ access to financial aid as one of their top two goals, compared with 35 percent of regular programs.

### 3.	Academic Orientation

UBMS projects try to provide academic enrichment beyond what students are exposed to in school (see Exhibit II.8). Very few projects emphasized remedial instruction in 1994. By 1998, the relative focus on enrichment was even greater, with 83 percent of UBMS projects citing enrichment as a major emphasis and only 1 percent citing remediation. Although about 1 in 4 regular Upward Bound programs emphasized providing remedial instruction, fewer than 1 in 10 UBMS projects reported doing the same.

The focus of UBMS on academic enrichment over remediation is consistent with the types of students served by the program. As described earlier, findings in Moore (1997) indicate that, on average, UBMS participants probably had less need for remedial support than regular Upward Bound participants.

Exhibit II.7

Most Important Goals of UPWARD BOUND math-science Projects, Summer 1994

Source: 1998 survey of Upward Bound Math-Science projects.

### 

Exhibit II.8

Academic Orientation of upward bound math-science Projects

Source: 1998 survey of Upward Bound Math-Science projects; Nancy J. Fasciano and Jon E. Jacobson, “Grantee Survey Report.” In *A 1990s View of Upward Bound: Programs Offered, Students Served and Operational Issues*, by U.S. Department of Education, Office of the Under Secretary (Washington, D.C.: U.S. Department of Education, May 1997). 

Note: Percentages do not total 100 because grantees were allowed to rate more than one approach as a major emphasis.

 † Excludes summer bridge programs for Upward Bound participants who have just graduated from high school. 

### 

### 4.	Summer Offerings

In accordance with program guidelines, UBMS projects offered instruction in a diverse array of academic subject areas (see Exhibit II.9). Seventy-five percent or more of these projects in 1994 offered instruction in the following subjects: writing and composition, algebra II, geometry, pre-calculus, computer applications and software use, physics, biology and chemistry. The average total number of offerings in 1994 was about 14, with a range of 2 to 22. The average number of offerings in math and science was about 7, with a range of 1 to 11. Thus, on average, math and science courses accounted for roughly half of UBMS projects’ total offerings.  

UBMS projects clearly differed from regular Upward Bound projects in their relative emphasis on certain subjects. First, as expected, they concentrated their offerings more on math and science. Although UBMS projects were no more likely than regular Upward Bound projects to offer certain math or science courses (for example, algebra II, geometry pre-calculus, calculus, biology and chemistry), UBMS projects were much less likely to offer instruction in areas outside of math and science, including social science or history courses or electives or nonacademic courses. Second, consistent with their greater emphasis on enrichment than on remediation, UBMS projects were less likely than regular Upward Bound programs to offer low-end courses such as reading comprehension and vocabulary, pre-algebra, and earth science (see Exhibit II.9).

To help prepare students for a postsecondary education and postcollegiate careers in math or science, UBMS projects also offered a range of support services and activities (see Exhibit II.10). Among the most common activities were field trips (e.g., to math or science facilities) and assistance with college and financial aid applications. The average number of these noninstructional offerings in 1994 was about 10, with a range of 3 to 15.

UBMS projects were substantially less likely than regular Upward Bound projects to offer services focused on preparing for college. Many regular Upward Bound projects would have provided these services during the academic year: both regular Upward Bound and UBMS programs focus on academics during the summer. Because many UBMS participants participated in other precollege programs during the academic year, UBMS project staff members could have reasonably expected that those other programs were assisting students in preparing for college.

Exhibit II.9

Instruction Offered by upward bound math-science Projects, by Subject Area

Sources: 1998 survey of Upward Bound Math-Science projects; Nancy J. Fasciano and Jon E. Jacobson, “Grantee Survey Report.” In *A 1990s View of Upward Bound: Programs Offered, Students Served and Operational Issues*, by U.S. Department of Education, Office of the Under Secretary (Washington, D.C.: U.S. Department of Education, May 1997), 39.

Note: UBMS projects offer instruction in many areas besides math and science, either to meet regulatory requirements or simply to ensure that their program will interest and benefit students in many ways.

a 1992 nonbridge summer programs or 1992–93 academic year.

b Neither survey listed statistics, trigonometry or Internet/Web page design, but enough project directors specified them under “other courses” that we present data on these courses separately. 

Exhibit II.10

Noninstructional Services Offered by upward bound math-science Projects

Sources: 1998 survey of UBMS projects; Nancy J. Fasciano and Jon E. Jacobson, “Grantee Survey Report.” In *A 1990s View of Upward Bound: Programs Offered, Students Served and Operational Issues*, by U.S. Department of Education, Office of the Under Secretary (Washington, D.C.: U.S. Department of Education, May 1997), 54.

a 1992 nonbridge summer programs or 1992–93 academic year.

b In the regular Upward Bound grantee survey, this item was phrased, “Identify sources of financial aid.”

c In the regular Upward Bound grantee survey, this item was phrased, “Site visit to employers or job shadowing.”

d Although the regular Upward Bound grantee survey had three separate items about Job Training Partnership Act positions, work-study arrangements, and other partnerships, the results were reported only in the aggregate, and it did not ask about math or science internships (Fasciano and Jacobson 1997, 54).

e The regular Upward Bound grantee survey asked about field trips of varying lengths, not the destinations.

### 5.	Academic-Year Offerings

Although UBMS projects also provided services to students during the academic year in the mid-1990s, these services were minimal compared with UBMS summer services and were typically far less numerous and less intense than academic year services provided in regular Upward Bound. During the 1994–95 academic year, about one-third of UBMS projects provided tutoring or study sessions, and just more than half provided assistance with 

college applications (see Exhibit II.11). The average UBMS project provided about three types of these services during 1994–95.

Exhibit II.11

Academic Year Services Offered by upward bound math-science Projects, 1994–95

Source: 1998 survey of Upward Bound Math-Science projects.

By 1998–99, the percentage of UBMS projects regularly providing these services had increased substantially. For example, the percentage providing tutoring or study sessions rose from 81 percent to 93 percent, and the percentage providing assistance with college applications rose from 52 percent to 78 percent. This increase probably reflects the establishment of more locally oriented UBMS projects. Not surprisingly, geographic proximity to their participating students influenced whether UBMS projects provided certain services during the academic year. UBMS projects were substantially more likely to provide tutoring and workshops during the academic year if a relatively large percentage of their participants lived in the same city or town as the program host. For example, more than three-quarters of UBMS projects with a relatively large percentage of participants from the local area (above the median) offered tutoring during the academic year versus less than half of UBMS projects with a relatively small percentage (below the median).

### 6.	Instructional Approaches

In 1994, the most common instructional approach taken by UBMS projects was the providing of instruction through courses in separate subjects. Four out of five UBMS projects offered courses in separate subjects (see Exhibit II.12, Panel A). In three out of four UBMS projects, the primary method of instruction was either the provision of these courses (37 percent, see Exhibit II.12, Panel B) or the combination of these courses with interdisciplinary courses (also 37 percent, see Exhibit II.12, Panel B).  

Exhibit II.12

Instructional Methods Used by upward bound math-science Projects, 1994

Source: 1998 survey of Upward Bound Math-Science projects.

Note: Percentages may not total 100 percent due to rounding.  

However, UBMS projects frequently used other instructional methods. The majority of projects (63 percent) offered interdisciplinary courses, and in a large minority (35 percent), at least some students worked on a large project or experiment that spanned multiple academic subject areas (see Exhibit II.12, Panel A).

UBMS projects vary considerably in how they sort students into classes or groups. In the summer of 1994, about half of the projects placed their students in instructional groups based on proficiency level (37 percent) or grade level (16 percent). About one-fourth placed students with diverse proficiency levels in the same group to facilitate learning (presumably the learning of less proficient students), and the remaining projects grouped students by their interests or in some other way.  

UBMS participants do not spend most of their time in traditional lecture-style classes. At the average project during the summer of 1994, only one-fourth of the time was spent in lecture-style classes such as those offered in most schools. The remaining time was spent in small-group, teacher-led instruction (32 percent), laboratories (29 percent), computer-based instruction (12 percent), and other settings (4 percent).  

## 7.	Intensity and Quantity of Services

The services that UBMS projects offer and the length of their summer residential summer programs suggest that these projects offer intensive programs that provide students with a “large dose” of services, at least for one summer. Furthermore, larger doses of effective services may yield larger impacts than smaller doses, as we found for regular Upward Bound (Myers et al. 2004). Summary measures of program intensity presented in this section indicate that UBMS projects offer an intensive program that might be expected to improve the math and science preparation of program participants. 

UBMS is a resource-intensive program. Program grants to UBMS projects provided an average of approximately $4,800 per student in FY 2004 (see Exhibit II.13). This level of funding is comparable with funding for regular Upward Bound—approximately $4,500 per student—and much more expensive than other****precollege programs. UBMS funding supports an extensive package of instruction and services, as described earlier in the chapter.

Participants devote a substantial amount of time to the program, and most of this time is spent on academics. At the average UBMS project in the summer of 1994, students spent about 29 hours per week receiving instruction and almost 11 hours per week on tutoring and homework. Thus, participating in UBMS over the summer is somewhat like having a full-time job requiring 40 hours per week. Because the vast majority of UBMS summer programs last six weeks, participants at the average project spend 240 total hours on academics during the summer. Only five UBMS projects reported a program of a different length: three had a five-week session, one had a seven-week session, and one had an eight-week session.

The amount of time devoted to core program activities varied across UBMS projects. For example, four projects reported that students spent more than 40 hours per week in instruction alone, while a handful of projects reported estimates of less than 10 hours per week. Estimates of 

24

time spent on tutoring or homework ranged from 0 to 20 hours per week. And the estimates of average total time spent on academics during the summer varied considerably, from fewer than 100 hours at some projects to more than 340 hours at some others.

Most UBMS participants completed the summer program. At the average project in the summer of 1994, 94 percent of participants completed all the requirements of the program. A few projects reported completion rates of 50 percent or less, but this information does not necessarily indicate a high dropout rate. Students might have attended the summer program for its full length but be counted as failing to “complete all the requirements of the program” (the wording in the questionnaire) because they did not, for example, turn in all their assignments.  

Exhibit II.13

Per-Capita Funding for Regular upward bound and for Upward Bound Math-science (UBMS) Projects, FY 2004

Note:            Regular Upward Bound cost per participant listed in January 2010 ($4,721) is slightly above that listed in FY2004 U.S. Department of Education program data Web pages presented in the figure above. 

Source:        U.S. Department of Education program data Web pages http://www.ed.gov/programs/trioupbound/funding.html and http://www.ed.gov/programs/triomathsci/funding.html (accessed January 2010).

Some students participate for multiple summers or receive services during the academic year. As indicated earlier, about 70 percent of UBMS projects operating in summer 1994 allowed students to return from the previous summer. Among those projects, an average of 35 percent of 1994 participants had also participated during 1993. In summer 1998, even more UBMS projects (more than 90 percent) allowed students to return; moreover, at those projects an average of 52 percent of students had also participated in summer 1997. Furthermore, some UBMS projects extended students’ exposure to the program beyond the summer by providing services during the academic year.  

# 

52

# III. The Impacts of the Upward Bound Math-Science Program

UBMS was established to increase economically disadvantaged students’ achievement in high school math and science courses, to increase the likelihood that they would major in math and science in college, and ultimately to increase their representation in math and science careers. Until now, no rigorous studies have measured the extent to which the program achieves its goals. In this chapter, we assess the effects of UBMS on important outcomes for program participants seven to nine years after they were scheduled to graduate from high school.  

In the remainder of this section, we describe the study design, the data sources and outcomes, as well as the results of the impact analysis. 

## A.	Study Design

UBMS provides intensive academic enrichment in math and science. Like regular Upward Bound projects, most UBMS projects also offer some assistance in preparing for college, including assistance with college applications (see Chapter II for more details). The combination of intensive academic enrichment in math and science with college preparation assistance suggests that UBMS might have positive effects on the following outcomes for program participants: (a) performance in high school, especially in math and science courses; (b) postsecondary attendance, persistence and completion; and (c) the likelihood of completing a postsecondary degree in a math and science field. Therefore, we designed the analysis to answer the following three research questions:

What are the effects of UBMS participation on student performance in high school overall and in math and science courses in particular?[footnoteRef:4]   [4: . The interim report (Olsen et al. 2007) presented detailed findings on the effects of UBMS on high school outcomes. By the 1999–2000 data collection on which that report was based, nearly all sample members had progressed out of high school and into college. Therefore, this report focuses only on postsecondary outcomes.] 

What are the effects of UBMS participation on college attendance, attendance at different types of colleges and universities, persistence, and college completion?

What are the effects of UBMS participation on the likelihood of enrolling and completing a degree in math or science? 

How are these findings influenced by prior participation in regular Upward Bound?

We have designed the analysis to measure the impacts of UBMS on two important subgroups: students who had previously participated in regular Upward Bound and students who had not participated. UBMS participants who previously participated in regular Upward Bound may have received a large dose of precollege services and academic preparation before participating in UBMS. However, most of the other UBMS participants entered UBMS without having received such intensive services. On the one hand, it is reasonable to expect UBMS to have larger effects on the students who had not previously received intensive services: students who have already received them may have already received the boost they needed to succeed. On the other hand, UBMS participants may be better prepared to benefit from their participation if they have previously participated in intensive services. In our analysis, we compute the effects separately for students who had previously participated in regular Upward Bound and students who had not participated to assess whether the effects of UBMS depend on the amount of precollege services students have received to that point.  

Regular Upward Bound is just one of the other programs in which UBMS participants could have participated. Appendix A of Moore (1997) includes a list of 28 alternative math and science precollege programs. Annual per student costs were available for 17 of the 28 alternative math and science programs; of these 17, only two were more expensive per student-year than UBMS.

It is important to clarify that we have not attempted to measure the effects of UBMS versus an absence of precollege services. The analysis was designed to measure the effects of participating in UBMS relative to what students would have participated in otherwise—which might include regular Upward Bound—*not* the effects of participating in UBMS relative to no program participation. Most UBMS participants in our sample did not participate in regular Upward Bound, and only a few other precollege programs are as intensive as Upward Bound. Therefore, most UBMS participants would have participated in less intensive precollege services if they had not participated in UBMS.

In this section, we describe the design of the analysis used to measure the impacts of UBMS. The impact analysis is based on a matched comparison group that attempts to reduce two types of bias common to many nonexperimental studies: selection bias and bias attributable to different data collection protocols for the participant and comparison groups. The strength of the impact analysis rests on three features of its design: 

1. How we selected our initial samples, particularly the comparison sample

1. How we collected “baseline” (preprogram) information on the two samples

1. How we used that information to select a matched comparison group 

In the remainder of this section, we describe these three features of the study and our approach to estimating the effects of UBMS on student participants.

### 1.	Selecting the Samples

For the impact analysis, we obtained our sample of program participants from the projects themselves. In 1998, we contacted the 62 Upward Bound MSCs that were operating at that time and that had been operating between 1993 and 1995. Although we would have been interested in obtaining lists of students from projects that were no longer operating in 1998, we believed that it would be very difficult to obtain such lists. If MSCs that closed before 1998 operated less effective programs than those that remained open, then the results presented in this chapter may overstate the effectiveness of MSCs operating between 1993 and 1995. From the MSCs that we contacted, we requested lists of the students who had participated in their program in the summer of 1993, the summer of 1994 or the summer of 1995; we received participant lists from all but one of these MSCs. To reduce the costs of collecting the necessary data, we randomly selected one out of every four of the students from these lists for our analysis sample. 

A primary feature of any nonexperimental evaluation is the choice of a comparison group. Experiments yield the best comparison groups because differences in outcomes between treated and untreated cases cannot be attributed to selection bias. In the absence of an experiment, the strength of an evaluation depends on the comparability between the participant and comparison groups.

The most convenient comparison group for the impact analysis is also a compelling one—and the one we used. For the comparison group, we selected students from the evaluation of regular Upward Bound who reported that they had not participated in an MSC.[footnoteRef:5] In doing, so, we selected a comparison group with three desirable attributes: [5: . If UBMS projects admitted eligible applicants on a fairly random basis, rejected applicants would probably constitute the best comparison group. However, MSCs are not required to select randomly from eligible applicants, and the evidence suggests that they do not. Case studies of 14 MSCs in the mid-1990s suggest that many MSCs impose additional eligibility requirements beyond the federal requirements, including an interest in math and science (Moore 1997). Therefore, it is unlikely that rejected applicants would have the virtues of a randomly selected control group for the purposes of this evaluation. Furthermore, the difficulty in obtaining lists of program participants from MSCs several years after they participated in UBMS suggests that many MSCs would not have been able to provide information on rejected eligible applicants.  ] 

1. ***Like UBMS participants, comparison students applied to participate in regular Upward Bound.***Therefore, both UBMS participants and comparison students revealed a high level of motivation to pursue precollege services through Upward Bound or UBMS. This attribute provides some protection against a common source of selection bias in nonexperimental studies—bias from comparing more motivated participants with less motivated nonparticipants.

***Like UBMS participants, comparison students met the federal eligibility requirements to participate in some type of Upward Bound program—either regular or math and science.***The federal eligibility requirements are the same for regular Upward Bound and for UBMS. To be included in the sample for the evaluation of regular Upward Bound—the sample from which we selected our comparison group—a student must have applied to a regular Upward Bound project and been determined eligible to participate. Therefore, both UBMS participants and comparison students in our sample must have come from either “low-income” families (income below 150 percent of the poverty line) or potential “first-generation” families (neither parent had earned a bachelor’s degree).

***Like UBMS participants, many comparison students would have met project-specific eligibility requirements imposed by some MSCs.*** Moore (1997) indicates that MSCs often apply additional admissions criteria in selecting applicants—criteria that include a minimum GPA in math and science. In this chapter, we show that many comparison students met the same criteria: many were successfully “matched” to UBMS participants who took similar courses and earned similar grades in math and science in ninth grade.  

Although the regular Upward Bound sample is a useful comparison group for measuring the effects of UBMS, it is not a perfect one. Data from the 1990s suggest that MSCs typically had more stringent minimum GPA requirements than regular Upward Bound projects. In addition, MSC staff members—who typically had recent experience with regular Upward Bound participants—reported that UBMS participants tended to be “more serious about school” than regular Upward Bound participants (Moore 1997, 26). Therefore, UBMS participants might fare better than regular Upward Bound participants, even without any assistance from UBMS, and simple differences in mean outcomes between the UBMS participant and comparison groups may overstate the effects of UBMS due to selection bias. 

To reduce selection bias, we selected a matched comparison sample from the regular Upward Bound sample. More specifically, we matched each UBMS participant to one or more regular Upward Bound sample members with similar characteristics based on data collected from student surveys and transcripts. Very few students in the regular Upward Bound sample participated in UBMS, and those who reported participating in UBMS were excluded from the comparison group. See Section 3 in this chapter for more details.   

### 2.	Collecting Baseline Data

The baseline variables constructed for the impact analysis characterize members of the two samples early in high school before UBMS participants in our sample entered UBMS. These variables are critical because they allow us to account—through a combination of matching and regression adjustments—for many preexisting differences between the two groups that might otherwise bias our impact estimates. To collect the information necessary to create baseline variables, we conducted student surveys and collected high school transcripts (see Appendix A).

An important strength of the study’s design is that the data collection strategy was similar for both samples. Heckman, Smith and Clements (1997) argue that different survey questionnaires for the participant and comparison samples can be an important source of bias in nonexperimental studies. In this evaluation, we developed the initial survey questionnaire for UBMS participants from the baseline survey questionnaire for the regular Upward Bound sample. Baseline characteristics were collected for comparison group members through the baseline survey for the evaluation of regular Upward Bound. Baseline information on many of the same characteristics was collected for UBMS participants through a follow-up survey conducted in 1999. Therefore, the survey questions from which we constructed baseline variables were often identical and always similar for the two samples.  

Even though the treatment and comparison groups were enrolled in high school at the same period of time, we had to address two differences in the surveys in constructing baseline variables for the MSC impact analysis: (a) the two survey questionnaires were different, and (b) the surveys were conducted at different times—after high school for the participant group and early in high school for the comparison group. To address the differences in the survey questionnaires, we used data from survey questions that are either identical or almost identical. To address the difference in the timing of the surveys, we used data from survey questions only when the timing of the survey was unlikely to affect the answer to the question. Furthermore, the approach to collecting and coding high school transcripts was the same for the two samples. Therefore, it is unlikely that differences in the data between the two samples have biased the impact estimates presented in this chapter.  

The baseline variables for the impact analysis fall into the following three categories: (a) demographic and family characteristics, (b) participation in other precollege programs, and (c) ninth-grade academic achievement not only in math and sciences but also more generally (see Exhibit III.1). We believe that the measures of students’ ninth-grade academic achievement are critical to the strength of the study. Given the findings in earlier reports, it seems entirely possible that even among students with similar demographic and family characteristics, students who participated in UBMS might have a higher academic aptitude and interest in math and science than students who participated in regular Upward Bound. Therefore, we use information on ninth-grade courses taken and ninth-grade GPA—overall and specifically math and science—to control for differences between the two samples in early high school achievement.

Exhibit III.1

Baseline Variables

Note: The initial surveys were conducted in 1999 for Upward Bound Math-Science (UBMS) participants and in 1992–94 for comparison students in the regular Upward Bound sample. To identify UBMS participants who had previously participated in regular Upward Bound, we used responses to the 1999 initial survey; to identify comparison students who participated in regular Upward Bound, we used participation information provided by projects. We collected high school transcripts in 2000 and 2003 for UBMS participants and in 2000 and earlier years for comparison students.

### 3.	Selecting a Matched Comparison Sample for the Impact Analysis

The data collected on UBMS participants and regular Upward Bound sample members are useful in identifying not only many similarities between the two groups but also some differences (see Exhibit III.2). In both groups, 37 percent of the students were African American and 11 percent had a sibling who had participated in Upward Bound. The two groups were also similar with respect to many other characteristics, including the percentage taking algebra in ninth grade. However, the two groups exhibit differences on several dimensions. For example, UBMS participants were more likely to be male and tended to have higher grades than regular Upward Bound sample members.

To address possible selection bias, we selected a matched comparison group from the regular Upward Bound sample using propensity score matching methods. Many studies adjust for baseline differences of these types using standard covariance adjustments—that is, by controlling for these variables in a regression analysis. However, the differences between the UBMS participant sample and the regular Upward Bound sample are too large to expect covariance adjustment to be reliable. Regression adjustments are likely to be unreliable if the means of the propensity scores are more than half a standard deviation apart (Rubin 2002). For the participant sample, the mean and standard deviation of the propensity score are 0.58 and 0.22, respectively; for the comparison sample, the mean and standard deviation of the propensity score are 0.25 and 0.23, respectively. Therefore, the difference in mean propensity scores is more than one standard deviation, and regression adjustments alone are likely to be unreliable.

The goal in matching was to select a matched comparison sample from the regular Upward Bound sample such that the distributions of the baseline variables for the UBMS participant sample and the matched comparison sample were similar. Matching was conducted separately for sample members who had previously participated in regular Upward Bound and for those who had not:

UBMS participants who had previously participated in regular Upward Bound were matched to members of the *treatment* group (those offered the opportunity to participate in Upward Bound) in the experimental evaluation of regular Upward Bound that was conducted as part of the National Evaluation of Upward Bound.  

UBMS participants who had *not* previously participated in regular Upward Bound were matched to members of the *control* group (those *not* offered the opportunity to participate in Upward Bound) in the experimental evaluation of regular Upward Bound that was conducted as part of the National Evaluation of Upward Bound.

Exhibit III.2

Summary Statistics from the Baseline Variables

*/**/*** Significantly different from Upward Bound Math-Science participants at the 0.10/0.05/0.01 levels, respectively.

	Regular Upward Bound sample members that were matched to at least one of the 689 UBMS participants were included in the matched comparison sample (988 of the 1,134 eligible). The propensity score matching and the impact analysis are restricted to sample members who entered high school between 1991 and 1993. Although some UBMS participant sample members entered high school before 1991 and after 1993, relatively few comparison sample members did so. Furthermore, the high school cohort is related to the likelihood of participating in the program and to the outcomes of interest because earlier cohorts had more time to enter college and select a field of study by the time they were interviewed for the evaluation. Therefore, to protect the internal validity of the study, we focused the analysis on students who entered high school between 1991 and 1993.  

	UBMS participants could be matched to more than one regular Upward Bound sample member, and regular Upward Bound sample members could be matched to more than one UBMS participant. To be matched, a pair of students must satisfy the following condition: the difference between matched students in the log odds of the propensity scores was less than 0.20 times the standard deviation of the log odds. Using matching with replacement within a fixed radius, each UBMS participant had, on average, 23.1 comparison group members matched to it, and each comparison group member matched to 16.1 UBMS participants. The weights of each UBMS participant were distributed equally to all comparison group members to whom that participant was matched, and the weight for a comparison group member is equal to the sum of the portions of the weights that group member received from each participant to whom it was matched.

Using matching procedures, we were able to select a matched comparison group that is highly similar to the sample of UBMS participants on many dimensions (see the last column of Exhibit III.2). Only two of the differences between the groups are statistically significant—the difference in the percentage of sample members who are native English speakers and the percentage whose mother has lived in the United States for all or almost all of her life. Furthermore, given the number of baseline variables, two is a small number of significant differences: we would expect about two significant differences even if the differences between the two groups were purely random. Although there is no guarantee that matching removed all unmeasured differences between the two samples, matching removed differences on a broad range of baseline variables—differences that might otherwise bias the impact estimates.  

### 4.	Estimating the Impacts of UBMS Participation

To measure the effects of UBMS participation on participating students, we used a regression-based approach that allows us to (a) adjust for the small remaining differences between the UBMS participant sample and the matched comparison group and (b) increase the precision of our impact estimates. The regression models yield estimates of the effect of UBMS on students who participated in the program. We regressed each outcome on a set of control variables and an indicator of whether the student participated in UBMS. The control variables included the variables used in selecting the matched comparison group: prior participation in regular Upward Bound; siblings in Upward Bound; sex; race; ethnicity; mother’s native language and immigrant status; high school cohort; region of the country; and several variables describing academic achievement in ninth grade, including GPA (separately for math and science courses and for other courses), math course taken, and science course taken. For continuous variables such as number of college credits, we estimated linear regression models; for categorical outcomes such as whether the sample member pursued postsecondary studies in math or science, we estimated logistic regression models or “logit” models. In estimating standard errors, we accounted for clustering by project and implemented the Taylor series linearization methods used by the SUDAAN statistical analysis software. 

We estimated the effects of UBMS for the entire sample. However, the effects of UBMS may depend on the amount of other precollege services received. Therefore, we present separate impact estimates for those who participated in regular Upward Bound and those who did not participate.    

## b.	DATA SOURCES and OUTCOMES MEASURES

### 1.	Outcome Measures

The outcomes for which impact estimates are presented in this report can be grouped into four areas: postsecondary enrollment, persistence, completion, and field of study. 

**Postsecondary enrollment.**We estimate the impacts of UBMS on enrollment at any type of postsecondary educational institution, along with the highest level of postsecondary institution attended as well as the selectivity of four-year colleges and universities attended. Highest level of enrollment was defined as *four-year* for sample members who attended a public or private, nonprofit, four-year college or university; *two-year* for sample members who attended a public or private, nonprofit, two-year college, but not a four-year college or university; and *vocational* for sample members who attended a for-profit institution, but no two- or four-year institution. 

Selectivity of four-year colleges and universities attended was measured by using school ratings from *Barron’s Profiles of American Colleges* (Barron’s Educational Series 2003). If a school was rated as “most competitive,” “highly competitive” or “very competitive,” we classified the school as *more selective*. If a school was rated as “competitive,” “less competitive,” “noncompetitive,” “special” or unrated, or was excluded from *Barron’s*, we classified the school as *less selective*. According to the classification system, *more selective* colleges and universities generally accept less than 75 percent of applicants, and students at *more selective* institutions were generally in the top half of their high school class. *Less selective* postsecondary institutions generally admit more than 75 percent of their applicants. The values of the four-year college or university selectivity outcome variables are set to zero for sample members who did not attend a four-year college or university.

**Postsecondary persistence.**To assess progress toward completing and attaining postsecondary credentials, we requested transcripts from the relevant institutions for the sample members who reported having attended them. Credits were tabulated by level of institution and field of study.

**Postsecondary completion.**We estimate the impacts of UBMS on completion of any postsecondary credential and on the highest postsecondary credential (degree, certificate or license) earned. Highest credential was defined as a *four-year degree* for sample members who earned a bachelor’s degree or higher; a *two-year degree* for sample members who earned an associate’s degree, but not a bachelor’s degree; and a *certificate or license* for sample members who earned a postsecondary certificate or license, but no higher degree. 

**Field of study.**To determine whether a sample member had pursued postsecondary studies in math and science, we asked sample members for their “most recent or intended field of study,” and we classified their responses according to the same classification system used in the National Science Foundation’s Scientists and Engineers Statistical Data System. Furthermore, for the analysis, we separated the social sciences from other math and science fields (which we refer to as “math or science”) because the objectives of the program are more closely tied to math and science than to social sciences.  

The fields classified as science and engineering were biological sciences, computer science, engineering, mathematics, physical sciences and technical fields. The fields classified as nonscience and engineering were agriculture, arts, business, education, clerical or legal assistance, communications, health-related fields, humanities, trade and industry, protective services, and consumer or personal services. A small number of fields reported by sample members could not be classified as either science and engineering or nonscience and engineering. (For more details about NSF’s classification of fields of study, see http://www.nsf.gov/sbe/srs/nsf99337/pdf/appa.pdf.)

To measure these postsecondary outcomes, we use data from the fifth follow-up survey and from administrative records. We describe these different data sources below, along with their strengths and weaknesses in providing valid information for correctly measuring these outcomes of interest.

### 2.	Data Sources

The analyses described in this report are based on information provided (a) by UBMS participants and comparison group members during the follow-up surveys and by the postsecondary institutions that they reported attending as well as (b) by two administrative data sources. 

**Surveys and transcripts.**UBMS sample members provided retrospective baseline information during the 1998–1999 survey, and similar information was obtained for comparison group members from their baseline questionnaire when they applied to Upward Bound. The estimates in this report rely substantially on data from the follow-up survey conducted in 2003–2004. This survey focused on obtaining information from sample members about their postsecondary educational attainment. After the follow-up survey, we also collected transcripts from high schools and postsecondary educational institutions attended by sample members. 

**Administrative data.** We collected data from other sources that allow us to augment our survey and transcript data. The NSC collects enrollment and degree information from the majority of colleges and universities in the United States, enabling it to provide verification of these activities by institution and semester. The federal SFA records are based on the Free Application for Federal Student Aid (FAFSA) filled out by most college aspirants, and include information on aid application and receipt of Pell funding.  

### 3.	Construction of the Outcome Measures

The data available from the follow-up surveys, the NSC and the SFA records were used to construct various outcome measures in three different ways: using only the fifth follow-up survey, using only administrative records, and blending data from the surveys and the administrative sources in different combinations. Because data from the NSC were available for a period of time after the fifth follow-up survey was completed, we constructed two versions of an outcome when data from the NSC records were used: one using all the information available from the NSC records (NSC Full) and the other using information available from the NSC by the end of calendar year 2004, when the fifth follow-up survey was complete (NSC Truncated). A more detailed discussion about the construction of various outcome measures using these different data sources is provided in Appendix B of Seftor, Mamun, and Schirm (2009). 

We used the different data sources because they have different relative strengths and weaknesses. In conducting the impact analysis for this report, our basic principle was to use the maximum amount of information that was available on the sample members. Although the follow-up surveys provided data on a broad range of outcomes, we faced the problem of not having data for survey nonrespondents, and the nonrespondents might be systematically different from respondents, potentially leading to nonresponse bias in our estimates. The NSC and the SFA data are two convenient resources to mitigate this problem because we could get information on both survey respondents and nonrespondents from these administrative records.  

However, these administrative sources have their own limitations. The NSC does not cover the entire universe of postsecondary schools and does not cover all member schools for the entire relevant time period. Nationally, current rates of coverage are 87 percent for students attending a two-year institution and 90 percent for students attending a four-year institution. The coverage rates were lower in earlier years (the NSC data go back to 1993–1994); in terms of total U.S. college enrollment, the NSC data rose from 57 percent in 1997 to 88 percent in 2002, with small increases in subsequent years. Thus, the NSC might be missing data for a sample member who attended and potentially completed his or her education at a postsecondary institution because the institution was not covered by the NSC during the relevant years. SFA records provide data on all sample members; however, they do not have information on postsecondary completion, and they provide information on enrollment for only some students (those who received a Pell grant). 

## c.	Analysis and Findings 

As documented in detail in this section, UBMS affected the types of institutions participants attended, both in level and selectivity. UBMS increased enrollment at four-year and selective colleges, and some of this increase can be attributed to students who would otherwise have attended only two-year colleges. We found evidence that UBMS increased persistence and completion rates, primarily through the completion of four-year degrees. In terms of field of study, we found that UBMS raised the likelihood of pursuing postsecondary studies and of completing a four-year degree in social science, but we found no detectable effect that the program increased these outcomes in math and science fields.

### 1. 	The Effect of UBMS on Postsecondary Attendance, Persistence and Completion

Although UBMS focuses on preparing students to major in math and science and to complete a degree in a math and science field, a person must enroll in college before choosing a major and must complete college to earn a degree in a math and science field. Even if UBMS had little effect on students’ choice of major, UBMS might be a cost-effective strategy to increase college enrollment and completion for disadvantaged students. Therefore, we assessed whether UBMS promotes postsecondary attendance, persistence and completion before we examined its effects on college major. 

Exhibit III.3 presents the effect of the participation in UBMS on any postsecondary enrollment (four-year, two-year or other), using a variety of enrollment measures. We present findings on a range of measures, based on a number of data sources and assumptions; a detailed description of the measures can be found in Appendix A. Each row in the exhibit presents the results from a separate analysis, using one version of the outcome measure; thus, Exhibit III.3 presents 27 versions of the “any postsecondary enrollment” outcome. The “Data Source” column describes the sources that were used in the construction of the particular measure and, where relevant, the order of their use. The “Uncoded” column provides details on how we dealt with cases for which the data do not provide definitive evidence of enrollment. The remaining columns present the findings from the analysis, including the regression-adjusted UBMS participants mean, the comparison group mean, the impact of UBMS on the measure, an indicator for statistical significance, and the *p*-value underlying the significance indicator.

As noted earlier, the evidence suggests that, on average, UBMS serves students who perform well in high school and attend college at higher rates than average low-income students. Estimates for the matched comparison group using the fifth follow-up survey suggest that 91 percent of UBMS participants would have attended a postsecondary institution if they had not participated in UBMS, leaving little room for improvement because of UBMS participation. We find that the impact of UBMS on overall postsecondary enrollment is unclear, with mixed results that are sensitive to the data source. More specifically, we either could not compute an impact or found small negative effects when the data source included the NSC data because the postsecondary enrollment rate for the matched comparison group was 100 percent.  

However, underlying the cloudy picture for overall enrollment is a much clearer finding of significant effects of UBMS participation on the types of postsecondary institutions attended. The impact estimates were (a) all positive (ranging from 8.6 to 18.2, with an average of 12.0) and significant for attending a four-year institution; (b) all negative (ranging from –9.6 to –3.7, with an average of –6.06 and only one estimate not statistically different from zero) for attending a two-year college, but not a four-year college or university; and (c) nearly all negative (ranging from –1.7 to 0.2, with an average of –1.0 and one estimate significant) for attending a school other than a two- or four-year institution (Exhibits III.4 through III.6). In other words, it appears that some of the increase in four-year college enrollment can be attributed to students who would otherwise have attended two-year colleges. 

Our analysis by previous participation in the regular Upward Bound provides additional insight. Almost all impacts on attending a four-year institution are larger for UMBS participants without previous participation in regular Upward Bound (see Exhibit B.2 in Appendix B). Moreover, the overall negative effect on enrollment in a two-year college is driven almost exclusively by the impact for students who did not participate in Upward Bound (see Exhibit B.3 in Appendix B). These findings suggest that UBMS increases four-year enrollment for both regular Upward Bound participants and nonparticipants, but leads to an additional shift from two-year to four-year enrollment for those who did not have prior Upward Bound participation. 

We also examined whether UBMS affected the level of selectivity of the four-year colleges and universities attended by participants. A substantial portion of UBMS participants who attended four-year institutions attended more-selective institutions, and the difference between treatment and comparison means is positive (ranging from 12.2 to 22.9, with an average of 18.6) and statistically significant for all of the alternative measures (Exhibit III.7). Our subgroup analysis by previous participation in the Regular Upward Bound confirms these findings (see Exhibit B.5 in Appendix B).

Transcript information shows that the number of credits earned within seven to nine years after scheduled graduation for the comparison group was 83 and for the treatment group 95 (Exhibit III.8). This 12-point difference was not statistically significant in our analyses. 

Finally, data from survey responses was combined with information from the NSC to create measures of postsecondary completion at various types of schools. UBMS increased the likelihood of completing any postsecondary credential (ranging from 9.0 to 12.1, with an average of 11.0) (Exhibit III.9). This increase in overall postsecondary completion is primarily attributable to an increase in the percentage of sample members whose highest credential was a bachelor’s degree (ranging from 11.8 to 16.8, with an average of 15.7, and statistically significant for all of the measures); the program decreases the likelihood of completing an associate’s degree (ranging from –5.0 to –2.7, with an average of –3.8, and most negative impacts significant). UBMS did not have a detectable effect on the likelihood of completing a certificate or license (ranging from –0.8 to 0.5, with an average of -0.1 and only one estimate significant). 

### 

### 2.	The Effect of UBMS on Postsecondary Field of Study

A primary objective of UBMS is to prepare students for postsecondary studies in math and science. The evidence suggests that UBMS participation encourages students to pursue postsecondary studies in math, science or the social sciences. UBMS increased the likelihood of pursuing postsecondary studies in the social sciences from 8.2 percent to 12.8 percent overall and from 7.4 percent to 12.5 percent at four-year institutions, though neither is statistically significant. UBMS also increased the likelihood of majoring or intending to major in math or science from 25.8 percent to 31.6 percent overall and from 20.2 percent to 24.7 percent at four-year colleges and universities, although neither of these impacts is statistically different from zero at conventional levels (see Exhibit III.10). 

The study found that 8.2 percent of the comparison group majored or intended to major in social sciences, and 12.8 percent of the UBMS participants so reported. Among those at four-year institutions, 7.4 percent of the comparison group and 12.5 percent of the UBMS participants majored in the social sciences. Neither of these differences was statistically significant in our analyses.

Considering math and sciences, the percentage majoring in or intending to major in these fields is 25.8 for the comparison group and 31.6 percent for the UBMS group overall. Again, neither of these differences was statistically significant.  

Exhibit III.3

Impact of Upward Bound Math-Science (UBMS) on Any Postsecondary Enrollment, percentage of students

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. ! Impacts could not be assessed due to complete or semicomplete separation. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound. 

**/*** Statistically significant at the 0.05/0.01 levels, respectively.

Exhibit III.4

Impact of Upward Bound Math-Science (UBMS) on Highest Level of Postsecondary Enrollment: Four-Year Institution, Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

*** Statistically significant at the 0.01 level.

Exhibit III.5

Impact of Upward Bound MathScience (UBMS) on Highest Level of Postsecondary Enrollment: Two-Year Institution, Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

**/*** Statistically significant at the 0.05/0.01 levels, respectively.

Exhibit III.6

Impact of Upward Bound Math-Science (UBMS) on Highest Level of Postsecondary Enrollment: Other Institution, Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

* Statistically significant at the 0.1 level.

Exhibit III.7

Impact of Upward Bound Math-Science on Attendance at a Highly-Selective Four-Year Postsecondary Institution, Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

**/*** Statistically significant at the 0.05/0.01 levels, respectively.

Exhibit III.8

Impact of Upward Bound Math-Science (UBMS) on Total Postsecondary Credits Earned, Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

** Statistically significant at the 0.05 level.

Exhibit III.9

Impact of Upward Bound Math-Science (UBMS) on Completion of Any Credential and Highest Credential Completed (Intent To Treat), Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

*/**/*** Statistically significant at the 0.1/0.05/0.01 levels, respectively.

Exhibit III.10

Impact of Upward Bound Math-Science (UBMS) on Postsecondary Field of Study (Intent To Treat), Percentage of Students

Source: Mathematica Policy Research, Inc. analysis file “ITT_UBMS.sas.”

Notes: PS indicates postsecondary. Impact may not exactly equal the difference between Upward Bound Math-Science Participants and the Matched Comparison Group due to rounding. All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4). See Exhibits B.1 through B.8 in Appendix B for estimates by prior participation in Regular Upward Bound.

*/** Statistically significant at the 0.1/0.05 levels, respectively.

# IV. inteRpretation of findings

The estimated effects of UBMS paint a fairly consistent picture of UBMS improving participants’ postsecondary educational outcomes. Specifically, it increases enrollment at four-year and more selective institutions, increases math and science course taking, and increases postsecondary degree completion overall and at four-year institutions.

Although we took several steps to reduce selection bias, it is certainly possible that the true effects of participating in UBMS are smaller than our estimates suggest. As reported earlier in the chapter, UBMS participants may be more serious about school than regular Upward Bound participants on average. This difference is reflected in our data: UBMS participants in our sample had higher GPAs and took more advanced math and science courses in ninth grade than members of the regular Upward Bound sample. We accounted for these differences by selecting a matched comparison group that resembled the UBMS participant sample in grades earned and in course taking in ninth grade. However, it is possible that despite earning similar grades and taking similar courses early in high school, the UBMS participant sample is somewhat more serious about school, more serious about math and science, or is different from the matched comparison group in some other way that would lead our analysis to overstate the effects of UBMS.

Although we cannot measure the extent of selection bias, some informed speculation is helpful in interpreting the impact estimates. As we indicated earlier, students in both groups—the UBMS participant sample and the matched comparison group—exhibited some motivation to improve academically by applying to participate in Upward Bound. Therefore, motivational differences between the two groups are likely to be small and unlikely to bias the impact estimates. However, the impact estimates would overstate the true impacts if any of the following were true:  

***Members of the UBMS participant sample were higher achievers than members of matched comparison group.*** By the end of high school and after students in the UBMS sample had participated in UBMS, the average GPA was slightly but significantly higher for UBMS participants (3.14) than for matched comparison students (3.06). Even if the entire difference were attributable to selection bias instead of the effects of the program, the average student in each group was a B student. Therefore, if the UBMS participant sample contains higher achieving students than the matched comparison group, the difference seems to be small.

***Members of the UBMS participant sample were better at math and science than members of the matched comparison group.*** If students in the UBMS participant sample were better at math and science than students in the matched comparison group, it is not reflected in their math and science course taking or in their grades during ninth grade: the two groups had similar grades in math and science, were equally likely to take algebra or geometry and were equally likely to take biology, chemistry or physics. Although we did not explicitly examine whether UBMS participants were more likely to enroll in the advanced sections of courses before entering UBMS, other findings suggest that UBMS participants were no more likely to have taken Advanced Placement classes in math or science by the time they finished high school. Finally, focus groups conducted in 1996 suggest that UBMS participants do not view themselves as strong in math and science (Moore 1997, 28). Therefore, the information we have collected provides no reason to believe that when they entered UBMS, the participant sample was better at math and science than the matched comparison group.

***Members of the UBMS participant group were more interested in math and science than members of the matched comparison group.*** None of the information that we extracted from student transcripts would suggest that the UBMS participant sample had******greater interest in math and science. It is possible that UBMS participants had greater interest in careers in math and science that simply was not reflected in their high school course taking or in their grades early in high school. In 1996 focus groups, many UBMS participants expressed interest in pursuing careers in scientific fields such as engineering, medicine, and nursing (Moore 1997, 28). However, many of these students indicated that their career interests had developed just that summer, and the expression of those interests could have been influenced by the fact that the focus groups were conducted on site at projects that emphasized math and science.

We suspect that the impacts most vulnerable to selection bias are those that are most closely related to a person’s interest in pursuing careers in a math or science field. The ability to gauge students’ interest in math and science early in high school would probably require conducting assessments or survey interviews at that time. However, we first interviewed members of the UBMS participant sample after they had completed high school. The possibility that UBMS participants might have had greater interest in pursuing a career in science than matched comparison students raises the question of whether our estimates overstate the effects of UBMS on the outcomes that are most closely related to one’s career interests such as majoring in math or science in college.

Although the findings in this report are promising, a note of caution is appropriate. We speculate that the selection bias is likely to be largest for outcome variables most closely tied to one’s interest in pursuing math and science careers, but it is not possible to measure the selection bias. Although we took several steps to reduce selection bias, the estimated effects of UBMS may overstate the true effects of the program.  

However, it is also worth noting that much of the impact of UBMS on postsecondary enrollment is through the type of institution attended, with a shift from two-year to four-year institutions. This finding was true for all UBMS participants, but was even stronger for those who had not participated in a regular Upward Bound program. The vast majority of UBMS projects are hosted by four-year institutions, and it is conceivable that part of the impact of UBMS is through exposure to that environment and its subsequent role in the enrollment decisions of participants.  

# References

Barron’s Educational Series, College Division. 2003. *Barron’s profiles of American colleges.* Woodbury, NY: Author. 

Curtin, Thomas R., and Margaret W. Cahalan. 2004, December. *A profile of the Upward Bound Math-Science Program: 2000–2001*. Washington, D.C.: U.S. Department of Education, Office of Postsecondary Education.

Fasciano, Nancy J., and Jon E. Jacobson. 1997, May. Grantee survey report. In *A 1990s view of Upward Bound: Programs offered, students served and operational issues*, by U.S. Department of Education, Office of the Under Secretary. Washington, D.C.: U.S. Department of Education.

Heckman, James J., Jeffrey Smith, and Nancy Clements. 1997. Making the most out of programme evaluations and social experiments: Accounting for heterogeneity in programme impacts. In Evaluation of Training and Other Social Programmes, special issue, *Review of Economic Studies* 64 (4):487–535. http://www.jstor.org/stable/2971729 (accessed January 10, 2010).

Ingels, Steven J., T. R. Curtin, Phillip Kaufman, Martha Naomi Alt, and Xianglei Chen. 2002. Coming of age in the 1990s: The eighth-grade class of 1988 12 years later. (NCES 2002–321). U.S. Department of Education, National Center for Education Statistics. Washington, D.C.: U.S. Government Printing Office. 

Moore, Mary T. 1997, September. *Developing math and science skills among disadvantaged youth: A review of the Upward Bound precollege math/science centers*. Washington, D.C.: U.S. Department of Education, Planning and Evaluation Service.

Myers, David. 1997, May. *Measuring the impact of the Upward Bound math/science initiative: Feasibility of an evaluation*. Washington, D.C.: U.S. Department of Education, Policy and Program Studies Service.

Myers, David, Rob Olsen, and Neil Seftor. 2002, September. The impacts of Upward Bound: High school experiences and college access. Presentation at the annual conference of the Council for Opportunity in Education, Washington, D.C.

Myers, David, Robert Olsen, Neil Seftor, Julie Young, and Christina Tuttle*.*2004, April. *The impacts of regular Upward Bound: Results from the third follow-up data collection*. Washington, D.C.: U.S. Department of Education, Policy and Program Studies Service.

Myers, David, and Allen Schirm. 1999, April. *The impacts of Upward Bound: Final report for Phase I of the national evaluation*. Washington, D.C.: U.S. Department of Education, Planning and Evaluation Service.

National Science Foundation. n.d. Women, minorities and persons with disabilities in science and engineering, table C-6, minority share of S&E and non-S&E bachelor’s degrees: 1998–2007. http://www.nsf.gov/statistics/wmpd/race.cfm#degrees (accessed November 10, 2009).

Olsen, Robert, Neil Seftor, Tim Silva, David Myers, David DesRoches, and Julie Young. 2007. *Upward Bound Math-Science: Program description and interim impact estimates.* Washington, D.C.: U.S. Department of Education, Office of Planning, Evaluation, and Policy Development.

Rubin, Donald B. 2002. Using propensity scores to help design observational studies: Application to the tobacco litigation. *Health Services and Outcomes Research Methodology* 2:169–88.

Seftor, Neil, Arif Mamun, and Allen Schirm. 2009. *The impacts of regular Upward Bound on postsecondary outcomes 7–9 years after scheduled high school graduation.* Washington, D.C.: U.S. Department of Education, Policy and Program Studies Service.

U. S. Census Bureau. 2001. Population by race and Hispanic or Latino origin for the United States: 1990 and 2000. http://www.census.gov/population/www/cen2000/briefs/tablist.html (accessed November 10, 2009).

U.S. Department of Education, Office of Educational Research and Improvement, National Center for Education Statistics*.* 1994.*The condition of education 1994*. NCES 94-149. Washington, D.C.: Author.

———.**1996a. *The condition of education 1996*. NCES 96-304. Washington, D.C.: Author.

———. 1996b. *National education longitudinal study of 1988: High school seniors’ instructional experiences in science and mathematics*. NCES 95-278. Washington, D.C.: Author.

———. 1999. *Digest of education statistics 1998*. NCES 1999-036. Washington, D.C.: Author.

U.S. Department of Education, Office of the Under Secretary. 1997. *A 1990s view of Upward Bound: Programs offered, students served and operational issues.* Washington, D.C.: U.S. Department of Education.

54

# APPENDIX A

# DATA COLLECTION AND OUTCOMES MEASURES

This appendix describes and assesses the procedures for collecting the data that we used to measure the outcomes examined in this report. The data come from two types of sources:

1. Follow-up surveys of sample members 

2. Administrative records (National Student Clearinghouse and Student Financial Aid)

This appendix focuses on procedures for obtaining completed interviews in the fifth follow-up survey, collecting administrative records, and constructing outcome measures from multiple sources. 

## A.	Fifth Follow-Up Survey of Students

The fifth follow-up survey was conducted between July 2003 and December 2004. It was designed to collect information on postsecondary outcomes. 

### 1.	Data Collection Modes

One week before we began interviewing, we sent a letter to all study participants. The letter stated that we would call them to complete an interview for an important study. In addition, the letter encouraged the individuals’ participation in the survey and noted that we would pay respondents $10 for completing the interview. Toward the end of the data collection, the U.S. Department of Education (ED) requested that the Office of Management and Budget approve an increase in the incentive to $25 in September 2004; the final 180 interviews were collected under the higher incentive scheme.

Most interviews were administered using computer-assisted telephone interviewing (CATI). CATI interviews took about 30 minutes to complete. When a CATI interview was not possible, we attempted to obtain a completed questionnaire through the mail. Study participants also had the option of completing the survey on the Internet. Throughout the data collection process, we mailed questionnaires to study participants when respondents requested it or when we determined they could not be reached by telephone. We conducted three follow-up mailings after the first mailing, with the last set of questionnaires sent out in December 2004. 

### 2.	Locating

Throughout the data collection period, locating staff members used services such as LexisNexis and Internet databases to obtain updated addresses and telephone numbers for difficult-to-reach study participants. 

### 3.	Incentives

To obtain a high response rate, we used financial incentives for survey completion. As noted, we offered study participants $10 or $25 for responding to the survey. We mailed incentive checks after the sample member completed the interview.

### 4.	Response Rates 

The eligible sample consisted of 687 UBMS participants (two were deceased) and 988 sample members from the evaluation of regular Upward Bound. (See Chapter III for more details on the samples.) We obtained completed interviews for 534 UBMS participants and 804 regular Upward Bound sample members for response rates of 78 percent and 81 percent, respectively.  

## B.	Transcript Data Collection

For the fifth follow-up analysis of Upward Bound Math Science, we collected postsecondary transcripts between January 2004 and May 2006. We made transcript requests to institutions that were reported by sample members in either the fifth follow-up survey of sample members or earlier surveys. 

### 1.	Preparation for Requesting Transcripts

Information about students’ secondary and postsecondary enrollment was primarily obtained from follow-up interviews. Students reported the secondary and postsecondary institutions that they had attended. Secondary transcripts were requested only from UBMS sample members selected for the impact analysis; postsecondary transcripts were requested from all sample members—both UBMS and regular Upward Bound—who reported or confirmed having attended a particular postsecondary institution.[footnoteRef:6]   [6: . We did not collect secondary transcripts for regular Upward Bound sample members because we had already collected these transcripts for a large percentage of the sample in previous waves of data collection.   ] 

To obtain mailing addresses for the schools that were attended by sample members, we matched schools that were reported by survey respondents to directories of secondary and postsecondary schools maintained by ED. Secondary schools were matched to the Common Core of Data; postsecondary schools were matched to the integrated Postsecondary Education Data System.[footnoteRef:7]  [7: . Students were asked to provide the name and state of each secondary and postsecondary school they attended, but sometimes misspellings or incomplete information resulted in some invalid requests for student transcripts because schools were matched with an incorrect address and transcripts were requested from the wrong school. When a school indicated that they could not fill a request because they had no record of the student whose transcript we requested, it was sometimes because of such mismatches. In these cases, we attempted to learn the correct name and address of the school where the student was enrolled and make a new transcript request.] 

### 2.	Procedures for Requesting Transcripts

Each school was sent a transcript request packet that included the following: 

A letter, printed on ED letterhead, which explained the purpose of the study and the reason we were requesting transcripts

A statement of Authorization and Confidentiality, which cited the *Family Educational Rights and Privacy Act* and included questions and answers regarding consent and confidentiality

A transcript checklist of all the materials that we requested from the school, including student transcripts, a course catalog, grade descriptions and a transcript reimbursement form, which would indicate the reimbursement that the school required for providing the requested transcripts 

A postage-paid business reply envelope for sending the transcripts

A disclosure notice to be placed in each student’s file, indicating that a copy of his or her transcript was released to Mathematica Policy Research, Inc. (Mathematica) as an agent to ED 

### 3.	Follow-Up Procedures 

In instances when schools did not respond to our initial request for transcripts, we mailed another request for student transcripts. These mailings were done periodically as we tracked the schools that had not yet sent the requested transcripts and corrected requests that contained errors.  

As the targeted end date for collecting transcripts approached, interviewers started calling schools directly to inquire about the status of our requests. Many schools responded to these calls by faxing us the requested transcripts. When the school indicated that they could not provide one or more of the requested transcripts, the interviewer completed a problem sheet indicating the reason. The reason generally fell into one of the following categories:

***The student was never enrolled at the school according to the school’s records.***When this situation occurred, our first response was to call the school and provide more information on the student (e.g., provide or verify date of birth and dates of attendance) to see whether a transcript could be located with additional information. In many cases, the school was able to locate and provide transcripts once additional information was provided. In other cases, the school provided some information that helped us determine where we might obtain the needed transcripts.[footnoteRef:8] If the school had no record of the student having ever attended and we were unable to obtain additional information, then we marked the case as an invalid request.   [8: . For example, some school principals and registrars indicated that their school was often confused with another school having the same or a similar name and suggested that we direct our request to the other school. In this case, we would call the alternate school to find out whether the student was ever enrolled there. If so, we made a correction to the database and sent a request to the newly identified school.] 

***Transcripts were held by the school district.*** Some schools held only the transcripts of currently enrolled students, and all other transcripts were sent to the school district. In this situation, the school would sometimes forward the request packet to the district. Other times, the school returned the materials to us, and we sent them to the school district.

***The student transferred to another school.*** When the student had transferred to another school, a transcript was requested from the school to which the student had transferred. In some cases, the registrar or school secretary forwarded the request materials to the transfer school. In other cases, the request materials were sent back to us and we sent a new request to the transfer school.

***The school would not release any transcript without student’s written consent.*** A few schools returned the transcript request materials with no transcripts, indicating that they required written consent from each student whose transcript we were requesting. A problem sheet was completed for these cases, and they were forwarded to the survey manager for follow-up. As a first step, the survey manager called the school to explain that, as an agent of ED, Mathematica was authorized to collect student transcripts for the purposes of this study and that, according to the laws of the *Family Educational Rights and Privacy Act*, schools are permitted to release student transcripts to ED without the written consent of students participating in the study. It was also explained that students had given verbal consent over the telephone or written consent when they completed the mail survey and that we did not request transcripts for any students who refused consent. Some schools agreed to send the requested transcripts on hearing this explanation. Others reiterated that signed consent was required by school policy. In this case, we sent written consent forms to the students for them to sign and return to Mathematica so that we could obtain their student transcript for the impact study. A postage-paid return envelope was included with the consent form. A small number of students did sign and return the consent form, but most of the letters came back unopened because we no longer had a valid address for the student.

***The school would not release transcripts without advance payment.***In these cases, we sent a check to cover the cost of each transcript, along with a list of the students whose transcripts we were requesting.

***The school would not release a transcript until the student paid an outstanding debt.*** In some cases we were eventually able to obtain these transcripts as students paid whatever bills they owed the school. When the debt remained unpaid, however, there was no way we could get the transcript. These cases were marked as unfilled requests. 

### 4.	Response Rates

From the samples used in the impact analysis described in Chapter III, 1,201 students reported having attended at least one postsecondary institution that could be matched to the Postsecondary Education Data System. For each of these students, we requested a transcript from each of the postsecondary institutions that he or she reported attending. We received 1,756 of the 2,295 transcripts requested (77 percent).  

## C.	National Student Clearinghouse Data

The National Student Clearinghouse (NSC) is a repository of information on enrollment, loans and degrees awarded for postsecondary institutions that join the NSC as members. The NSC is a nonprofit organization that began in 1993 with support from ED to verify enrollment for student loan recipients. Currently, the NSC is an enrollment and degree verification resource for ED, colleges and universities, and employers. It is supported by fees paid by the member institutions as well as by fees for enrollment and degree searches. Because the NSC began operations only two years before most of the sample members in this study were graduating from high school (1995–1997), caution is needed in using the data from this period. The NSC Web site reports that it reached about 25 percent coverage for enrollment data in 1996, and that it did not begin to collect data on degrees awarded until sometime later. Coverage is more complete for four-year institutions than for two-year and less-than-two-year institutions. By 2009, the NSC reported coverage of about 10,000 institutions representing about 92 percent of enrollment.[footnoteRef:9] We submitted identifying information for all 1,675 eligible sample members from the treatment and control groups; however, as noted above, because of the early time frame, NSC data should be used with caution because it probably underreports postsecondary entrance and degree attainment in this period.  [9: . Figures are based on http://www.studentclearinghouse.org.] 

## D.	Federal Student Financial Aid Data

The Office of Postsecondary Education in ED provided consistent, comprehensive data on applications for federal financial aid. We were able to obtain files including information on all sample members who applied for aid for each year of the follow-up period. The data source for all applications for federal financial aid is the Free Application for Federal Student Aid. The main data items of interest were application for financial aid and receipt of a Pell grant. The primary approach for searching for someone in the Student Financial Aid (SFA) data is based on the person’s Social Security number. For cases with a missing or obviously invalid Social Security number, attempts were made to match using name and date of birth. During the years of interest, 1,400 (84 percent, unweighted) of our sample members applied for aid in at least one year, and 1,108 (66 percent, unweighted) received a Pell grant. 

## E.	Combining Data

Each of our three data sources—the follow-up surveys conducted for the evaluation, the NSC, and the federal SFA files—contain valuable information for measuring postsecondary educational outcomes. However, each also has some important limitations. The limitations of the surveys include nonresponse and, potentially, response error. For the fifth follow-up survey, 26 percent of sample members did not respond at all. Although there is very little item nonresponse among respondents, some respondents might not have answered questions correctly, forgetting, for example, a brief period of college enrollment several years earlier. The main limitation of the NSC is undercoverage; that is, not all postsecondary institutions are in the NSC. Because a substantial fraction of postsecondary enrollees never receive financial support from a Pell grant, the main limitation of the SFA data is measurement error and, specifically, the fact that Pell grant receipt or nonreceipt is not equivalent to postsecondary enrollment or nonenrollment.

In using these data sources and determining how to address effectively their relative limitations, several questions arise, including the following. How might we combine the data sources? How much of the available data do we use, recognizing that the different data sources do not cover exactly the same periods of time? How do we handle cases for which the data that are used do not provide definitive evidence about postsecondary enrollment status? Because none of these questions has a single, unambiguously correct answer, our approach has been to develop many different measures of outcomes that reflect different reasonable answers to the questions.

### 1.	Enrollment Measures

We constructed measures using only the fifth follow-up survey (Measure 1), only the NSC (Measure 2), and only the SFA data (Measure 3). We used the Pell receipt indicator variable from the SFA data for the construction of measures of enrollment because it was an indication that a school actually received Pell grant money after a student’s enrollment. The indicator of application for financial aid, in contrast, is not necessarily indicative of subsequent enrollment. We also created measures using each possible combination of data, including NSC and SFA (Measure 4), SFA and survey (Measure 6), NSC and survey (Measure 7), and all three data sources (Measures 5 and 8). The last two measures differ only in the order in which the data were used: Measure 5 uses all data sources simultaneously while Measure 8 uses the survey data first. The final combination, Measure 9, augments all three current sources with data from the third and fourth follow-up surveys.

Although the survey and SFA data available cover roughly the same reference periods, the NSC data are available for a period of time after the completion of the fifth follow-up survey and the years for which financial aid records were available to us. One advantage of truncating the NSC data at the end of 2004 is that we have a similar reference period for all data sources; however, this approach also means discarding relevant data for sample members who may have taken longer to enroll in school. Therefore, for all measures using NSC data, we created versions using the full data through 2006 and truncated data through 2004.

For each combination of data sources that we consider, we ascertain the enrollment status of as many cases as we can with the information that is available in the applicable data sources. With Measure 5, for example, we code a sample member as an enrollee if he or she is found to be an enrollee in the NSC data *or* if he or she is a Pell recipient according to the SFA data *or* if the sample member said in the survey that he or she was enrolled at some time. The sample member is not an enrollee if he or she does not appear in the NSC data (and is therefore not an enrollee) *and* has not been a Pell recipient *and* said in the survey that he or she had never been enrolled. This approach leaves uncoded the survey nonrespondents who are not in the NSC data and did not receive a Pell grant. For them, we apply the following assumptions: (a) not enrolled, (b) not enrolled if never applied for financial aid (otherwise, left as enrollment status missing), and (c) left as enrollment status missing. With assumptions (b) and (c), cases left as enrollment status missing get dropped from the analysis, and weights for the remaining cases are adjusted to compensate, as described elsewhere in this appendix. When creating measures based on the truncated NSC data, we follow the same steps described above, but ignore all NSC records that have a starting date after July 31, 2004.

Measures 4, 6 and 7 are constructed in similar fashion. Measure 4 combines the NSC and SFA data and has versions based on whether to use the full or truncated NSC file. Measure 6 combines the fifth follow-up survey with the SFA data and has variations based on the assumptions applied to survey nonrespondents. Measure 7 combines the fifth follow-up survey with the NSC data and includes variations based on both the amount of NSC data to include and how to code nonrespondents; it does not, however, have assumption (b) for dealing with uncoded data because that assumption relies on SFA data.

Measures 8 and 9 are similar to Measure 5, with some small exceptions. Rather than treat all data equivalently, as in Measure 5, Measure 8 establishes preference for the fifth follow-up survey. Variables are defined first using survey responses, and data from the other two sources are used only for survey nonrespondents. This approach implies that if we have inconsistent data from multiple sources, the survey data take precedence. Measure 9 uses Measure 5 as a starting point, and augments it with data from the third and fourth follow-up surveys. Any remaining uncoded observations use the following modified assumptions: (e) enrolled if reported enrollment in either the fourth or third follow-up surveys, otherwise not enrolled; (f) not enrolled if reported lack of enrollment in either the fourth or third follow-up surveys and if never applied for financial aid, otherwise left as enrollment status missing; and (g) left as enrollment status missing. 

### 2.	Completion Measures

Just as the NSC data were unable to provide information on financial aid application and receipt, the SFA data were unable to provide information on postsecondary completion. In the creation of measures of completion, we are therefore limited to those that can be constructed using only the fifth follow-up survey and NSC data: Measure 1 (fifth follow-up survey only), Measure 2 (NSC only), and Measure 7 (the two sources combined). Furthermore, the degree completion information in the NSC is less comprehensive than enrollment information (just more than half of the schools provide degree information), adding to the limitations of the data for our analysis.

A-9

# APPENDIX B

# PROGRAM IMPACTS BY SUBGROUPS 

EXHIBIT B.1. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON ANY POSTSECONDARY ENROLLMENT, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”   

Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financ- 3 -ial Aid. All estimates are weighted to account for missing data.  

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. The symbol ! indicates that impacts could not be assessed due to complete or semi-complete separation.

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*/**/*** Statistically significant at the 0.1/0.05/0.01 levels, respectively.

 (B-2)

EXHIBIT B.2. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON HIGHEST LEVEL OF POSTSECONDARY ENROLLMENT, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND: FOUR-YEAR INSTITUTION (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*/**/*** Statistically significant at the 0.1/0.05/0.01 levels, respectively.****

**(B-3)**

EXHIBIT B.3. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON HIGHEST LEVEL OF POSTSECONDARY ENROLLMENT, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND: TWO-YEAR INSTITUTION (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*** Statistically significant at the 0.01 level.

 (B-4)

EXHIBIT B.4. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON HIGHEST LEVEL OF POSTSECONDARY ENROLLMENT, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND: OTHER INSTITUTION (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*** Statistically significant at the 0.01 level.

 (B-5)

EXHIBIT B.5. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON ATTENDING A HIGHLY SELECTIVE FOUR-YEAR POSTSECONDARY INSTITUTION, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*/**/*** Statistically significant at the 0.1/0.05/0.01 levels, respectively.

 (B-6)

EXHIBIT B.6. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON TOTAL POSTSECONDARY CREDITS EARNED, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

* Statistically significant at the 0.1 level.

 (B-7)

    EXHIBIT B.7. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON COMPLETION OF ANY CREDENTIAL AND    HIGHEST CREDENTIAL COMPLETED, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND (ITT)

         Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

         Note: ITT = Intent To Treat; NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*/**/*** Statistically significant at the 0.1/0.05/0.01 levels, respectively.

 (B-8)

EXHIBIT B.8. IMPACT OF UPWARD BOUND MATH-SCIENCE (UBMS) ON POSTSECONDARY FIELD OF STUDY, BY PRIOR PARTICIPATION IN REGULAR UPWARD BOUND (ITT)

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: ITT = Intent To Treat. All estimates are weighted to account for missing data. 

a Impact may not exactly equal the difference between UBMS Participants and the Matched Comparison Group due to rounding. 

b All estimates are weighted to account for missing data. The comparison group estimates and impact estimates are regression adjusted (see Chapter III, Section A.4).

*/** Statistically significant at the 0.1/0.05 levels, respectively.

 (B-9)

# APPENDIX C

# SAMPLE SIZES AND STANDARD ERRORS

EXHIBIT C.1. SAMPLE SIZES AND STANDARD ERRORS FOR REPORTED IMPACT ESTIMATES IN 

EXHIBIT III.3

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

        Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Statistical significance could not be assessed due to complete or semi-complete separation.

a Standard errors account for project clustering and were estimated using Taylor series linearization methods.

b The symbol ! indicates that impacts could not be assessed due to complete or semi-complete separation.****

****

EXHIBIT C.2. SAMPLE SIZES AND STANDARD ERRORS FOR REPORTED IMPACT

ESTIMATES IN EXHIBIT III.4

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. 

† Standard errors account for project clustering and were estimated using Taylor series linearization methods

EXHIBIT C.3. SAMPLE SIZES AND STANDARD ERRORS FOR REPORTED IMPACT ESTIMATES IN EXHIBIT III.5

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. 

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.4. SAMPLE SIZES AND Standard Errors for Reported Impact Estimates IN Exhibit III.6

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

       Exhibit C.5. SAMPLE SIZES AND STANDARD Errors for Reported Impact Estimates IN Exhibit III.7

        Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

         Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

        † Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.6. Sample Sizes and Standard Errors for Reported Impact Estimates IN Exhibit III.8

            Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

           † Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.7. Sample Sizes and Standard Errors for Reported Impact Estimates IN Exhibit III.9

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF=National Student Clearinghouse full; NSCT=National Student Clearinghouse truncated; SFA=Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.8. SAMPLE SIzes and Standard Errors for Reported Impact Estimates IN Exhibit III.10

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

EXHIBIT C.9. SAMPLE SIZES AND STANDARD ERRORS FOR REPORTED IMPACT ESTIMATES IN APPENDIX B, EXHIBIT B.1

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid. Statistical significance could not be assessed due to complete or semi-complete separation.

a Standard errors account for project clustering and were estimated using Taylor series linearization methods. 

b The symbol ! indicates that impacts could not be assessed due to complete or semi-complete separation.****

        Exhibit C.10. Sample Sizes and Standard Errors for Reported Impact Estimates IN    APPENDIX B, Exhibit B.2

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.11. Sample Sizes and Standard Errors for Reported Impact Estimates IN APPENDIX B, Exhibit B.3

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.12. Sample Sizes and Standard Errors for Reported Impact Estimates IN APPENDIX B, Exhibit B.4

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Notes: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

     Exhibit C.13. Sample Sizes and Standard Errors for Reported Impact Estimates IN  APPENDIX B, Exhibit B.5

         Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

          Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

         † Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.14. Sample Sizes and Standard Errors for Reported Impact Estimates IN APPENDIX B, Exhibit B.6

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

Exhibit C.15. Sample Sizes and Standard Errors for Reported Impact Estimates IN APPENDIX B, Exhibit B.7

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

Note: NSCF = National Student Clearinghouse full; NSCT = National Student Clearinghouse truncated; SFA = Student Financial Aid.

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

C-16

Exhibit C.16. Sample Sizes and Standard Errors for Reported Impact Estimates IN APPENDIX B, Exhibit B.8

Source: Mathematica Policy Research, Inc., analysis file “ITT_UBMS.sas.”

† Standard errors account for project clustering and were estimated using Taylor series linearization methods.

 (C-16)

Majority Asian1%

Majority black 30%

Majority Hispanic or Latino3%

Majority white54%

No majority racial and ethnic group12%

Percent of Upward Bound Math-Science Projects

Local Only11%

Statewide or Regional45%

Both45%

No majorityracial and ethnic group37%

MajorityAmerican Indian or Alaskan Native1%

Majority black42%

MajorityHispanic or Latino4%

Percent of UBMS Projects

Percent of UBMS Projects

A.  Use of Different Instructional Methods

Percent of UBMS Projects

B.  Primary Method Used

Separate subjects and interdisciplinary courses37%

Separate subjects37%

Large project11%

Interdisciplinary courses9%

Other6%

$4,514

# image29.wmf

# Microsoft_Excel_97_-_2004_Worksheet14.xls

# image30.wmf

# Microsoft_Excel_97_-_2004_Worksheet15.xls

# image31.wmf

# Microsoft_Excel_97_-_2004_Worksheet16.xls

# image1.wmf

# Microsoft_Excel_97_-_2004_Worksheet1.xls

# image2.wmf

# Microsoft_Excel_97_-_2004_Worksheet2.xls

# image3.wmf

# Microsoft_Excel_97_-_2004_Worksheet3.xls

# image4.wmf

# Microsoft_Excel_97_-_2004_Worksheet4.xls

# image5.wmf

# Microsoft_Excel_97_-_2004_Worksheet5.xls

# image6.wmf

# Microsoft_Excel_97_-_2004_Worksheet6.xls

# image7.wmf

# Microsoft_Excel_97_-_2004_Worksheet7.xls

# image8.wmf

# Microsoft_Excel_97_-_2004_Worksheet8.xls

# image9.emf

# image10.emf

# image11.emf

# image12.emf

# image13.emf

# image14.emf

# image15.emf

# image16.emf

# image17.wmf

# Microsoft_Excel_97_-_2004_Worksheet9.xls

# image18.wmf

# Microsoft_Excel_97_-_2004_Worksheet10.xls

# image19.emf

# image20.emf

# image21.wmf

# Microsoft_Excel_97_-_2004_Worksheet11.xls

# image22.emf

# image23.wmf

# Microsoft_Excel_97_-_2004_Worksheet12.xls

# image24.emf

# image25.emf

# image26.emf

# image27.wmf

# Microsoft_Excel_97_-_2004_Worksheet13.xls

# image28.emf