A Study of Implementation and Outcomes in Upward Bound and 

Other TRIO Programs

Part B: Collection of Information Employing Statistical Methods

December 3, 2012

Prepared for

Institute of Education Sciences 

U.S. Department of Education

Prepared by

Decision Information Resources, Inc.

Abt Associates

Contract Number:

ED-IES-10-R-0015
. Page i

Contents

Part B: Collection of Information Using Statistical Methods

B.1. Respondent Universe and Sampling Methods .................................................................... 3

B.2. Information Collection Procedures..................................................................................... 4

B.3. Methods to Maximize Response Rates............................................................................... 5

B.4. Test of Procedures............................................................................................................... 5

B.5. Individuals Consulted on Statistical Aspects of Design ..................................................... 6
Page 1

INTRODUCTION

This clearance request is submitted to the Office of Management and Budget (OMB) in 

support of data collection and analysis planned as part of A Study of Implementation and 

Outcomes of Upward Bound and Other TRIO Programs, sponsored by the Institute of Education 

Sciences (IES) at the U.S. Department of Education (ED). This request is a resubmission on 

behalf of this project, after ED and OMB determined that an earlier plan for the study was not 

feasible.1

Overview of the Upward Bound Program

The Upward Bound (UB) Program, the oldest and largest of the federal TRIO programs, 

was initiated under Title IV of the Higher Education Act of 1965 as a pre-college program 

designed to help economically disadvantaged high school students (low-income and/or firstgeneration college going) prepare for, enter, and succeed in postsecondary education. With grant 

awards approximating $5,000 per student per year, Upward Bound provides an intensive set of 

services. In the most recent (FY 2012) grant competition, ED sought to make the provision of 

those services more productive and cost effective by incentivizing project applicants to propose 

new delivery methods and approaches that could reduce the cost per student without sacrificing 

implementation quality. Awards were made to 820 Upward Bound grantees totaling $268.2 

million to serve more than 62,500 students. 

The structure of Upward Bound projects is well established, defined largely by specific 

statutory language prescribing project and student eligibility and the services offered. Students 

usually enter the program while in the ninth or tenth grade. Although students may participate in 

Upward Bound through the summer following twelfth grade (for three to four years total), 

participants spend an average of 21 months in the program.2 Upward Bound projects are 

generally operated by two- or four-year colleges. Projects must offer academic instruction as 

well as college counseling, mentoring, and other support services. In addition to regularly 

scheduled meetings throughout the school year, projects also offer an intensive instructional 

program that meets daily for about six weeks during the summer. Upward Bound projects may

provide additional services or supports to students, beyond the set of required core activities. 

Currently, very little is known about the intensity, duration, and mix of services, or how they are 

delivered, particularly for FY 2012 grantees that may have changed implementation strategies 

under the new grant requirements

Overview of the Study and the Role of the Proposed Data Collection 

The Study of Implementation and Outcomes of Upward Bound and Other TRIO 

Programs was awarded in August 2010 as a first step towards launching a congressionally

mandate study. The 2008 reauthorization of the Higher Education Act (HEA) required ED to 

 

1 The original OMB submission (1850-NEW (04471)), for site visits to 20 Upward Bound grantees, was submitted 

on 4/20/11 and later withdrawn.

2 U.S. Department of Education, Office of the Under Secretary, Policy and Program Studies Service, The Impacts of 

Regular Upward Bound: Results from the Third Follow-Up Data Collection, Washington, D.C., 2004.
Page 2

conduct a study that would identify particular institutional, community, and program or project 

practices that are effective in improving Upward Bound student outcomes (20 U.S.C. 1070a-18) 

(see Appendix A). However, the law also prohibited any study of a TRIO program that would 

require grantees to “recruit additional students beyond those that the program or project would 

normally recruit” or that would result “in the denial of services for an eligible student under the 

program or project.” These prohibitions were initially seen as eliminating the possibility of 

conducting a randomized control trial within the Upward Bound program.

The new contract was intended to explore possible designs for and the feasibility of 

conducting a study of the effectiveness of one or more promising practices in Upward Bound. 

Design work completed highlighted the weaknesses of popular, program-supported study

approaches such as matching grantees on certain project characteristics and then assessing 

variation in student outcomes that could be attributed to differences in project implementation.3 

In consultation with OMB in summer 2011, ED moved forward to investigate the options for 

testing out certain “enhanced” practices or alternative strategies for implementing core 

components of Upward Bound using a randomized control trial. 

The data collection for which we are now seeking OMB approval is part of that effort.4 

The survey of FY 2012 Upward Bound grantee project directors has two objectives:

1. To help identify one or more strategies to test as part of a random assignment 

demonstration. One challenge in pursuing a rigorous study of promising practices in 

Upward Bound, given that it is a service-rich program, is understanding how (not just 

whether) projects implement core components, where gaps in services might exist, and 

the prevalence of both the implementation approaches and gaps. In order not to violate 

the statutory prohibition against denying regular Upward Bound services to student 

participants, we have to ensure that any promising strategy we evaluate is not already 

widely implemented but yet is sufficiently attractive to Upward projects to allow us to 

recruit grantees not already implementing the strategy to participate in a research 

demonstration involving random assignment.

2. To provide new and up-to-date information about the services and strategies that 

Upward Bound grantees implement, particularly under the new grant productivity 

requirements. The last survey of program implementation was conducted in 19975and 

the program office has an interest in obtaining information that is more current. The 

 

3 This type of quasi-experimental study would not yield reliable or conclusive results because of the inability to fully 

isolate the effects of any particular implementation strategies. Upward Bound projects differ in the students they 

serve, the relationship between the grantee “host institution” and its participating high schools, the level of external 

resources, and many other factors in addition to their approaches to implementing core components and services. To 

measure the effects of implementing a particular (potentially promising) approach to a UB component an evaluation 

would need to statistically “control for” everything else that might differ among projects and influence outcomes; 

there would inevitably be unknown or not easily measured factors that could have a powerful effect on outcomes.

4

IES has also awarded a separate contract with an option to conduct an RCT of a promising practice within Upward 

Bound, once such a practice or strategy can be identified under the current contract.

5 Moore, Mary T. “The National Evaluation of Upward Bound: A 1990's View of Upward Bound Programs 

Offered, Students Served, and Operational Issues.” Washington, D.C.: U.S. Department of Education, 1997.
Page 3

survey will attempt to capture approaches to program services and supports that 

participants experience and which may improve their prospects of successfully competing 

high school and entering and/or college.

The survey is designed to address the following research questions:

 How are the key components and service areas implemented within Upward Bound?

 What is the prevalence of specific strategies or approaches for implementing service 

areas or program components across grantee projects? 

 Where are there gaps in services that might be linked to improving student outcomes?

 What strategies or approaches may be amenable to testing through an experimental 

design to measure impact on student outcomes? 

The project director survey will be administered to all 820 regular Upward Bound 

grantees in the spring of 2013. Because of the need to collect information on how services are 

implemented, the instrument is detailed and designed to take approximately 40 minutes to 

complete. We are soliciting input from the TRIO community and other college access experts to 

improve the instrument during the public comment period. The results of the data collection will 

be tabulated and released in a summary report.

B.1. Describe (including a numerical estimate) the potential respondent universe and any 

sampling or other respondent selection methods to be used. 

All 820 Upward Bound grantees that received grants in 2012 will be included in the 

survey. The list of funded grantees maintained by the U.S. Department of Education’s Office of 

Postsecondary Education (OPE) will be used to identify the universe of UB projects. 

The decision to conduct surveys with a census of UB grantees, as opposed to a smaller 

statistical sample, is based on the following factors:

 Need for subgroup estimates. One goal of the survey is to help identify an important 

and promising practice to assess rigorously. Key to achieving that goal is determining 

how commonly the practice is already implemented both overall and across various types 

of grantees. In order to tabulate subgroup means and distributions it is necessary to have 

a significant sample of grantees in the survey. 

 Respondent cooperation. Because of concern among UB grantees and stakeholders 

about the use of data from a past UB evaluation in which sampling and weighting were 

used, we believe that greater acceptance of and cooperation with the survey effort will be 

garnered if all UB projects are invited to participate. Because of the interaction among 

UB grantees through their regional and national association activities, we anticipate 

higher response rates if all grantees are required to participate in the survey as part of 

their grant obligations under EDGAR rather than ask a sample of grantees to participate.

 Cost efficiency. Like all surveys, the fixed costs of conducting this survey will be 

substantial. However, the marginal cost of including additional sample members in the 

survey is small. In addition, selecting a sample and using sampling weights that account 

for the sampling design would add substantial cost to the government.
Page 4

In summary, the survey will include a census of Upward Bound projects because of the 

higher anticipated response rate, need for detail and a sample sufficiently large for subgroup 

analysis, and the limited marginal cost to the government of including all projects instead of a 

smaller sample of projects are quite small. In addition, given that it has been more than 15 years 

since the last national survey of Upward Bound projects, we believe that survey including all

Upward Bound projects is warranted.

B.2. Describe the procedures for the collection of information including statistical 

methodology for stratification and sample selection, estimation procedure, degree of 

accuracy needed for the purpose described in the justification, unusual problems 

requiring specialized sampling procedures, and any use of periodic (less frequent 

than annual) data collection cycles to reduce burden.

The survey will include the current universe of Upward Bound grantees so no sampling 

will be employed. No special statistical procedures will be used in the analysis, only tabulations 

of means and distributions overall and for key subgroups of grantees. Because we close to 100 

percent, tests of statistical significance for group differences are not warranted.6 This is a onetime data collection effort, with the most recent similar effort conducted more than 15 years ago.

The proposed plan for deployment of the grantee survey and subsequent analysis is 

summarized in Table 1.

Table 1. Anticipated Schedule of Data-Collection and Reporting Activities

Weeks from Launch Activity

Launch (estimated 

4/15/2013)

Email and hard copy invitations mailed

Weekly from weeks 2-6 Email reminder messages

Weeks 3-6

Begin reminder phone calls to non-responding PDs 

(Estimated 60% of sample)

Weeks 6-7

Send letter plus hard copy survey to non-responders with 

option to complete hard copy, fax return, complete on Web, 

or call into DIR (Estimated 30% of sample)

Weeks 7-8 Troubleshoot final non-respondents with senior staff calls

Weeks 2-8 Prepare data files for analysis

Week 10 Internal memo summarizing initial findings

Week 44 Final data file with documentation

 

6

In the event that less 95 percent of the universe completes the survey we will assume that we have a sample of the 

universe and use appropriate tests of statistical significance (t-tests and chi-square) for comparing results across 

groups.
Page 5

B.3. Describe methods to maximize response rates and to deal with issues of nonresponse. The accuracy and reliability of information collected must be shown to be 

adequate for intended uses. For collections based on sampling, a special 

justification must be provided for any collection that will not yield "reliable" data 

that can be generalized to the universe studied.

The study will aim to achieve a response rate of 95 percent. All data for this study will 

be collected through a web-based survey. Our approach to implementing web-based surveys will 

help to ensure we attain the high response rate desired to minimize non-response bias. The study 

team will employ a number of features in the design and administration of the survey that have 

been demonstrated to be effective in web-based surveys. These features include: 

 Clearly identifying the survey sponsorship and support by key stakeholders. The 

letter requesting completion of the survey will be sent by the Office of Postsecondary

Education and will clearly identify the need and goals of the survey. The study team will 

also seek the endorsement of the Council for Opportunity in Education, whose active 

support of the study could help improve the survey items and be instrumental in helping 

achieve high response rates. 

 Provide for multiple modes for completion. Respondents will be able to completed the 

survey online, hard-copy, and by phone.

 Make use of reminders. The study team will make use of reminder emails, reminder 

phone calls, and mailed reminders.7

B.4. Describe any tests of procedures or methods to be undertaken. Testing is 

encouraged as an effective means of refining collections of information to minimize 

burden and improve utility. Tests must be approved if they call for answers to 

identical questions from 10 or more respondents. A proposed test or set of test may 

be submitted for approval separately or in combination with the main collection of 

information.

During the 60-day comment period, the instrument will be refined through information 

gathering with a set of project directors and input from college access experts serving as a 

Technical Working Group. Once revisions to this initial draft of the instrument are approved and 

question phrasing is finalized, the study contractor will pilot the web-based survey with no more 

than 8 respondents during the OMB review period to ensure that the survey is easy to navigate, 

functions well, all skips operate appropriately, and the resulting data file is accurate. 

 

7 Millar, Morgan M. and Donald Dillman. “Improving Response to Web and Mixed-Mode Surveys.” Public Opinion 

Quarterly. Volume 75, Number 2, Summer 2011; Heerwegh, Dirk. “Effects of Personal Salutations in Email 

Invitations to Participate in a Web Survey. Public Opinion Quarterly. Volume 69, Number 4, Winter 2005.

http://survey.sesrc.wsu.edu/dillman/papers/websurveyppr.pdf, accessed 6 November 2002.
Page 6

B.5. Provide the name and telephone number of individuals consulted on statistical 

aspects of the design and the name of the agency unit, contractor(s), grantee(s), or 

other person(s) who will actually collect and/or analyze the information for the 

agency.

The data-collection analysis plans were developed by Decision Information Resources, 

Inc. (DIR) and Abt Associates. 

The following individuals will be responsible for data collection and analysis:

 Dr. Russell Jackson, Decision Information Resources, Inc., 713-650-1425

 Dr. Sylvia Epps, Decision Information Resources, Inc., 713-650-1425

 Dr. Carol Pistorino, Decision Information Resources, Inc., 713-650-1425

 Dr. Rob Olsen, Abt Associates, 301-634-1716

 Ms. Radha Roy, Abt Associates, 301-347-5722

 Dr. Eleanor Harvill, Abt Associates, 301-347-5638