## A Study of Implementation and Outcomes in Upward Bound and

**Other TRIO Program s**  **Part B**  **: Collection of Information Employing Statistical Methods**  December 3, 2012

Prepared for Institute of Education Sciences U.S. Department of Education Prepared b y Decision Information Resources, Inc. Abt Associates Contract Number: ED - IES
- 10
- R - 0015. Page i 

## Contents **Part B: Collection of Information Using Statistical Methods**  B.1. Respondent Universe and Sampling Methods
................................ ................................ .... 3 B.2. Information Collection Procedures
................................ ................................ ..................... 4 B.3. Methods to Maximize Response Rates
................................ ................................ ............... 5 B.4. Test of Procedures
................................ ................................ ................................ ............... 5 B.5. Individuals Consulted on Statistical Aspects of Design
................................ ..................... 6Page 1

**INTRODUCTION** This clearance request is submitted to the Office of Management and Budget ( OMB ) in support of data collection and anal ys is planned as part of A Study of Implementation and Outcomes of Upward Bound and Other TRIO Programs, sponsored b y the Institute of Education Sciences (IES) at the U.S. Department of Education (ED). This request is a resubmission on behalf of this project, after ED and OMB determined that an earlier plan for the study was not feasi ble. 1 **Overview of the Upward Bound Program** The Upward Bound (UB) Program, the oldest and largest of the federal TRIO programs , was initiated under Title IV of the Higher Education Act of 1965 as a pre - college program designed to help economically disadvantaged high school students (low
- income and/or first
generation college going) prepare for, enter, and succeed in postsecondary education
. With grant awards approximating $5,000 per student per ye ar, Upward Bound provides an intensive set of services. In the most recent (FY 2012) grant competition, ED sought to make the provision of those services more productive and cost effective by incentivizing project applicants to propose new delivery

methods and approaches that could reduce the cost per student without sacrificing implementation quality . Awards were made to 820 Upward Bound grantees totaling $268.2 million to serve more than 62,500 students
. The structure of Upward Bound projects is well established, defined largel y by specific statutory languag e prescribing project and student eligibility and the services offered. Students usually enter the program while in the ninth or tenth grade. Although students may participate in Upward Bound through the summer following twelfth grade (for three to four ye ars total), participants spend an average of 21 months in the program. 2 Upward Bound projects are generall y operated b y two - or four
- year colleges. Projects must offer academic instruction as well as college counseling, mentoring, and other support serv ices. In addition to regularl y scheduled meetings throughout the school ye ar, projects also offer an intensive instructional program that meets dail y for about six weeks during the summer. Upward Bound projects may provide additional services or supports t o students, beyond the set of required core activities. Currently , very

little is known about the intensity , duration, and mix of services , or how they are delivered, particularl y for FY 2012 grantees that may have changed implementation strategies under the new grant requirements **Overview of the Study and the Role of the Proposed Data Collection** The Study of Implementation and Outcomes of Upward Bound and Other TRIO Programs was awarded in August 2010 as a first step towards launching a congressionally mandate study . The 2008 reauthorization of the Higher Education Act (HEA) required ED to

1 The original OMB submission (1850
-NEW (04471)), for site visits to 20 Upw ard Bound grantees, was submitted on 4/20/11 and later withdrawn. 2 U.S. Department of Education, Office of the Under Secretary, Policy and P rogram Studies Service, The Impacts of Regular Upward Bound: Results from the Third Follow
- Up Data Collection , Washington, D.C., 2004.Page 2

conduct a stud y that would identify particular institutional, community , and program o r project practices that are effective in improving Upward Bound student outcomes ( 20 U.S.C. 1070a
- 1 8 ) (see Appendix A)
. However, the law also prohibit ed an y study of a TRIO program that would require grantees to “ recruit additional students beyond those that the program or project would normally recruit” or that would result “in the denial of services for an eligible student under the program or project.” These prohibitions were initially seen as eliminating the possibility of conducting a randomized control trial within the Upward Bound program. The new contract was intended to explore possible designs for and the feasibility of conducting a study

of the effectiveness of one or more promising practices in Upward Bound. Design work completed highlighted the weaknesses of popular, program
- supported study approaches such as matching grantees on certain project characteristics and then assessing variation in student outcomes that could be attributed to differences in project implementation. 3 In consultatio n with OMB in summer 2011, ED moved forward to investigate the options for testing out certain “enhanced” practices or alternative strategies for implementing core components of Upward Bound using a randomized control trial. The data collection for whic h we are now seeking OMB approval is part of that effort. 4 The survey

of FY 2012 Upward Bound grantee project directors has two objectives: 1. **To help identify**  **one or more strateg**  **ies to test as part of a random assignment**  **demonstration .**  One challenge in pursuing a rigorous study of promising practices in Upward Bound, given that it is a service
- rich program , is understanding how (not just whether) projects implement core components , where gaps in services might exist, and the prevalence of both the implem entation approaches and gaps. In order not to violate the statutory

prohibition against denying regular Upward Bound services to student participants, we have to ensure that an y promising strategy we evaluate is not already widely implemented but yet is su fficiently attractive to Upward projects to allow us to recruit grantees not already implementing the strategy to participate in a research demonstration involving random assignment. 2. **To provide new and up -**  **to -**  **date information about the services and strateg ies that**  **Upward Bound grantees implement, particularly under the new grant productivity requirements.**  The last survey of program implementation was conducted in 1997 5 and the program office has an interest in obtaining information that is more current. The

3 Th is type of quasi
- experimental study wo uld not yield reliable or conclusive results because of the inability to fully isolate the effects of any particular implementation strategies. Upward Bound projects differ in the students they serve, the relationship between the grantee “host institution” and its participating high schools, the level of external resources, and many other factors in addition to their approaches to implementing core components and services. To measure the effects of implementing a particular (potentially promising) approach to a UB component an evaluation wo uld need to statistically “control for” eve ry thing else that might differ among projects and influence outcomes; there would inevitably be unknown or not easily measured factors that could have a pow erful effect on outcomes. 4 IES has also awarded a separate contract with an option to conduct an RC T of a promising practice within Upward Bound, once such a practice or strategy can be identified under the current contract. 5 Moore, Mary T. “The National Evaluation of Upward Bound: A 1990's View of Upward Bound Programs Offered, Students Served, and O perational Issues.” Washington, D.C.: U.S. Department of Education, 1997.Page 3

survey will attempt to capture approaches to program services and supports that participants experience and which may improve their prospects of successfully competing high school and entering and/or college. The survey is designed to address the foll owing research questions:  How are the key components and service areas implemented within Upward Bound ?  What is the prevalence of specific strategies or approaches for implementing s ervice areas or program components across grantee projects ?  Where are the re gaps in services that might be linked to improving student outcomes?  What strategies or approaches may be amenable to testing through an experimental design to measure impact on student outcomes? The project director survey will be administered to all 820 regular Upward Bound grantees in the spring of 2013. Because of the need to collect information on how services are implemented, the instrument is detailed and designed to take approximately 40 minutes to complete. We are soliciting input from th e TRIO community and other college access experts to improve the instrument during the public comment period. The results of the data collection will be tabulated and released in a summary report. **B.1. Describe (including a numerical estimate) the poten**  **tial respondent universe and any sampling or other respondent selection methods to be used.**  All 820 Upward Bound grantees that received grants in 2012 will be included in the survey . The list of funded grantees maintained b y the U.S. Department of Education’s Office of Postsecondary

Education (OPE) will be used to identify the universe of UB projects. The decision to conduct survey s with a census of UB grantees, as opposed to a smalle r statistical sample , is based on the following factors:

 **Need for subgroup estimates .**  One goal of the survey is to help identify

an important and promising practice to assess rigorousl y . Key to achieving that goal is determining how commonly

the practic e is alread y implemented both overall and across various types of grantees. In order to tabulate subgroup means and distributions it is necessary to have a significant sample of grantees in the survey
.
 **Respondent cooperation.** Because of concern among UB grantees and stakeholders about the use of data from a past UB evaluation in which sampling and weighting were used, we believe that greater acceptance of and cooperation with the survey effort will be garnered if all UB projects are invited to particip ate. Because of the interaction among UB grantees through their regional and national association activities, we anticipate higher response rates if all grantees are required to participate in the survey as part of their grant obligations under EDGAR rathe r than ask a sample of grantees to participate.  **Cost efficiency.** Li ke all survey s, the fixed costs of conducting this survey will be substantial. However, the marginal cost of including additional sample members in the survey is small. In addition, selec ting a sample and using sampling weights that account for the sampling design would add substantial cost to the government.Page 4

In summary , the survey

will include a census of Upward Bound projects because of the higher anticipated response rate, need for det ail and a sample sufficiently large for subgroup anal ys is, and the limited marginal cost to the government of including all projects instead of a smaller sample of projects are quite small. In addition, given that it has been more than 15 years since the l ast national survey of Upward Bound projects, we believe that survey including all Upward Bound projects is warranted. **B.2. Describe the procedures for the coll**  **ection of information including s tatistical**  **methodology for stratification and sample selection,**  **e stimation procedure,**  **d egree of**  **accuracy needed for the purpose described in the justification, u**  **nusual problems requiring specialized sampling procedures, and**  **a ny use of periodic (less frequent**  **than annual) data collection cycles to reduce burd en.**  The survey

will include the current universe of Upward Bound grantees so no sampling will be employed. No special statistical procedures will be used in the analy sis, only tabulations of means and distributions overall and for key subgroups of grante es. Because we close to 100 percent, tests of statistical signifi cance for group differences are not warranted. 6 This is a one
time data collection effort, with the most recent similar effort conducted more than 15 ye ars ago. The proposed plan for deplo ym ent of the grantee survey and subsequent analy sis is summarized in Table 1. Table 1. Anticipated Schedule of Data
- Collection and Reporting Activities **Weeks from Launch Activity**  Launch (estimated 4/15/2013) Email and hard copy invitations mailed Weekly from weeks 2
- 6 Email reminder messages Weeks 3 - 6 Begin reminder phone calls to non
- responding PDs (Estimated 60% of sample) Weeks 6 - 7 Send letter plus hard copy survey to non
- responders with option to complete hard copy, fax return, complete on Web, or call into DIR (Estimated 30% of sample) Week s 7 - 8 Troubleshoot final non
- respondents with senior staff calls Weeks 2
- 8 Prepare data files for analysis Week 10 Internal memo summarizing initial findings Week 44 Final data file with documentation

6 In the event that less 95 percent of the universe completes the survey we will assume that we have a sample of the universe and use appropriate tests of statistical significance ( t - tests and chi
- square) for comparing results across groups.Page 5

**B.3. Describe methods to maximize response rates and to deal with issues of non**  **** **response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special**  **justification mu st be provided for any collection that will not yield "reliable" data**  **that can be generalized to the universe studied.** The study will aim to achieve a response rate of 95 percent. All data for this study will be collected through a web
- based survey . Our approach to implementing web
- based survey s will help to ensure we attain the high response rate desired to minimize non
- response bias. The stud y team will employ

a number of features in the design and administration of the survey that have been demonstrate d to be effective in web
- based survey s. These features include:

 **C learly identifying the survey sponsorship and support by key stakeholders**  **.** The letter requesting completion of the survey will be sent by

the Office of Postsecondary Education and will clearly identify the need and goals of the survey . The study team will also seek the endorsement of the Council for Opportunity in Education, whose active support of the study

could help improve the survey items and be instrumental in helping achieve high resp onse rates.

 **P rovid**  **e for multiple modes for completion**  **.** Respondents will be able to completed the survey online , hard
- cop y, and by phone
.
 **Make use of reminder**  **s.** The study team will make use of reminder emails, reminder phone calls, and mailed reminders. 7 **B.4. Describe any tests of procedures or methods to be undertaken. Testing is**  **encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to**  **identica l questions from 10 or more respondents. A proposed test or set of test may**  **be submitted for approval separately or in combination with the main collection of information.**  During the 60
- day

comment period, the instrument will be refined through informati on gathering with a set of project directors and input from college access experts serving as a Technical Working Group
. Once revisions to this initial draft of the instrument are approved and question phrasing is finalized, the study contractor will pilo t the web
- based survey with no more than 8 respondents during the OMB review period to ensure that the survey is easy to navigate, functions well, all skips operate appropriately , and the resulting data file is accurate.

7 Millar, Morgan M. and Donald Dillman. “Improving Response to Web and Mixed
- Mode Surveys.” Public Opinion Quarterly. Volume 7 5 , Number 2, Summer 20 11; Heerwegh, Dirk. “Effects of Personal Salutations in Email Invitations to Participate in a Web Survey. Public Opinion Quarterly. Volume 69 , Number 4 , Winter 2005. http://survey.sesrc.wsu.edu/dillman/papers/websurveyppr .pdf , accessed 6 November 2002.Page 6

**B.5. Provide the name and teleph**  **one number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or**  **other person(s) who will actually collect and/or analyze the information for the agency.**  The d ata - collection anal ysis p lans were developed by Decision Information Resources, In c. (DIR) and Abt Associates
. **The following individuals will be responsible for data collection and analysis:**

 Dr. Russell Jackson , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Sylvia Epps , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Carol Pistorino , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Rob Olsen , Abt Associates , 301
- 634
- 1716

 Ms. Radha Roy , Abt Associates , 301
- 347
- 5722

 Dr. Eleanor Harvill , Abt Associates , 301
- 347
- 5638