## A Study of Implementation and Outcomes in Upward Bound and

**Other TRIO Program s**  **Part B**  **: Collection of Information Employing Statistical Methods**  April 10 , 2013

Prepared for Institute of Education Sciences U.S. Department of Education Prepared b y Decision Information Resources, Inc. Abt Associates Contract Number: ED
- IES
- 10
- R
- 0015. Page i 

## Contents **Part B: Collection of Information Using Statistical Methods**  B.1. Respondent Universe and Sampling Methods
................................ ................................ .... 3 B.2. Information Collection Procedures
................................ ................................ ..................... 4 B.3. Methods to Maximize Response Rates
................................ ................................ ............... 5 B.4. Test of Procedures
................................ ................................ ................................ ............... 6 B.5. Individuals Consulted on Statistical Aspects of Design
................................ ..................... 6Page 1

**INTRODUCTION** This clearance request is submitted to the Office of Management and Budget ( OMB ) in support of data collection and anal ys is planned as part of A Study of Implementation and Outcomes of Upward Bound and Other TRIO Programs, sponsored b y the Institute of Edu cation Sciences (IES) at the U.S. Department of Education (ED). This request is a resubmission on behalf of this project, after ED and OMB determined that an earlier plan for the study was not feasible. 1 **Overview of the Upward Bound Program** The Upward Bo und (UB) Program, the oldest and largest of the federal TRIO programs , was initiated under Title IV of the Higher Education Act of 1965 as a pre - college program designed to help economically disadvantaged high school students (low
- income and/or first
gener ation college going) prepare for, enter, and succeed in postsecondary education
. With grant awards approximating $5,000 per student per ye ar, Upward Bound provides an intensive set of services. In the most recent (FY 2012) grant competition, ED sought to make the provision of those services more productive and cost effective by incentivizing project applicants to propose new delivery

methods and approaches that could reduce the cost per student without sacrificing implementation quality . Awards were made to 820 Upward Bound grantees totaling $268.2 million to serve more than 62,500 students
. The structure of Upward Bound projects is well established, defined largel y by specific statutory language prescribing project and student eligibility and the serv ices offered. Students usually enter the program while in the ninth or tenth grade. Although students may participate in Upward Bound through the summer following twelfth grade (for three to four y ears total), participants spend an average of 21 months i n the program. 2 Upward Bound projects are generall y operated b y two - or four
- year colleges. Projects must offer academic instruction as well as college counseling, mentoring, and other support services. In addition to regularl y scheduled meetings througho ut the school y ear, projects also offer an intensive instructional program that meets dail y for about six weeks during the summer. Upward Bound projects may provide additional services or supports to students, beyond the set of required core activities. C urrently , very

little is known about the intensity , duration, and mix of services , or how they are delivered, particularl y for FY 2012 grantees that may have changed implementation strategies under the new grant requirements **Overview of the Study and the Role of the Proposed Data Collection** The Study of Implementation and Outcomes of Upward Bound and Other TRIO Programs was awarded in August 2010 as a first step towards launching a congressionally mandate study . The 2008 re authorization of the Higher Education Act (HEA) required ED to

1 The original OMB submission (1850
-NEW (04471)), for site visits to 20 Upw ard Bound grantees, was submitted on 4/20/11 and later withdrawn. 2 U.S. Department of Education, Office of the Under Secretary, Policy and Program Studies Service, The Impacts of Regular Upward Bound: Results from the Third Follow
- Up Data Collection , Washington, D.C., 2004.Page 2

conduct a stud y that would identify particular institutional, community , and program or project practices that are effective in improving Upward Bound student outcomes ( 20 U.S.C. 1070a
- 1 8 ) (see Appendix A)
. However, the law also prohibit ed an y study of a TRIO program that would require grantees to “ recruit additional students beyond those that the program or project would normally recruit” or that would result “in the denial of services for an eligible student under the program or project.” These prohibitions were initially seen as eliminating the possibility of conducting a randomized control trial within the Upward Bound program. The new contract was intended to explore possible designs for and the feasibility of conducting a study

of the effectiveness of one or more promising practices in Upward Bound. Design work completed highlighted the weaknesses of popular, program
- supported study approaches such as matching grantees on certain project characteristics and then assessing variation in student outcomes that could be attributed to differences in project implementation. 3 In consultation with OMB in summer 2011, ED moved forward to investigate the options for testing out certain “enhanced” p ractices or alternative strategies for implementing core components of Upward Bound using a randomized control trial. The data collection for which we are now seeking OMB approval is part of that effort. 4 The survey

of FY 2012 Upward Bound grantee proj ect directors has two objectives: 1. **To help identify**  **one or more strateg**  **ies to test as part of a random assignment**  **demonstration .**  One challenge in pursuing a rigorous study of promising practices in Upward Bound, given that it is a service
- rich program , is understanding how (not just whether) projects implement core components , where gaps in services might exist, and the prevalence of both the implementation approaches and gaps. In order not to violate the statutory

prohibition against denying regular Upw ard Bound services to student participants, we have to ensure that an y promising strategy we evaluate is not already widely implemented but yet is sufficiently attractive to Upward Bound projects to allow us to recruit grantees not already implementing the strategy to participate in a research demonstration involving random assignment. 2. **To provide new and up**  **- to**  **- date information about**  **the services and strategies that Upward Bound**  **grantee s implement**  **, particularly under the new grant productivity requirements**  **.** The last survey of program implementation was conducted in 1997 5 and the program office has an interest in obtaining information that is more current. The

3 Th is type of quasi
- experimental study wo uld n ot yield reliable or conclusive results because of the inability to fully isolate the effects of any particular implementation strategies. Upward Bound projects differ in the students they serve, the relationship betw een the grantee “host institution” and its participating high schools, the level of external resources, and many other factors in addition to their approaches to implementing core components and services. To measure the effects of implementing a particular (potentially promising) approach to a UB component an evaluation wo uld need to statistically “control for” everything else that might differ among projects and influence outcomes; there would inevitably be unknown or not easily measured factors that could have a pow erful effect on outcomes. 4 IES has also awarded a separate contract with an option to conduct an RCT of a promising practice within Upward Bound, once such a practice or strategy can be identified under the current contract. 5 Moore, Mary T. “The National Evaluation of Upward Bou nd: A 1990's View of Upward Bound Programs Offered, Students Served, and Operational Issues.” Washington, D.C.: U.S. Department of Education, 1997.Page 3

survey will attempt to capture approaches to program services and supports that participants expe rience and which may improve their prospects of successfully competing high school and entering and/or college. The survey is designed to address the following research questions:

 How are the key components and service areas implemented within Upward Boun d ?

 What is the prevalence of specific strategies or approaches for implementing s ervice areas or program components across grantee projects ?

 Where are there gaps in services that might be linked to improving student outcomes?

 What strategies or approaches may be amenable to testing through an experimental design to measure impact on student outcomes? The project director survey will be administered to all 820 regular Upward Bound grantees in the summer of 2013 , so that grantees can report on Upward Bound activities and services during their summer program as well as during the school y ear . Because of the need to collect information on how services are implemented, the instrument is detailed and designed to take approximately

40 minutes to co mplete
. We solicited and have received input from the TRI O community

and other college access experts to improve and revise the instrument during the public comment period. 6 The results of the data collection will be tabulated and released in a summary r eport. **B.1. Describe (including a numerical estimate) the potential respondent universe and any**  **sampling or other respondent selection methods to be used.** All 820 Upward Bound grantees that received grants in 2012 will be included in the survey . The list of funded grantees maintained b y the U.S. Department of Education’s Office of Postsecondary

Education (OPE) will be used to identify the universe of UB projects. The decision to conduct survey s with a census of UB grantees, as opposed to a smal ler statistical sample , is based on the following factors:

 **Need for subgroup estimates .**  One goal of the survey is to help identify

an important and promising practice to assess rigorousl y . Key to achieving that goal is determining how commonly

the practice is already

implemented both overall and across various ty pes of grantees. In order to tabulate subgroup means and distributions it is necessary to have a significant sample of grantees in the survey
.

6 During the 60
- day

public comment period, comments were received from four members of the TRIO community, including the advocacy group representing UB and other TRIO programs. Three experts were solicited. Comments and input did not exceed eight individuals.Page 4

 **Respondent cooperation.** Because of concern among UB grantees and stakeholders about the use of data from a past UB evaluation in which sampling and weighting were used, we believe that greater acceptance of and cooperation with the surv ey effort will be garnered if all UB projects are invited to participate. Because of the interaction among UB grantees through their regional and national association activities, we anticipate higher response rates if all grantees are required to participate in the survey as part of their grant obligations under EDGAR rather than asking a sample of grantees to participate.

 **Cost efficiency.** Li ke all survey s, the fixed costs of conducting this survey will be substantial. However, the marginal cost of inclu ding additional sample members in the survey is small. In addition, selecting a sample and using sampling weights that account for the sampling design would add substantial cost to the government. In summary , the survey

will include a census of Upward Bound projects because of the higher anticipated response rate, need for detail and a sample sufficiently large for subgroup anal ys is, and the limited marginal cost to the government of including all projects instead of a smaller sample of projects are quite small. In addition, given that it has been more than 15 years since the last national survey of Upward Bound projects, we believe that survey including all Upward Bound projects is warranted. **B.2. Describe the procedures for the coll**  **ection of information including s**  **tatistical methodology for stratification and sample selection,**  **e stimation procedure,**  **d egree of**  **accuracy needed for the purpose described in the justification, u**  **nusual problems requiring specialized sampling procedures, and**  **a ny use of periodic**  **(less frequent than annual) data collection cycles to reduce burden.**  The survey

will include the current universe of Upward Bound grantees so no sampling will be employed. No special statistical procedures will be used in the analy sis, only tabulations o f means and distributions overall and for key subgroups of grantees. Because we expect response from close to 100 percent of the grantees , tests of statistical signifi cance for group differences are not warranted. 7 This is a one
- time data collection effort, with the most recent similar effort conducted more than 15 years ago. The proposed plan for deploy ment of the grantee survey and subsequent analy sis is summarized in Table 1. Table 1. Anticipated Schedule of Data
- Collection and Reporting Activities **Weeks from Launch Activity**  Launch (estimated 6 /1/2013) Email and hard copy invitations mailed Weekly from weeks 2
- 6 Email reminder messages

7 In the event that less 95 percent of the universe completes the survey we will assume that we have a sample of the universe and use appropriate tests of statistical significance (t
- tests and chi
- square) for comparing results across groups.Page 5

Weeks 3 - 6 Begin reminder phone calls to non - responding PDs (Estimated 60% of sample) Weeks 6 - 7 Send letter plus hard copy survey to non
- responders with option to complete hard copy, fax return, complete on Web, or call into DIR (Estimated 30% of sample) Week s 7 - 8 Troubleshoot final non - respondents with senior staff calls Weeks 2
- 8 Prepare data files for analysis Week 10 Internal memo summarizing initial findings Week 44 Final data file with documentation **B.3. Describe methods to maximize response rates and to deal with issues**  **of non**   ** response. The accuracy and reliability of information collected must be shown to be** **adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data**  **that can be generalized to the universe studied.**  The study will aim to achieve a response rate of 95 percent with the expectation of a 100% response rate since grantee participation in a national evaluation collection is required under ED’s grant regulations (EDGAR). We expect policy officials at OPE to communicate this requirement to Upward Bound grantees. All data for this study will be collected through a web
based survey . Our approach to implementing web
- based survey s will h elp to ensure we attain the high response rate desired to minimize non
- response bias. The stud y team will employ a number of features in the design and administration of the survey that have been demonstrated to be effective in web
- based survey s. These features include:

 **C learly identifying the survey sponsorship and support by key stakeholders**  **.** The letter requesting completion of the survey will be sent by

the Office of Postsecondary Education and will clearly identify the need and goals of the survey . Th e study team will also seek the endorsement of the Council for Opportunity in Education, whose active support of the study

could help improve the survey items and be instrumental in helping achieve high response rates.

 **P rovid**  **e for multiple modes for comp**  **letion .**  Respondents will be able to completed the survey online , hard
- cop y, and by phone
.
 **Make use of**  **reminder s.**  The study team will make use of weekl y reminder emails, as well as reminder phone calls and mailed reminders as needed
. 8

8 Millar, Morgan M. and Donald Dillman. “Improving Response to Web and Mixed
- Mode Surveys.” Public Opinion Quarterly. Volume 7 5 , Number 2, Summer 20 11; Heerwegh, Dirk. “Effects of Personal Salutations in Email Invitations to Participate in a Web Survey. Public Opinion Quarterly. Volume 69 , Number 4 , Winter 2005. http://survey.sesrc.wsu.edu/dillman/papers/websurveyppr.pdf , accessed 6 November 2002.Page 6

**B.4. Describe any**  **tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize**  **burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or**  **more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of**  **information.** During the 60
- day

comment period, the instrument was refined through comments and input solicited from seven individuals total

—three project directors , the advocacy group representing UB and other TRIO programs, and three college access experts serving as a Technical Working Group
. Their feedback has resulted in a substantial revision of the survey instrum ent that is still consistent with our burden estimates. T he study contractor will pilot the web
- based version of the revised survey with no more than eight respondents during the OMB review period to ensure that the survey is eas y to navigate, functions w ell, all skips operate appropriatel y, and the resulting data file is accurate. Based on the nature of the comments received fr om current project directors and the TRI O community , and the revision of the instrument in response to those comments, i t is our e xpectation that changes to the instrument will be primarily

operational
. For more information on the individuals who provided feedback and the nature of the comments please reference Supporting Statement A, section A.8. **B.5. Provide the name and telep**  **hone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or**  **other person(s) who will actually collect and/or analyze the information for the agency.**  The d ata - collection anal ysis plans were developed by Decision Information Resources, In c. (DIR) and Abt Associates
. **The following individuals will be responsible for data collection and analysis:**

 Dr. Russell Jackson , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Sylvia Epps , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Carol Pistorino , Decision Information Resources, Inc. , 713
- 650
- 1425

 Dr. Rob Olsen , Abt Associates , 301
- 634
- 1716

 Ms. Radha Roy , Abt Associates , 301
- 347
- 5722

 Dr. Eleanor Harvill , Abt Associates , 301
- 347
- 5638