U.S. DEPARTMENT OF EDUCATION

National Evaluation of 

Student Support Services:

Examination of Student Outcomes

After Six Years

Final Report
National Evaluation of

Student Support Services: 

Examination of Student Outcomes After Six Years

Final Report

By

Bradford W. Chaney

Westat

Rockville, Md.

Prepared for:

U.S. Department of Education

Office of Planning, Evaluation and Policy Development

Policy and Program Studies Service

2010
This report was prepared for the U.S. Department of Education under Contract No. ED-03-PO2561. The project monitor was Michael Fong in the Policy and Program Studies Service. The 

views expressed herein are those of the contractor. No official endorsement by the U.S. 

Department of Education is intended or should be inferred.

U.S. Department of Education

Arne Duncan

Secretary

Office of Planning, Evaluation and Policy Development

Carmel Martin

Assistant Secretary

Policy and Program Studies Service

Alan L. Ginsburg

Director

April 2010

This report is in the public domain. Authorization to produce it in whole or in part is granted. 

Although permission to reprint this publication is not necessary, the citation should be: U.S. 

Department of Education, Office of Planning, Evaluation and Policy Development, Policy and 

Program Studies Service, National Evaluation of Student Support Services: Examination of 

Student Outcomes After Six Years, Washington, D.C., 2010.

This report is available on the Department’s Web site at 

http://www.ed.gov/about/offices/list/opepd/ppss/reports.html.

On request, this publication is available in alternative formats, such as Braille, large print, or 

computer diskette. For more information, please contact the Department’s Alternate Format 

Center at 202-260-0852 or 202-260-0818.
iii

CONTENTS

LIST OF EXHIBITS ........................................................................................................................ v

ACKNOWLEDGMENTS ................................................................................................................ ix

EXECUTIVE SUMMARY .............................................................................................................. xi

1. INTRODUCTION ....................................................................................................................... 1

Organization of This Report ..................................................................................................... 2

Description of SSS.................................................................................................................... 3

The SSS Services...................................................................................................................... 5

Findings From Earlier Years of This Study .............................................................................. 7

2. STUDY DESIGN.......................................................................................................................... 9

Outcome Measures Used .......................................................................................................... 9

Overview of the Study Design .................................................................................................. 14

The Longitudinal Study ............................................................................................................ 15

3. RESEARCH FINDINGS BASED ON DICHOTOMOUS MEASURES OF 

PARTICIPATION....................................................................................................................... 31

Highlights of Findings .............................................................................................................. 31

Introduction............................................................................................................................... 32

Multivariate and Logistic Regressions...................................................................................... 33

Hierarchical Linear Modeling................................................................................................... 36

Summary ................................................................................................................................... 38

4. RESEARCH FINDINGS BASED ON CONTINUOUS MEASURES OF 

PARTICIPATION....................................................................................................................... 41

Highlights of Findings .............................................................................................................. 41

Introduction............................................................................................................................... 42

Multivariate and Logistic Regressions...................................................................................... 44

Without propensity scores ................................................................................................ 45

With propensity scores...................................................................................................... 48

Hierarchical Linear Modeling................................................................................................... 51

Without propensity scores ................................................................................................ 51

With propensity scores...................................................................................................... 54

Types of Services...................................................................................................................... 57

Summary ................................................................................................................................... 61

5. SUMMARY AND CONCLUSIONS .......................................................................................... 63

Overview................................................................................................................................... 63

Choosing Among the Six Models............................................................................................. 67

Policy Findings Concerning SSS .............................................................................................. 70

REFERENCES.................................................................................................................................. 73
iv

APPENDIXES

A DETAILED TABLES............................................................................................................. 75

B METHODOLOGY.................................................................................................................. 113
v

EXHIBITS

Exhibit

E-1 Estimated improvement in academic outcomes associated with receiving first-year SSS 

services and with receiving supplemental services from any source and in any of the six 

academic years, using six analytic approaches: Six-year outcomes after freshman entry 

in 1991–92 ............................................................................................................................. xvi

1-1 List of SSS services and the frequency that they were received in 1991–92 ........................ 6

2-1 Outcome measures for the sixth-year evaluation after freshman entry in 1991–92 .............. 9

2-2 Percentage distribution of all SSS participants, by mean GPA levels six years after 

freshman entry in 1991–92 .................................................................................................... 9

2-3 Percentage of SSS participants who earned various amounts of credits over six years

after freshman entry in 1991–92............................................................................................ 10

2-4 Retention and attainment of degrees by SSS participants six years after freshman entry 

in 1991–92 ............................................................................................................................. 12

2-5 Comparison of SSS students with comparison group and all freshmen, by student 

characteristics......................................................................................................................... 18

2-6 Correlation of the number of hours received of each SSS service as freshmen in 1991–

92 with the number of hours received each other SSS service in the same year................... 22

2-7 Percent of SSS and non-SSS students receiving services in the six-year period starting 

with freshman entry in 1991–92 ............................................................................................ 26

2-8 Number of students receiving SSS services in 1991–92 in database used for outcomes 

analysis .................................................................................................................................. 28

3-1 Summary of findings for each outcome using dichotomous measures of participation, 

by method used: Six-year outcomes after freshman entry in 1991–92................................. 31

3-2 Regression coefficients concerning effects of SSS and non-SSS supplemental services, 

by outcome measure: Six-year outcomes after freshman entry in 1991–92......................... 34

3-3 Estimated effects of SSS and non-SSS supplemental services on SSS students, by 

outcome measure: Six-year outcomes after freshman entry in 1991–92.............................. 35

3-4 HLM regression coefficients concerning effects of SSS and non-SSS supplemental 

services, by outcome six years after freshman entry in 1991–92 .......................................... 36
vi

3-5 Estimated effects of SSS and non-SSS supplemental services on SSS students, by 

outcome six years after freshman entry in 1991–92 (HLM analysis).................................... 37

4-1 Summary of findings for each outcome using continuous measures of participation, by 

method used: Six-year outcomes after freshman entry in 1991–92...................................... 41

4-2 Regression coefficients concerning effects of SSS services using continuous measures 

of SSS participation and no measures of propensity to receive services, by outcome six 

years after freshman entry in 1991–92 .................................................................................. 46

4-3 Estimated effects of SSS and non-SSS supplemental services on SSS students using 

continuous measures of SSS participation and no measures of propensity to receive 

services, by outcome six years after freshman entry in 1991–92 .......................................... 47

4-4 Regression coefficients concerning the effects of SSS services using continuous 

measures of SSS participation and adding propensity scores, by outcome six years after 

freshman entry in 1991–92 .................................................................................................... 49

4-5 Estimated effects of SSS and non-SSS supplemental services on SSS students using 

continuous measures of SSS participation and adding propensity scores, by outcome six 

years after freshman entry in 1991–92 .................................................................................. 50

4-6 HLM regression coefficients concerning the effects of SSS services using continuous 

measures of SSS participation and no measures of propensity, by outcome six years

after freshman entry in 1991–92............................................................................................ 52

4-7 Estimated effects of SSS and non-SSS supplemental services on SSS students when 

using continuous measures and no propensity scores, by outcome six years after 

freshman entry in 1991–92 (HLM analysis).......................................................................... 53

4-8 HLM regression coefficients concerning the effects of SSS services using continuous 

measures of SSS participation with propensity scores, by outcome six years after 

freshman entry in 1991–92 .................................................................................................... 55

4-9 Estimated effects of SSS and non-SSS supplemental services on SSS students when 

using continuous measures and propensity scores, by outcome six years after freshman 

entry in 1991–92 (HLM analysis).......................................................................................... 56

4-10 First-year SSS services that showed positive and statistically significant effects on 

student outcomes, by outcome six years after freshman entry in 1991–92 ........................... 59

4-11 Other supplemental services that showed positive and statistically significant effects on 

student outcomes, by outcome six years after freshman entry in 1991–92 ........................... 60

5-1 Comparison of estimated effects of first-year SSS services using each of six 

methodologies: Six-year outcomes after freshman entry in 1991–92 .................................. 64

5-2 Comparison of estimated effects of supplemental services using each of six 

methodologies: Six-year outcomes after freshman entry in 1991–92 .................................. 66
vii

5-3 First-year SSS services that showed positive and statistically significant effects on 

student outcomes when only statistically significant propensity scores are retained, by 

outcomes six years after freshman entry in 1991–92 ............................................................ 70

A-1 HLM analysis to predict cumulative GPAs six years after freshman entry in 1991–92

using dichotomous measures of participation........................................................................ 77

A-2 HLM analysis to predict the number of credits earned six years after freshman entry in 

1991–92 using dichotomous measures of participation......................................................... 78

A-3 HLM analysis to predict retention or baccalaureate degree completion at the same 

institution six years after freshman entry in 1991–92 using dichotomous measures of 

participation ........................................................................................................................... 79

A-4 HLM analysis to predict retention or baccalaureate degree completion at any institution 

six years after freshman entry in 1991–92 using dichotomous measures of participation .... 80

A-5 HLM analysis to predict baccalaureate degree completion (or higher) at any institution 

six years after freshman entry in 1991–92 using dichotomous measures of 

participations.......................................................................................................................... 81

A-6 HLM analysis to predict associate’s degree completion (or higher) at any institution six 

years after freshman entry in 1991–92 using dichotomous measures of participation.......... 82

A-7 HLM analysis to predict transfers from two-year to four-year institutions using 

dichotomous measures six years after freshman entry in 1991–92of participation............... 83

A-8 HLM analysis to predict cumulative GPAs six years after freshman entry in 1991–92

using continuous measures of participation and no measures of propensity to receive 

services .................................................................................................................................. 84

A-9 HLM analysis to predict the number of credits earned six years after freshman entry in 

1991–92 using continuous measures of participation and no measures of propensity to 

receive services...................................................................................................................... 86

A-10 HLM analysis to predict retention or baccalaureate degree completion at the same 

institution six years after freshman entry in 1991–92 using continuous measures of 

participation and no measures of propensity to receive services........................................... 88

A-11 HLM analysis to predict retention or baccalaureate degree completion at any institution 

six years after freshman entry in 1991–92 using continuous measures of participation 

and no measures of propensity to receive services................................................................ 90

A-12 HLM analysis to predict baccalaureate degree completion (or higher) at any institution 

six years after freshman entry in 1991–92 using continuous measures of participation 

and no measures of propensity to receive services................................................................ 92

A-13 HLM analysis to predict associate’s degree completion (or higher) at any institution six 

years after freshman entry in 1991–92 using continuous measures of participation and 

no measures of propensity to receive services....................................................................... 94
viii

A-14 HLM analysis to predict transfers from two-year to four-year institutions six years after 

freshman entry in 1991–92 using continuous measures of participation and no measures 

of propensity to receive services............................................................................................ 96

A-15 HLM analysis to predict cumulative GPAs six years after freshman entry in 1991–92

using continuous measures of participation and adding propensity scores........................... 98

A-16 HLM analysis to predict the number of credits earned six years after freshman entry in 

1991–92 using continuous measures of participation and adding propensity scores............ 100

A-17 HLM analysis to predict retention or baccalaureate degree completion at the same 

institution six years after freshman entry in 1991–92 using continuous measures of 

participation and adding propensity scores............................................................................ 102

A-18 HLM analysis to predict retention or baccalaureate degree completion at any institution 

six years after freshman entry in 1991–92 using continuous measures of participation 

and adding propensity scores................................................................................................. 104

A-19 HLM analysis to predict baccalaureate degree completion (or higher) at any institution 

six years after freshman entry in 1991–92 using continuous measures of participation 

and adding propensity scores.................................................................................................106

A-20 HLM analysis to predict associate’s degree completion (or higher) at any institution six 

years after freshman entry in 1991–92 using continuous measures of participation and

adding propensity scores........................................................................................................ 108

A-21 HLM analysis to predict transfers from two-year to four-year institutions six years after 

freshman entry in 1991–92 using continuous measures of participation and adding 

propensity scores ................................................................................................................... 110

B-1 Percent of students in SSS programs in 1991–92 that provided each of nine SSS 

services .................................................................................................................................. 122
ix

ACKNOWLEDGMENTS

As a large multiyear effort, this study depended on the contributions of many people. The 

National Evaluation of Student Support Services was conducted under the direction of the Policy and 

Program Studies Service of the U.S. Department of Education. David Goodwin provided technical 

oversight as for the initial phases of the study, and Michael Fong provided technical oversight in the final 

phase of the study. The study was performed by Westat, under contract to the U.S. Department of 

Education. Margaret Cahalan directed the initial study design, data collection, and analysis at Westat, and 

Mary Jo Nolin directed the final phase of the study. A large team of individuals provided statistical 

support, computer programming, data collection and processing, and site visits. Of course, the study 

could not have been conducted without the assistance of the participating higher education institutions, 

which collected and provided much of the data used in this report, and the students in the study, who 

completed multiple questionnaires describing their educational experiences over the course of the study.
x
xi

Executive Summary

This is the final report of the National Evaluation of Student Support Services (SSS). SSS is 

one of eight federally funded grant programs that are administered as part of the Federal TRIO Programs 

within the U.S. Department of Education (ED). The SSS program, in particular, focuses on students 

while they are enrolled in college. In general, SSS provides the most services to first-year college 

students, though it does also provide services in later years. 

The purpose of the study was to estimate the effects of SSS on the outcomes of the student 

participants. The full report discusses five academic outcomes. For brevity, this summary focuses only 

on the key outcomes: retention in college, transfers from two-year to four-year institutions, and degree 

completion.

Overview of SSS

The purposes of the SSS program, as stated in the Higher Education Act of 1965, as 

reauthorized by the Higher Education Opportunity Act of 2008, are to (1) increase college retention and 

graduation rates for eligible students, (2) increase the transfer rates of eligible students from two-year to 

four-year institutions, and (3) foster an institutional climate supportive of success for low-income and 

first-generation college students and individuals with disabilities. Two-thirds of the students served by an 

SSS project must be low-income (defined as at or below 150 percent of the poverty level) and firstgeneration college students or students with disabilities. The other third must be low-income or firstgeneration college students. One-third of the disabled students also must be low income. 

SSS projects have great latitude to custom-design their services to fit the particular needs of 

their student population. All SSS projects provide academic advising as one of their services (although 

there is great variation in the amount students received), but the projects differ greatly with respect to 

offering other services to SSS students such as tutoring, labs, workshops, special instructional courses, 

and services specifically for students with disabilities. As a rule, SSS students are in full control of 

determining both the types and the amounts of services they receive, as long as the services are offered by 

the institutions and the students qualify for them (e.g., a student must have a disability in order to receive 

special services for individuals with disabilities). 

Main Findings

The study used a variety of statistical methodologies to estimate the effects of participating 

in the SSS program as a college freshman (the only year for which it was possible to collect program 

participation data) and also the more general effect of receiving supplemental services during college. 

The major finding is that analytic models that account for differences in service levels 

generally showed positive and statistically significant effects. Participation in SSS projects as measured 

by the amount of services received during the freshman year is associated with moderate increases on the 

key measures of college retention and degree completion but neither increases nor decreases student 

transfers from two-year to four-year institutions and neither increases nor decreases the outcomes on 

some of the key measures in the HLM models. Although these models controlled for student 

demographics and, whenever possible, prior achievement, one limitation of this model is the potential 

selection bias of participants who received more services. 

Models that measure supplemental services regardless of whether they were offered by the 

SSS project or were offered by some other service provider on campus are associated with positive and 

statistically significant effects on all outcome measures of retention, transfers from two-year to four-year 

institutions, and degree completion. 
xii

In addition this report includes analyses that simply consider whether or not the student was 

classified as being in SSS as a college freshman, although this comparison was considered of limited 

validity given the structure of SSS grants. A major limitation of this analysis is that it does not account 

for the level of service received by SSS participants; nor does it account for the fact that comparison 

students may have received similar services that were not funded by the federal SSS program. This 

measure did not show any effect from participating in SSS as a college freshman. 

Study Design

This report focuses on a statistical comparison of SSS and non-SSS students in order to 

estimate the effects of SSS. The study was designed as a longitudinal study, in which the academic 

progress of 5,800 freshman students in 1991–92 was tracked for six years. Half of the students were 

participants in SSS, and the other half were statistically chosen using propensity models to have similar 

characteristics to the SSS participants. Both quantitative and qualitative data were collected. It should be 

noted that this study utilizes data from the 1990s and may not capture the full nature of the SSS program 

as it operates today.

The sample was chosen in several steps. First, a nationally representative, stratified random 

sample of 200 SSS projects was surveyed in 1991–92. Second, 30 of these projects were randomly 

subsampled for site visits and other data collection activities, and 20 higher education institutions without 

SSS grants were selected to match the SSS sites, based on institutional characteristics. Three of the 

institutions later dropped out of the study. The 47 remaining institutions provided basic information on 

student characteristics, and the SSS projects also provided participant service records on each service 

contact through SSS for the sampled students, including the types of service, length in minutes, date, and 

number of students receiving the specific services. Student surveys were conducted at three different time 

points: 1991–92, 1994–95, and 1997–98. Student transcripts were collected from all institutions the 

students attended (not just the 47 original institutions) at the end of the first, third, and sixth years. 

Methodology

This study had a quasi-experimental design. Regression models and propensity scores were 

used to select a group of comparison students that matched the SSS students as closely as possible on a 

variety of student characteristics, and both the SSS and comparison students were monitored through 

student surveys to determine what other supplemental services they received. Although the comparison 

students were highly disadvantaged, the SSS participants were even more so, and additional statistical 

adjustments were required in the analysis to allow appropriate comparisons between the two groups. 

Exposure to SSS was not randomized because of the way in which SSS is structured. By design, SSS 

students may receive supplemental services outside of SSS, and any attempt to deny services to needy 

students would raise ethical questions, threaten people’s support of the institution, and potentially conflict 

with the institution’s mission. 

The lack of a uniform SSS experience, with considerable variation even within each 

institution, combined with the receipt of equivalent services outside of SSS, made it difficult to design 

statistical models that properly described students’ experience. Rather than choosing a single 

methodology, therefore, this study used multiple approaches. All models included separate measures of 

SSS services for first-year students, supplemental services received outside of SSS or received after the 

first year, and measures of student and school characteristics. The models differed in the statistical 

techniques that were used, in the ways that SSS and other supplemental services were measured, and in 

the use of propensity scores.

 Some models used standard multivariate regression analysis while others use 

hierarchical linear modeling (HLM). 
xiii

 Some models treated SSS participation as dichotomous (i.e., either a student was in 

SSS or he or she was not), while other models treated SSS participation as a collection 

of nine separate services, with each service measured separately in terms of the 

number of hours of participation by each student. 

 Some models adjusted for differences among students solely through individual 

measures of student characteristics (such as academic strength, background, and 

attitudes), while other models also included propensity measures that are designed to 

estimate students’ probability of receiving services.

This report includes all of the above approaches so that readers can see the implications of 

the various methodological choices. In principle, HLM provides better estimates than regressions 

because it is better able to handle the clustering of students within institutions. However, HLM did not 

always converge to a solution when using logistic models (for limited dependent variables), thus requiring 

the use of more standard regression models within HLM. Dichotomous measures of services avoid the 

difficulty that students’ levels of participation may be related to other factors (such as neediness or 

motivation) that might have confounding relationships with student outcomes. However, due to the high 

correlation between SSS participation and the receipt of other supplemental services (91 percent of the 

SSS students received services outside of SSS), dichotomous measures primarily are useful for examining 

supplemental services in general. Only the models with continuous measures appear able to discriminate 

between SSS and other services. Propensity scores usually serve as a substitute for individually fitting 

each regression equation based on student characteristics, and they should be less necessary in these 

models that were individually fitted. In practice, the propensity scores often were statistically 

insignificant, and estimated SSS effects were increased when the statistically insignificant propensity 

scores were dropped. Thus, it is questionable whether they provided useful information, and they may 

sometimes have had confounding effects because of their interrelation with the measures of SSS services.

The regression estimates are presented here in two different formats: as regression 

coefficients to aid researchers to understand the structure of the models and to replicate the results, and as 

impact estimates to aid policymakers in applying the findings, because the regression estimates often do 

not translate easily into effects. (One difficulty is that for some models, one must know the amount of 

services that students received in order to interpret the regression coefficients; another is that when 

logistic regression is used, neither the regression coefficients nor the log odds ratios are easy to interpret.)

Normally, estimates of program effects would be presented by indicating both what students 

would have achieved if they did not participate in SSS and what they did achieve after participating in 

SSS. Unfortunately, because the comparison group was more advantaged than the SSS participants, and 

because students in the comparison group often also received supplemental services, there is no pure 

measure of what outcomes would be expected if students had not participated in SSS. There are actual 

measures of final student outcomes after six years, but the alternative outcomes if students had not 

participated in SSS could only be estimated through statistical models. Moreover, because this report 

uses multiple models as a tool for examining the implications of the methodological choices involved, 

there are multiple estimates of the effects of SSS from which to choose. The results across the various 

models are often highly consistent but not identical. Therefore this report presents only the estimated 

effects of the services along with the final measured outcomes. Readers can calculate what students 

would have achieved without SSS by first deciding which model appears most trustworthy, and then 

subtracting the estimated effects from the final outcomes. 

This study was longitudinal in the sense that the selected students were tracked for six years 

with multiple data collections in order to determine their experiences during those six years and their 

academic outcomes. However, the data analysis has been largely cross-sectional, with the multiple data 

points all included in a single regression or HLM model rather seeking to model changes in the students 

from one time to another. An exception is that in the previously published analysis of student outcomes 
xiv

after three years, structural equations models were used to model the interactions between SSS 

participation, first-year college GPAs, and college retention. That is, one might anticipate that students’ 

academic successes in the first year, as measured by their GPAs, would influence their retention in 

college, but their academic successes could be influenced both by their own characteristics and by 

assistance they received from SSS or other sources. Thus, if first-year college GPAs are used to predict 

retention, a regression equation would tend to underestimate the effect of SSS and other supplemental 

services by only estimating that portion of the effect that was independent of students’ GPAs, while 

ignoring the indirect contribution of supplemental services through their effects on GPAs. For that 

reason, structural equations models were used to separate students’ GPAs into a latent GPA that would be 

expected without any receipt of supplemental services and the additional increment added through 

supplemental services; the latent GPA, rather than the actual GPA, was then used to predict retention so 

the model could estimate the full effects of SSS. 

The structural equations model confirmed that SSS did indirectly influence retention through 

its effect on GPAs, and that the effect of SSS on retention was underestimated unless this relationship was 

properly modeled. This analysis helps to demonstrate the importance of structural equations and path 

analytic models in longitudinal studies. However, the same analysis also revealed that when a second 

measure of students’ academic strengths (high school GPAs) was substituted for college GPAs (because 

high school GPAs would not have been influenced by SSS), essentially the same estimates of the effect of 

SSS were produced. If the purpose of this study was to develop a full theoretical model of what happens 

to students while in college, then it would be important to model the different paths through which SSS 

influences students. However, because the study goal was simply to estimate the total effects of SSS, and 

not to estimate how that effect was distributed among direct and indirect paths, this report uses surrogate 

measures such as high school GPAs that are independent of SSS rather than a structural equations 

approach. This choice avoided some technical complexities (such as imputation of missing data and the 

use of logistic regression within structural equations models) that would have been introduced with a 

structural equations approach. 

Findings From Earlier Years of This Study

This report focuses on student outcomes six years after freshmen began participating in SSS. 

As part of the same longitudinal study, earlier reports discussed the effects of SSS on student outcomes 

after one year and after three years. The outcomes that were examined earlier were largely the same 

outcomes as those discussed here (effects on grade point averages, credits earned, and retention, though 

over shorter time frames), except that the one- or three-year time periods were too short to examine the 

effects on degree completion and on transfers from two-year to four-year institutions.

SSS showed a small but positive and statistically significant effect on students’ GPAs, 

number of semester credits earned, and retention. The greatest effect generally occurred during the first

year, when the most SSS services were received, but some SSS services received in the first year showed 

persisting effects in later years, and some services received in later years (not necessarily through SSS) 

also showed positive and statistically significant effects. The size of the effect depended on the degree to 

which students participated in SSS, with greater levels of participation resulting in a greater effect. The 

estimated effects of SSS also varied based on which particular services each student received and the 

structure of the SSS projects.

The average effect was small because most students received only a modest amount of 

services. Nine percent of students had only one service in their freshman year. The mean number of 

hours of services received in the first year was 32, and the median was 14. The mean for nonfreshmen 

was 15 hours, and the median was six. SSS projects appeared successful in targeting those students who 

were most disadvantaged from among the overall student population. In comparison with the national 

averages for college freshmen, SSS participants tended to be older, to be members of a minority group, to 

have had lower prior academic achievement, and to have dependent children. For additional highlights 

from the previous study, see page 7 in the Final Report.
xv

Limitations

Except for services received during the first year, this study was not able to distinguish 

between services received through SSS and services received through other means. The reason is that to 

limit the burden on participating institutions and to obtain their cooperation, the data on later-year 

services were collected only through student questionnaires, and students were not expected to know the 

funding source of the services that they received (especially since SSS programs often were renamed). 

The third-year report found that effects from first-year services were greater on first-year outcomes than 

on later outcomes, so it is consistent that six-year outcomes show a stronger relationship to later-year 

services than to first-year services. Thus, though some findings specifically concerning SSS do appear in 

this report, many of the findings are more general in nature and apply to all supplemental services 

regardless of the source. In one sense, this is not a serious limitation; there is no specific reason that, say, 

peer tutoring would have a different level of effectiveness when provided through SSS than through other 

means. Findings about the effects of supplemental services can be interpreted as providing information 

about the effects of SSS when it supplies those same supplemental services. Also, participation in SSS 

was associated with receiving higher levels of services (whether through SSS or through other sources), 

suggesting that some of the improved outcomes from later-year services might be associated with SSS 

either because the services were provided by SSS or because SSS encouraged students to make fuller use 

of available services. However, the specific amount of the contribution provided by SSS in comparison 

to other sources of supplemental services generally cannot be determined.

In the case of services received after the first year, any student who received such services 

necessarily persisted beyond the first year, so that the presence of such services is partially correlated with 

academic success. This could lead to overstating the importance of later-year services. The problem is 

most likely to appear when dichotomous measures of SSS participation are used, while the greater amount 

of variation in the predictors when multiple and continuous measures are used helps to lessen the 

difficulty. To adjust for this issue, all of the models include a variable indicating whether the student 

attended higher education at any time during the five years after 1991–92. That way the changes in 

outcomes that are associated with persistence are measured through this variable, while the measure of the 

receipt of supplemental services will only capture any additional changes in student outcomes that are 

associated with the receipt of supplemental services. However, the addition of this variable may lead to 

underestimates of the effects of supplemental services (whether through SSS or through other means). 

Because persistence may be partially due to the receipt of supplemental services, the estimates for 

supplemental services may show only that effect that is independent of this measure of persistence, and 

not the effect that is related to the impact of supplemental services on persistence.

Some SSS services were received by only a small number of students, affecting the 

likelihood of finding statistically significant results. Of the 5,800 students originally in the study, 5,055 

ultimately had sufficient data for inclusion in the six-year outcomes analysis, with 2,380 being SSS 

participants. Depending on the service, a range from 63 to 1,892 students received each service. The 

number of students was particularly an issue for analyses that were limited to students at two-year 

institutions. Among students at two-year institutions, between eight and 415 students received each 

service. The small number of students receiving some services may also have made it more difficult to 

measure the effects of SSS services than of general supplemental services (i.e., using measures that do not 

distinguish between SSS and non-SSS services), because roughly twice the number of students would 

have received services when using these more general measures. 

Study Findings: Examination of Student Outcomes After Six Years

The single most consistent finding is that the receipt of supplemental services was correlated 

with improved student academic outcomes. This finding was consistent across all measures of academic 

outcomes and all types of statistical models (exhibit E-1). The findings for first-year SSS services in 
xvi

particular were also largely consistent and positive, except that the models using dichotomous measures 

did not show any effect from SSS. The poor performance of the dichotomous measures was anticipated 

because of the difficulty of differentiating between SSS and non-SSS services in those models, and 

appears to be a methodological artifact. The continuous measures of SSS participation showed positive 

and statistically significant effects for each of the outcome measures except transfers from two-year to 

four-year institutions and selected models using HLM. Participation in SSS was also associated with 

receiving a higher level of supplemental services, including both those services specifically designed for 

SSS students and supplemental services in general.

A second finding is that supplemental services continued to be important after the freshman 

year. In fact, the later-year services appear to show a stronger relationship to long-term outcomes than 

first-year services. SSS programs typically give the greatest emphasis to providing services for freshmen, 

though they include services for later years as well. Based on this finding, some SSS programs may wish 

to reexamine their distribution of services to be sure that some program is in place to provide services in 

later years. 

Exhibit E-1. Estimated improvement in academic outcomes associated with receiving first-year 

SSS services and with receiving supplemental services from any source and in any of 

the six academic years, using six analytic approaches: Six-year outcomes after 

freshman entry in 1991–92 

Retention or degree 

completion 

(combined)

Degree attainment

Methodology Six-year 

GPA

Credits 

earned Same 

institution

Any 

institution

Transfers 

from two-year 

to four-year 

institutions

Bachelor’s 

degree or 

higher

Associate’s 

degree or 

higher

Observed six-year outcome..................... 234 75.49 34% 63% 19% 38% 49%

(Increase) (Percentage point increase)

Dichotomous measures of 

services

Regressions

SSS services only ................................ 006 -- -- -- -- -- --

All supplemental services.................... 023 21.97 12% 16% 11% 10% 8%

HLM

SSS services only ................................ -- -- -- -- -- -- --

All supplemental services.................... 014 16.99 12% 18% 13% 10% 8%

Continuous measures of 

services

No propensity scores

Regressions

SSS services only............................ 012 2.60 4% 5% -- 6% 3%

All supplemental services ............... 025 18.77 13% 21% 7% 14% 11%

HLM

SSS services only............................ 004 3.19 5% 9% -- 2% 1%

All supplemental services ............... 017 19.17 15% 24% 10% 13% 12%

With propensity scores

Regressions

SSS services only............................ 014 2.39 7% 8% -- 5% --

All supplemental services ............... 029 18.48 16% 23% 8% 14% 9%

HLM

SSS services only............................ 002 -- 6% 4% -- -- --

All supplemental services ............... 016 16.27 17% 19% 10% 11% 11%

-- Not statistically significant. Statistical significance is based on p-value of 0.05.

NOTE: The six-year outcome is the mean outcome for all applicable SSS participants who were in the study, as measured through their 

transcripts and self-reports. The predicted outcome if students did not receive services can be calculated for any of the six analytic approaches by 

subtracting the percentage point improvement from the observed six-year outcome. For example, the mean retention rate at the same institution 

among all SSS participants in the study was 34 percent, and using the HLM model with propensity scores the mean retention rate is estimated at 

17 percent (34 percent minus 17 percent) if they had not received any supplemental services. 
xvii

Third, a few SSS services appeared to stand out by being related to improved student 

outcomes: home-based programs, blended programs, peer tutoring, labs, workshops, and services for 

students with disabilities. However, some additional types of services also were related to improved 

student outcomes, though they were not necessarily SSS services: counseling, field trips or cultural 

enrichment, referrals to outside resources, services for those with limited English ability, college reentrance counseling, and recent contacts with support services. Also, there is some evidence that what 

may be most important is that students receive an appropriate “package” of services, and some individual 

measures may not have shown independent benefits because their benefits were included in measures of 

whether the students had received packages of services (e.g., through home-based or blended programs).

Finally, the stronger findings for supplemental services in general when compared with firstyear SSS services in particular should not be interpreted as indicating that services outside of SSS were 

superior to SSS services. Instead, two reasons seem the best explanations for the differences: (1) lateryear services may be more effective than first-year services when seeking to predict long-term outcomes 

(i.e., because they are more contemporaneous with the long-term outcomes); and (2) some of the effects 

of SSS services may be captured through the other measures of supplemental services, so that the SSS 

effect is underestimated. The latter explanation seems likely both because some of the later-year services 

were likely to have been provided by SSS services, and because of the difficulty of differentiating SSS 

from non-SSS services when using dichotomous measures of participation. Also, the use of propensity 

measures may sometimes also have led to underestimating the effects of SSS. Often the propensity scores 

were insignificant, suggesting that they were not important parts of the models, and if the insignificant 

measures were dropped from the models, the estimated effect of SSS was increased. 
xviii
1

1. Introduction

Student Support Services (SSS) is one of eight federally funded grant programs that are 

administered as part of the Federal TRIO Programs within the U.S. Department of Education (ED).1 All 

eight programs are designed to help economically disadvantaged and first-generation college students 

achieve success at the postsecondary level. Some programs work with students at the middle school or 

high school level to help them complete high school and prepare for postsecondary education. Other 

programs help students while they are attending postsecondary institutions. The SSS program, in 

particular, focuses on students while they are enrolled in college and is designed to help them complete 

their postsecondary degrees or certificates, to transfer to four-year institutions if they start at two-year

schools, and to enroll in graduate school. Students may start participating in SSS immediately upon 

enrollment in postsecondary education or at a later stage. In general, SSS provides the most services to 

first-year college students, though it does provide services in later years as well. 

SSS has grown from 121 projects and approximately 30,000 participants in 1970–71 to 947 

projects and 198,940 participants in 2007–08. Its funding has increased from $10 million to $264 million 

($52 million in 1970 dollars). Participating institutions comprise roughly 22 percent of the four-year and 

two-year colleges and universities serving freshmen in the United States. SSS is the second largest of the 

Federal TRIO Programs in terms of the number of students served (after Talent Search), and the second 

largest in terms of funding (after Upward Bound).

This report presents the final results of a longitudinal study of students who began 

participating in SSS during the 1991–92 academic year. This study was mandated by Congress to 

“examine the effectiveness of current programs and to identify program improvements” (P.L. 101-166). 

Earlier results were presented in an implementation study report, completed in 1994, which described the 

characteristics of the SSS program and the SSS students at the institutions participating in the study, and 

in a third-year update, completed in 1997, which provided additional information on the implementation 

of SSS as well as an analysis of the effects of SSS on participating students with regard to their grade 

point averages (GPAs), the number of credits they earned, and their retention in higher education. This 

study updates the previous analysis by examining the effects of SSS after six years. It continues the 

analysis of GPAs, total credits earned, and retention from the earlier study, but also expands the analysis 

by examining degree completion and transfers to four-year institutions. These last two measures could 

not have been meaningfully examined in earlier years because students often had not had sufficient time 

to complete their education. The measures are much more meaningful now when applied to six years of 

time, although even that amount of time is not always enough for all students to have finished their 

education. 

This evaluation was conducted as a longitudinal study, in which both SSS participants and a 

comparison group of nonparticipants were tracked for six years following their entry into college. The 

students were selected by first selecting 200 SSS projects through a nationally representative random 

sample, subsampling 30 of those institutions and 20 comparable institutions without SSS projects for site 

visits, and selecting groups of SSS participants and nonparticipants from those institutions. Of these 50 

institutions, 47 ultimately participated for all phases of the study. From these sites and their participating 

 

1

 The eight federal programs are collectively known as the Federal TRIO Programs in the Higher Education Act of 1965, as 

reauthorized by the Higher Education Opportunity Act of 2008. The other TRIO programs are Upward Bound, Talent Search, 

the Educational Opportunity Centers (EOC), Staff Training, the McNair Fellowships, Dissemination Partnership Program, and 

Upward Bound Math-Science.
2

students, the following data were collected: student information files containing basic information on 

student characteristics; participant service records documenting each student’s contact with SSS during 

the first year; student surveys conducted in 1991–92, 1993–94, and 1996–97; and student transcripts for 

students’ six years of postsecondary education starting in 1991–92. Additional information about the 

research methodology is provided in chapter 2.

In many ways, this report builds upon the methodology and results of the third-year followup study. Some of the particularly important findings of that report that relate to the design of the study 

are listed below. Additional details about past findings are presented later in this chapter.

 The 2,900 non-SSS study participants tended to be relatively advantaged in comparison 

to the SSS students, in part due to a lack of sufficient information to identify similar 

students and in part due to the fact that some institutions did not have students who were 

comparable in their disadvantages to the SSS participants. As a result, multivariate 

analysis is used in this report (as it was previously) to statistically adjust for the 

differences between these two groups.

 SSS students were found to vary greatly in the types and amounts of services they 

received through SSS, both across SSS projects and within individual SSS projects. 

This report therefore uses a variety of methodological approaches to look at the 

importance of these variations. 

 Student demographic characteristics were important, with blacks, Hispanics, and males 

often showing poorer academic outcomes on average. Thus, these types of variables 

were also examined in this report and retained when they were statistically significant. 

 Similarly, the institution type (two-year or four-year) was associated with student 

outcomes and is retained here for use in the updated multivariate models.

This report makes selected use of the statistics and findings from the earlier reports but does 

not attempt to include all of them. Readers desiring additional information are particularly advised to 

examine the third-year follow-up report, which includes substantial information about the SSS program 

and the characteristics of SSS participants.

Organization of This Report

Because of the way that SSS programs are designed, with programs varying greatly from one 

institution to another, and students varying in the degree to which they participate, the design and analysis 

of the evaluation raised some complex methodological issues. The methodological choices potentially 

may have an important effect on the research findings, and researchers may legitimately disagree on the 

best methodological approach. Therefore, the methodological decision points are given special 

prominence in this report, and alternative methodologies are compared so that readers can determine the 

importance of the choices and give special attention to the approaches that they prefer. This report is 

structured to facilitate such comparisons. 

This chapter provides an overview of the SSS program and its students. Chapter 2 presents 

an overview of the study design and discusses the key methodological issues and their implications for 

the study. The remainder of the report progressively examines the data using each approach and 
3

discusses the findings in the context of each approach. Chapter 3 starts the analysis using the simplest 

statistical models, in which dichotomous measures of SSS and non-SSS participation are used. Some 

advantages of this approach are that it provides a single summary measure of the effects of SSS, and that 

differences in participation levels among SSS participants are ignored (because those differences might be 

indicators of differences in neediness or motivation, which also might be related to student outcomes). 

Chapter 3 includes both regression analysis and HLM so that readers can compare the two methodologies. 

Chapter 4 presents models in which SSS participation is measured through nine separate variables for 

each of the nine SSS services listed in exhibit 1-1, plus two organizational characteristics of SSS that 

were noted in the site visits. Some potential advantages of this approach are an increased ability to 

differentiate between SSS and non-SSS services (in order to measure the full effect of SSS), a better 

capacity to distinguish among SSS students to determine whether some students are helped more than 

others (e.g., based on their levels of participation in SSS), and the ability to compare features of SSS 

programs to determine whether some features are more effective than others. 

Similar to chapter 3, chapter 4 also uses both regression analysis and HLM to examine the 

nine SSS services and two organizational characteristics. Besides the use of multiple measures of 

participation, chapter 4 also differs from chapter 3 by sometimes including separate propensity scores to 

statistically adjust for the likelihood that students will receive each of the services. The primary purpose 

of adding propensity scores is to separate students’ tendency to receive services (which may in itself be 

related to student outcomes) from students’ actual receipt of services, so that measures of the relationship 

between participation and student outcomes will not be confounded by the relationship between the 

tendency to participate and student outcomes. Again, so that the reader can compare the different 

methodological choices, the statistical results are presented both with and without the use of propensity 

scores. Finally, chapter 5 attempts to synthesize the various approaches, looking at which findings are 

consistently supported regardless of the approach, which findings depend on the particular approach being 

used, and which findings (from our perspective) best describe the association between participation in 

SSS and student outcomes.

Description of SSS

The purposes of SSS are:

 To increase college retention and graduation rates for eligible students;

 To increase the transfer rates of eligible students from two-year to four-year institutions; 

and

 To foster an institutional climate supportive of success of low-income and firstgeneration college students and individuals with disabilities.

Two-thirds of the students served by an SSS project must be low-income (defined as at or 

below 150 percent of the poverty level)2 and first-generation college students or students with disabilities. 

The other third must be low-income or first-generation college students. One-third of the disabled 

students also must be low income. Services may include counseling, tutoring, workshops, laboratories, 

cultural events, special services for disabled students, and instructional courses that are solely for SSS 

students.

 

2

 In 1999, the income level was $25,050 for a family of four in the 48 contiguous states.
4

SSS projects have great latitude to custom-design their services to fit the particular needs of 

the institution. They thus vary considerably in terms of which specific services are offered through SSS 

and how they are organized. There is no single “SSS experience” that describes the participation of all 

SSS students. All SSS projects provide academic advising as one of their services (although even here 

there is great variation in the amount of academic advising received), while they vary greatly with respect 

to offering other services such as tutoring, labs, workshops, special instructional courses, and services 

specifically for students with disabilities.

SSS students are more disadvantaged than the student population as a whole, not only with 

regard to the criteria used for eligibility but also in other ways that are interrelated with their 

disadvantaged background. Following are the characteristics of the SSS students who participated in the 

national study:3

 Age. SSS students entered college later than typical students. Sixty percent of the SSS

students were either 18 or 19 years old, compared with 90 percent of all freshmen 

nationwide. 

 Gender. Two-thirds (67 percent) of SSS participants were female, compared with 53 

percent overall.

 Race and ethnicity. The three largest groups were whites (41 percent), African 

Americans (31 percent), and Hispanics (22 percent). By contrast, the respective 

percentages for freshmen overall are 80 percent, 9 percent, and 6 percent.

 Marital status. SSS participants were less likely than the general population of 

freshmen to be currently married (11 percent versus 27 percent), and more likely to be 

never married (80 percent versus 71 percent) or divorced or separated (9 percent versus 

2 percent).

 Disability. Seventeen percent of SSS participants reported having some kind of 

disability. The most common was a specific learning disability, with 6 percent of SSS 

participants reporting one (compared with 2 percent among freshmen overall).

 High school graduation. Ninety-one percent of SSS participants graduated from high 

school (compared with 98 percent overall), 9 percent obtained a GED, and 1 percent left 

high school without receiving a diploma.

 Academic major. When they were freshmen, 18 percent of SSS participants selected 

health-related fields as their major, 17 percent chose business, 11 percent chose 

education, and 11 percent chose social sciences. 

 Residence. Sixty-four percent of SSS participants attended colleges that were within 50 

miles of home, compared with 46 percent of all freshmen.

 

3

 More detailed information about the characteristics of the SSS participants is provided in chapter 2 and in National Study of 

Student Support Services, Report Number 2, Profile of Freshmen Participants and Project Services: 1991–92, U.S. Department 

of Education, 1994.
5

 Financial aid. SSS participants were much more likely to receive financial aid than 

undergraduates overall (82 percent versus 42 to 45 percent, depending on the 

dependency status of the students).

The SSS Services

Because SSS projects vary greatly, this study collected data on the number of services that 

each student received as a freshman in nine different areas (exhibit 1-1). The services that were most 

widely received were professional counseling (80 percent of all SSS students), peer tutoring (45 percent), 

instructional courses limited to SSS students (32 percent), and workshops (31 percent), while the 

remaining services were received by 17 percent or less of the students.

Institutions’ packages of services varied, both in terms of what services were offered and 

whether the services were offered through SSS or some other mechanism. Thus, sometimes students did 

not have the opportunity to participate in some of the services listed in exhibit 1-1. At other times, the 

services may have been available, but they were provided through a mechanism other than SSS. For 

example, in order to help demonstrate that the funds for SSS were not supplanting existing institutional 

programs, an institution may have chosen to create special new services for SSS students, while also 

referring the SSS students to additional services outside of SSS that had already been offered at the 

institution. Thus, a list of the SSS services that a student received typically is not a complete list of all of 

the supplemental services that student received at the institution. 

As a rule, SSS students are in full control of determining both the types and amounts of 

services they receive, as long as the services are offered by the institutions and the students qualify for 

those services (i.e., a student must have a disability in order to receive special services for individuals 

with disabilities). At least two exceptions were noted in the case studies, however: essentially all SSS 

students participated in academic advising at the time of their enrollment, and some institutions made a 

practice of calling in students for additional advising if their early performance at the college was poor. 

In addition, the case studies revealed that SSS projects differed in their basic organization, 

and the analysis in this report focuses on two characteristics in particular. First, home-based programs 

provided a home base on campus that served the “whole student” by providing a broader range of services 

to facilitate the students’ integration on the campus and by seeing that any needed supplemental services 

were provided. Often home-based programs made special attempts to have group activities for the SSS 

students, such as cultural events or service projects. They can be contrasted with dominant service

programs, which primarily focused on providing a single service, and all service programs, which served 

as the only (or at least primary) provider of support services at the institution.4 Second, SSS programs 

differed in the extent to which the SSS services were blended with other services on campus. In order to 

satisfy early federal requirements (up to 1992) for nonsupplanting and nonduplication, almost all of the 

programs had ways of maintaining their unique service and population served; still, some did this by 

coordination with other service providers and some by having a more separate service delivery model. 

SSS program regulations were later changed to allow a greater blending of services, but this distinction 

was important at least at the start and proved sometimes to be important in the statistical analysis.

 

4

These categories are not necessarily mutually exclusive. For this analysis, however, we chose to describe each program using 

only the single category that best described the program, rather than assigning multiple categories to a program. Further, 

because only three programs in our sample fit the all service category, and our initial investigation suggested that home-based 

programs deserved the closest analysis, we focus on the distinction between home-based programs and all other programs.
6

Exhibit 1-1. List of SSS services and the frequency that they were received in 1991–92

Service category and description Types of services

Instructional courses

Courses were developmental in focus

Only includes courses that were offered exclusively to SSS 

students

Courses may or may not have been for credit

study skills (16 percent of SSS students)

writing (12 percent)

developmental mathematics (11 percent)

reading (7 percent)

developmental English (4 percent)

English proficiency (1 percent)

Professional tutoring

Designed to provide assistance with course work

All tutors were paid, but this group was distinguished by the 

use of graduate students or faculty and staff to perform the 

tutoring

Includes both one-to-one and group tutoring

English (8 percent)

general tutoring (3 percent)

mathematics (3 percent)

science (1 percent)

Peer tutoring

Similar in design to professional tutoring but performed by 

undergraduates

mathematics (21 percent) 

English (17 percent)

science (9 percent)

social sciences (4 percent)

general tutoring (3 percent)

Professional counseling

Provision of advice or counseling (as distinct from content 

area instruction)

Performed by graduate students or faculty and staff

academic counseling/advising (60

percent)

personal counseling (20 percent)

financial aid counseling (19 percent)

career counseling (8 percent)

Peer counseling

Performed by undergraduates

Otherwise similar to professional counseling

academic counseling/advising (10

percent)

personal counseling (4 percent)

financial aid counseling (1 percent)

Labs

Supplemental assistance in content areas

Provided in group settings

Similar to group tutoring

mathematics (7 percent)

writing (4 percent)

reading (3 percent)

English (2 percent)

Workshops

Designed to provide skill enhancement rather than content 

knowledge (e.g., study skills and orientation)

orientation to college (18 percent)

study skills (8 percent)

career guidance (3 percent)

Cultural events

Group trips to concerts, museums, or other events

concerts (3 percent)

museums (1 percent)

lectures (1 percent)

other events (4 percent)

Services for the disabled

Special services, such as note-taking, counted as SSS services 

only if the services were provided through SSS (rather than 

referrals)

counseling (2 percent) 

note takers (1 percent)
7

Findings From Earlier Years of This Study

This report focuses on student outcomes six years after freshmen began participating in SSS. 

As part of the same longitudinal study, earlier reports discussed the effects of SSS on student outcomes 

after one year and after three years. The outcomes that were examined earlier were largely the same 

outcomes as those discussed here (effects on grade point averages, credits earned, and retention, though 

over shorter time frames), except that the one- or three-year time periods were too short to examine the 

effects on degree completion and on transfers from two-year to four-year institutions. Following are the 

main findings presented in those reports.

 SSS showed a small but positive and statistically significant effect on students’ GPAs, 

number of semester credits earned, and retention. The greatest effect generally 

occurred during the first year, when the most SSS services were received, but some 

SSS services received in the first year showed persisting effects in later years, and 

some services received in later years (not necessarily through SSS) also showed 

positive and statistically significant effects. 

─ Students’ college GPAs were increased by a mean of 0.15 in the first year, 

resulting in a mean GPA of 2.29 on a four-point scale (e.g., “A”=4.0 and 

“C+”=2.3). In the second year, the mean increase was 0.11 (to 2.44), and in 

the first three years combined the increase was also 0.11 (to 2.59). 

─ The number of semester credits earned was increased by a mean of 1.25 (to a 

total of 20.91 credits) in the first year, 0.79 (to 20.62) in the second year, 0.71 

(to 20.58) in the third year, and 2.25 (to 73.38) in the first three years 

combined. 

─ Retention was increased at the same institution by 7 percentage points (i.e., 

from 60 percent to 67 percent) for retention to the second year, and by 9 

percentage points (i.e., from 40 percent to 49 percent) for retention to the third 

year. Retention to the third year at any higher education institution was 

increased by 3 percentage points (i.e., from 74 percent to 77 percent).

 The average effect was small because most students received only a modest amount of 

services. Nine percent of students had only one service contact in their freshman year. 

The mean number of hours of services received in the first year was 32, and the 

median was 14. The mean for nonfreshmen was 15 hours, and the median was 6.

 The size of the effect depended on the degree to which students participated in SSS, 

with greater levels of participation resulting in a greater effect.

 The estimated effects of SSS also varied based on which particular services each 

student received and the structure of the SSS projects. 

 The effects of SSS generally appeared consistent across different subgroups of 

students. To the extent that some subgroups showed different effects than other 

subgroups, those differences appear attributable to differences in participation levels 

rather than to differing effects of individual services. In general, students who were 

more disadvantaged tended to participate more and thus experienced greater effects 

through SSS participation.
8

 SSS projects appeared successful in targeting those students who were most 

disadvantaged from among the overall student population. In comparison with the 

national averages for college freshmen, SSS participants tended to be older, to be 

members of a minority group, to have had lower prior academic achievement, and to 

have dependent children. Minority SSS participants composed 54 percent of the SSS 

student population, whereas minority populations represented only 25 percent of the 

total undergraduate population.

 SSS students received higher levels of supplemental services than did comparison 

students, including services offered outside of SSS. This suggests that SSS increased 

the amount of services obtained by students beyond what they would have received 

otherwise. This difference in service use declined substantially after the freshman 

year. For example, 63 percent of these students received tutoring at some point during 

their first three years compared with 36 percent of comparison group members. In the 

first term, 46 percent of SSS participants reported use of tutoring compared with 20 

percent of comparisons. By the spring 1994 term, 11 percent of SSS and 8 percent of 

comparisons reported use of tutoring. There was less difference between the SSS 

participants and the comparison group in levels of counseling use.

 Although SSS has increased greatly in size, when adjusted for inflation, the funding 

per program and per participant was less than in 1970. SSS program funding went 

from its initial funding level of $10 million in 1970 to $143.5 million in 1995. Over 

the same time period, the number of projects funded grew from 121 to 706, and the 

total number of students served by the SSS program from 30,000 to 165,561. The 

number of students served peaked at 181,368 in 1981. In constant 1990 dollars, the 

average grant size declined from $278,393 in 1970 to $174,365 in 1995. The level of 

per-participant funding (in constant 1990 dollars) was highest in 1970 at $1,123, 

declined to $507 in 1981, and reached $744 in 1995. In 1995 current dollars, funding 

per participant was $867. 

 SSS programs are concentrated at particular types of institutions. In 1994, 

approximately 24 percent of all higher education institutions serving freshmen had 

SSS projects. Because SSS projects tended to be located in larger schools, about 34 

percent of all freshmen attended institutions having SSS projects. SSS projects tended 

to be concentrated in four-year institutions, public institutions, institutions enrolling 

more than 20,000 students, and institutions with 50 percent or more minority 

enrollment. Over 40 percent of doctoral institutions compared with 15 percent of 

baccalaureate institutions and 22 percent of two-year institutions had SSS programs. 

Relatively few highly selective institutions (19 percent) had SSS projects. 
9

2. Study Design

This chapter discusses the key concepts in the design of this study: the outcome measures 

used, an overview of the study design, the longitudinal study, and the interpretation of the data. 

Additional information about the research methodology is provided in Appendix B.

Outcome Measures Used

Five types of measures were used as outcome measures for the study (see Exhibit 2-1). The 

following discussion provides additional information about each variable as it applies to the SSS students.

Exhibit 2-1. Outcome measures for the sixth-year evaluation after freshman entry in 1991–92

Variable Description Source

GPA (cumulative) Four-point scale (A=4.0) Postsecondary transcripts

Total credits earned (cumulative) Total number of semester credits Postsecondary transcripts

Retention or degree completion

 At the same institution

 At any institution

Student either earned a degree (bachelor’s degree or 

higher) within six years, or was still in college in the 

sixth year

Postsecondary 

transcripts, 1996–97

survey of students

Degree completion

 Bachelor’s degree or higher

 Associate’s degree or higher

Student earned the specified degree within six years Postsecondary 

transcripts, 1996–97

survey of students

Transfers from two-year to four-year

institutions

Student who originally enrolled at a two-year

institution later transferred (within six years) to a fouryear institution

Postsecondary 

transcripts, 1996–97

survey of students

Grade point average (GPA). After six years, SSS participants in the study had a mean 

GPA of 2.3 (i.e., about a C+ on a 4.0 scale), with most students (81 percent) getting Bs or Cs as their 

average grades (exhibit 2-2). 

Exhibit 2-2. Percentage distribution of all SSS participants, by mean GPA levels six years 

after freshman entry in 1991–92

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
10

Number of credits earned. On average, the SSS participants in the study earned 75.5 

semester credits over their first six years in college, or roughly the amount that would be earned through 

2.5 years of full-time enrollment.5 About one-fourth (28 percent) had earned more than enough to 

graduate (exhibit 2-3), although they had not necessarily met other graduation requirements (83 percent of 

them had completed a baccalaureate degree or higher). The total number of credits earned depends on 

several factors, including whether or not the students had graduated from college (having graduated 

means that students must have taken a certain minimum number of credits in order to qualify, but 

graduation also is a reason for students to consider their education completed and to stop earning further 

credits), whether the student was attending part-time or full-time, and whether the student had stopped 

attending (or dropped out) for one or more years. Disadvantaged students are more likely to be part-time 

and to interrupt their education than other students, so in general they would be expected to earn fewer 

credits over the same amount of time. Measures of the amount of credits earned can vary from one 

institution to another because institutions sometimes vary in which courses are for credit and which earn 

no credits; however, this distinction is probably less important in later years than in the first years, 

because some of the introductory courses show the most variation in how they are treated.

Exhibit 2-3. Percentage of SSS participants who earned various amounts of credits over six 

years after freshman entry in 1991–92

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.

Retention, degree attainment, and transfers to four-year institutions. Two of the 

primary goals of SSS are to help disadvantaged students remain in college and ultimately to earn degrees 

(especially four-year, or baccalaureate, degrees). These goals are interrelated since students must remain 

in college for some minimum amount of time in order to earn degrees, and once students do earn degrees, 

they are likely to stop being enrolled in college (unless they seek more advanced degrees). A third goal 

also is closely related to these goals: encouraging students at two-year institutions to transfer to four-year

institutions so they can earn four-year degrees. Transferring to four-year institutions is related to both 

retention (because one must remain in education a longer time) and degree attainment (especially because

 

5

 Credits earned under the quarter system were converted to semester credits to provide comparable measures across students and 

schools.
11

degree attainment is here usually defined in terms of receiving a bachelor’s degree or higher). Because of 

the interrelationships among these three outcomes, all three are discussed together.

Now, with six years of data available, the analysis of retention can be expanded to cover 

more years, and it is possible to examine degree attainment and transfers to four-year institutions. Even 

six years is not enough time to come to a complete conclusion for all students, however. While four years 

is considered the traditional amount of time to complete a bachelor’s degree, only about a third (36 

percent) of college graduates complete a degree in that time after starting college.6 By contrast, about 

one-fourth (26 percent) takes more than six years to complete their degrees. Further, disadvantaged and 

minority students tend to take longer to complete their degrees than other students. While 25 percent of 

white college graduates took more than six years to complete a bachelor’s degree, 32 percent of blacks, 

35 percent of Hispanics, and 43 percent of American Indians and Alaskan Natives took more than six 

years.7 

After six years, 14 percent of the SSS participants had completed a baccalaureate degree and 

were continuing in postsecondary education, 23 percent had completed a baccalaureate degree and were 

no longer in college, 5 percent had completed an associate’s degree and were continuing in college, 7 

percent had completed an associate’s degree and were no longer in college, 20 percent were still in 

college without completing a degree, and 31 percent neither were in college nor had completed a degree 

(exhibit 2-4). This rate of baccalaureate degree completion (37 percent after six years) was higher than 

that found for the most disadvantaged students in the Beginning Postsecondary Students (BPS) study, 

which showed completion rates of 25.7 percent for those with family incomes lower than $25,000, and 

32.7 percent for those with incomes between $25,000 to $44,999.8

 

6

 Thomas M. Smith, The Condition of Education 1996, U.S. Department of Education, National Center for Education Statistics, 

NCES 96-304 (Washington, D.C.: 1996), 60.

7

 Ibid., p. 215. Asians and Pacific Islanders were an exception among minorities, with only 17 percent taking more than six years 

to complete a bachelor’s degree. The three racial or ethnic categories used here are highly correlated with being disadvantaged 

and are used because statistics on the education status of these groups are more readily available.

8

 National Center for Education Statistics, Descriptive Summary of 1995–96 Beginning Postsecondary Students: Six Years Later, 

U.S. Department of Education, (Washington, D.C.: 2002), 57.
12

Exhibit 2-4. Retention and attainment of degrees by SSS participants six years after 

freshman entry in 1991–92

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.

Degree completion alone. While retention and degree completion can be examined through 

combined measures, encouraging degree completion by itself is one of the primary goals of SSS. This 

study therefore examined participants’ degree completion rates, regardless of whether they were still in 

college. Because this analysis is based on six years of data, and disadvantaged students often take a

longer period of time to complete their degrees, these data reflect the effects of SSS after six years, but 

not necessarily the final SSS effect. The final effect might be larger (if additional SSS participants 

complete their degrees later, due in part to their participation) or smaller (if, say, SSS provides a 

temporary advantage, but nonparticipants eventually catch up).

This analysis uses three basic measures of degree attainment—whether participants earned a 

bachelor’s degree or higher (a specific goal of SSS), whether participants earned any degree (including an 

associate’s degree), and whether participants at two-year institutions earned any degree (again, including 

an associate’s degree). The third measure is used to better understand the effects of SSS at two-year

institutions, since students at such schools may be less likely to earn four-year degrees. While it would be 

possible to examine attainment of associate’s degrees at two-year institutions (rather than an associate’s 

degree or higher), such an analysis might produce misleading results: students who did not earn 

associate’s degrees but did earn bachelor’s degrees would be considered “failures” in such a measure (i.e., 

they had not earned associate’s degrees), and the group of “failures” thus could include both students on 

whom SSS had no effect and students on whom SSS had a great effect. 
13

Such a measure could easily lead to improper estimates of the effects of SSS. In fact, 65 

percent of community college beginners who transferred to a four-year institution did so without first 

earning a degree at the two-year institution.9 By instead counting the attainment of any degree (i.e., an 

associate’s degree or higher), all students whose degree attainment may have been affected by SSS are 

included. However, data based on two-year institutions only should be considered as less reliable, 

particularly for testing statistical significance. With fewer cases available for analysis, combined with the 

fact that some SSS services were received by only a small number of SSS students, there is an increased 

prospect that some measures of SSS may not be statistically significant based on the number of cases 

alone. In fact, the standard demographic and attitudinal variables that were incorporated in the 

multivariate analysis also performed poorly and were often statistically insignificant. Further, diagnostic 

tests indicate that the logistic regression model may be providing questionable statistics, which again may 

be due to the reduced number of cases available for analysis.10

Transfers from two-year to four-year institutions. Because a purpose of SSS is to 

encourage disadvantaged students to earn four-year degrees, encouraging transfers from two-year to fouryear institutions is a subsidiary goal. Among the SSS participants in the sample at two-year institutions, 

19 percent transferred to four-year institutions. This percentage appears to be somewhat higher than 

would be expected based on other databases. According to data from the Beginning Postsecondary 

Students Longitudinal Study, 13 percent of full-time low-income freshmen at two-year institutions had 

transferred to four-year institutions within five years of starting their college careers.11

Two factors complicate the measurement of the SSS effect on transfers in this study. First, 

because the sample was not limited to two-year colleges, fewer cases were available for the analysis of 

transfers than for the preceding analyses, making statistically significant findings less likely to appear. 

Second, transfers to four-year institutions typically occur at the end of the second year (i.e., the typical 

end of a two-year program) or later, since many SSS students switched to part-time status and appeared to 

take longer to complete their academic careers. In that context, the focus on first-year SSS services may 

not necessarily be the best basis for examining transfers; it may be that decisions on transfers are made at 

a much later time, so that first-year services are not particularly relevant. For example, SSS advisors may 

primarily concentrate on survival and general academic strategies in the first year, and shift their focus to 

assisting with transfers in later years.

Employment. Another way of evaluating the effects of SSS participation is to examine the 

patterns of participants’ employment. If participation in SSS helps students to be more successful in 

college (as indicated by the last two chapters in this report), and if attending college helps students to be 

more successful in employment, there logically should be some connection between SSS participation 

and students’ success in their employment.

However, several factors complicate the search for such connections. First, many students 

might be expected to take more than six years to complete their college education, so students may not yet 

 

9

 A. McCormick and C.D. Carroll, Transfer Behavior Among Beginning Postsecondary Students: 1989–94, U.S. Department of 

Education, National Center for Education Statistics, NCES 97-266 (Washington, D.C.: 1997), vii.

10 The models converge properly when the full database is used, but they do not consistently do so when used only for two-year 

institutions.

11 1989–90 Beginning Postsecondary Students Longitudinal Study, Second Follow-up (BPS:90/94), National Center for 

Education Statistics, U.S. Department of Education. Low-income freshmen here are defined as freshmen whose families were 

in the bottom quartile in their socioeconomic status.
14

have had time for their employment to be greatly affected by their college attendance. Second, in some 

ways employment and education reflect two competing ways for people to spend their time, and their 

choices depend in part on their families’ financial resources. Employment may be an indicator of 

financial need rather than of success in education. Because SSS targets disadvantaged students, a high 

employment rate is likely as a response to the students’ generally greater financial needs and may 

possibly interfere with academic progress rather than being a result of it. Third, the six-year period of this 

study is too short for many students to have become established in their careers (especially considering

that much of that time may have been consumed by education), so that students’ current careers may be 

poor indicators of their ultimate careers. Fourth, in the short term employment success sometimes is 

correlated more with seniority than education; people who have been employed continuously for six years 

may have greater temporary employment success (e.g., higher salaries due to experience and seniority) 

than people just entering employment with a college degree, even though the college degree ultimately

may be correlated with greater lifetime earnings. Thus, if SSS encourages students to stay in college, in 

the short term the employment effects might appear to be negative. Fifth, there is no single perfect 

indicator of employment success; for example, while annual salaries are one potential measure, two 

people may be equally successful in their chosen careers and yet have very different incomes.

Given all of these limitations, this report does not look at employment outcomes for potential 

effects from SSS.

Overview of the Study Design

SSS provides a relatively moderate-sized intervention in students’ educational experiences, 

and the nature of the SSS experience varies from one institution to another and from one student to 

another. These qualities make it a difficult program to evaluate. One would not expect to find large 

effects on students, given the moderate nature of the SSS experience, and may not necessarily find 

consistent effects, given the wide variation in projects and levels of participation. 

Because of the potential difficulties in measuring the effects of SSS, the evaluation study 

was designed to provide multiple perspectives on SSS:

 A nationally representative, stratified random sample of 200 SSS projects was 

surveyed in 1991–92.

 Site visits were made to 50 higher education institutions, with 30 having SSS projects 

and 20 not having SSS grants. The 30 were randomly subsampled from the sample of 

200 institutions for the project survey. The 20 non-SSS sites were selected to match 

the SSS sites, based on enrollment size, geographic region, selectivity, percentage 

receiving Pell Grants, institution type (two-year, four-year), and institutional control 

(public, private). Three of the institutions later dropped out of the study, leaving 47

institutions that became the primary focus of the analysis in this report.

 Project-reported data were collected from the annual performance reports submitted 

by each project, providing information on the students served and the types of services 

provided.
15

 Other basic data were collected from the Department of Education, the Council for 

Opportunity in Education (COE; formerly called the National Council of Educational 

Opportunity Associations, or NCEOA), the Integrated Postsecondary Education Data 

System (IPEDS), and a nationally representative Higher Education Survey (HES) on 

institutions’ retention practices.

 A longitudinal study was conducted to compare the outcomes of 2,900 SSS 

participants and 2,900 comparison students who did not participate in SSS but had 

similar background characteristics. The students were selected from 28 of the 30 

sampled SSS projects, and 19 of the 20 non-SSS institutions.

The Longitudinal Study

The longitudinal study is the primary focus of this report, though data from the other 

components of the study were used to help develop the longitudinal study and are also used here to 

supplement the data and help in interpreting the statistical results. Just as the evaluation study as a whole 

sought multiple perspectives, the longitudinal study also collected data from multiple sources. It should 

be noted that this study utilizes data from the 1990s and may not capture the full nature of the SSS 

program as it operates today.

 Student information files provided by the 47 participating institutions provided basic 

information on student characteristics, though the amount of information that was 

available varied from one site to another.

 Participant service records provided detailed information on each service contact 

through SSS with the sampled students, including the types of service, length in 

minutes, date, and number of students in the service.

 Student surveys were conducted at three different time points: (1) to provide baseline 

measures during students’ initial enrollment as freshmen at the participating 

institutions in 1991–92, (2) in the third year (1994–95) to provide updated information 

on students’ college experiences, plans, and attitudes, and (3) in the sixth year (1997–

98) to provide further updates on the students, including potentially their graduation 

and attendance at postsecondary institutions.

 Student transcripts were collected at the end of the first year, at the end of the third 

year, and at the end of the sixth year to provide detailed information on students’ 

academic progress. Because many students transferred to other institutions during the 

six-year course of the study, the total number of institutions involved in the transcript 

collection increased from an initial base of 47 institutions to roughly 850 institutions 

by the end of the third year, and to roughly 1,050 by the end of the sixth year.

The design of the longitudinal study raised a number of complex methodological issues. In 

the remainder of this section, we discuss these issues, the study design that was chosen, and the 

implications of the methodological issues for this study.

This study used a quasi-experimental design, in which two groups of students were chosen to 

have roughly similar characteristics, with one group receiving SSS services and the other not receiving 
16

such services (at least during their freshman year). By comparing the outcomes of both groups, and also 

allowing for other supplemental services that might have been received outside of SSS, the study 

developed estimates of the impact of SSS and of supplemental services in general. If exposures to SSS 

had been randomized, the study could be classified as an experimental design, but there was no 

randomization. Additional discussion of the differences between experimental and quasi-experimental 

design, and on the reasons for choosing a quasi-experimental design for this study, is provided in 

Appendix B.

The presence of self-selection of services in SSS raised difficult but unavoidable issues for 

the evaluation study. Therefore, this study used a variety of data and statistical approaches in order to 

estimate the importance of the self-selection on student outcomes. It presents separate analyses that either 

(1) treat all SSS students the same regardless of what services they have received or (2) specifically 

model the amount and type of services that each student received. It also used a large amount of 

supplemental data on student characteristics both to statistically adjust for the interrelationship between 

those characteristics and student outcomes and to model students’ likelihood of participation.

To monitor students’ use of supplemental services, the SSS study sites maintained detailed 

records of each service contact with the sampled students during the first year. (Students’ level of

participation in SSS is typically highest during the freshman year, though they can and sometimes do 

continue to receive services in later years. No official service records were maintained after the freshman 

year in order to reduce the burden on the participating institutions.) Also, the student surveys in the first, 

third, and sixth year of the study were used to collect students’ self-reports on their receipt of 

supplemental services to capture information on SSS students’ receipt of non-SSS services and services 

received after the first year, and on comparison students’ receipt of any supplemental services. 

Inclusion of institutions without SSS programs. To further strengthen the study design, 

and at the request of the U.S. Office of Management and Budget, this study included 19 institutions that 

did not have SSS programs as a source for additional comparison students. This design aspect has two 

main advantages:

 As noted, in order for random selection to be maximally useful, the randomization 

should be applied to all relevant characteristics. It is possible that institutions that 

have SSS programs differ systematically from other institutions; for example, they 

may have a greater orientation to accepting minority or disadvantaged students, they 

may be especially well administered (e.g., in order to win an SSS grant against 

competing institutions), or they may have a strong orientation to providing students 

with supplemental assistance and working to see that all students are integrated into 

the campus community. All of these factors could potentially be related to student 

outcomes, and their importance may affect the viability of expanding SSS to 

additional institutions, as has been the trend. Some of these characteristics also might 

lead to (or arise from) SSS institutions having a fundamentally different mixture of 

students than other institutions (at least with regard to certain characteristics). By 

including non-SSS institutions in the study, one has the capacity to check for 

differences in student characteristics, and the study can better be generalized to a 

larger set of institutions. 

 There is the potential that SSS programs may “skim” the most disadvantaged students 

in an institution, so that there is not a truly comparable set of nonparticipating students 

facing the same disadvantages within that institution. By including institutions that 
17

are similar to SSS institutions except that they lack SSS programs, one has a better 

potential for finding comparison students who collectively are similar to the SSS 

participants, even if SSS students and non-SSS students are not necessarily 

comparable within any particular institution. 

Selection of comparison group. Because randomization was not used to produce 

equivalent sets of treatment and comparison students, it was necessary in some other way to select an 

appropriate group of comparison students. The comparison group was chosen by using regression 

analysis to calculate propensity scores of students' likelihood of participation in SSS based on 

demographic data that were available from the colleges.12 The derived formulas were then used to choose 

a comparison group at each institution of students whose propensity scores showed similar distributions. 

As shown in Exhibit 2-5, the propensity scoring was useful in selecting students who were 

more like the SSS participants than like typical students, but the differences were only partially overcome. 

In particular, SSS students tended to be more disadvantaged than the students in the comparison group; 

for example, they were more likely to have household incomes of $10,000 or less (30 percent versus 15 to 

17 percent) and to have parents with less than a high school education (e.g., 35 percent versus 17 to 20 

percent with regard to the father’s education background).13 The primary reason the SSS students and the 

comparison group were not more similar appears to be the lack of adequate data for fully comparing the 

two groups of students. For example, some institutions did not have information about the race or 

ethnicity of their students or about the students’ finances, especially for those who were not receiving 

assistance. Thus, the fewer the items that were available, the less powerful were the propensity models, 

and most institutions were able to supply only a limited amount of data. Also, as revealed by case 

studies, it was sometimes difficult to select comparable non-SSS students because that the SSS programs 

were so highly targeted at a few institutions that there were no comparable non-SSS students with similar 

characteristics. The number of SSS students at such institutions, however, was sufficiently small as to 

have only a minor effect on the overall averages. Also, the students in the highly targeted programs were 

actually less likely to be disadvantaged than the other SSS students (e.g., 77 percent were white, 

compared with 35 percent of other SSS students), so that targeting was not the major source of the 

differences between the SSS and comparison groups. Whatever the cause, there were systematic 

differences between the SSS participants and the comparison group, with the SSS participants being more 

disadvantaged. The analysis therefore required the use of statistical adjustments to correct for these 

differences. 

 

12The formulas that were used for calculating propensity scores varied from one institution to another because institutions 

differed in the amount of information that they were able to provide for use in the statistical model.

13 The cited differences are all statistically significant at the 0.01 level based on t-tests.
18

Exhibit 2-5

Comparison of SSS students with comparison group and all freshmen, by student characteristics

Comparison group All

Student characteristic SSS SSS schools Non-SSS freshmen

Demographic data

Percent male ........................................................................... 33% 40% 40% 47%

Never married......................................................................... 81% 87% 86% 71%

Have dependent children ........................................................ 22% 15% 15% 24%

Percent white .......................................................................... 42% 50% 55% 80%

Over 100 miles from home..................................................... 18% 19% 24% 36%

Age 19 or under...................................................................... 75% 79% 70% 92%

Household income $10,000 or less......................................... 30% 17% 15% 7%

Other language besides English spoken at home ................... 31% 23% 17% --

Understand English very well ................................................ 71% 88% 83% --

Speak English very well ......................................................... 59% 82% 69% --

Write English very well.......................................................... 48% 78% 67% --

Read English very well........................................................... 60% 82% 72% --

Parental information

Lived with father/male guardian ............................................ 72% 77% 75% --

Lived with mother/female guardian ....................................... 94% 96% 97% --

Father was manager/proprietor............................................... 18% 24% 24% --

Mother in service occupation ................................................. 51% 39% 36% --

Father had less than high school education ............................ 35% 20% 17% 12%

Mother had less than high school education........................... 30% 19% 15% 9%

Academic information

Have taken SAT or ACT ........................................................ 81% 87% 85% --

SAT Verbal (mean) ................................................................ 399.0 455.3 445.0 422

SAT Math (mean)................................................................... 428.7 463.0 462.4 474

ACT Composite (mean) ......................................................... 18.8 20.2 20.4 21

"A" as average grade .............................................................. 10% 11% 17% 24%

Years of math (mean) ............................................................. 3.3 3.4 3.4 --

Took courses at other college ................................................. 29% 36% 40% --

Credits earned at this institution (mean)................................. 21.1 21.5 22.3 --

Full-time student..................................................................... 90% 91% 90% --

Work-study job....................................................................... 16% 13% 11% --

College finances

Received financial aid ............................................................ 82% 73% 73% 45%

High school counselor helped assemble................................. 26% 19% 19% --

College counselor helped assemble........................................ 27% 26% 16% --

Self-assembled........................................................................ 50% 68% 60% --

Listed as dependent by parents............................................... 43% 62% 64% --

Received assistance of $600 or more ..................................... 27% 43% 48% --

Financing college is major concern ........................................ 41% 34% 31% --

Attitudes

BA/higher as highest planned degree (this college)............... 67% 72% 68% 67%

BA/higher highest planned at any college.............................. 91% 95% 95% 90%

Definitely able to complete college ........................................ 71% 79% 80% --

Academic ability above average............................................. 43% 59% 57% 54%

Drive to achieve above average.............................................. 65% 72% 73% 67%

Emotional health above average............................................. 58% 67% 71% 55%

Leadership ability above average ........................................... 51% 62% 65% 50%

Mathematical ability above average ....................................... 30% 42% 41% 37%

Physical health above average................................................ 57% 74% 73% 56%

Popularity above average ....................................................... 40% 52% 57% 38%

Intellectual self-confidence above average............................. 57% 71% 71% 51%

Social self-confidence above average..................................... 56% 66% 66% 45%

Writing ability above average ................................................ 38% 56% 54% 40%

--Data not available.

SOURCE: U.S. Department of Education, Planning and Evaluation Service, National Study of Student Support Services (SSS), Service Record 

Analysis, 1991-92.
19

It is important to note that though the distribution of characteristics among the SSS and 

comparison groups was different, both groups were diverse (though disadvantaged) and there was no

discernable group of SSS students who were so disadvantaged that the comparison group lacked such 

students. If such a subgroup did exist, then statistical models might be limited in their ability to discuss 

program effects for such students. Fortunately, the two groups did overlap in their coverage of students, 

thus improving the chances that the results would be generalizable and that multivariate regression 

models could successfully adjust for these differences. The pooling of data across multiple institutions is 

especially helpful in this regard because it helps to provide groups of students with similar characteristics 

in both the SSS and comparison groups even if SSS students were highly distinct in a few institutions. 

The inclusion of students from institutions without SSS programs also helped by providing institutions in 

which skimming of the most disadvantaged students by SSS was not an issue. In order to maximize 

diversity among both SSS students and comparison students, the analyses were conducted with all of the 

institutions combined. A limited number of analyses were conducted on an institution-by-institution basis 

in the analysis after three years, but these models were considered inferior both because of the less 

adequate coverage of student characteristics and the reduced number of cases available for analysis. The 

multivariate regressions did include variables to measure some institution characteristics (whether the 

institutions were two-year, four-year, or doctorate-granting institutions, whether the SSS program 

included home-based or blended services, and whether the institutions had SSS programs), and the use 

HLM did model differences in variances on an institution-by-institution basis).

Exhibit 2-5 also shows that there generally were only small differences between the 

comparison students based on whether they were at SSS or non-SSS institutions. Because statistical 

adjustments for the differences between SSS and non-SSS students are required in any case, and these 

adjustments can also compensate for these small differences between the two types of institutions, the 

distinction between the two comparison groups is unnecessary. Accordingly, this report combines both 

groups of comparison students into a single group.

Intent to treat. One of the critical issues in conducting an evaluation can be the 

determination of which people are considered to have received treatment. For example, in a medical 

study, some patients might drop out because of the side effects they experience from the medication being 

studied. If those patients are ignored, then one would not have a proper measure of either the frequency 

of side effects or of the success of the medication. The solution is to consider the initial enrollment of a 

person into a treatment group as being the critical factor (i.e., that there was an “intent to treat”) even if a 

person does not receive the full treatment. However, sometimes exceptions are appropriate, and it can be 

difficult to know precisely where to draw the line. Using another medical example, many antibiotics 

require multiple doses (e.g., often over a 10-day period), and not only is it ineffective to terminate the 

medicine prematurely, but it may also be counterproductive (if the infection is not eliminated completed, 

the bacteria may develop a resistance to the antibiotic and be harder to treat afterward). Should a patient 

who received only a partial treatment still be considered as part of the treatment group? The answer, 

which depends on whether the withdrawal was in some way related to the treatment or disease, may not 

always be straightforward.

Because SSS allows students to set their own levels of participation, and because the SSS 

programs also vary from one institution to another, the actual amount and types of services received vary 

greatly. Many students might be considered as dropouts from the program, whether based on the number 

of contacts or the hours of contacts: 9 percent received only a single service contact over the entire 

freshman year, 7 percent received less than 1 hour of total services in the first year, and 22 percent 

received between 1 and 5 hours of services. Though it is conceivable that even a single contact might be 
20

sufficient to resolve a key intellectual problem or impart a needed skill, it would not be surprising if most 

or all of these students experienced little effect from their participation in SSS. In fact, many education 

policies are based on the assumption that the amount of time devoted to education is important, including 

many reform efforts aimed at increasing the time devoted to reading and mathematics, and even the way 

that graduation requirements are specified (in terms of the number of hours or credits of instruction 

received). Particularly if the students dropped out because they perceived the program as providing no 

benefit, it is reasonable to consider the students as “failures” in the sense that SSS failed to impart needed 

skills or to motivate students sufficiently to benefit from the SSS offerings. On the other hand, these 

students also consumed few of the resources in terms of the dollars spent for supplemental services. If 

one were conducting a cost-benefit analysis, one might even argue that allowing dropouts actually 

increases the program’s efficiency by reducing the use of resources for those who do not benefit from 

them (e.g., assuming either that a reason that students drop out is that they do not perceive a program as 

being effective, or that the factor that leads to nonparticipation—such as low motivation—would lessen 

the likelihood of the student being helped). Thus, though it is important if some students are not helped 

through SSS, it would not necessarily be a complete failure if some students are helped and some are not, 

particularly if the resources are concentrated on those who are helped.

The question of how to measure participation is important because of how statistical 

significance is calculated. If some students benefit from SSS and other students do not, then the average 

benefit across all students may or may not be statistically significant when a dichotomous measure of 

participation is used, depending on the size of the benefit, the distribution of any benefits across the SSS 

population, and the number of students in the study. Estimates of the average benefit also would not 

provide any information on the differential effects of SSS. On the other hand, if there is an a priori means 

of predicting which students would benefit, and separate tests are run for each group, then the added 

precision is more likely to result in statistically significant findings for those students who benefit.14 Two 

such methods are examined in this report: differentiating among students based on the amount of services 

they received, and also on the specific types of services they received. Potentially these two measurement 

tools can provide more precise information about which students and how students are helped.

In order to address these conflicting needs to consider dropouts as potential failures of the 

program but also to test for differential effects among students, this study uses multiple statistical 

measures of participation to provide multiple perspectives. More specifically,

 The study uses both dichotomous (yes or no) measures of participation that group all 

SSS participants together and continuous measures that differentiate among students 

in terms of the amount and types of services they received.

 Regardless of the measure used, average estimates of the total effect always include 

all SSS participants, including those who might be considered as dropouts.

 

14 In statistical analyses, it is possible (but incorrect) to first identify those students with superior outcomes, and then calculate 

“impacts” just for these students. This approach could result in making even random differences in outcomes appear to support 

findings of positive and statistically significant outcomes. Thus, it is critical not to separate students based on their outcomes 

but rather to use prespecified and independent measures of participation. Differentiating among students based on the amounts 

and types of services they have received is one such a priori approach and has been part of this study since the initial design 

phase, when the decision was made to collect such information on the students.
21

 Some measures in this report provide estimates for those students who a priori might 

be considered to be most likely to have improved outcomes based on their 

participation in SSS.

However, the use of continuous measures of participation based on the amount of services 

received does create a risk of conflating participation with motivation or need (or other factors that might 

influence participation). This topic is discussed in greater detail in the section that immediately follows.

Separating motivation and need from participation. To the extent that students’ levels of 

participation are based on student characteristics that themselves may be correlated with student 

outcomes, there is a risk that the measure of participation may serve as a surrogate for these other factors, 

and that any statistical association that is found between participation and student outcomes actually is 

due to these other factors and not participation in SSS itself. For example, students may participate more 

if they are highly motivated to succeed in their education, and this high degree of motivation might be 

associated with improved student outcomes. Alternatively, students may participate more if they had a 

high degree of academic need, and that neediness might be associated with poor student outcomes. These 

factors work in opposite directions but either could lead to misleading statistical results. A high 

association between motivation and participation could result in making participation highly correlated 

with positive student outcomes, thus overstating the effects of participation on student outcomes. 

Alternatively, a high association between academic need and participation could result in making 

participation highly correlated with poor student outcomes, thus understating the effects of participation 

on student outcomes (or, at the extreme, resulting in a negative association between participation and 

outcomes, so that it would appear that participation is associated with reduced student outcomes).

To investigate this issue, we attempted to predict students’ levels of participation using a 

variety of statistical models:

 Logistic regressions to predict whether or not a student is a participant in SSS;

 Multiple regression to predict the total number of hours of services received; and

 Multiple regressions to predict the number of hours of services received for each 

specific service.

We found that participation is a complex phenomenon that is related both to measures associated with 

academic need and measures associated with motivation. We also found that there is no single propensity 

to participate that works uniformly across all types of services, but rather that participation in one type of 

service often shows little or no relationship to participation in another (exhibit 2-6). 
22

Exhibit 2-6. Correlation of the number of hours received of each SSS service as freshmen in 1991–

92 with the number of hours received each other SSS service in the same year

Service

Instructional

courses

Professional 

tutoring

Peer 

tutoring

Professional 

counseling

Peer 

counseling

Workshops Labs Cultural 

events

Services 

for 

disabled

Instructional 

courses -- 0.08 -0.01 0.28 -0.12 -0.07 0.23 0.03 -0.04

Professional 

tutoring 0.08 -- 0.40 0.22 0.29 0.13 0.24 0.09 0.19

Peer 

tutoring -0.01 0.40 -- 0.22 0.36 0.05 0.13 0.12 0.10

Professional 

counseling 0.28 0.22 0.22 -- 0.01 0.15 0.57 0.32 0.02

Peer 

counseling -0.12 0.29 0.36 0.01 -- -0.02 0.06 0.08 0.01

Workshops -0.07 0.13 0.05 0.15 -0.02 -- 0.23 0.33 0.10

Labs 0.23 0.24 0.13 0.57 0.06 0.23 -- 0.23 0.04

Cultural 

events 0.03 0.09 0.12 0.32 0.08 0.33 0.23 -- 0.23

Services for 

disabled -0.04 0.19 0.10 0.02 0.01 0.10 0.04 0.23 --

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.

Based on these findings, students’ propensity to participate is not a monolithic force, and to 

the extent that measures of participate act as surrogates for these other factors, it is not clear whether there 

would be a bias or the direction of that bias.

To lessen the potential for bias, this study used several procedures:

 One of the measures of participation is dichotomous, so that SSS participants with 

high propensities to participate will not be treated separately from SSS participants 

with low propensities. (However, even a dichotomous measure can be subject to bias 

if there are different propensities between SSS and non-SSS students.)

 Multivariate models were used to adjust for students’ demographic differences, 

academic background, and attitudes.

 Separate propensity measures were created and included in some multivariate models 

to further adjust statistically for differences in the propensity to participate. 

Services received outside of SSS. One of the complicating factors for this study is that 

institutions also offer supplemental services through other means than SSS. In fact, one SSS program 

may offer a specific service through SSS, while another institution refers students to an equivalent 

program outside of SSS. Further, non-SSS students often receive services that are equivalent to those 

provided in SSS. In the longitudinal sample for this study, 91 percent of the SSS students and 53 percent 

of the comparison students received services outside of SSS. 

One potential approach to the outside services is to ignore them, based on the logic that the 

purpose of the evaluation is to determine what extra difference is made by SSS. However, then the study 
23

would not be measuring the effects of SSS (versus not receiving services) but rather the comparative 

effect of SSS relative to other supplemental services, which would be risky. For example, this approach 

in a medical setting could lead to saying that one antibiotic is not effective because its cure rate for strep 

throat is little different from that of another while, in fact, both may be effective. Thus, failing to account 

for alternative treatments could lead to incorrect inferences. Presumably, policymakers do not expect 

that, say, peer tutoring offered through SSS must be superior to peer tutoring offered through other 

sources but, rather, are satisfied if peer tutoring (or any other SSS service) offered through SSS is 

effective. Thus, to accurately assess the association between SSS participation and student outcomes, it is 

necessary to adjust for the receipt of other services.

All of the models in this analysis therefore adjust in some way for students’ receipt of 

supplemental services outside of SSS. The method that is used depends on how SSS participation is 

measured in order to provide a consistent approach. When dichotomous measures of SSS participation 

are used, a dichotomous measure of non-SSS participation is also used, and when multiple measures of 

SSS participation are used, multiple measures of non-SSS services are also used. Unfortunately, because 

essentially all of the SSS participants also received non-SSS services, the models using dichotomous 

measures are not very powerful with respect to distinguishing effects received through SSS from those 

received from supplemental services in general. There are two main alternatives, but each encounters the 

same difficulty:

 The measure of participation in non-SSS services could be assigned a value of 1 for 

all comparison students receiving such services, and 0 for all other comparison 

students and also for all SSS students. If supplemental services have positive effects 

on student outcomes, then both the measure of SSS services and this measure of nonSSS services should be positive and statistically significant. However, because the 

measure of non-SSS services would not include any non-SSS services received by 

SSS students, the measure of SSS services would include the average effects of the 

non-SSS services as well as the effects of the SSS services. The measure therefore 

would tend to overstate the effects of the SSS services. If there is a sizable difference 

between the effects for SSS students and non-SSS students receiving supplemental 

services, one might be tempted to infer that the added benefit was associated with SSS 

participation; however, the measure would not be adequate to prove this because there 

would be no way of separating the SSS and non-SSS services received by the 

participant.15 Similarly, if the estimated effects are basically equivalent, one might be 

tempted to infer that both groups received an equivalent package of services, but the 

model would not indicate whether the SSS services were an important part of the 

package. 

 Alternatively, the measure of non-SSS supplemental services could be assigned a 

value of 1 for both SSS and non-SSS students who received services outside of SSS, 

and 0 otherwise. This approach has the advantage that the measure of SSS services 

should in principle show only the effects of SSS and not of the other services. If a 

substantial number of SSS students did not receive non-SSS services, such a measure 

might be successful. However, because 91 percent of SSS students also received nonSSS services, the two measures of services are largely identical for SSS students, and 

 

15 One cannot assume that non-SSS services have the same effect on SSS students as on non-SSS students because the measure 

does not indicate whether both groups received equivalent amounts of non-SSS services. 
24

the statistical model is unlikely to be able to differentiate between the two sources of 

services. Suppose, for example, that both SSS students and those non-SSS students 

receiving supplemental services ultimately received the same amount of services, 

though from different sources, and therefore showed the same total effect with regard 

to the supplemental services. Because the single measure of non-SSS services would 

largely be effective in identifying almost all students who received supplemental 

services, it would tend to show the full value of all supplemental services combined, 

while the SSS variable would show no added benefit of SSS services. The lack of 

differentiation between the two measures for SSS students would probably result in 

the SSS measure being statistically insignificant, even if the SSS services made a 

positive difference.

For this report, the second alternative was chosen because it potentially provides a greater ability to 

differentiate between SSS and non-SSS services. However, it is likely to understate the effects of SSS 

because some of effect is likely to instead be captured through the other measure of supplemental 

services.

By contrast, when multiple measures of SSS participation (and non-SSS participation) are 

used, there is a much greater ability to differentiate among students. For example, while 91 percent of 

SSS students received some non-SSS supplemental service, the percentage receiving non-SSS peer 

tutoring (or whatever service being considered) is much smaller. As a result, there is a greater ability to 

differentiate between the effects of SSS and non-SSS services.

Services received after the first year. To reduce burden on the institutions, the SSS 

projects only collected detailed information on students’ service contacts during the first year. Thus, all 

information that is available on services received in later years is based on students’ responses to the 

questionnaires. Students were not asked about the source of those services, both to reduce the 

questionnaire length and complexity and because students were not expected to know.16 Thus, while this 

study is able to examine whether supplemental services after the first year are associated with improved 

student outcomes, it is not able to examine the effects of SSS services in later years. 

As with both the SSS services and the first-year services received outside of SSS, the 

treatment of these services varies depending on the particular model being used. When participation in 

SSS is treated as a dichotomous variable, then these services are also measured through a single separate 

variable for all services received after the first year.17 As with the measure of non-SSS first year services, 

there is substantial overlap between this measure and the other two measures of supplemental services: 

80 percent of SSS students received supplemental services, along with 82 percent of students receiving 

non-SSS first year services. Thus, if either this measure of later-year services or the measure of non-SSS 

first year services is positive and statistically significant, the variable may be capturing some of the SSS 

 

16 One reason is that students might think only of the services as being provided by the institution, rather than by a particular 

office within the institution, and another is that SSS projects sometimes create special institution-specific names for the 

projects, so that students might not know that the projects were SSS projects even if they remembered the specific source of the 

services.

17 Another option would have been to use a single dichotomous variable to account both for first-year non-SSS services and all 

later services. This latter option was not chosen because the other first-year services can be distinctly labeled as non-SSS 

services, while the services in later years cannot, and also because such a large proportion of students would have received 

either a first-year non-SSS service or a later year supplemental service that the variable would provide little capacity to 

distinguish one student from another.
25

effect, possibly making the measure of SSS services statistically insignificant. If so, the lack of statistical 

significance for the SSS variable would not necessarily indicate that SSS had no effect but, rather, that no 

additional benefit of SSS could be identified beyond that benefit that was already captured in the other 

two measures of supplemental services.

Especially when the dichotomous measure of non-first-year services is used, there is some 

risk that the variable acts as a surrogate for other factors that may be correlated with academic success. 

For example, the fact that a student received services after the first year necessarily requires that the 

student “persisted” for at least one year.18 Among the combined group of SSS and non-SSS students in 

the study, 92 percent of those who persisted also received at least one supplemental service in those later 

years, making these two measures largely similar. Alternatively, the variable might also be considered a 

measure of academic need and motivation because it applies to students who have chosen to receive

supplemental services. Still another possibility is that to the degree the measure is related to persistence, 

it may also be modeling some of the effects of SSS participation rather than allowing the SSS measure to 

show all SSS-related effects; this might be an issue mostly when examining retention, and the estimate of 

the SSS effect might therefore be an underestimate. 

The problem is especially an issue for the dichotomous models because then the measures of 

persistence and receipt of supplemental services are the most similar. When the measure of supplemental 

services is divided into multiple continuous measures, the greater amount of variation provides a better 

capacity for relating specific services to variations in student outcomes. As an adjustment in all of the 

models (i.e., using dichotomous measures and using multiple continuous measures of participation), an 

additional dichotomous variable is added that measures whether students attended higher education at any 

time during the five years after 1991–92. Thus, the changes in outcomes that are associated with 

persistence are measured through this variable, while the measure of the receipt of supplemental services 

will only capture any additional changes in student outcomes that are associated with the receipt of 

supplemental services. 

A weakness of this approach, however, is that the model is still somewhat recursive: SSS 

participation may affect persistence, which in turn may affect the student outcome being examined, so 

that the total effect associated with participation in SSS may be some combination of the estimate for SSS 

participation plus some component of the estimate for persistence.19 Thus, the models may tend to 

understate the effects of SSS. Conceivably, one could create a latent variable to separate that component 

of persistence that is associated with participation in SSS from the level of persistence that would appear 

in any case; however, given the wide overlap between the three measures of supplemental services (firstyear SSS services, first-year non-SSS services, and later services from any source), the dichotomous 

models would not be well designed for measuring the effects of SSS even after creating the latent 

variable. The dichotomous models are primarily useful for testing whether supplemental services (of 

some kind) are associated with improved student outcomes, while the more precise measures of 

participation used elsewhere in this report are the best for estimating the effects specifically associated 

with participation in SSS. 

 

18 This is a liberal definition of persistence because it includes students who stopped out for a time and then later returned to 

postsecondary education. It is therefore not the same as retention to the second year.

19 The problem would remain even if the measure of persistence is omitted because the measure of services received in later years 

would then have much the same effect. The only way to completely avoid this issue is to avoid any reference to services 

received in later years, but that would create a different problem: SSS participants would be compared to other students as if 

the other students had not received supplemental services, and the estimated effects of SSS (for some students) would therefore 

be the estimated difference between SSS and other services rather than the estimate difference between SSS and no services.
26

The three types of services (first-year SSS services, other first-year services, and later 

services) all show substantial overlap, especially when measured through dichotomous variables. In fact, 

only 2 percent of SSS students did not receive some type of supplemental service besides first-year SSS 

services. 

Because of the inability to distinguish between SSS and non-SSS services received after the 

first year, parts of this report focus on the more general question of whether the receipt of supplemental 

services is associated with improved academic outcomes. This general question has policy relevance for 

SSS. First, if supplemental services are found to improve student outcomes, then it is logical to assume 

that supplemental services provided through SSS also improve student outcomes. That is, there is no 

clear reason why services provided outside of SSS should be superior to SSS. If anything, given that 

institutions must win the SSS grants through a competitive process while the other services may not go 

through such a process, there are reasons to expect the SSS services to have better quality control. 

Second, to the extent that later-year services are found to be important, that information could lead SSS 

programs to increase the level of services provided in later years, thus potentially increasing the effects of 

SSS. Third, because part of the function of SSS is to refer students to needed services even if those 

services are outside of SSS, the effects of those other services is relevant to SSS and in some ways might 

be attributed to SSS. In fact, SSS students received a higher level of services than the comparison group 

both in the first year and also in next two years (Exhibit 2-7). The rates of receiving services were 

roughly comparable for both groups after 1993–94, except that the SSS students received more tutoring. 

Unfortunately, the study did not provide data on why the SSS students received more services (e.g., 

whether it was through the influence of SSS or for other reasons such as greater academic need).

Exhibit 2-7. Percent of SSS and non-SSS students receiving services in the six-year period starting 

with freshman entry in 1991–92

Non-SSS SSS

First-year services from any source

Services for disabled 1 8

Counseling services 50 93

Classroom instruction 20 74

Cultural events 16 34

Tutoring 25 73

Services for limited-English ability 1 6

Student orientation 82 85

Referrals to agencies/resources 12 26

Services after the first year

Counseling in 1992–93 41 48

Tutoring in 1992–93 18 30

Counseling in 1993–94 37 40

Tutoring in 1993–94 12 18

Tutoring after 1993–94 30 38

Advising/counseling after 1993–94 51 51

Services for disabled after 1993–94 2 5

Field trip after 1993–94 23 20

Contact with support services after 1993–94 50 48

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
27

Hierarchical clustering of data. Both the SSS participants and comparison group students 

were chosen by first selecting the institutions and then selecting the applicable students in the institutions. 

Though highly efficient, this selection method violates the classical statistical assumption that each 

student observation is independent of the others. To some degree, the inclusion of variables at the 

institution level within the regression models helps to adjust for such clustering. Still, the nesting of 

students within schools creates the potential that standard multivariate regression may not properly 

estimate the variances. This report therefore used both standard multivariate regression and HLM 

modeling so that readers may compare the results using both approaches.

To develop the HLM modeling approach, we first used HLM to examine several preliminary 

multivariate models, using a variety of measures and models. HLM was not able to estimate a solution 

unless most of the variables were treated as fixed. We therefore used a stepwise procedure of adding 

variables to test for each variable whether including a residual provided significant benefits. Based on 

these tests, we determined that for all HLM models for this report, only the intercept would not be treated 

as fixed. All of the independent variables were centered around the grand mean in order to increase the 

stability of the model. The HLM software that we used does offer an option for limited dependent 

variables, but this approach generally did not converge to a solution. Thus, ordinary HLM models were 

used instead for all outcome variables, whether or not they had limited dependent variables.

Number of cases. As noted, the longitudinal study was of 2,900 SSS participants and 2,900 

comparison students. Because of nonresponse, the actual database for examining six-year student 

outcomes consisted of 5,055 students, with 2,380 SSS participants and 2,675 comparison students. 

However, as shown in Exhibit 1-1 the number of students receiving any particular service varied widely, 

and the receipt of some services was relatively rare. The wide variations occurred for a variety of 

reasons: some services were not relevant for all students (especially services for the disabled), institutions 

varied in the packages of services they provided, and students could choose which services they received. 

Because of these variations, and because the amounts of services received often were also small (so that 

only small effects were likely), modeling the effects of specific supplemental services sometimes was 

hampered by the lack of adequate variation. Sometimes the finding of statistically significant results for 

one service and insignificant results for another may be based more on the numbers of students receiving 

each service rather than indicating that one service was more effective than another. The small number of 

students receiving some services could also be a reason why general measures of supplemental services 

(that do not distinguish between SSS and non-SSS students) would perform better than measures of SSS 

services, because the general measures would have roughly twice as many students receiving services. It 

especially imposed limitations on those analyses that were limited students at two-year institutions; that 

is, models to predict transfers from two-year to four-year institutions and to predict degree completion at 

two-year institutions were tested using substantially fewer students and the models performed less well. 

Many variables that were statistically significant when all students were included were not significant 

when restricted to students at two-year institutions, and the models did not always converge to a solution. 

Exhibit 2-8 shows the number of students receiving each SSS service for students in the dataset.
28

Exhibit 2-8. Number of students receiving SSS services in 1991–92 in database used for outcomes 

analysis

Category of student

Instructional 

courses

Professional 

tutoring

Peer 

tutoring

Professional 

counseling

Peer 

counseling

Labs

WorkshopsCultural 

events

Services 

for the 

disabled

All students 746 384 1,082 1,892 318 390 723 226 63

At two-year

institutions 110 56 250 415 23 157 90 8 11

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010.

Presentation of effect estimates. The regression estimates are presented here in two 

different formats: as regression coefficients to aid researchers to understand the structure of the models 

and to replicate the results, and as effect estimates to aid policymakers in applying the findings, because

the regression estimates often do not translate easily into effects. (One difficulty is that for some models, 

one must know the amount of services that students received in order to interpret the regression 

coefficients; another is that when logistic regression is used, neither the regression coefficients nor the log 

odds ratios are easy to interpret.)

Normally, estimates of program effects would be presented by indicating both what students 

would have achieved if they did not participate in SSS and what they did achieve after participating in 

SSS. Unfortunately, because the comparison group was more advantaged than the SSS participants, and 

since students in the comparison group often also received supplemental services, there is no pure 

measure of what outcomes would be expected if students had not participated in SSS. There are actual 

measures of final student outcomes after six years, but the alternative outcomes if students had not 

participated in SSS could only be estimated through statistical models. Moreover, because this report 

uses multiple models as a tool for examining the implications of the methodological choices involved, 

there are multiple estimates of the effects of SSS from which to choose. The results across the various 

models are often highly consistent but not identical. Therefore this report presents only the estimated 

effects of the services along with the final measured outcomes. Readers can calculate what students 

would have achieved without SSS by first deciding which model appears most trustworthy, and then 

subtracting the estimated effect from the final outcomes.

Negative and statistically significant findings. In some cases, SSS or other supplemental 

services showed negative but statistically significant findings. As a general rule these findings seem 

implausible, though there are situations in which negative findings could be the result of interactions 

between variables. 

 For example, if SSS helps students to graduate, this could lead to their no longer being 

enrolled in higher education; thus, one might simultaneously find positive effects on 

degree attainment and negative effects on retention. To handle this particular 

situation, this report primarily focuses on a composite measure (i.e., students either 

received a degree or were retained in higher education) as the best measure in this 

area. 

 Similarly, in the short term it is not clear how to interpret the relationship between 

SSS and employment; for example, those students who dropped out of higher 

education might be the ones who have had the most time to get established in their 

careers, possibly leading to higher employment rates and higher salaries. Thus, again 

a positive effect on retention could be correlated with an apparent negative effect on 
29

employment. Because of the difficulty of interpreting employment outcomes in the 

short term, and because employment outcomes are not specifically a goal of SSS, this 

report does not look at such outcomes. 

 More generally, negative findings can be the result of differences between the SSS 

and comparison groups. To the extent that the SSS students were more 

disadvantaged, one would expect them to have worse academic outcomes unless they 

receive supplemental services that compensate for their disadvantages. If the 

statistical models do not adequately adjust for these disadvantages and the effects of 

SSS are not sufficiently large to make up for the differences between the two groups, 

then one could find negative correlations between SSS participation and student 

outcomes. In earlier analyses, this particularly appeared to be a factor with regard to 

first-year professional counseling received by SSS students because of the way in 

which it was administered. While some models in this study do show positive 

outcomes from counseling (especially for counseling that is not specifically associated 

with SSS), first-year counseling through SSS often showed negative relationships to 

student outcomes. The reason appears to be that some SSS institutions proactively 

brought in students for counseling if they were doing badly in their academics; this 

action had the statistical effect of making high participation in counseling correlated 

with poor academic outcomes. 

All three of these explanations could result in underestimating the positive effect of SSS, either because 

of a failure to model a positive contribution from SSS (such as improved retention) or a failure to fully 

adjust for students’ disadvantages.

Of course, one could generate alternative hypotheses as to why SSS might be harmful. For 

example, the tutors could be so poorly qualified that they teach incorrect material, or the sessions could be 

so ineffective that they take time away from more constructive uses of students’ time. These latter 

explanations seem implausible, though, both because the institutions won competitive SSS grants based 

on the quality of their proposals, and because the students would be unlikely to keep returning for 

additional sessions if they found the sessions not to be helpful (or even counterproductive). 

For completeness, and so that readers may be aware of such indicators of inadequacies in the 

models, all negative regression coefficients and low odds ratios (i.e., odds ratios below 1.0) for SSS 

variables are included in the tables if they were statistically significant. However, because they are not 

considered reliable indicators of the effects of SSS, they are not included in the effect estimates. That is, 

the statistical findings are always presented through a combination of two tables, with the first showing 

regression coefficients and odds ratios and the second showing effect estimates. The negative findings 

are included in the first, but not in the second. For the third-year report, we also examined alternative 

models in which the SSS variables with negative coefficients were dropped from the models to see if the 

estimates for the other SSS variables would be affected; both types of models produced similar results, 

suggesting that the dropping of the negative coefficients does not artificially result in high effect 

estimates.
30
31

3. Research Findings Based on Dichotomous

Measures of Participation

Highlights of Findings

 Exhibit 3-1 summarizes the major results from the statistics presented in this chapter. 

There were consistent and statistically significant improvements in students’ academic 

outcomes if they received supplemental services in general, whether using regression 

or HLM analysis. By contrast, the regression analysis showed positive effects from 

SSS participation only with regard to cumulative GPA, and the HLM analysis, which 

better accounts for the clustering of students in institutions, showed no effects that are 

specifically associated with SSS. However, the dichotomous models lack sufficient 

power to discriminate among the different types of supplemental services, and the 

effects that were identified for supplemental services may include effects from SSS. 

Exhibit 3-1. Summary of findings for each outcome using dichotomous measures 

of participation, by method used: Six-year outcomes after freshman 

entry in 1991–92

Regressions HLM

First-year services First-year services Outcome measure

SSS Other

Later 

services SSS Other

Later 

services

GPA ■ ■ ■

Credits ■ ■

Retention or degree completion at 

the same institution

 All institutions ■ ■

 Four-year institutions ■ ■

 Two-year institutions

Retention or degree completion at 

any institution

 All institutions ■ ■

 Four-year institutions ■ ■

 Two-year institutions ■ ■

Bachelor’s degree or higher ■ ■

Associate’s degree or higher

 All institutions ■ ■

 Two-year institutions ■ ■ ■

Transfers to four-year institutions ■ ■

■=Positive and statistically significant; ─ =Negative and statistically significant.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of 

Student Support Services (SSS): Examination of Student Outcomes After Six Years, 2010.

 The average total amount of improvement from the HLM analysis that could be 

associated with the receipt of supplemental services (from any source) ranged from 12 

to 19 percentage points with regard to retention or degree completion, 8 to 10 

percentage points with regard to degree attainment, 16 percentage points with regard 

to transfers from two-year to four-year institutions, GPA increases of 0.14 on a 4.0 

scale, and an increase of 17 credits in the total credits earned over the six years. 
32

Introduction

This chapter starts the presentation of the research findings by examining the simplest 

method of modeling student participation in SSS: through a dichotomous (yes or no) measure of whether 

students participated in SSS.20 Such an approach has both advantages and disadvantages.

Advantages

 This method provides a single summary measure of the effects of SSS, and when 

multivariate linear regression is used, the applicable coefficient can be directly 

interpreted as the average overall effect among all SSS students.

 The “intent to treat” issue is resolved by treating all SSS participants in the same way, 

regardless of whether some students participated more or in different ways than 

others. Thus, if there are confounding factors such as motivation or academic 

neediness that affect students’ level of participation, those factors will not be allowed 

to give some SSS students greater weight than others.

Disadvantages

 This approach treats dosage as unimportant, as if all students are expected to 

experience the same effect regardless of which services they received or how much 

they received.

 Because only a single measure of SSS participation is used, and because the 

corresponding measures of alternative services are also dichotomous and highly 

interrelated with the receipt of SSS services, the statistical ability to distinguish the 

effects of SSS from the effects of other services is greatly lessened. Often it may only 

be possible to draw a conclusion about supplemental services in general, rather than to 

specifically estimate the association between SSS participation and student outcomes.

 The estimates of the SSS effect shown here may be underestimates because of a 

failure to account for the potential indirect effects of SSS through its effects on 

persistence after the second year. This is a greater issue when using dichotomous 

measures than it would be with more refined measures.

Limitations

 This approach to the “intent to treat” issue does not eliminate the need to statistically 

adjust for such factors as motivation and academic neediness. Because the SSS 

 

20 Though this is the simplest method of modeling SSS participation, it is not the simplest method overall. The simplest approach 

would be a comparison of the mean outcomes for SSS and non-SSS outcomes. However, because the SSS students were more 

disadvantaged than the comparison group, such results would be likely to be misleading. All of the statistics presented here are 

based on multivariate models in which student and institution characteristics (such as academic background, demographic 

characteristics, and student attitudes) are included to adjust for the differences between the two groups.
33

students tended to be more disadvantaged than non-SSS students (and possibly may 

also have differed in motivation as shown by their choice to participate), it is 

necessary to adjust for such characteristics anyway. 

The dichotomous measures were examined using two different statistical techniques: (1) 

multivariate and logistic regression, and (2) hierarchical linear modeling.

Multivariate and Logistic Regressions

Exhibits 3-2 and 3-3 present the findings using the dichotomous approach, with exhibit 3-2 

showing the regression coefficients or log-odds ratios, and exhibit 3-3 showing the estimated effects 

across all SSS students. In general, the exhibits show that the receipt of supplemental services was 

associated with improved student outcomes for every academic outcome but one (the one exception was 

retention at the same institution for students starting at two-year institutions). However, the findings were 

much more limited with regard to first-year SSS participation in particular. First-year SSS participation 

was associated with improvements in grade point averages and receipt of an associate’s degree or higher 

among students at two-year institutions but not with improvements on any of the other outcome measures. 

Because the three measures of supplemental services are highly interrelated, it is possible that some of the 

effects of first-year SSS participation may be captured through the other variables. Also, the measure of 

services after the first year does include SSS services as well as non-SSS services, so any effect of SSS 

participation after the first year would be included within this more general measure of participation. The 

following improvements were identified as being associated with the receipt of supplemental services:

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.11 (if they had received no supplemental services) to an 

actual 2.34, with 0.06 of that increase being associated with first-year SSS 

participation and the remainder with supplemental services received after the first 

year. 

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 53.5 credits (if no supplemental services were received) to 

an actual mean of 75.5. None of that increase could be specifically associated with 

first-year SSS participation.

 Retention or baccalaureate degree completion. SSS students were 12 to 18 

percentage points more likely to be either still in college during the sixth year or to 

have received a baccalaureate degree than would be estimated if they had not received 

services. (However, no statistically significant improvements were found for 

retention or degree completion at the same institution for students starting at two-year

institutions, possibly in part because of the reduced number of cases available for 

analysis.) For example, 22 percent are estimated to have remained at the same 

institution or completed a degree if they had not received supplemental services, 

compared with an actual 34 percent. None of the increases could be specifically 

associated with first-year SSS participation.
34

Exhibit 3-2. Regression coefficients concerning effects of SSS and non-SSS supplemental services, 

by outcome measure: Six-year outcomes after freshman entry in 1991–92

Outcome measure SSS participation in 

first year

Non-SSS supplemental 

services

Services after the 

first year

Cumulative (six-year) GPA 0.063** -- 0.178**

Cumulative (six-year) credits earned -- -- 21.967**

Retention or degree completion (combined)1

 Same institution

 All institutions -- -- 2.271**

 Four-year institutions only -- -- 2.523**

 Two-year institutions only -- -- --

 Any institution

 All institutions -- -- 3.090**

 Four-year institutions only -- -- 4.074**

 Two-year institutions only -- -- 3.861**

Degree attainment1

 Bachelor’s degree or higher -- -- 2.135**

 Associate’s degree or higher

 All institutions -- -- 1.814**

 Two-year institutions only -- 1.603** 1.594*

Transfers from two-year to four-year institutions1 -- -- 4.177**

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown. 1

 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of 

completing a degree) can be expressed as an odds (e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio 

expresses the improvement in the probability that is associated with participation in SSS (e.g., if the odds ratio is 2, then doubling the odds of 1.5 

would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). A log-odds ratio that is less than 1 

reflects a negative association between the receipt of supplemental services and the outcome indicated.

NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 

models and alternative measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010.
35

Exhibit 3-3. Estimated effects of SSS and non-SSS supplemental services on SSS students, by 

outcome measure: Six-year outcomes after freshman entry in 1991–92

Outcome measure

Estimated 

outcome without 

supplemental 

services

Improvement 

associated with 

SSS participation 

in first year

Improvement 

associated with 

other services

Observed 

outcome

Cumulative (six-year) GPA 2.11 0.063 0.178 2.34

Cumulative (six-year) credits earned 53.52 -- 21.967 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 22% -- 12% 34%

 Four-year institutions only 26% -- 16% 42%

 Two-year institutions only 8% -- 0% 8%

 Any institution

 All institutions 47% -- 16% 63%

 Four-year institutions only 52% -- 18% 70%

 Two-year institutions only 19% -- 16% 35%

Degree attainment

 Bachelor’s degree or higher 28% -- 10% 38%

 Associate’s degree or higher

 All institutions 41% -- 8% 49%

 Two-year institutions only 23% 7% 6% 36%

Transfers from two-year to four-year

institutions

8% -- 11% 19%

--Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the 

required p-value.

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may 

not sum to the total effect, due to the transformation used to estimate probabilities based on the logistic regression coefficients. This table 

presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative models and alternative 

measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010.

 Degree completion. SSS students were 8 to 13 percentage points more likely to have 

received either a bachelor’s degree or higher or associate’s degree or higher than if 

they had not received supplemental services. For example, 28 percent are estimated to 

have received a bachelor’s degree or higher if they had received no supplemental 

services, compared with 38 percent as the actual rate. None of the increases could be 

specifically associated with first-year SSS participation.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 8 percent (if no services 

had been received) to an actual 19 percent. None of that increase was specifically 

associated with first-year SSS participation.

Because the three types of supplemental services are so highly interrelated when measured through 

dichotomous variables, it is difficult to differentiate among them. One cannot be confident either that the 

two findings of positive and statistically significant relationships with first-year participation measure 
36

only SSS effects, or that the other measures exclude SSS effects. The safest conclusion is that 

supplemental services in general appear to be associated with improved student outcomes. 

Hierarchical Linear Modeling

As discussed in chapter 2, the variance estimates for the statistics in exhibit 3-1 may be 

incorrect due to a failure to account for the nesting of students within institutions, possibly leading to 

incorrect conclusions about the statistical significance of the findings. To examine this possibility, the 

regressions in exhibit 3-2 were rerun using HLM. Exhibit 3-4 presents the regression coefficients from 

the HLM runs, and exhibit 3-5 presents the average estimated effect on the various academic outcomes.

Exhibit 3-4. HLM regression coefficients concerning effects of SSS and non-SSS supplemental 

services, by outcome six years after freshman entry in 1991–92

Outcome measure SSS participation in 

first year

Non-SSS supplemental 

services

Services after the 

first year

Cumulative (six-year) GPA -- -- 0.177**

Cumulative (six-year) credits earned -- -- 22.206**

Retention or degree completion (combined)

 Same institution

 All institutions -- -- 0.151**

 Four-year institutions only -- -- 0.198**

 Two-year institutions only -- -- --

 Any institution

 All institutions -- -- 0.229**

 Four-year institutions only -- -- 0.213**

 Two-year institutions only -- -- 0.255**

Degree attainment

 Bachelor’s degree or higher -- -- 0.128**

 Associate’s degree or higher

 All institutions -- -- 0.103*

 Two-year institutions only -- 0.093* --

Transfers from two-year to four-year institutions -- -- 0.212**

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown.

NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 

models and alternative measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010.
37

Exhibit 3-5. Estimated effects of SSS and non-SSS supplemental services on SSS students, by 

outcome six years after freshman entry in 1991–92 (HLM analysis) 

Outcome measure

Estimated 

outcome without 

supplemental 

services

Improvement 

associated with 

SSS participation

Improvement 

associated with 

other services

Observed 

outcome

Cumulative (six-year) GPA 2.20 -- 0.136 2.34

Cumulative (six-year) credits earned 58.50 -- 16.991 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 22% -- 12% 34%

 Four-year institutions only 26% -- 16% 42%

 Two-year institutions only 8% -- 0% 8%

 Any institution

 All institutions 45% -- 18% 63%

 Four-year institutions only 53% -- 17% 70%

 Two-year institutions only 19% -- 16% 35%

Degree attainment

 Bachelor’s degree or higher 28% -- 10% 38%

 Associate’s degree or higher

 All institutions 41% -- 8% 49%

 Two-year institutions only 28% -- 8% 36%

Transfers from two-year to four-year

institutions

6% -- 13% 19%

--Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the 

required p-value.

NOTE: This table presents only the findings when dichotomous measures of SSS participation were used. See other chapters for alternative 

models and alternative measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): 

Examination of Student Outcomes After Six Years, 2010.

The HLM results indicate that no statistically significant improvements can specifically be 

associated with SSS participation on any of the student outcomes. Still, despite the lack of positive 

findings concerning SSS, the HLM results do indicate that supplemental services are associated with 

improved student outcomes: on two measures with regard to non-SSS services received in the first year 

(receiving an associate’s degree, when measured either across all institutions or only among two-year

institutions) and on 10 measures with regard to services received after the first year. Note, however, that 

because of the broadness of the dichotomous measures and the high degree to which they are interrelated, 

the findings should be interpreted as providing general support on the benefits of supplemental services 

rather than as indicating that one form of supplemental services is superior to another. Following are the 

specific estimates of the amount of improvement in outcomes that was associated with receiving 

supplemental services:

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.20 (if they had received no supplemental services) to an 

actual 2.34. None of that increase could be specifically attributed to first-year 

participation in SSS.

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 58.5 credits (if no supplemental services were received) to 
38

an actual mean of 75.5. None of that increase could be specifically associated with 

first-year SSS participation.

 Retention or baccalaureate degree completion. SSS students were 12 to 18 

percentage points more likely to be either still in college during the sixth year or to 

have received a baccalaureate degree than would be estimated if they had not received 

services. (The one exception is that no statistically significant improvements were 

found for retention or degree completion at the same institution for students starting at 

two-year institutions, possibly because of the reduced number of cases available for 

analysis.) For example, 22 percent are estimated to have remained at the same 

institution or completed a degree if they had not received supplemental services, 

compared with an actual 34 percent. None of the increases could be specifically 

attributed to first-year participation in SSS.

 Degree completion. SSS students were 8 to 10 percentage points more likely to have 

received either a bachelor’s degree or higher or associate’s degree or higher than if 

they had not received supplemental services. For example, 28 percent are estimated to 

have received a bachelor’s degree or higher if they had received no supplemental 

services, compared with 38 percent as the actual rate. None of the increases were 

specifically associated with first-year SSS participation.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 6 percent (if no services 

had been received) to an actual 19 percent. None of that increase was specifically 

associated with first-year SSS participation.

Summary

Though some minor differences appeared when using HLM rather than with multivariate 

and logistic regressions, both approaches largely produced similar results. The HLM results did not show 

positive and statistically significant associations between SSS participation and any of the 12 outcome 

measures, while the regression results showed such associations for only one of the 12 outcome measures. 

In general, the HLM estimates should be considered superior because they better account for the 

clustering of students within institutions. Both HLM and the regression results show positive and 

statistically significant associations for the same 11 academic outcomes when using general measures of 

supplemental services, and failed to show such associations for the remaining outcomes (i.e., retention or 

degree institution at the same institutions for students at two-year institutions only). The size of the 

estimated effect was highly similar using both approaches, with the total effect of all supplemental 

services typically varying by only 1 or 2 percentage points.

Both statistical approaches also present the same difficulty with regard to drawing 

conclusions about the effects of SSS. Because of the substantial overlap among students in receiving all 

three types of supplemental services (SSS services during the first year, non-SSS services received during 

the first year, and SSS or non-SSS services received during later years), some effects of SSS may be 

included within the measures of supplemental services. Also, the measure of services received after the 

first year (which is the measure that most consistently showed positive and statistically significant 

relationships) includes SSS services as well as non-SSS services. Thus, the results cannot properly be 
39

interpreted as indicating that SSS services do not show any effect while other supplemental services do. 

Rather, the best interpretation based on these results is that supplemental services in general have a 

positive effect, and we need more precise measures that better differentiate among the various types of 

services in order to estimate the specific contribution of each. 
40
41

4. Research Findings Based on Continuous Measures

of Participation

Highlights of Findings

 Exhibit 4-1 summarizes the major results from the statistics presented in this chapter. 

There were consistent and statistically significant improvements in students’ academic 

outcomes if they received supplemental services. Unlike when dichotomous measures 

of participation were used, academic improvements often were specifically associated 

with SSS participation. The HLM analysis, which better accounts for the clustering of 

students in institutions, showed effects on cumulative GPA, retention or degree 

completion at the same institution, retention or degree completion at any institution, 

and degree attainment.21

Exhibit 4-1. Summary of findings for each outcome using continuous measures 

of participation, by method used: Six-year outcomes after freshman 

entry in 1991–92

Regressions HLM

First-year services First-year services Outcome measure

SSS Other

Later 

services SSS Other

Later 

services

GPA ■ ■ ■ ■ ■ ■

Credits ■ ■ ■ ● ■ ■

Retention or degree completion at 

the same institution

 All institutions ■ ■ ■ ■

 Four-year institutions ■ ○ ■ ■ ○ ■

 Two-year institutions ● ■ ● ■ ■

Retention or degree completion at 

any institution

 All institutions ■ ■ ■ ■

 Four-year institutions ■ ■ ■ ■ ● ■

 Two-year institutions ■ ■ ■

Bachelor’s degree or higher ■ ■ ● ■

Associate’s degree or higher

 All institutions ● ■ ● ■

 Two-year institutions ■ ■

Transfers to four-year institutions ● ■ ● ■

■ Significant both with and without propensity scores; ● Significant only without propensity scores; 

○ Significant only with propensity scores.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support 

Services (SSS): Examination of Student Outcomes After Six Years, 2010.

 Using the HLM estimates, the average total amount of improvement that could be 

associated with the receipt of supplemental services (from any source) ranged from 15 

to 24 percentage points with regard to retention or degree completion across all 

institutions, 11 to 13 percentage points with regard to degree attainment across all 

institutions, 10 percentage points with regard to transfers from two-year to four-year

 

21 All of the estimates concerning employment outcomes should be considered preliminary because the time period may have 

been too short to properly reflect the effect of education on employment.
42

institutions, GPA increases of 0.16 to 0.17 on a 4.0 scale, and an increase of 16 to 18 

credits in the total credits earned over the six years. 

 Using the HLM estimates, first-year participation in SSS was correlated with the 

following improvements in student outcomes: an increase of 0.02 to 0.04 in the 

cumulative GPA, and increases of 4 to 9 percentage points in the likelihood of either 

remaining in college or completing a baccalaureate degree or higher across all 

institutions combined. For degree attainment, the HLM models were split on whether 

or not first-year participation in SSS was related to improvements in outcomes: 

models that included propensity scores showed no effect, while models that excluded 

propensity scores show improvements of 1 to 2 percent in degree attainment across all 

institutions. First-year participation did not show any association with improvements 

in transfers from two-year to four-year institutions. 

Introduction

Chapter 3 provided evidence that the receipt of supplemental services is associated with 

improved academic outcomes but at best only weak evidence concerning the effect of SSS. Still, the 

dichotomous measures of participation that were used in chapter 3 were not well designed to differentiate 

between SSS participation and the receipt of other supplemental services: once participation in 

supplemental services was known, no additional benefit could be found that was associated with 

participating in SSS services, perhaps because most or all of the SSS benefit was already captured 

through the other measures. In this chapter, the extent and nature of SSS participation are measured more 

precisely through the use of nine distinct measures of SSS services, each based on the number of hours of 

services received in that area. The nine types of SSS services examined are instructional courses that are 

specially designated for SSS students, professional tutoring, peer tutoring, professional counseling, peer 

counseling, labs, workshops, cultural events, and services for the disabled. 

In addition, two dichotomous measures describing the SSS programs are also used: whether 

the programs were home-based, and whether the SSS services were blended with other services. 

Similarly, just as first-year SSS participation is modeled through multiple variables, the receipt of other 

supplemental services is also modeled through multiple variables, though in most cases (except for some 

of the measures of tutoring and counseling) participation is measured through a dichotomous variable 

rather than quantifying the extent of participation. The use of multiple, continuous measures of SSS 

participation has both advantages and disadvantages.

Advantages

 The use of multiple, continuous measures provides a better capacity than the original 

dichotomous measures to differentiate between SSS services and other supplemental 

services. For example, while 91 percent of the SSS students received services outside 

of SSS, the correlations would tend to be much lower if one related a particular SSS 

service to a particular non-SSS service. Thus, there is a greater potential for 

measuring the effects specifically of SSS services rather than of supplemental services 

in general. 
43

 The resulting regression coefficients represent the improvements that are associated 

with specific services and thus provide a means of determining whether some services 

might especially be helpful.

 The statistical models are more realistic in that they take account of dosage, so that 

students who receive more services might be found to have different levels of 

improvement than students who receive few services. 

Disadvantages

 It is more difficult to determine the average effects across all SSS students because 

there is no single summary measure of participation that applies to all students. 

Because no service was received universally by all SSS students, any particular 

regression coefficient applies only to a subset of SSS students rather than to all of 

them. Because a student may receive more than one service, determining the total 

effect may also require summing multiple effects from individual services. Further, 

the total effect also depends on the amount of services received, so the regression 

coefficient must be multiplied by the amount of services rather than directly providing 

the average benefit from receiving the service. 

 Because the amount of services received is being modeled, there is the potential that 

confounding factors (such as motivation or academic neediness) may be associated 

with both the level of services and students’ academic outcomes, possibly creating 

false associations between the receipt of services and student outcomes. To explore 

the importance of this possibility and to adjust the estimates appropriately, the 

statistical models were estimated both with and without the use of separate propensity 

measures that were created to measure students’ likelihood of receiving services. In 

principle, the propensity measures should capture the importance of students’ 

tendency to receive services, while the measures of participation would then only be 

associated with any additional change in student outcomes beyond that change that is 

associated with the propensity to receive services.

Limitations

 It continues to be important to statistically adjust for such factors as motivation and 

academic neediness. Because the SSS students tended to be more disadvantaged than 

non-SSS students (and possibly may also have differed in motivation as shown by 

their choice to participate), it is necessary to adjust for such characteristics even when 

propensity measures are added to the models. 

 The estimates of the SSS effect shown here still may be underestimates because of a 

failure to account for the potential indirect effect of SSS through its effect on 

persistence after the second year. Though this is a greater issue when using the 

dichotomous measures in chapter 3 than with the more refined measures used here, it 

remains a risk.

Part of the structural design of this report is to use multiple research methodologies in order 

to allow readers to evaluate the implications of the various methodologies that might be applied. Thus, 

like chapter 3, this chapter presents the results of both multivariate and logistic regression, and 
44

hierarchical linear modeling—which should provide the most accurate estimates because HLM is better 

able to account for the clustering of students within institutions. In addition to presenting these two 

methodological alternatives, this chapter presents the results both with and without propensity scores 

added to the model. Propensity scores are used here to estimate the number of hours of a particular type 

of service that a student would take if that option were available (i.e., if the student were in SSS and the 

service were offered at that institution).22 

The purpose of including the propensity scores is to separate a student’s general tendency to 

receive a service from the actual effect of that service, because the tendency itself could be correlated 

with student outcomes (e.g., a high level of motivation might be associated with improved student 

outcomes, and a high level of academic need might be associated with lower student outcomes). 

However, determining the value of including propensity scores is more difficult than determining the 

value of using HLM. Clearly it is necessary in some way to account for factors such as students’ 

motivation and academic need, and otherwise the measures of participation may be conflated with these 

other factors. Still, it is arguable whether it is sufficient to adjust for these factors through the addition of 

variables measuring these student characteristics, or whether it is also necessary to adjust for these 

characteristics by adding propensity scores to the model. Both types of adjustments have similar 

purposes, so their simultaneous use may not be necessary. One reason for providing statistics both with 

and without propensity measures is to allow researchers to evaluate these changes to the statistical 

models.

Combining both of these methodological variations, the chapter is organized using the 

following structure, starting with the simplest models and advancing to the most complex:

• Multivariate and logistic regressions

– Without propensity scores

– With propensity scores

• Hierarchical linear modeling

– Without propensity scores

– With propensity scores

This chapter primarily seeks to describe the results for each of the four models, and compare 

how they are different, without coming to a conclusion about which approach provides the best estimates. 

A final synthesis of these results and those from chapter 3 is provided in chapter 5.

Multivariate and Logistic Regressions

This section presents the research results when using multivariate and logistic regressions. 

As indicated, multivariate and logistic regressions are here used in two ways: without the use of 

propensity scores, and with their use.

 

22 Propensity scores are also discussed in chapter 2 as part of the discussion on research methodology.
45

Without propensity scores

Exhibits 4-2 and 4-3 present the findings using multiple measures of participation and no 

separate measures of propensity to receive services, with exhibit 4-2 showing the regression coefficients 

or log-odds ratios and exhibit 4-3, the estimated effects across all SSS students. To simplify the 

presentation of the effect estimates, and because ED’s primary interest is in the net effect of SSS and 

other supplemental services, only the net effects are presented in exhibit 4-3. (One still can determine 

which particular first-year SSS services showed positive and statistically significant relationships by 

examining exhibit 4-2.) 

These measures differ substantially in their performance from the dichotomous measures 

used in chapter 3 with regard both to finding effects of supplemental services in general (statistically 

significant improvements in performance appeared for all of the 12 measures, compared with 11 of the 12 

measures in chapter 3) and especially the ability to differentiate SSS effects from general effects (separate 

first-year SSS effects appeared for 11 of the 12 measures, compared with 1 of the 12 when using the 

dichotomous measures). The improved performance of the SSS participation measures is consistent with 

the increased capacity to differentiate among the different types of services, while the dichotomous 

models had little ability to distinguish between SSS and non-SSS services. 

The following improvements were identified as being associated with the receipt of 

supplemental services:

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.09 (if they had received no supplemental services) to an 

actual 2.34, with 0.12 of that increase being associated with first-year SSS 

participation and the remainder with other supplemental services. 

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 56.7 credits (if no supplemental services were received) to 

an actual mean of 75.5. First-year SSS participation was associated with 2.6 credits of 

that increase, while the remainder was associated with the receipt of other 

supplemental services. 

 Retention or baccalaureate degree completion. SSS students were 13 to 21 

percentage points more likely (across all institutions) either to be still in college 

during the sixth year or to have received a baccalaureate degree than would be 

estimated if they had not received services. For example, 21 percent are estimated to 

have remained at the same institution or completed a degree if they had not received 

supplemental services, compared with an actual 34 percent. First-year SSS 

participation was associated with improvements of between 4 and 5 percentage points 

across all institutions. 

 Degree completion. SSS students were 11 to 14 percentage points more likely 

(across all institutions) to have received either a bachelor’s degree or higher or 

associate’s degree or higher than if they had not received supplemental services. For 

example, 24 percent are estimated to have received a bachelor’s degree or higher if 

they had received no supplemental services, compared with 38 percent as the actual 
46

Exhibit 4-2. Regression coefficients concerning effects of SSS services using continuous measures of SSS participation and no 

measures of propensity to receive services, by outcome six years after freshman entry in 1991–92

Outcome measure

Instructional 

courses

Professional 

tutoring

Peer 

tutoring

Professional 

counseling

Peer 

counseling

Labs Workshops

Cultural 

events

Services 

for the

disabled

Homebased

Blended

Cumulative (six-year) GPA -- -- 0.006** -0.019** -- -- -- -- -- 0.159** --

Cumulative (six-year) credits 

earned

-- -- 0.281** -1.176** -4.581** -- -- 1.763** -- -- --

Retention or degree 

completion (combined)1

 Same institution

 All institutions -- -- -- 0.929** -- -- 1.044* -- 1.048** -- 1.454**

 Four-year institutions only -- -- -- 0.926** -- -- 1.044* -- -- -- 1.466**

 Two-year institutions only -- -- -- -- -- -- -- -- -- 3.515* --

 Any institution

 All institutions -- -- 1.017** -- -- -- -- -- -- -- 1.649**

 Four-year institutions only -- -- 1.025** 0.923** -- -- -- -- -- -- 1.581**

 Two-year institutions only -- -- -- -- -- -- -- -- -- 3.641** 1.859*

Degree attainment1

 Bachelor’s degree or higher 1.004* -- 1.018** 0.930** -- 0.945** -- -- -- 1.440** --

 Associate’s degree or higher

 All institutions 1.003* -- 1.013** 0.941** -- 0.963** -- -- -- -- --

 Two-year institutions only -- -- -- -- -- -- -- -- -- -- --

Transfers from two-year to 

four-year institutions1

-- -- -- -- -- -- -- -- -- -- --

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown.

1

 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 

(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 

odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). The estimated effect of SSS is based on 

observed retention rates at the same institution of 34 percent across all institutions, and 43 percent across two-year institutions.

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 

services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 

This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 

of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
47

Exhibit 4-3. Estimated effects of SSS and non-SSS supplemental services on SSS students using continuous measures of SSS 

participation and no measures of propensity to receive services, by outcome six years after freshman entry in 1991–92

Outcome measure

Estimated outcome 

without supplemental 

services

Improvement associated 

with SSS participation in 

first year

Improvement associated 

with other services Observed outcome

Cumulative (six-year) GPA 2.09 0.121 0.131 2.34

Cumulative (six-year) credits earned 56.72 2.601 16.170 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 21% 4% 10% 34%

 Four-year institutions only 25% 5% 12% 42%

 Two-year institutions only 6% 1% 2% 8%

 Any institution

 All institutions 42% 5% 18% 63%

 Four-year institutions only 47% 5% 16% 70%

 Two-year institutions only 19% 6% 11% 35%

Degree attainment

 Bachelor’s degree or higher 24% 6% 3% 38%

 Associate’s degree or higher

 All institutions 38% 3% 8% 49%

 Two-year institutions only 31% 0% 5% 36%

Transfers from two-year to four-year institutions 12% 0% 7% 19%

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 

transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 

chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 

when using 0.05 as the required p-value.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
48

rate. First-year SSS participation was associated with an improvement of 6 percentage 

points with regard to completing a bachelor’s degree or higher, 3 percentage points 

with regard to completing an associate’s degree or higher at any institution, and no 

improvement with regard to completing an associate’s degree or higher at two-year

institutions.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 12 percent (if no 

services had been received) to an actual 19 percent. None of that increase was 

specifically associated with first-year SSS participation.

With propensity scores

Repeating the same structure used previously, but now using models that include propensity 

scores, exhibits 4-4 and 4-5 present the findings using multiple measures of participation: Exhibit 4-4 

shows the regression coefficients or log-odds ratios, and exhibit 4-5 shows the estimated effects across all 

SSS students. Like exhibits 4-2 and 4-3, these measures show more a more consistent association 

between the receipt of supplemental services and students outcomes than the dichotomous measures used 

in chapter 3, with regard both to finding effects of supplemental services in general (statistically 

significant improvements in performance appeared for all of the 12 measures, compared with 11 of the 12 

measures in chapter 3) and the ability to differentiate SSS effects from general effects (separate first-year 

SSS effects appeared for 8 of the 12 measures, compared with 1 of the 12 when using the dichotomous 

measures). The results for first-year SSS participation were somewhat weaker with propensity scores 

than without them, with statistically significant improvements appearing for 8 of 12 measures (rather than 

11 of 12), and fewer SSS services showing positive and statistically significant relationships with student 

outcomes. 

The following improvements were identified as being associated with the receipt of 

supplemental services:

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.05 (if they had received no supplemental services) to an 

actual 2.34, with 0.14 of that increase being associated with first-year SSS 

participation and the remainder with other supplemental services. 

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 57.0 credits (if no supplemental services were received) to 

an actual mean of 75.5. First-year SSS participation was associated with 2.4 credits of 

that increase, while the remainder was associated with the receipt of other 

supplemental services. 

 Retention or baccalaureate degree completion. SSS students were 2 to 28 

percentage points more likely either to be still in college during the sixth year or to 

have received a baccalaureate degree than would be estimated if they had not received 

services. For example, 18 percent are estimated to have remained at the same 
49

Exhibit 4-4. Regression coefficients concerning the effects of SSS services using continuous measures of SSS participation and 

adding propensity scores, by outcome six years after freshman entry in 1991–92

Outcome measure

Instructional 

courses

Professional 

tutoring

Peer 

tutoring

Professional 

counseling

Peer 

counseling Labs Workshops Cultural events

Services 

for the 

disabled

Homebased Blended

Cumulative (six-year) GPA -- -- 0.006** -0.036** -0.056* -- -- -- -- 0.195** --

Cumulative (six-year) credits 

earned

-- -- 0.366** -1.864* -4.098** -- -- -- -- -- --

Retention or degree completion 

(combined)1

 Same institution

 All institutions -- -- -- 0.908* -- -- -- -- -- 1.558** 1.369**

 Four-year institutions only -- -- -- 0.874* -- -- -- -- -- 1.687** 1.356*

 Two-year institutions only -- -- -- -- -- 0.925* -- -- -- -- --

 Any institution

 All institutions 0.995* -- -- -- -- -- -- -- -- 1.657** 1.605**

 Four-year institutions only -- -- -- 0.816** -- -- -- -- -- 1.860** 1.581**

 Two-year institutions only 0.989* -- -- -- -- -- -- -- -- 2.822* --

Degree attainment1

 Bachelor’s degree or higher -- 1.655** --

 Associate’s degree or higher

 All institutions -- -- -- -- -- -- -- -- -- -- --

 Two-year institutions only -- -- -- -- -- -- -- -- -- -- --

Transfers from two-year to fouryear institutions10.9834* -- -- -- -- -- -- -- -- -- --

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown.

1

 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 

(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 

odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). A log-odds ratio that is less than 1 reflects a 

negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 

services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 

This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 

of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
50

Exhibit 4-5. Estimated effects of SSS and non-SSS supplemental services on SSS students using continuous measures of SSS 

participation and adding propensity scores, by outcome six years after freshman entry in 1991–92

Outcome measure

Estimated outcome 

without supplemental 

services

Improvement associated 

with SSS participation in 

first year

Improvement associated 

with other services Observed outcome

Cumulative (six-year) GPA 2.05 0.143 0.144 2.34

Cumulative (six-year) credits earned 57.01 2.386 16.091 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 18% 7% 9% 34%

 Four-year institutions only 21% 9% 12% 42%

 Two-year institutions only 6% 0% 2% 8%

 Any institution

 All institutions 40% 8% 17% 63%

 Four-year institutions only 42% 9% 16% 70%

 Two-year institutions only 32% 2% 1% 35%

Degree attainment

 Bachelor’s degree or higher 24% 5% 10% 38%

 Associate’s degree or higher

 All institutions 40% 0% 9% 49%

 Two-year institutions only 29% 0% 7% 36%

Transfers from two-year to four-year institutions 11% 0% 8% 19%

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 

transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 

chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 

when using 0.05 as the required p-value.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
51

institution or completed a degree if they had not received supplemental services, 

compared with an actual 34 percent. First-year SSS participation was associated with 

improvements of between 7 and 9 percentage points at four-year institutions and 

across all institutions, and of 0 to 2 percentage points at two-year institutions. 

 Degree completion. SSS students were 7 to 14 percentage points more likely to have 

received either a bachelor’s degree or higher or associate’s degree or higher than if 

they had not received supplemental services. For example, 24 percent are estimated to 

have received a bachelor’s degree or higher if they had received no supplemental 

services, compared with 38 percent as the actual rate. First-year SSS participation 

was associated with an improvement of 5 percentage points with regard to completing 

a bachelor’s degree or higher, and no improvement with regard to completing an 

associate’s degree or higher.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 11 percent (if no 

services had been received) to an actual 19 percent. None of that increase was 

specifically associated with first-year SSS participation.

Hierarchical Linear Modeling

As suggested earlier, HLM is better designed to estimate variances than ordinary least 

squares regression when students are clustered within institutions. This section repeats the analysis in the 

previous section of the report, but using HLM.

Without propensity scores

Exhibit 4–6 presents the regression coefficients from the HLM runs when no propensity 

scores are included in the HLM models, and exhibit 4-7 presents the average estimated effects on the 

various academic outcomes. These results are quite similar to the results in the previous section of this 

chapter, though one outcome (retention or degree completion at any institution for students at two-year

institutions) did not show statistically significant relationships with the receipt of SSS. However, the 

results were very unlike the HLM results in chapter 3, with first-year SSS participation showing positive 

and statistically significant associations with 9 of the 12 outcomes (compared with 0 of 12 when using the 

dichotomous models). The specific findings were as follows:

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.17 (if they had received no supplemental services) to an 

actual 2.34. Of that increase, 0.04 was associated with first-year participation in SSS, 

and the remainder with the receipt of other supplemental services.

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 56.3 credits (if no supplemental services were received) to 

an actual mean of 75.5. Three credits of the 19-credit increase were specifically 

associated with first-year SSS participation.
52

Exhibit 4–6. HLM regression coefficients concerning the effects of SSS services using continuous measures of SSS participation and 

no measures of propensity, by outcome six years after freshman entry in 1991–92

Outcome measure

Instructional 

courses

Prof. 

tutoring

Peer 

tutoring

Prof. 

counseling

Peer 

counseling

Labs

WorkshopsCultural 

events

Services 

for the 

disabled

Homebased

Blended

Cumulative (six-year) GPA -- -- 0.006** -0.014* -- -- -- -- -- -- --

Cumulative (six-year) credits 

earned

-- -- 0.391** -0.887* -- -- -- 1.476** -- -- --

Retention or degree completion 

(combined)1

 Same institution

 All institutions -- -- 0.003** -0.011** -- -- -- -- 0.006** -- 0.064*

 Four-year institutions only -- -- 0.003** -0.012** -- -- -- -- -- -- 0.074*

 Two-year institutions only -- -0.016** -- -- -- -- -- -- 0.009** 0.085* 0.058*

 Any institution

 All institutions -- -- 0.003** -- -- -- -- -- -- 0.077** 0.065*

 Four-year institutions only -- -- 0.003** -- -- -- -- -- -- 0.059* 0.069*

 Two-year institutions only -- -- -- -- -- -- -- -0.061** -- --

Degree attainment1

 Bachelor’s degree or higher -- -- 0.002** -0.009* -- -- -- -- -- --

 Associate’s degree or higher -- -- -- -- --

 All institutions -- -- 0.002* -0.009* -- -0.005** -- -- -- -- --

 Two-year institutions only -- -- -- -- -- -- -- -- -- --

Transfers from two-year to fouryear institutions1-- -- -- -- -- -- -0.020* -0.039* -- -- --

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown.

1

 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 

(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 

odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). A log-odds ratio that is less than 1 reflects a 

negative association between the receipt of supplemental services and the outcome indicated. 

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 

services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 

This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 

of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
53

Exhibit 4-7. Estimated effects of SSS and non-SSS supplemental services on SSS students when using continuous measures and no 

propensity scores, by outcome six years after freshman entry in 1991–92 (HLM analysis) 

Outcome measure

Estimated outcome 

without supplemental 

services

Improvement associated 

with SSS participation in 

first year

Improvement associated 

with other services Observed outcome

Cumulative (six-year) GPA 2.17 0.039 0.136 2.34

Cumulative (six-year) credits earned 56.32 3.191 15.978 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 19% 5% 10% 34%

 Four-year institutions only 24% 5% 13% 42%

 Two-year institutions only 1% 4% 3% 8%

 Any institution

 All institutions 39% 9% 15% 63%

 Four-year institutions only 46% 9% 15% 70%

 Two-year institutions only 24% -- 11% 35%

Degree attainment

 Bachelor’s degree or higher 25% 2% 11% 38%

 Associate’s degree or higher

 All institutions 37% 1% 11% 49%

 Two-year institutions only 28% -- 8% 36%

Transfers from two-year to four-year institutions 9% -- 10% 19%

--Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p-value.

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 

transformation used to estimate probabilities based on the logistic regression coefficients. 

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
54

 Retention or baccalaureate degree completion. Overall, SSS students were 15 to 

24 percentage points more likely (across all institutions) either to be still in college 

during the sixth year or to have received a baccalaureate degree than would be 

estimated if they had not received services. For example, 19 percent are estimated to 

have remained at the same institution or completed a degree if they had not received 

supplemental services, compared with an actual 34 percent. Between 5 and 9 

percentage points of the increases were associated with first-year participation in SSS.

 Degree completion. Overall, SSS students were 12 to 13 percentage points more 

likely (across all institutions) to have received either a bachelor’s degree or higher or 

associate’s degree or higher than if they had not received supplemental services. For 

example, 25 percent are estimated to have received a bachelor’s degree or higher if 

they had received no supplemental services, compared with 38 percent as the actual 

rate. Between 1 and 2 percentage points of that increase were associated with firstyear participation in SSS.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 9 percent (if no services 

had been received) to an actual 19 percent. None of that increase was specifically 

associated with first-year SSS participation.

With propensity scores

Because of the potential that the level of participation in SSS might be confounded with 

factors such as motivation and academic need, the HLM models were also run with propensity scores 

included. Exhibit 4-8 presents the regression coefficients from the HLM runs, and exhibit 4-9 presents 

the average estimated effects on the various academic outcomes.

Similar to the regression results, the HLM results show that the receipt of supplemental 

services was associated with improved student outcomes for every academic outcome that was tested. 

However, the results were weaker with regard to outcomes that could specifically be associated with firstyear SSS participation: there was an association between such participation and improved outcomes with 

regard to cumulative GPA, and retention or degree completion (at four-year institutions and at all 

institutions combined), but no SSS effect was identified with regard to degree attainment (if examined 

alone and not in combination with retention), transfers from two-year to four-year institutions, and 

retention or degree completion at two-year institutions. The results with regard to first-year SSS 

participation were also weaker when compared with the HLM models that did not use propensity 

measures: they showed positive and statistically significant relationships to only 5 of the 12 outcomes, 

compared with 11 of 12 when no propensity scores were included, and typically only one SSS measure 

achieved statistical significance per outcome, rather than up to three measures as shown in exhibit 4–6. 

Still, these results were stronger than when dichotomous measures of participation were used (in chapter 

3), suggesting that the greater ability to differentiate among students was helpful in identifying the effects 

of services. 
55

Exhibit 4-8. HLM regression coefficients concerning the effects of SSS services using continuous measures of SSS participation 

with propensity scores, by outcome six years after freshman entry in 1991–92

Outcome measure

Instructional 

courses

Prof. 

tutoring

Peer 

tutoring

Prof. 

counseling

Peer 

counseling

Labs Workshops

Cultural 

events

Services 

for the 

disabled

Homebased

Blended

Cumulative (six-year) GPA -- -- -- -- -- 0.010* -- -- -- -- --

Cumulative (six-year) credits 

earned

-- -- -- -- -- -- -- -- -- -- --

Retention or degree completion 

(combined)

1

 Same institution

 All institutions -- -- -- -- -- -- -- -- -- 0.063* 0.060*

 Four-year institutions only -- -- -- -- -- -- -- -- -- 0.074* --

 Two-year institutions only -- -- -- -- -- -- -- -- -- -- --

 Any institution

 All institutions -- -- -- -- -- -- -- -- -- 0.077** --

 Four-year institutions only -- -- -- -- -- -- -- -- -- 0.072* --

 Two-year institutions only -- -- -- -- -- -- -- -- -- -- --

Degree attainment1

 Bachelor’s degree or higher -- -- -- -- -- -- -- -- -- -- --

 Associate’s degree or higher

 All institutions -- -- -- -- -- -- -- -- -- -- --

 Two-year institutions only -- -- -- -- -- -- -- -- -- -- --

Transfers from two-year to fouryear institutions1-- -- -- -- -- -- -- -- -- -- --

-- Not statistically significant.

*p<.05. **p<.01. Only outcomes that were statistically significant at the 0.05 level are shown.

1

 For these measures, logistic regression was used, and the statistics shown are log-odds ratios. A student’s probability or being retained (or of completing a degree) can be expressed as an odds 

(e.g., if the probability is 60 percent, the odds is 60/(100-60)=60/40=1.5). The odds ratio expresses the improvement in the probability that is associated with participation in SSS (e.g., if the 

odds ratio is 2, then doubling the odds of 1.5 would result in an odds of 3, which is equivalent to a probability of 75 percent (75/(100-75)=75/25=3). A log-odds ratio that is less than 1 reflects a

negative association between the receipt of supplemental services and the outcome indicated.

NOTE: In order to present the results compactly, only those variables that showed statistically significant relationships (either positive or negative) are shown. The complete list of SSS 

services that were tested is instructional courses, professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, cultural events, and services for the disabled. 

This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures 

of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
56

Exhibit 4-9. Estimated effects of SSS and non-SSS supplemental services on SSS students when using continuous measures and 

propensity scores, by outcome six years after freshman entry in 1991–92 (HLM analysis) 

Outcome measure

Estimated outcome 

without supplemental 

services

Improvement associated 

with SSS participation in 

first year

Improvement associated 

with other services Observed outcome

Cumulative (six-year) GPA 2.18 0.021 0.137 2.34

Cumulative (six-year) credits earned 59.22 0.0 16.265 75.49

Retention or degree completion (combined)

 Same institution

 All institutions 17% 6% 11% 34%

 Four-year institutions only 24% 5% 13% 42%

 Two-year institutions only 6% 0% 2% 8%

 Any institution

 All institutions 44% 4% 15% 63%

 Four-year institutions only 51% 4% 15% 70%

 Two-year institutions only 24% 0% 11% 35%

Degree attainment

 Bachelor’s degree or higher 27% 0% 11% 38%

 Associate’s degree or higher

 All institutions 38% 0% 11% 49%

 Two-year institutions only 28% 0% 8% 36%

Transfers from two-year to four-year institutions 9% 0% 10% 19%

NOTE: The separate estimates for the improvement associated with SSS participation and the improvement associated with other services may not sum to the total effect, due to the 

transformation used to estimate probabilities based on the logistic regression coefficients. This table presents only the findings when continuous measures of SSS participation were used. See 

chapter 2 and the summary in chapter 5 for the findings when using dichotomous measures of SSS. All effect estimates are based on regression coefficients that were statistically significant 

when using 0.05 as the required p-value.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
57

Following are the specific estimates of the amount of improvement in outcomes that was 

associated with receiving supplemental services: 

 Grade point averages. SSS students’ cumulative GPAs over six years were 

increased from an estimated 2.18 (if they had received no supplemental services) to an 

actual 2.34. Of that increase, 0.02 was specifically associated with first-year 

participation in SSS.

 Total credits earned. The total number of credits earned for six years was increased 

from an estimated mean of 59.2 credits (if no supplemental services were received) to 

an actual mean of 75.5. None of that increase could be specifically associated with 

first-year SSS participation.

 Retention or baccalaureate degree completion. SSS students were 17 to 19 

percentage points more likely (across all institutions) either to be still in college 

during the sixth year or to have received a baccalaureate degree than would be 

estimated if they had not received services. Between 4 and 6 percentage points of that 

increase was specifically associated with first-year SSS participation. 

 Degree completion. SSS students were 8 to 11 percentage points more likely to have 

received either a bachelor’s degree or higher or associate’s degree or higher than if 

they had not received supplemental services. For example, 27 percent are estimated to 

have received a bachelor’s degree or higher if they had received no supplemental 

services, compared with 38 percent as the actual rate. None of the increases were 

specifically associated with first-year SSS participation.

 Transfers from two-year to four-year institutions. The rate of transfer from twoyear to four-year institutions was increased from an estimated 9 percent (if no services 

had been received) to an actual 19 percent. None of that increase was specifically 

associated with first-year SSS participation.

Types of Services

Because the measures of supplemental services in this chapter were tied to specific services, 

it is possible to examine whether some types of services were especially likely to be effective. Exhibit 4-

10 shows which specific first-year SSS services were found to have positive effects for each outcome 

measure. Similarly, because the focus is on types of services, and a given type of service presumably 

could be effective whether offered through SSS or through other sources, exhibit 4-11 shows which other 

supplemental services were found to have positive effects for each outcome measure. For brevity, only 

those variables that were positive and statistically significant are shown.23 Also for brevity, only the 

findings from the HLM analyses are included, because these models should have the most reliable 

standard errors and thus provide the best test of whether a service had a significant effect. However, 

because deciding whether or not to include propensity scores is a less straightforward decision, the 

exhibits cover both methodologies, and show whether the statistical significance was based on either or 

both methodologies.

 

23 Statistical significance is based on the 0.05 level.
58

Some services stood out as being statistically significant both with and without the use of 

propensity scores:24 

 Home-based SSS programs (for two outcome measures); 

 Blended programs (for one outcome measure); 

 College re-entrance counseling (for one outcome measure);

 Cultural enrichment (outside of SSS) in the first year (for two outcome measures) or 

cultural trips in later years (for 10 outcome measures); 

 Referrals to resources in the first year (for one outcome measure);

 Counseling in 1993–94 (for 9 outcome measures), or in later years (for 11 outcome 

measures); 

 Tutoring in 1992–93 (for one outcome measure) or 1993–94 (for 7 outcome 

measures); or 

 Recent contacts with supplemental services (for 11 outcome measures). 

A few other services were statistically significant either one way or the other, but not both:

Significant only without the use of propensity measures:

 SSS peer tutoring

 SSS cultural events in the first year

 SSS services for the disabled in the first year 

 Non-SSS services in the first year for students with limited English ability

Significant only with the use of propensity measures:

 SSS labs in the first year

 Non-SSS services for the disabled in the first year

A few other patterns also are apparent in exhibits 4-10 and 4-11:

 Services received after the first year appear to be significantly related to long-term 

student outcomes much more frequently than services received in the first year.

 Among those services received in the first year, SSS services more frequently were 

statistically significant that non-SSS services.

 

24 The counts provided here are the counts of outcome measures that showed statistical significance both with and without the use 

of propensity measures. In addition, some of the variables listed below also were significant for other outcome measures using 

only one approach (i.e., either with or without the use of propensity measures but not for both for the same outcome measure). 
59

Exhibit 4-10. First-year SSS services that showed positive and statistically significant effects on student outcomes, by outcome six 

years after freshman entry in 1991–92

Outcome measure Peer tutoring Labs Cultural events

Services for the 

disabled

Home-based Blended

Cumulative (six-year) GPA ● ○

Cumulative (six-year) credits earned ● ●

Retention or degree completion (combined)

 Same institution

 All institutions ● ● ○ ■

 Four-year institutions only ● ○ ●

 Two-year institutions only ● ● ●

 Any institution

 All institutions ● ■ ●

 Four-year institutions only ● ■ ●

 Two-year institutions only

Degree attainment

 Bachelor’s degree or higher ●

 Associate’s degree or higher

 All institutions ●

 Two-year institutions only

Transfers from two-year to four-year institutions

■ Significant both with and without propensity scores; ● Significant only without propensity scores; ○ Significant only with propensity scores.

NOTE: Statistical significance is based on the 0.05 level. This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary 

in chapter 5 for the findings when using dichotomous measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
60

Exhibit 4-11. Other supplemental services that showed positive and statistically significant effects on student outcomes, by outcome 

six years after freshman entry in 1991–92

Non-SSS first-year services Later-year services

Outcome measure Services 

for the 

disabled

Limited 

English 

ability

College reentrance 

counseling

Cultural 

enrichment

Referrals 

to 

resources

Counseling in 

1992

Counseling in 

1993

Tutoring in 

1992

Tutoring in 

1993

Recent 

counseling

Recent 

trip

Recent 

contact

Cumulative (six-year) GPA ■ ■ ■ ■ ■ ■

Cumulative (six-year) 

credits earned

■ ■ ■ ■ ■ ■ ■

Retention or degree 

completion (combined)

 Same institution

 All institutions ■ ○ ■ ■ ■ ■

 Four-year institutions 

 only

○ ■ ■ ■ ■ ■

 Two-year institutions 

 only

■

 Any institution

 All institutions ■ ■ ■ ■ ■

 Four-year institutions 

 only

● ■ ■ ■ ■

 Two-year institutions 

 only

■ ■ ■ ■ ■

Degree attainment

 Bachelor’s degree or 

higher

■ ■ ■ ■

 Associate’s degree or 

higher

 All institutions ■ ■ ■ ■

 Two-year institutions 

 only

■ ■ ■

Transfers from two-year to 

four-year institutions

● ■ ■ ■ ■

■ Significant both with and without propensity scores; ● Significant only without propensity scores; ○ Significant only with propensity scores;

NOTE: Statistical significance is based on the 0.05 level. This table presents only the findings when continuous measures of SSS participation were used. See chapter 2 and the summary 

in chapter 5 for the findings when using dichotomous measures of SSS.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
61

 SSS peer tutoring in the first year is unique among first-year services in reaching 

statistical significance for 8 of the 12 measures, although only when propensity scores 

were not used.

 Home-based and blended SSS programs were the other SSS characteristics that were 

statistically significant for more than two student outcomes.

Summary

All of the models considered in this chapter models showed statistically significant 

improvements for the 12 student outcomes if students had received supplemental services. The measures 

of SSS participation, which were limited to the first year, performed less well than many of the other 

measures of supplemental services. This is consistent with a general pattern that the relationships were 

stronger for services received after the first year than for first-year services. (Students’ participation in 

SSS was not limited to the first year, but the data do not discriminate between SSS and non-SSS services 

after the first year.) Still, depending on the methodological approach that was used, between 5 and 10 

student outcomes were positively related to first-year SSS participation. 

The regression and HLM models differed primarily by the HLM models being less likely to 

show statistically significant relationships for first-year SSS participation: fewer outcomes showed such 

relationships, and fewer SSS services had positive and statistically significant coefficients. 

Fundamentally, however, the results were not greatly different. Perhaps the greatest difference was that 

the negative but statistically significant results that appeared in the regression models largely disappeared 

in the HLM models.

Much greater differences appeared when comparing the models with and without propensity 

measures. The models that excluded propensity measures showed a greater variety of first-year SSS 

services as being positive and statistically significant, and across a greater number of outcomes. In fact 

the models that included propensity measures primarily indicated the importance of home-based programs 

for selected student outcomes, rather than showing a mixture of services as affecting student outcomes. 
62
63

5. Summary and Conclusions

This chapter compares the findings from the four types of approaches used in chapters 3 and 

4 to determine which findings are consistently supported regardless of the approach, which findings 

depend on the particular approach being used, and the importance of the alternative methodological 

approaches. It then presents an overall conclusion about which findings (from our perspective) best 

describe the association between participation in SSS and student outcomes, and the implications of the 

findings for the SSS program.

Overview

The results varied depending on whether one focuses specifically on first-year SSS services 

or on supplemental services in general (exhibit 5-1). With regard to first-year SSS services, none of the 

12 outcome measures showed consistent positive findings across all six methodologies. This was 

primarily due to the poor performance of the SSS measures when using the dichotomous measures of 

participation, because none of the dichotomous HLM models and only one of the dichotomous regression 

models showed positive effects from SSS. The poor performance of the dichotomous models was 

expected because of the weaknesses of the dichotomous measures of participation. The remaining four 

models showed consistent positive effects for five outcome measures, mixed results for five measures, 

and no significant effects for two measures. However, even the models using continuous measures of 

SSS participation performed less well with regard to first-year SSS participation than they did for 

supplemental services in general. This was most noticeable when the propensity scores were included, 

with the HLM findings showing positive effects for 5 of the 12 student outcomes and the regression 

results showing positive effects for 8 of the 12 outcomes. First-year SSS participation did better when the 

propensity scores were excluded (i.e., with positive outcomes for 9 of the 12 outcomes in the HLM results 

and 10 of the 12 outcomes in the regression results), though still less well than other supplemental 

services (11 of 12 outcomes). Also, though the differences were often small, the regression results tended 

to result in higher effects for SSS services than did the HLM results (13 times versus 5 times in which the 

HLM effects were greater). 
64

Exhibit 5-1. Comparison of estimated effects of first-year SSS services using each of six methodologies: Six-year outcomes after 

freshman entry in 1991–92

Continuous measures of services

Dichotomous measures of services

No propensity scores With propensity scores

Regressions HLM Regressions HLM Regressions HLM

Outcome measure

Start Gain Start Gain Start Gain Start Gain Start Gain Start Gain

Six-year

outcome

Cumulative (six-year) GPA (A=4.0) 2.28 0.06 2.34 -- 2.22 0.12 2.3 0.04 2.2 0.14 2.32 0.02 2.34

Cumulative (six-year) credits earned 75.49 -- 75.49 -- 72.89 2.60 72.3 3.19 73.1 2.39 75.49 -- 75.49

Retention or degree completion 

(combined)

 Same institution

 All institutions 34% -- 34% -- 30% 4% 29% 5% 27% 7% 28% 6% 34%

 Four-year institutions only 42% -- 42% -- 37% 5% 37% 5% 33% 9% 37% 5% 42%

 Two-year institutions only 8% -- 8% -- 7% 1% 4% 4% 8% -- 8% -- 8%

 Any institution

 All institutions 63% -- 63% -- 58% 5% 54% 9% 55% 8% 59% 4% 63%

 Four-year institutions only 70% -- 70% -- 65% 5% 61% 9% 61% 9% 66% 4% 70%

 Two-year institutions only 35% -- 35% -- 29% 6% 35% -- 33% 2% 35% -- 35%

Degree attainment

 Bachelor’s degree or higher 38% -- 38% -- 32% 6% 36% 2% 33% 5% 38% -- 38%

 Associate’s degree or higher

 All institutions 49% -- 49% -- 46% 3% 48% 1% 49% -- 49% -- 49%

 Two-year institutions only 36% -- 36% -- 36% -- 36% -- 36% -- 36% -- 36%

Transfers from two-year to four-year

institutions 19% -- 19% -- 19% -- 19% -- 19% -- 19% -- 19%

-- Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p-value.

NOTE: The column labeled start shows the estimated outcome if the SSS students had not participated in SSS but had still received the same other supplemental services as in the study, while 

the gain shows the estimated improvement according to the statistical model that was used. Note that the starting level is a statistical estimate, calculated by subtracting the estimated 

improvement from the actual (six-year) outcome. For this reason, the starting level varies depending on the statistical model that was used. The six-year outcome is the mean outcome for all 

applicable SSS participants who were in the study, as measured through their transcripts and self-reports. For example, the mean six-year cumulative GPA among all SSS participants in the 

study was 2.34, and, among all SSS students in the study who started as freshmen in two-year institutions, 19 percent transferred to a four-year institution at some time during the six years of 

the study. Shaded cells indicate positive and statistically significant improvements in student outcomes.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
65

The results were stronger and more consistent when looking at supplemental services in 

general (exhibit 5-2). Often the results were quite similar across all six alternative methodologies. For 11 

outcomes, all six methodologies showed significant improvement in outcomes, while they were split on 

one outcome. For the only outcome for which there were mixed results, it was the dichotomous models 

that failed to find significant improvements, while the continuous models did find significant 

improvements. The effect estimates tended to fall within a small range and generally could not be 

considered as being significantly different from each other. Counting even the smallest differences in 

order to detect overall patterns, the HLM results produced higher estimates of effects 15 times and the 

regression results did so 14 times. The models that did not include propensity scores tended to produce 

higher estimates of effects slightly more often (10 times) than the models with propensity scores (eight

times), but this difference is too small to be meaningful. The continuous measures producing higher 

estimates of effects 32 times and the dichotomous measures producing higher estimates 13 times. 

When comparing exhibits 5-1 and 5-2, it is clear that there is a stronger case that 

supplemental services make a difference than that SSS services do. Several alternative explanations 

might be offered for such a difference.

 Non-SSS services may be more effective. However, there is no clear reason for such a 

difference. Since institutions need to submit plans for their SSS programs to win 

grants, and since the grants are awarded competitively, it seems likely that they are 

administered in a competent manner; if anything one might expect that they would 

perform better rather than doing more poorly than other supplemental services.

 SSS services may be hindered by federal requirements for the programs. Again, such 

an explanation is theoretically possible but seems premature without evidence of such 

hindrances, especially given that the programs have an extra source of funds (i.e., 

federal funding for SSS).

 Later-year services may be more effective than first-year services, especially with 

regard to long-term outcomes. This explanation seems more likely than the previous 

two, and in fact some researchers might be surprised to see first-year services still 

showing an effect in the sixth year. In fact, this could be a reason why SSS program 

characteristics (home-based programs, and blended programs) in some models seem 

to be more important than measures of specific first-year services; these general 

program characteristics may be acting as surrogates for SSS participation over 

multiple years.

 The models may not be fully differentiating between SSS and non-SSS services, so that 

some of the effect of SSS services is captured through the alternative measures. 

• One reason is that any SSS services received after the first year are included 

within the alternative measures (because of an inability to determine the 

source of the services), rather than being treated as SSS services. In fact, it is 

possible that because the comparison group was chosen to have comparable 

disadvantages as the SSS students, non-SSS students in the study may have 

qualified for and received SSS services in later years, so even the comparison 

group may have benefited from SSS services after the first year. 
66

Exhibit 5-2. Comparison of estimated effects of supplemental services using each of six methodologies: Six-year outcomes after 

freshman entry in 1991–92

Continuous measures of services

Dichotomous measures of services

No propensity scores With propensity scores

Regressions HLM Regressions HLM Regressions HLM

Outcome measure

Start Gain Start Gain Start Gain Start Gain Start Gain Start Gain

Six-year

outcome

Cumulative (six-year) GPA (A=4.0) 2.11 0.23 2.20 0.14 2.09 0.25 2.17 0.17 2.05 0.29 2.18 0.16 2.34

Cumulative (six-year) credits earned 53.52 21.97 58.50 16.99 56.72 18.77 56.32 19.17 57.01 18.48 59.22 16.27 75.49

Retention or degree completion 

(combined)

 Same institution

 All institutions 22% 12% 22% 12% 21% 13% 19% 15% 18% 16% 17% 17% 34%

 Four-year institutions only 26% 16% 26% 16% 25% 17% 24% 18% 21% 21% 24% 18% 42%

 Two-year institutions only 8% -- 8% -- 6% 2% 1% 7% 6% 2% 6% 2% 8%

 Any institution

 All institutions 47% 16% 45% 18% 42% 21% 39% 24% 40% 23% 44% 19% 63%

 Four-year institutions only 52% 18% 53% 17% 47% 23% 46% 24% 42% 28% 51% 19% 70%

 Two-year institutions only 19% 16% 19% 16% 19% 16% 24% 11% 22% 13% 24% 11% 35%

Degree attainment

 Bachelor’s degree or higher 28% 10% 28% 10% 24% 14% 25% 13% 24% 14% 27% 11% 38%

 Associate’s degree or higher

 All institutions 41% 8% 41% 8% 38% 11% 37% 12% 40% 9% 38% 11% 49%

 Two-year institutions only 23% 13% 28% 8% 31% 5% 28% 8% 29% 7% 28% 8% 36%

Transfers from two-year to four-year

institutions

8% 11% 6% 13% 12% 7% 9% 10% 11% 8% 9% 10% 19%

-- Not statistically significant. All effect estimates are based on regression coefficients that were statistically significant when using 0.05 as the required p-value.

NOTE: The column labeled start shows the estimated outcome if the SSS students had not participated in supplemental services, while the gain shows the estimated improvement according to 

the statistical model that was used. Note that the starting level is a statistical estimate, calculated by subtracting the estimated improvement from the actual (six-year) outcome. For this reason, 

the starting level varies depending on the statistical model that was used. The six-year outcome is the mean outcome for all applicable SSS participants who were in the study, as measured 

through their transcripts and self-reports. For example, the mean six-year cumulative GPA among all SSS participants in the study was 2.34, and, among all SSS students in the study who 

started as freshmen in two-year institutions, 19 percent transferred to a four-year institution at some time during the six years of the study. Shaded cells indicate positive and statistically 

significant improvements in student outcomes.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services (SSS): Examination of Student Outcomes After Six Years, 

2010.
67

• With the dichotomous models, the overlap between receiving SSS 

services and other services was so great that apparently little variation 

was left to explain after the other supplemental services had been 

modeled. Though the better differentiation between services with the 

use of multiple continuous measures should have lessened this problem 

(and in fact these models did show a greater effect for SSS when 

compared with the dichotomous models), it may not have eliminated it 

completely. It may be that the most critical question is whether a 

student is a recipient of a package of services, and that the measures of 

later-year services and first-year non-SSS services are largely 

sufficient to identify students receiving such packages. In that case, 

there may be little variation left to be explained by the first-year SSS 

variables.

 The inclusion of a measure of whether students had attended school some time 

after the first year could result in underestimating the effect of SSS. This 

danger was noted in chapter 2. However, the inclusion of this variable might 

have similar effects on the other measures of supplemental services, and thus it 

is not clear why this should result in weaker findings for SSS than for the other 

measures of supplemental services.

 The use of propensity scores may have contributed to underestimating the 

effects of first-year SSS services. In the models without propensity scores, the 

first-year SSS services were more likely to be significant than when the 

propensity scores were included, though they were still less likely to show 

significant relationships than some of the measures of recent services. This 

issue is discussed in greater detail later in this chapter.

Of all of these explanations, the third and fourth appear to provide the most credible 

explanations for the weaker findings concerning SSS than for supplemental services in general: 

later-year services may be more critical with regard to affecting long-term outcomes, or the models 

may be failing to fully differentiate between SSS and non-SSS services, so that some of the effect 

of SSS is captured through the other measures. The last explanation concerning the use of 

propensity scores might also help to partially explain the weaker findings for SSS, though it would 

not provide a complete explanation because the results were still somewhat weaker even when 

propensity scores were excluded.

Choosing Among the Six Models

Though the models are largely consistent in finding positive effects from 

supplemental services in general, and more limited positive effects from first-year SSS 

participation in particular, the differences in the models raise the question as to which estimates 

should be considered the most reliable. 

 One part of the answer should be relatively straightforward: the HLM 

estimates should generally be considered to be superior to the regression 

estimates because HLM is better designed for accounting for the clustering of 

students within institutions. The greatest handicap of HLM is that no 
68

accommodation was made for the fact that some outcome variables were 

dichotomous, because HLM did not converge to a solution when using this 

option. In this regard, it is reassuring that the logistic regressions (which did 

account for the limited dependent variables) often produced similar results.

 Choosing between the dichotomous measures and continuous measures may be 

more difficult. The advantage of the dichotomous measures is that one does 

not risk introducing other factors (such as academic need and motivation) that 

may be correlated with students’ levels of participation. On the other hand, 

such a large proportion of SSS students also received supplemental services 

from other sources that the dichotomous models are likely to underestimate the 

effects of SSS and, instead, absorb some of that effect within the other 

measures of supplemental services. The continuous measures, through their 

greater level of detail, are better designed to discriminate between SSS and 

other services, while the inclusion of propensity scores or a variety of measures 

of student characteristics should help to statistically adjust for any bias 

associated with conflating participation with academic need or motivation. 

Thus, the continuous measures seem to be the best designed for measuring the 

effects of SSS.

 Perhaps the most difficult choice is the decision whether to include or not 

include propensity measures. The argument for including propensity measures 

is that students’ level of involvement in SSS may be interrelated with factors 

such as motivation and academic need; thus, there is a risk that the statistical 

models might measure the importance of these factors with regard to student 

outcomes rather than giving accurate measures of the effects of SSS. The 

purpose of adding propensity measures is to model the likelihood of 

participation separately, so that the SSS participation measures will only 

capture the added benefit from participating in SSS, distinct from these other 

factors. This argument is an important one, and there clearly is a need to 

somehow account for the importance of these other factors. On the other hand, 

there are also arguments against the need for or benefit from using propensity 

measures.

• Factors such as motivation and academic need are included in the 

statistical models in any case, and the regression equations should 

be better able to individually fit those variables when they are 

measured as multiple variables in the context of a particular student 

outcome, rather than through standardized formulas that are the 

same for every model.

• While the propensity scores were derived individually (i.e., one 

score at a time), they are used in combination. The reason for using 

nine separate propensity measures is that participation in one 

service showed little relation to participation in another. On the 

other hand, when all nine are used together in a single model, they 

may interact in a way that might confuse the measurement process.
69

• If irrelevant variables are added to a regression correlation and they 

are correlated with other variables of importance, their inclusion 

may result in increased standard errors that may interfere with 

examining those other variables. The propensity scores do show a 

fairly high correlation with the measures of first-year SSS services 

(ranging from 0.22 for services for disabled students to 0.64 for 

peer tutoring), so this is a real possibility. The facts that peer 

tutoring especially had a high correlation and that the results for 

peer tutoring were the most sensitive to the inclusion of propensity 

scores (i.e., peer tutoring was the SSS variable that performed the 

best when propensity scores were excluded but that never was 

statistically significant when including propensity scores) are 

consistent with this explanation.

• As one test of whether the propensity scores belonged in the model, 

few if any of the propensity scores were statistically significant. 

For six of the 12 outcome measures, no propensity score was 

statistically significant, while one was significant for four outcome 

measures, two were significant for one outcome measures, and three 

were significant for one outcome measure. The propensity scores 

were retained under the principle that they were theoretically 

necessarily. However, an alternative approach is to consider the 

other measures of student characteristics as also adjusting for 

students’ propensity to participate, in which case it would be 

reasonable to drop those propensity scores that were not significant. 

Exhibit 5-3 shows which SSS services would be considered to have 

positive and statistically significant effects in that case.

• The actual performance of the propensity measures is in some ways 

counterintuitive. The SSS variable that stood out most strongly was 

home-based programs, while the actual services generally were not 

statistically significant when propensity scores were included. Yet 

even if the organization of the programs is important, presumably 

the actual effect comes from the receipt of supplemental services; 

one would not expect a student who officially belonged to SSS but 

never received any SSS services to experience much benefit.25 

There is at least one possible explanation why the variable for 

home-based programs may be capturing all of the first-year effects: 

it is possible that it acts as a surrogate for receiving a full package 

of services (because one of the purposes of a home-based program 

is to see that students receive such a package), and that there is little 

or no extra benefit to be measured once the effect of the “package” 

is measured. Alternatively, the absence of significant results for 

individual services but not home-based programs could be 

 

25 It is possible that simply belonging to SSS may help to create some identification with the institution and with other 

students, so home-based services may provide some benefit that extends beyond the receipt of services. Still, it seems 

unlikely that all of the benefit from SSS is independent of the receipt of services. 
70

considered as a sign that the statistical model was not performing 

well.

Weighing all these factors together, the best choice appears to be to include 

propensity scores only when they are statistically significant. 

Policy Findings Concerning SSS

The single largest finding is that both supplemental services in general and first-year 

SSS services in particular have a positive effect on students’ academic outcomes, though SSS 

effects were not detected for as many outcome measures as supplemental services in general. This 

section reviews which particular outcomes were affected by SSS or by supplemental services in 

general. It also examines what implications these findings have for how SSS might best be 

administered. More specifically, these findings involve the relative importance of specific SSS 

services and the importance of receiving services beyond the first year.

Exhibit 5-3. First-year SSS services that showed positive and statistically significant effects 

on student outcomes when only statistically significant propensity scores are 

retained, by outcomes six years after freshman entry in 1991–92

Outcome measure Peer tutoring

Services for 

the disabled

Home-based Blended

Cumulative (six-year) GPA 

Cumulative (six-year) credits earned

Retention or degree completion (combined)

 Same institution

 All institutions   

 Four-year institutions only  

 Two-year institutions only   

 Any institution

 All institutions 

 Four-year institutions only   

 Two-year institutions only

Degree attainment

 Bachelor’s degree or higher

 Associate’s degree or higher

 All institutions 

 two-year institutions only

Transfers from two-year to four-year institutions

 Statistically significant at 0.05 level.

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.

Types of outcomes affected by SSS. Based on the hypothesis that the HLM models 

with continuous measures of SSS services provide the most reliable measures of effect, and that 

propensity scores are used only when statistically significant, SSS appeared to have a positive 

effect on seven measures of student outcomes, while the remaining five outcomes showed no 

effect. However, supplemental services in general appeared to have positive effects on all 12 

outcomes, providing some justification for inferring that SSS might be helpful in those areas as 
71

well. Following are the complete findings, together with some speculations about why some 

outcome measures showed no SSS effects.

 GPAs. Both first-year SSS services and other supplemental services were 

shown to have a positive effect on students’ cumulative GPAs. 

 Credits earned. No effect from first-year SSS participation was found on the 

total credits earned, though other supplemental services did have an effect.

 Retention or degree completion (combined measure). First-year SSS 

participation had a positive effect on retention or degree completion for every 

measure but retention or degree completion at any institution (for two-year

institutions only). Other supplemental services were found to have a positive 

effect for the remaining measure (as well as on the measures where first-year 

SSS participation showed a positive association with improved student 

outcomes). 

 Degree attainment (alone). First-year SSS participation was positively 

related to earning an associate’s degree or higher at all institutions but not to 

the other two measures of degree attainment. Supplemental services were 

found to have a positive effect on all three measures. Possibly the weaker 

findings for degree attainment alone as compared with degree attainment when 

combined with retention are because the six-year time span may not be 

adequate for many disadvantaged students to complete a degree. As shown in 

exhibit 2-4, 25 percent of SSS students were still in college though they had 

not yet attained a baccalaureate degree.

 Transfers from two-year to four-year institutions. No specific effect from 

first-year SSS participation was detected, though supplemental services were 

found to have a positive effect. All models that were limited only to two-year

institutions tended to perform poorly, very likely due to the limited number of 

cases available for analysis.

The importance of specific SSS services. The specific SSS services that were found 

to have a positive effect were home-based programs, blended programs, peer tutoring, and services 

for the disabled. Other services that had a positive effect, and that may or may not have been 

offered through SSS, were counseling, field trips or cultural enrichment, referrals to outside 

resources, services for the disabled and for those with limited English ability, college reentrance 

counseling, and recent contacts with support services.

One should be wary of judging that any remaining services failed to have an effect. 

From a statistical viewpoint, the failure to find a statistically significant difference only means that 

the difference could have occurred by chance, but not that it did. Further, some of the findings, 

including the performance of the measures of home-based programs, referrals to other services, 

and recent contacts with support services, may suggest both that it is important that students 

receive a package of services and that the contributions of specific services may not appear 

independently but rather be incorporated within indicators that students received such a package.
72

The importance of services beyond the first year. As noted elsewhere, some of the 

services that most consistently were statistically significant were those received after the first year, 

and especially in the most recent years. No data are available on whether or not those services 

were offered through SSS. However, these findings suggest that SSS should not be viewed simply 

as a first-year program but that the continued provision of resources is an important tool in 

improving student outcomes.
73

References

Chaney, B., Muraskin, L., Cahalan, M. and Rak, R. (1998). Helping the Progress of 

Disadvantaged Students in Higher Education: The Federal Student Support Services Program. 

Educational Evaluation and Policy Analysis, 20(3), 197–215.

U.S. Department of Education, Office of the Under Secretary. (1994). National Study of Student 

Support Services, Interim Report: Volume 2: Profile of Freshman Participants and Project 

Services: 1991–92, by Margaret Cahalan, Bradford Chaney and Selma Chen. Washington, D.C.: 

Author.

U.S. Department of Education, Office of the Under Secretary. (1997). National Study of Student 

Support Services: Third-Year Longitudinal Study Results and Program Implementation Study 

Update, by Bradford Chaney, Lana Muraskin, Margaret Cahalan and Rebecca Rak. Washington, 

D.C.: Author.
74
75

Appendix A: Detailed Tables

NOTE: This appendix provides the complete regression models used in the analysis. For brevity, 

because of the large number of models that were tested, only the results of the HLM analysis are 

shown (under the assumption that they provide more accurate estimates of the standard errors). 
76
77

Exhibit A-1. HLM analysis to predict cumulative GPAs six years after 

freshman entry in 1991–92 using dichotomous measures of 

participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 2.4102 0.0246 98.0840 0.0000

Two-year institution 0.0695 0.0656 1.0600 0.2960

Doctorate-granting institution -0.0721 0.0538 -1.3400 0.1870

Age in 1991 0.0273 0.0027 10.0700 0.0000

Black student -0.3321 0.0354 -9.3730 0.0000

Hispanic student -0.1301 0.0450 -2.8900 0.0040

American Indian student -0.2584 0.0800 -3.2310 0.0020

Asian student -0.0733 0.0577 -1.2720 0.2040

Female student 0.1410 0.0238 5.9170 0.0000

High school GPA 0.2944 0.0274 10.7410 0.0000

Percentile on SAT/ACT 0.2387 0.0558 4.2790 0.0000

Took any developmental course -0.1558 0.0238 -6.5370 0.0000

Hours on school activities 0.0036 0.0006 6.1760 0.0000

Attended college after first year 0.6618 0.0577 11.4650 0.0000

No parental education beyond high school 0.0259 0.0203 1.2710 0.2040

Family income greater than $20,000 0.0248 0.0246 1.0110 0.3120

Above average academic ability 0.1408 0.0193 7.3030 0.0000

Would attend free tutoring -0.0779 0.0255 -3.0520 0.0030

Have major concerns about ed. finances -0.0375 0.0181 -2.0750 0.0380

Once I start something, I finish it 0.0467 0.0190 2.4540 0.0140

Feel comfortable on this campus 0.0597 0.0253 2.3590 0.0180

Participated in SSS in first year 0.0282 0.0339 0.8300 0.4110

Received non-SSS services in first year 0.0186 0.0292 0.6370 0.5240

Received services after first year 0.1775 0.0362 4.9060 0.0000

Missing data on student age 0.6446 0.0790 8.1570 0.0000

Missing data on student race 0.0432 0.0741 0.5840 0.5590

Missing data on high school GPA 0.7922 0.1186 6.6770 0.0000

Missing data on SAT/ACT 0.0259 0.0409 0.6330 0.5260

Missing data on developmental courses -0.2461 0.0657 -3.7490 0.0000

Missing data for hours on school activities 0.1413 0.0425 3.3230 0.0010

Missing data on first generation -0.0922 0.0576 -1.6000 0.1090

Missing data on income -0.0362 0.0350 -1.0340 0.3020

Missing data on perceived ability 0.1391 0.1084 1.2830 0.2000

Missing data on attending free tutoring -0.0864 0.1661 -0.5200 0.6020

Missing data on financial concerns -0.0562 0.1084 -0.5190 0.6040

Missing data on finishing tasks 0.1792 0.1206 1.4850 0.1370

Missing data on comfort on campus -0.1804 0.1848 -0.9760 0.3300

Missing data on services in later years 0.2412 0.0896 2.6920 0.0080

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
78

Exhibit A-2. HLM analysis to predict the number of credits earned six 

years after freshman entry in 1991–92 using dichotomous 

measures of participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 76.1699 1.7481 43.5730 0.0000

Two-year institution -20.0641 3.5524 -5.6480 0.0000

Doctorate-granting institution 4.5546 5.0934 0.8940 0.3760

Black student -6.2885 1.4786 -4.2530 0.0000

Hispanic student 0.8139 1.8453 0.4410 0.6590

American Indian student -18.5136 3.6791 -5.0320 0.0000

Asian student 5.1328 2.9719 1.7270 0.0840

Female student 3.1824 1.4458 2.2010 0.0280

Student was full-time 14.0263 3.1303 4.4810 0.0000

High school GPA 10.5841 1.4056 7.5300 0.0000

Percentile on SAT/ACT 10.0523 4.0474 2.4840 0.0130

Took any developmental course -6.3317 1.6473 -3.8440 0.0000

Hours on school activities 0.2194 0.0381 5.7560 0.0000

Attended college after first year 41.0143 2.5731 15.9400 0.0000

No parental education beyond high 

school -1.2227 1.2949 -0.9440 0.3450

Family income greater than $20,000 -1.7380 1.6204 -1.0730 0.2840

Above average academic ability 2.5914 1.4739 1.7580 0.0780

Have major concerns about ed. finances -3.6605 1.3501 -2.7110 0.0070

Once I start something, I finish it 4.5540 1.4359 3.1720 0.0020

Feel comfortable on this campus 7.2484 1.9053 3.8040 0.0000

Participated in SSS in first year 3.6342 2.3208 1.5660 0.1170

Received non-SSS services in first year 0.6991 1.2667 0.5520 0.5810

Received services after first year 22.2065 1.9249 11.5360 0.0000

Missing data on student race -2.2149 4.4667 -0.4960 0.6200

Missing data on full-time status 25.4795 7.2609 3.5090 0.0010

Missing data on high school GPA 29.2348 5.2492 5.5690 0.0000

Missing data on SAT/ACT -2.3667 1.9763 -1.1980 0.2310

Missing data on developmental courses -6.9558 4.8878 -1.4230 0.1550

Missing data for hours on school 

activities 6.0046 3.5506 1.6910 0.0900

Missing data on first generation -3.2014 2.5629 -1.2490 0.2120

Missing data on income -2.4127 1.7562 -1.3740 0.1700

Missing data on perceived ability 1.4748 5.2493 0.2810 0.7790

Missing data on financial concerns -9.8968 4.4124 -2.2430 0.0250

Missing data on finishing tasks 9.5735 7.2156 1.3270 0.1850

Missing data on comfort on campus -1.0909 8.4039 -0.1300 0.8970

Missing data on services in later years 1.7242 5.2594 0.3280 0.7430

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
79

Exhibit A-3. HLM analysis to predict retention or baccalaureate degree 

completion at the same institution six years after freshman entry in 

1991–92 using dichotomous measures of participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.3188 0.0111 28.6890 0.0000

Two-year institution -0.2671 0.0254 -10.5100 0.0000

Doctorate-granting institution 0.0550 0.0290 1.8960 0.0640

Female student 0.0260 0.0138 1.8850 0.0590

Student was full time 0.0463 0.0260 1.7790 0.0750

High school GPA 0.0669 0.0129 5.1790 0.0000

Percentile on SAT/ACT 0.1297 0.0435 2.9840 0.0030

Took any developmental course -0.0533 0.0160 -3.3360 0.0010

Hours on school activities 0.0010 0.0003 3.0730 0.0030

Attended college after first year 0.1462 0.0254 5.7620 0.0000

No parental education beyond high 

school -0.0006 0.0145 -0.0390 0.9690

Family income greater than $20,000 -0.0196 0.0158 -1.2370 0.2160

Once I start something, I finish it 0.0313 0.0133 2.3520 0.0190

Feel comfortable on this campus 0.0622 0.0181 3.4380 0.0010

Participated in SSS in first year 0.0180 0.0222 0.8110 0.4170

Received non-SSS services in first year -0.0071 0.0125 -0.5720 0.5670

Received services after first year 0.1505 0.0216 6.9710 0.0000

Missing data on full-time status 0.0886 0.0739 1.1990 0.2310

Missing data on high school GPA 0.2175 0.0454 4.7870 0.0000

Missing data on SAT/ACT 0.0209 0.0205 1.0190 0.3090

Missing data on developmental courses -0.1219 0.0294 -4.1460 0.0000

Missing data for hours on school 

activities 0.0065 0.0309 0.2100 0.8340

Missing data on first generation 0.0146 0.0291 0.5020 0.6150

Missing data on income -0.0233 0.0209 -1.1170 0.2650

Missing data on finishing tasks 0.0135 0.0850 0.1580 0.8750

Missing data on comfort on campus -0.0003 0.0892 -0.0030 0.9970

Missing data on services in later years 0.0723 0.0386 1.8710 0.0610

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
80

Exhibit A-4. HLM analysis to predict retention or baccalaureate degree 

completion at any institution six years after freshman entry in 

1991–92 using dichotomous measures of participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.6187 0.0115 53.8190 0.0000

Two-year institution -0.2134 0.0280 -7.6180 0.0000

Doctorate-granting institution 0.0485 0.0292 1.6600 0.1040

Black student 0.0211 0.0189 1.1210 0.2630

Hispanic student 0.0622 0.0237 2.6250 0.0090

American Indian student -0.1069 0.0567 -1.8860 0.0590

Asian student 0.0967 0.0342 2.8250 0.0050

Female student 0.0453 0.0141 3.2080 0.0020

Newly married -0.1065 0.0284 -3.7430 0.0000

High school GPA 0.0530 0.0121 4.3800 0.0000

Percentile on SAT/ACT 0.0876 0.0339 2.5810 0.0100

Took any developmental course -0.0561 0.0146 -3.8470 0.0000

Hours on school activities 0.0012 0.0004 2.6610 0.0080

Attended college after first year 0.3285 0.0301 10.9110 0.0000

No parental education beyond high 

school -0.0568 0.0146 -3.8940 0.0000

Family income greater than $20,000 0.0032 0.0152 0.2140 0.8310

Above average academic ability 0.0258 0.0139 1.8500 0.0640

Participated in SSS in first year 0.0202 0.0158 1.2760 0.2020

Received non-SSS services in first year -0.0029 0.0160 -0.1800 0.8580

Received services after first year 0.2292 0.0228 10.0660 0.0000

Missing data on student race 0.0986 0.0526 1.8760 0.0600

Missing data on full-time status -0.0258 0.0722 -0.3570 0.7210

Missing data on high school GPA 0.1514 0.0486 3.1150 0.0020

Missing data on SAT/ACT -0.0438 0.0212 -2.0700 0.0380

Missing data on developmental courses -0.0568 0.0333 -1.7070 0.0870

Missing data for hours on school 

activities -0.0011 0.0408 -0.0270 0.9790

Missing data on first generation -0.0522 0.0359 -1.4560 0.1450

Missing data on income -0.0234 0.0229 -1.0200 0.3080

Missing data on perceived ability 0.0350 0.0587 0.5970 0.5500

Missing data on services in later years 0.0700 0.0511 1.3680 0.1710

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
81

Exhibit A-5. HLM analysis to predict baccalaureate degree completion (or 

higher) at any institution six years after freshman entry in 1991–92

using dichotomous measures of participations 

Variable Coefficient Standard 

Error

T-ratio P-value

Intercept 0.3988 0.0148 27.0210 0.0000

Two-year institution -0.2272 0.0309 -7.3620 0.0000

Doctorate-granting institution 0.0654 0.0369 1.7700 0.0830

Black student -0.0369 0.0241 -1.5300 0.1260

Hispanic student 0.0041 0.0221 0.1840 0.8540

American Indian student -0.1613 0.0542 -2.9730 0.0030

Asian student 0.0929 0.0285 3.2660 0.0010

Female student 0.0309 0.0156 1.9750 0.0480

Live in college housing or frat./sorority 0.0703 0.0190 3.6990 0.0000

Student was full-time 0.0597 0.0272 2.2000 0.0280

Newly married -0.0715 0.0298 -2.4000 0.0170

High school GPA 0.0962 0.0145 6.6330 0.0000

Percentile on SAT/ACT 0.1246 0.0315 3.9540 0.0000

Took any developmental course -0.0904 0.0163 -5.5320 0.0000

Hours on school activities 0.0017 0.0005 3.1120 0.0020

Attended college after first year 0.1574 0.0335 4.7040 0.0000

No parental education beyond high school -0.0256 0.0150 -1.7060 0.0880

Family income greater than $20,000 0.0203 0.0141 1.4320 0.1520

Above average academic ability 0.0419 0.0134 3.1360 0.0020

Have major concerns about ed. finances -0.0398 0.0121 -3.2770 0.0010

Once I start something, I finish it 0.0541 0.0143 3.7860 0.0000

Participated in SSS in first year 0.0196 0.0197 0.9990 0.3180

Received non-SSS services in first year 0.0042 0.0132 0.3210 0.7480

Received services after first year 0.1279 0.0201 6.3780 0.0000

Missing data on student race 0.0485 0.0496 0.9780 0.3290

Missing data on college housing 0.0790 0.1173 0.6740 0.5000

Missing data on full-time status 0.0966 0.1388 0.6960 0.4860

Missing data on high school GPA 0.2809 0.0409 6.8670 0.0000

Missing data on SAT/ACT -0.0233 0.0241 -0.9630 0.3360

Missing data on developmental courses -0.1171 0.0321 -3.6470 0.0000

Missing data for hours on school activities 0.0329 0.0381 0.8640 0.3880

Missing data on first generation -0.0459 0.0346 -1.3270 0.1850

Missing data on income 0.0159 0.0201 0.7910 0.4290

Missing data on perceived ability 0.0460 0.0555 0.8280 0.4080

Missing data on financial concerns -0.1006 0.0486 -2.0700 0.0380

Missing data on finishing tasks -0.0372 0.0612 -0.6080 0.5430

Missing data on services in later years 0.0800 0.0598 1.3390 0.1810

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
82

Exhibit A-6. HLM analysis to predict associate’s degree completion (or higher) 

at any institution six years after freshman entry in 1991–92 using 

dichotomous measures of participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.5401 0.0151 35.8150 0.0000

Two-year institution -0.0412 0.0349 -1.1790 0.2450

Doctorate-granting institution 0.0445 0.0399 1.1160 0.2710

Black student -0.0764 0.0280 -2.7240 0.0070

Hispanic student -0.0499 0.0237 -2.1020 0.0350

American Indian student -0.2518 0.0551 -4.5710 0.0000

Asian student 0.0282 0.0302 0.9320 0.3520

Live in college housing or frat./sorority 0.0534 0.0181 2.9440 0.0040

Student was full-time 0.0872 0.0308 2.8300 0.0050

High school GPA 0.1184 0.0157 7.5480 0.0000

Took any developmental course -0.0960 0.0146 -6.5700 0.0000

Hours on school activities 0.0014 0.0006 2.4830 0.0130

Attended college after first year 0.3060 0.0388 7.8940 0.0000

No parental education beyond high school -0.0217 0.0169 -1.2810 0.2000

Family income greater than $20,000 0.0190 0.0174 1.0890 0.2770

Above average academic ability 0.0451 0.0158 2.8530 0.0050

Have major concerns about ed. finances -0.0357 0.0142 -2.5140 0.0120

Feel comfortable on this campus 0.0480 0.0206 2.3320 0.0200

Participated in SSS in first year 0.0005 0.0197 0.0230 0.9820

Received non-SSS services in first year 0.0207 0.0129 1.6030 0.1090

Received services after first year 0.1034 0.0248 4.1730 0.0000

Missing data on student race -0.0175 0.0544 -0.3210 0.7480

Missing data on college housing -0.0279 0.1464 -0.1900 0.8490

Missing data on full-time status 0.1408 0.1860 0.7570 0.4490

Missing data on high school GPA 0.3089 0.0597 5.1710 0.0000

Missing data on developmental courses -0.1356 0.0384 -3.5270 0.0010

Missing data for hours on school activities 0.0219 0.0400 0.5490 0.5830

Missing data on first generation -0.0665 0.0344 -1.9310 0.0530

Missing data on income 0.0147 0.0190 0.7730 0.4400

Missing data on perceived ability 0.0250 0.0721 0.3460 0.7290

Missing data on financial concerns -0.0214 0.0899 -0.2380 0.8120

Missing data on comfort on campus 0.0387 0.0836 0.4620 0.6430

Missing data on services in later years 0.1240 0.0632 1.9630 0.0490

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
83

Exhibit A-7. HLM analysis to predict transfers from two-year to four-year

institutions using dichotomous measures six years after 

freshman entry in 1991–92 of participation

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.2124 0.0176 12.0660 0.0000

Female student -0.0728 0.0261 -2.7880 0.0060

Live in college housing or frat./sorority 0.3242 0.0785 4.1290 0.0000

Newly married -0.0960 0.0368 -2.6110 0.0090

High school GPA 0.0483 0.0181 2.6680 0.0080

Took any developmental course -0.0789 0.0259 -3.0440 0.0030

Attended college after first year 0.0355 0.0358 0.9920 0.3220

No parental education beyond high 

school -0.1133 0.0265 -4.2700 0.0000

Family income greater than $20,000 0.0160 0.0261 0.6120 0.5400

Expect harder time than most -0.0531 0.0281 -1.8880 0.0590

Participated in SSS in first year -0.0043 0.0284 -0.1530 0.8790

Received non-SSS services in first year 0.0410 0.0269 1.5270 0.1270

Received services after first year 0.2118 0.0311 6.8200 0.0000

Missing data on college housing -0.0140 0.0926 -0.1510 0.8800

Missing data on high school GPA 0.0714 0.0696 1.0250 0.3060

Missing data on developmental courses -0.1050 0.0613 -1.7120 0.0860

Missing data on first generation -0.0707 0.0494 -1.4310 0.1520

Missing data on income 0.0122 0.0359 0.3390 0.7350

Missing data on expect harder time -0.0348 0.0768 -0.4530 0.6500

Missing data on services in later years -0.0149 0.0708 -0.2110 0.8330

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
84

Exhibit A-8. HLM analysis to predict cumulative GPAs six years after freshman entry 

in 1991–92 using continuous measures of participation and no measures 

of propensity to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 2.4089 0.0248 97.1910 0.0000

SSS services are home-based 0.1215 0.0667 1.8220 0.0750

Two-year institution 0.1369 0.0623 2.1980 0.0340

Doctorate-granting institution -0.0758 0.0653 -1.1600 0.2530

No SSS programs at institution 0.0371 0.0667 0.5570 0.5800

SSS services are blended -0.0494 0.0687 -0.7190 0.4760

Age in 1991 0.0272 0.0020 13.9180 0.0000

Black student -0.3281 0.0307 -10.7000 0.0000

Hispanic student -0.1416 0.0393 -3.6030 0.0010

American Indian student -0.2643 0.0880 -3.0050 0.0030

Asian student -0.0551 0.0537 -1.0250 0.3060

Female student 0.1365 0.0219 6.2450 0.0000

High school GPA 0.2883 0.0194 14.8380 0.0000

Percentile on SAT/ACT 0.2148 0.0547 3.9230 0.0000

Took any developmental course -0.1453 0.0235 -6.1770 0.0000

Hours on school activities 0.0034 0.0007 4.9680 0.0000

Attended college after first year 0.7027 0.0338 20.7800 0.0000

No parental education beyond high school 0.0243 0.0230 1.0560 0.2910

Family income greater than $20,000 0.0249 0.0239 1.0430 0.2980

Above average academic ability 0.1384 0.0219 6.3230 0.0000

Would attend free tutoring -0.0824 0.0233 -3.5320 0.0010

Feel comfortable on this campus 0.0641 0.0266 2.4080 0.0160

Number of hours: instr. Courses 0.0003 0.0005 0.5550 0.5790

Number of hours: profl. tutoring -0.0032 0.0090 -0.3570 0.7210

Number of hours: peer tutoring 0.0059 0.0013 4.5090 0.0000

Number of hours: profl. counseling -0.0141 0.0057 -2.4650 0.0140

Number of hours: peer counseling -0.0331 0.0217 -1.5240 0.1270

Number of hours: labs 0.0009 0.0024 0.3790 0.7040

Number of hours: workshops 0.0106 0.0062 1.7180 0.0850

Number of hours: cultural events 0.0086 0.0094 0.9060 0.3650

Number of hours: handicapped srvc. 0.0006 0.0034 0.1670 0.8670

Services for physically disabled -0.0126 0.0353 -0.3560 0.7210

Services for limited-English ability 0.0121 0.0603 0.2000 0.8420

Student orientation -0.0347 0.0295 -1.1750 0.2400

College reentrance counseling -0.0521 0.0292 -1.7860 0.0740

Classroom instr.--basic skills -0.0523 0.0358 -1.4610 0.1440

Classroom instr.--dvlpmntl. English 0.0038 0.0408 0.0940 0.9260

Classroom instr.--dvlpmntl. math 0.0086 0.0361 0.2370 0.8130

Cultural enrichment activities 0.0118 0.0053 2.2050 0.0270

Referrals to agencies/resources 0.0649 0.0286 2.2720 0.0230

# of counseling sessions in 1991–92 -0.0042 0.0027 -1.5520 0.1200

Continues to next page.
85

Continued from previous page.

# of counseling sessions in 1992–93 0.0002 0.0027 0.0870 0.9310

# of counseling sessions in 1993–94 0.0076 0.0027 2.8290 0.0050

# of tutoring sessions in 1991–92 -0.0002 0.0016 -0.1480 0.8830

# of tutoring sessions in 1992–93 -0.0005 0.0017 -0.3010 0.7630

# of tutoring sessions in 1993–94 0.0013 0.0019 0.6960 0.4860

Some tutoring in years 4–6 -0.0116 0.0034 -3.3910 0.0010

Some counseling in years 4–6 0.0111 0.0037 2.9710 0.0030

Some services for disabled in years 4–6 -0.0637 0.0555 -1.1490 0.2510

Field trip in years 4–6 0.1254 0.0265 4.7270 0.0000

Contact with support services in years 4–6 0.0164 0.0031 5.3740 0.0000

Missing data on student age 0.5991 0.0855 7.0050 0.0000

Missing data on student race 0.0296 0.0856 0.3460 0.7290

Missing data on high school GPA 0.7650 0.0791 9.6740 0.0000

Missing data on SAT/ACT 0.0356 0.0348 1.0230 0.3070

Missing data on developmental courses -0.1948 0.0809 -2.4080 0.0160

Missing data for hours on school activities 0.1400 0.0581 2.4080 0.0160

Missing data on first generation -0.0917 0.0516 -1.7760 0.0750

Missing data on income -0.0212 0.0355 -0.5990 0.5490

Missing data on perceived ability 0.1268 0.1051 1.2060 0.2280

Missing data on attending free tutoring 0.0243 0.1935 0.1250 0.9010

Missing data on comfort on campus -0.2178 0.1928 -1.1290 0.2590

Missing data on services in later years 0.2291 0.0686 3.3380 0.0010

Missing data on services for limited-English -0.1189 0.4893 -0.2430 0.8080

Missing data on student orientation -0.1051 0.1237 -0.8490 0.3960

Missing data on re-entrance counseling 0.1733 0.0712 2.4350 0.0150

Missing data on classroom instr.--basic skills -0.1131 0.1043 -1.0850 0.2790

Missing data on cultural enrichment activities -0.2931 0.2373 -1.2350 0.2170

Missing data on referrals to agencies/resources 0.5182 0.5420 0.9560 0.3390

Missing data on tutoring in 1991–92 -0.0611 0.0753 -0.8110 0.4180

Missing data on tutoring in years 4–6 -0.0929 0.1311 -0.7090 0.4780

Missing data on counseling in years 4–6 -0.0660 0.0882 -0.7490 0.4540

Missing data on services for disabled in years 4–6 -0.0711 0.1298 -0.5480 0.5830

Missing data on field trip in years 4–6 0.4276 0.1805 2.3700 0.0180

Missing data on contact with support services in yrs.

4–6 -0.2436 0.1759 -1.3850 0.1660

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
86

Exhibit A-9. HLM analysis to predict the number of credits earned six years after 

freshman entry in 1991–92 using continuous measures of participation 

and no measures of propensity to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 76.1473 1.7824 42.7210 0.0000

SSS services are home-based 4.5830 4.8039 0.9540 0.3460

Two-year institution -14.5154 4.4250 -3.2800 0.0020

Doctorate-granting institution 4.1163 4.7266 0.8710 0.3890

No SSS programs at institution 3.1919 4.7665 0.6700 0.5070

SSS services are blended -2.5910 4.9978 -0.5180 0.6060

Black student -6.3235 1.7780 -3.5560 0.0010

Hispanic student -1.3367 2.2859 -0.5850 0.5580

American Indian student -18.8912 5.1443 -3.6720 0.0000

Asian student 6.4593 3.0902 2.0900 0.0360

Female student 3.9224 1.2552 3.1250 0.0020

Student was full-time 13.7099 2.2520 6.0880 0.0000

High school GPA 10.2556 1.1129 9.2150 0.0000

Percentile on SAT/ACT 11.3196 3.1250 3.6220 0.0010

Took any developmental course -6.0028 1.3548 -4.4310 0.0000

Hours on school activities 0.1716 0.0395 4.3460 0.0000

Attended college after first year 43.7484 1.9238 22.7410 0.0000

No parental education beyond high school -0.7331 1.3206 -0.5550 0.5780

Family income greater than $20,000 -1.8817 1.3812 -1.3620 0.1730

Have major concerns about ed. finances -4.1052 1.2510 -3.2820 0.0010

Once I start something, I finish it 3.6691 1.4058 2.6100 0.0090

Feel comfortable on this campus 6.8203 1.5466 4.4100 0.0000

Number of hours: instr. courses 0.0514 0.0278 1.8520 0.0640

Number of hours: profl. tutoring -0.3803 0.5231 -0.7270 0.4670

Number of hours: peer tutoring 0.3910 0.0758 5.1580 0.0000

Number of hours: profl. counseling -0.8865 0.3333 -2.6600 0.0080

Number of hours: peer counseling -0.6247 1.2633 -0.4940 0.6210

Number of hours: labs -0.0248 0.1371 -0.1810 0.8570

Number of hours: workshops 0.0276 0.3594 0.0770 0.9390

Number of hours: cultural events 1.4762 0.5487 2.6910 0.0080

Number of hours: handicapped srvc. 0.1528 0.1954 0.7820 0.4340

Services for physically disabled 1.5046 2.0381 0.7380 0.4600

Services for limited-English ability 0.3152 3.4657 0.0910 0.9280

Student orientation -0.3259 1.7005 -0.1920 0.8480

College reentrance counseling -4.2336 1.6798 -2.5200 0.0120

Classroom instr.--basic skills -5.0660 2.0673 -2.4500 0.0140

Classroom instr.--dvlpmntl. English 1.2470 2.3485 0.5310 0.5950

Classroom instr.--dvlpmntl. math -1.6766 2.0798 -0.8060 0.4200

Cultural enrichment activities 0.6065 0.3078 1.9700 0.0480

Referrals to agencies/resources -1.7362 1.6520 -1.0510 0.2940

# of counseling sessions in 1991–92 -0.1195 0.1567 -0.7620 0.4460

Continues to next page.
87

Continued from previous page.

# of counseling sessions in 1992–93 0.0498 0.1554 0.3200 0.7480

# of counseling sessions in 1993–94 0.4968 0.1562 3.1810 0.0020

# of tutoring sessions in 1991–92 -0.1099 0.0927 -1.1850 0.2360

# of tutoring sessions in 1992–93 0.4678 0.0980 4.7740 0.0000

# of tutoring sessions in 1993–94 0.3040 0.1106 2.7480 0.0060

Some tutoring in years 4–6 0.0091 0.1977 0.0460 0.9640

Some counseling in years 4–6 1.4312 0.2165 6.6100 0.0000

Some services for disabled in years 4–6 -12.2753 3.2296 -3.8010 0.0000

Field trip in years 4–6 12.9644 1.5348 8.4470 0.0000

Contact with support services in years 4–6 2.1192 0.1768 11.9850 0.0000

Missing data on student race -3.9126 4.9972 -0.7830 0.4340

Missing data on full-time status 20.6079 7.1663 2.8760 0.0040

Missing data on high school GPA 26.7383 4.5542 5.8710 0.0000

Missing data on SAT/ACT 0.2429 1.9648 0.1240 0.9020

Missing data on developmental courses -2.6660 4.6790 -0.5700 0.5680

Missing data for hours on school activities 5.3797 3.5652 1.5090 0.1310

Missing data on first generation -2.2715 3.0565 -0.7430 0.4570

Missing data on income -1.8068 2.0460 -0.8830 0.3770

Missing data on perceived ability 1.6111 6.1057 0.2640 0.7920

Missing data on financial concerns -8.8075 5.6964 -1.5460 0.1220

Missing data on finishing tasks 9.3706 6.0911 1.5380 0.1240

Missing data on comfort on campus -5.2128 7.5539 -0.6900 0.4900

Missing data on B32 -1.6215 3.9603 -0.4090 0.6820

Missing data on services for limited-English -20.8979 28.1960 -0.7410 0.4590

Missing data on student orientation 0.8228 7.1110 0.1160 0.9080

Missing data on reentrance counseling -1.0399 4.1092 -0.2530 0.8000

Missing data on classroom instr.--basic skills 7.7590 5.8813 1.3190 0.1870

Missing data on cultural enrichment activities -29.4697 13.6924 -2.1520 0.0310

Missing data on referrals to agencies/resources 45.3459 31.2358 1.4520 0.1470

Missing data on tutoring in 1991–92 -3.0130 4.3513 -0.6920 0.4880

Missing data on tutoring in years 4–6 -0.0991 7.6381 -0.0130 0.9900

Missing data on counseling in years 4–6 -5.3820 5.0868 -1.0580 0.2910

Missing data on services for disabled in years 4–6 -8.6346 7.5275 -1.1470 0.2520

Missing data on field trip in years 4–6 7.6529 10.6411 0.7190 0.4720

Missing data on contact with support services in yrs.

4–6 6.0809 10.3418 0.5880 0.5560

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
88

Exhibit A-10. HLM analysis to predict retention or baccalaureate degree completion 

at the same institution six years after freshman entry in 1991–92 using 

continuous measures of participation and no measures of propensity 

to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.3184 0.0109 29.2890 0.0000

SSS services are home-based 0.0545 0.0291 1.8760 0.0670

Two-year institution -0.2288 0.0279 -8.2000 0.0000

Doctorate-granting institution 0.0433 0.0284 1.5240 0.1350

No SSS programs at institution 0.0561 0.0297 1.8900 0.0650

SSS services are blended 0.0692 0.0292 2.3730 0.0230

Female student 0.0295 0.0128 2.3050 0.0210

High school GPA 0.0620 0.0111 5.6070 0.0000

Percentile on SAT/ACT 0.1309 0.0314 4.1620 0.0000

Took any developmental course -0.0511 0.0139 -3.6810 0.0000

Attended college after first year 0.1635 0.0197 8.3030 0.0000

No parental education beyond high school 0.0029 0.0135 0.2160 0.8290

Family income greater than $20,000 -0.0198 0.0139 -1.4250 0.1540

Feel comfortable on this campus 0.0630 0.0156 4.0250 0.0000

Number of hours: instr. courses 0.0003 0.0003 1.0160 0.3100

Number of hours: profl. tutoring -0.0021 0.0053 -0.3960 0.6920

Number of hours: peer tutoring 0.0026 0.0008 3.3280 0.0010

Number of hours: profl. counseling -0.0109 0.0034 -3.2410 0.0020

Number of hours: peer counseling -0.0009 0.0127 -0.0730 0.9420

Number of hours: labs 0.0003 0.0014 0.2480 0.8050

Number of hours: workshops 0.0051 0.0036 1.4090 0.1590

Number of hours: cultural events 0.0029 0.0055 0.5180 0.6040

Number of hours: handicapped srvc. 0.0058 0.0020 2.8930 0.0040

Services for physically disabled 0.0343 0.0209 1.6410 0.1000

Services for limited-English ability 0.0266 0.0353 0.7550 0.4500

Student orientation 0.0127 0.0173 0.7330 0.4640

College reentrance counseling -0.0184 0.0172 -1.0680 0.2860

Classroom instr.--basic skills -0.0190 0.0213 -0.8940 0.3720

Classroom instr.--dvlpmntl. English 0.0032 0.0241 0.1320 0.8960

Classroom instr.--dvlpmntl. math -0.0048 0.0214 -0.2250 0.8220

Cultural enrichment activities -0.0022 0.0032 -0.7050 0.4810

Referrals to agencies/resources -0.0198 0.0169 -1.1750 0.2400

# of counseling sessions in 1991–92 -0.0022 0.0016 -1.4050 0.1600

# of counseling sessions in 1992–93 -0.0005 0.0016 -0.2880 0.7730

# of counseling sessions in 1993–94 0.0036 0.0016 2.2370 0.0250

# of tutoring sessions in 1991–92 -0.0010 0.0009 -1.0520 0.2930

# of tutoring sessions in 1992–93 0.0017 0.0010 1.7350 0.0820

# of tutoring sessions in 1993–94 0.0025 0.0011 2.1610 0.0300

Some tutoring in years 4–6 0.0015 0.0020 0.7350 0.4620

Some counseling in years 4–6 0.0113 0.0022 5.0640 0.0000

Continues to next page.
89

Continued from previous page.

Some services for disabled in years 4–6 -0.0735 0.0327 -2.2450 0.0250

Field trip in years 4–6 0.0740 0.0158 4.6990 0.0000

Contact with support services in years 4–6 0.0177 0.0018 9.7580 0.0000

Missing data on high school GPA 0.1997 0.0449 4.4490 0.0000

Missing data on SAT/ACT 0.0361 0.0197 1.8270 0.0670

Missing data on developmental courses -0.1330 0.0479 -2.7750 0.0060

Missing data for hours on school activities -0.0070 0.0318 -0.2200 0.8260

Missing data on first generation 0.0206 0.0297 0.6930 0.4880

Missing data on income -0.0148 0.0207 -0.7160 0.4740

Missing data on comfort on campus -0.0059 0.0514 -0.1150 0.9090

Missing data on services in later years 0.0615 0.0406 1.5150 0.1290

Missing data on services for limited-English -0.4768 0.2914 -1.6370 0.1010

Missing data on student orientation 0.0104 0.0714 0.1450 0.8850

Missing data on reentrance counseling 0.0076 0.0416 0.1820 0.8560

Missing data on classroom instr.--basic skills 0.0391 0.0601 0.6510 0.5150

Missing data on cultural enrichment activities 0.2479 0.1411 1.7570 0.0790

Missing data on referrals to agencies/resources 0.1910 0.3226 0.5920 0.5540

Missing data on tutoring in 1991–92 0.0173 0.0447 0.3870 0.6990

Missing data on tutoring in years 4–6 -0.0060 0.0766 -0.0780 0.9380

Missing data on counseling in years 4–6 -0.0510 0.0523 -0.9760 0.3300

Missing data on services for disabled in years 4–6 -0.0059 0.0772 -0.0770 0.9390

Missing data on field trip in years 4–6 0.0540 0.1070 0.5050 0.6130

Missing data on contact with support services in yrs.

4–6 0.0102 0.1045 0.0970 0.9230

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
90

Exhibit A-11. HLM analysis to predict retention or baccalaureate degree completion 

at any institution six years after freshman entry in 1991–92 using 

continuous measures of participation and no measures of propensity to 

receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.6167 0.0101 61.2400 0.0000

SSS services are home-based 0.0766 0.0269 2.8420 0.0070

Two-year institution -0.1518 0.0262 -5.7910 0.0000

Doctorate-granting institution 0.0323 0.0259 1.2470 0.2200

No SSS programs at institution 0.0701 0.0276 2.5370 0.0150

SSS services are blended 0.0647 0.0266 2.4300 0.0200

Female student 0.0463 0.0131 3.5450 0.0010

High school GPA 0.0459 0.0114 4.0430 0.0000

Percentile on SAT/ACT 0.0827 0.0317 2.6110 0.0090

Took any developmental course -0.0526 0.0138 -3.8160 0.0000

Hours on school activities 0.0005 0.0004 1.2350 0.2170

Attended college after first year 0.3491 0.0234 14.9450 0.0000

No parental education beyond high school -0.0468 0.0136 -3.4370 0.0010

Family income greater than $20,000 0.0008 0.0141 0.0590 0.9530

Above average academic ability 0.0114 0.0130 0.8740 0.3820

Number of hours: instr. courses -0.0002 0.0003 -0.8800 0.3790

Number of hours: profl. tutoring -0.0026 0.0057 -0.4580 0.6470

Number of hours: peer tutoring 0.0029 0.0008 3.7890 0.0000

Number of hours: profl. counseling -0.0015 0.0034 -0.4320 0.6650

Number of hours: peer counseling -0.0083 0.0129 -0.6460 0.5180

Number of hours: labs -0.0006 0.0014 -0.4640 0.6420

Number of hours: workshops -0.0003 0.0036 -0.0860 0.9320

Number of hours: cultural events -0.0014 0.0054 -0.2620 0.7930

Number of hours: handicapped srvc. 0.0017 0.0019 0.9320 0.3520

Services for physically disabled -0.0141 0.0238 -0.5910 0.5540

Services for limited-English ability 0.0647 0.0370 1.7460 0.0800

Student orientation 0.0170 0.0177 0.9640 0.3350

College reentrance counseling 0.0052 0.0191 0.2730 0.7850

Classroom instr.--basic skills -0.0068 0.0216 -0.3150 0.7530

Classroom instr.--dvlpmntl. English -0.0237 0.0243 -0.9770 0.3290

Classroom instr.--dvlpmntl. math -0.0137 0.0217 -0.6330 0.5270

Cultural enrichment activities 0.0030 0.0035 0.8450 0.3980

Referrals to agencies/resources -0.0280 0.0170 -1.6440 0.1000

# of counseling sessions in 1991–92 -0.0032 0.0016 -1.9630 0.0490

# of counseling sessions in 1992–93 -0.0015 0.0015 -1.0020 0.3170

# of counseling sessions in 1993–94 0.0033 0.0015 2.1780 0.0290

# of tutoring sessions in 1991–92 -0.0011 0.0009 -1.1760 0.2400

# of tutoring sessions in 1992–93 0.0005 0.0010 0.5460 0.5850

# of tutoring sessions in 1993–94 0.0047 0.0011 4.1940 0.0000

Some tutoring in years 4–6 -0.0016 0.0019 -0.8310 0.4060

Continues to next page.
91

Continued from previous page.

Some counseling in years 4–6 0.0209 0.0021 10.0440 0.0000

Some services for disabled in years 4–6 -0.0308 0.0306 -1.0060 0.3150

Field trip in years 4–6 0.1337 0.0147 9.0900 0.0000

Contact with support services in years 4–6 0.0193 0.0017 11.3650 0.0000

Missing data on full-time status 0.0064 0.0662 0.0960 0.9240

Missing data on high school GPA 0.1174 0.0456 2.5760 0.0100

Missing data on SAT/ACT -0.0267 0.0200 -1.3360 0.1820

Missing data on developmental courses 0.0302 0.0540 0.5590 0.5760

Missing data for hours on school activities 0.0017 0.0384 0.0450 0.9640

Missing data on first generation -0.0338 0.0334 -1.0110 0.3130

Missing data on income -0.0159 0.0214 -0.7430 0.4580

Missing data on perceived ability 0.0640 0.0563 1.1370 0.2560

Missing data on B32 -0.0522 0.0468 -1.1150 0.2650

Missing data on services for limited-English -0.1851 0.2716 -0.6810 0.4950

Missing data on student orientation 0.0444 0.0743 0.5970 0.5500

Missing data on reentrance counseling 0.0125 0.0406 0.3070 0.7580

Missing data on classroom instr.--basic skills -0.0182 0.0603 -0.3020 0.7630

Missing data on cultural enrichment activities -0.2675 0.1320 -2.0260 0.0420

Missing data on referrals to agencies/resources 0.4141 0.3010 1.3760 0.1690

Missing data on tutoring in 1991–92 -0.0901 0.0572 -1.5760 0.1150

Missing data on tutoring in years 4–6 0.1907 0.0720 2.6510 0.0080

Missing data on counseling in years 4–6 -0.0330 0.0487 -0.6760 0.4990

Missing data on services for disabled in years 4–6 -0.1130 0.0719 -1.5720 0.1160

Missing data on field trip in years 4–6 0.2344 0.0997 2.3510 0.0190

Missing data on contact with support services in yrs.

4–6 0.1208 0.0975 1.2400 0.2150

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
92

Exhibit A-12. HLM analysis to predict baccalaureate degree completion (or higher) at any 

institution six years after freshman entry in 1991–92 using continuous 

measures of participation and no measures of propensity to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.3973 0.0137 28.9700 0.0000

SSS services are home-based 0.0520 0.0370 1.4060 0.1670

Two-year institution -0.1663 0.0351 -4.7320 0.0000

Doctorate-granting institution 0.0633 0.0358 1.7660 0.0840

No SSS programs at institution 0.0413 0.0370 1.1160 0.2710

SSS services are blended 0.0251 0.0377 0.6650 0.5090

Black student -0.0411 0.0191 -2.1500 0.0310

Hispanic student -0.0047 0.0241 -0.1960 0.8450

American Indian student -0.1626 0.0549 -2.9600 0.0040

Asian student 0.0989 0.0337 2.9330 0.0040

Student was female 0.0249 0.0136 1.8260 0.0670

Live in college housing or frat./sorority 0.0580 0.0168 3.4560 0.0010

High school GPA 0.0960 0.0119 8.0580 0.0000

Percentile on SAT/ACT 0.1474 0.0334 4.4130 0.0000

Took any developmental course -0.0875 0.0143 -6.1000 0.0000

Hours on school activities 0.0011 0.0004 2.4640 0.0140

Attended college after first year 0.1448 0.0242 5.9870 0.0000

No parental education beyond high school -0.0229 0.0143 -1.6020 0.1090

Family income greater than $20,000 0.0161 0.0149 1.0800 0.2810

Have major concerns about ed. finances -0.0451 0.0136 -3.3200 0.0010

Once I start something, I finish it 0.0424 0.0151 2.8080 0.0050

Number of hours: instr. courses 0.0002 0.0003 0.6140 0.5390

Number of hours: profl. tutoring 0.0008 0.0060 0.1390 0.8900

Number of hours: peer tutoring 0.0023 0.0008 2.8690 0.0050

Number of hours: profl. counseling -0.0089 0.0035 -2.4980 0.0130

Number of hours: peer counseling -0.0011 0.0139 -0.0790 0.9370

Number of hours: labs -0.0004 0.0015 -0.2420 0.8090

Number of hours: workshops -0.0012 0.0038 -0.3210 0.7480

Number of hours: cultural events 0.0052 0.0057 0.9130 0.3620

Number of hours: handicapped srvc. -0.0004 0.0020 -0.2050 0.8380

Services for physically disabled 0.0179 0.0247 0.7260 0.4680

Services for limited-English ability 0.0098 0.0391 0.2510 0.8020

Student orientation 0.0083 0.0186 0.4490 0.6530

College reentrance counseling -0.0192 0.0198 -0.9700 0.3330

Classroom instr.--basic skills -0.0048 0.0225 -0.2150 0.8300

Classroom instr.--dvlpmntl. English -0.0010 0.0254 -0.0410 0.9680

Classroom instr.--dvlpmntl. math -0.0150 0.0227 -0.6610 0.5090

Cultural enrichment activities -0.0016 0.0036 -0.4530 0.6500

Referrals to agencies/resources -0.0035 0.0179 -0.1950 0.8460

# of counseling sessions in 1991–92 -0.0033 0.0017 -1.9530 0.0500

# of counseling sessions in 1992–93 0.0021 0.0016 1.3360 0.1820

Continues to next page.
93

Continued from previous page.

# of counseling sessions in 1993–94 0.0041 0.0016 2.5750 0.0100

# of tutoring sessions in 1991–92 0.0005 0.0010 0.4700 0.6380

# of tutoring sessions in 1992–93 0.0011 0.0010 1.0490 0.2940

# of tutoring sessions in 1993–94 0.0003 0.0012 0.2270 0.8210

Some tutoring in years 4–6 -0.0042 0.0020 -2.1470 0.0320

Some counseling in years 4–6 0.0118 0.0022 5.4970 0.0000

Some services for disabled in years 4–6 -0.1077 0.0316 -3.4100 0.0010

Field trip in years 4–6 0.1345 0.0152 8.8600 0.0000

Contact with support services in years 4–6 0.0163 0.0018 9.2790 0.0000

Missing data on student race 0.0205 0.0531 0.3860 0.6990

Missing data on college housing 0.1069 0.0781 1.3690 0.1710

Missing data on high school GPA 0.2676 0.0481 5.5660 0.0000

Missing data on SAT/ACT -0.0050 0.0211 -0.2390 0.8110

Missing data on developmental courses -0.0298 0.0575 -0.5180 0.6040

Missing data for hours on school activities 0.0468 0.0402 1.1650 0.2450

Missing data on first generation -0.0530 0.0351 -1.5080 0.1310

Missing data on income 0.0216 0.0225 0.9630 0.3360

Missing data on financial concerns -0.0428 0.0645 -0.6630 0.5070

Missing data on finishing tasks -0.0294 0.0557 -0.5280 0.5970

Missing data on B32 0.0144 0.0503 0.2850 0.7750

Missing data on services for limited-English 0.5156 0.2800 1.8410 0.0650

Missing data on student orientation 0.0193 0.0771 0.2510 0.8020

Missing data on reentrance counseling 0.0088 0.0422 0.2090 0.8340

Missing data on classroom instr.--basic skills 0.0192 0.0622 0.3090 0.7580

Missing data on cultural enrichment activities -0.1149 0.1364 -0.8420 0.4000

Missing data on referrals to agencies/resources -0.4777 0.3103 -1.5400 0.1230

Missing data on tutoring in 1991–92 -0.0837 0.0617 -1.3560 0.1750

Missing data on tutoring in years 4–6 0.1515 0.0748 2.0260 0.0420

Missing data on counseling in years 4–6 0.0394 0.0503 0.7830 0.4330

Missing data on services for disabled in years 4–6 -0.1299 0.0742 -1.7500 0.0800

Missing data on field trip in years 4–6 0.3293 0.1033 3.1890 0.0020

Missing data on contact with support services in yrs.

4–6 0.0853 0.1007 0.8470 0.3970

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
94

Exhibit A-13. HLM analysis to predict associate’s degree completion (or higher) at any 

institution six years after freshman entry in 1991–92 using continuous 

measures of participation and no measures of propensity to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.5381 0.0148 36.3290 0.0000

SSS services are home-based 0.0258 0.0398 0.6490 0.5200

Two-year institution 0.0005 0.0372 0.0120 0.9900

Doctorate-granting institution 0.0523 0.0388 1.3490 0.1850

No SSS programs at institution 0.0099 0.0399 0.2480 0.8050

SSS services are blended -0.0124 0.0408 -0.3050 0.7620

Black student -0.0929 0.0202 -4.5970 0.0000

Hispanic student -0.0658 0.0259 -2.5350 0.0120

American Indian student -0.2512 0.0593 -4.2380 0.0000

Asian student 0.0385 0.0363 1.0610 0.2890

Student was female 0.0536 0.0148 3.6190 0.0010

Student was full-time 0.0939 0.0265 3.5400 0.0010

High school GPA 0.1035 0.0129 8.0380 0.0000

Took any developmental course -0.0891 0.0153 -5.8220 0.0000

Attended college after first year 0.2742 0.0262 10.4650 0.0000

No parental education beyond high school -0.0221 0.0154 -1.4290 0.1530

Family income greater than $20,000 0.0160 0.0161 0.9940 0.3210

Above average academic ability 0.0345 0.0146 2.3670 0.0180

Have major concerns about ed. finances -0.0391 0.0147 -2.6620 0.0080

Number of hours: instr. courses 0.0002 0.0003 0.4710 0.6370

Number of hours: profl. tutoring -0.0066 0.0064 -1.0400 0.2990

Number of hours: peer tutoring 0.0021 0.0009 2.4340 0.0150

Number of hours: profl. counseling -0.0088 0.0039 -2.2820 0.0220

Number of hours: peer counseling 0.0008 0.0150 0.0510 0.9600

Number of hours: labs -0.0046 0.0016 -2.8440 0.0050

Number of hours: workshops 0.0062 0.0041 1.5050 0.1320

Number of hours: cultural events 0.0010 0.0062 0.1550 0.8770

Number of hours: handicapped srv. 0.0014 0.0021 0.6790 0.4970

Services for physically disabled 0.0115 0.0261 0.4410 0.6590

Services for limited-English ability 0.0065 0.0421 0.1540 0.8780

Student orientation 0.0179 0.0201 0.8930 0.3720

College reentrance counseling -0.0084 0.0214 -0.3900 0.6960

Classroom instr.--basic skills -0.0145 0.0244 -0.5960 0.5510

Classroom instr.--dvlpmntl. English -0.0100 0.0275 -0.3620 0.7170

Classroom instr.--dvlpmntl. math -0.0221 0.0246 -0.8980 0.3690

Cultural enrichment activities -0.0016 0.0039 -0.4030 0.6860

Referrals to agencies/resources 0.0153 0.0193 0.7940 0.4270

# of counseling sessions in 1991–92 -0.0018 0.0018 -0.9660 0.3340

# of counseling sessions in 1992–93 0.0007 0.0017 0.4050 0.6850

# of counseling sessions in 1993–94 0.0052 0.0017 3.0190 0.0030

# of tutoring sessions in 1991–92 0.0004 0.0011 0.3640 0.7160

Continues to next page.
95

Continued from previous page.

# of tutoring sessions in 1992–93 0.0013 0.0011 1.1350 0.2570

# of tutoring sessions in 1993–94 0.0002 0.0013 0.1790 0.8580

Some tutoring in years 4–6 -0.0052 0.0021 -2.4370 0.0150

Some counseling in years 4–6 0.0088 0.0023 3.7350 0.0000

Some services for disabled in years 4–6 -0.1053 0.0344 -3.0620 0.0030

Field trip in years 4–6 0.1188 0.0165 7.1870 0.0000

Contact with support services in years 4–6 0.0179 0.0019 9.4140 0.0000

Missing data on student race -0.0624 0.0577 -1.0810 0.2800

Missing data on full-time status 0.1267 0.0800 1.5830 0.1130

Missing data on high school GPA 0.2653 0.0519 5.1100 0.0000

Missing data on developmental courses -0.1008 0.0625 -1.6130 0.1060

Missing data on first generation -0.0720 0.0380 -1.8930 0.0580

Missing data on income 0.0111 0.0243 0.4570 0.6470

Missing data on perceived ability 0.0441 0.0674 0.6550 0.5120

Missing data on financial concerns 0.0093 0.0700 0.1320 0.8950

Missing data on B32 0.0174 0.0538 0.3240 0.7460

Missing data on services for limited-English 0.1449 0.3052 0.4750 0.6350

Missing data on student orientation 0.0142 0.0834 0.1710 0.8650

Missing data on reentrance counseling -0.0060 0.0458 -0.1320 0.8950

Missing data on classroom instr.--basic skills 0.0349 0.0677 0.5160 0.6060

Missing data on cultural enrichment activities -0.1672 0.1486 -1.1260 0.2610

Missing data on referrals to agencies/resources -0.0284 0.3382 -0.0840 0.9340

Missing data on tutoring in 1991–92 -0.0116 0.0672 -0.1730 0.8630

Missing data on tutoring in years 4–6 0.1392 0.0818 1.7020 0.0880

Missing data on counseling in years 4–6 0.0586 0.0548 1.0690 0.2850

Missing data on services for disabled in years 4–6 0.0078 0.0809 0.0970 0.9230

Missing data on field trip in years 4–6 0.1999 0.1123 1.7800 0.0750

Missing data on contact with support services in yrs.

4–6 0.0568 0.1097 0.5180 0.6040

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
96

Exhibit A-14. HLM analysis to predict transfers from two-year to four-year institutions six 

years after freshman entry in 1991–92 using continuous measures of 

participation and no measures of propensity to receive services

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.2155 0.0175 12.3210 0.0000

SSS services are home-based 0.0473 0.0603 0.7830 0.4490

No SSS programs at institution 0.0868 0.0438 1.9820 0.0700

SSS services are blended 0.0142 0.0491 0.2890 0.7770

Female -0.0837 0.0245 -3.4150 0.0010

High school GPA 0.0531 0.0178 2.9920 0.0030

Took any developmental course -0.0555 0.0253 -2.1990 0.0280

Attended college after first year 0.0923 0.0294 3.1430 0.0020

No parental education beyond high school -0.1112 0.0255 -4.3570 0.0000

Family income greater than $20,000 -0.0106 0.0251 -0.4220 0.6720

Once I start something, I finish it -0.0337 0.0261 -1.2910 0.1970

Number of hours: instr. courses -0.0003 0.0008 -0.3800 0.7040

Number of hours: profl. tutoring -0.0159 0.0091 -1.7430 0.0810

Number of hours: peer tutoring 0.0021 0.0017 1.2020 0.2300

Number of hours: profl. counseling 0.0097 0.0060 1.6040 0.1080

Number of hours: peer counseling 0.0572 0.0565 1.0130 0.3120

Number of hours: labs 0.0008 0.0015 0.5440 0.5860

Number of hours: workshops -0.0196 0.0084 -2.3370 0.0200

Number of hours: cultural events -0.0388 0.0170 -2.2870 0.0220

Number of hours: handicapped srvc. -0.0060 0.0031 -1.9520 0.0510

Services for physically disabled 0.0202 0.0328 0.6160 0.5370

Services for limited-English ability -0.0480 0.0564 -0.8520 0.3940

Student orientation 0.0096 0.0256 0.3730 0.7080

College reentrance counseling 0.0542 0.0261 2.0750 0.0380

Classroom instr.--basic skills -0.0690 0.0392 -1.7590 0.0780

Classroom instr.--dvlpmntl. English 0.0332 0.0417 0.7970 0.4260

Classroom instr.--dvlpmntl. math -0.0769 0.0383 -2.0080 0.0440

Cultural enrichment activities -0.0134 0.0181 -0.7420 0.4580

Referrals to agencies/resources -0.0028 0.0328 -0.0870 0.9310

# of counseling sessions in 1991–92 -0.0044 0.0031 -1.3850 0.1660

# of counseling sessions in 1992–93 -0.0052 0.0030 -1.7260 0.0840

# of counseling sessions in 1993–94 0.0162 0.0035 4.6120 0.0000

# of tutoring sessions in 1991–92 0.0030 0.0022 1.3730 0.1700

# of tutoring sessions in 1992–93 -0.0043 0.0023 -1.8530 0.0630

# of tutoring sessions in 1993–94 0.0078 0.0024 3.2140 0.0020

Some tutoring in years 4–6 -0.0030 0.0045 -0.6820 0.4950

Some counseling in years 4–6 0.0303 0.0051 5.9350 0.0000

Some services for disabled in years 4–6 0.0233 0.0647 0.3600 0.7190

Field trip in years 4–6 0.1411 0.0368 3.8400 0.0000

Contact with support services in years 4–6 0.0064 0.0042 1.5150 0.1300

Missing data on high school GPA 0.0741 0.0676 1.0970 0.2730

Continues to next page.
97

Continued from next page.

Missing data on developmental courses 0.0304 0.0798 0.3810 0.7030

Missing data on first generation -0.0722 0.0465 -1.5530 0.1200

Missing data on income -0.0016 0.0344 -0.0450 0.9640

Missing data on finishing tasks -0.0670 0.0665 -1.0070 0.3140

Missing data on B32 -0.0618 0.0673 -0.9190 0.3580

Missing data on services for limited-English -0.0977 0.1664 -0.5870 0.5570

Missing data on student orientation 0.0671 0.1350 0.4970 0.6180

Missing data on reentrance counseling 0.0183 0.0668 0.2730 0.7850

Missing data on classroom instr.--basic skills -0.0155 0.1661 -0.0930 0.9260

Missing data on tutoring in 1991–92 -0.0855 0.0704 -1.2150 0.2250

Missing data on tutoring in years 4–6 0.1713 0.1432 1.1970 0.2320

Missing data on counseling in years 4–6 -0.1320 0.0814 -1.6210 0.1050

Missing data on services for disabled in years 4–6 -0.1946 0.1608 -1.2100 0.2270

Missing data on field trip in years 4–6 0.3026 0.2087 1.4500 0.1470

Missing data on contact with support services in yrs.

4–6 -0.2193 0.2083 -1.0530 0.2930

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
98

Exhibit A-15. HLM analysis to predict cumulative GPAs six years after freshman entry in 

1991–92 using continuous measures of participation and adding propensity 

scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 2.4096 0.0251 95.9680 0.0000

SSS services are home-based 0.1059 0.0677 1.5640 0.1250

Two-year institution 0.1204 0.0635 1.8960 0.0650

Doctorate-granting institution -0.1433 0.0938 -1.5280 0.1340

No SSS programs at institution 0.0283 0.0682 0.4140 0.6810

SSS services are blended -0.0524 0.0697 -0.7510 0.4570

Age in 1991 0.0274 0.0020 13.3710 0.0000

Black student -0.3484 0.0374 -9.3130 0.0000

Hispanic student -0.1754 0.0655 -2.6760 0.0080

American Indian student -0.2598 0.0900 -2.8880 0.0040

Asian student -0.0920 0.0662 -1.3890 0.1650

Female student 0.1384 0.0220 6.2780 0.0000

High school GPA 0.2885 0.0199 14.5140 0.0000

Percentile on SAT/ACT 0.2007 0.0549 3.6570 0.0000

Took any developmental course -0.1403 0.0235 -5.9650 0.0000

Hours on school activities 0.0035 0.0007 5.0620 0.0000

Attended college after first year 0.7135 0.0339 21.0210 0.0000

No parental education beyond high school 0.0208 0.0242 0.8620 0.3890

Family income greater than $20,000 0.0279 0.0265 1.0550 0.2920

Above average academic ability 0.1446 0.0249 5.8090 0.0000

Would attend free tutoring -0.0803 0.0265 -3.0320 0.0030

Feel comfortable on this campus 0.0590 0.0274 2.1550 0.0310

Number of hours: instr. courses -0.0009 0.0007 -1.3460 0.1780

Number of hours: profl. tutoring 0.0205 0.0478 0.4290 0.6680

Number of hours: peer tutoring -0.0005 0.0027 -0.1690 0.8660

Number of hours: profl. counseling 0.0052 0.0124 0.4180 0.6760

Number of hours: peer counseling -0.0452 0.0305 -1.4850 0.1370

Number of hours: labs 0.0096 0.0040 2.4360 0.0150

Number of hours: workshops 0.0207 0.0111 1.8690 0.0610

Number of hours: cultural events -0.0255 0.0459 -0.5570 0.5770

Number of hours: handicapped srvc. 0.0077 0.0097 0.7930 0.4280

Propensity to receive instr. courses 0.0002 0.0009 0.1990 0.8420

Propensity to receive profl. tutoring 0.0109 0.0274 0.3990 0.6900

Propensity to receive peer tutoring 0.0010 0.0027 0.3710 0.7100

Propensity to receive profl. counseling -0.0109 0.0270 -0.4040 0.6860

Propensity to receive peer counseling 0.0357 0.0195 1.8290 0.0670

Propensity to receive labs -0.0011 0.0029 -0.3660 0.7140

Propensity to receive workshops -0.0056 0.0067 -0.8390 0.4010

Propensity to receive cultural events 0.0066 0.0167 0.3970 0.6910

Propensity to receive handicapped srvc. 0.0031 0.0028 1.1160 0.2650

Services for physically disabled -0.0042 0.0345 -0.1210 0.9040

Continues to next page.
99

Continued from previous page.

Services for limited-English ability 0.0272 0.0605 0.4490 0.6530

Student orientation -0.0403 0.0296 -1.3620 0.1730

College reentrance counseling -0.0479 0.0292 -1.6420 0.1000

Classroom instr.--basic skills -0.0570 0.0359 -1.5880 0.1120

Classroom instr.--dvlpmntl. English 0.0005 0.0409 0.0130 0.9900

Classroom instr.--dvlpmntl. math -0.0014 0.0362 -0.0390 0.9690

Cultural enrichment activities 0.0128 0.0052 2.4590 0.0140

Referrals to agencies/resources 0.0607 0.0286 2.1210 0.0340

# of counseling sessions in 1991–92 -0.0040 0.0028 -1.4600 0.1440

# of counseling sessions in 1992–93 -0.0006 0.0027 -0.2210 0.8250

# of counseling sessions in 1993–94 0.0084 0.0027 3.1170 0.0020

# of tutoring sessions in 1991–92 -0.0012 0.0016 -0.7230 0.4700

# of tutoring sessions in 1992–93 0.0004 0.0017 0.2310 0.8180

# of tutoring sessions in 1993–94 0.0010 0.0019 0.5410 0.5880

Some tutoring in years 4–6 -0.0111 0.0034 -3.2530 0.0020

Some counseling in years 4–6 0.0114 0.0037 3.0550 0.0030

Some services for disabled in years 4–6 -0.0625 0.0544 -1.1490 0.2510

Field trip in years 4–6 0.1279 0.0266 4.8070 0.0000

Contact with support services in years 4–6 0.0159 0.0031 5.1840 0.0000

Missing data on student age 0.5994 0.0871 6.8820 0.0000

Missing data on student race 0.0143 0.0932 0.1540 0.8780

Missing data on high school GPA 0.7643 0.0863 8.8590 0.0000

Missing data on SAT/ACT 0.0318 0.0350 0.9090 0.3640

Missing data on developmental courses -0.1736 0.0812 -2.1390 0.0320

Missing data for hours on school activities 0.1479 0.0584 2.5330 0.0120

Missing data on first generation -0.0885 0.0535 -1.6540 0.0980

Missing data on income -0.0210 0.0372 -0.5650 0.5710

Missing data on perceived ability 0.1212 0.1069 1.1340 0.2570

Missing data on attending free tutoring -0.0017 0.1941 -0.0090 0.9930

Missing data on comfort on campus -0.1926 0.1933 -0.9960 0.3200

Missing data on services in later years 0.2334 0.0688 3.3920 0.0010

Missing data on services for limited-English -0.0591 0.4906 -0.1210 0.9050

Missing data on student orientation -0.0896 0.1243 -0.7210 0.4710

Missing data on reentrance counseling 0.1698 0.0723 2.3500 0.0190

Missing data on classroom instr.--basic skills -0.1490 0.1041 -1.4310 0.1520

Missing data on cultural enrichment activities -0.3003 0.2369 -1.2680 0.2050

Missing data on referrals to agencies/resources 0.4740 0.5429 0.8730 0.3830

Missing data on tutoring in 1991–92 -0.0946 0.0754 -1.2540 0.2100

Missing data on tutoring in years 4–6 -0.0940 0.1313 -0.7160 0.4740

Missing data on counseling in years 4–6 -0.0617 0.0883 -0.6980 0.4850

Missing data on services for disabled in years 4–6 -0.0701 0.1301 -0.5390 0.5890

Missing data on field trip in years 4–6 0.4124 0.1808 2.2810 0.0230

Missing data on contact with support services in yrs.

4–6 -0.2307 0.1762 -1.3090 0.1910

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
100

Exhibit A-16. HLM analysis to predict the number of credits earned six years after 

freshman entry in 1991–92 using continuous measures of participation and 

adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 76.1127 1.8124 41.9950 0.0000

SSS services are home-based 4.5506 4.8904 0.9310 0.3580

Two-year institution -14.5028 4.5225 -3.2070 0.0030

Doctorate-granting institution 10.5614 6.1001 1.7310 0.0900

No SSS programs at institution 3.4131 4.8785 0.7000 0.4880

SSS services are blended -3.1692 5.0891 -0.6230 0.5370

Black student -5.6227 2.2409 -2.5090 0.0120

Hispanic student 1.1283 4.6401 0.2430 0.8080

American Indian student -20.4032 5.2886 -3.8580 0.0000

Asian student 4.2210 4.1520 1.0170 0.3100

Female student 4.5071 1.2706 3.5470 0.0010

Student was full-time 10.6999 4.6545 2.2990 0.0220

High school GPA 10.5388 1.1386 9.2560 0.0000

Percentile on SAT/ACT 10.0531 3.1568 3.1850 0.0020

Took any developmental course -5.7366 1.3569 -4.2280 0.0000

Hours on school activities 0.1775 0.0397 4.4690 0.0000

Attended college after first year 43.8911 1.9326 22.7110 0.0000

No parental education beyond high school -0.5416 1.3889 -0.3900 0.6960

Family income greater than $20,000 -1.2707 1.6757 -0.7580 0.4480

Have major concerns about ed. finances -3.8501 1.2561 -3.0650 0.0030

Once I start something, I finish it 3.7426 1.4220 2.6320 0.0090

Feel comfortable on this campus 7.1031 1.5924 4.4600 0.0000

Number of hours: instr. courses -0.0339 0.0406 -0.8350 0.4040

Number of hours: profl. tutoring -2.9658 2.6790 -1.1070 0.2690

Number of hours: peer tutoring 0.2252 0.1579 1.4270 0.1540

Number of hours: profl. counseling 0.6705 0.7233 0.9270 0.3540

Number of hours: peer counseling 0.4628 1.7842 0.2590 0.7950

Number of hours: labs 0.1704 0.2287 0.7450 0.4560

Number of hours: workshops -0.0199 0.6439 -0.0310 0.9760

Number of hours: cultural events -3.9044 2.6594 -1.4680 0.1420

Number of hours: handicapped srvc. -0.0996 0.5648 -0.1760 0.8600

Propensity to receive instr. courses 0.0626 0.0491 1.2730 0.2030

Propensity to receive profl. tutoring 4.5855 1.5797 2.9030 0.0040

Propensity to receive peer tutoring -0.2307 0.1380 -1.6720 0.0940

Propensity to receive profl. counseling 1.3736 2.7236 0.5040 0.6140

Propensity to receive peer counseling -0.7014 1.1361 -0.6170 0.5370

Propensity to receive labs -0.0658 0.1668 -0.3940 0.6930

Propensity to receive workshops 0.1878 0.3882 0.4840 0.6280

Propensity to receive cultural events 0.0710 0.9661 0.0740 0.9420

Propensity to receive handicapped srvc. -0.0636 0.1636 -0.3880 0.6970

Services for physically disabled 1.7497 1.9916 0.8790 0.3800

Continues to next page.
101

Continued from previous page.

Services for limited-English ability 0.5564 3.4790 0.1600 0.8730

Student orientation -0.9701 1.7039 -0.5690 0.5690

College reentrance counseling -3.6612 1.6849 -2.1730 0.0300

Classroom instr.--basic skills -5.0852 2.0723 -2.4540 0.0140

Classroom instr.--dvlpmntl. English 1.3197 2.3592 0.5590 0.5750

Classroom instr.--dvlpmntl. math -2.2437 2.0880 -1.0750 0.2830

Cultural enrichment activities 0.7130 0.2992 2.3830 0.0170

Referrals to agencies/resources -2.0481 1.6587 -1.2350 0.2170

# of counseling sessions in 1991–92 -0.1042 0.1588 -0.6560 0.5110

# of counseling sessions in 1992–93 0.0058 0.1559 0.0370 0.9710

# of counseling sessions in 1993–94 0.5198 0.1565 3.3210 0.0010

# of tutoring sessions in 1991–92 -0.1417 0.0924 -1.5330 0.1250

# of tutoring sessions in 1992–93 0.5115 0.0976 5.2380 0.0000

# of tutoring sessions in 1993–94 0.2977 0.1107 2.6890 0.0080

Some tutoring in years 4–6 0.0301 0.1982 0.1520 0.8800

Some counseling in years 4–6 1.4482 0.2171 6.6710 0.0000

Some services for disabled in years 4–6 -11.6425 3.1724 -3.6700 0.0000

Field trip in years 4–6 13.0440 1.5407 8.4660 0.0000

Contact with support services in years 4–6 2.1016 0.1777 11.8280 0.0000

Missing data on student race -0.7630 5.8918 -0.1300 0.8970

Missing data on full-time status 19.4546 7.5037 2.5930 0.0100

Missing data on high school GPA 27.2416 4.9852 5.4650 0.0000

Missing data on SAT/ACT -0.0586 1.9861 -0.0290 0.9770

Missing data on developmental courses -1.3646 4.6997 -0.2900 0.7710

Missing data for hours on school activities 5.4407 3.5825 1.5190 0.1290

Missing data on first generation -2.5116 3.1752 -0.7910 0.4290

Missing data on income -0.3377 2.2521 -0.1500 0.8810

Missing data on perceived ability 0.6781 6.2966 0.1080 0.9150

Missing data on financial concerns -9.5040 5.7126 -1.6640 0.0960

Missing data on finishing tasks 10.2830 6.1636 1.6680 0.0950

Missing data on comfort on campus -5.1408 7.6061 -0.6760 0.4990

Missing data on services in later years -1.6456 3.9744 -0.4140 0.6780

Missing data on services for limited-English -17.7201 28.3003 -0.6260 0.5310

Missing data on student orientation 0.5878 7.1507 0.0820 0.9350

Missing data on reentrance counseling 0.5508 4.1750 0.1320 0.8950

Missing data on classroom instr.--basic skills 6.2460 5.8836 1.0620 0.2890

Missing data on cultural enrichment activities -30.0858 13.6867 -2.1980 0.0280

Missing data on referrals to agencies/resources 42.7787 31.3177 1.3660 0.1720

Missing data on tutoring in 1991–92 -4.6920 4.3594 -1.0760 0.2820

Missing data on tutoring in years 4–6 0.1428 7.6608 0.0190 0.9850

Missing data on counseling in years 4–6 -5.5397 5.1007 -1.0860 0.2780

Missing data on services for disabled in years 4–6 -8.8703 7.5542 -1.1740 0.2410

Missing data on field trip in years 4–6 7.4848 10.6760 0.7010 0.4830

Missing data on contact with support services in yrs. 4–6 6.0580 10.3757 0.5840 0.5590

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
102

Exhibit A-17. HLM analysis to predict retention or baccalaureate degree completion at the 

same institution six years after freshman entry in 1991–92 using continuous 

measures of participation and adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.3182 0.0111 28.6840 0.0000

SSS services are home-based 0.0632 0.0298 2.1220 0.0400

Two-year institution -0.2221 0.0288 -7.7180 0.0000

Doctorate-granting institution 0.1084 0.0466 2.3240 0.0250

No SSS programs at institution 0.0590 0.0308 1.9140 0.0620

SSS services are blended 0.0603 0.0300 2.0070 0.0510

Female student 0.0368 0.0130 2.8400 0.0050

High school GPA 0.0623 0.0112 5.5550 0.0000

Percentile on SAT/ACT 0.1144 0.0319 3.5890 0.0010

Took any developmental course -0.0506 0.0139 -3.6500 0.0000

Attended college after first year 0.1640 0.0198 8.2930 0.0000

No parental education beyond high school 0.0048 0.0140 0.3440 0.7310

Family income greater than $20,000 -0.0124 0.0154 -0.8050 0.4210

Feel comfortable on this campus 0.0683 0.0160 4.2610 0.0000

Number of hours: instr. courses 0.0001 0.0004 0.1730 0.8630

Number of hours: profl. tutoring 0.0146 0.0275 0.5330 0.5940

Number of hours: peer tutoring 0.0019 0.0016 1.1990 0.2310

Number of hours: profl. counseling 0.0009 0.0072 0.1230 0.9030

Number of hours: peer counseling 0.0004 0.0176 0.0210 0.9830

Number of hours: labs 0.0010 0.0023 0.4470 0.6540

Number of hours: workshops -0.0110 0.0065 -1.6910 0.0900

Number of hours: cultural events -0.0020 0.0269 -0.0750 0.9410

Number of hours: handicapped srvc 0.0067 0.0057 1.1690 0.2430

Propensity to receive instr. courses 0.0008 0.0004 1.8690 0.0610

Propensity to receive profl. tutoring 0.0122 0.0153 0.8000 0.4240

Propensity to receive peer tutoring -0.0009 0.0014 -0.6850 0.4930

Propensity to receive profl. counseling 0.0128 0.0117 1.0990 0.2720

Propensity to receive peer counseling -0.0119 0.0106 -1.1140 0.2660

Propensity to receive labs -0.0009 0.0016 -0.5600 0.5750

Propensity to receive workshops 0.0029 0.0038 0.7770 0.4370

Propensity to receive cultural events 0.0072 0.0096 0.7430 0.4570

Propensity to receive handicapped srvc. 0.0002 0.0016 0.1380 0.8910

Services for physically disabled 0.0279 0.0205 1.3630 0.1730

Services for limited-English ability 0.0270 0.0356 0.7600 0.4470

Student orientation 0.0068 0.0174 0.3910 0.6960

College reentrance counseling -0.0100 0.0173 -0.5810 0.5610

Classroom instr.--basic skills -0.0162 0.0213 -0.7600 0.4470

Classroom instr.--dvlpmntl. English 0.0053 0.0242 0.2210 0.8250

Classroom instr.--dvlpmntl. math -0.0063 0.0214 -0.2930 0.7700

Cultural enrichment activities -0.0008 0.0031 -0.2450 0.8070

Referrals to agencies/resources -0.0225 0.0169 -1.3290 0.1840

Continues to next page.
103

Continued from previous page.

# of counseling sessions in 1991–92 -0.0021 0.0016 -1.2870 0.1980

# of counseling sessions in 1992–93 -0.0009 0.0016 -0.5480 0.5830

# of counseling sessions in 1993–94 0.0038 0.0016 2.3700 0.0180

# of tutoring sessions in 1991–92 -0.0014 0.0009 -1.5020 0.1330

# of tutoring sessions in 1992–93 0.0020 0.0010 1.9940 0.0460

# of tutoring sessions in 1993–94 0.0024 0.0011 2.1430 0.0320

Some tutoring in years 4–6 0.0016 0.0020 0.7870 0.4310

Some counseling in years 4–6 0.0110 0.0022 4.9310 0.0000

Some services for disabled in years 4–6 -0.0597 0.0321 -1.8600 0.0620

Field trip in years 4–6 0.0737 0.0158 4.6660 0.0000

Contact with support services in years 4–6 0.0176 0.0018 9.6740 0.0000

Missing data on high school GPA 0.2100 0.0482 4.3550 0.0000

Missing data on SAT/ACT 0.0343 0.0200 1.7160 0.0860

Missing data on developmental courses -0.1286 0.0481 -2.6750 0.0080

Missing data for hours on school activities -0.0081 0.0320 -0.2520 0.8010

Missing data on first generation 0.0302 0.0308 0.9800 0.3270

Missing data on income -0.0038 0.0216 -0.1760 0.8610

Missing data on comfort on campus -0.0010 0.0520 -0.0190 0.9850

Missing data on services in later years 0.0610 0.0407 1.4990 0.1340

Missing data on services for limited-English -0.4886 0.2921 -1.6730 0.0940

Missing data on student orientation 0.0232 0.0718 0.3230 0.7460

Missing data on reentrance counseling 0.0197 0.0423 0.4660 0.6410

Missing data on classroom instr.--basic skills 0.0296 0.0601 0.4930 0.6220

Missing data on cultural enrichment activities 0.2612 0.1408 1.8560 0.0630

Missing data on referrals to agencies/resources 0.1782 0.3231 0.5520 0.5810

Missing data on tutoring in 1991–92 0.0069 0.0447 0.1550 0.8770

Missing data on tutoring in years 4–6 -0.0046 0.0768 -0.0600 0.9530

Missing data on counseling in years 4–6 -0.0517 0.0524 -0.9870 0.3240

Missing data on services for disabled in years 4–6 -0.0082 0.0774 -0.1060 0.9160

Missing data on field trip in years 4–6 0.0505 0.1072 0.4710 0.6370

Missing data on contact with support services in yrs. 4–6 0.0119 0.1048 0.1130 0.9100

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
104

Exhibit A-18. HLM analysis to predict retention or baccalaureate degree completion at any 

institution six years after freshman entry in 1991–92 using continuous 

measures of participation and adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.6196 0.0099 62.6400 0.0000

SSS services are home-based 0.0766 0.0266 2.8760 0.0070

Two-year institution -0.1315 0.0262 -5.0220 0.0000

Doctorate-granting institution 0.1365 0.0437 3.1240 0.0040

No SSS programs at institution 0.0675 0.0278 2.4250 0.0200

SSS services are blended 0.0437 0.0264 1.6580 0.1050

Female student 0.0540 0.0132 4.0980 0.0000

High school GPA 0.0426 0.0115 3.6920 0.0000

Percentile on SAT/ACT 0.0939 0.0319 2.9410 0.0040

Took any developmental course -0.0525 0.0138 -3.8140 0.0000

Hours on school activities 0.0005 0.0004 1.1720 0.2420

Attended college after first year 0.3501 0.0234 14.9680 0.0000

No parental education beyond high school -0.0508 0.0141 -3.5960 0.0010

Family income greater than $20,000 0.0114 0.0155 0.7340 0.4630

Above average academic ability 0.0211 0.0139 1.5180 0.1290

Number of hours: instr. courses -0.0006 0.0004 -1.5740 0.1150

Number of hours: profl. tutoring 0.0079 0.0284 0.2780 0.7810

Number of hours: peer tutoring 0.0010 0.0016 0.6170 0.5370

Number of hours: profl. counseling 0.0089 0.0071 1.2500 0.2120

Number of hours: peer counseling -0.0008 0.0173 -0.0440 0.9650

Number of hours: labs 0.0009 0.0024 0.3630 0.7160

Number of hours: workshops -0.0070 0.0064 -1.0960 0.2740

Number of hours: cultural events -0.0219 0.0267 -0.8200 0.4120

Number of hours: handicapped srvc. -0.0007 0.0059 -0.1170 0.9070

Propensity to receive instr. courses 0.0015 0.0004 3.4700 0.0010

Propensity to receive profl. tutoring 0.0224 0.0166 1.3480 0.1780

Propensity to receive peer tutoring -0.0005 0.0014 -0.3500 0.7260

Propensity to receive profl. counseling -0.0121 0.0125 -0.9750 0.3300

Propensity to receive peer counseling 0.0070 0.0106 0.6670 0.5050

Propensity to receive labs -0.0016 0.0017 -0.9350 0.3500

Propensity to receive workshops -0.0018 0.0037 -0.4750 0.6340

Propensity to receive cultural events -0.0074 0.0095 -0.7810 0.4350

Propensity to receive handicapped srvc. 0.0002 0.0016 0.1140 0.9100

Services for physically disabled -0.0090 0.0231 -0.3910 0.6960

Services for limited-English ability 0.0490 0.0374 1.3110 0.1900

Student orientation 0.0143 0.0177 0.8100 0.4180

College reentrance counseling 0.0123 0.0191 0.6440 0.5190

Classroom instr.--basic skills -0.0059 0.0216 -0.2740 0.7840

Classroom instr.--dvlpmntl. English -0.0221 0.0243 -0.9080 0.3640

Classroom instr.--dvlpmntl. math -0.0166 0.0217 -0.7660 0.4440

Cultural enrichment activities 0.0029 0.0035 0.8380 0.4020

Continues to next page.
105

Continued from previous page.

Referrals to agencies/resources -0.0340 0.0171 -1.9940 0.0460

# of counseling sessions in 1991–92 -0.0027 0.0016 -1.6500 0.0990

# of counseling sessions in 1992–93 -0.0020 0.0015 -1.3150 0.1890

# of counseling sessions in 1993–94 0.0034 0.0015 2.1990 0.0280

# of tutoring sessions in 1991–92 -0.0016 0.0009 -1.6800 0.0930

# of tutoring sessions in 1992–93 0.0008 0.0010 0.8440 0.3990

# of tutoring sessions in 1993–94 0.0048 0.0011 4.3290 0.0000

Some tutoring in years 4–6 -0.0016 0.0019 -0.8320 0.4060

Some counseling in years 4–6 0.0208 0.0021 9.9940 0.0000

Some services for disabled in years 4–6 -0.0191 0.0299 -0.6390 0.5230

Field trip in years 4–6 0.1320 0.0147 8.9590 0.0000

Contact with support services in years 4–6 0.0194 0.0017 11.4010 0.0000

Missing data on fulltime status -0.0055 0.0673 -0.0820 0.9350

Missing data on high school GPA 0.1319 0.0486 2.7170 0.0070

Missing data on SAT/ACT -0.0172 0.0202 -0.8500 0.3950

Missing data on developmental courses 0.0418 0.0541 0.7730 0.4390

Missing data for hours on school activities 0.0023 0.0385 0.0590 0.9530

Missing data on first generation -0.0442 0.0343 -1.2890 0.1980

Missing data on income -0.0028 0.0222 -0.1250 0.9010

Missing data on perceived ability 0.0566 0.0569 0.9940 0.3210

Missing data on services in later years -0.0473 0.0468 -1.0110 0.3120

Missing data on services for limited-English -0.1851 0.2720 -0.6800 0.4960

Missing data on student orientation 0.0319 0.0744 0.4290 0.6670

Missing data on reentrance counseling 0.0089 0.0413 0.2150 0.8300

Missing data on classroom instr.--basic skills -0.0155 0.0602 -0.2580 0.7970

Missing data on cultural enrichment activities -0.2483 0.1314 -1.8900 0.0580

Missing data on referrals to agencies/resources 0.3955 0.3010 1.3140 0.1890

Missing data on tutoring in 1991–92 -0.1117 0.0572 -1.9550 0.0500

Missing data on tutoring in years 4–6 0.2011 0.0720 2.7940 0.0060

Missing data on counseling in years 4–6 -0.0378 0.0488 -0.7750 0.4380

Missing data on services for disabled in years 4–6 -0.1129 0.0720 -1.5690 0.1160

Missing data on field trip in years 4–6 0.2226 0.0997 2.2330 0.0250

Missing data on contact with support services in yrs. 4–6 0.1243 0.0975 1.2740 0.2030

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
106

Exhibit A-19. HLM analysis to predict baccalaureate degree completion (or higher) at any 

institution six years after freshman entry in 1991–92 using continuous 

measures of participation and adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.3985 0.0143 27.9370 0.0000

SSS services are home-based 0.0452 0.0386 1.1710 0.2490

Two-year institution -0.1713 0.0365 -4.6890 0.0000

Doctorate-granting institution 0.1128 0.0542 2.0810 0.0430

No SSS programs at institution 0.0415 0.0390 1.0630 0.2940

SSS services are blended 0.0228 0.0394 0.5790 0.5650

Black student -0.0311 0.0231 -1.3470 0.1780

Hispanic student 0.0275 0.0393 0.6980 0.4850

American Indian student -0.1825 0.0559 -3.2670 0.0010

Asian student 0.1006 0.0417 2.4150 0.0160

Female student 0.0344 0.0138 2.4920 0.0130

Live in college housing or frat./sorority 0.0466 0.0222 2.1000 0.0350

High school GPA 0.0980 0.0121 8.0870 0.0000

Percentile on SAT/ACT 0.1359 0.0335 4.0630 0.0000

Took any developmental course -0.0889 0.0143 -6.2070 0.0000

Hours on school activities 0.0009 0.0004 2.1000 0.0350

Attended college after first year 0.1436 0.0242 5.9330 0.0000

No parental education beyond high school -0.0253 0.0150 -1.6840 0.0920

Family income greater than $20,000 0.0305 0.0163 1.8750 0.0600

Have major concerns about ed. finances -0.0418 0.0136 -3.0770 0.0030

Once I start something, I finish it 0.0434 0.0153 2.8480 0.0050

Number of hours: instr. courses 0.0000 0.0004 0.0590 0.9540

Number of hours: profl. tutoring -0.0236 0.0297 -0.7940 0.4270

Number of hours: peer tutoring 0.0019 0.0017 1.0980 0.2720

Number of hours: profl. counseling 0.0025 0.0077 0.3210 0.7480

Number of hours: peer counseling -0.0099 0.0188 -0.5270 0.5980

Number of hours: labs 0.0031 0.0026 1.2090 0.2270

Number of hours: workshops -0.0064 0.0068 -0.9470 0.3440

Number of hours: cultural events -0.0426 0.0281 -1.5200 0.1280

Number of hours: handicapped srvc. 0.0065 0.0062 1.0510 0.2940

Propensity to receive instr. courses 0.0008 0.0005 1.4430 0.1490

Propensity to receive profl. tutoring 0.0270 0.0183 1.4780 0.1390

Propensity to receive peer tutoring -0.0024 0.0015 -1.5940 0.1110

Propensity to receive profl. counseling 0.0392 0.0156 2.5200 0.0120

Propensity to receive peer counseling 0.0100 0.0121 0.8250 0.4090

Propensity to receive labs -0.0018 0.0018 -0.9690 0.3330

Propensity to receive workshops -0.0019 0.0045 -0.4280 0.6680

Propensity to receive cultural events 0.0165 0.0102 1.6130 0.1060

Propensity to receive handicapped srvc -0.0028 0.0017 -1.6050 0.1080

Services for physically disabled 0.0161 0.0239 0.6730 0.5010

Services for limited-English ability 0.0055 0.0392 0.1400 0.8890

Continues to next page.
107

Continued from previous page.

Student orientation 0.0007 0.0186 0.0380 0.9700

College reentrance counseling -0.0130 0.0198 -0.6580 0.5100

Classroom instr.--basic skills 0.0000 0.0225 -0.0010 0.9990

Classroom instr.--dvlpmntl. English -0.0014 0.0255 -0.0550 0.9560

Classroom instr.--dvlpmntl. math -0.0146 0.0227 -0.6410 0.5220

Cultural enrichment activities -0.0023 0.0036 -0.6470 0.5180

Referrals to agencies/resources -0.0066 0.0178 -0.3700 0.7110

# of counseling sessions in 1991–92 -0.0029 0.0017 -1.7140 0.0860

# of counseling sessions in 1992–93 0.0017 0.0016 1.0540 0.2920

# of counseling sessions in 1993–94 0.0041 0.0016 2.6240 0.0090

# of tutoring sessions in 1991–92 0.0000 0.0010 0.0470 0.9630

# of tutoring sessions in 1992–93 0.0013 0.0010 1.2830 0.2000

# of tutoring sessions in 1993–94 0.0005 0.0012 0.4490 0.6530

Some tutoring in years 4–6 -0.0040 0.0020 -2.0250 0.0430

Some counseling in years 4–6 0.0117 0.0022 5.4390 0.0000

Some services for disabled in years 4–6 -0.1063 0.0309 -3.4350 0.0010

Field trip in years 4–6 0.1334 0.0152 8.7810 0.0000

Contact with support services in years 4–6 0.0162 0.0018 9.2270 0.0000

Missing data on student age 0.0582 0.0568 1.0240 0.3060

Missing data on student race 0.1168 0.0790 1.4790 0.1390

Missing data on high school GPA 0.3029 0.0521 5.8140 0.0000

Missing data on SAT/ACT -0.0046 0.0213 -0.2170 0.8280

Missing data on developmental courses -0.0250 0.0576 -0.4340 0.6640

Missing data for hours on school activities 0.0389 0.0402 0.9680 0.3340

Missing data on first generation -0.0496 0.0360 -1.3780 0.1680

Missing data on income 0.0444 0.0234 1.8990 0.0570

Missing data on financial concerns -0.0480 0.0648 -0.7410 0.4590

Missing data on finishing tasks -0.0245 0.0567 -0.4330 0.6640

Missing data on B32 0.0099 0.0504 0.1970 0.8440

Missing data on services for limited-English 0.4974 0.2802 1.7750 0.0750

Missing data on student orientation 0.0115 0.0771 0.1490 0.8820

Missing data on reentrance counseling 0.0258 0.0429 0.6020 0.5470

Missing data on classroom instr.--basic skills 0.0037 0.0621 0.0600 0.9530

Missing data on cultural enrichment activities -0.1003 0.1359 -0.7380 0.4600

Missing data on referrals to agencies/resources -0.4757 0.3101 -1.5340 0.1250

Missing data on tutoring in 1991–92 -0.0910 0.0617 -1.4760 0.1400

Missing data on tutoring in years 4–6 0.1561 0.0747 2.0890 0.0360

Missing data on counseling in years 4–6 0.0357 0.0503 0.7090 0.4780

Missing data on services for disabled in years 4–6 -0.1335 0.0742 -1.7990 0.0720

Missing data on field trip in years 4–6 0.3262 0.1032 3.1600 0.0020

Missing data on contact with support services in yrs.

4–6 0.0886 0.1007 0.8800 0.3790

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
108

Exhibit A-20. HLM analysis to predict associate’s degree completion (or higher) at any 

institution six years after freshman entry in 1991–92 using continuous 

measures of participation and adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.5382 0.0153 35.2720 0.0000

SSS services are home-based 0.0238 0.0411 0.5780 0.5660

Two-year institution -0.0041 0.0387 -0.1070 0.9160

Doctorate-granting institution 0.0596 0.0582 1.0240 0.3120

No SSS programs at institution 0.0073 0.0417 0.1750 0.8620

SSS services are blended -0.0127 0.0422 -0.3000 0.7650

Black student -0.1138 0.0286 -3.9750 0.0000

Hispanic student -0.1422 0.0775 -1.8340 0.0660

American Indian student -0.2444 0.0611 -3.9990 0.0000

Asian student -0.0278 0.0566 -0.4910 0.6230

Female student 0.0573 0.0150 3.8240 0.0000

Student was full-time 0.1766 0.0790 2.2360 0.0250

High school GPA 0.1005 0.0141 7.1390 0.0000

Took any developmental course -0.0891 0.0153 -5.8200 0.0000

Attended college after first year 0.2774 0.0263 10.5680 0.0000

No parental education beyond high school -0.0183 0.0162 -1.1270 0.2600

Family income greater than $20,000 -0.0030 0.0217 -0.1360 0.8920

Above average academic ability 0.0468 0.0238 1.9690 0.0490

Have major concerns about ed. finances -0.0371 0.0147 -2.5150 0.0120

Number of hours: instr. courses -0.0002 0.0005 -0.3400 0.7340

Number of hours: profl. tutoring -0.0324 0.0322 -1.0050 0.3150

Number of hours: peer tutoring 0.0012 0.0018 0.6530 0.5130

Number of hours: profl. counseling -0.0008 0.0083 -0.1020 0.9190

Number of hours: peer counseling 0.0050 0.0205 0.2450 0.8070

Number of hours: labs -0.0031 0.0028 -1.1110 0.2670

Number of hours: workshops -0.0074 0.0073 -1.0120 0.3120

Number of hours: cultural events 0.0067 0.0302 0.2210 0.8250

Number of hours: handicapped srvc. 0.0084 0.0067 1.2500 0.2120

Propensity to receive instr. courses 0.0001 0.0006 0.1890 0.8500

Propensity to receive profl. tutoring 0.0238 0.0197 1.2120 0.2260

Propensity to receive peer tutoring -0.0028 0.0016 -1.7080 0.0870

Propensity to receive profl. counseling -0.0555 0.0500 -1.1090 0.2680

Propensity to receive peer counseling 0.0141 0.0130 1.0860 0.2780

Propensity to receive labs -0.0003 0.0020 -0.1580 0.8750

Propensity to receive workshops 0.0032 0.0044 0.7290 0.4660

Propensity to receive cultural events 0.0099 0.0113 0.8780 0.3800

Propensity to receive handicapped srvc. -0.0013 0.0018 -0.7300 0.4660

Services for physically disabled 0.0032 0.0253 0.1270 0.8990

Services for limited-English ability 0.0098 0.0423 0.2310 0.8170

Student orientation 0.0158 0.0201 0.7840 0.4330

College reentrance counseling -0.0063 0.0215 -0.2920 0.7700

Continues to next page.
109

Continued from previous page.

Classroom instr.--basic skills -0.0139 0.0244 -0.5710 0.5680

Classroom instr.--dvlpmntl. English -0.0093 0.0276 -0.3370 0.7360

Classroom instr.--dvlpmntl. math -0.0222 0.0247 -0.9000 0.3680

Cultural enrichment activities -0.0015 0.0038 -0.4040 0.6860

Referrals to agencies/resources 0.0118 0.0194 0.6080 0.5430

# of counseling sessions in 1991–92 -0.0018 0.0019 -0.9760 0.3300

# of counseling sessions in 1992–93 0.0005 0.0017 0.2580 0.7960

# of counseling sessions in 1993–94 0.0053 0.0017 3.0440 0.0030

# of tutoring sessions in 1991–92 0.0000 0.0011 0.0020 0.9990

# of tutoring sessions in 1992–93 0.0016 0.0011 1.4080 0.1590

# of tutoring sessions in 1993–94 0.0003 0.0013 0.2320 0.8170

Some tutoring in years 4–6 -0.0048 0.0021 -2.2590 0.0240

Some counseling in years 4–6 0.0087 0.0023 3.7000 0.0000

Some services for disabled in years 4–6 -0.1038 0.0338 -3.0750 0.0030

Field trip in years 4–6 0.1184 0.0166 7.1450 0.0000

Contact with support services in years 4–6 0.0178 0.0019 9.2790 0.0000

Missing data on student race -0.1322 0.0795 -1.6630 0.0960

Missing data on fulltime status 0.1719 0.0883 1.9470 0.0510

Missing data on high school GPA 0.2664 0.0602 4.4270 0.0000

Missing data on developmental courses -0.0948 0.0627 -1.5120 0.1300

Missing data on first generation -0.0832 0.0399 -2.0880 0.0360

Missing data on income 0.0027 0.0289 0.0940 0.9250

Missing data on perceived ability 0.0888 0.0729 1.2170 0.2240

Missing data on financial concerns 0.0077 0.0703 0.1090 0.9140

Missing data on services in later years 0.0210 0.0539 0.3890 0.6970

Missing data on services for limited-English 0.1387 0.3061 0.4530 0.6500

Missing data on student orientation 0.0084 0.0836 0.1000 0.9210

Missing data on reentrance counseling 0.0045 0.0466 0.0970 0.9230

Missing data on classroom instr.--basic skills 0.0210 0.0677 0.3100 0.7560

Missing data on cultural enrichment activities -0.1688 0.1482 -1.1380 0.2550

Missing data on referrals to agencies/resources -0.0264 0.3387 -0.0780 0.9380

Missing data on tutoring in 1991–92 -0.0254 0.0672 -0.3780 0.7050

Missing data on tutoring in years 4–6 0.1311 0.0820 1.5990 0.1100

Missing data on counseling in years 4–6 0.0627 0.0549 1.1410 0.2540

Missing data on services for disabled in years 4–6 0.0034 0.0811 0.0420 0.9670

Missing data on field trip in years 4–6 0.1918 0.1125 1.7050 0.0880

Missing data on contact with support services in yrs. 4–6 0.0721 0.1099 0.6570 0.5110

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
110

Exhibit A-21. HLM analysis to predict transfers from two-year to four-year institutions six 

years after freshman entry in 1991–92 using continuous measures of 

participation and adding propensity scores

Variable Coefficient Standard 

Error T-ratio P-value

Intercept 0.2186 0.0122 17.9280 0.0000

SSS services are home-based 0.0375 0.0447 0.8410 0.4170

No SSS programs at institution 0.1060 0.0336 3.1530 0.0090

SSS services are blended 0.0285 0.0354 0.8050 0.4370

Female student -0.0704 0.0251 -2.8050 0.0050

High school GPA 0.0629 0.0178 3.5340 0.0010

Took any developmental course -0.0609 0.0252 -2.4150 0.0160

Attended college after first year 0.0930 0.0295 3.1510 0.0020

No parental education beyond high school -0.1195 0.0271 -4.4080 0.0000

Family income greater than $20,000 0.0025 0.0281 0.0880 0.9300

Once I start something, I finish it -0.0320 0.0262 -1.2190 0.2230

Number of hours: instr. courses -0.0012 0.0006 -1.9150 0.0550

Number of hours: profl. tutoring -0.0167 0.0589 -0.2840 0.7770

Number of hours: peer tutoring 0.0039 0.0037 1.0410 0.2980

Number of hours: profl. counseling 0.0169 0.0120 1.4060 0.1600

Number of hours: peer counseling -0.0917 0.1648 -0.5560 0.5780

Number of hours: labs 0.0040 0.0027 1.4680 0.1420

Number of hours: workshops 0.0344 0.0321 1.0720 0.2840

Number of hours: cultural events 0.1990 0.1162 1.7130 0.0860

Propensity to receive instr. courses 0.0021 0.0007 3.1170 0.0020

Propensity to receive profl. tutoring 0.0421 0.0342 1.2300 0.2190

Propensity to receive peer tutoring -0.0049 0.0033 -1.4830 0.1380

Propensity to receive profl. counseling 0.0159 0.0187 0.8500 0.3950

Propensity to receive peer counseling -0.0461 0.0642 -0.7190 0.4720

Propensity to receive labs 0.0000 0.0021 0.0130 0.9890

Propensity to receive workshops 0.0355 0.0139 2.5540 0.0110

Propensity to receive cultural events -0.1200 0.0584 -2.0540 0.0400

Propensity to receive handicapped srvc. 0.0027 0.0068 0.3940 0.6940

Services for physically disabled 0.0465 0.0312 1.4910 0.1360

Services for limited-English ability -0.0761 0.0570 -1.3340 0.1820

Student orientation 0.0055 0.0254 0.2160 0.8290

College reentrance counseling 0.0486 0.0257 1.8890 0.0590

Classroom instr.--basic skills -0.0604 0.0392 -1.5410 0.1230

Classroom instr.--dvlpmntl. English 0.0152 0.0418 0.3640 0.7150

Classroom instr.--dvlpmntl. math -0.0750 0.0384 -1.9540 0.0500

Cultural enrichment activities -0.0258 0.0146 -1.7640 0.0770

Referrals to agencies/resources -0.0028 0.0329 -0.0870 0.9310

# of counseling sessions in 1991–92 -0.0033 0.0032 -1.0310 0.3030

# of counseling sessions in 1992–93 -0.0072 0.0030 -2.3530 0.0190

# of counseling sessions in 1993–94 0.0159 0.0035 4.5450 0.0000

# of tutoring sessions in 1991–92 0.0034 0.0021 1.6050 0.1080

Continues to next page.
111

Continued from previous page.

# of tutoring sessions in 1992–93 -0.0051 0.0023 -2.2700 0.0230

# of tutoring sessions in 1993–94 0.0082 0.0024 3.3640 0.0010

Some tutoring in years 4–6 -0.0030 0.0044 -0.6710 0.5020

Some counseling in years 4–6 0.0289 0.0051 5.6540 0.0000

Some services for disabled in years 4–6 -0.0181 0.0618 -0.2940 0.7690

Field trip in years 4–6 0.1334 0.0366 3.6420 0.0000

Contact with support services in years 4–6 0.0067 0.0042 1.5800 0.1140

Missing data on high school GPA 0.1558 0.0753 2.0690 0.0380

Missing data on developmental courses 0.0056 0.0797 0.0710 0.9440

Missing data on first generation -0.0854 0.0505 -1.6900 0.0910

Missing data on income 0.0341 0.0357 0.9540 0.3400

Missing data on finishing tasks -0.0764 0.0682 -1.1200 0.2630

Missing data on B32 -0.0618 0.0679 -0.9110 0.3630

Missing data on services for limited-English -0.2639 0.1734 -1.5220 0.1280

Missing data on student orientation 0.1009 0.1358 0.7430 0.4570

Missing data on reentrance counseling 0.0212 0.0684 0.3100 0.7570

Missing data on classroom instr.--basic skills 0.1128 0.1686 0.6690 0.5030

Missing data on tutoring in 1991–92 -0.0747 0.0695 -1.0760 0.2820

Missing data on tutoring in years 4–6 0.1830 0.1451 1.2610 0.2080

Missing data on counseling in years 4–6 -0.1256 0.0818 -1.5370 0.1240

Missing data on services for disabled in years 4–6 -0.2264 0.1616 -1.4010 0.1610

Missing data on field trip in years 4–6 0.3364 0.2092 1.6080 0.1080

Missing data on contact with support services in yrs.

4–6 -0.2524 0.2092 -1.2060 0.2280

SOURCE: U.S. Department of Education, Policy and Program Studies Service, National Evaluation of Student Support Services 

(SSS): Examination of Student Outcomes After Six Years, 2010.
112
113

Appendix B: Methodology

Highlights

• This study examined two types of outcomes to evaluate the effects of SSS. These were 

students’ academic performance (i.e., college grade point averages and total number of credits 

earned), and retention in higher education and degree attainment (including transfers from 

two-year to four-year institutions). 

• The study used transcript data to track the academic performance of about 2,900 freshman SSS 

participants over six years. Propensity scoring was used to select a comparison group of about 

2,900 students with similar demographic characteristics who also were tracked over six years.

• Researchers collected detailed information about students’ participation in SSS, including the 

number of hours of services received in each of nine categories: instructional courses, 

professional tutoring, peer tutoring, professional counseling, peer counseling, labs, workshops, 

cultural events, and services for the disabled. Both SSS participants and the comparison group 

also provided self-reports about services they received.

• Because students in the comparison group were still relatively advantaged compared with the 

SSS participants, additional statistical adjustments were used to correct for these differences. 

The variables chosen for these adjustments include students’ demographic characteristics, 

academic background, and attitudes, and institutional characteristics of the schools they 

attended.

• Using regression analysis, the study developed statistical models to examine the relationship 

between the SSS services received and the measures of effect. 

• Regression coefficients were used to calculate the effects of SSS for each student based on the 

number of hours of services that each received. Summary statistics then were developed to 

describe the average effect of individual SSS services and of SSS overall.
114

Introduction

This section provides a description of the study design and of the issues that were 

involved in the data analysis. Although there is some overlap in topics, it does not attempt to 

repeat all of the discussion provided in chapter 2 on the study design. Chapter 2 is intended to 

provide the general background that is needed to understand the structure of the report, while this 

section provides additional historical information about the development of the study and of the 

statistical models, along with some additional details on the topics discussed in chapter 2. (There 

necessarily also is some discussion of the research methodology when interpreting the statistical 

models used for the evaluation, especially in chapter 5, which weighs the various statistical models 

and attempts to determine whether some analytic approaches are more effective or trustworthy 

than others. Again, this section does not attempt the repeat that analysis but focuses on more 

general issues concerning the study design.)

While this report adds new methodological approaches that have not been used in

earlier reports (i.e., the use of propensity scores within the statistical models, and the use of 

hierarchical linear modeling), in many ways the form of the statistical models still builds upon 

research that was conducted in earlier phases of the study. We did not attempt to reinvestigate 

every methodological choice that had been examined in the past. Thus, some parts of this 

appendix refer to earlier investigations (e.g., comparing the value of various approaches when 

investigating the effects of SSS on first-year GPAs) and do not attempt to update the analysis 

using the final data set (e.g., looking at six-year cumulative GPAs). 

Response Rates

Following is a list of the different data sources used in preparing this analysis, 

together with the dates the data were collected and the response rates.

Data source Dates of data collection Response rate

Lists of all SSS participants ...................... 1991–92 28 of 30 sites

Freshman files for all freshmen in each 

SSS and non-SSS study site ..................

1991–92 28 of 30 SSS sites; 19 of 20 non-SSS 

sites

First-year baseline student survey............. 1991–92 86 percent

Service records of SSS participants .......... 1991–92 86 percent

Follow-up student survey.......................... 1994–95 86 percent

Student transcripts..................................... 1994 97 percent from initial 47 institutions; 92 

percent from additional institutions

Follow-up student survey.......................... 1997–98 77 percent

Student transcripts..................................... 1997–98 92 percent of schools; 97 percent of 

transcripts

Longitudinal Analysis

This study was designed as a longitudinal analysis of 2,900 SSS participants chosen 

when they first entered college and then tracked over time. Students were regularly contacted to 

obtain information about their attitudes, characteristics, academic progress, and employment, and 

their academic transcripts were collected and summarized in order to measure their academic 
115

performance. At the time of this report, six years of data about the students were available. For 

this report, students’ performance is measured in the following ways: 

• Academic performance

 GPAs earned in college each year and over all six years combined

 Number of credits earned each year and over all six years

• Retention and degree attainment

 Retention at the same institution to the second, third, and sixth years

 Retention at any institution of higher education in the third year and sixth 

year

 Transfer from two-year to four-year institutions

For several reasons, it was determined that a longitudinal analysis of college 

freshmen would be the most effective means of measuring the short-term and long-term effects of 

SSS participation. If students were sampled at a later point in their college enrollment, then little 

information would be available about their first years in college, except for information recorded 

on academic transcripts or that students are able to remember and report. Also, because many 

students drop out of college before their second and third years, and because promoting retention 

in college is one of the major goals of SSS, a sample of non-freshmen potentially would be 

statistically biased by excluding those students who were not retained. Finally, an examination of 

SSS programs revealed that SSS services are primarily provided in the freshman year, and that 

there is little difference in the receipt of services after the freshman year; by selecting students as 

freshmen, detailed information could be collected about their participation in SSS services, which 

forms the primary basis for the analysis that follows.

Depending on the effect measure used, the effects are examined in individual years 

and cumulatively across all six years. Generally, given that SSS participation is typically greatest 

during the freshman year, one might expect that the greatest effect of SSS would also be found at 

that time, though it is also possible that some effects (perhaps especially for retention) may take 

more time to accumulate. On the other hand, since SSS has specific long-term goals such as 

increasing degree completion and transfers to four-year institutions, it is appropriate to evaluate 

SSS in terms of such goals. By looking at each year individually as well as at all years in 

combination, both possibilities can be examined. Moreover, this approach also allows the 

examination of the persistence of benefits of SSS participation: if SSS does improve students’ 

academic performance, is that effect limited to the specific courses and time period when those 

services are received, or are the students learning knowledge and study skills that will help their 

performance in later years as well?

Quasi-Experimental Design

Many studies are handicapped by the lack of adequate statistical controls, so that 

when program effects appear to be identified, there may be a strong possibility of other 

confounding explanations for any differences that were found. For example, some possible 

confounding factors are maturation, high motivation to succeed, and differing levels of academic 

skills and knowledge. To limit the danger of such confounding factors, it is best to study two 

roughly equivalent groups, one subjected to the program treatment and the other which is not. By 

comparing the outcomes of the two groups, one can estimate whether the program treatment was 
116

associated with improved performance. For this approach to work, it is especially important that 

the two groups be equivalent, and the process for choosing the groups can be critical to the success 

of the study. If people are assigned to the groups on a random basis, then the design is considered 

a true experiment. If exposures to the treatment are not randomized, then the design is quasiexperimental.

Selection of an appropriate comparison group is especially important for an 

evaluation of programs for disadvantaged students. Disadvantaged students are less likely both to 

attend college and to graduate from college. A study that compared SSS students to a typical 

mixture of students would be likely to show negative correlations between participation in SSS 

and student outcomes, not because SSS participation was counterproductive, but because SSS 

students could be expected to have worse outcomes than other students on average. 

Random assignment is often considered the “gold standard” for evaluation studies. 

The random assignment of people to both groups is helpful because there is always the possibility 

that some factor might make one group systematically different from another, and thus explain in 

part any differences that are found. Researchers can consciously try to make the two groups 

equivalent by using some type of matching process, but there is the possibility that the matching 

may not work well (e.g., because data are not available, the data are of poor quality, or researchers 

failed to anticipate all of the characteristics that might be important). The use of randomization 

helps to helps to prevent any systematic differences from appearing between the two groups.

Despite the strong advantages of random assignment, there are times where it is 

impractical or ineffective. These situations are particularly relevant with regard to the SSS 

evaluation.

 Failure of randomization to control for critical characteristics. It is 

necessary that the random assignment process be designed to span across all 

important dimensions. Though simple in concept, however, it not always easy 

in practice, and especially if one does not know which dimensions might be 

important. One way that this issue is relevant for the SSS evaluation is that the 

assignment of SSS grants to institutions is itself not random. Institutions must 

choose to apply for the SSS grants, and the grants are awarded in a competitive 

fashion after evaluating the quality of the proposals. It is possible that 

characteristics that affect an institution’s interest and ability to prepare a strong 

proposal might also be important in shaping the institution’s program design 

and implementation. In this particular case, the potential for bias is not 

necessarily critical as long as the way in which grants are distributed does not 

change, but it may affect the generalizability of the results to new institutions if 

the program is expanded or the selection criteria are altered. 

 Ethical issues with the use of randomization. Sometimes there are ethical 

reasons why it is difficult to impose randomization. For example, there are 

ethical difficulties with telling a student in the comparison group that he or she 

cannot receive tutoring or some other service from any source while in the 

study. Further, institutions would be likely to consider such a policy as being 

counter to their mission and potentially risky with regard to maintaining 

support and interest from prospective students, parents, alumni, and financial 

supporters. Thus, institutions do often offer supplementary services outside of 
117

SSS, and needy students often participate in them, making it difficult or 

impossible to select students who have equivalent experiences except for the 

participation in SSS. 

 Problems in implementation. Sometimes it is difficult to implement a study 

in a way that will make the desired distinction between the treatment and 

control groups. One difficulty with regard to SSS has already been noted: the 

distinction between SSS students and the comparison group is not ideal 

because the comparison students also often received supplemental services. 

Further, even among those students receiving SSS services, SSS is structured 

to allow students freedom in determining which services they receive and how 

much they receive; thus, one cannot force them all to receive equivalent 

amounts of services, and there is a potential that students who choose a 

particular type of service or who choose high levels of services may be 

systematically different from other study participants. SSS students also may 

receive supplemental services from other sources than SSS, and it would be 

both impractical and potentially unethical to prevent the receipt of such 

services (especially since some SSS programs are designed with the 

expectation that the students will also receive supplemental services from other 

sources).

 Placebo effects. In medical and other research, it has been well documented 

that the process of being monitored or treated can itself have an effect. 

Researchers try to compensate for this issue in a variety of ways; for example, 

the act of observing a comparison group helps to ensure that all groups have 

the feeling of being monitored, and the distribution of sugar pills (or other 

substitutes) helps to make people feel they are being treated. However, 

depending on the nature and invasiveness of the procedure, it is not always 

possible to hide that one group is being treated differently than another. The 

nature of participation in SSS is such that one cannot disguise whether a 

student is receiving a service, and one could easily hypothesize that the process 

of participating may change a student’s attachment to the institution or 

motivation to study.26 

Because of the difficulties with implementing a random assignment approach, this 

study instead used a quasi-experimental approach. The study did not attempt to influence either 

students’ inclusion within SSS or their level of participation in supplemental services. It used a 

comparison group of students who were not participants in SSS but were chosen because they had 

similar characteristics to the participants.27 The method and effectiveness of the selecting the 

comparison students is discussed in greater detail in a following section. The use of such a quasiexperimental design had both advantages and disadvantages:

 

26 Students often are unaware of the source of the service (e.g., whether it was provided through SSS or some other 

source), but the critical point is that such students are aware that the institution provided special supplemental services 

to them.

27 A few schools did accept some marginal students for enrollment into the institution only if those students agreed to 

participate in SSS as a condition of their enrollment. The study did not attempt to change institutions’ policies in either 

direction, neither forbidding conditional enrollments nor requiring them.
118

 The study was able to evaluate SSS in the same form as it was actually being 

implemented. If the conditions of SSS had been changed (e.g., by requiring a 

certain level of participation), the nature of the SSS program might be have 

been so radically changed that the study results might no longer be applicable 

toward estimating the effects of SSS. For example, it is possible that students 

select the type and amount of services they receive based on their perceptions 

of the effectiveness of the services (e.g., by discontinuing contacts if a service 

does not seem to be helpful), and that students vary in terms of which service is 

most effective (e.g., based on differences in learning style or academic skills 

and background). Forcing all students to receive uniform experiences might 

not only be difficult to implement (because of student resistance) but might 

obscure the effects of SSS. One difficulty is that students may not be as likely 

to benefit from a service if their attendance is forced, and another is that if 

students do differ in terms of which services will be most helpful, denying the 

possibility of self-selection may water down the estimated effect of each 

service by including students who are less able to benefit. Whatever the 

findings, it would be difficult to extrapolate those findings to SSS programs as 

they are actually implemented, when students can determine for themselves the 

amounts and types of services that they receive. The actual effect might be 

either higher (because of students’ self-selection into those services that are 

most helpful) or lower (because of lower participation levels) than those 

observed under controlled conditions.

 The fact that students’ receipt of services was not controlled or randomized 

does create a potential for bias, if the factors that are associated with 

participation are also related to student outcomes. For example, if those 

students who participate most are highly motivated to succeed, that motivation 

might help the students to be successful regardless of what supplemental 

services they receive. Conversely, if the students who participate the most are 

the neediest students (who seek more help because they need more help), then 

their outcomes may tend to be worse because of their greater need. Note that 

this issue is very different from the one raised in the first bullet. If students’ 

self-selection is associated with whether the services are helpful, then it is 

critical to allow such self-selection to properly estimate program effects. On 

the other hand, if students’ self-selection is associated with other factors that 

may affect student outcomes, then it may be difficult or even impossible to 

separate the effect of the service from the “effect” of the student 

characteristics. This issue is discussed in greater detail later within this 

chapter.

The Comparison Group of Non-SSS Students

In order to measure the effects of SSS on students’ academic performance, one needs 

some basis for knowing how the students would have performed if they had not been in SSS. It is 

not appropriate to compare SSS students to typical college students because the academic or 

economic disadvantages that are required for SSS eligibility are known to be negatively associated 

with students’ academic performance. Depending on the degree to which SSS helps participants 
119

to overcome their disadvantages, they might be helped through their SSS participation without 

necessarily performing as well as “typical” students.

Method of Creation of Comparison Group

The study design included the selection of a comparison group of freshman students 

who were chosen to be as similar as possible to the SSS participants. The comparison group was 

chosen by using regression analysis to calculate propensity scores of students' likelihood of 

participation in SSS based on demographic data that were available from the colleges.28 The 

derived formulas were then used to choose at each institution a comparison group of students 

whose propensity scores showed similar distributions. To lessen the risk that an institution might 

not have non-SSS students who were comparable to the SSS participants, the study also used 

propensity scores to select a comparison group of students from a similar institution that had no 

SSS programs. In this way, even if the SSS programs "skimmed" the most disadvantaged 

students, the study would include some comparison students for whom no such skimming was 

possible.29

No constraints were placed on the comparison group in terms of whether they 

participated in other non-SSS services, because if the SSS students had not participated in SSS, 

one might expect that they would have received many of the same non-SSS services. However, 

both the SSS participants and the comparison group were asked in the student questionnaires to 

describe all of the services that they received so that the effects of these services could be 

measured. 

Comparison With SSS Students

As the 1997 report documented, while the propensity scoring was effective in 

selecting students who were more like the SSS participants than typical students, the differences 

were only partially overcome. In particular, SSS students tended to be more disadvantaged than 

the students in the comparison group. The primary reason the SSS students and the comparison 

group were not more similar appears to be the lack of adequate data for fully comparing the two 

groups of students. For example, some institutions did not have information about the race or 

ethnicity of their students or did not have information about the students’ finances, especially for 

those who were not receiving assistance. The fewer the items that were available, the less 

powerful were the propensity models that were developed; most institutions were able to supply 

only a limited amount of data. Another reason that it was sometimes difficult to select comparable 

non-SSS students is that at a few institutions, case studies revealed that the SSS programs were so 

highly targeted that there were no comparable non-SSS students with similar characteristics. The 

number of SSS students at such institutions, however, was sufficiently small as to have only a 

 

28The formulas that were used for calculating propensity scores varied from one institution to another because 

institutions differed in the amount of information that they were able to provide for use in the statistical model.

29However, some institutions only admitted certain types of students if they participated in SSS. These institutions by 

definition did not have comparable students outside of SSS. To the degree that the matching non-SSS institutions had 

similar admissions criteria, even those institutions might not have students with comparable disadvantages because 

without an SSS program to compensate for the students' deficiencies, those institutions might not accept such students 

on even a conditional basis.
120

minor effect on the overall averages. Also, the students in the highly targeted programs were 

actually less likely to be disadvantaged than the other SSS students (e.g., 77 percent were white, 

compared with 35 percent of other SSS students) so that targeting was not the major source of the 

differences between the SSS and comparison groups. Whatever the cause, there were systematic 

differences between the SSS participants and the comparison group, with the SSS participants 

being more disadvantaged. The analysis therefore required the use of statistical adjustments to 

correct for these differences (discussed below).

The 1997 report also shows that there generally were only small differences between 

the comparison students based on whether they were at SSS or non-SSS institutions. Because

statistical adjustments for the differences between SSS and non-SSS students are required in any 

case, and these adjustments can also compensate for small differences between the two types of 

institutions, the distinction between the two comparison groups is unnecessary. Accordingly, this 

report combines both groups of comparison students into a single group for most analyses.

Use of Statistical Adjustments

Because the comparison group still might be anticipated to outperform the SSS 

students on academic measures, additional statistical adjustments were necessary. The use of 

statistical adjustments also was necessary for another reason: even among the pool of SSS 

participants, there was substantial variation in student characteristics that might reasonably be 

expected to affect the level of services that students use. For example, a student's high use of 

tutoring or counseling might reflect a high level of academic need so that the use of SSS services 

could be negatively correlated with measures of academic performance.

In order to adjust for these differences in student characteristics, two steps were 

taken. First, the analysis pooled the student data across all of the participating institutions; in this 

way, even if comparable non-SSS students could not always be found at the same institution as a 

particular group of SSS students, students with similar characteristics could often be found at one 

or more of the other institutions. Second, multivariate regression models were used to statistically 

adjust for differences in student characteristics. Because the purpose was not to measure the effect 

of a specific student characteristic, but rather to adjust for the cumulative effect of many 

characteristics, a relatively large number of variables were used in the regression models. 

Specifically, a variable was included in the model if it was of theoretical interest (e.g., measures of 

SSS participation and eligibility) or it proved statistically significant at the 0.05 level. 

Because the intention of the model was to statistically adjust for factors that might be 

related both to SSS participation and to students' GPAs or retention, a relatively comprehensive 

list of variables was included in the model. If the intention was to measure the specific effects of 

one of these factors, such a procedure might result in biased estimates. For this particular study, 

however, the goal of providing a complete adjustment was deemed more important. 
121

Measurement Issues

Variability of SSS Services

One theoretical decision that has important implications for evaluating SSS is 

whether to treat SSS as a relatively uniform intervention into students' lives or as a set of services 

that varies tremendously from one institution to another, and from one student to another even 

within an institution. From the standpoint of policy evaluation, there is considerable value in 

stating an overall effect of the SSS program and, thus, treating SSS as being relatively uniform. 

However, both our qualitative case studies and our quantitative data indicate that SSS services are 

not received in a uniform manner. 

• SSS programs differed in the way they were organized. Some primarily focused 

on providing a single service, some were the only (or at least primary) provider of 

all support services at the institution, and some provided a home base on campus 

that served the “whole student” by providing a broader range of services to 

facilitate the students’ integration on campus and by seeing that any needed 

supplemental services were provided. Because only three programs fit the “all 

service” category, and because the case studies suggested that the home-based 

programs deserved the closest analysis, the analysis in this study contrasts homebased programs with all other programs.

• SSS programs also differed in the extent to which the SSS services were blended 

with other services on campus. The analysis in the report uses the degree of 

blending as another way of differentiating between programs.

• SSS programs differed greatly in the types and amount of services they provided. 

For example, some programs provided separate instructional courses (e.g., 

developmental courses) that were offered exclusively to SSS students, while 

others offered similar courses that made no distinction between SSS and non-SSS 

students. Exhibit B-1 shows the percentage of SSS programs that provided each 

of nine SSS services to at least one freshman student in the sample.30 While 

some services, such as professional counseling and peer tutoring, were almost 

universally available to students (with over 90 percent of the students at programs 

that provided the service), four of the nine services were offered by less than half 

of the programs.

• SSS students differed greatly in their use of services even at a single institution. 

This is most obviously true of services offered to the handicapped, because

relatively few students were eligible for such services, but it was also true of 

more general services such as peer tutoring: some SSS students received no 

services, others received only small amounts, and still others received large 

amounts. No service was used by all students for whom the service theoretically 

was available. For example, while 96 percent of all programs offered 

 

30In some programs, a service may have been available, even though no sampled freshman had received the service. For 

example, the service might be offered only to nonfreshmen, or it might be such a specialized service (e.g., for 

handicapped students) that no sampled students used it during the study. The definition used here requires specifically 

that the service be available to the SSS freshmen in the sample, as measured by the fact that at least one such freshman 

received the service. Additional information about the nine services is provided in exhibit 1-1 in chapter 1.
122

professional counseling to freshmen, only 82 percent of all the sampled freshmen 

in those programs received professional counseling.31 In fact, only two 

services—professional counseling (82 percent) and workshops (62 percent)—

were received by a majority of the students to whom they were available, while 

five of the nine services were received by less than a third of the students that 

might have received them. And only one service (professional counseling at 80 

percent) was received by a majority of all SSS students.

Exhibit B-1. Percent of students in SSS programs in 1991–92 that provided each of nine 

SSS services

SOURCE: U.S. Department of Education, Planning and Evaluation Service (now known as the Policy and Program Studies Service), 

National Study of Student Support Services (SSS), Service Record Analysis, 1991–92.

Because of this great variation in the availability and use of the SSS services, a 

measure of the effects of SSS can be misleading unless the variation is taken into account. The 

group of SSS students includes many who received very few services through SSS and thus who 

were not very different from the comparison group in their educational experiences. Also, if some 

SSS services are more effective than others (as suggested later in this report), then even students 

who appeared to receive many SSS services may not have received the services that were most 

valuable, so that their educational experiences again may not be meaningfully different from those 

of the comparison group.

An implication of these differences is that it is difficult to describe SSS in terms of 

the "average" effects. Because there are many students who participate in SSS in only a minimal 

way (9 percent had only one service contact over the entire freshman year, 7 percent received less 

than one hour of total services in the first year, and 22 percent received between one and five

hours), the average tends to understate the effect of SSS on those with whom it is most involved. 

 

31A slightly lower number—80 percent—received professional counseling if one also includes SSS students who 

attended colleges that did not offer professional counseling. Typically, for services that were less widely offered than 

professional counseling, the gap between the two percentages is much greater.

32

%

36

%

43

%

46

%

61

%

61

%

71

%

93

%

96

%

0 10 20 30 40 50 60 70 80 90 100

Services for 

disabled

Cultural 

events

Lab

s

Peer 

counseling

Inst. 

Courses

Prof. 

tutoring

Workshop

s

Peer 

tutoring

Prof. 

counseling

Percent of SSS students at an institution offering the 

service
123

The financial costs connected with SSS are most associated with students who are heavy users of 

the services, while the average effect would instead give a substantial weight to students who 

neither required many resources nor would be expected to feel much benefit. On the other hand, 

excluding those students who barely participated in SSS could also lead to misinterpretations of 

the data. If the reason that those students showed minimal participation is because they perceived 

that SSS was not helping them, then excluding them would be biasing the study results by 

excluding the “failures” (i.e., those who the program failed to help, regardless of how they did on 

the various outcome measures). 

This analysis therefore uses multiple approaches. Some statistical models 

differentiate between SSS students based on the types of programs in which they were involved 

(home-based versus other, and blended versus separate), the types of services they received, and 

the number of hours of services they received. As an alternative, we also examined SSS 

participation through a single dichotomous measure that was based only on whether students 

participated in SSS during their freshman year, and that did not take account of either the type or 

amount of services they received. We also considered alternative means of measuring SSS 

participation, such as summing each student's participation across all nine services to create a 

single total for each student. This approach has the advantage of simplifying the measurement of 

SSS participation while still allowing for variations in the use of services. However, such an 

approach assumes that an hour of one type of service is equivalent to an hour of another, and tends 

to especially place a high weight on those services that consumed the most time (especially on 

development courses that were set aside only for SSS students). Both assumptions might be 

considered questionable.

One might speculate that the presence of certain types of SSS services provides 

important information about the general characteristics of the SSS programs. For example, only 

some SSS programs offered participation in cultural events as part of their SSS program, which 

might be an indicator that the program was sensitive to students’ integration on the campus as well 

as to academic issues. Thus, it is possible that the effect measures used here might sometimes 

measure more general program characteristics than the specific services noted here. For example, 

it is possible that participation in cultural events by itself has no effect on student outcomes, but 

that rather it is the orientation of the program (which is measured through the offering of cultural 

events) that is the key factor. Possibly the “service” presenting the greatest such risk is the 

measure of home-based programs, because the presence of such a home base on campus helps to 

assure that students receive a coordinated package of benefits; thus, this measure is not really a 

measure of a particular service but of a set of services. The measure of blended services could 

present similar risks. Except for these two measures, however, the wide variation in the receipt of 

SSS services even within individual institutions lessens the likelihood that the measures are only 

serving as surrogates for more general program characteristics. It is certainly possible that other 

services might provide some of the same effects (or even increased effects) as the services 

examined here. In fact, this would be true regardless of whether the measures are surrogates for 

more general program characteristics. Nevertheless, it seems reasonable to conclude that the 

effects are still the result of participation in particular services rather than being more general 

indicators of program effects.
124

Measurement of SSS and Other Student Services

The measures of first-year SSS services were based on service records provided by 

the cooperating institutions that indicated the number of minutes (here converted to hours) of 

students' participation in each service. Data were collected for nine different categories of 

services: instructional courses, professional tutoring, peer tutoring, professional counseling, peer 

counseling, labs, workshops, cultural events, and services to the disabled. Additionally, data were 

collected using a number of subcategories of these services; for example, the category of peer 

tutoring was subdivided into general peer tutoring and tutoring in each of five different specific 

subject areas (English, mathematics, science, social science, and general). These subcategories 

and the percentage of SSS students using each type of service are presented in exhibit 1-1 in 

chapter 1. Some analyses were conducted using selected subcategories to determine whether the 

general categories or subcategories were more useful. The general categories are used in this 

report for simplicity and because the subcategories provided no useful additional information; 

generally the main effect of using the subcategories was to reduce the statistical significance of the 

findings, probably because of the reduced number of students getting a service when such detailed 

subcategories were used.

In order to reduce the burden the participating institutions, and because the freshman 

year was viewed as the primary year in which students participate in SSS, the study only obtained 

service records for the first year (1991–92). For services received in later years, only students’ 

self-reports were used. Students were not asked to indicate the source of the services (they would 

not necessarily know the source, especially since many SSS projects adopt unique names at 

individual schools rather than using the SSS name), so these data do not distinguish between SSS 

and non-SSS services.

In order to properly measure the effects of the SSS services, it was judged necessary 

to also measure students' use of non-SSS services because SSS programs often referred students to 

non-SSS services, and because students often could receive equivalent services without 

participating in SSS. For example, if two students had equivalent abilities and backgrounds and 

one student benefited from SSS services while the other benefited from equivalent non-SSS 

services, an analysis that excluded the non-SSS services might falsely conclude that SSS had no 

effect because there was no measurable difference in outcomes between the two students. In fact, 

they each may have benefited compared with how they would have performed with no services. 

Similarly, because SSS students may receive non-SSS services, even an analysis that was limited 

to SSS participants would require the measurement of non-SSS services. Otherwise, one SSS 

student could benefit from an SSS service while another SSS student benefited from an equivalent 

non-SSS service, and the SSS service would falsely appear to show no effect because there would 

be no difference in the effect on outcomes. 

Unfortunately, it was not possible to collect data about non-SSS services with the 

same precision as for SSS services, partly because of the wide range of sources of services. 

Instead, this study used students' self-reports of the services they had received, which was typically 

a dichotomous (yes or no) measure. (An exception is for tutoring and counseling, in which the 

students gave more detailed categorical responses—e.g., weekly, monthly—that were converted to 

numeric estimates.) The dichotomous measures were not expected to perform as well as the more 

precise measures of the SSS services (because they lumped together students receiving high and 

low amounts of the services), and an analysis confirmed that they did not. For example, the 
125

measure of non-SSS tutoring showed statistically significant effects on first-year GPA when used 

as a continuous measure but not when it was recoded as a dichotomous variable. Thus, this 

analysis may understate the value of non-SSS services. Nevertheless, the statistical results for the 

non-SSS services were consistent with those for the SSS services, with the same types of services 

often being statistically significant for both SSS and non-SSS students. 

The primary difference between the two groups of measures is that in a few cases the 

non-SSS measures were positive and statistically significant when the comparable SSS measures 

were not: for example, for two outcomes cultural enrichment activities were statistically 

significant when the comparable SSS measure was not, and measures of counseling often 

performed better than the measure of first-year SSS counseling. These small and occasional 

differences might be attributed to extraneous factors, such as differences in the number of students 

receiving services, or the use of targeted SSS counseling to help students facing academic 

difficulties. Given the consistency of the findings for SSS and non-SSS services, it does not seem 

that the differences in metrics had a harmful effect. Further, the primary goal of including the 

measures of non-SSS services was not to precisely measure their effects but to obtain an overall 

measure of the mean effect of each service so that the regression intercept (in combination with the 

other demographic variables) reflected the estimated result if a student received no services; this 

approach allowed a better estimate of whether SSS students were helped. 

The measurement issue is further complicated by the fact that students' self-reports 

did not indicate whether the services they received were provided through SSS or through some 

other mechanism. One alternative was to leave the students' self-reports unadjusted, which would 

risk double-counting an SSS service as being provided both through SSS and also outside of SSS, 

while another alternative was to assume that whenever a student received an SSS service, the 

student’s self-report must be referring to that SSS service and not to some alternative source. The 

case studies suggested that it is rare for a student to receive the same service through both SSS and 

outside of SSS; rather, when students receive both SSS and non-SSS services, the non-SSS 

services are typically in areas in which no comparable SSS services are provided at the institution. 

Thus, the second alternative was judged the most reasonable, and student self-reports were 

adjusted to eliminate any double-counting of services.32 

The Use of Multiple Measures of SSS Services

By using multiple measures of SSS services, there is a risk that if two services were 

highly correlated, their joint inclusion might result in increased standard errors or a 

mismeasurement of the relative effects of the services. To check whether the use of multiple 

measures simultaneously had an effect on the results, each of the nine measures of SSS services 

was also run separately in regressions to estimate the effects of SSS on students’ first-year GPAs. 

This procedure had no effect on which variables showed positive and statistically significant 

results; the only important difference was that one variable (professional counseling) that 

 

32One reviewer had the concern that the combination of using a highly precise measure of SSS services with a 

dichotomous measure of non-SSS services would result in the effects of non-SSS services being absorbed by the more 

precise SSS measures. However, the strategy that was used to avoid double-counting helps to prevent this problem 

from occurring, because only one measure of a particular type of service can be nonzero for any given student. There 

is a risk that the measurement of the SSS effects will be overstated if a student received both an SSS service and an 

equivalent non-SSS service, but the likelihood of such an occurrence is so low that this is not a serious issue.
126

otherwise showed negative and statistically significant results was still negative but no longer 

statistically significant. 

There are probably two explanations for this. First, because SSS overall had a 

positive effect and no other variable was included in the equation to measure that effect, the 

variable for SSS professional counseling picked up some of that positive effect, canceling out 

some of the negative association between the need for counseling and students' GPAs. Second, 

when professional counseling alone was in the model, the remaining variables in the regression 

equation were better able to adjust for the differences between students who received counseling 

and those who did not, while when all nine SSS services were included together, the coefficients 

that were appropriate for most SSS services were not as appropriate for professional counseling. 

The primary change was in the intercept, suggesting that students who received professional 

counseling started with some type of disadvantage relative to other SSS students that could not be 

captured through the other measures of student characteristics. Though the negative coefficient for 

professional counseling is undesirable (in the sense that some aspect of students' performance is 

not being properly explained), this finding best supports the conclusion that a more complex 

model is needed that more fully captures the differences between students who received 

counseling and those who did not, not that there is a problem with including individual measures 

of each SSS service.

Measures of Student Outcomes

Several measures of student outcomes were developed for this report: students’ 

GPAs, the total number of credits earned, students’ retention at the same college or within any 

college, students’ retention or completion of a baccalaureate degree, and students’ transfer from 

two-year to four-year institutions. Following is a more detailed description of the derivation of 

these measures.

Students’ GPAs were calculated for each year individually and for all six years 

combined using the transcript data. In order to standardize the grading system, all grades were 

converted to a standard four-point scale, with an increment of 0.3 used for pluses and minuses.33 

The converted numeric score was multiplied by the number of credits attempted in order to 

compute a weighted GPA. 

The number of credits earned was based only on courses taken for regular credit. 

Institutions sometimes differed in the way that credits were assigned, with developmental courses 

counting for regular credit at some institutions and not at others. Each institution’s definition was 

accepted, without attempting to convert all credits to a similar coding scheme. For those colleges 

that divided the academic year into three terms (rather than two semesters), the credits earned were 

multiplied by two-thirds in order to create a standardized semester-based measure.

Retention at the same institution was measured by the presence of a GPA for that 

year at the sampled institution, without setting a minimum number of courses. Thus, part-time 

students were counted as retained, as were students who enrolled only for a single term in the 

academic year. Retention was measured through students’ self-reports, except that students were 

 

33E.g., “A”=4.0 and “C+”=2.3.
127

counted as retained if their transcript data indicated they were enrolled, even if the students 

reported they were not enrolled or the students did not complete the student questionnaire.34 

Retention by itself could be a misleading outcome measure since a student who had 

graduated with a baccalaureate degree would typically no longer be in college (unless he or she 

was obtaining additional schooling for a teaching certificate or for graduate education). At the 

same time, many students had not had sufficient time to complete a baccalaureate degree. Thus, a 

combined measure was created that indicated those students who either were still in college or who 

had completed a baccalaureate degree or higher. The study also included separate outcome 

measures based only on what degrees the students had earned (without looking at retention); these 

measures sometimes might be premature (in the sense that some students may not have had 

sufficient time to complete their degrees), but they do not present the same difficulty that a 

retention-only measure would present (i.e., that a success—completing a degree—might be 

interpreted as a failure—no longer being enrolled in higher education). 

The measure of students’ transfers was limited to those students who started out at 

two-year institutions and whose transcripts showed that they had at some time attended a four-year

institution. The level of the institution was obtained from the Integrated Postsecondary Education 

Data System (IPEDS).

Missing Data

In general, if a questionnaire item had substantial missing data, then a two-step 

procedure was followed: the missing values were set to 0, and a new dummy variable was set to 1

if there had been missing values, and 0 otherwise. In this way, the cases could be retained in the 

analysis, while the dummy variable summarizes the degree to which the cases with missing data 

differed from the overall mean.

A special situation occurred when structural equations models were used to estimate 

the effect of SSS on retention to the third year. Second-year GPAs were missing for roughly 30 

percent of the students (i.e., because the student at least temporarily left higher education or 

transferred, and the student either did not provide any information or did not provide sufficient 

information for the transfer grades to be obtained), but the previous solution could not be used 

because the second-year GPA was used as a dependent variable in one of the equations. Rather 

than remove these students from the analysis, second-year GPAs were instead imputed by first 

introducing the first-year GPAs, and then iteratively using regression equations to calculate an 

estimated second-year GPA. Without the use of these additional data, the structural equations 

 

34There would be some risk of bias if retention rates were calculated through this mechanism, because students who 

remained at the same institution would be counted as enrolled, but students who transferred to another institution might 

not be detected (without the student’s report that he or she had attended another college, the transcript would not have 

been collected). Also, the imputation was only in one direction; no students were coded as not enrolled if we lacked 

transcript records to confirm the enrollment. However, the focus of the regression analysis was not to calculate 

retention rates but to calculate the incremental effect of SSS on retention. Assuming that the effects of SSS did not 

vary greatly from one student to another (except based on the differing amounts of services that each received, which 

was included in the model), this recoding of data was helpful by increasing the accuracy of the data and increasing the 

number of cases available for analysis.
128

model could not be estimated.35 The results from the structural equations models were not greatly 

different from logistic regression models that used high school GPAs in place of college GPAs as 

a measure of academic strength, so unlike the third-year report, this report uses the logistic 

regression models without any imputation of data both for the third-year and sixth-year measures 

of retention.

Development of the Statistical Models

In this section, we describe the four types of statistical models that were used in the 

outcomes analysis. Additional discussion of the modeling process, including some alternative 

models that were considered, is provided in the 1997 report. A later section discusses how the 

models were interpreted.

Choice of Statistical Techniques

At times, relatively complex statistical techniques were required to solve some of the 

methodological issues that appeared in this study. Because of the possibility that the choice of a 

particular methodological approach might be responsible for the research findings, the 1997 report 

examined each student outcome from multiple statistical perspectives—at the least, by using 

multiple regression to look both for overall effects and for effects associated with particular SSS 

services, but also through using several methods to analyze retention: (1) logistic regression, (2) 

allowing for indirect effects of SSS through its effects on college GPAs, and (3) structural 

equations. In general, the comparison of the different statistical approaches showed that with only 

a few exceptions, the different approaches produced highly similar results, and the techniques that 

appeared the best from theoretical grounds were also the most powerful with regard to their 

predictive power. Thus, rather than repeating the use of multiple perspectives, this analysis builds 

upon the techniques that were identified as best in the 1997 report. This section briefly 

summarizes the findings of the 1997 report, while a more complete discussion of the approach, 

including the results of using the various alternative approaches, is provided in that report.

One of the key findings from the comparison of multiple approaches was that the 

statistical models were more powerful when each SSS service was treated as a separate variable in 

the overall model than when SSS participation was measured though a simple dichotomous 

measure (i.e., either the student was an SSS participant, or he or she was not). This finding was 

expected because the dichotomous measure mixes together students who were highly involved in 

SSS with students who were barely different in their participation from the non-SSS comparison 

group, and it is not realistic to expect both students to experience the same effects. Further, the 

use of a dichotomous measure ignores how the resources within SSS are distributed; the heavy 

users of SSS are also the ones requiring the greatest resources, so the costs of SSS are largely 

associated with one group, while the effects would be measured by a much larger and very 

different group. This report continues the practice of examining both a dichotomous measure of 

 

35However, the results of logistic regression models that omitted second-year GPAs (as an alternative way of handling 

indirect effects of SSS through GPAs) were consistent with the structural equations models, so the results presented 

here do not appear to be an artifact of the imputation process. 
129

participation and multiple, continuous measures of participation, so that readers may see the 

importance of the methodological choices involved. 

Second, this analysis repeats the use of logistic regression analysis when the 

dependent variable is dichotomous, because the statistical assumptions for ordinary least squares 

regression analysis are violated when the dependent variable is limited to only two values. 

(However, the HLM models did not use logistic regression because HLM generally was unable to 

converge to a stable estimate. Fortunately, some studies have indicated that OLS regression is 

fairly robust, and tends to produce similar results to logistic regression. The same appeared to be 

true in this study, with the various methodological approaches often producing highly similar 

estimates of effect.) 

In the 1997 report, only the measures of retention required logistic regression 

analysis, but the new measure of transfers to four-year institutions also is dichotomous, so that 

logistic regression analysis is used. In logistic regression analysis, coefficients have a somewhat 

different interpretation than do regression coefficients; they are not as directly interpretable, but 

rather can be transformed to reflect the odds ratio of a student's probability of retention or transfer. 

For example, suppose that a student would normally have a probability of 60 percent of being 

retained until the next year. This probability might be expressed as an odds (60 percent versus 40 

percent, or 60/40 = 1.5). If the logistic regression indicates that the receipt of an SSS service 

might double the odds (i.e., if the odds ratio is 2), then the student's estimated new probability of 

retention (given the receipt of the service) would be 75 percent (i.e., the odds would be doubled 

from 1.5 to 3, and the probability distribution that would produce odds equal to 3 is 75 percent 

versus 25 percent). Thus, the amount by which students’ probability of retention is increased will 

depend upon the original predicted retention rates; if the students are very unlikely to be retained, 

then even a doubling of the (low) probability will still result in a low number, while if the students 

are equally likely to be retained or not retained, the same odds ratio would be associated with a 

much larger change in the probability of retention. 

To estimate the effects of SSS, one therefore needs to know not only the odds ratio, 

but also at least a rough approximation of the students’ probability of retention. To simplify the 

discussion, this analysis often inserts a base rate into the calculation, so that it will easier to judge 

the magnitude of the change in probability. The base rates that were used are 14 percent for 

retention to the sixth year at the same institution, 42 percent for retention to the sixth year at any 

institutions, 34 percent for retention to the sixth year or the earning of a baccalaureate degree or 

higher at the same institution, 60 percent for retention to the sixth year or the earning of a 

baccalaureate degree or higher at any institutions, 6 percent for retention to the sixth year at the 

same institution at two-year institutions, 30 percent for retention to the sixth year at any 

institutions at two-year institutions, 6 percent for retention to the sixth year or the earning of a 

baccalaureate degree or higher at the same institution at two-year institutions, 34 percent for 

retention to the sixth year or the earning of a baccalaureate degree or higher at any institutions, and 

15 percent for transfers from two-year to four-year institutions.36

 

36The choice of the base retention rate has no effect on the statistical significance of the findings; it only affects how the 

coefficients are interpreted in terms of the size of the SSS effects. Further, even when interpreting the size of the SSS 

effects, the choice of the base retention rate does not have much effect as long as the alternatives are all relatively close 

to each other, as was the case here. Only a rough approximation is needed, especially given the low odds ratios that 

appeared in this analysis (no higher than 1.5 for the SSS variables). For example, if the odds ratio was 1.5, any base 

retention rate from 0.34 to 0.57 would produce an estimate of a 10 percent increase in retention, and any base retention 
130

A third finding was that unless great care was used, introducing students’ college 

GPAs as explanatory variables in the regression analysis had the effect of understating estimates of 

SSS on retention. Because students’ college GPAs were affected by participation in SSS, the 

variables that were intended to measure the effect of SSS actually only measured the residual 

effect after the influence of SSS through students' GPAs was removed, not the total impact of SSS. 

Two different approaches produced similar results: leaving college GPAs out of the regression 

(instead using high school GPAs as a measure of academic strength) and using structural equations 

to calculate a base GPA (i.e., the predicted GPA if students had not participated in SSS) that could 

be used rather than the actual GPA (which included the added impact of SSS) to predict retention. 

Because the results were quite similar and the structural equations approaches added several 

complexities (i.e., the software for structural equations models was not designed for logistic 

regression analysis to compensate for the use of a dichotomous dependent variable, and it was 

necessary to impute college GPAs for many students in order to provide enough cases for stable 

estimates), this report uses logistic regression analysis without the use of college GPAs rather than 

using structural equations models.

Interpretation of Models

This section discusses how the summary measures of the effects of SSS were derived, 

and how the occasional appearance of negative but statistically significant regression coefficients 

should be interpreted.

 

rate from 0.66 to 0.71 would produce an estimated increase of 8 percent. If the odds ratio was 1.2, any base retention 

rate from 0.43 to 0.53 would produce an estimated increase of 5 percent, and any base retention rate from 0.54 to 0.72 

would produce an estimated increase of 4 percent. The base retention rates were chosen based on the observed 

retention rates of the SSS students and the comparison group.
131

Translation of Coefficients into Estimated Effects

The regression coefficients for SSS services that appear in Appendix A may be 

interpreted as the average change in the student outcome that is associated with each hour of 

service received.37 However, because SSS students varied greatly in the amount of services 

received, the actual impact of the SSS programs cannot be determined from those statistics alone. 

To provide a better summary measure of the impact of SSS, those coefficients that were positive 

and statistically significant were multiplied by the actual hours of services that each student 

received, and the mean among all students was then calculated.38 This procedure was conducted 

both for each individual service that was statistically significant and for the sum of all statistically 

significant services that each student received. The latter statistic, labeled “total program effect” 

in this report, can be viewed as a single summary statistic that describes the total impact of the 

SSS program when all services are combined together.

In previous reports, the above calculation typically was performed for two groups of 

students. First, in order to provide an overall measure of the impact of SSS, the calculation was 

performed over all SSS students in the sample, even if a particular student did not participate in 

any services that showed statistically significant effects. This is the most useful statistic if one 

wishes to describe the total impact of SSS as a national program; however, because many SSS 

students participate in only a minimal way, this statistic does not necessarily describe the impact 

on those students most responsible for using SSS resources. Further, especially since some of the 

statistically significant services were not even available to many SSS students, the overall statistic 

fails to indicate the impact that might be produced if SSS were restructured to make greater use of 

services that are shown to be effective. For these reasons, in previous reports the calculation was 

also performed only for students who received statistically significant services (e.g., only for 

students receiving peer tutoring or, for the measurement of the “overall program effect,” only for 

students receiving at least one statistically significant service). This latter statistic is a better 

measure of the value of a specific service, though it does not show the national impact of the 

service. For example, typically each of the nine SSS services was only offered at some of the SSS 

projects and not at others. If a service is offered, say, at half of the SSS projects, it might be 

accurate to say that the nationwide impact of that service was x percent, but it is also misleading in 

the sense that many SSS students never had the option of receiving that service through SSS. In 

this report, the primary focus is on the overall impact of SSS after summing across all SSS 

services rather than on individual services. Also, in order to be conservative with regard to the 

“intent to treat” issue (i.e., that omitting people who did not participate might bias the results by 

omitting failures of the project to help students), it seemed best to include all students.39 

 

37The coefficients for the logistic regressions are interpreted somewhat differently and reflect changes in the odds ratio 

that are associated with each hour of service. Odds ratios are discussed in greater detail earlier in this appendix.

38In the case of home-based and blended programs, the coefficient was multiplied by 1 if a student was in that type of 

program, and by 0 if the student was in a different type of program.

39 However, one could differentiate between institutions that do not offer a particular service and students who do not 

participate when it is offered. A student’s choice not to participate might be considered a failure of the service (e.g., if 

the reason is that the student has heard negative reports from other SSS participants), but the lack of availability of the 

service might simply indicate that the institution has chosen to offer the service though some means other than SSS.
132

Negative Regression Coefficients

For some models, and particularly using the measure of the amount of SSS 

professional counseling received by each student, one or more regression coefficients for the SSS 

services were negative but statistically significant. While it seems reasonable that some SSS 

services might be ineffective and show no relationship to student outcomes, it seems much less 

reasonable that a service was actually detrimental to students (e.g., because of quality control 

procedures to win and maintain the program grants, and because students would not be likely to 

participate unless they perceived a benefit from their participation); a more likely explanation is 

that some unmeasured variable (e.g., some academic deficiency) was responsible for both students 

receiving relatively large amounts of a service and also performing poorly on a measure of student 

outcomes. This, in fact, is a common risk of studies of programs to help disadvantaged students, 

because the disadvantages that make a student eligible for special programs also are related to poor 

academic performance; unless appropriate statistical adjustments are made, a negative relationship 

between participation and achievement may simply mean that a program failed to fully 

compensate for students’ disadvantages, even if the participation might have been helpful. It is 

because of this risk that considerable effort was given to selecting a comparison group that had 

similar characteristics to the SSS participants, and for using additional statistical adjustments in the 

regression models.

If negative regression coefficients appear only rarely and there is no systematic 

pattern to their appearance, they might appear by chance given the large number of outcome 

measures used in this report. (While this study uses seven categories of outcome measures, it uses 

12 separate measures across those categories; for example, retention/degree completion is 

measured both overall and also separately for two-year and four-year institutions.) For example, if 

a significance level of 0.05 is used for hypothesis testing, then one out of 20 of the findings might 

be expected to be statistically significant simply by chance.40 This report will interpret negative 

relationships within the larger context of the findings: to the extent that they appear idiosyncratic 

and contradicted by more frequent positive findings, they will be ignored and assumed to reflect 

either the disadvantaged status of the SSS students or random aberrations, while consistent 

patterns will be examined to determine the source of the negative relationships.

The SSS service variable that most regularly produced negative and statistically 

significant coefficients was professional counseling. Additional analysis of the data indicated that 

there were systematic differences between students who received counseling and those who did 

not. For example, students who received no professional counseling had the strongest prior 

academic backgrounds (i.e., based on their mean SAT and ACT percentiles of 0.43). More 

generally, the strength of the students' prior backgrounds was inversely related to the amount of 

counseling they received (with SAT and ACT percentiles of 0.34 for students receiving up to one

hour of professional counseling, 0.27 for one to two hours, and 0.24 for two to four hours; an 

exception to this pattern is that the percentile was 0.36 for the 6 percent of students receiving more 

than four hours of professional counseling). Similar patterns appeared for other types of 

counseling besides professional counseling. Students who received no peer counseling had the 

 

40 Not all of the measures are completely distinct. For example, retention or degree completion at four-year institutions 

is likely to be similar to retention or degree completion overall, and perhaps even to retention or degree completion at 

two-year institutions. Thus, it is probably an overstatement to label these as 12 separate measures. Still, to the degree 

that a finding appears highly idiosyncratic rather than showing a consistent pattern across multiple measures, one 

should be wary of giving too much importance to that finding.
133

strongest prior backgrounds with mean SAT or ACT percentile scores of 0.40, while students who 

received up to one hour of peer counseling had mean percentiles of 0.28, and those who received 

over one hour had mean percentiles of 0.25. For non-SSS counseling, the results were mixed 

depending on the type of counseling received. College reentrance counseling showed a similar 

pattern to those above, with the students who received counseling having weaker backgrounds 

(0.35) than those who did not (0.39). For other non-SSS counseling, however, the pattern 

sometimes held and sometimes did not: students who received one to three sessions had stronger 

backgrounds (0.45) than students with more than three sessions (0.38), but also stronger 

backgrounds than those with no counseling (0.37). 

In principle, it should be possible to statistically adjust for the systematic differences 

in student characteristics so that the effect of counseling can be measured. In practice, however, 

the statistical adjustments that were appropriate for the remaining SSS service variables did not 

completely compensate for the relative disadvantage of students receiving professional counseling. 

Even when the remaining SSS service variables were deleted from a regression to predict students’ 

first-year GPAs (under the assumption that different statistical adjustments are required for 

professional counseling than for other services), the relationship remained negative (though 

statistically insignificant). The primary change among the rest of the variables in the regression 

model was a slightly lower intercept, suggesting that students who need professional counseling 

were somehow academically more disadvantaged, but this disadvantage was not well captured by 

the other explanatory variables. Thus, it appears that the proper modeling of characteristics 

associated with receiving counseling would require data that are not currently available. One 

reason for this finding may be that professional counseling was often administered in a manner 

that was distinct from other SSS services; students who appeared to be in academic trouble were 

often called in for professional counseling, so that the receipt of professional counseling was often 

an indicator of academic distress. 

Given the above difficulties, it seems the best interpretation of negative regression 

coefficients is to group them together with other SSS services that failed to show positive and 

statistically significant results; they all are services for which no benefit to students can be 

statistically proven. This does not mean that services that fail to show positive effects are proven 

to have no effect; that is, a failure to prove that a coefficient is different from 0 does not mean that 

the coefficient is equal to 0. For the particular case of negative regression coefficients, we 

especially have reason to think that the model is not working properly, and thus not to give 

excessive attention to those statistics. In this report, the focus is primarily on those services that 

do show positive and statistically significant results, rather than discussing each SSS service 

individually.

Another question that was examined was the treatment of negative but statistically 

significant regression coefficients when calculating the total impact of SSS. As described in an 

earlier section, the total impact was estimated by multiplying the positive and statistically 

regression coefficients by the actual number of hours of services received for each student, and 

then calculating the total across all services. The negative but statistically significant coefficients 

should not be considered reliable estimates of the effects of SSS, and in that sense they should not 

be included in the calculations. On the other hand, it might be that the size of the positive and 

statistically significant coefficients is due in part to the presence of negative coefficients, and that 

they would be smaller (with a correspondingly smaller total effect of SSS) if the variables showing 

a negative relationship were not included. To investigate this possibility, the regressions in the 
134

third-year report were recalculated with the negative but statistically significant variables 

excluded. Overall, the coefficients that previously were statistically significant continued to be 

statistically significant, with coefficients that were roughly the same as those that were found 

previously. For example, the estimated impact of SSS on students GPAs in the first year would 

generally be 0.01 lower than reported in chapter 2 if this procedure is followed. Thus, the 

presence of negative but statistically significant coefficients does not seem to have had an 

important effect on estimates of the effects of other SSS services or on the calculation of the total 

impact.

For this report, the sixth-year regressions also were examined through alternative 

models in which the negative but statistically significant variables were excluded. Overall, the 

exclusion of the variables typically had little effect on the regression estimates. The two major 

exceptions are that in one model (the impact of SSS on retaining or completing a baccalaureate 

degree), there was a service (SSS workshops) that stopped being statistically significant when 

negative but statistically significant variables were excluded, and in another model (the impact of 

SSS on whether participants at two-year institutions received an associate’s degree or higher), 

there was a service (peer tutoring) that became statistically significant when negative but 

statistically significant variables were excluded. In general, deleting variables is riskier than 

including irrelevant variables because deleting a variable may result in biased estimates, while 

including irrelevant variables affects the standard errors but does not bias the estimates. The 

presence of negative but statistically significant coefficients suggests that there is wide variation in 

the impact or distribution of SSS that is not adequately being measured in the model. Most likely, 

the regression model is too small (e.g., it is not including important information about how the 

services are distributed, as noted with regard to professional counseling) rather than too large. The 

approach used in this report is to focus on the statistics resulting from the full model, while 

footnotes discuss the results of the alternative models in which negative but statistically significant 

variables were excluded. 

Estimating Program Impacts

The method used to estimate program impact varies slightly depending on the 

particular methodology that is used. When dichotomous measures of SSS participation are used in 

multivariate regressions, the regression coefficient for the measure of SSS participation can be 

directly interpreted as the average change in student outcomes that is associated with participation 

in the program. For example, a coefficient of 0.03 in the analysis of GPAs could be interpreted as 

an average improvement (across all SSS students) of 0.03 using a four-point grading scale, and a 

coefficient of 3.1 in the analysis of total credits earned could be interpreted as an average 

improvement of 3.1 credits. When logistic regression is used, the coefficients are more difficult to 

interpret because they represent a change in the probability that the student will achieve the stated 

outcome. 

More specifically, the statistics shown in the tables are not the actual coefficients but 

the log-odds ratio. The odds ratio expresses the improvement in the probability that is associated 

with participation in SSS. For example, if a student’s probability of achieving the outcome is 60 

percent, that probability could also be expressed as an odds of 1.5 (i.e., 60/(100-60)=60/40=1.5). 

An odds ratio of 2 would imply that participation in SSS is associated with doubling the student’s 

odds (i.e., from 1.5 to 3), which would be equivalent to a probability of 75 percent (i.e., 75/(100-

75)=75/25=3). For that student, then, the increase in probability of achieving the outcome is 15 
135

percent (i.e., 75–60). Note that the amount of change in a student’s probability depends on the 

student’s base level of probability of achieving the outcome if he/she does not participate in SSS. 

If the base probability of achieving the outcome instead was 75 percent, the odds would be 

expressed as 3.0 (75/(100-75)=75/25=3.0), and a doubling of the odds to 6.0 would be equivalent 

to a probability of approximately 85.7 percent (85.7/(100-85.7)=85.7/14.3≈6.0), or an increase of 

about 12 percent. For this report, when a dichotomous measure of SSS participation is used, the 

base probability was calculated by determining what probability level, when combined with the 

estimated log-odds ratio that was obtained, would result in the actual outcome probability as 

observed among the SSS students.41

When nondichotomous measures of SSS participation are used, the regression 

coefficients cannot automatically be interpreted as the average impact across all students. Rather, 

the regression coefficient represents the level of improvement that is associated with receiving one

hour of the specified service. Thus, one way of estimating the average impact is to multiply the 

regression coefficient by the average number of hours of that service that were received across all 

SSS students. If multiple services showed positive and statistically significant results, one also 

needs to sum all such estimates in order to estimate the total SSS impact (rather than just the 

impact of one particular service). When logistic regression is used, one would take a somewhat 

comparable approach, except that one must also conduct the appropriate transformations to switch 

from log-odds ratios to changes in the probability of achieving the specified outcome. However, 

when multiple SSS services show positive and statistically significant results, the log-odds ratios 

cannot simply be added. The estimates in this report were computed by multiplying the logistic 

regression coefficients by the actual number of hours for each student, summing the results across 

all SSS variables for each student, estimating the level of impact per student, and averaging those 

results to produce an overall estimate.

Data on Two-year Institutions

Several of the outcome measures used in this report were limited to students who 

were attending two-year institutions at the start of the study. The measure of transfers from twoyear to four-year institutions was defined only for such students, while the measures of retention 

and degree completion were computed for two-year institutions only, as well as across all 

institutions and (sometimes) for four-year institutions only. However, because only 1,200 of the 

5,055 students in the database started at two-year institutions, the models that were limited to such 

students were less likely to show statistically significant relationships. In fact, not only were the 

supplemental services less likely to be significantly related to student outcomes, but all variables 

in the models tended to perform poorly. The final statistical models generally contained only a 

relatively small number of variables, and several variables were retained in the final models only 

because the selection procedures required that at least one variable be retained in each major 

category of variables, regardless of statistical significance. Thus, statistics on two-year institutions 

are not highly accurate, and readers should be wary of overinterpreting the findings. In particular, 

a failure to find statistically significant relationships may only indicate that the number of cases 

 

41 This approach is a simplification because it ignores the fact that most SSS students also received non-SSS services in 

the first year, plus SSS or non-SSS services after the first year, and both types of services may be associated with 

improved student outcomes. The estimated change in probability that is associated with SSS participation depends on 

what base probability of achieving the outcome is assumed, and the presence of positive and statistically significant 

results for these other supplemental services would suggest that the base probability should be somewhat lower than if 

one assumes that SSS was the only factor producing improved student outcomes.
136

available was not adequate for identifying statistical relationships unless those relationships were 

quite strong. The small number of cases was especially an issue for examining retention or 

baccalaureate degree completion at the same institution. With only a few exceptions, students 

were not able to earn baccalaureate degrees at two-year institutions, and students at two-year

institutions were not likely to remain enrolled for six years unless they also transferred to four-year

institutions. Only a total of 84 students at two-year institutions had either remained enrolled at 

that institution or earned a baccalaureate degree at that institution. Thus, the statistical models for 

this student outcome consistently performed poorly, and few variables of any type were 

statistically significant. 
The Department of Education’s mission is to promote student achievement 

and preparation for global competitiveness by fostering educational 

excellence and ensuring equal access.

www.ed.gov