Student Transcript-Enhanced Placement Study (STEPS)Technical Report
September 2013

Prepared byTerrence Willett
RP Researcher

Summary














The Student Transcript-Enhanced Placement Study (STEPS) helps colleges analyze theassociation between students’ high school performance in English and math and theirachievement in college English and math courses to inform discussions about practicalways to improve college placement policy and practices.
Previous studies indicate that the use of information contained on high schooltranscripts may be a viable option to improve our current placement process.
Long Beach City College (LBCC) developed data management and analysis infrastructure
useful to other colleges analyzing high school transcript data.
Grassroots efforts lead to the improvement of this infrastructure.
Many colleges encountered data availability and data quality issues.
Ten of the original twenty colleges were able to complete their analyses.
o High school test scores associated most strongly with college placement inEnglish and math.
o High school grades associated most strongly with college grades for English.
o Predicting college math grades is complex given the greater number of levels andtest types in high school.
Overall, these findings suggest that high school transcript data are at least partiallypredictive of college performance and could be used for a variety of purposes:
o As a part of the multiple measures component of existing community college
placement systems
o As the primary measure for placement of recent high school students
o To measure the impact of curriculum alignment effortso To create advising guides for high school students
o To inform broader policies at regional or state levels
Other implications include the importance of data quality, research capacity,
anticipating costs of policy changes, and the potential benefits of aligning high school
and college curricula.
Further research on a statewide dataset is currently underway.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

1

Introduction
The first day of school can be exhilarating with its blend of anticipation and anxiety. Recent highschool graduates entering a California community college for the first time often confront anadditional burden—assessment results that tell them they have to repeat an English or mathclass already completed at the secondary level. As John Hetts, Director of Institutional Researchfor Long Beach City College often notes in his presentations, telling incoming students just outof high school that “you’re not ready for college” can be discouraging. Of particular concern isthe fact that while our community colleges place the “remedial” label on these students, it may
be our own placement processes that are in need of remediation.
When critically examining California community college placement systems and K-12 curricula,
we may ask how aligned are California community college English and math courses to
California K-12 standards? To what extent do the K-12 standards prepare students for the core
English composition and college-level math courses that are articulated among California public
community colleges and universities? How effective are our assessments in informing us about
students’ readiness for college-level work? Of course, some students do need remediation.
Some did not have a successful high school experience. Others graduated long ago and theiracademic skills have faded with time. However, when we consider the academic preparation ofrecent high school graduates from California who earned passing grades or higher in their highschool courses, we may be underestimating their ability to succeed in college-level work—inpart due to the underutilization of their high school course and test data for advising andplacement.
In the context of these questions, a group of colleges, foundations, a system office andresearchers have come together to explore responses and strategies to improve student
progression through our educational systems through the Student Transcript-Enhanced
Placement Study (STEPS)1. This report documents the process and current state of STEPS,
findings and implications discovered to date, and planned continuing efforts and will inform acompanion practitioner brief for the field. This document begins with historical background andrelated efforts before turning to detailed and technical descriptions of the work in whichcolleges engaged. The technical sections are useful for researchers wishing to replicate or
extend the work described but can be skipped by readers more interested in the final section
with findings, implications, and future activities.

1

http://www.rpgroup.org/projects/STEPS

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

2

What is the STEPS Project?
The RP Group—with funding from The Rosalinde and Arthur Gilbert Foundation, the Walter S.
Johnson Foundation and LearningWorks and with support from the California Partnership forAchieving Student Success (Cal-PASS) and the California Community College Chancellor’s Office(CCCCO)—has been conducting a pilot project providing technical assistance on data
management and analysis to a group of invited California community colleges who wish toenhance their placement and advising systems for recent local high school graduates byutilizing high school transcript information. This effort, called the Student Transcript-Enhanced
Placement Study (STEPS) began in early 2012 and focuses on helping colleges replicate analyses
conducted by Cal-PASS and Long Beach City College (LBCC) examining the association between
students’ high school performance in English and math and their achievement in college English
and math courses, respectively. STEPS analyses are intended to drive discussions about
practical ways to improve college placement policy and practices.
Why are Multiple Measures for Assessment and Placement Important?
Historically, California community colleges have emphasized placement testing as an importantcomponent of addressing basic skills, also referred to as developmental or remedial education.
The assumption is that accurate placement would allow for students with similar skill levels to
be in the same course and receive the instruction most appropriate for advancing them to thenext level in an English or math sequence. Over the years, each community college has created
its own placement system under matriculation regulations that guide placement systemdevelopment and validation (Matriculation Handbook 2011). These regulations were initiallycreated by the Matriculation Act passed in 1988 and legally prevent incoming students frominitially enrolling in higher-level English or math courses unless they earn a sufficient score on aplacement test or have successfully completed the prerequisite course at the college. Concerns
about and challenges to this approach emerged immediately. A suit (Romero-Frias et al. v.
Mertes et al., 1988) brought by the Mexican American Legal Defense and Education Fund
(MALDEF) contested the California community college assessment and prerequisite process,suggesting it was biased and unfair and that it ignored other information such as priorcoursework at other schools and colleges (Wiseley, 2009). The suit was dropped after asuccessful negotiation between the California Community College Chancellor’s Office (CCCCO)
and MALDEF that resulted in a number of changes to matriculation practices including arequirement that community colleges not use a single test score for placement:
Multiple measures ensure that no single test score or assessment measure is used toplace students into courses. Course placement on the basis of a test score by itself is
prohibited [Title 5, § 55521(a)(3)] because a test score alone is not as informative andStudent Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

3

valid for placement purposes as when applying it with another measure. (Matriculation
Handbook 2011, page 2.5)
These “multiple measures” can consist of additional standardized tests, performanceassessments such as essays and interviews, surveys of students’ self-efficacy, prior coursework,and other background information. A recent survey of California community colleges found thatthe strong majority of respondents (81%) reported using high school educational backgroundinformation as a multiple measure (REL WestEd 2011). Of those, most (over 70%) reported
using surveys to collect high school information while only about a quarter to a third usedtranscripts to verify high school background data.
Despite the changes to assessment and placement policies and practices, concerns about theefficacy of these key matriculation services persist. Test scores can seem to overshadow these
additional measures; as stated by one student participating in a related study about assessment
and placement in California’s community colleges, “You spent four years in your high school,
and they’re judging your four years just off of that test. It puts you backwards” (Venezia,
Bracco, & Nodine, 2010).
In addition, many students do not seem to appreciate the consequences of the placements
until after they receive their placement test scores and realize that they cannot take collegelevel courses until after one or more terms of preparatory work (Venezia, Bracco, & Nodine2010). Students often do not prepare for these tests and may not perceive the placement
process as connected to their high school coursework or college registration and matriculation.
Moreover, an unintended consequence of the multiple measures requirement has been toenhance the variability in college placement systems. While there are only a handful ofcommercial assessments and a minority of institutions develops their own tests2, colleges
generally set their own cut scores, use a wide range of multiple measures, employ numerous
methods for collecting and verifying background information, and place variable emphases on
test scores versus multiple measures (WestEd REL, 2011). Another measure used by many
California community colleges is the Early Assessment Program (EAP)3 lead by the California
State University (CSU) system in partnership with the California Department of Education (CDE).
This program consists of an additional set of question items on the California Standards Tests(CST) administered to 11th grade students in the subjects of 11th grade English and Algebra II or
Summative Math for students enrolled in Pre-Calculus. Students receive a score indicating if
they are “ready for college” at a CSU. Those receiving “not ready for college” feedback can thenaccess a series of support efforts through EAP to achieve readiness for CSU by graduation.
Currently, 76 California community colleges accept either English or math EAP scores for23

http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/Assessment/ApprovedInstrumentsApril2009RevSept2011.pdf
http://www.calstate.edu/eap/

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

4placement4 with some accepting only one subject or the other and some accepting both English
and math EAP results. While the number of high school students taking the EAP test has been
growing, the majority of students do not take the math EAP as a student has to be enrolled in a
specific level of math in 11th grade in order to be eligible to take the assessment.
Multiple measures and traditional placement tests are not the only approaches California
community colleges are using for placing recent local high school graduates. There are different
local and regional approaches throughout the state as high schools and colleges partner toexperiment with different approaches. For example, Grossmont College, the Grossmont Union
High School District (GUHSD), San Diego State University, and local elementary and middle
schools with support from Cal-PASS created the English Curriculum Alignment Project (ECAP).
This multi-year effort saw faculty from elementary through university level meet and align theirEnglish sequences. In fall 2009 and 2010, a pilot was conducted to allow students earning a
grade of A or B in 12th grade English to enroll directly in college level composition at thecommunity college. The fall 2009 pilot cohort had an 83% success rate (grade of C or better)compared to 70% for comparison group students and 67% for all other composition students(Linfor, Harris, & Jendian, 2011). The fall 2010 pilot cohort had an 86% success rate compared
to 68% for other composition students. These early successes have been followed by an
expansion of the pilot to more students. This model represents an alternative approach toplacement testing for incoming high school graduates in the form of directly articulating highschool coursework with college coursework. The amount of time invested by faculty to do thenecessary alignment work is substantial, but is counterbalanced by cost savings in a reduced
number of testing units that must be purchased and less demand for developmental courses.
The great variation in California community college placement systems can create an additional
challenge for students who move among campuses. While some colleges honor each other’splacement recommendations, it is not consistent or required. Not being able to transport
placement results from one campus to another creates a barrier for students enrolling at morethan one college and also incentivizes “placement shopping,” where students take placement
tests at several different colleges to achieve a higher placement. To help create moreconsistency for students, the Student Success Task Force5 recommendations call for the
implementation of a statewide common assessment. In addition, there are efforts to create astatewide centralized repository of assessment data to help placement scores be more portable
and to support validation research. Further, there is a growing body of evidence outlined belowthat detailed high school test and course performance data can provide a valuable
enhancement to current multiple measure practices.
45

http://extranet.cccco.edu/Divisions/StudentServices/EAP/AcceptingEAP.aspx
http://californiacommunitycolleges.cccco.edu/PolicyinAction/StudentSuccessTaskForce.aspx

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

5How Useful Might High School Data Be for College Placement?
Several studies indicate that the use of information contained on high school transcripts, suchas grade point average (GPA) and standardized test scores and grades in math and English
courses, may be a viable option to improve our current placement process (Belfield & Crosta,
2012; Jaffe, 2012; Brown & Conley, 2007; Brown & Niemi, 2007; Willett, Hayward, & Dahlstrom,
2008; Wurtz, 2008). However, these findings have not been incorporated into placement policy
or practice on a large scale in California. In 2011, the Long Beach City College (LBCC)
institutional research office replicated and extended one of these earlier studies (Willett,Hayward, & Dahlstrom, 2008) on their campus (Fuenmayor, Hetts & Rothstein, 2011). LBCC
researchers used data from the California Partnership for Achieving Student Success (CalPASS)
intersegmental data system6 to link high school transcript data from high schools in LBCC’slargest feeder school district, Long Beach Unified School District (LBUSD), to LBCC course
records.
The LBCC researchers used high school data including the highest level of high school English
and math completed, scores on the California Standards Test (CST), and overall grades. College
variables included level of first English or math course attempted and grade received in thatcourse. Using regression analyses to assess the predictive power of high school information,
they found significant associations between high school and college performance in English andmath. LBCC researchers presented their findings to the LBCC community, which lead to a seriesof discussions about how to improve the college’s placement process. In turn, the institution
launched a pilot effort called Promise Pathways 7 that included placing recent LBUSD graduates
based upon their high school English and math performance, which resulted in more students
being placed into transfer-level English and math coursework when enrolling at LBCC.
Guaranteed space in core courses incentivized students to attend full-time and to take English
and math in their first year.
The first semester of the pilot showed a significant increase in the number of students taking
and successfully completing transfer-level English and math (Hetts and Willett, 2012). Prior to
the pilot, 13% of LBUSD graduates placed into transfer-level English in 2011 while 60% of LBUSD
graduates entered transfer-level English under the Promise Pathways initiative in fall 2012.
Despite the much greater number of students enrolling in transfer-level English, success rates(earning a grade of C or better) held steady with 64% of former LBUSD students in 2011 being
successful compared to 62% of the Promise Pathways students in 2012. For math, 9% of former
LBUSD students in 2011 placed into transfer level math compared to 30% of 2012 Promise
6

CalPASS is a voluntary data system of student level transcript data with encrypted identifiers to enhance security and privacy.
More information on this system can be found at www.calpass.org .
7
http://lbcc.edu/promisepathways/

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

6Pathway students. Again, despite tripling the number of students in transfer-level math, successrates remained relatively consistent with a 55% success rate for former LBUSD students in 2011
and a 51% success rate for Promise Pathway students in 2012. These results suggest that a verylarge number of recent high school graduates had been underplaced and in fact were prepared
to engage in transfer-level work. These findings have fueled the continuation of the pilot andmovement towards institutionalization. A key factor that primed this pilot was a long-standing
collaborative effort called the Long Beach Promise that, among other activities, facilitatedcoordinated curricula alignment between K-12 and postsecondary education segments in theLong Beach area. Further, while exploring the utility of high school transcripts in college placedrove the initial discussions, the actual policies put into place included a broad array ofstrategies. Given that LBCC implemented a suite of changes simultaneously, it would be difficult
to impossible to isolate the impact of any single alteration in policy or practice.
LBCC Creates the Foundation of STEPSIn the process of conducting their analyses, the LBCC research office developed infrastructurethat would prove useful to other colleges wishing to replicate this study. In particular, they
developed a Microsoft Access module that automated the importation of CalPASS Plus data
files into a data mart and created a file for analysis in the SPSS statistical program. They also
developed a series of programming scripts in SPSS to conduct the regression analyses. Finally,
they created extensive documentation that allows others to replicate their analyses andgenerously shared their work with the field. The infrastructure LBCC created provided thefoundation for the statewide pilot replication study described in this report.
STEPS Statewide Pilot Research Questions and Implementation Technical Details
About twenty colleges were invited to participate in STEPS with resources from the Rosalinde
and Arthur Gilbert Foundation, the Walter S. Johnson Foundation and LearningWorks and withsupport Cal-PASS and the California Community College Chancellor’s Office (CCCCO). The RP
Group provided technical assistance to colleges to download, process, and analyze CalPASS
data from their own campus and local feeder high schools. The primary research questions
driving the analyses were:
1. What is the relationship between high school performance measures both in generaland specifically for English and the level of the first English course attempted at thecommunity college?
2. What is the relationship between high school performance measures both in generaland specifically for English and success in the first English course attempted at thecommunity college?

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

7

3. What is the relationship between high school performance measures both in generaland specifically for math and the level of the first math course attempted at thecommunity college?
4. What is the relationship between high school performance measures both in generaland specifically for math and success in the first math course attempted at thecommunity college?
The effort also explored data availability and data quality issues. In addition, opportunities forand barriers to making improvements in placement practices received consideration.
STEPS 1.0 – Research Limitations and Lessons Learned. During the early stages of the process,the first institutions to engage in the pilot provided feedback on the extent to which theMicrosoft Access module and SPSS scripts developed for LBUSD and LBCC would function withdata from their own schools and colleges. Some of the earliest analyses were hampered by avariety of issues. One challenge was inaccurate or ambiguous course coding primarily with highschool courses. These courses are coded according to the California Basic Educational Data
System (CBEDS) schema8. Originally designed for mapping faculty assignments, CBEDS codes
were then applied to courses and used not only to distinguish different disciplines but also to
map sequences within a discipline such as English or math. In the absence of clear and accurate
CBEDS codes, these sequences have to be derived from course titles and other information that
requires much more time and introduces a source of error in cases where a course title does
not clearly indicate the level of a course.
Another barrier to the analysis consisted of instances where an insufficient number of years ofdata from a high school to adequately track students through high school and into college.
While California’s community colleges have a centralized data system (the Chancellor’s Office
Management Information System or “COMIS”) with data from all institutions that dates back to
1994, the analogous K-12 system (California Pupil Achievement Data System or “CalPADS”) is
relatively new and still in development. The Cal-PASS data system receives voluntarysubmissions from K-12 districts that vary greatly in their capacity to submit multiple years ofdata or any data at all. In some cases, these challenges are due to information technology
resource limitations or technical issues such as recent changes in information systems thatmake historical data difficult to access. In addition, some K-12 districts have missing data
elements for some years. For example, a district may have five years of enrollment and grade
data but only two years of testing data submitted to the Cal-PASS system. In cases where there
are only a few hundred or fewer student records and/or several years of data missing for8

http://www.cde.ca.gov/ds/sp/cl/documents/assncodemap2010.doc

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

8certain data elements, regression models can suffer and institutions struggle to draw
conclusions from the results.
A third major constraint discovered through the early statewide pilot was the ability of acommunity college research office to commit sufficient time to the analyses, especially in theface of challenges with data availability and quality. These barriers resulted in about half of thecolleges not being able to participate fully in the study.
STEPS 2.0—Improvements to the Transcript Analysis Methodology. At the same time, theseexperiences also inspired a grassroots effort to improve the efficiency of data screening andanalysis required for the transcript analysis initiative. This effort—lead by Dan Lamoree (Mount
San Antonio College) with input and assistance from the LBCC research team, Dustin Tamashiro(Pasadena City College) and RP researchers—a “STEPS 2.0” Microsoft Access module wasreleased. This new module underwent field testing among a subset of the pilot colleges leading
to further refinements. Some of the significant improvements included:1) A more generalized structure to better apply to any school and college
2) Incorporation of all file manipulation into the Access module, leaving SPSS scripts solely
for performing statistical functions
3) A user-friendly screen to review and easily correct course codings
4) Expanded inclusion criteria to capture a larger number of students
5) Incorporation of additional variables of interest such as time between high school andcollege English and math courses
6) Additional documentation including a training video
While the 2.0 version began as a grassroots effort among the research community, the RP
Group staff formally incorporated these enhancements into the pilot project and supported
Dan Lamoree’s efforts. To accompany the module, the RP Group staff created a new set of SPSS
scripts that included LBCC’s regressions, alternate forms of the regressions to accommodate
colleges that had more limited software licenses, and two additional analyses. The first analysis
was a set of bivariate correlations (both parametric and non-parametric) to help those with asmall number of cases and/or a large number of missing data elements. While the regressions
consider all variables simultaneously, bivariate correlations compare only two variables at atime and allow for interpretation of the more robust data elements. The drawback to thisapproach is that sometimes simple two-way associations can be misleading if there are othervariables with a strong influence. For example, correlating a high school grade point with acollege grade point alone does not account for the level of the course taken in college.
The second analysis created a set of two tables that cross the last English or math course taken
in high school with the first English or math class respectively attempted in college. One table
Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

9populates the cells with row percentages to assess the level to which students are transitioningin college in relation to the highest level they completed in high school. The second table
populates the cells with college success rates (grade of C- or better) to assess the achievementof students in the college course attempted. These tables were intended to provide researcherswith an intuitive display of student transition and achievement for a broad audience (seeexample table for math in Appendix A).
The release of STEPS 2.0 was accompanied by webinars and presentations at the RP Group
researcher conference. User feedback indicated that the new version significantly reduced dataprocessing time while increasing accuracy through more management screening and correction
tools. Some of the earliest participants reran their analyses to include the new students in the
expanded inclusion criteria and apply some of the changes made to the regression modules. By
the end of the analysis phase, a total of ten colleges were able to participate in addition to
LBCC’s efforts.
What Did the Statewide Pilot Study Find?
Eleven pilot colleges were able to complete regression analyses. Each college conducted four
regressions to:Predict level of first college English course attempted
Predict level of first college math course attempted
Predict success in first college English course attempted
Predict success in first college math course attempted

Predictor variables included:







Most recent California Standards Test (CST) score in English (or math) (standardized)
Count of "A-G" or college preparatory courses completed9
Level of most recent high school course in English (or math) (ordinal rank)
Grade point in most recent high school course in English (or math)High school grade point average excluding English (or math)Level of first college course attempted in English (or math) (ordinal rank, only used forpredicting success in college course)

High schools and colleges had differing numbers of levels of English and math offerings
although three to four levels was common for English and five to eight levels was typical formath for both high schools and community colleges. Ordinal regressions were used to predict
9

California has seven categories of high school courses to complete in order to be eligible for admission to a University ofCalifornia or California State University.
http://admission.universityofcalifornia.edu/freshman/requirements/a-g-requirements/

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

10the level of college English or math for all 11 colleges. To predict success, eight colleges usedlogistic regression, two used ordinal regression, and one college was unable to conduct
regressions for predicting success in English or math in the time frame of the pilot. Table 1
shows the count of students in the regressions and regression strength indices.
Table 1. Counts of colleges and students in the regression models with sample size weighted
mean pseudo R2s.

College
English levelvariable
Math level
predicted English success
Math success

Count ofColleges
11
11
10
10

Count of Students Within
Weighted mean
Community College
pseudo R2
Average Cox and
Minimum Maximum (Mean)
Snell
Nagelkerke
128
9,717
1,844
0.32
0.36
65
9,717
1,985
0.11
0.16
65
8,299
1,362
0.35
0.37
65
8,299
1,462
0.13
0.18

The number of students ranged from 65 to nearly 10,000 with an average around 1,500 to
2,000. One measure of strength of ordinal and logistic regression models is the pseudo R2 (rsquared). This index ranges from zero to one (asymptotically) where zero indicates thepredictors have almost no predictive value and indices close to one indicate the predictors havealmost perfect predictive value. There are several different pseudo R2 values reported forordinal and logistic regressions with the Cox and Snell and Nagelkerke being two of the mostcommonly reported. The pseudo R2 values for predicting English level and success weremoderately strong in the context of education research and suggest the models may include
many of the most important predictor variables. The pseudo R2 values for predicting math leveland success were moderately weak in the context of education research and suggest themodels could benefit from either additional variables or other changes to the models.
One challenge in the math models results from the California Standards Test (CST) forms that
relate directly to the type of math taken. For example, if a high school student’s last math class
was Algebra II in 11th grade, they would take the Algebra II CST. If a high school students last
math class was Pre-Calculus in 11th grade, they would take the “summative math” CST. In
contrast, all high school students in 11th grade English take the same form of the English CSTwith the exception of students taking alternative test forms for special circumstances. One
strategy to address these challenges in math is to conduct separate regressions for eachdifferent CST form. However, this approach requires a larger number of students. Anotherpossible approach would be to include the CST form as an addition categorical variable or as aseries of indicator variables.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

11

To convey to a broad audience how the utility of different predictors varies among the colleges,
a set of descriptive charts was created. In each chart, the horizontal axis shows predictor
categories and the vertical axis indicates the number of colleges for which that predictor had astatistically significant association with the predicted outcome (level of or success in college
English or math). The color of the bars indicates the strength of significant predictor variables
with “warmer” colors (orange) being stronger predictors and “cooler” colors (green to blue)
being weaker predictors. The strength of predictors was determined by an RP Group staff
researcher inspecting regression coefficients within each college and assigning the coefficientsto one of the three categories of strongest, intermediate, and weakest. This method involves
human judgment and arbitrary categories and therefore the charts should be viewed as
illustrative rather than as an objective meta-analysis. As described later in this report, astatewide analysis is in progress that will conduct a series of regressions among a large numberof high schools and community colleges to more objectively discern the variability of predictor
utility among educational institutions.
When predicting level of college English (Figure 1):







The high school English CST predictor category is the most consistent and strongest as
seen by the bar indicating that the variable was significant for all 11 colleges and theshading being orange for all colleges.
The count of A-G courses was significant for six colleges but was weak for all six of thesecolleges.
High school English course level was significant for only four colleges and was anintermediate predictor for most of those colleges.
High school grade in English was significant for only four colleges but was a weakpredictor for most of those colleges.
High school GPA excluding English grades was significant for seven colleges and was anintermediate predictor for most of the colleges.

For predicting college English level, the CSTs were clearly the most important predictor variable.
As college English levels were generally determined by placement processes that typicallyweight placement test scores heavily, one could simplify this interpretation to “tests predict
tests”. While a concise statement, it should be cautioned that other variables had predictive
value that varied among colleges.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

12

Count of Colleges
Showing SignificancePredicting College English Level
11
10
9
8
7
6
5
4
3
2
1
0

WeakestIntermediate
Strongest

CSTs

A-G
CoursesHS
CourseLevel

HS
CourseGrade

HS GPA*

Predictor Variable Category* High school GPA excluding English grades

Figure 1. Count of colleges with significant predictor variables for each predictor category andtheir predicting strength within each college for predicting level of college English.
When predicting level of college math (Figure 2):







The high school math CST predictor category is the most consistent as seen by the bar
indicating that variable was significant for all 11 colleges; however, the strength varied
as seen by the variation in shading.
The count of A-G courses was significant for six colleges but was weak for most of thecolleges.
High school math course level was significant for seven colleges and, unlike when
predicting English level, was a strong predictor for most of these colleges.
High school grade in math was significant for eight colleges but was a weak predictor for
all of those colleges.
High school GPA excluding math was significant for seven colleges and was anintermediate predictor for most of the colleges.

For predicting college math level, the CSTs were clearly the most consistent predictor variable
but the strength varied. As with predicting the level of college English, one could also stateStudent Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

13

“tests predicts tests” for math but the highest level of math completed in high school is also a
powerful predictor for the majority of colleges in the pilot study.

Count of Colleges
Showing SignificancePredicting College Math Level
11
10
9
8
7
6
5
4
3
2
1
0

WeakestIntermediate
Strongest

CSTs

A-G
CoursesHS
CourseLevel

HS
CourseGrade

HS GPA*

Predictor Variable Category* High school GPA excluding math grades

Figure 2. Count of colleges with significant predictor variables for each predictor category andtheir predicting strength within each college for predicting level of college math.
When predicting success in college English (Figure 3):






The high school English CST predictor category is the least consistent and weakest
predictor being significant for only three colleges and was weak for all three.
The count of A-G courses was significant for seven colleges and was of intermediate
strength for most colleges.
High school English course level was significant for five colleges and was an intermediate
predictor for most of those colleges.
High school grade in English was significant for eight colleges but the predictive strength
varied among colleges.
High school GPA excluding English grades was significant for nine colleges and was astrong predictor for all of those colleges.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

14The level of college English attempted was significant for six colleges and was a strong
predictor for most colleges.

In general, students attempting high-level courses experienced lower success rates. For
predicting college English success, the non-English high school grades were clearly the mostimportant predictor variable. This result suggests that general student skills rather than subject
specific skills may be an important factor in college success. Grades in English also had fairly
consistent predictive value. Another simplified interpretation could be “grades predicts
grades”. Again, it should be cautioned that other variables had predictive value that variedamong colleges.

Count of Colleges
Showing SignificancePredicting College English Success
11
10
9
8
7
6
5
4
3
2
1
0

WeakestIntermediate
Strongest

CSTs

A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level

Predictor Variable Category* High school GPA excluding English grades

Figure 3. Count of colleges with significant predictor variables for each predictor category andtheir predicting strength within each college for predicting success in college English.
When predicting success in college math (Figure 4):
The high school math CST predictor category is among the least consistent and weakest
predictor being significant for five colleges and was weak for the majority of thosecolleges.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

15







The count of A-G courses was significant for only four colleges and was of intermediate
strength for those colleges.
High school math course level was significant for seven colleges and was anintermediate predictor for most of those colleges.
High school grade in math was significant for six colleges and was of intermediate
strength.
High school GPA excluding math grades was significant for seven colleges but thepredictive strength varied among colleges.
The level of college math attempted was significant for seven colleges and was anintermediate predictor for most colleges.

In general, students attempting higher level college courses experienced lower success rates.
For predicting college math success, predictors had less consistency of utility among colleges
than seen for English. Levels of math completed in high school and attempted in college
appeared to be the most important predictors of success in college math. However, similar topredicting success in college English, the non-math grades were also among the most important
predictor variables again suggesting that general student skills rather than subject specific skills
may be an important factor in college success. The more complex situation for predicting mathsuccess qualifies the “grades predicts grades” statement, raising instead the more ungainly“predictors of college math success vary among high schools and colleges.” One reason for thisadditional complexity relates back to the possible reason for the math regressions being weaker
than English with the greater diversity of CST forms in math and more variability in the numbersof levels of math in high school and college.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

16

Count of Colleges
Showing SignificancePredicting College Math Success
11
10
9
8
7
6
5
4
3
2
1
0

WeakestIntermediate

Strongest

CSTs

A-G
HS
HS
HS College
Courses Course Course GPA* Course
Level Grade
Level

Predictor Variable Category* High school GPA excluding math grades

Figure 4. Count of colleges with significant predictor variables for each predictor category andtheir predicting strength within each college for predicting success in college math.
Overall, these findings suggest that high school transcript data are at least partially predictive ofcollege performance. A key theme emerging from these results could be “more information onhigh school performance is better than less when placing students at college”.
Barriers to Implementation of the Use of High School Transcripts in Placement Processes
After conducting these regressions, community college researchers shared the findings within
their colleges and, in some cases, also with feeder high schools. College personnel engaged in
discussions of the findings typically included discipline faculty, assessment staff, administrators,
and researchers. These conversations resulted in the identification of a number of barriers to
implementing transcript analysis in the placement process.
Some colleges reporting feeling challenged to address the implications of these results in theface of barriers to changing practice and the demands of maintaining day-to-day operations.
One key barrier to utilizing high school information in placement processes was the challenge ofincorporating high school transcripts into current practices. While community colleges have
Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

17

processes for generating transcripts, the open access nature of these institutions means that
unlike selective universities, there are generally not processes in place to receive and process
high school transcripts. Adding this capacity to community colleges is a significant task that
requires a number of steps including developing agreements with feeder high schools and/or
the parent of high school students and/or adult former high school students to transfer
identifiable data, either electronically or in hard copy. Electronic transfers require information
technology infrastructure while hard copy transfers require secure office space resulting in
demands for facilities resources. In addition, training is needed and staff time allocated to these
new tasks. For at least one college attempting to incorporate high school transcripts into theirplacement systems, a group of admissions and records staff had to be trained, reclassified andpromoted in order to interpret transcripts.
At the same time, no colleges reported strong opposition by any group of faculty, staff, or
administrators to the general concept of using high school information in placement. The
attitudes and experiences of the personnel involved would need to be monitored during alonger period of planning and implementation to fully assess if other concerns might arise. In
general, it appears the most important short-term barriers to initiating the use of high schooltranscripts in college placement are one-time and ongoing facilities and personnel costs.
Implications for Practice
Importance of Quality Data and Capacity to Manage and Analyze Data
This pilot has a series of implications for practice at both high schools and colleges. The earlychallenges with the limited extent of data submissions and accuracy of some high school course
data underline the fundamental importance of accurate and complete data for the planning,
implementation, and evaluation of using high school transcripts in college placement. The data
management and analysis infrastructure created for the project reduced the costs associatedwith conducting analyses. However, it was apparent that many colleges lacked the capacity to
conduct the analyses. Capacity in this case is not necessarily the skill or ability to manage andanalyze the data but rather the time available to perform this work in addition to the tasks
already required by the institution. One skill area that was developed for many of thecommunity college researchers was awareness and knowledge of CBEDS code and English andmath offerings in general at their local feeder high schools.
Unexpected Costs
A second implication is the recognition that such an implementation can have unexpected costs
such as changes in job descriptions, training, new equipment acquisitions, or expanded officespace requirements. These costs are in addition to more readily anticipated costs such as theStudent Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

18

need for participation of high school and college personnel, students, parents, and the broader
community in collaborative discussions of proposed changes and the development ofagreements and protocols.
Potential Benefit of Aligning Curricula
A third key implication is the potential benefit of aligning English and math curricula amonghigh schools and community colleges. Aligned curricula would allow students to progress
forward in their English and math sequences after transitioning from high school to college.
Increases in alignment of sequences between high school and college would be expected toincrease the predictive validity of high school information. Curricular alignment that achieves
the level of virtual articulation between high school and college would obviate the need forplacement tests for recent high school students. However, placement systems would still beneeded for out-of-state students who were educated under a different framework of standard
or re-entry college students whose high school coursework may be dated. Exactly how long ahigh school course remains a sufficient prerequisite for the next level of coursework in college
is unclear. Expiration dates for prior coursework should be established based on clear evidence
and may vary widely among individuals complicating efforts to create simple, uniform policies
on recency requirements. While the basic principle of aligned curricula appears simple, it
requires significant effort to establish and maintain. K-12 curricula in the past have been guided
by state standards and are currently shifting to meet new Common Core State Standards10.
Community colleges have to meet articulation agreements for transfer-level English and mathcourses with universities. Within each of these shifting frameworks, discipline faculty from highschools and college have to commit substantial time and effort to determine how to alignsequences and then establish on-going collaborative work to maintain alignment in the face ofchanges in standards, articulation agreements, and other factors. To help the alignment effort
and assess its effectiveness, utilizing high school transcripts as a multiple measure in current
community college placement systems will allow for the collection of data for monitoring andresearch as institutions increase the weight given to high school transcripts over time.
Possible Uses of High School Transcript Data
The associations between high school and college performance could be used for a variety of
purposes:
10

As a part of the multiple measures component of existing community college placement
systems
As the primary measure for placement of recent high school students

For information on California’s implementation, see http://www.cde.ca.gov/re/cc/ .

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

19





To measure the impact of curriculum alignment efforts
To create advising guides for high school students
To inform broader policies at regional or state levels

The items in this list are not mutually exclusive or exhaustive. Findings from STEPS effort cannot
suggest a single most effective strategy or change in practice or policy as curriculum,
placement, and advising are composed of many complex issues most of which were not directly
addressed by STEPS. Each region will have to engage with their data and determine the best
course of action for their students and community.
Current Developments
Common Core
There are several current developments that have impacts on these efforts. One previously
mentioned impending change is California’s adoption of the Common Core State Standards 11.
This comes not only with new standards but new tests to assess students on these standards.
College placement systems that begin to utilize high school transcripts will have to prepare forvalidating their systems with a new set of K-12 tests and grades based on a different set ofstandards. The Common Core promises updated standards for K-12 and greater consistency
among states choosing to adopt that framework. This change could result in K-12 students who
move among Common Core states experiencing a more consistent education. Further,
California community colleges that align their curricula with the Common Core would
theoretically be aligning not just within California but also with other states utilizing theCommon Core framework.
Student Success and Support Programs
For the California community colleges, one change includes the Chancellor’s Office renaming
the processes and services bundled under of the name of Matriculation to Student Success andSupport Programs. Beyond a name change, this signals shifts in priorities and strategies andmay impact placement procedures and resources. In addition, two longstanding members of
the Matriculation Advisory Committee, Douglass Glasnapp and John Poggio, who developed theplacement test guidelines currently in use, are retiring from the committee. Their role will be
replaced by the Buros Center for Testing who may update the guidelines. The Chancellor’s
Office is also developing a statewide multiple measures assessment warehouse that promises
to lower the burden on colleges wishing to expand their multiple measures to include highschool transcripts by providing centralized access to student information for placement,
11

Mike Kirst provides a brief history of California standards and policy implications of the common core:
http://www.edpolicyinca.org/publications/common-core-meets-state-policy-changes-almost-everything

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

20

advising and research. This assessment warehouse is connected to the Chancellor’s Office plan
to design and implement a common set of placement tests for all California community
colleges. These centralization efforts are intended to create more consistency and portability ofplacement results for students.
Accelerated Developmental English and MathSome faculty, such as Katie Hern of Chabot College and Myra Snell of Los Medanos College,
have called for substantial changes in remedial English and math course sequences to reduce
the number of levels students must progress through to reach transfer level. The argument isthat each level of a sequence creates an opportunity for a student to drop out and that ashortened pathway will help “accelerate” students to college level coursework. One such effort
led by Hern and Snell is the California Acceleration Project (CAP) 12. A key to the CAP effort is
identifying the core content in English and math required for success in transfer level (or college
level) coursework. Outdated or peripheral content is identified and de-prioritized to streamline
and focus the remedial courses on preparing students for either college composition or
statistics, as appropriate. Because the accelerated math sequence focuses on a pathway tostatistics, students who are preparing for science, technology, engineering, or math (STEM)
majors would be expected to take additional math coursework. One accelerated math program,developed by the Carnegie Foundation for the Advancement of Teaching, includes two
accelerated curricula; one is for statistics preparation while the other is for liberal arts mathpreparation13.
CAP is currently being evaluated by the RP Group at a group of over 15 colleges that implement
accelerated curricula in the 2011-2012 academic year. In addition to the question of the impact
on student outcomes for those in the accelerated courses versus the traditional sequence,questions remain as to how student placement would be impacted.
One possible consequence is that placement could be used solely to determine whether astudent is ready for transfer-level work. Students who are not ready would take a single
preparatory course as opposed to potentially facing two or more semesters of remediation.
This change could potentially simplify placement decision systems as there would be only twolevels into which to place students. Another possibility is for colleges to have both accelerated
and non-accelerated curricula thereby increasing the complexity of placement and advising but
providing more options for students from a variety of backgrounds. The next several years will

1213

http://cap.3csn.org/
http://www.carnegiefoundation.org/developmental-mathStudent Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

21

reveal how accelerated curricula efforts perform, how they are perceived, and how they areincorporated into colleges’ English and math offerings.
Northern California STEPS
A group of community colleges in the far northern reaches of California have formed a researchalliance to conduct STEPS analyses at their colleges. One of the colleges involved was in theinitial STEPS pilot but encountered technical barriers during the time frame of the study. The
colleges in this sparsely populated region are providing support to each other to shareproblems, solutions, and findings to accomplish the analyses and participate in regional
discussions of how to improve placement systems. The Northern California STEPS grouprepresents yet another grassroots effort coalescing around the issue of improving placement
for students at California community colleges. These colleges are receiving technical assistancefrom Cal-PASS and the RP Group with Dan Lamoree providing the direct support on the STEPS
2.0 Microsoft Access module.
Statewide STEPS Study
Currently, an effort is underway between the RP Group and Cal-PASS Plus to conduct an
expanded analysis of the relationship between high school and community college
performance. This study will include a large number of schools and colleges that were unable toparticipate in the pilot. The project will add additional variables and explore questions that
arose in the pilot phase such as the impact on college performance of 1) the time elapsed fromwhen students take English and math in high school and when they take it in college, 2)
students’ EAP scores, and 3) various student characteristics. Results of this study will be ofparticular use to colleges unable to participate in the STEPS pilot. Findings will also be used to
inform the development of the statewide multiple measures assessment warehouse bysuggesting data elements of the greatest value and creating a baseline placement algorithm
that colleges can utilize or modify for their placement systems.
Conclusion
The STEPS effort revealed the intense interest and motivation on the part of a large number ofcollege personnel to improve community college placement systems. This work resulted in arefined infrastructure lowering the burden for those wishing to conduct STEPS research at theircampus. The findings from the pilot colleges suggest that high school information can be usefulin college placement but that local validation studies are important to capture variation among
student populations and curricula and to generate locally appropriate strategies.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

22

References
Belfield, C. & Crosta, P. (2012). Predicting Success in College: The Importance of Placement
Tests and High School Transcripts. Community College Research Center. New York, NY. CCRC
Working Paper No. 42.
Brown, R. & Conley, D. (2007). Comparing state high school assessments to standards for
success in entry-level university courses. Eugene, OR: Center for Educational Policy
Research. Retrieved June 28, 2007, from http://www.s4s.org/cepr.articles.php.
Brown, R. & Niemi, D. (2007). Investigating the alignment of high school and community college
assessments in California. San Jose, CA: National Center for Public Policy in Higher
Education.
California Community College Chancellor’s Office Student Services and Special Programs
Division. (2011). Matriculation Handbook. Sacramento, CA: California Community College
Chancellor’s Office. Retrieved August 9, 2013 from,http://extranet.cccco.edu/Portals/1/SSSP/Matriculation/MatriculationHandbookRevSeptember2011.pdf

Fuenmayor, A.; Hetts, J., & Rothstein, K. (2011). Assessing Assessment: Evaluating Models ofAssessment and Placement. Long Beach, CA: Long Beach City College.
Geiser, S., & Santelices, M. V. (2007). Validity of high-school grades in predicting student
success beyond the freshman year: High-school record vs. standardized tests as indicatorsof four-year college outcomes. Research & Occasional Paper Series CSHE.6.07. Berkeley, CA:
University of California, Center for Studies in Higher Education.
Hetts, J., & Willett, T. (2012). Assessing Transcript-Based Placement. Long Beach, CA:
Association of Institutional Research Annual Forum. Retrieved fromhttp://forum.airweb.org/2013/Documents/Presentations/1708_94318854-0d27-410b9e32-41fc835de98a.pdf
Jaffe, L. (2012). Mathematics from High School to Community College: Preparation, Articulation,
and College Un-readiness (Unpublished doctoral dissertation). Los Angeles, CA: University ofCalifornia at Los Angeles.
Linfor, C.; Harris, C.; & Jendian, M. (2011). Reconceptualizing Secondary English Instruction: A
New Professional Development Model. Anaheim, CA: California Education Research
Association Conference. Retrieved from http://www.cera-web.org/wpcontent/uploads/presentations/nancy_brynelson_cali_linfor_tracy_huebner/CERA%20presentation%20combined.pdf
Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

23

REL West at WestEd. (2011). 1.2.110 Types of Multiple Measures Used in California Community
College Mathematics, English, and English as a Second Language Course Placement:
Summary Report of Survey Results. San Francisco, CA: WestEd.
Romero-Frias et al. v. Mertes et al., No. 502341, (S. C. Sacramento County 1988) in Wisely, W. C.
(2009). Effectiveness of contextual approaches to developmental math in California
community colleges (Unpublished doctoral dissertation). Stockton, CA: University of the
Pacific.
Venezia, A., Bracco, K. R., & Nodine, T. (2010). One-shot deal? Students’ perceptions ofassessment and course placement in California’s community colleges. San Francisco:
WestEd.
Willett, T.; Hayward, C.; & Dahlstrom, E. (2008). An Early Alert System for Remediation Needs ofEntering Community College Students: Leveraging the California Standards Test, Report
2007036. Encinitas, CA: California Partnership for Achieving Student Success.
Wisely, W. C. (2009). Effectiveness of contextual approaches to developmental math inCalifornia community colleges (Unpublished doctoral dissertation). Stockton, CA: Universityof the Pacific.
Wurtz, K. (2008). Relations among high school math performance and college placements andachievement. Rancho Cucamonga, CA: Chaffey Community College.

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

24

Appendix A. Sample High School to Community College Math Transition Tables Using Artificial
Data.

Highestlevel
successfullycompletedin highschoolTotal

Highestlevel
successfullycompletedin highschoolPre-Alg
Algebra I
Geometry
Algebra II
Pre-Calc
Calculus
Percent
Count

First level attempted in community college
Pre-Alg
Beg Alg
Int Alg
Stats
Pre-Calc
67%
33%
0%
0%
0%
25%
50%
25%
0%
0%
25%
50%
25%
0%
0%
10%
19%
38%
19%
10%
8%
17%
25%
25%
17%
0%
0%
20%
20%
40%
15%
25%
28%
15%
11%
80
130
150
80
60

Pre-Alg
Algebra I
Geometry
Algebra II
Pre-Calc
Calculus
Total

Success in first level attempted in community college
Pre-Alg
Beg Alg
Int Alg
Stats
Pre-Calc
Calc
60%
50%
na
na
na
na
65%
60%
55%
na
na
na
65%
60%
55%
na
na
na
70%
65%
55%
80%
65%
60%
75%
70%
60%
85%
75%
70%
na
na
60%
70%
75%
65%
66%
62%
56%
81%
72%
65%

Calc
0%
0%
0%
5%
8%
20%
6%
30

Total
Percent
Count
100%
30
100%
80
100%
40
100%
210
100%
120
100%
50
100%
530

Total
57%
60%
60%
64%
73%
69%
65%

Notes: Geometry is not shown in the community college sequence as it is generally not required to progress in themath sequence.

Key to Abbreviations
na = Not Applicable
Pre-Alg = Pre-Algebra
Beg Alg = Beginning Algebra (note this is similar to Algebra I in high school)
Int Alg = Intermediate Algebra (note this is similar to Algebra II in high school)
Stats = Statistics and other non-science major math such as Finite Math
Pre-Calc = Pre-Calculus
Calc = Calculus

Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

25

STEPS Team Members
All participating colleges
Kathy Booth, RP Group
Andrew Fuenmayor, Long Beach City College
John Hetts, Long Beach City College
Craig Hayward, RP Researcher
Dan Lamoree, Lead RP Developer
Emily Lawrence, CalPASSplus Researcher
Nathan Pellegrin, RP Researcher
Karen Rothstein, Long Beach City College
Ken Sorey, CalPASSplus Manager
Nick Wade, CalPASSplus Developer
Terrence Willett, Lead RP Researcher
Alex Zakharenkov, CalPASSplus Developer
Special Thanks to Dustin Tamashiro of Pasadena City College

For more information…
Contact Terrence Willett, Senior Researcher
twillett@rpgroup.org

Who Supports This Project?
The Rosalinde and Arthur Gilbert Foundation
Walter S. Johnson Foundation
LearningWorks
With in-kind assistance from:The California Partnership for Achieving Student Success (Cal-PASS)
The California Community College Chancellor’s Office (CCCCO)
Student Transcript-Enhanced Placement Study (STEPS) Progress Report - August 2013 – www.rpgroup.org

26