STEP-UP PROGRAM 

SOCIAL INNOVATION FUND 

YEAR 4 (FINAL) IMPACT AND IMPLEMENTATION 

EVALUATION REPORT 

May, 2018 

Prepared by: 

Cindy Reich, PhD 

Research Associate 
 

 

 

 

 

 

 

 

 

 

 

 

 

Research, Development and Engagement to Improve Education 

Acknowledgements 

The authors wish to acknowledge the leadership and staff at AchieveMpls and the City of 

Minneapolis involved with the STEP-UP program. Thanks also to the Department of Research, 

Evaluation, Assessment, and Assessment at Minneapolis Public Schools for their support. And 

we are grateful to the STEP-UP students who completed trainings and internships for 

participating in the research. 

The Social Innovation Fund (SIF) was a program that received funding from 2010 to 

2016 from the Corporation for National and Community Service, a federal agency that 

engages millions of Americans in service through its AmeriCorps, Senior Corps, and 

Volunteer Generation Fund programs, and leads the nation’s volunteer and service 

efforts. Using public and private resources to find and grow community-based nonprofits 

with evidence of results, SIF intermediaries received funding to award subgrants that 

focus on overcoming challenges in economic opportunity, healthy futures, and youth 

development. Although CNCS made its last SIF intermediary awards in fiscal year 2016, 

SIF intermediaries will continue to administer their subgrant programs until their federal 

funding is exhausted. 

This material is based upon work supported by the Corporation for National and 

Community Service (CNCS) under Grant No. 12SIHMN001 and Generation Next and 

Greater Twin Cities United Way. Opinions or points of view expressed in this document 

are those of the authors and do not necessarily reflect the official position of, or a position 

that is endorsed by, the Corporation, Generation Next, Greater Twin Cities United Way 

or the Social Innovation Fund. 

How to Cite This Report 

Reich, C. F. (2018). STEP-UP Program, Social Innovation Fund, year 4 final impact and 

implementation evaluation report. Saint Paul, MN: University of Minnesota, College of 

Education and Human Development, Center for Applied Research and Educational 

Improvement. 

Contact Information 

Cindy Reich, PhD, Research Associate 

Center for Applied Research and Educational Improvement 

College of Education and Human Development 

University of Minnesota 

1954 Buford Avenue, Suite 425 

Saint Paul, MN 55108 

Telephone: 952-239-5032 

Email: reic0086@umn.edu 

Center for Applied Research and Educational Improvement, University of Minnesota i 
 

 

Table of Contents 

Executive summary..........................................................................................................................1
 

Introduction......................................................................................................................................7
 

Method: Implementation................................................................................................................16
 

Method: Impact..............................................................................................................................23
 

Findings: Implementation ..............................................................................................................43
 

Findings: Impact ............................................................................................................................64
 

Summary of findings......................................................................................................................94
 

Interpretations and limitations .......................................................................................................98
 

Conclusions and recommendations..............................................................................................102
 

References....................................................................................................................................104
 

List of Tables 

Table 1: Demographics for Interns and Matched Students............................................................31
 

Table 2: STEP-UP Targets and Actuals.........................................................................................44
 

Table 3: Intern Demographics 2014-2016 .....................................................................................47
 

Table 4: Demographics of Studied Sample ...................................................................................65
 

Table 5: Comparison of Mean GPAs for Interns and Matched Students ......................................66
 

Table 6: Comparison of Rate of Attendance for Interns and Matched Students ...........................67
 

Table 7: Comparison of Rate of Attendance, Controlling for SY16 .............................................67
 

Table 8: Comparison of Achieve Interns and Matched Students on GPA, Attendance, ACT 

Scores, MCA Reading Scores, MCA Math scores, and Behavior..............................................72
 

Table 9: Comparison of Discover Interns and Matched Students on GPA, Attendance, ACT 

Scores, MCA Reading Scores, MCA Math scores, and Behavior..............................................73
 

Center for Applied Research and Educational Improvement, University of Minnesota ii 
 

 

 

 

 

 

 

 

 

 

Table 10: Comparison of Achieve Interns and Matched Students on Behavior Referrals, 

Removals, Suspensions, On-Track to Graduate, and Graduation Rates.....................................73
 

Table 11: Comparison of Discover Interns and Matched Students on Behavior Referrals, 

Removals, Suspensions, On-Track to Graduate, and Graduation Rates.....................................74
 

Table 12: Comparison of 2014 and 2015 Participants and Comparison Students on Postsecondary Enrollment .................................................................................................................74
 

Table 13: SEL data for analysis conducted by REAA at Minneapolis Public Schools.................75
 

Table 14: Comparison of STEP-UP interns’ outcomes Pre (School Year 2015-16) and Post 

(School Year 2016-17) ...............................................................................................................81
 

Table 15: Analyses of School Outcomes for Interns Who Worked Different Numbers of Hours 

per Week ....................................................................................................................................84
 

Table 16: Analyses of School Outcomes for Interns Who Worked Different Numbers of Weeks
 

.....................................................................................................................................................86
 

Table 17: Analyses of School Outcomes for Achieve Interns Who Had 1 Internship or More
 

Than 1 Internship ...........................................................................................................................89
 

Table 18: Analyses of School Outcomes for Discover Interns Who Had 1 Internship or More
 

Than 1 Internship ...........................................................................................................................90
 

Table 19: Analysis of Employability Item with Statistical Significance for Interns Who Worked 

Different Number of Hours per Week ...........................................................................................90
 

Table 20: Analysis of Employability Item with Statistical Significance for Interns Who Worked 

Different Number of Weeks...........................................................................................................91
 

Table 21: Analysis of Employability Items with Statistical Significance for Achieve Interns Who 

Had 1 Internship or More Than 1 Internship .................................................................................92
 

Table 22: Analysis of Employability Items with Statistical Significance for Discover Interns 

Who Had 1 Internship or More Than 1 Internship ........................................................................93
 

Table 23: Comparison of Intern Non-Involvement in the Disciplinary System, 2015 and 2016 

Cohorts.........................................................................................................................................100
 

List of Figures
 

Figure 1: Sources of Data for Implementation Evaluation ............................................................20
 

Figure 2: Percentage of Interns by Gender ....................................................................................45
 

Center for Applied Research and Educational Improvement, University of Minnesota iii 
 

 

 

 

 

 

 

 

Figure 3: Percentage of Interns by Eligibility for Free and Reduced Lunch .................................45
 

Figure 4: Number and Percentage of Interns by Training Group ..................................................46
 

Figure 5: Number and Percentage of Interns by Racial/Ethnic Group ..........................................46
 

Figure 6: Intern Comments on Placements....................................................................................53
 

Figure 7: Achieve Intern Satisfaction of Work Experience...........................................................54
 

Figure 8: Discover Intern Satisfaction of Work Experience..........................................................54
 

Figure 9: A Selection of Benefits of STEP-UP Reported by Interns.............................................55
 

Figure 10: Involvement of Interns and Matched Students in MPS Disciplinary System ..............67
 

Figure 11: Involvement of STEP-UP Interns in MPS Disciplinary System..................................78
 

Figure 12: Percentages of Interns with NO Referrals in S16 and SY17........................................80
 

Figure 13: Percentages of Interns with NO Removals in S16 and SY17 ......................................80
 

Figure 14: Percentages of Interns with NO Suspensions in S16 and SY17...................................81
 

Appendices
 

Appendix A: STEP-UP Logic Model ......................................................................................... 105
 

Appendix B: Training Curriculum ..............................................................................................106
 

Appendix C: Program Changes in 2017 ......................................................................................107
 

Appendix D: Career Enrichments and Special Trainings ............................................................111
 

Appendix E: Survey Questions Used to Assess Program Implementation..................................114
 

Appendix F: Perceptions Study ...................................................................................................121
 

Appendix G: Propensity Score Match .........................................................................................124
 

Appendix H: Employability Outcomes Survey Instrument .........................................................129
 

Appendix I: Demographic Data on Interns..................................................................................131
 

Appendix J: Industries: Intern Placements ..................................................................................133
 

Appendix K: Program Exposure ..................................................................................................134
 

Center for Applied Research and Educational Improvement, University of Minnesota iv 
 

 

Appendix L: Program Satisfaction Data ......................................................................................147
 

Appendix M: GPA ANCOVA.....................................................................................................159
 

Appendix N: Full model analyses................................................................................................160
 

Appendix O: Employability Analysis Data .................................................................................166
 

Appendix P: Employability Analyses for Differential Experiences ............................................168
 

Center for Applied Research and Educational Improvement, University of Minnesota v 
 

About this Report 

This evaluation report is a Final Report for the STEP-UP program and is intended to fulfill the 

SIF requirements to determine at least a moderate level of evidence for funded projects. It 

includes impact and implementation studies focused on the cohort of interns who completed 

internships in the summer of 2016. 

Center for Applied Research and Educational Improvement, University of Minnesota vi 
 

 

 

 

 

 

 

 

 

 

 

Executive Summary 

Program and Background Information 

STEP-UP is a youth employment program for Minneapolis youth ages 14-21. STEP-UP serves 

youth who face some of the greatest barriers to employment, including youth from low income 

families, youth from immigrant families, and youth with disabilities. STEP-UP provides two 

levels of coordinated support and internship opportunities. Achieve is for students ages 16 – 21 

while Discover is for ages 14 – 15. STEP-UP is a partnership among the City of Minneapolis, 

AchieveMpls, Department of Employment and Economic Development (DEED) and Project for 

Pride in Living. STEP-UP recruits, trains, and places youth in internships1 with a wide range of 

Twin Cities businesses, nonprofits, and public agencies. The intended short-term outcomes for 

participants include an increase in professional and life knowledge and skills (employability) and 

improved school outcomes. Longer term outcomes are increased graduation rates, increase in 

college or other post-secondary enrollment, increase in college persistence and obtaining a postsecondary degree, increase in employment rates, and increase in earnings. In the summer of 

2016, 1,272 youth completed internships. 

Prior Research, Targeted Evidence 

The literature search by Michlin and Schultz (2016) found limited evidence for the effect of 

summer youth employment programs (SYEP). The authors concluded that while there is funding 

and implementation nationwide for these programs, the data on program effects are mixed. 

Among the studies that have been conducted, “there seems to be a lack of emphasis on 

evaluating and reporting data on the effects of SYEPs.” 

Prior to this report, three interim implementation evaluations (for the summers of 2014, 2015, 

and 2016) and two interim impact evaluations (for 2014 and 2015) were conducted and reports 

submitted to CNCS. This report examines STEP-UP impact on students who participated in the 

program in the summer of 2016 and compares results with participants from the summer of 

2015.2 The report also looks at implementation data for the summer of 2016 and compares 

results to the summer sessions 2014-2015. 

The impact study includes two designs to examine the effects of STEP-UP on school outcomes 

and employability. The first, a quasi-experimental matched-groups design, examined the 

program’s effect on educational outcomes and post-secondary enrollment and targeted a 

1 Throughout the report the summer work experiences of STEP-UP are called jobs and internships interchangeably.
 

Program staff members use both terms and we follow their practice in this report. 2 While an impact study was conducted in 2014, the analyses were not separated for the Discover and Achieve 

levels, making the results not directly comparable. 

Center for Applied Research and Educational Improvement, University of Minnesota 

1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

moderate level of evidence. The confirmatory question was, “Does the STEP-UP summer jobs 

program improve school outcomes?” 

The second design was a quasi-experimental one-group pretest-posttest within participants 

design. It addressed the exploratory questions, “Do the Achieve participants achieve better 

school outcomes pre- to post-program?” and “Do the Discover students achieve better school 

outcomes pre- to post-program?” The same design was used to address the exploratory questions, 

“Do the Achieve participants achieve better employability outcomes pre- to post-program?” and 

“Do the Discover participants achieve better employability outcomes pre- to post-program?” and 

“Do participants with different experiences (number of hours worked per week, number of weeks 

worked, participating in the program for one or multiple years) achieve better school and 

employability outcomes?” The second design targeted a preliminary level of evidence. 

Evaluation Design Overview 

The first design compared interns to non- STEP-UP Minneapolis Public School (MPS) middle 

school and high school students. Intern comparison groups were formed based on propensity 

score matching using demographic variables. School outcomes included GPA (grade point 

average), ACT scores, Minnesota Comprehensive Assessment (MCA) scores3

, attendance, 

behavior (referrals, removals and suspensions4), graduation rates, on-track to graduate status, and 

post-secondary enrollment. Except for post-secondary enrollment, school-based outcomes were 

obtained through administrative and demographic data collected by MPS and STEP-UP. Postsecondary data were provided by MN SLEDS5 (Minnesota Statewide Longitudinal Educational 

Data System). Employability outcomes were collected using a student survey developed by 

CAREI and STEP-UP staff. 

For the between-groups design we used independent t tests, ANCOVA, logistic regression, 

3 The ACT is a college admissions test, used to assess academic readiness for college. It is administered to eleventh 

graders in Minneapolis Public Schools. The Minnesota Comprehensive Assessments (MCA) are the state tests that 

help districts measure student progress toward Minnesota’s academic standards and also meet federal and state 

legislative requirements. Reading is given in tenth grade and math in eleventh. 

4 A suspension is when a student is sent home and misses more than one day of school. A removal is when a student 

is removed from the classroom for a specified period of time or is sent home for a day or less. A referral is any other 

type of administrative response that is not either a suspension or a removal. 

5 SLEDS matches student data from pre-kindergarten through completion of postsecondary education and into the 

workforce to identify pathways for individuals in achieving successful outcomes in education and work; to inform 

decisions to support and improve education and workforce policy and practice, and to assist in creating a more 

seamless education and workforce system for all Minnesotans. The Minnesota P-20 Education Partnership governs 

the SLEDS system. The project is managed jointly by the Minnesota Office of Higher Education (OHE) (opens new 

window), Minnesota Departments of Education (MDE), and Employment and Economic Development (DEED). 

Center for Applied Research and Educational Improvement, University of Minnesota 2 
 

 

 

 

 

 

 

Mann-Whitney U, and chi-square analyses to determine if there were significant differences on 

school outcome measures between STEP-UP interns and the comparison non-STEP-UP students. 

The analysis included students enrolled in Minneapolis Public Schools and analyzed outcomes 

for 270 interns and 186 matched students at the Achieve level and 227 interns and 153 matches 

at the Discover level. The propensity score models did not include variables representing prior 

academic achievement, and therefore we did not know if there was baseline equivalence between 

interns and matched students on the outcome variables in the year prior to the program. 

Therefore, we refit our models to control for prior achievement in the assessment of outcomes in 

the school year following the program. 

The second design, used to document program specific gains for interns, compared educational 

data from the school year prior to the internship to data from the school year after the internship. 

Employability gains were from the time of training (February/March) to the end of the internship 

(mid-August) of the same year and were measured using a survey in which interns rated 

themselves on a series of items. Employability outcomes included attitude, occupational or 

professional knowledge, future orientation, 21st Century skills such as teamwork and 

communication, and professional networks. 

For the within-group design we used paired-samples t tests, McNemar’s test, and chi-square tests 

to compare school outcome variables for the school year before the internship and the school 

year after the internship. We analyzed data for 270 Achieve interns and 227 Discover interns. 

On employability outcomes, we used the Wilcoxon signed-rank test to examine if the rank orders 

on the disagree/agree Likert type scale were the same or different for students pre- and postinternship. Pre- and post were the nominal level independent variables and the dependent 

variable was the response to each question on the Likert type scale. Employability analyses used 

data on 696 interns, 333 from Achieve and 363 from Discover. 

Research Questions 

Impact Evaluation Questions: Confirmatory 

1. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes than comparison group 

students?
 

b. Do the Discover participants achieve better school outcomes than comparison group 

students?
 

c. Do STEP-UP participants have a higher postsecondary enrollment than the comparison 

group? 

Impact Evaluation Questions: Exploratory 

1. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes pre- to post-program? 

b. Do the Discover students achieve better school outcomes pre- to post-program? 

Center for Applied Research and Educational Improvement, University of Minnesota 3 
 

 

 

 

 

 

 

 

 

2. Does the STEP-UP summer jobs program improve employability outcomes? 

a. Do the Achieve participants achieve better employability outcomes pre- to post-program? 

b. Do the Discover participants achieve better employability outcomes pre- to post-program? 

3.	 Do differential experiences in the STEP-UP summer job program – internship quality, length, 

longitudinal experience, and/or additional training – lead to differential outcomes on school 

and employability measures? 

a. Do students who work longer work weeks and/or more weeks in the program have better 

outcomes? 

b. Do students who work for STEP-UP multiple years in a row have better outcomes? 

Implementation 

1.	 Has the STEP-UP program implemented all of its major components with fidelity? 

2.	 To what extent has the STEP-UP program increased its outputs over time? 

3.	 How do perceptions of the STEP-UP Program impact applications and retention?6 

Study Logistics 

The study proceeded as planned. There were no issues or changes to the timeline or budget. 

There were no changes to evaluation personnel at CAREI. The research/evaluation staff position 

at AchieveMpls changed over the period of the grant; five different people held the position. 

Those changes did not affect the conduct of the study. There were no problems with the IRB. 

Findings 

Impact: Confirmatory/Moderate Evidence. The evaluation for 2016 for all STEP-UP 

participants (and for Achieve in 2015) demonstrated moderate evidence of a small positive effect 

on students’ being on track to graduate in four years. The evaluation did not find moderate 

evidence of positive program effects on other school outcomes, academic or behavioral, 

including post-secondary enrollment. The results did not change when controlling for prior 

achievement. One possible explanation for these results is that STEP-UP does not specifically 

target school-related skills in its trainings or internships. Using school outcomes, then, may be 

neither a desirable nor a realistic way of measuring the impact of the program on its participants. 

If changes in school-related metrics were the goal of the program, these school outcomes would 

need to be targeted more explicitly. 

In the area of behavior outcomes, most interns were not involved with the disciplinary system 

either before or after the internship, though in almost all categories a higher percentage avoided 

6 This set of questions about perceptions was added to the SEP modification that was approved in September, 2016. 

In light of the recognition by program staff that there are eligible students who miss the opportunity to do a STEPUP internship (including students who apply and even participate in training), project staff wanted to explore 

perceptions of the program by teens and to use their learning for improving program recruitment and retention. See 

the methods and findings sections for further information about the study. 

Center for Applied Research and Educational Improvement, University of Minnesota 4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

the disciplinary system the year following the internship than the year before. Of those students 

who had behavior incidents in the years prior to the internship, many had none the year after.7 In 

contrast, there were students who did not have incidents in the year prior to the internship who 

did have them in the year after. It may be that participation in STEP-UP influenced students 

whose behavior improved. In light of the fact that some students with no behavior incidents 

before the internship had them the following year, we have to consider that factors other than the 

program had an influence on behavior change. 

Impact: Exploratory/Preliminary. The exploratory impact study sought preliminary 

evidence about whether interns achieved better employability outcomes pre- to post-program. In 

2016, both Achieve and Discover interns improved in job-related skills: knowing what questions 

to ask in an interview, what questions to expect in an interview, and identifying people to use as 

professional references. These outcomes were consistent with the results of 2015. 

In addition to the skills mentioned above, in 2016 Achieve interns showed positive changes in 

self-assessments of a number of personal skills.8 Other than the interview and references skills 

Achieve interns did not show change in employment-specific skills pre- to post.9 In 2016 

Discover interns showed positive changes in their self-assessments of other employment-specific 

skills: knowing what clothes to wear to work, oral and written communication in a professional 

setting, and creating a professional resume.10 They did not show positive changes in their selfassessments of personal skills (e.g., taking initiative, problem solving). It is possible that 

Discover students, because they most likely had less experience in a work environment, had 

more to learn about workplace decorum and practices. Achieve interns, older and more mature, 

may have been more self-aware or more open to personal growth. 

The exploratory analyses examining differences for interns who worked more hours per week or 

more weeks per year showed no differences in employability outcomes. There were some 

7 At the Achieve level, of students with referrals in SY15, 66% fewer had them in SY16; of students with referrals in 

SY16, 88% fewer had them in SY17. Of students with removals in SY15, 68% fewer had them in SY16; of students 

with removals in SY16, 100% fewer had them in SY17. Of students with suspension in SY15, 79% fewer had them 

in SY16; of students with suspensions in SY16, 75% fewer had them in SY17. 

At the Discover level, of students with referrals in SY15, 68% fewer had them in SY16; of students with referrals in 

SY16, 74% fewer had them in SY17. Of students with removals in SY15, 82% fewer had them in SY16; of students 

with removals in SY16, 65% fewer had them in SY17. Of students with suspension in SY15, 59% fewer had them in 

SY16; of students with suspensions in SY16, 47% fewer had them in SY17. 

8 Staying calm under stress, accepting criticism, looking for additional tasks when work is complete, and breaking 

problems into small parts to solve them. The difference on breaking problems into small parts to solve them was not 

statistically significant after correcting for multiple comparisons. 

9 After correcting for multiple comparisons, the difference on naming references was not statistically significant 

from pre to post. 

10 After correcting for multiple comparisons there was not a statistically significant difference pre to post on resume 

writing. 

Center for Applied Research and Educational Improvement, University of Minnesota 5 
 

 

 

 

 

differences for interns with more than a single year of experience. Both Achieve and Discover 

interns had better outcomes about interview skills, and two personal skills and one employmentspecific skill. This suggests that having an internship for more than a single year may improve 

some employability skills. 

Implementation. STEP-UP implemented all components of the program with fidelity. 

The numbers of eligible applicants (3,447), students accepted (3,447), students completing workreadiness training (2,036), students placed in internships (1,389), students completing internships 

(1,272), number of companies (226), and number of supervisors (513) all increased in 2016 

compared to 2015. Most of the numbers were lower in 2016 than in 2014. In all categories where 

there were targets, actual numbers were lower than the 2016 targets. 

Among students who completed internships in 2016, there were more girls than boys (57% vs. 

43%). The racial/ethnic breakdown was 33% African Americans, 29% ethnic Africans, 11% 

Asians, 9% Hispanic, 6% Native Americans, 6% white, and 6% other or unidentified. Of 

completing interns, 90% were eligible for free or reduced price lunch. Just over 2/3 of Achieve 

interns held paying jobs prior to the internship; nearly 2/3 of Discover interns had never held a 

paying job. These statistics are roughly comparable to 2014 and 2015. 

There were high levels of satisfaction with the program among youth who completed an 

internship and among supervisors. Interns reported benefits such as personal learning and 

growth, professional learning, and the building of relationships. Supervisors reported benefits 

such as supporting the development of youth, energy and perspectives of youth in the workplace, 

expanding their supervisory/mentoring skills, learning about interns’ cultures, and getting help at 

their sites. 

Recommendations 

While this report concludes the SIF external evaluation, there are several areas of research that 

may be beneficial to AchieveMpls, the City of Minneapolis, and others in the field of youth 

employment regarding the benefits of internships on youth. 

1.	 Continue to examine longitudinal data to see if there are differences between a 

comparison group and interns. Because of the length of the grant, this evaluation was not 

able to access data on many participants who completed post-secondary education and 

entered the workforce. It is possible that participation in STEP-UP internships will have 

an impact on these outcomes, but we were not yet able to access data to examine this 

possibility. Tracking more youth over a longer trajectory and examining variables such as 

college graduation, type of employment, and wages could demonstrate longer-term 

impacts of the program. 

2.	 Focus on the use of other outcome measures to measure program impact. Short-term 

Center for Applied Research and Educational Improvement, University of Minnesota 6 
 

 

 

 

 

 

 

 

school outcomes do not show many positive effects of the program. This is logical given 

that the elements of the program (training, internship, career enrichments/specialized 

training) do not specifically target these outcomes. We know from survey responses that 

the program is having an influence on interns—expanding their consciousness about what 

the world of work is like, developing their networks in the professional community, 

helping them imagine what kind of career they might like and what path they need to take 

to achieve it, and becoming more comfortable in the workplace. 

Developing an instrument to measure these findings and comparing them to peers over 

time may provide useful information for the program going forward. We encourage 

STEP-UP to move in the direction they are considering. A retrospective pre-post 

instrument on which program participants can think more carefully about their growth 

rather than simply rating themselves could be helpful. One way to develop an instrument 

is to conduct interviews and/or focus groups to help develop pre-post questions. It may 

also be possible to collaborate with Minneapolis Public Schools on its measures of social 

and emotional learning instruments. The results could compare supervisor ratings of 

students. 

3.	 Consider a rigorous exploration of the dynamic between the supervisor and the intern as a 

contributing factor to success. This could draw on and contribute to the extensive 

research base on coaching/mentoring youth (and connects to other programs run by 

AchieveMpls). Interviewing both supervisors and interns could identify key factors that 

influence a productive internship as well as some of the longer-term goals of the 

program. 

Growing out of the implementation and impact studies, there are a number of steps the program 

could consider for strengthening the work of STEP-UP. 

1.	 As the curriculum for training interns is revamped, align training curriculum for interns 

more closely with the desired employability outcomes. Having seen changes in the 

students’ self-ratings on skills specifically in the training (e.g., interviewing skills), the 

evidence suggests that even over a short period of time it is possible to influence the 

growth of participants in the program. 

2.	 Better align training and oversight of supervisors with the desired employability 

outcomes for interns. Provide increased support throughout the internship period that 

focuses on these outcomes. In the 2016 and 2017 program seasons, STEP-UP staff 

members took steps to work with the supervisors to encourage their intentional work with 

interns on employability outcomes. This continues to be an area of potential development 

for the program, though not a simple one to achieve given the challenges of finding 

supervisors, constraints on the time supervisors have available for working directly with 

Center for Applied Research and Educational Improvement, University of Minnesota 7 
 

interns, skills of the supervisors, constraints on STEP-UP staff time, and of course, the 

complex nature of many of the employability outcomes. 

3.	 Revamp the logic model to eliminate school outcomes that are not impacted by the 

program and continue to revise it as other outcome are demonstrated to have positive 

effects. 

Center for Applied Research and Educational Improvement, University of Minnesota 8 
 

 

 

 

 

Introduction
 

Purpose and Audience 

The purpose of the evaluation was to examine the implementation and impact of the STEP-UP 

internship program for the summer of 2016, outlining the enduring elements of the program over 

time, as well as changes to the program during the period of funding from SIF through Greater 

Twin Cities United Way in partnership with Generation Next. This evaluation report is submitted 

to the Corporation for National and Community Service (CNCS) and its Social Innovation Fund 

(SIF). Its findings are also directed to SIF grantees, Generation Next and the Greater Twin Cities 

United Way, and to the subgrantee, AchieveMpls, and their partner in the STEP-UP program, the 

City of Minneapolis. 

Theory of Change 

Intended outcomes. The intended short-term outcomes for STEP-UP student participants 

include improved attitude, satisfaction of experience, increase in work-related skills, increase in 

occupational/professional knowledge, improvement in 21st Century skills like teamwork and 

communication and increase in professional capital. Moderate-range outcomes are improved 

educational achievement, improved school attendance, improved school behavior, and improved 

school engagement. Longer term outcomes are increased graduation rates, increase in college or 

other post-secondary enrollment, increase in college persistence and obtaining a post-secondary 

degree, increase in employment rates and increase in earnings. The anticipated impacts are a 

more diverse and skilled workforce in entry-level professional careers, particularly in the 

targeted pipeline areas; program alumni with high wages, high quality occupations, and high 

qualities of life; and employers better prepared to employ a more diverse workforce (see 

Appendix A for the STEP-UP theory of change and logic model). 

Prior research and level of evidence. Prior to this evaluation, the evidence for STEPUP’s efficacy was at a preliminary level. One study conducted by University of Minnesota 

researchers used propensity score matching to examine academic outcomes of STEP-UP students 

(Maruyama, VanBoekel and Cobie, 2013). The results showed that the STEP-UP program had a 

marginally significant impact on standardized test scores in certain sub-groups but did not have 

an impact on behavior or attendance. The present study used only MPS students, which account 

for about 60 percent of the STEP-UP students. 

Targeted level of evidence. This study used two designs. First, a quasi-experimental 

matched-groups design examined the program’s effect on educational outcomes and postsecondary enrollment and targeted a moderate level of evidence. The confirmatory question was 

Center for Applied Research and Educational Improvement, University of Minnesota 

7 
 

 

 

 

 

 

 

 

 

“Does the STEP-UP summer jobs program improve school outcomes?” See a fuller explanation 

on pg. 23 below. 

The second design was a quasi-experimental one-group pretest-posttest within participants 

design. It addressed the confirmatory questions, “Do the Achieve participants achieve better 

school outcomes pre- to post-program?” and “Do the Discover students achieve better school 

outcomes pre- to post-program?” The same design was used to address the exploratory questions, 

“Do the Achieve participants achieve better employability outcomes pre- to post-program?” and 

“Do the Discover participants achieve better employability outcomes pre- to post-program?” The 

second design targeted a preliminary level of evidence. 

Program Description 

STEP-UP is a youth employment program for Minneapolis youth ages 14-21. STEP-UP serves 

youth who face some of the greatest barriers to employment, including youth from low income 

families, youth from immigrant families, and youth with disabilities. STEP-UP provides two 

levels of coordinated support and internship opportunities. Achieve is for students ages 16 – 21 

while Discover is for ages 14 – 15. STEP-UP is a partnership among the City of Minneapolis, 

AchieveMpls, Department of Employment and Economic Development (DEED) and Project for 

Pride in Living. STEP-UP recruits, trains, and places low-income youth in paying jobs with a 

wide range of Twin Cities businesses, nonprofits, and public agencies. The program has both 

school year and summer internships; the latter serve the bulk of participants. The work readiness 

training is credentialed by the Minneapolis Regional Chamber of Commerce. STEP-UP helps 

interns explore diverse career interests, gain vital skills, make professional connections, and 

prepare for meaningful careers. At the Achieve level, most interns’ salaries are paid by the 

employers.11 At the Discover level, salaries are paid by the City of Minneapolis. 

The STEP-UP programming cycle for the summer internships program runs from December 

through August. Beginning in December, STEP-UP staff members recruit students to apply to 

the program before the application deadline in late February. Generally, all eligible students are 

accepted to the program and invited to job trainings occurring throughout the spring. In 2016, 

STEP-UP received 3,447 eligible applications. Two-thousand two-hundred students completed 

work-readiness training in 2016. During the training period, older students (16+) submit an 

additional online application to provide information that will be used to identify what types of 

job opportunities would be most appropriate and interesting for each student. (In 2014 and 2015, 

younger students (14-15) also submitted this additional application.) In May and June, students 

11 AchieveMpls also secures some foundation funding to subsidize wages. 

Center for Applied Research and Educational Improvement, University of Minnesota 

8 
 

 

 

 

 

 

are placed with employers.12 Students complete their internships by mid-August, although exact 

start and end dates are determined by each employer. In 2016, 1,389 students started internships 

and 1,272 completed internships. 

Recruitment. STEP-UP recruits both students and employers. During the fall, winter, 

and spring, recruiting employers and identifying specific internship positions are key activities of 

STEP-UP staff from AchieveMpls and the City of Minneapolis. Student recruitment takes place 

during the first half of the school year. Students submit applications electronically. 

The majority of recruitment efforts take place when the online application opens in December. 

Although the program serves youth 14 to 21 years old, STEP-UP program associates are 

assigned schools for recruitment purposes, including the seven comprehensive Minneapolis 

Public Schools district (MPS) high schools and MPS alternative schools. STEP-UP staff work 

closely with school leadership in developing recruitment plans, a process that starts with the 

MPS Superintendent sending a letter of endorsement of the program and request for support to 

all high school principals. Staff members use several different methods including classroom 

presentations, targeted after school group presentations, lunchroom tabling, school 

announcements, signs in schools, parent and student newsletters, posters, flyers, a video, and 

meetings with school staff. Staff hold regular after school application help sessions to assist 

students with their applications and answer questions about the program. STEP-UP also directs 

communication about the internship program and resources (a recruitment “toolkit”) to MPS 

middle schools, Minneapolis’ charter and private schools, and schools in nearby suburbs where 

some Minneapolis students attend, and to community based and education focused nonprofit 

partners whose youth workers help spread the message and support application completion. 

In 2016, STEP-UP revamped student recruitment flyers and presentations based on findings of a 

2015 qualitative study of student perceptions of the program. Staff focused on targeted 

participant populations and developed new messaging themed around (a) empowering and 

unlocking potential, (b) making money, and (c) the opportunity for students to shape their own 

experience within the program. The new approach also focuses on clarifying the breadth of 

opportunities available across program components without burdening recruits with excessive 

information regarding the process. 

The majority of students who apply are accepted and invited to participate in training. Each year 

there are students who are not accepted into the program because they do not meet the following 

eligibility requirements: live in the City of Minneapolis, between the ages of 14 and 21, not 

12 Due to the number of available internships, not all students who are eligible and interested are placed in 

internships. 

Center for Applied Research and Educational Improvement, University of Minnesota 

9 
 

 

 

 

 

 

 

 

 

 

enrolled in college and either have a qualifying income or meet standard barriers to employment 

indicators (in foster care, have an IEP, pregnant or parenting, English language learner, homeless 

or highly mobile, or involved in the criminal justice system). 

Training. STEP-UP provides four distinct training tiers, with student training schedules 

and curricula differentiated by participant age and prior experience in the program. While 

learning targets are distinct across the 4 tiers, objectives for the training focus on distinguishing 

between jobs and careers, networking, landing a job, employment documentation and onboarding 

procedures, resumes, interviewing, workplace communications, workplace 

expectations/professional etiquette, attitude and character, problem-solving and decision-making, 

workplace ethics, supervisor role, and the importance of feedback/developing a continues 

improvement mindset. The curriculum for the Discover trainings was not changed from 2015. 

This curriculum was originally revised and piloted prior to Summer 2013. Many of the 

curriculum modules were selected from an existing US Government resource13, “Soft Skills to 

Pay the Bills,” and adapted for the STEP-UP context. To a lesser extent they included pieces 

from “Learning for Life.14” 

In 2016 the structure of the training was the same as in 2015. The trainings were day-long events 

on Saturdays and took place at Minneapolis Community and Technical College in downtown 

Minneapolis. 

Trainers for all the high school students were either staff of AchieveMpls or individuals with 

whom Achieve contracts to conduct training programs. Train the trainer sessions were held prior 

to the launch of the student training. Overall, 2,036 students participated in training. 

STEP-UP Achieve (Youth 16-21 years of age). Trainings were offered on four 

consecutive Saturdays; each youth was expected to participate in one day of training. Intake 

interviews were scheduled on the same day as training. Mock interviews were held separately 

during the following month. Training and interviews totaled 9 hours. 

STEP-UP Achieve: Advanced (Youth 16-21 years of age who have already completed 

an Achieve level internship). Training for Advanced level interns was revamped in 2015. Areas 

of skill development were similar to the Achieve level, but interns worked on attaining a higher 

13 www.dol.gov/odep/topics/youth/softskills/ 14 The Learning for Life Corporation offers seven programs designed to support schools and community-based 

organizations in their efforts to prepare youth to successfully handle the complexities of contemporary society and to 

enhance their self-confidence, motivation, and self-esteem. The programs focus on character education and career 

education. 

Center for Applied Research and Educational Improvement, University of Minnesota 

10 
 

 

 

level of competency. Workshops to address specific skills were conducted by volunteers in 

training/talent development roles at companies that hire interns. In addition to the workshops, the 

Advanced training included keynote addresses by professionals and former STEP-UP interns, 

making the format more like a professional conference. Students also had the opportunity to 

choose sessions that reflected their interests. Training and interviews totaled 9 hours. 

STEP-UP Discover: High School (Youth 14-15 years old and in high school). Trainings 

were held on four consecutive Saturdays; each youth was expected to participate in two days of 

training. Mock interviews were held separately during the following month. High School 

Discover trainings were conducted by AchieveMpls. Training and interviews totaled 14 hours. 

STEP-UP Discover: Middle School (14-15 years old and in middle school). The training 

and placement of interns in the Discover section of STEP UP is coordinated by the City of 

Minneapolis. The city contracts with Project for Pride in Living (PPL) to conduct trainings for 

Middle School Discover students. The curriculum for Middle School Discover student training is 

coordinated with that of the high school training. Topics are similar, but the content is more basic 

since it is designed for younger students who are less likely to have had experience in the world 

of work. Some of the materials used are from the curriculum Soft Skills to Pay the Bills. In 2016 

these trainings took place at Minneapolis Community and Technical College (MCTC) on two 

Saturdays in March for a total of 12 hours of training. Middle school youth do not attend a mock 

interview. 

The training culminates in a series of mock job interviews with hundreds of volunteers from the 

city’s professional community. All graduates of the training who go on to successfully complete 

a summer internship receive a work-readiness credential certified by the Minneapolis Chamber 

of Commerce. A detailed summary of the training curriculum is provided in Appendix B. 

Summer Internships. The paid summer internship is the key component of the STEPUP program. Most internships run for nine weeks from mid-June to mid-August. Some 

internships, however, are shorter in length while others are longer. Interns’ work opportunities 

are in either STEP-UP Achieve (for youth ages 16-21) or STEP-UP Discover (for youth ages 14 

and 15). STEP-UP Discover, which is run by the City of Minneapolis in coordination with 

AchieveMpls and the Minnesota Workforce Centers, offers foundational, entry-level jobs in 

nonprofit organizations for youth with limited work experience. STEP-UP Achieve offers more 

challenging internships; Achieve interns are placed at private companies, government agencies, 

educational institutions and nonprofits and employed directly by those companies. The City of 

Minneapolis contracts with Workforce Centers to provide job coaches to support Discover 

interns and employers during the internship experience. AchieveMpls staff serve as job coaches 

Center for Applied Research and Educational Improvement, University of Minnesota 

11 
 

 

 

 

to offer training and support both to Achieve interns and their supervisors. In the summer of 

2016, 1,389 students were placed in internships and 1,272 completed internships. 

Placement. Once a student has completed training and submitted required paperwork, he 

or she is eligible to be matched with internship positions and placed with employers. Students 

who are not matched to positions are provided with a list of resources for finding a job without 

the program. The process of matching students to internships differs for Achieve and Discover. 

Achieve. Achieve interns are matched with jobs by AchieveMpls staff members. To be 

eligible, a student must have completed training. Achieve has developed a philosophy of 

placement; its goal is to create the best matches for the benefit of all interns and partner 

companies. The best match or placement for any single position is balanced with what is best for 

the entire pool of interns and available positions. 

Interns are placed in a wide range of industries. STEP-UP Achieve interns work in careeroriented internships in private sector businesses, public agencies, and nonprofits on a rolling 

basis throughout the month of May. As companies submit their internship position description 

via an online form, Achieve staff review them for fit, assign them to a select list of “job types” 

and designate some as being “high priority” placements (based on position/employer 

requirements and timeline required for intern onboarding). These priority placements are the first 

positions Achieve staff fill in the placement period. Achieve staff members use a database with 

job descriptions and information about the students (including applications and resumes) to 

create matches. They make matches on the basis of the interns’ industries of interest; their skills; 

their availability; their transportation opportunities; and whether they have completed specific 

training for certain career areas. These must align with the internship positions’ requirements for 

work schedule, location of employment, skills required, and industry of employment. STEP-UP 

sends companies the resumes and contact information for their matched interns. Interns receive 

notification of their placement and a job description. Supervisors are expected to contact the 

intern candidates within 3 days to schedule an interview. The interview is meant to ensure that 

the intern and the supervisor feel secure that the internship will be successful. If after the 

interview either party determines that it is not a good fit, STEP-UP works to place another 

candidate in the position, and makes the youth available for placement into another position. 

Center for Applied Research and Educational Improvement, University of Minnesota 

12 
 

 

 

 

 

 

 

 

 

 

 

 

Funding for Achieve internships comes primarily from the employers. AchieveMpls also secures 

some foundation funding to subsidize wages. Achieve interns are paid at least state minimum 

wage, and salaries vary depending on the internship. The pay rate for each position is set by the 

company or organization, based on their assessment of the value of the position and the work 

they expect to be produced. 

As in the past, there were fewer jobs available in 2016 than students who applied and completed 

training. Some students, because of family commitments or summer school, withdraw their 

applications even after completing training. 

Discover. Middle school and high school Discover youth who complete work readiness 

training have an intake interview with the Minneapolis Workforce Center (contracted by the City 

of Minneapolis) at the Minneapolis Convention Center. Students are required to bring 

employment eligibility, age and identity documents with them to the appointment. Each student 

meets with a job counselor, reviews the internship application and talks about availability, 

interests and skills. The job counselor reviews open jobs and helps the student select a job that is 

a good fit. The student is then given a job description and told that their supervisor will contact 

them in the next few weeks. STEP-UP staff later follow up with supervisors to provide a list of 

youth and asks them to call the youth to come in for an introduction or interview. Interviews are 

optional, but encouraged, at the Discover level 

STEP-UP Discover interns, who are younger or have limited work experience, serve in entry 

level jobs in nonprofit organizations. Their program is funded and run by the City of 

Minneapolis with significant involvement by The Workforce Center.15 All jobs have the same 

hourly wage which the city sets. 

In 2016 the staff of STEP-UP Discover decided to limit internships to 20 hours per week plus 

three hours for class (down from 30 hours in 2015). By making the change the City of 

Minneapolis STEP-UP staff placed 100+ additional Discover youth into paid internships using 

the same funding for wages as in the prior summer. Achieve interns worked 20 to 40 hours per 

week. 

Preparing and Supporting Supervisors. Prior to the summer, STEP-UP Achieve and 

STEP-UP Discover provides an employer orientation and a supervisor handbook. 

The Minnesota Workforce Center team assists with youth recruitment, matches nearly 1,000 youth into positions, 

processes payroll paperwork and manages time reporting throughout the summer. 

Center for Applied Research and Educational Improvement, University of Minnesota 

13 

15 
 

 

 

 

 

Orientation. The purpose of the orientation is to inform, engage, prepare, and inspire 

supervisors to establish a work plan and strategy for a successful summer student internship 

experience—for both parties involved. The orientation is considered mandatory for companies 

that pledged student placements. In reality, not all supervisors complete the orientation. 

AchieveMpls revised and expanded its STEP-UP Achieve supervisor orientation in 2016 to make 

the orientation more practical and engaging for participants and to align it more closely with the 

interns’ work readiness training. The program also offered continuing education for supervisors 

as a collaborative effort between STEP-UP and a number of other youth employment programs 

in the Twin Cities (Right Track, and The Brand Lab, Genesys Works, and Brooklynk). For 

Discover supervisors, orientation is a mandatory training for staff members tasked with regularly 

supervising an intern. Topics include STEP-UP overview and partnerships, supervisor 

responsibilities, tips and resources for how to be a good mentor, and payroll and safety 

procedures. 

Supervisor Handbook. The handbook is provided in hard copy at orientations and is also 

available electronically to all supervisors. It addresses these topics: adolescent development, 

STEP-UP work readiness training, the role of the supervisor, building and maintaining the 

supervisor-intern relationship, and templates to use during the internship (e.g., hire letter, work 

plan, weekly check-in). 

Job Coaches. During the summer internships, STEP-UP staff from AchieveMpls serve as 

job coaches to the Achieve interns. Discover interns are coached by temporary summer staff 

from the Workforce Center team. 

This oversight helps the program staff to monitor the quality of the internships. In addition, 

coaches address troubles that arise, both from the perspectives of the employers and of the 

interns. They mediate difficult situations and coach interns to understand what behaviors are and 

are not acceptable in the workplace (e.g., being on time, appropriate dress, responding to 

feedback). They handle intern concerns, such as paychecks and the payment 

process, as well as difficulties with supervisors. Discover job coaches visit each worksite at least 

once over the course of the summer, and often twice to check in and troubleshoot challenges 

with the intern as needed and to ensure the worksite is meeting program expectations. Achieve 

and Discover sent weekly emails to supervisors with information on upcoming STEP-UP events, 

best practices and tips for working with interns. Approaches to communicating with supervisors 

shifted over the course of the grant period, from Constant Contact to electronic newsletters, 

varying the amount of content and number of images with the goal of increasing readership and 

understanding of the program. 

Center for Applied Research and Educational Improvement, University of Minnesota 

14 
 

 

 

 

 

 

 

 

 

Specialized Training/Career Enrichment. In addition to general pre-internship required 

training, STEP-UP, in collaboration with various companies and organizations, offers targeted 

trainings focused on specific career-related knowledge and skills, career exposure events, and 

industry-recognized certifications. These take place before and during the summer internship 

experience. Some are required of interns in particular career fields and open to those interns 

only, such as Scrubs Camp which serves youth in healthcare internships; others are open to 

anyone interested. Most are optional. They are promoted to interns both by STEP-UP staff and 

by employers and are coordinated by AchieveMpls. AchieveMpls categorizes and tracks student 

participation in experiences that are under 20 hours and ones that last 20 hours or more. Over the 

SIF subgrant period, some training and enrichment opportunities were offered each year, while 

some were newly introduced or discontinued each summer. For a description of career 

enrichment and special trainings offered to STEP-UP interns in the summer of 2016, see 

Appendix D. In 2014-16, Discover interns participated in weekly training classes (see above, pg. 

11). For changes made to Discover summer training, see Appendix C. 

Discover: High School and Middle School Summer Training. In addition to the training 

prior to the summer, Discover interns participate in classes during the internship period. In the 

summers of 2014-16 interns attended classes were held once a week for eight weeks; this was a 

vestige in the program from the time when there was not training in the spring for Discover 

interns. Based on feedback from participants, this piece of the program was changed in 2017 (see 

Appendix C). 

Center for Applied Research and Educational Improvement, University of Minnesota 

15 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Method: Implementation Evaluation
 

Implementation Evaluation Questions 

The following questions guided the implementation evaluation of the STEP-UP program. 

1. Has the STEP-UP program implemented all of its major components with fidelity? 

a. To what extent have the youth output targets been met? (comparison of current 

year’s outputs to current year’s targets) 

i. How many youth applied to the program? 

ii. How many youth completed the Work-Readiness training? How many 

internships did youth complete? 

iii. How many youth attended extra enrichments? (<20 hours; >20 hours) 

iv. How many interns completed specialized career trainings? 

v. How many interns earned work-based learning credit? 

vi. How much did youth earn in wages in internships? 

vii. How many youth attended weekly classes? 

b. To what extent have the employer output targets been met? 

i. How many companies employed STEP-UP interns? 

ii. How many supervisors supported STEP-UP interns? 

iii. In what types of settings did STEP-UP interns work? 

c. To what extent have systems building output targets been met? 

i. What enrichment opportunities were offered as part of systems building? 

ii. What is meant by systems building? 

iii. How many youth were engaged in the various kinds of enrichment 

opportunities? 

d. How do the components of the implemented STEP-UP program compare to those 

described in the proposal? 

e. What was the nature of student training for STEP-UP? 

i. How many training programs were run? 

ii. What were the purposes of the different trainings? 

iii. How were the training programs developed? 

iv. What were the components of the trainings? 

v. Who participated in the different trainings? 

vi. How did staff perceive the training: materials, implementation, and 

reactions of participants? What changes would they make if the trainings 

are offered again? 

vii. How did participants perceive the training? 

Center for Applied Research and Educational Improvement, University of Minnesota 

16 
 

 

 

 

 

 

 

 

viii. How could the quality of training be assessed? 

f.	 Who were the participants in the program? 

g.	 What was the participants’ satisfaction with the program? 

h.	 What was the quality of the student internships? 

2.	 To what extent has the STEP-UP program increased its outputs over time? 

a.	 For students? 

b.	 With employers? 

c.	 Regarding systems-building? 

3. How do perceptions of the STEP-UP Program impact applications and retention?16 

a.	 How do youth and personnel in MPS perceive the STEP-UP program? 

b.	 What factors influence youth to apply to STEP-UP? 

c.	 What factors influence youth retention in the STEP-UP program (from application 

through training; from one year to another) 

Implementation Modifications to SIF Evaluation Plan (SEP) 

Some modifications to the implementation questions were made in October, 2015 and others 

were included in the modification approved in September, 2016. Questions that were 

inadvertently repeated in the original SEP were omitted. The question about extra enrichments 

was adjusted to reflect the program’s differentiation between experiences of 20 hours or less and 

those of more than 20 hours. The question about academic elective credits was subsumed by 

another question in the approved SEP with the remaining question reflecting more accurately the 

language used by program staff and participants. A question was added about the number of 

internships available to youth because this is the one type of employer output with specific 

targets. Questions about systems buildings and pipelines were adjusted to reflect the program’s 

shift to an output-oriented focus in this aspect of their work. A question about recognition of 

work and training experiences were omitted due to lack of data. Finally, a set of questions was 

added to address an area of significant interest to the program staff—perceptions of the program 

and their influence on applications and retention in the program. 

Data Collection, Measurement, & Analysis: Implementation 

16 This set of questions about perceptions was added to the SEP modification that was approved in September, 2016. 

In light of the recognition by program staff that there are eligible students who miss the opportunity to do a STEPUP internship (including students who apply and even participate in training), project staff wanted to explore 

perceptions of the program by teens and to use their learning for improving program recruitment and retention. See 

the methods and findings sections for further information about the study. 

Center for Applied Research and Educational Improvement, University of Minnesota 

17 
 

 

 

 

 

 

 

 

 

Data collection, measurement, and analysis for the implementation evaluation focused on 

outputs, fidelity to program design, dosage, program exposure, participant satisfaction, and 

perceptions of the program. In terms of satisfaction, the perspectives of both interns and 

supervisors were explored. 

Data Collection. There were several sources of data for this implementation evaluation. 

Staff members at AchieveMpls and the City of Minneapolis provided program information 

through written correspondence and in-person interviews. We also drew on program documents 

such as handbooks and email correspondence, as well as the program’s website. 

Data about training were collected from 1,916 student participants (94% response rate) at the end 

of the training period in person using a survey prepared by STEP-UP staff in InspiroScan. It was 

administered as a paper and pencil survey, and results were compiled electronically using 

InspiroScan. Data about satisfaction with training come from the perspective of the students. Due 

to other demands on staff, and since the training curriculum had changed very little in 2016, 

there was not a debriefing for the trainers in 2016. 

Baseline information about interns on employability skills (primarily used for the impact 

evaluation) was gathered using a survey prepared in InspiroScan Survey developed by STEP-UP 

and CAREI staff. The questions on the baseline survey in 2016 were revised to align with the 

MHA Labs17 Hirability Skills framework (see pg. 37). Program staff hoped that this framework 

would be used by supervisors over the summer to support their supervision of interns and as a 

way to measure intern growth on these targeted skills. The baseline survey was administered as 

a paper and pencil survey to students at the beginning of training; results were compiled 

electronically and analyzed in SPSS. Students completed the baseline survey (n=2,041), 

including 85% of those who completed a summer internship. 

End of summer intern surveys, developed by STEP-UP and CAREI staff, were administered 

electronically through SurveyMonkey. Each intern received a unique link to the survey in order 

to identify results. Staff at AchieveMpls emailed the surveys to all interns and followed up by 

text messages to increase the response rate. The same employability questions from the baseline 

survey were included, as well as other questions about experiences in the internship. Results 

were compiled in Excel through SurveyMonkey and were analyzed in SPSS. Four hundred 

seventy-three Achieve interns (77% response rate) and 406 Discover interns (62% response rate) 

completed the end of summer survey. 

17 MHA (Means and Measures of Human Achievement) Labs is a public/private research and development nonprofit 

that designs products and services for 21st century skills development with a focus on youth. Since 2011, the 

organization has been developing and refining a work readiness/employability assessment tool. 

Center for Applied Research and Educational Improvement, University of Minnesota 

18 
 

 

 

 

 

End of summer supervisor surveys, developed by STEP-UP and CAREI staff, were also 

administered electronically through SurveyMonkey. Unique links were emailed to supervisors, 

and reminder emails were sent to increase response rates. The link was not individualized. 

Results were compiled in Excel through SurveyMonkey and were analyzed in SPSS. These 

surveys provided data on the experience of supervisors with their interns and with the program 

and were used for the implementation evaluation. 

Of the 513 supervisors from the summer of 2016, 341 (66%) responded to an end-of-summer 

survey (105 Discover—82%; 236 Achieve—61%). 

Demographic data about students, as well as information about training levels and placements, 

were drawn from the STEP-UP database and provided to the evaluation team by STEP-UP 

program staff. We also drew on program documents such as handbooks and email 

correspondence, as well as the program’s website. 

Prior to the summer of 2016, STEP-UP staff worked with their CAREI evaluators to revamp the 

baseline survey for assessing employability outcomes. The change was made to accomplish two 

goals: (1) to use items from an instrument that had been tested extensively and that had the 

potential to provide data with stronger validity, and (2) to align the tool for measuring this 

change with a tool that supervisors were using to provide feedback to and about their interns. 

The materials from MHA Labs were streamlined and further tested prior to the 2016 program 

season. The staff’s intention was to have supervisors focus on the skills with strong correlations 

to employability, to use the framework of hirability skills to give feedback to interns, and then to 

measure growth on the pre- and post-internship survey on the same skills. Supervisor 

orientations devoted time to the hirability framework from MHA Labs, and weekly emails sent 

by STEP-UP staff addressed its use as well. Data from the end-of-summer surveys of both 

interns and supervisors indicated that many supervisors did not use the framework. 

Data about perceptions were gathered from teens in seven semi-structured group interviews in 

the spring and summer of 2016. Four were held at two different high schools in late May and 

early June. Three group interviews were held in July at different work sites. Interview 

participants were grouped according to their experience with STEP-UP: students who never 

participated in STEP-UP, students who applied to the program but did not do training, students 

who did a Discover internship but did not continue into Achieve the following summer, and 

students who did Achieve internships. The CAREI evaluators worked with staff from 

AchieveMpls to conceptualize the study, develop data gathering instruments, and analyze data. 

Achieve staff arranged for and conducted the interviews, led the work on data analysis, and drew 

conclusions from the study. Interviews lasted approximately 30 minutes and groups ranged in 

size from 2-9 students. Staff from the high school Career and College Centers assisted with the 

interviews and took notes in the spring interviews. Staff from AchieveMpls assisted with the 

Center for Applied Research and Educational Improvement, University of Minnesota 

19 
 

 

 

 

 

 

summer interviews and took notes. The interviews were recorded for analysis purposes. The 

notes and recordings were analyzed to identify themes. 

Measurement. The implementation evaluation examined (a) outputs (b) fidelity to 

program design; (c) program dosage; (d) program quality—including exposure and satisfaction; 

and (e) perceptions of the program. Figure 1 outlines the sources of data for each of these areas. 

See Appendix E for specific survey questions used to measure various program components. 

OUTPUTS 

Program component Data source 

Training STEP-UP database 

Pipelines/Career Enrichment 

Activities 

STEP-UP database 

Internships: Interns STEP-UP database; baseline survey; end-of-summer intern survey 

Internships: Supervisors STEP-UP database 

Internships: Placements STEP-UP database 

FIDELITY OF IMPLEMENTATION 

Program component Data source 

Training Staff interviews; program website; program documents 

Pipelines/Career Enrichment Staff interviews; program website; program documents 

Internships Staff interviews; program website; program documents 

PROGRAM DOSAGE 

Program component Data source 

Training Program documents 

Internships Program documents 

Pipelines/Career Enrichment Program documents; STEP-UP database 

PROGRAM EXPOSURE/SATISFACTION 

Program component Data source 

Training Post-training survey; end-of-summer intern survey; program 

documents 

Internships End-of-summer intern survey; end-of-summer supervisor survey; 

program documents 

PERCEPTIONS OF THE PROGRAM 

Program component Data source 

Center for Applied Research and Educational Improvement, University of Minnesota 

20 
 

 

 

 

Training Group interviews conducted by AchieveMpls staff 

Internships Group interviews conducted by AchieveMpls staff 

Figure 1: Sources of data for implementation evaluation 

Analysis. The following outlines analysis procedures for each of the following parts of 

the implementation evaluations: (a) outputs (b) fidelity to program design; (c) program dosage; 

(d) program exposure; (e) participant satisfaction and (f) perceptions of the program. 

Outputs. Program outputs were analyzed in SPSS using data drawn from the STEP-UP 

database. 

Fidelity to program design. Current practices were compared to the description of the 

program in the SEP. 

Program dosage. Dosage is defined as the experiences provided to participants by 

AchieveMpls and the City of Minneapolis. These were training and the assignment of internships 

for all participants, required career enrichment activities for some participants, and optional 

career enrichment activities. For the training component of the program, dosage was the same for 

all participants at each training level and was reported by program staff; number of hours were 

drawn from program documents. Completing an internship was considered the internship dosage. 

Since the program did not control the number of hours per week that interns worked nor the 

number of weeks worked over the course of the summer, they cannot be considered dosage. 

Dosage for career enrichment activities was determined by counting the number of enrichment 

activities in which an intern participated that were 20 hours or fewer (if any) and the number of 

activities interns participated in that were more than 20 hours (if any). 

Program quality. Quality of internship was nearly impossible to measure given the vast 

range of settings and experiences that constituted STEP-UP internships. Using the guidelines and 

recommendations that program staff provided to supervisors in orientations, handbooks, and 

emails during the summer, it is possible to infer what staff believe contribute to a high-quality 

internship. Elements of quality that were examined were exposure and satisfaction. 

Exposure. One element that contributes to program quality is exposure. We define 

exposure as opportunities interns provided to interns at the internship placement. While the 

STEP-UP program recommended that supervisors provide particular types of opportunities 

during the summer, program staff had no control over whether they were provided, nor, if 

provided, the frequency or extent of those opportunities. 

Center for Applied Research and Educational Improvement, University of Minnesota 

21 
 

 

 

 

 

 

 

Recommended experiences for interns were communicated to supervisors in the orientation, 

supervisor handbook and weekly emails to supervisors. There is no way to know exactly how 

much time was devoted to any of these things in any given internship. Using intern and 

supervisor end-of-summer surveys, we were able to calculate estimates of the extent and or 

frequency of particular activities and experiences for participants who completed the surveys. 

The items used a six-point Likert scale, and the categories were collapsed to three—weak (1 or 2 

on the Likert scale), moderate (3 or 4 on the Likert scale), and strong (5 or 6 on the Likert scale), 

and percentages were calculated for each of the three categories on each item. These were the 

items examined: 

	 Providing a job description; creating a work plan (expectations and goals) 

	 Implementing a strategy for supporting the intern (giving clear directions, checking in 

regularly and reflecting on the experience together, talking to them about future plans, 

helping them develop professional skills, talking to them about the company and its 

activities, talking about their own education and work history) 

 Establishing a project for the intern to undertake
 

 Providing a mentor (either the supervisor or another adult)
 

 Giving regular feedback to the intern (including a mid-internship check-in and a
 

performance evaluation) 

 Opportunities to learn about a career in the specific fields through activities such as job 

shadow, attending meetings, informational interviews 

 Developing networks (e.g., lunch with supervisor, attending social gatherings with 

colleagues, meeting the president or CEO, attending meetings) 

 Specialized training (e.g., Excel, Outlook) 

Comparisons were made between supervisor and intern responses about the same activities. 

In the supervisor end-of-summer survey, respondents estimated the amount of time they spent in 

a typical week on a range of activities with their intern using a response scale from not at all to 

more than two hours. The eight categories were collapsed to three (not at all, 15-45 minutes, and 

1 hour or more) and percentages of responses were calculated for each of the three categories for 

each item. 

Participant satisfaction. Another element of program quality is satisfaction of 

participants—both interns and supervisors. We looked at satisfaction with training and with 

internships. 

Training. Responses to the Likert-scale and multiple-choice items on the post-training 

survey were tallied and percentages calculated using Excel for each response option (Strongly 

Disagree/Disagree/Agree/Strongly Agree). 

Center for Applied Research and Educational Improvement, University of Minnesota 

22 
 

 

 

 

 

 

 

Internships. Survey questions with Likert scale and multiple-choice responses were
 

tallied and percentages calculated in SPSS. Responses to open-ended questions were coded to 

identify patterns and themes. 

Perceptions of the program. The interviews were recorded for analysis purposes. Using a 

coding system, the notes and recordings were analyzed to identify themes. See Appendix F for 

more on information on the perceptions study. 

Study Logistics 

The study proceeded as planned. There were no issues or changes to the timeline or budget. 

There were no changes to evaluation personnel at CAREI. The research/evaluation staff position 

at AchieveMpls changed over the period of the grant; five different people held the position. 

Those changes did not affect the conduct of the study. There were no problems with the IRB. 

Method: Impact Evaluation 

Impact Evaluation Questions: Confirmatory 

1. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes than comparison group 

students?
 

b. Do the Discover participants achieve better school outcomes than comparison group 

students?
 

c. Do STEP-UP participants have a higher postsecondary enrollment than the comparison 

group? 

Impact Evaluation Questions: Exploratory 

2. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes pre- to post-program? 

b. Do the Discover students achieve better school outcomes pre- to post-program? 

2. Does the STEP-UP summer jobs program improve employability outcomes? 

a. Do the Achieve participants achieve better employability outcomes pre- to post-program? 

b. Do the Discover participants achieve better employability outcomes pre- to post-program? 

Center for Applied Research and Educational Improvement, University of Minnesota 

23 
 

 

 

 

 

4.	 Do differential experiences in the STEP-UP summer job program – internship quality, length, 

longitudinal experience, and/or additional training – lead to differential outcomes on school 

and employability measures? 

a. Do students with higher quality internship experiences have better outcomes? 

b. Do students who work longer work weeks and/or more weeks in the program have better 

outcomes? 

c. Do students who work for STEP-UP multiple years in a row and/or during the summer and 

school year have better outcomes? 

d. Do students who received additional training, support services and/or certifications have 

better outcomes than those students who do not?

 Impact Modifications to SIF Subgrantee Evaluation Plan (SEP) Adjustments were 

made in the questions proposed in the original SEP approved in 2013. These modifications were 

submitted in July, 2016, returned with a request for further information, and resubmitted 

September 12, 2016. They were approved September 21, 2016. One confirmatory impact 

question was omitted. It asked about “Fast Track” program participants; the question was 

removed because the program was discontinued. Several exploratory impact questions related to 

the Fast Track program were also removed. 

One confirmatory impact question about postsecondary enrollment was added. While originally 

intended to be included given the program’s theory of change and logic model, the question was 

removed because of the timeframe of the study and data availability. Because the study’s 

timeline was extended, and because the data became available, the question could be added to 

the evaluation. The postsecondary outcome of program enrollment is examined below. Other 

postsecondary outcomes mentioned in the original SEP (college persistence, obtaining a 

secondary degree, job placement, and income) are not explored since they will only be 

measurable and/or accessible after the period of the SIF subgrant has concluded. 

When conducting the analysis for the final report, it became clear that there were not data 

available to answer some of the exploratory impact sub-questions listed in the SEP about 

differential experiences and whether they lead to differential outcomes. The questions that were 

addressed were about students who worked more weeks or longer weeks and about students who 

participated in STEP-UP multiple years. Since we were unable to develop a measure of quality, 

it was not possible to examine outcomes for interns in internships of differing quality. 

Impact Study Design 

First design: A quasi-experimental between-groups design with matched 

comparison groups based on propensity score matching. To address the confirmatory 

Center for Applied Research and Educational Improvement, University of Minnesota 

24 
 

 

 

 

 

 

 

questions: “Do STEP-UP participants achieve better school outcomes than comparison group 

students?” and “Do STEP-UP participants have a higher postsecondary enrollment than the 

comparison group?” a quasi-experimental between-groups design with matched comparison 

groups from non-STEP-UP Minneapolis Public School (MPS) middle school and high school 

students was used to study the program’s impact on school outcomes. A comparison group was 

formed based on propensity score matching. 

The Research, Evaluation, Assessment, and Accountability staff at Minneapolis Public Schools 

also conducted an internal exploratory analysis of STEP-UP participants and matched 

comparison groups on data related to social-emotional learning, one of the targeted outcomes 

included in the SEP. These analyses looked at participants and their matches for 2014, 2015, and 

2016. 

Rationale. With the quasi-experimental design with matched comparison groups, the 

study sought to reach a moderate level of program outcome evidence. Random assignment of 

applicants to program or control groups was not feasible for this evaluation. Briefly, more youth 

apply than AchieveMpls has the capacity to train; only about half the applicants complete 

training; in the end, the available internships can only accommodate about three-quarters of those 

who complete training; those slots –1,389 in 2016– are filled by a most-qualified and best-fit 

selection process. This process of applying for and matching students to internships prevents the 

use of random assignment. 

Design. Four hundred ninety-seven (497) STEP-UP interns18 in the summer of 2016 were 

matched with 339 similar non-STEP-UP Minneapolis Public School students to study the 

program’s effectiveness on educational outcomes. The group of interns was compared with the 

group of matched MPS students on school outcomes (see below). The SEP anticipated a higher 

number of participants and matched students19. In order to be included in the analysis, 

participants had to be enrolled in Minneapolis Public schools for at least 75% of the year for two 

consecutive years, both before and after the internship, and they had to have completed a STEPUP internship. Using those criteria reduced the total number of students to match. There is 

adequate power for analyses that includes 497 participants and 339 matched students. The 

reduced numbers do impact the external validity of the analyses. It is possible that students who 

did not attend Minneapolis Public Schools or did not complete an internship were different from 

those included in our analyses. 

Threats to internal validity. The quasi-experimental design with matched comparison 

18 This is out of 1,272 students who completed an internship in 2016. 

19 The original estimate was 800 program participants and 800 matched students. 

Center for Applied Research and Educational Improvement, University of Minnesota 

25 
 

 

 

 

 

groups using propensity score matching techniques enables the formation of matched groups that 

have balance on a set of observed covariates (Stuart & Rubin, 2007). The covariate distributions 

in the program and comparison groups are only randomly different from each other in respect to 

the included covariates. Conceptually, this approach replicates a randomized experiment where 

study participants are balanced on characteristics across groups. There are potential limitations 

with this design. The most common limitation is the omission of important covariates that can 

lead to bias in statistical comparisons (Fan & Nowell, 2011). Although the evaluation team and 

AchieveMpls identified key covariates that predict the likelihood of students receiving their 

services, there may have been hidden covariates that should have been considered. For example, 

the internal threat of history to validity could be a potential problem. We know that STEP-UP 

interns are receiving support services and are working at their job placements. We were not able 

to determine other interventions that students in our comparison groups may have been 

receiving. In addition, the propensity score model did not include prior academic achievement. 

Propensity score matching. The goal of our matching strategy was to select comparative 

groups that are as similar as possible to STEP-UP interns. To reduce selection bias and enhance 

internal validity of the design, propensity score matching was used to improve group equivalence 

or comparability (Song & Herman, 2009). 

Covariates. The first step in propensity score matching is to determine the best covariates 

to generate propensity scores (Fan & Nowell, 2011). Literature on propensity score matching 

procedures recommends that corresponding variables include preexisting characteristics that are 

strongly related to the outcome variable(s) (Song & Herman, 2009; Stuart & Rubin, 2007). 

Relevant variables related to school performance outcomes include pre-assessment measures, 

demographics, socioeconomic status, family background, and geographical location (Barth, Guo, 

& McCrae, 2008; Fan & Nowell, 2011). 

The evaluation team worked closely with AchieveMpls to identify key covariates that would best 

represent characteristics of the program’s students. Additionally, the variables were thought to 

influence academic outcomes. The covariates and rationales for including them were: 

 Socioeconomic status (Free or reduced-price lunch status). An indicator of living below 

federal poverty guidelines for school children is receiving free or reduced-price lunch 

through the National School Lunch Program. Eligibility is based on household size and 

family income. Interns were compared to students with similar free or reduced lunch 

status. 

 Ethnicity. STEP-UP interns come from diverse backgrounds, including African 

American, ethnic Africans, Asian, Latino, Native American and white. Matching 

techniques equated youth with similar ethnicity backgrounds. 

Center for Applied Research and Educational Improvement, University of Minnesota 

26 
 

 

 

 

 

 

 

 

 

 

	 Primary language (English language learner status). Similar to ethnicity, primary 

language reflects cultural background and the language most often used in homes. 

English is a secondary language for some program interns. Since school performance 

measures were completed in English, students were matched in both groups on primary 

language. 

	 Gender. To accommodate any differences between groups on gender, this variable was 

included in the matching procedures. 

	 Grade Level. Although interns vary somewhat on age across grade level, this variable 

prevented the matching of a 14-year-old STEP-UP intern with an 18-year-old non-STEPUP MPS student. 

	 Special education status. Special education status may influence academic progress and, 

therefore, was included in the matching procedure. We wanted to know if interns and 

potential MPS student matches had active Individualized Educational Plans (IEPs) or 504 

plans and were receiving specialized instruction and related services sometimes in 

restrictive environments outside the regular classroom. Special education status has 

implications for the AchieveMpls training and internships as well school performance. 

Propensity score matching procedures. AchieveMpls and the evaluation team worked with 

Minneapolis Public Schools to obtain (matched) samples of students and their covariate and 

outcome data. Data for each student was associated with a unique identifier (MARSS) number 

used by the MPS database to track all MPS students. Measures were taken to ensure there was no 

overlap between the two groups. Staff of the Minneapolis Public Schools Department of 

Research, Evaluation, Assessment and Accountability conducted logit score matching with a 

nearest-neighbor 1-to-1 ratio with replacement method. A separate model was run for each group 

by grade-level: Grade 10 Discover, Grade 11 Discover, Grade 10 Achieve, Grade 11 Achieve 

and Grade 12 Achieve. The same approach to matching was also used for the SEL study 

conducted by MPS and provided to CAREI. Variables used for the match were socio-economic 

status (using free or reduced-price lunch eligibility), ethnicity, primary language, grade level, 

gender and special education status. In four of the five models, baseline equivalence was 

obtained for each of the covariates for students and their matches after matching. One of the 

models (Model 3) did not reach complete equivalence on the variables of gender and one of the 

home languages. For details about the distribution of propensity scores in the treatment and 

comparison groups, the proportion of cases matched, and the standardized mean differences in 

the baseline characteristics between treatment and comparison groups, please see Appendix G. 

The models did not include variables representing prior academic achievement, and therefore we 

did not know if there was baseline equivalence between interns and matched students on the 

outcome variables in the year prior to the program. Therefore, we refit our models to control for 

prior achievement in the assessment of outcomes in the school year following the program. The 

results did not change when controlling for prior achievement. 

Center for Applied Research and Educational Improvement, University of Minnesota 

27 
 

 

 

 

 

 

 

 

 

 

 

 

 

Second design: A quasi-experimental one-group pretest-posttest within participants 

design. To address the confirmatory questions, “Do the Achieve participants achieve better 

school outcomes pre- to post-program?” and “Do the Discover students achieve better school 

outcomes pre- to post-program?” a quasi-experimental one-group pretest-posttest within 

participants design was used (Shadish, Cook & Campbell, 2002). The same design was used to 

address the exploratory questions, “Do the Achieve participants achieve better employability 

outcomes pre- to post-program?” and “Do the Discover participants achieve better employability 

outcomes pre- to post-program?” 

Rationale. With a quasi-experimental one-group pretest-posttest within participants 

design, AchieveMpls sought preliminary program outcome evidence on pre- to post-program 

specific gains on school and employability outcomes for MPS STEP-UP interns. 

Design. Approximately 330 (53% of Achieve interns who completed an internship) STEPUP Achieve interns and 362 (55% of Discover interns who completed an internship) STEP-UP 

Discover interns were compared with themselves pre- to post-program on school outcomes. In 

this design, pre-program data stand in for the comparison group or control mechanism. School 

outcomes are data collected by MPS. Pre-program were the data collected through the school 

year completed before the 2016 summer internships, during school year 2015-16 (SY16). Postprogram were the data collected through the following school year, 2016-17 (SY17). The 

students in the analysis are students for whom we had data from both school years.20 School 

outcomes included improved academic outcomes—GPA; attendance; behavior (referrals, 

removals, suspensions); graduation rates; and on-track to graduate. 

Employability outcomes were data collected by the STEP-UP program. Pre-program data were 

collected before training and internship placement. Post-program data were collected during the 

exit phase of the summer internships (from mid-August through September) and included 

attitude, occupational or professional knowledge, future orientation, 21st Century skills such as 

teamwork and communication, and professional networks. 320 Achieve and 360 Discover 

interns were compared with themselves pre- to post-program on employability outcomes. These 

were the interns who completed an internship and who completed both the pre- and post-program 

survey. 

20 We have school outcome data only for Minneapolis Public School (MPS) students. If students did not attend MPS 

or attended only in SY15 or SY16 we could not include them in the analysis. The number of students for whom we 

had both years of school outcome data available was lower than anticipated in the SEP, approved in 2014. The SEP 

expected the comparison of 750 STEP-UP Achieve interns and 650 STEP-UP Discover interns. 

Center for Applied Research and Educational Improvement, University of Minnesota 

28 
 

 

 

 

 

 

 

 

 

 

Threats to internal validity. There were few or no threats to internal validity from 

maturation or history since this was within subjects, although history effects may have occurred 

if some of the participants received concurrent and subsequent school-year additional outside 

interventions. The major threats to internal validity for the one group pretest-posttest within 

participants design (both school outcomes and employability) were instrumentation, testing 

effects, missing data, and attrition (Shadish, Cook & Campbell, 2002). As with the quasiexperimental between subjects matched comparison design, external validity here is minimal. 

Evidence. The program evaluation of STEP-UP was designed to provide a moderate 

level of evidence. The quasi-experimental propensity score matching design provided a rigorous 

approach and allowed the study to consider evidence of causality. As mentioned earlier, 

randomization of students to program or control groups was not possible for this study. With the 

second design, the intent was to provide preliminary program outcome evidence on pre- to postprogram specific gains on school outcomes for STEP-UP interns. 

Prior to this evaluation, the evidence for STEP-UP’s efficacy was at a preliminary level. One 

study conducted by University of Minnesota researchers used propensity score matching to 

examine academic outcomes of STEP-UP students (Maruyama, VanBoekel and Cobie, 2013). 

They reported that the STEP-UP program had a marginally significant impact on standardized 

test scores in certain sub-groups but did not have an impact on behavior or attendance. This 

study used only MPS students, which account for about 60 percent of the STEP-UP students. 

The study also did not look at more distal outcomes such as graduation rates, postsecondary 

outcomes, or workforce outcomes. 

Additional Exploratory Questions: Differential Experiences. We addressed the 

exploratory questions about different experiences (more than one summer as an intern, longer 

internships) by analyzing the relationship between students with different experiences in 

relationship to school and employability outcomes. Because Discover interns worked a 

standardized number of hours per week and weeks per summer, we looked at length of internship 

differences (hours and weeks) only for Achieve interns. 

Participant Flow Description 

Sample size. In the summer of 2016, 1,272 youth completed a STEP-UP internship, 617 

Achieve and 655 Discover. 

Of the 1,272, 664 were students in Minneapolis Public Schools in 2016-17 (the summer 

following the internship). In order to have adequate data and meaningful analyses, we examined 

only those who were enrolled at least 75% of the school years prior to and following the 

Center for Applied Research and Educational Improvement, University of Minnesota 

29 
 

 

 

 

 

internship. This criterion for inclusion yielded 270 Achieve students and 227 Discover students 

in the analysis of the treatment group. 

Using the same criterion for inclusion, enrolled at least 75% of the school years prior to and 

following the summer of 2016, the match yielded 186 matches for the Achieve participants and 

153 matches for the Discover participants. 

Employability analyses included 696 interns—those for whom we had pre-program and postprogram data. Nearly all interns completed the pre-program survey, but a smaller number of 

interns completed the post-program survey. Of those for who we have pre- and post-program 

data, 333 were Achieve interns and 363 were Discover interns. 

In the exploratory analysis comparing outcomes for interns who worked different numbers of 

weeks and hours per week, we had data on 154 Achieve interns. In the exploratory analysis 

comparing outcomes for interns who were in the program more than one summer, we had data 

for 420 interns. 

Demographics of the sample. Table 1 presents the demographics for all students who 

completed a STEP-UP internship in 2016, in 2016, as well as the subset of interns included in the 

study and their matched students. The final sample for the study included interns and matches 

who were enrolled in Minneapolis Public Schools at least 75% of the school year in 2015-16 and 

2016-17. While there are some differences in the demographic data for those enrolled 75% of 

both years, the patterns are the same in the reduced sample as in the larger group for whom we 

have data. The patterns are similar to all Achieve and Discover interns who completed the 

program in 2016. 

In terms of gender, there were more females than males. The largest racial group was African 

Americans. Around 4/5 of the sample was eligible for free and reduced lunch. A very small 

percentage of the sample was homeless or highly mobile. In the Achieve sample, about 20% 

were special education students; in the Discover sample there were about 14% special education 

students. The home language of the majority of Discover students and matches (60%) was 

English; for Achieve that was the case for about 45%. The next most commonly spoken home 

language was Somali. 

Center for Applied Research and Educational Improvement, University of Minnesota 

30 
 

 

 

 

 

 

 

 

 

 

 

Table 1 

Demographics for Interns and Matched Students 

All 

Interns^ 

All 

Achieve 

Interns^ 

Achieve 

MPS All 

Data* 

Achieve 

MPS 

Sample** 

Achieve 

Matched 

All 

Data# 

Achieve 

Matched 

Sample## 

All 

Discover 

Interns^ 

Discover 

MPS All 

Data* 

Discover 

MPS 

Sample** 

Discover 

Matched 

All Data# 

Discover 

Matched 

Sample## 

GENDER 

Male 

43% 

(552) 

42% 

(260) 

47% (127) 45% 

(292) 

48% (108) 

Female 

57% 

(720) 

58% 

(357) 

53% (143) 55% 

(363) 

52% (119) 

RACE/ETHNICITY African 

American 

62% 

(795) 

60% 

(372) 

62% 

(174) 

60% (163) 55% 

(144) 

51% (95) 65% 

(426) 

60% 

(146) 

60% (135) 55% 

(126) 

50% (77) 

Native 

American 

6% (73) 4% (27) 2% (6) 2% (6) 5% (13) 3% (6) 7% (46) 4% (10) 4% (10) 6% (14) 7% (10) 

Asian 11% 

(145) 

13% (81) 15% 

(42) 

15% (41) 17% (44) 17% (32) 10% (64) 15% (37) 16% (36) 12% (28) 14% (21) 

Hispanic 9% (110) 10% (60) 11% 

(31) 

12% (31) 13% (35) 17% (31) 8% (50) 11% (26) 10% (23) 16% (36) 20% (30) 

Caucasian 6% (78) 8% (47) 10% 

(29) 

11% (29) 10% (27) 12% (22) 5% (31) 11% (26) 10% (23) 11% (25) 10% (15) 

Missing or 

Other 

6% (71) 5% (30) 6% (38) 

FRL Yes 81% 

(1,082) 

78% 

(482) 

80% 

(226) 

79% (214) 80% (21) 76% 

(142) 

83% 

(546) 

80% 

(196) 

79% (180) 82% 

(187) 

80% (123) 

No 9% 

(114) 

9% 

(57) 

20% 

(56) 

21% (56) 20% (53) 24% (44) 9% 

(57) 

20% (49) 21% (47) 18% (42) 20% (30) 

n/a or 

missing 

10% 

(129) 

13% 

(77) 

8% 

(52) 

HOMELESS Yes 3% (42) 2% (15) .4% (1) .4% (1) .4% (1) .5% (1) 4% (27) 0 0 0 0 

No 97% 

(1,230) 

98% 

(602) 

99.6% 

(281) 

99.6% 

(269) 

99.6% 

(262) 

99.5% 

(185) 

96% 

(628) 

100% 

(245) 

100% 

(227) 

100% 

(229) 

100% 

(153) 

SPECIAL ED Yes 14% 

(172) 

15% (90) 20% 

(56) 

20% (54) 19% (49) 20% (38) 12.5% 

(82) 

13.5% 

(33) 

14% (31) 13% (33) 14% (22) 

No 86% 

(1,100) 

85% 

(527) 

80% 

(226) 

80% (216) 81% 

(214) 

80% 

(148) 

87.5% 

(573) 

86.5% 

(212) 

86% (196) 87% 

(196) 

86% (131) 

Center for Applied Research and Educational Improvement, University of Minnesota 31 
 

 

 

 

 

 

 

All All Achieve Achieve Achieve Achieve All Discover Discover Discover Discover 

Interns^ Achieve MPS All MPS Matched Matched Discover MPS All MPS Matched Matched 

Interns^ Data* Sample** All Sample## Interns^ Data* Sample** All Data# Sample## 

Data# 

HOME English 44% 44% (119) 46% 46% (85) 61% 60% (135) 63% 60% (92) 

LANG (125) (122) (149) (144) 

Hmong 13% 

(36) 

13% (35) 11% (29) 12% (23) 11% (27) 12% (26) 9% (21) 9% (14) 

Somali 24% 

(67) 

23% (63) 24% (63) 22% (40) 16% (38) 16% (37) 15% (34) 15% (23) 

Spanish 10% 

(29) 

11% (29) 10% (26) 12% (22) 9% (22) 9% (20) 10% (22) 12% (18) 

Other/ 

Unknown 

9% (25) 9% (24) 9% (23) 9% (16) 4% (9) 4% (9) 4% (8) 4% (6) 

^Data drawn from STEP-UP database 

*All interns from 2016 for whom MPS supplied data 

**Interns who were enrolled at least 75% of both SY16 and SY17 for whom we had data on school outcomes 

#All matched students for whom MPS supplied data 

##Matched students who were enrolled at least 75% of both SY16 and SY17 

Center for Applied Research and Educational Improvement, University of Minnesota 32 
 

 

 

 

 

 

 

 

 

Changes to the sample in SEP. The SEP anticipated approximately 750 Achieve and 

650 Discover interns for the study, with approximately the same number of matched students. 

Because not all program participants were enrolled in the Minneapolis Public Schools, we were 

able to get school outcome data on a smaller number of students (numbers indicated above). 

Sample recruitment and retention. For the treatment group, 3,447 students applied to 

the program and were eligible for training. Of those, 2,036 students completed work readiness 

training. Of those, 1,389 were placed in internships and 1,272 completed internships. From the 

students who completed internships, we examined students who were enrolled in Minneapolis 

Public Schools at least 75% of the school years 2015-16 (prior to the internship) and 2016-17 

(following the internship). See above for final sample size. 

Overall and differential attrition. The initial data set from Minneapolis Public Schools 

yielded 664 students in the treatment group and 492 matches. After examining the dataset, we 

recognized that both in the treatment group and the matched group there were students who were 

enrolled both years or who were enrolled only for a limited number of days in one or both years. 

All variables were examined descriptively for differences between the sample when all students 

were included versus when only students enrolled for at least 75% of days in both SY16 and 

SY17 were included. The results indicated no difference between the two samples. We therefore 

made the decision to include in our sample only students who had been enrolled for 75% of both 

years. This criterion for inclusion yielded 497 students in the treatment group (a reduction of 

25%) and 339 students in the matched group (a reduction of 31%). For Achieve, the number of 

treatment students went from 410 to 270 (a reduction of 34%) and matched students went from 

263 to 186 (a reduction of 29%). For Discover, the number of treatment students went from 254 

to 227 (a reduction of 11%) and matched students went from 229 to 153 (a reduction of 33%). 

We also ran each analysis on outcomes (GPA, attendance, behavior, test scores) with the full 

sample and with the restricted sample to see if there were differences between the full group and 

those enrolled 75% of the year using t-tests and Chi-square tests. Results regarding program 

effects were the same. 

In looking at differential experiences (more hours/weeks worked, more than one year in the 

program), we had data for 154 Achieve interns on hours and weeks.21 There were data on fewer 

students on ACT and MCA tests because not all interns took the exams; the tests are 

administered only to single grade level cohorts each year. For the ACT analysis there were data 

on 63 interns. For Minnesota Comprehensive Assessment (MCA) Match there were data on 18 

interns. 

21 To be included in the analyses on hours and weeks worked, interns needed to have been MPS students and to have 

completed end-of-summer surveys. 

Center for Applied Research and Educational Improvement, University of Minnesota 

33 
 

 

 

 

 

 

 

 

 

 

 

 

For conducting the comparisons of interns with more than one year in the program to ones who 

participated in the summer of 2016 only, we had data for 420 interns, 234 Achieve (137 with one 

year of experience, 97 with more than one) and 186 Discover (153 with one year of experience, 

33 with more than one). 

For the analyses of post-secondary enrollment, we had data only for the cohorts who participated 

in the program in the summers of 2014 and 2015. While most of this report presents an analysis 

of the 2016 cohort, we are not able to report on their post-secondary enrollment at this time. For 

the 2014 cohort, we had data on 450 students, 128 participants and 322 comparison students.22 

We examined students who were in 11th th and 12 grades the summer following the internship. 

For the 2015 cohort, we had data on 117 participants and 94 comparison students. We examined 

students who were in 12th grade the summer following the internship. 

Non-response bias and missing data. Descriptive statistics were generated and 

compared for the full data set of treatment and matched students and for those enrolled in 

Minneapolis Public Schools at least 75% of the year in both 2015-16 (SY16) and 2016-17 

(SY17). An examination of the data indicated no differences between the two samples, and the 

results are presented for students enrolled 75% or more of both years. 

In cases where data was missing or we found data entry mistakes for a particular outcome (either 

school or employability), we removed those cases from the analysis of that outcome only. 

Results of assessment and adjustment for potential biases due to non-consent and 

data non-response. There were no issues of non-consent. In cases with non-response, we 

eliminated cases on an outcome-by-outcome basis. 

Statistical procedures used to adjust for missing data. In cases where there was 

missing outcome data we used list-wise deletion for each outcome separately. 

Data Collection & Measurement: Impact 

First design (between-groups): measures/instruments. Outcome measures used for the 

first design were the following school outcomes: GPA (grade point average), ACT test scores23, 

Minnesota Comprehensive Assessment (MCA) scores24, attendance, behavior (referrals, 

22 The propensity score match in 2014 drew 3 matches for every participant. 

23 The ACT is a college admissions test, used to assess academic readiness for college. It is administered to eleventh 

graders in Minneapolis Public Schools. 

24 The Minnesota Comprehensive Assessments (MCA) are the state tests that help districts measure student progress 

toward Minnesota’s academic standards and also meet federal and state legislative requirements. Reading is given in 

tenth grade and math in eleventh. 

Center for Applied Research and Educational Improvement, University of Minnesota 

34 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

removals, and suspensions25), graduation rates, and being on-target for graduation. The postsecondary outcome measure used was whether students had ever enrolled in a post-secondary 

program (certificate or degree-bearing). 

The social-emotional outcome measure used was academic persistence, selected because 

it is a measure for which MPS collects data from all students every year. 26 Data about socialemotional learning was drawn from an academic persistence survey, developed by REAA.27 

First design (between-groups): data collection. STEP-UP provided demographic data 

on interns from their database to the evaluators. The Department of Research, Evaluation, 

Assessment and Accountability (REAA) of the Minneapolis Public Schools provided deidentified school outcome data to the evaluators. These data are collected regularly by the 

department for all Minneapolis Public School students and were provided to CAREI evaluators 

through a data-sharing agreement established with Greater Twin Cities United Way for this 

evaluation. Data for 2016 participants and matched students were gathered throughout the 2015-

16 and 2016-17 school years and were shared through a secured file sharing system. 

STEP-UP program staff trained to use the STEP-UP database, in consultation with CAREI 

evaluators, provided a list of internship completers to REAA following the summer of 2016. 

25 A suspension is when a student is sent home and misses more than one day of school. A removal is when a student 

is removed from the classroom for a specified period of time or is sent home for a day or less. A referral is any other 

type of administrative response that is not either a suspension or a removal. 

26 Academic persistence can best be defined as ongoing task engagement even in the face of difficulties and failures 

(Guan, Kiang, McBride & Bruene, 2006). Persistence in academics is represented by continued task engagement 

even when confronted with failures (Guan, Xiang, McBride & Bruene, 2006). The study of academic persistence 

has a rich history where academic persistence has been shown to be related to GPA, adoption of mastery goals, 

school completion and retention in higher education (Bordes-Edgar, Arredondo, Robinson Kurpius & Rund, 2011; 

Elliot, McGregor & Gable, 1999). 

Minneapolis Public Schools defines persistence as refusal to give up in spite of difficulty. Persistence can also be 

thought of as a measure of a student’s desire to beat-the-odds. Persistence pushes students to meet long-term goals, 

stay motivated. 

27 This tool includes 10 items measuring students’ refusal to give up in spite of difficulty. Items are answered on a 

five-point scale, ranging from not like me (1) to always like me (5). Sample items include: “If I am stuck on 

something, I will not stop until I find a solution,” “I do high-quality work in school all year long,” and “I like to do 

things that are challenging.” Factor analyses found evidence that all ten persistence items loaded on to a single 

factor. The scale had good internal consistency reliability (as measured by coefficient alpha; α2015 = .93, α2016 = .94, 

and α2017 = .94) indicating that students were responding consistently to these items. 

Center for Applied Research and Educational Improvement, University of Minnesota 

35 
 

 

 

 

 

 

 

 

 

 

 

REAA drew the Minneapolis Public School students from this list both to generate the match and 

to furnish school-related data for the school years prior to and following the internship summer. 

School outcome and demographic data on both the students who completed their internship and 

matches were provided to CAREI by REAA through a secure file-sharing system and were deidentified. These data were stored in a password-protected system at the University of MN. For 

participants, data from the STEP-UP data base were merged with files on demographics and 

school outcomes provided by MPS. Separate files were created for interns and matched students 

who were enrolled in MPS at least 75% of both SY16 and SY17. 

Data on post-secondary enrollment were provided by SLEDS (Minnesota Statewide Longitudinal 

Educational Data System).28 CAREI, in collaboration with Minneapolis Public Schools and 

AchieveMpls, submitted an application for data access. De-identified data for all Minneapolis 

Public School students were provided by SLEDS to CAREI. SLEDS provided a key to the data 

to MPS so they could re-identify the files of students in the study. The researcher at CAREI then 

merged the post-secondary outcome data with other data on interns and matched students. 

In order to conduct the analyses on behavior, several new variables were created. 

	 Total number of removals was calculated by adding out-of-school removals and in-school 

removals 

	 For referrals, a binary variable was created (Yes/No) if a student had any referrals 

(regardless of number). The same was done for removals and suspensions. The new 

variables were used to perform Chi-square analyses. 

In order to conduct analyses on differential experiences, several new variables were created. 

 For number of weeks worked, students were assigned a grouping variable based on the 

number they indicated on their end-of-summer survey 

 For number of hours worked per week, students were assigned a grouping variable based 

on the range they indicated on their end-of-summer survey 

 For multiple year participation, a new variable indicated whether students completed an 

internship in STEP-UP only in 2016 or if they did so in 2016 PLUS either 2014 or 2015. 

28 SLEDS matches student data from pre-kindergarten through completion of postsecondary education and into the 

workforce to identify pathways for individuals in achieving successful outcomes in education and work; to inform 

decisions to support and improve education and workforce policy and practice, and to assist in creating a more 

seamless education and workforce system for all Minnesotans. The Minnesota P-20 Education Partnership governs 

the SLEDS system. The project is managed jointly by the Minnesota Office of Higher Education (OHE) (opens new 

window), Minnesota Departments of Education (MDE), and Employment and Economic Development (DEED). 

Center for Applied Research and Educational Improvement, University of Minnesota 

36 
 

 

 

 

 

The academic persistence survey was administered to MPS students for three consecutive school 

years: 2014-2015 (10,896 students), 2015-2016 (11,458 students), and 2016-2017 (11,002 

students). 

Second design (within-participants) school outcomes: measures/instruments. 

Achieve and Discover interns were compared with themselves pre- to post-program on these 

school outcomes: GPA, attendance, and behavior (referrals, removals, and suspensions). Because 

students do not take the same standardized test every year, test scores were not an outcome 

measure used in the second design. 

Second design (within-participants) school outcomes: data collection. Pre-program 

school measures were the data collected through the school year completed before the 

internships in the summer of 2016, that is SY15 (2015-16). Conceptually, pre-program data 

stood in for a comparison group or control group. Post-program were the data collected through 

the following school year, that is SY17 (2016-17). As indicated above, data were provided by 

Minneapolis Public Schools Department of Research, Evaluation, Assessment and 

Accountability from their student database to the evaluation team. The same procedures 

described above were used to insure data security and for data storage. 

Second design (within-participants) employability outcomes: measures/instruments. 

Employability outcomes were measured using a survey in which students rated themselves on a 

five-point agree/disagree scale prior to training and again at the end of the internship on 25 

items. See Appendix H for a copy of the instrument. 

In the original SEP, the plan was to develop an instrument in-house to measure 

employability outcomes. CAREI developed the survey, drawing on literature, tested 

instruments, and STEP-UP’s previously used survey. It was used in the summers of 2014 

and 2015. Students rated themselves prior to training and again at the end of the 

internship on items related to attitude, work related skills, future orientation, occupational 

knowledge, 21st Century skills such as conflict resolution, teamwork, and professional

networks. Because of its use of a four-point response scale, the nature of the questions, 

the students’ tendency to rate themselves highly, and the difficulty to establish the 

reliability and validity of the instrument, it was difficult to use it to measure change over 

time. 

Center for Applied Research and Educational Improvement, University of Minnesota 

37 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

A new student survey on employability skills, based on one developed by MHA Labs, 

29 30 31 was used in the summers of 2016 and 2017. MHA Labs’ statistical analysis focuses 

on how well skill survey items predict a youth's Hirability Status. Hirability Status is the 

average of 3 items: 

 If I had a job opening, I would hire this person 

 I would recommend this person to a colleague, for a similar position 

 I would seek out this person to be on my next project. 

Because the items in the MHA Labs instrument were intended to be used by employers to rate 

workers, some of the items needed to be adapted for a youth survey. Some of the items for the 

STEP-UP survey were taken directly from the MHA tool and others were reworded. Some items 

were carried over from a tool developed by CAREI and Achieve staff and used in 2014 and 

2015. Validation studies were not conducted on the instrument. Many items were drawn from the 

MHA instrument on which validation studies were conducted. 

Second Design (Within-participants) Employability Outcomes: Data Collection. For 

employability outcomes, the pre-internship data were collected using a paper and pencil survey 

distributed by trainers on the first day of training in spring 2016. Results were compiled 

electronically using InspiroScan, compiled in Excel, and analyzed in SPSS. The same questions 

were included in a longer end of summer instrument administered online through 

29 The staff’s intention was to have supervisors focus on the skills with strong correlations to employability, to use 

the framework of hirability skills to give feedback to interns, and then to measure growth on the pre- and postinternship survey on the same skills. Supervisor orientations devoted time to the hirability framework from MHA 

Labs and weekly emails addressed its use as well. 

30 MHA (Means and Measures of Human Achievement) Labs is a public/private research and development nonprofit 

that designs products and services for 21st century skills development with a focus on youth. Since 2011, the 

organization has been developing and refining a work readiness/employability assessment tool. 

31 By calculating the R Squared values for each survey, MHA Labs was able to identify the fraction of the Hirability 

Status that can be explained by adding together the contributions of (in this case) all the skill-related survey items 

from a given survey. R Squared was obtained through multiple linear regression. MHA Labs additionally adjusted 

for the length of the survey. Skills survey research was conducted over a 7 years period beginning with a 4,000item 

skill research database. MHA Labs collaborated with research analysts, subject matter experts and practitioners to 

isolate and test 102 different items on 6 unique employer survey designs with a total of 8950 summer job employer 

surveys. The survey is a performance review of an individual youth employee. 

The 2015 version tested 40 items with an N of 2,353 and an R Squared Adjusted value of 0.73. With the goal to 

further shorten the tool, only 12 items with the highest R Squared value comprised the 2016 survey. This version 

with only 12 items had an N of 3496 and an R Squared Adjusted value of 0.83. 

Center for Applied Research and Educational Improvement, University of Minnesota 

38 
 

 

 

 

 

 

 

 

SurveyMonkey. Each intern received a unique link to the survey in order to identify results. Staff 

at AchieveMpls and the City of Minneapolis emailed the surveys to all interns in August prior to 

the end of the internship and to supervisors to provide time at work for the interns to complete 

the surveys. Staff at Achieve and the City followed up by text message throughout September to 

increase the response rate. Results were compiled in Excel through SurveyMonkey and analyzed 

in SPSS. 

Pre-internship data were available from 1,084 interns who completed internships. Even with 

these efforts, post-internship data were available from 828 interns, a response rate of 65%. The 

analysis examined interns for whom we had both a pre- and a post survey (approximately 330 

Achieve interns and 360 Discover interns; there was variation across items). The data included 

in this analysis included both MPS students and student enrolled in other school districts. 

Analysis 

First design: quasi-experimental between-groups design with matched comparison 

groups based on propensity score matching. The unit of analysis in all analyses was the 

individual student. The independent variable for the analyses was STEP-UP internship 

participation versus non-participation. This was a fixed effect. The dependent variables were 

school outcomes: GPA, attendance, ACT test scores, MCA scores, behavior records (referrals, 

removals, and suspensions), being on track to graduate, and graduation rates. The dependent 

variable for post-secondary enrollment was whether students had ever enrolled in a postsecondary program (certificate or degree-bearing). 

First, we calculated descriptive statistics for the STEP-UP interns who were enrolled in MPS in 

2015-16, completed internships in Summer 2016, and were again enrolled in MPS in 2016-17 as 

well as the comparison groups. 

We used independent t tests, Mann-Whitney U, chi-square, ANCOVA, and logistic regression 

analyses to determine if there were significant differences on school outcome measures and postsecondary enrollment between STEP-UP interns and the comparison non-STEP-UP students. 

Where possible we controlled for SY16 outcome measures when looking at the differences 

between participants and non-participants on SY17 outcome measures. Tests were selected on 

the basis of the scale of the outcome variable or on modeling assumptions. 

For all of the analyses, Discover and Achieve students were examined as separate groups. In 

order to do so, the matched students were selected by age. 

With some outcome variables, there were limited numbers of cases to analyze. Standardized tests 

Center for Applied Research and Educational Improvement, University of Minnesota 

39 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

were only given at a single grade level. With regard to behavior outcomes (referrals, removals, 

suspensions), many students were never referred, removed, or suspended. Graduation rates could 

only be examined for students who were in 11th grade in SY16. In all analyses we present 

descriptive statistics, and where numbers of cases in the analysis allow32, we also report on 

statistically significant differences and effect sizes. 

The postsecondary analysis was conducted only for the 2014 and 2015 cohorts since there were 

not postsecondary data available for the 2016 cohort. 

SEL study analysis. Independent-samples t tests were conducted to determine how 

STEP-UP students’ level of academic persistence compared to that of a matched comparison 

sample. The matched comparison samples were created using propensity score matching (PSM). 

Across all three cohorts, participants were matched within grade and on several characteristics: 

race/ethnicity, gender, English language services, home language, special education status, 

homeless/highly mobile status, free/reduced price lunch eligibility, and school. In the 2015 and 

2016 samples, participants were also matched within STEP-UP Achieve and Discover groups. 

Each PSM matched with replacement and utilized the nearest neighbor matching method. 

Independent-samples t tests were conducted within each of the three cohorts, with participation 

in STEP-UP serving as the independent variable and academic persistence serving as the 

dependent variable. Many participants were missing persistence data: 1,916 in the 2014 cohort 

(426 STEP UP, 1,490 non); 347 in the 2015 cohort (183 STEP UP, 164 non); and 426 in the 

2016 cohort (187 STEP UP, 239 non). T tests were conducted within group (i.e., 

Achieve/Discover) for the 2015 and 2016 cohorts – Achieve/Discover group data were not 

available for the 2014- cohort – for a total of five t tests. 

Second design: quasi-experimental one-group pretest-posttest within participants 

design. The unit of analysis in all analyses was the individual intern. The independent variable 

for the analyses was time of data collection, namely, pre- or post-program. This is a fixed effect. 

The dependent variables were school and employability outcomes. 

School outcomes. First, we calculated descriptive statistics for the STEP-UP interns on 

the school measures pre-program and post-program who were enrolled for at least 75% of the 

year in MPS in 2015-16, completed internships in Summer 2016, and were again enrolled in 

MPS for at least 75% of the year in 2016-17. There were 497 students who met these criteria, 

270 Achieve interns and 227 Discover interns. 

32 In analyses with thirty or fewer cases, we have low power to detect anything but a large effect size and we are 

expecting small to moderate effects at most based on the previous year’s evaluation. 

Center for Applied Research and Educational Improvement, University of Minnesota 

40 
 

 

 

 

 

 

 

 

 

 

 

 

 

These statistics included means and standard deviations. We used paired sample t tests (with 

appropriate effect sizes), McNemar’s test, and chi-square tests to determine if there were 

significant differences on school outcome measures before and after participation in the program. 

We analyzed the pre- versus post-program differences of the STEP-UP Achieve interns and the 

STEP-UP Discover interns for each dependent variable for school measures. 

With some outcome variables, there were limited numbers of cases to analyze. With regard to 

behavior outcomes (referrals, removals, suspensions), many students were never referred, 

removed, or suspended. We were unable to examine ACT or MCA test scores in the withinsubject analyses because students took these exams once, not year after year. 

It is important to note certain limitations in the conclusions that can be drawn from the withinparticipants analyses. It is likely there were factors that influenced changes in student outcome 

measures like development/growth and transitions from one school to another that cannot be 

accounted for, so our results of the analyses should be interpreted with caution. It is critical that 

when we find differences from one year to the next, whether positive or negative, that we do not 

assign causation for these changes to the program. In all analyses we present descriptive 

statistics, and where numbers of cases in the analysis allow33, we also report on statistically 

significant differences and effect sizes. 

Employability outcomes. Because survey responses to the employability questions use a 

Likert scale, we used the Wilcoxon signed-rank test to analyze the pre- versus post-program 

differences for interns with both surveys for those items. The Wilcoxon signed-rank test is the 

nonparametric test equivalent to the dependent t test. It is used to compare two sets of scores that 

come from the same participants when the data are not normally distributed, and the dependent 

variable is measured at the ordinal level. The data met the assumption for a symmetrically 

shaped distribution of differences for the pre- and post- scores for all interns for all items, as 

assessed by histograms. 

Wilcoxon signed-rank tests were run on the employability data for 2016 interns who completed 

an internship and for whom we had pre-training and post-internship surveys. Discover and 

Achieve students were examined as separate groups. 

Differential experiences analyses. To examine whether there was a relationship between 

the number of weeks worked or the number of hours per week worked on school outcomes and 

33 In analyses with a small number of cases, we have low power to detect anything but a large effect size. 

Center for Applied Research and Educational Improvement, University of Minnesota 

41 
 

 

 

 

 

 

employability outcomes, we calculated descriptive statistics for the Achieve interns in 2016 with 

at least 75% enrollment in Minneapolis Public Schools in SY16 and SY17 for whom we also had 

2016 end-of-summer survey data (the source of information on weeks and hours worked). There 

were 154 interns who met these criteria. There were a number of school outcomes that lacked 

adequate data for an analysis—on track to graduate, MCA reading scores, referrals, removals, 

and suspensions. For the other school outcomes (GPA, attendance, MCA math, and ACT) we 

conducted ANOVA tests, and where assumptions were violated we also ran the nonparametric 

Kruskal Wallis test. For employability outcomes we used the Kruskal-Wallis H test. 

To examine whether there was a relationship between multiple-year participation and school and 

employability outcomes, we calculated descriptive statistics for interns in 2016 with at least 75% 

enrollment in MPS in SY16 and SY17. For the 420 interns for whom we had data, we compared 

students who did an internship only in 2016 with those who did an internship in 2016 and in 

2014 and/or 2015. These analyses were run separately for Achieve and Discover students, and 

analyses were conducted using independent samples t tests and the nonparametric Mann Whitney 

tests. Analyses were performed on GPA, attendance, ACT scores, and MCA scores. For 

employability outcomes we used a Chi-square test. 

Intent-to-treat analyses were not performed due to the nature of the designs. Comparison groups 

were formed through propensity score matching. Randomization of students to program or 

control groups was not possible for this study. 

Study Logistics 

The study proceeded as planned. There were no issues or changes to the timeline or budget. 

There were no changes to evaluation personnel at CAREI. The research/evaluation staff position 

at AchieveMpls changed over the period of the grant; five different people held the position. 

Those changes did not affect the conduct of the study. There were no problems with the IRB. 

Center for Applied Research and Educational Improvement, University of Minnesota 

42 
 

 

 

 

Findings: Implementation Evaluation 

Descriptive Participant and Program Statistics/Outputs 

Program delivery timeline. The STEP-UP programming cycle for the summer 

internships program runs from December through August. Beginning in December, 

STEP-UP staff members recruit students to apply to the program. Students apply to the 

program and complete STEP-UP job training throughout the spring. After training, 

students interested in having an internship submit an additional application detailing 

their availability, job skills and work preferences. In May and June, students are placed 

with employers. Students complete their internships in mid-August, although exact start 

and end dates are determined by each employer. 

Training. In 2016, STEP-UP received 3,447 eligible 

applications; 3,447 students were accepted and invited to attend training, 

and 2,036 completed work readiness training (the SEP target was 2,200). 

These numbers are higher than in 2015; total eligible applicants increased 

by about 400 students, and 157 more who completed training. 

Among Discover interns, 420 attended weekly training classes during the summer and earned a 

total of 719 credits. Interns were paid for their time in training. See Table 2 for a comparison 

across years of program targets and actuals. 

Center for Applied Research and Educational Improvement, University of Minnesota 

43 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Table 2 

2014 2015 2016 

Target Actual Target Actual Target Actual 

Eligible Applications 3,754 3,678 4,000 3,062 4,000 3,447 

Accepted to program 3,678 3,582 3,800 3,003 3,100 3,447 

Completed Work-Readiness 

Training 

2,200 2,076 2,125 1,879 2,200 2,036 

Internships started 1,500 1,425 1,400 1,238 1,420 1,389 

Completed internships 1,278 1,222 1,300 1,132 1,310 1,272 

Companies/Organizations 

participating 

268 212 226 

Supervisors 539 505 513 

Specialized Training/Career 

Enrichment Activities (20 

hours or fewer) 

250 576* 470 491 500 424 

Specialized Training/Career 

Enrichment Activities (more 

than 20 hours) 

82 124 85 120 

STEP-UP Targets and Actuals 

*2014 was prior to the distinction between activities of more or less than 20 hours. 

Specialized Training/Career Enrichment Activities. A description of specialized 

training programs and career enrichment activities can be found in Appendix D. In 2016 there 

were 424 completed career enrichment activities of 20 hours or fewer and 120 completed 

activities of more than 20 hours. Some individuals participated in more than one, so the numbers 

reflect “units of service” rather than discrete interns. There were at least 315 different students 

who completed career enrichment activities. Due to a glitch in the record keeping for one of the 

large enrichment programs, we were unable to calculate a precise number; the total number of 

students who participated in at least one activity is probably 75-90 more if we use 2015 records as 

a guideline. Shorter programs included Golden Gopher Day at the University of MN and a 

Financial Education Day. Longer programs included Scrubs Camp (careers in health sciences) 

and Microsoft Excel training. Of the 544 who completed enrichment activities, 247 completed 

Center for Applied Research and Educational Improvement, University of Minnesota 

44 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

specialized career training. Forty-three earned professional credentials (28 Excel, 15 IT Training 

Credential). See Table 2 for a comparison across years of program targets and actuals. 

Internships. We have assembled descriptive statistics about interns, supervisors, and 

placements. 

Interns. Of the 2,036 students who completed work readiness training, 

1,389 (680 Achieve; 709 Discover) were placed in more than 226 companies. 

This number was 31 short of the 2016 target for internships begun but was an 

increase of about 150 students over 2015. Summer internship providers 

included businesses, public agencies, and nonprofits, of which 64 were new 

partners or were returning after a hiatus. Of the 1,389 students placed, 1,272 

completed internships (617—91% Achieve; 655—91% Discover). During the 

summer of 2016, STEP-UP interns earned approximately $2.29 million in 

wages ($1,346,340 through Achieve; $944,886 through Discover).34 See Table 

2 for a comparison across years of program targets and actuals. 

Of the students who completed an internship in 2016, there were more girls than boys. The largest 

of the four training groups was Achieve (34%), followed by HS Discover, MS Discover, and 

Advanced. Most of the interns (90%) were eligible for free or reduced lunch and a very small 

number were homeless or highly mobile.35 There were more African American participants than 

any other racial group. Details on student demographics for all youth who completed an 

internship in 2016 are presented in Figures 2-5. For further details, refer to Appendix I. The 

overall demographics are the same across the three years as shown in Table 3. 

34 Data on wages earned by Achieve interns are an estimate based on data provided by those Achieve interns who 

completed the end-of-summer survey. Because of the way the data were reported (ranges), that we do not have data 

on all the internships, and we are unsure of the reliability of the data we do have, we cannot calculate descriptive 

statistics such as means or medians for total wages or hourly wages. 35 There are two reasons why the number is not 100% of participants. Interns may meet other eligibility criteria such 

as having an IEP or being pregnant or parenting. It is also possible that some of the interns’ families meet the income 

requirement, but have not submitted the required paperwork for “free and reduced lunch” status. 

Center for Applied Research and Educational Improvement, University of Minnesota 

45 
 

 

 

Figure 2. Percentage of interns by gender. Figure 3. Percentage of interns eligible for FRL 

Figure 4. Number and percentage of interns by training group. 

Center for Applied Research and Educational Improvement, University of Minnesota 

46 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Figure 5. Number and percentage of interns by racial/ethnic group. 

Table 3 

Intern Demographics, 2014-2016 

2014: All Interns 

N = 1,242 

2015: All Interns 

N = 1,132 

2016: All Interns 

N = 1,271 

GENDER Male 42% (513) 42% (473) 43% (552) 

Female 58% (709) 58% (659) 57% (720) 

RACE/ETHNICITY African 

American 

79% (847) 62% (700) 62% (795) 

Native American 3% (31) 9% (102) 6% (73) 

Asian 13% (159) 11% (123) 11% (145) 

Hispanic 7% (87) 7% (82) 9% (110) 

Caucasian 7% (81) 5% (58) 6% (78) 

Missing or Other 6% (67) 6% (71) 

FRL Yes 85% (1,039) 89% (1,007) 81% (1,028) 

No 14% (171) 11% (125) 9% (114) 

Not Applicable 

or Missing 

1% (12) 13% (77) 

HOMELESS Yes 1% (12) 2% (24) 3% (42) 

Center for Applied Research and Educational Improvement, University of Minnesota 

47 
 

 

 

 

 

 

 

 

 

 

 

No 97% (1,185) 98% (1,108) 97% (1230) 

Missing Data 2% (24) 

SPECIAL ED Yes 11% (132) 13% (144) 14% (172) 

No 88% (1,070) 86% (970) 86% (1,100) 

Missing Data 2% (20) 2% (18) 

Among Achieve interns who responded to the survey, 71% worked eight weeks or more. Some 

internships were shorter in length. Achieve interns worked 20 to 40 hours per week. Twenty-eight 

percent of interns who responded to the survey worked 16 to 25 hours, 20% worked 26 to 34 

hours per week, and 27% worked 35-40 hours per week. Discover interns worked nine weeks. 

Discover interns were limited to 23 hours a week in 2016 (including three hours of class per 

week); this was a change from previous years. In making the change, the 

program placed over 100 additional youth using the same funding. Prior 

to the summer of 2016, 32% of Achieve participants and 63% of Discover 

participants had never held paying jobs. (More details about intern 

characteristics can be seen in tabular form in Appendix I.) 

Supervisors. There were 513 supervisors to STEP-UP interns. In the 

Discover track, 128 people supervised interns. Of the 128 supervisors, 29 

worked with a single intern and 99 had multiple interns. There were 385 Achieve supervisors. 

Among Achieve supervisors, 298 worked with a single intern and 87 had multiple interns. 

Approximately 93% of Achieve supervisors (358) and 95% of Discover supervisors (122) 

attended an orientation before the summer program began; this was the highest rate of attendance 

in the history of the program. 

Placements. Achieve and Advanced interns worked in placements across 

different types of industries. The industries with the largest concentration of Achieve 

interns were Government/Public Administration, Health Science, and Education/Training. 

Discover interns worked in the nonprofit sector only; over 67% of Discover interns 

worked in Human Services or Outdoor/Natural Resources. The types of organizations in 

which they were placed also varied and included Arts, AV/Technology and 

Communication and Education/Training. (The industries for intern placement are 

enumerated in Appendix J.) 

Fidelity to Program Design 

In all of its major components (training, placement, supervision, and career enrichment activities) 

the STEP-UP program demonstrated fidelity to the plan outlined in the 2013 SEP. The 

requirement for training, number of hours of training, training content, procedures for placing 

Center for Applied Research and Educational Improvement, University of Minnesota 

48 
 

 

 

 

 

 

 

interns, the nature of the work with supervisors, the types and lengths of internships, and the 

offering of a range of career enrichment activities were consistent with that plan.36 

Dosage 

Training. Middle School Discover training is 12 hours. High School Discover training is 

13 hours including a mock interview. Achieve and Advanced trainings are 9 hours including a 

mock interview. Discover interns attended 3 hours a week of training throughout the summer 

internships. 

Internships. Each intern had an internship experience over the course of the summer. The 

analysis included interns who completed an internship. 

Career enrichments/pipelines. Some interns participated in multiple enrichment 

activities, some in one activity, and some not at all. Length of activities varied from two hours to 

over twenty hours. 

Quality 

Given the vast range of settings and experiences of internships, the limited control of STEP-UP 

staff over interns’ experiences, and limitations on the amount and quality of data related to those 

experiences, it was not possible to measure internship quality. We were able to examine the 

guidelines and recommendations that program staff provided to supervisors in orientations, 

handbooks, and emails during the summer, and to infer what the staff consider contributing 

factors to a high-quality internship. We examined those elements of quality for which data were 

available (exposure and satisfaction) and present those findings below. 

Exposure 

Training. Because the STEP-UP staff planned and delivered training and it was consistent 

for all students at each training level, we considered training part of dosage and not exposure. 

Internship exposure: time worked. The internship site controlled the number of hours 

and number of weeks Achieve interns worked; time at internship is therefore considered 

exposure. Interns work an average of eight weeks during the summer, but some internships are 

shorter. The average hours worked per week ranged from fewer than 10 hours to more than 40 

hours. Among Achieve interns, 28% of those who responded to the survey worked 16 to 25 hours, 

36 There were tweaks to the training curriculum made over time, including the creation of a special track for 

Advanced students (see program description, pg. 49. The basic components of the program, including training, 

remained consistent with the plan described in the SEP. 

Center for Applied Research and Educational Improvement, University of Minnesota 

49 
 

 

 

 

 

 

 

 

20% worked 26 to 34 hours per week, and 27% worked 35-40 hours per week. Discover 

internships in 2016 were limited to 20 hours per week plus three hours of classroom experience. 

Pipeline, or career enrichment (referred to in the program logic model as systems building), 

activities were mostly optional. Some interns did not participate in any, and some participated in 

several. Of those who participated, experiences ranged from two hours to over twenty hours. 

Internship exposure: activities. The STEP-UP program staff recommended ways for 

supervisors to structure internships and experiences to provide to interns; the supervisors at the 

sites had control over what actually happened and the nature of the experiences varied as a result 

of the particular industry, job, and supervisor. Recommendations from STEP-UP staff were 

communicated in the orientation, supervisor handbook and weekly emails to supervisors. The 

recommendations were: 

 Providing a job description; creating a work plan (expectations and goals) 

 Implementing a strategy for supporting the intern (giving clear directions, checking in 

regularly and reflecting on the experience together, talking to them about future plans, 

helping them develop professional skills, talking to them about the company and its 

activities, talking about their own education and work history) 

 Establishing a project for the intern to undertake 

 Assigning of a mentor (either the supervisor or another adult) 

 Giving regular feedback to the intern (including a mid-internship check-in and a 

performance evaluation) 

 Providing interns with opportunities to learn about a career in the specific fields through 

activities such as job shadow, attending meetings, informational interviews 

 Encouraging and providing opportunities to the interns to develop their network (e.g., 

lunch with supervisor, attending social gatherings with colleagues, meeting the president 

or CEO, attending meetings) 

 Offering specialized training (e.g., Excel, Outlook) 

For the purposes of analysis, these structures and activities can be grouped into five categories: 

structure and orientation, interpersonal experiences, supervision, opportunities for learning, and 

intern voice. 

In the post-internship survey approximately 457 out of the 617 (74%) Achieve interns who 

completed an internship responded to questions about their internship experiences and their 

supervisors. In the post-internship survey, approximately 379 out of the 657 (58%) Discover 

interns who completed an internship responded to questions about their internships and 

supervisors, including questions about exposure. In the end-of-summer survey approximately 236 

of the 385 (61%) Achieve supervisors and 105 of the 128 (82%) Discover supervisors responded 

to questions about their experiences with interns, including questions about internship exposure. 

Center for Applied Research and Educational Improvement, University of Minnesota 

50 
 

 

 

 

 

 

Highlights of reporting on internship exposure follow. Details can be found in Appendix K. 

Orientation/structure. Discover and Achieve interns felt they were well-orientated to their 

internships. About 3/5 of each group had structured work plans with measurable goals. 

Supervisors also responded to a question on the use of a structured work plan with measurable 

goals. Their responses showed that there was variation across internships in the extent to which 

interns had structured work plans with measurable goals. About two-fifths of Achieve supervisors 

gave strong ratings to this item and another two-fifths gave moderate ratings. There was some 

misalignment here with the interns’ responses (Interns—59% strong vs. Supervisors—44% 

strong). Just over half of Discover supervisors gave strong ratings to this item, whereas 61% of 

interns gave it a strong rating. In examining the discrepancies between supervisor and intern 

perspectives, an important consideration is that we did not have intern and supervisor respondents 

from exactly the same internships completing the surveys. While it is possible that interns and 

supervisors had different perspectives, the differences may be a result of respondents describing 

different internships. 

Interpersonal experience. Most Discover and Achieve interns indicated they were 

comfortable at work and were treated respectfully. About 1/3 of each group indicated their 

supervisors could be more sensitive to cultural differences. 

Opportunities for Learning. More than 3/4 of Discover interns and nearly 4/5 of Achieve 

interns felt strongly that they learned job-related skills; supervisors had similar responses, though 

Achieve interns gave stronger responses than the supervisors. More than half of Achieve 

supervisors (57%) and Discover supervisors (57%) chose strong responses about having a mix of 

short and long-term projects. This aligned fairly closely with the interns’ responses. 

Approximately 3/4 of interns reported that someone talked to them about future plans and a career 

in the field at least a few times during the internship. Interns and supervisors gave different 

feedback on how challenging tasks were in their internships. Supervisors tended to think tasks 

were more challenging than interns did. More than half of Achieve supervisors (54%) gave strong 

ratings, whereas 43% of interns did. The differences between Discover supervisors and interns 

were greater. Sixty percent of Discover supervisors gave strong ratings; only 37% of interns did. 

Some combination of factors may explain the discrepancies including the fact that some of the 

data refer to different internships (e.g., we may have a survey response from a supervisor but not 

the corresponding intern or vice versa) and that the responses are estimates or impressions. On 

opportunities to learn about a career in the field, about 4/5 of Achieve supervisors said interns 

had experiences at least a few times over the summer; this was about the same as interns’ 

responses. About 3/4 of Discover supervisors said interns had such experiences; the interns 

responded similarly. 

Center for Applied Research and Educational Improvement, University of Minnesota 

51 
 

 

 

 

 

 

 

Voice and Choice. About 2/3 of interns in each group felt strongly that they had input and 

choices about what they did during their internships. 

Supervision. Many supervisors did not use the Hirability Skills framework in working 

with their interns. Among those who used the Hirability Skills framework, about half said it was 

moderately helpful. More than half of Achieve supervisors and two-thirds of Discover supervisors 

reported spending an hour or more each week teaching job-related skills to their interns. Threefifths of supervisors reported spending 15-45 minutes per week on each of the following: 

developing or reviewing work plans, giving feedback, checking in, and talking about the career 

field. 

Interns reported getting more frequent feedback than supervisors reported giving it. The majority 

of Achieve supervisors reported giving feedback to their interns a few times over the summer or 

once a week. Nearly a third gave feedback a few times a week or every day. Achieve interns 

reported getting feedback more frequently; close to half (45%) said they got feedback a few times 

a week or daily. Half of Discover supervisors also reported giving feedback to their interns a few 

times over the summer or once a week. Nearly half of Discover interns (47%) said they got 

feedback a few times a week or daily. 

Participant Satisfaction: Interns 

Training. Students at all training levels rated their experiences highly—both on the 

trainers and the content. Those completing an internship used their skills from training in their 

jobs and said that training prepared them for their experiences. Students reported that training 

content was not new to them, especially those in the Advanced training level. In open-ended 

questions about how to improve the program, interns made numerous suggestions about how 

training could be improved. The most common recommendation was that the training be 

shortened. See Appendix L for a more detailed analysis of the trainee response data, data in 

tabular form, and suggestions for improvement. 

Internships. Most Achieve interns had internships related to their interests. Almost all 

interns agreed that their internships were a valuable learning experience. Almost all interns felt 

they made a valuable contribution to their workplace. Internships helped 78% of Achieve interns 

and 69% of Discover interns decide what career to pursue in the future. Almost half of Achieve 

interns indicated a likelihood of staying in touch with their STEP-UP supervisors. About 37% of 

Discover interns expressed that intention. Interns reported many benefits from participating in 

STEP-UP. These included personal learning and growth, professional learning, and the building 

of relationships. Most STEP-UP supervisors (93% Discover, 85% Achieve) felt their interns made 

a valuable contribution to their organization. Most (88% Discover, 83% Achieve) also felt their 

interns were a good fit for the position. Supervisors also reported benefits from their participation. 

Center for Applied Research and Educational Improvement, University of Minnesota 

52 
 

 

 

 

 

These included supporting the development of youth, harnessing energy and perspectives of youth 

in the workplace, expanding their supervisory/mentoring skills, learning about interns’ cultures, 

and getting help at their sites. 

Placements. Over 4/5 of Achieve interns found their job placements related to their 

interests (82%). Slightly fewer Discover interns found their job placement was a good fit with 

their personal interests (3/4 agreed or strongly agreed (75%). See Appendix L for details. 

Interns commented about the placements in open-ended questions. Many of them expressed 

gratitude for the placements, and for placements in their fields of interest. About her placement 

one intern said, “I love it there and I’m making a huge difference in the community.” Some were 

unhappy with their placements. Others wished there were more options and job opportunities in 

different fields. A sample of comments about the relationship between interns’ interests and their 

placements can be found in Figure 6. Some interns found that the internship helped them 

recognize an area of interest for further exploration. Some reported being placed in an area that 

they thought interested them, only to learn through the internship that this was not really a field 

for them. For others, the internship helped confirm their desire to work in the field.

 Figure 6. Intern comments on placements. 

Center for Applied Research and Educational Improvement, University of Minnesota 

53 
 

 

 

 

 

Work Experience. As Figure 7 shows, Achieve interns found their internships to be 

valuable learning experiences (96% agree or strongly agree). More than three quarters of Achieve 

interns (78%) who responded to the survey reported that their jobs helped them decide what 

career to pursue in the future. Ninety-five percent of Achieve interns felt they made valuable 

contributions to their workplaces. These results are consistent with responses to surveys in 2014 

and 2015. 

Discover interns, whose responses are shown in Figure 8 below, also found their internships to be 

valuable learning experiences (95% agree or strongly agree). Over two-thirds (69%) of Discover 

interns who responded to the survey reported that their jobs helped them decide what career to 

pursue in the future. Ninety-six percent of Discover interns felt like they made a valuable 

contribution to their places of work. These results are consistent with responses to surveys in 

2014 and 2015. 

Figure 7. Achieve satisfaction of work experience Figure 8. Discover satisfaction of work 

experience 

Experience with Supervisors. In open-ended comments about supervisors, interns 

mentioned specific things they valued in their supervisors: providing help and support, finding 

appropriate and interesting things for the intern to work on, talking about careers and future 

planning, giving advice about future education. Nearly all the comments about supervisors 

expressed appreciation. In a few comments, though, interns had negative feedback about 

supervisors. These interns said they did not get along well with their supervisor, that their 

supervisors weren’t adequately prepared to have an intern, the supervisor was not present 

consistently, or didn’t provide challenging tasks, direction or feedback. One mentioned the need 

for an understanding of Ramadan and what it means for Muslim interns. 

Support from STEP-UP. For information on the support interns received from STEP-UP 

during the summer, gathered primarily for the purpose of program improvement, see Appendix L. 

Center for Applied Research and Educational Improvement, University of Minnesota 

54 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Perceived Benefits. Many interns who completed the survey responded to open-ended 

questions about how they benefited from their work experiences. A summary of perceived 

benefits appears in Figure 9 below; the benefits fell into the categories of personal learning, 

professional learning and relationship-building. A more detailed analysis appears in Appendix L. 

Suggestions from interns for improving the program can be found in Appendix L. 

Personal Learning Professional 

Learning 

Relationship 

Building 

• Self-understanding • Specific skills • Networking 

• Increased (e.g., photography) • Working with 

responsibility • Career paths people from 

• Time management • Professional different 

• Social skills writing backgrounds 

• Accepting • Interviewing & • Adult guidance 

criticism resume writing and support 

• Managing money • Dressing for work 

Figure 9. A selection of benefits of STEP-UP reported by interns. 

Participant Satisfaction: Supervisors 

From the perspective of the majority of supervisors, STEP-UP was a success in their 

organizations. Among the Achieve supervisors responding to the end-of-summer survey 

(n = 224), 80% said STEP-UP was moderately to very successful at their site. Of the Discover 

supervisors responding to survey (n = 95), 94% said STEP-UP was moderately to very successful 

at their site. (See Appendix L for tabulated results.) 

Placement. Supervisors responded to the question, “Was the student matched to your 

company a good fit for the job?” Of the Discover supervisors who responded to the question on 

the survey, 88% felt their interns were a good fit. Of the Achieve supervisors, 83% felt the fit was 

good. Among the supervisors who said the fit was not good, the most frequently cited reasons 

they gave were that the students lacked the needed skills or were not reliable. In open-ended 

Center for Applied Research and Educational Improvement, University of Minnesota 

55 
 

 

 

 

 

 

 

questions about how to improve the program, several suggestions were offered about how the 

matching process could be improved (See Appendix L). 

Supervisors also responded to the question, “Overall, did your intern make a valuable contribution 

to your organization this summer?” Among the supervisors who responded to the question, 93% 

of Discover supervisors (n = 102) and 85% of Achieve supervisors (n = 224) felt their intern 

made a valuable contribution to their organization. There was closer alignment on this question 

between Discover supervisors and interns (96% of interns) than between Achieve supervisors and 

interns (95% of interns). 

Time commitment. In the end-of-summer survey, supervisors were asked how the actual 

time they spent training and supervising interns compared to the time they expected to spend. The 

majority of supervisors in both Achieve and Discover spent about the same amount of time 

supervising interns as they expected. 

Support from STEP-UP. For information on the support supervisors received from 

STEP-UP during the summer, gathered primarily for the purpose of program improvement, see 

Appendix L. 

Perceived benefits. In response to an open-ended question about what they most enjoyed 

about serving as a STEP-UP supervisor, respondents reported that they valued helping young 

people learn about the workplace, supporting the development of their skills, and watching them 

grow. Many of them appreciated the energy and perspectives the young people brought to the 

workplace. For many of the supervisors, the experience of mentoring a young person was an 

opportunity to expand their own skill sets. They reported learning from the students, including 

learning about the cultures from which the interns came. Many appreciated the opportunity to 

develop relationships with youth. Some supervisors mentioned the contributions the interns made 

to projects at their workplace. 

Suggestions from supervisors for improving the program can be found in Appendix L. 

Student Perceptions of the Program 

Achieve staff members reported on the findings that emerged from the group interviews they 

conducted about perceptions of the STEP-UP program (see description of the study in the section 

on Method on pg. 18). The interviews grew out of a concern by program staff that there are 

eligible students who miss the opportunity to do a STEP-UP internship (including students who 

apply and even participate in training). The interviews explored students’ perceptions of the 

program and staff planned to use what they learned to improve program recruitment and retention. 

Non-participants and Discover interns. The first set of findings came from students who 

Center for Applied Research and Educational Improvement, University of Minnesota 

56 
 

 

 

 

 

 

had not participated in the program at all or who had participated only in Discover. These students 

saw STEP-UP as a way to get a job and earn money when they were not old enough to get one on 

their own, and a means for developing job skills and clarifying their interests. They were 

influenced to apply by people (parents, friends, teachers) and by the prospect of earning money. 

STEP-UP’s messaging about developing a long-term career trajectory did not appear to be an 

influence with these students. 

Retention (from training through internship and from one year to the next) appeared to be 

negatively influenced by a number of factors: 

 training (e.g., work readiness training was too long, repetitive) 

 assurance of a job (e.g., there is no guarantee of a job even if a student completes 

training) 

 time lag (there is a long wait between the end of training and notification of hiring; 

students feared that if they waited and did not get a job it would be too late to find 

something else) 

 perceived lack of choice of job (not adequate choice or being placed in a job unrelated to 

their interests) 

 limited duration of internship opportunities (e.g., students wanted work year-round and 

not just in the summer) 

Retention from one year to the next also appeared to be influenced by previous experiences in 

STEP-UP, including the relationship with one’s supervisor and the extent to which the interns 

enjoyed the work they did. Many of the students who saw STEP-UP as a program for youth too 

young to get jobs otherwise expressed interest in “getting a job on their own,” once they got some 

experience and were old enough to do so. Students did not understand the differences between 

Discover and Achieve except that one is for younger and the other for older students. They were 

unaware of the variety of jobs in Achieve, for example. 

Achieve interns. There was some overlap in these perceptions and those of interview 

participants who had Achieve internships. They echoed the idea that students generally get started 

in STEP-UP as a way to have a job and earn money as a young teen. At this age they did not see 

the longer-term value of the program such as career development and networking. These Achieve 

interns listed numerous benefits of the program, including the development of both soft skills and 

job-related skills. They saw training as a benefit, though they also found it too long and 

repetitive. These students were able to differentiate between Discover and Achieve (e.g., Achieve 

involves more options for work, more responsibility, greater choice of work), and acknowledged 

understanding the differences only when they participated in Achieve. These interns also talked 

about the desirability of jobs that extended beyond the summer into the school year. 

Center for Applied Research and Educational Improvement, University of Minnesota 

57 
 

 

 

 

 

 

 

 

 

Summary of Study Findings by Evaluation Questions 

The discussion that follows summarizes the findings according to the research questions upon 

which the study was based. 

1. Has the STEP-UP program implemented all of its major components with fidelity? 

a. To what extent have the youth output targets been met? (comparison of current 

year’s outputs to current year’s targets) 

Youth output numbers were generally higher in 2016 than in 2015. In all categories 

where there were targets, actual numbers were lower than the 2016 targets. 

Eligible applicants. Compared to targets for 2016, the actual number of eligible 

applications fell short of targets by 553 (4,000 vs. 3,447). There were more applicants 

in 2016 than in 2015 (3,062) but not as many as in 2014 (3,678). 

Work-readiness training completers: Compared to targets for 2016, the number of 

students completing work-readiness training fell slightly short of the target (2,200 vs. 

2,036). More students completed work-readiness training than in 2015 (1,879) and was 

close to the number in 2014 (2,076). 

Internships completed. In 2016, the number of youth completing internships fell shy 

of the target (1,310 vs. 1,272). However, more youth completed internships in 2016 

than in 2015 (1,132) and in 2014 (1,222). 

Extra enrichments/specialized career training. For activities of 20 or fewer hours, 

424 interns participated in 2016. Due to a glitch in the record keeping for one of the 

large enrichment programs, we were unable to calculate a precise number; the total 

number of students who participated in at least one activity is probably 75-90. For this 

reason, it is difficult to compare the number of actual participants to the target of 500 

or to the number for 2015 (470). For activities longer than 20 hours, 120 interns 

participated in 2016, more than the target. This was a slight drop from 2015 (124).37 

At least 315 different students completed pipeline/career enrichment activities. 

Work-based learning credit. Four hundred twenty Discover interns attended weekly 

training classes during the summer and earned a total of 719 credits. No targets were 

37 Comparisons to 2014 are not included. Prior to 2014 distinctions were not made between activities on the basis of 

program length. The number of youth who participated in any enrichment in 2014 was 522. 

Center for Applied Research and Educational Improvement, University of Minnesota 

58 
 

 

 

 

 

 

 

 

set for learning credits. In 2015, 522 attended weekly training classes and earned 859 

credits. In 2014, 394 interns earned 750 credits. 

Wages. In 2016, STEP-UP interns earned an estimated $2.29 million in wages 

($1,346,340 through Achieve; $944,886 through Discover). There were no targets for 

wages. The overall number is about the same as in 2015, but in 2016, Achieve interns 

earned more than the previous year and Discover interns earned less. ($1,164,563 

through Achieve; $1,102,895 through Discover). In 2014 total wages were higher— 

approximately $2.4 million ($1,130,688 through Achieve and $1,315,376 through 

Discover). 

b. To what extent have the employer output targets been met? 

Number of available internships. In 2016, 1,389 internships were available, falling 

short of the target (1,420) yet exceeding the number available in 2015 (1,238). 

However, the number of available internships in 2016 still represented a reduction 

from the number that was available in in 2014 (1,425). 

Number of companies. In 2016, 226 companies had STEP-UP interns. This was an 

increase over 2015 (212 companies) and a decrease from 2014 (268). While the 

program sought to increase the number of companies in 2016 over previous years, they 

did not have exact targets. Targets were set for the number internships and not 

companies. 

Number of supervisors. In 2016, 513 supervisors worked with STEP-UP interns. 

This was a small increase over 2015 (505) and was fewer than in 2014 (539). No 

targets were set for the number of supervisors. 

Types of settings. Achieve interns served in a range of industries as well as non-profit 

and government settings. While Discover interns worked in the nonprofit sector only, 

the types of organizations in which they were placed also varied. See Appendix J for a 

list of industries and the numbers/percentages of interns in each. 

c. To what extent have systems building output targets been met? 

Enrichment opportunities. Enrichment opportunities included programs of under 20 

hours to those over twenty hours. Some provided general information for youth 

(getting ready for college; financial literacy), others about specific careers (included 

required training for particular internships), and others for general skill acquisition. 

For a complete list of these opportunities, see Appendix D. 

Center for Applied Research and Educational Improvement, University of Minnesota 

59 
 

 

 

 

 

 

Systems building. The program has moved from using the language of systemsbuilding and now talks about enrichment opportunities for youth in various fields of 

employment. 

See section 1(a) above on youth for enrichment targets. 

d.	 How do the components of the implemented STEP-UP program compare to those 

described in the proposal? 

The components of the program (training, internships, enrichment opportunities) are 

the same as those described in the SEP proposal of 2013. There have been program 

refinements in the components based on staff learning over time. 

e.	 What was the nature of student training for STEP-UP? 

There were four levels of training—Advanced, Achieve, HS Discover, and MS 

Discover. AchieveMpls runs the first three; the City of Minneapolis contracts Project 

for Pride in Living to run the fourth. Training takes place in the spring prior to the 

internship on Saturdays. A separate component of the training process is a weekday 

mock interview for the Advanced, Achieve, and HS Discover levels. A detailed 

description of the training process is located on pages 10-11 in this report. An outline 

of the training curriculum can be found in Appendix B. 

There was not a formal evaluation of training by the training staff in 2016. Based on 

experiences with training, informal feedback from trainers, and feedback from youth 

on end-of-summer surveys, the curriculum for training is being rewritten for the spring 

of 2018. 

Students at all training levels rated their training experiences highly—both on the 

trainers and the content. Those completing an internship used their skills from training 

in their jobs and said that training prepared them for their experiences. Students 

reported that training content was not new to them, especially those in the Advanced 

training level. In open-ended questions about how to improve the program, interns 

made numerous suggestions about how training could be improved. The most common 

recommendation was that the training be shortened. See Appendix L for a more 

detailed analysis of the trainee response data, data in tabular form, and suggestions for 

improvement. 

f.	 Who were the participants in the program? 

 More girls than boys completed internships in 2016 (57% vs. 43%) 

Center for Applied Research and Educational Improvement, University of Minnesota 

60 
 

 

 

 

 

	 The racial/ethnic breakdown was 33% African Americans, 29% ethnic 

Africans, 11% Asians, 9% Hispanic, 6% Native Americans, 6% white, and 6% 

other or unidentified 

 90% of interns were eligible for free or reduced price lunch 

 Just over 2/3 of Achieve interns held paying jobs prior to the 2016 internship; 

nearly 2/3 of Discover interns had never held a paying job 

g.	 What was the participants’ satisfaction with the program? 

In summary, there were high levels of satisfaction with the program among youth who 

completed an internship and among supervisors. The data are presented in 

Appendix L. 

 Most Achieve interns had internships related to their interests. The relationship 

was less strong for Discover interns, but the majority of all interns had jobs 

related to their interests 

	 Almost all interns agreed that their internships were a valuable learning 

experience 

 Almost all interns felt they made a valuable contribution to their workplace 

 The majority of interns said their internships helped them decide what career to 

pursue in the future 

	 Almost half of Achieve interns indicated a likelihood of staying in touch with 

their STEP-UP supervisors. Over one-third of Discover interns expressed that 

intention 

	 Most STEP-UP supervisors felt their interns made a valuable contribution to 

their organization. Most also felt their interns were a good fit for the position 

	 Interns reported many benefits from participating in STEP-UP. These included 

personal learning and growth, professional learning, and the building of 

relationships 

	 Supervisors reported these benefits: supporting the development of youth, 

harnessing the energy and perspectives of youth in the workplace, expanding 

their supervisory/mentoring skills, learning about interns’ cultures, and getting 

help at their sites. 

h. What was the quality of the student internships? 

Center for Applied Research and Educational Improvement, University of Minnesota 

61 
 

 

 

 

We were not able to measure internship quality given variation in settings and 

availability and quality of data. We were able to gather data on elements of quality— 

satisfaction (see (g) above), and exposure. 

Staff recommended that supervisors provide the following to support a quality 

internship experience for students: 

	 A job description and work plan with expectations and goals 

	 A strategy for supporting the intern (giving clear directions, checking in 

regularly and reflecting on the experience together, talking to them about 

future plans, helping them develop professional skills, talking to them about 

the company and its activities, talking about their own education and work 

history) 

 A project for the intern to undertake 

 A mentor (either the supervisor or another adult) 

 Regular feedback to the intern (including a mid-internship check-in and a 

performance evaluation) 

 Opportunities to learn about a career in the specific fields through activities 

such as job shadow, attending meetings, informational interviews 

 Opportunities to develop networks (e.g., lunch with supervisor, attending social 

gatherings with colleagues, meeting the president or CEO, attending meetings) 

 Specialized training (e.g., Excel, Outlook) 

Data from intern and supervisor surveys at the end of the summer indicated the 

following about exposure to these recommended opportunities: 

	 About 3/5 of Achieve and Discover interns reported having structured work 

plans with measurable goals; supervisors reported somewhat differently 

	 Most Discover and Achieve interns indicated they were comfortable at work 

and were treated respectfully. About 1/3 of each group indicated their 

supervisors could be more sensitive to cultural differences 

 More than 3/4 of Discover interns and nearly 4/5 of Achieve interns felt 

strongly that they learned job-related skills 

 Supervisors tended to think tasks assigned to interns were more challenging 

than interns did 

	 About 4/5 of Achieve supervisors and interns reported opportunities to learn 

about a career in the field and about 3/4 of Discover supervisors and interns 

reported similarly 

	 About 2/3 of interns (Achieve and Discover) reported having had input and 

choice about what they did in their internships 

Center for Applied Research and Educational Improvement, University of Minnesota 

62 
 

 

 

 

 

 

 

	 More than half of Achieve supervisors and two-thirds of Discover supervisors 

reported spending an hour or more each week teaching job-related skills to 

their interns 

	 Three-fifths of supervisors reported spending 15-45 minutes per week on each 

of the following: developing or reviewing work plans, giving feedback, 

checking in, and talking about the career field 

2.	 To what extent has the STEP-UP program increased its outputs over time? 

The numbers of eligible applicants, students accepted, students completing work-readiness 

training, students placed in internships, students completing internships, number of 

internships, and number of supervisors all increased in 2016 compared to 2015. Most of the 

numbers were lower in 2016 than in 2014. See Table 2 for more details. 

3.	 How do perceptions of the STEP-UP Program impact applications and retention?38 

a.	 How do youth and personnel in MPS perceive the STEP-UP program? 

	 Students who had not participated in the program at all or who had participated 

only in Discover saw STEP-UP as a way get a job and earn money when they 

could not get one on their own, and as a means for developing job skills and 

clarifying their interests. They did not tend to see the longer-term value of the 

program such as career development and networking. 

	 Achieve interns echoed the idea that students generally get started in STEP-UP as 

a way to have a job and earn money as a young teen. At this age or stage of 

development they did not see the longer-term value of the program such as career 

development and networking. 

	 Students who had the experience of Achieve enumerated benefits of the program, 

including the development of both soft skills and job-related skills. They saw 

training as a benefit, though they also found it too long and repetitive. 

38 This set of questions about perceptions was added to the SEP modification that was approved in September, 2016. 

In light of the recognition by program staff that there are eligible students who miss the opportunity to do a STEP-UP 

internship (including students who apply and even participate in training), project staff wanted to explore perceptions 

of the program by teens and to use their learning for improving program recruitment and retention. See the methods 

and findings sections for further information about the study. 

Center for Applied Research and Educational Improvement, University of Minnesota 

63 
 

 

 

 

 

	 Achieve interns were able to differentiate between Discover and Achieve (e.g., 

Achieve involves more options for work, more responsibility, greater choice of 

work) and acknowledged understanding the differences only once they participated 

in Achieve. 

	 Discover interns or non-participants did not seem to understand the differences 

between Discover and Achieve, except that one is for younger and the other for 

older students. They were unaware of the variety of jobs, for example. 

b.	 What factors influence youth to apply to STEP-UP? 

 Discover interns were influenced to apply by people (parents, friends, teachers) 

and by the prospect of earning money. 

	 STEP-UP’s messaging about developing a long-term career trajectory did not 

appear to be an influence with Discover students. 

c.	 What factors influence youth retention in the STEP-UP program (from application 

through training; from one year to another) 

 The following appeared to be challenges with regard to retention: 

o	 training (e.g., work readiness training was too long, repetitive) 

o	 assurance of a job (e.g., there is no guarantee of a job even if a student 

completes training) 

o	 time lag (there is a long wait between the end of training and notification of 

hiring; students feared that if they waited and did not get a job it would be too 

late to find something else) 

o	 perceived lack of choice of job (not adequate choice or being placed in a job 

unrelated to their interests) 

o	 limited duration of internship opportunities (e.g., students wanted work yearround and not just in the summer) 

	 Previous STEP-UP experience could support or interfere with retention, in 

particular: 

o	 relationship with supervisor 

o	 enjoyment of job 

Findings: Impact Evaluation 

Demographics of the Studied Sample 

Details about the demographics of the sample are presented in Table 4. The largest racial group 

was African Americans. Around 4/5 of the sample was eligible for free and reduced lunch. A very 

Center for Applied Research and Educational Improvement, University of Minnesota 

64 
 

 

 

 

small percentage of the sample was homeless or highly mobile. In the Achieve sample about 20% 

were special education students; in the Discover sample there were about 14% special education 

students. The home language of the majority of Discover students and matches was English; for 

Achieve that was the case for about 45%. The next most commonly spoken home language was 

Somali. 

Table 4 

Demographics of Studied Sample 

Achieve 

MPS Sample** 

Achieve 

Matched 

Sample## 

Discover 

MPS Sample** 

Discover 

Matched 

Sample## 

GENDER Male 47% (127) 48% (108) 

Female 53% (143) 52% (119) 

RACE/ETHNICITY African 

American 

60% (163) 51% (95) 60% (135) 50% (77) 

Native 

American 

2% (6) 3% (6) 4% (10) 7% (10) 

Asian 15% (41) 17% (32) 16% (36) 14% (21) 

Hispanic 12% (31) 17% (31) 10% (23) 20% (30) 

Caucasian 11% (29) 12% (22) 10% (23) 10% (15) 

Missing or 

Other 

FRL Yes 79% (214) 76% (142) 79% (180) 80% (123) 

No 21% (56) 24% (44) 21% (47) 20% (30) 

HOMELESS Yes .4% (1) .5% (1) 0 0 

No 99.6% (269) 99.5% (185) 100% (227) 100% (153) 

SPECIAL ED Yes 20% (54) 20% (38) 14% (31) 14% (22) 

No 80% (216) 80% (148) 86% (196) 86% (131) 

HOME 

LANGUAGE 

English 44% (119) 46% (85) 60% (135) 60% (92) 

Hmong 13% (35) 12% (23) 12% (26) 9% (14) 

Somali 23% (63) 22% (40) 16% (37) 15% (23) 

Spanish 11% (29) 12% (22) 9% (20) 12% (18) 

Other/ 9% (24) 9% (16) 4% (9) 4% (6) 

Center for Applied Research and Educational Improvement, University of Minnesota 

65 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Unknown 

**Interns who were enrolled at least 75% of both SY16 and SY17 for whom we had data on school outcomes 

##Matched students who were enrolled at least 75% of both SY16 and SY17 

First Design, Moderate Evidence: Between-Groups Design with Matched 

Comparison Groups Based on Propensity Score Matching 

Grade point average (GPA). After removing cases with mistakes in data entry for 

GPA,39 a one-way ANCOVA was conducted to compare SY17 GPA for students who completed 

an internship in 2016 and matched students after controlling for GPA in the school year before the 

internship. As illustrated in Table 5, average GPA’s were higher for STEP-UP students than for 

non-participants, with larger differences between Discover participants and matched students 

prior to the internship. This suggests that coming into the program and following it, students who 

participated in STEP-UP in 2016 were stronger students than matched students. After adjustment 

for GPA in SY16, Achieve interns’ GPA’s were significantly higher than matched students in 

SY17, F(1, 448) = 9.391, p = .002, partial η2 = .021, but overall GPA decreased for both interns 

and matched students. After adjustment for GPA in SY16, there was no significant difference 

found between interns and non-participants in GPA in SY17 for the Discover level, F(1, 374) = 

.038, p = .845, partial η2 < .001. See Appendix M for information on testing of assumptions and 

adjusted means. See Appendix N for full model outputs. 

Table 5 

Comparison of Mean GPA for Interns and Matched Students SY16 and SY17 

Mean 

GPA 

SY2016 

SD N Mean 

GPA 

SY2017 

SD N 

ACHIEVE Intern 2.90 .750 270 2.89 .709 270 

Matched 2.68 .846 181 2.63 .815 181 

DISCOVER Intern 2.86 .775 226 2.79 .782 226 

Matched 2.72 .994 151 2.64 1.003 151 

Attendance. We examined the outcome of school attendance using a logistic regression 

model that controlled for attendance in SY16. The analysis included only students who were 

enrolled at least 75% of both SY16 and SY17. As illustrated in Table 6, both Achieve and 

Discover interns had higher percentage attendance in both SY16 and SY 17 and all groups had a 

lower rate of attendance in SY17 than in SY16. Perhaps the most important outcome is that 

39 At the Achieve level, 5 cases were removed (3.6%); all were matched students. At eh discover level 2 cases were 

removed (1.3%); both were matched students. 

Center for Applied Research and Educational Improvement, University of Minnesota 

66 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

students in all groups had average attendance rates above 90%. We found a statistically 

significant difference for both Achieve (p = 0.0186) and Discover (p <.01) on attendance in SY17 

when controlling for attendance in SY16 between interns and matches, and Table 7 provides 

results of the analysis. While statistically significant, the effect of the difference in practical terms 

is slight (odds ratio for Achieve, 1.077; odds ratio for Discover, 1.217). The difference between 

interns and matches at the Achieve level is a difference between an attendance rate of 93% and 

91% and at the Discover level between 94% and 92%. 

Table 6 

Comparison of Rate of Attendance for Interns and Matched Students SY16 and SY17 

Mean % 

attendance 

SY16 

SD N Mean % 

attendance 

SY17 

SD N 

ACHIEVE Intern 0.9495 0.0624 225 0.9294 0.0815 225 

Matched 0.9337 0.0721 136 0.9130 0.0946 136 

DISCOVER Intern 0.9508 0.0576 178 0.9370 0.0708 178 

Matched 0.9477 0.0582 111 0.9235 0.0890 111 

Table 7 

Comparison of Attendance for Interns and Matched Students, Controlling for Attendance in SY16 

Estimate Standard Error z-value p 

ACHIEVE Intercept -5.31870 0.14593 -36.447 <2e-16 *** 

Attendance 16 8.41456 0.16073 52.352 <2e-16 *** 

Participant or not 0.07440 0.03162 2.353 0.0186* 

DISCOVER Intercept -6.34490 0.19894 -31.894 <2e-16 *** 

Attendance 16 9.50747 0.21536 44.146 <2e-16 *** 

Participant or not 0.19654 0.03691 5.325 1.01e-07*** 

*Estimate –the estimated parameter is on the log odds scale. 

**z – the Z statistic 

*** 0.001 

Center for Applied Research and Educational Improvement, University of Minnesota 

67 
 

 

 

 

 

 

 

 

 

 

 

 

Behavior: referrals, removals, and suspensions. As shown in Figure 10, the data about 

Achieve and Discover interns and matched students show that the majority (and in some cases 

nearly all) STEP-UP interns had no involvement with the disciplinary system in Minneapolis 

Public Schools. Findings about changes in the number of referrals, removals, and suspensions 

should be considered in light of the larger finding that most STEP-UP interns and their matches 

had no disciplinary records. 

Figure 10: Involvement of Interns and Matched Students in MPS Disciplinary System in SY17. 

Behavior Referrals.40 Proportions of STEP-UP interns and matched students with 

behavior referrals were calculated and compared using a chi-square test. All expected cell 

frequencies were greater than five. Of 270 Achieve participants, only 8 (3%) had referrals in 

SY17; among matched students, 5 of 186 (3%) had referrals. Achieve students who participated 

in STEP-UP were no less likely to have behavior referrals than matched students (X2 (1) = .03, p 

= .862). Of 227 Discover participants, 18 (8%) had referrals; among matched students, 8 of 153 

(5%) had them. Discover students who participated in STEP-UP were no less likely to have 

behavior referrals than matched students (X2 (1) = 1.046, p = .306). We regressed41 the outcome, 

referrals in SY17, onto participation in the program while controlling for referrals in SY16. There 

were not significant differences between interns and matched students for either Achieve (Wald 

test statistic = .034, df = 1, p =.854) or Discover (Wald test statistic = .695, df = 1, p =.405) on 

behavior referrals. The reader should keep in mind that the numbers of students analyzed was 

very small. 

40 Further information about Minneapolis Public Schools’ behavior standards and code of conduct can be found at 

http://policy.mpls.k12.mn.us/uploads/policy_5200_2014.pdf. Behavior data for the district can be found at 

http://www.mpls.k12.mn.us/data_behavior and 

http://www.mpls.k12.mn.us/uploads/ocr_behavior_guidebook_website.pdf 

Details about levels of behavior interventions and responses can be found at 

http://policy.mpls.k12.mn.us/uploads/mps_levels_of_behaviors_interventions_and_responses.pdf 

41 We used logistic regression. See Appendix N for the full model output. 

Center for Applied Research and Educational Improvement, University of Minnesota 

68 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

A Mann-Whitney U test was run to determine if there were differences in the number of 

behavioral referrals (for those students who had them) between STEP-UP participants and 

matched students. Distributions of the number of referrals for interns and matched students were 

similar, as assessed by visual inspection. For Achieve, the difference in number of referrals for 

interns (8) and matched students (5) was not statistically significantly, U = 19, z = -.233, 

p = .943. For Discover, the difference in number of referrals for interns (18) and matched students 

(8) was not statistically significantly, U = 62, z = -.623, p = .605. These results suggest that 

participation in STEP-UP did not influence the likelihood or rate of referrals. 

Removals. Proportions of STEP-UP interns and matched students with removals (inschool or out of school <1 day) were calculated and compared using a chi-square test. Of 270 

Achieve participants, 6 (2%) had removals in SY17; among 186 matched students 5 (3%) had 

removals. Achieve interns were no less likely to have removals from class than matched students 

(X2 (1) = .10, p = .75). Of 227 Discover interns, 18 (8%) had removals in SY17; among the 153 

matched students 10 (7%) had removals. Discover interns were no less likely to have removals 

from class than matched students (X2 (1) = .26, p = .61). We regressed42 the outcome, removals in 

SY17, onto participation in the program while controlling for removals in SY16. There were not 

significant differences between interns and matched students for either Achieve (Wald test 

statistic = .003, df = 1, p =.957) or Discover (Wald = .004, df = 1, p =.949) The reader should 

keep in mind that the numbers of students analyzed was very small. 

Because of the distribution of the data, it was not possible to run a Mann-Whitney U test to 

compare the differences in number of behavioral removals for those students who had them 

between STEP-UP participants and matched students. 

Suspensions. Proportions of STEP-UP interns and matched students with school 

suspensions were calculated and compared using a chi-square test. All expected cell frequencies 

were greater than five. Of 270 Achieve participants, 8 (3%) had suspensions in SY17; among 186 

matched students 10 (5%) had suspensions. Achieve interns were no less likely to be suspended 

from school than matched students (X2 (1) = 1.692, p = .193). Of 227 Discover participants, 31 

(14%) had suspensions in SY17; of the 153 matched students, 9 (6%) had suspensions. Discover 

interns were more likely to be suspended from school than matched students (X2 (1) = 5.865, p = 

.015), but the effect is minimal (Cramer’s V = .124). We regressed43 the outcome, suspensions in 

SY17, onto participation in the program while controlling for suspensions in SY16. There were 

not significant differences between interns and matched students for Achieve (Wald test statistic = 

.530, df = 1, p =.466) when we controlled for the prior year’s suspensions. At the Discover level, 

42 We used logistic regression. See Appendix N for the full model output. 

43 We used logistic regression. See Appendix N for the full model output. 

Center for Applied Research and Educational Improvement, University of Minnesota 

69 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

there was still a statistically significant difference between interns and matched students (Wald 

test statistic = 6.405, df = 1, p =.011) 

A Mann-Whitney U test was run to determine if there were differences in the number of 

suspensions (for those students who had them) between STEP-UP participants and matched 

students. For Achieve, distributions of the number of suspensions for interns and matched 

students were similar, as assessed by visual inspection. The difference in number of suspensions 

for interns (8) and matched students (10) was not statistically significantly, U = 33.5, z = -.796, p 

= .573. For Discover, distributions of the number of suspensions for interns and matched students 

were similar, as assessed by visual inspection. The difference in number of suspensions for 

interns (31) and matched students (9) was not statistically significantly, U = 108, z = -1.541, p = 

.321. 

These results suggest that participation in STEP-UP was related to the likelihood of suspensions 

but not the rate of suspensions for Discover students, and did not influence the likelihood or rate 

of suspensions for Achieve students. 

ACT composite scores. An independent-samples t test was conducted to compare ACT 

Composite test scores of STEP-UP interns who completed an internship in 2016 and matched 

students. ACT tests were taken by 11th graders. For Achieve interns there was not a significant 

difference in the scores for interns (N = 110, M = 16.65, SD = 4.801) and matched students (N = 

67, M = 16.67, SD = 4.584); t(175) = .023, p = .981. These results suggest doing a STEP-UP 

internship had no influence on ACT scores for Achieve level students. For Discover interns there 

was a smaller number of cases with ACT data; there was also not a significant difference between 

scores for interns (N = 32, M = 16.81, SD = 5.27) and matched students (N = 21, M = 15.71, SD = 

3.649); t(51) = -.832, p = .409. To control for prior achievement, we also conducted a one-way 

ANCOVA to examine differences between students and matches on ACT composite scores (using 

GPA in SY16 to represent prior academic achievement since students take this test only once 

during high school and could not, therefore, use prior scores as a co-variate). There were no 

differences at either the Achieve level (F(1, 174) = .201, p = .656, partial η2 = .005) or at the 

Discover level (F(1, 50) = .945, p = .332, partial η2 = .004). See Appendix N for full model 

outputs. 

MCA reading scores. An independent-samples t test was conducted to compare MCA 

reading test scores of STEP-UP interns who completed an internship in 2016 and matched 

students. MCA reading tests were taken by 10th graders. For Discover students, there was not a 

statistically significant difference in scores for interns (N = 67, M = 1038.64, SD = 16.242) and 

matched students (N = 60, M = 1039.77, SD = 17.486); t(125) = .376, p = .708. To control for 

prior achievement with the Discover level, we also conducted a one-way ANCOVA to examine 

Center for Applied Research and Educational Improvement, University of Minnesota 

70 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

differences between students and matches on MCA reading tests (using GPA in SY16 to represent 

prior academic achievement since students take this test only once during high school and could 

not, therefore, use prior scores as a co-variate). There were no differences at the Discover level 

(F(1, 127) = .210, p = .648, partial η2 = .002). These results suggest doing a STEP-UP internship 

did not have an influence on MCA reading scores for Discover students. For Achieve students we 

had data for only 7 cases. The mean score for interns (N = 4, M = 1027.25, SD = 16.86) was 

slightly higher than for matched students (N = 3, M = 1024.33, SD = 12.055). Due to small 

sample size at the Achieve level we did not run a t test or an ANCOVA because of the instability 

of parameter estimates. Cohen’s d for the difference between mean scores was .20. See Appendix 

N for full model outputs. 

MCA math scores. An independent-samples t test was conducted to compare MCA math 

test scores of STEP-UP interns who completed an internship in 2016 and matched students. For 

Achieve there was no significant difference in the scores for interns (N = 34, M = 1126.38, SD = 

15.65) and matched students (N = 19, M = 1127.74, SD = 18.187; t(32.9) = .273, p = .787. These 

results suggest doing a STEP-UP internship did not influence MCA Math scores. At the Discover 

level we had data for only 18 students. Levene’s test for the Discover level was significant 

(p = .006) and we therefore could not assume equal variances. The mean score for matched 

students (N = 8, M = 1127.13, SD = 7.259) was lower than for interns (N = 10, M = 1137.30, SD = 

22.48); t(11.25) = -1.346., p = .205 and the difference was not significant. To control for prior 

achievement, we also conducted a one-way ANCOVA to examine differences between students 

and matches on MCA math tests (using GPA in SY16 to represent prior academic achievement 

since students take this test only once during high school and could not, therefore, use prior scores 

as a co-variate). There were no differences at either the Achieve level (F(1, 50) = .153, p = .698, 

partial η2 = .110) or at the Discover level (F(1, 15) = .061, p = .808, partial η2 = .310). See 

Appendix N for full model outputs. 

On track to graduate (credit-ready). Chi-square analyses were conducted to compare 

the difference in the proportion of STEP-UP interns and matched students who were on track to 

graduate in four years, based on whether or not they had earned the requisite number of credits for 

their grade. All expected cell frequencies were greater than five at both the Achieve and Discover 

levels. At the Achieve level, there was a statistically significant association between participation 

in STEP-UP and being on track to graduate on time. Achieve interns were more likely to be on 

track to graduate than matched students (X2 (1) = 10.728, p = .001). Cramer’s V = .153, p = .001, 

shows a small effect size. At the Discover level, there was also a statistically significant 

association between participation in STEP-UP and being on track to graduate on time. Discover 

interns were more likely to be on track to graduate than matched students (X2 (1) = 12.513, p < 

.001). Cramer’s V = .181, p <.001, shows a small effect size. 

Center for Applied Research and Educational Improvement, University of Minnesota 

71 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

We also conducted a logistic regression to control for differences between student and matches on 

being on track to graduate in SY16. After controlling for being on track to graduate in SY16, 

there was a statistically significant difference between Achieve interns and their matches (Wald 

test statistic = 4.853, df = 1, p = .028 ) The odds of being credit ready in SY17 for Achieve interns 

were 2.7 times greater than the odds for matched students. Being an intern explained 47% of the 

variance (Nagelkerke R-Square = .467). There was also a significant difference for Discover 

interns (Wald test statistic = 3.801, df = 1, p = .051, ). The odds of being credit ready in SY17 for 

Discover interns were 2.5 times greater than the odds for matched students. Being an intern 

explained 54% of the variance (Nagelkerke R-Square = .54). See Appendix N for full model 

outputs. 

Graduation rates. A chi-square analysis was conducted to compare the difference in the 

proportion of STEP-UP interns and matched students who were in eleventh grade in SY2016 and 

received a diploma in SY2017. All expected cell frequencies were greater than five. Students who 

participated in STEP-UP were no more likely to graduate than matched students (X2 (1) = .017, 

p = .895). 

Tables 8, 9, 10, and 11 present a summary of the differences between interns and matched 

students for each of the school outcomes listed above. 

Table 8 

Comparison of Achieve Interns and Matched Students on GPA, Attendance, ACT Scores, MCA 

Reading Scores, MCA Math Scores, and Behavior 

ACHIEVE Group N Test 

Statistic 

df p Effect 

size 

Location 

of 

analysis 

in report 

Grade Point 

Average: GPA* 

Interns 270 9.391 1, 448 .002 0.021(eta 

squared) p. 65 Matched 181 

Attendance* 

Interns 225 

2.353 .0186 

1.077 

(odds 

ratio) 

Matched 136 p. 66

ACT Composite 

Scores 

Interns 110 .023 175 .981 X p. 69 Matched 67 

MCA Reading Scores Interns 4 No analysis performed due to power 

considerations p. 70 Matched 3 

MCA Math Scores Interns 34 .273 32.9 .787 X p. 70 Matched 19 

Behavior Referrals Interns 8 U = 19 

z = 

.943 X p. 68 Matched 5 

Center for Applied Research and Educational Improvement, University of Minnesota 

72 
 

 

 

 

 

 

 

 

 

-233 

Suspensions Interns 8 U = 

33.5 

z = 

-.796 

.573 X p. 69 Matched 10 

*Analysis controlled for prior year’s outcomes 

Table 9 

Comparison of Discover Interns and Matched Students on GPA, Attendance, ACT Scores, MCA 

Reading Scores, MCA Math Scores, and Behavior 

DISCOVER Group N Test 

Statistic 

df p d Location 

of 

analysis 

in report 

Grade Point 

Average: GPA* 

Interns 227 .038 1, 374 .845 X p. 65 Matched 153 

Attendance* 

Interns 178 

5.325 <.01 

1.217 

(odds 

ratio) 

Matched 111 p. 66

ACT Composite 

Scores 

Interns 32 -.832 51 .409 X p. 69 Matched 21 

MCA Reading Scores Interns 67 .376 125 .708 X p. 70 Matched 60 

MCA Math Scores Interns 8 -1.346 11.25 .205 X p. 70 

Center for Applied Research and Educational Improvement, University of Minnesota 

73 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Matched 10 

Behavior Referrals Interns 18 U = 62 

z = 

-.623 

Matched 8 .605 X p. 68

Suspensions Interns 31 U = 108 

z = -

1.541 

Matched 9 .321 X p. 69

*!nalysis controlled for prior year’s outcomes 

Table 10 

Comparison of Achieve Interns and Matched Students on Behavior Referrals, Removals, Suspensions, OnTrack to Graduate, and Graduation Rates 

ACHIEVE N X2 df p Cramer’s V Location 

of 

analysis 

in report 

Behavioral 

Referrals 456 .03 1 .862 X p. 68

Removals 456 .10 1 .75 X p. 68

Suspensions 456 1.692 1 .193 X p. 69

On-track to 

Graduate 456 10.782 1 .001 .153 p. 71

Graduation 

Rates 

230 .017 1 .895 X 

p. 71

X: Not included since p > .05

Table 11 

Comparison of Achieve Interns and Matched Students on Behavior Referrals, Removals, Suspensions, OnTrack to Graduate, and Graduation Rates 

DISCOVER N X2 df p Cramer’s V Location 

of 

analysis 

in report 

Behavioral 

Referrals 380 1.046 1 .306 X p. 68

Removals 380 .26 1 .61 X p. 68

Suspensions 380 5.865 1 .015 .124 p. 69

On-track to 

Graduate 380 12.513 1 <.001 .181 p.71 

Graduation 

Rates 

No analysis performed; not applicable at the Discover 

level 

X: Not included since p > .05

Center for Applied Research and Educational Improvement, University of Minnesota 

74 
 

 

 

 

 

 

 

 

 

 

 

 

 

Post-Secondary Enrollment. A chi-square analysis was conducted to compare the 

difference in the proportion of STEP-UP interns and matched students in the 2014 and 2015 

cohorts who enrolled in post-secondary programs. All expected cell frequencies were greater than 

five. Students who participated in STEP-UP were no more likely to enroll in a post-secondary 

program than matched students, 2014 cohort: (X2 (1) = 1.892, p = .169) and 

2015 cohort: (X2 (1) = .003, p =.955).44 

Table 12 presents a summary of the differences between interns and matched students for each of 

the cohorts for post-secondary enrollment. 

Table 12 

Comparison of post-secondary enrollment data for participants and comparison students 

for 2014 and 2015 cohorts 

N X2 df p Cramer’s V Location 

of 

analysis 

in report 

2014 Cohort 450 1.892 1 .169 .065 p. 74 

2015 Cohort 211 .003 1 .955 .004 p. 74 

Social-Emotional Learning: Persistence. Of the five independent t tests conducted for 

the three cohorts, one showed statistically significant results. Discover interns from the 2016 

cohort scored significantly higher on academic persistence than non-STEP-UP students. (Interns 

N = 172, M = 4.07, SD = 0.69) and matched students (N = 123, M = 3.80, SD = 0.90; t(293) = -

2.84, p=.005. Cohen’s d = 0.34 suggested a small effect. See Table 13 for details. 

Table 13 

SEL data for analysis conducted by REAA at Minneapolis Public Schools 

2014 2015 2016 

STEP-UP Non STEP-UP Non STEP-UP Non 

n M 

(SD) 

n M 

(SD) 

t 

(df) 

N M 

(SD) 

n M 

(SD) 

t 

(df) 

n M 

(SD) 

n M 

(SD) 

t 

(df) 

44 We conducted a sensitivity analysis using a multiple regression model that controlled for all variables in the 

original propensity score match. Results remained the same—there was no effect of program participation in 

postsecondary enrollment. 

Center for Applied Research and Educational Improvement, University of Minnesota 

75 
 

 

ACHIEVE - - - - - 152 4.21 

(.69) 

139 4.11 

(.75) 

-1.17 

(289) 

340 4.11 

(.75) 

253 4.11 

(.72) 

.04 

(296) 

DISC. - - - - - 148 4.00 

(.72) 

147 3.90 

(.77) 

-1.14 

(293) 

172 4.07 

(0.69) 

123 3.80 

(.90) 

-2.48* 

(293) 

ALL 381 4.12 426 4.09 -.70 

(1310) 

*p<.05 

Center for Applied Research and Educational Improvement, University of Minnesota 

76 
 

 

 

 

Effects 

ACHIEVE 

A comparison of interns and matched (non-interns) showed these effects in 2016: 

 Grade Point Average (GPA)—small positive effect; GPAs decreased 

 Attendance—minimal 

 Being on track to graduate on time—small positive effect 

The comparison showed no effects on the following outcomes in 2016: 

 ACT Scores 

 MCA Math Scores 

 Likelihood to have behavioral referrals 

 Number of behavioral referrals for those who had them 

 Likelihood of removals from class 

 Likelihood of suspension from school 

 Number of suspensions for those who had them in the past 

DISCOVER 

A comparison of interns and matched (non-interns) for showed these effects in 

2016: 

 Attendance—minimal 

 Being on track to graduate on time—small positive effect 

 Likelihood of suspension from school—minimal (interns more likely than 

matched) 

The comparison showed no effects on the following outcomes in 2016: 

 Grade Point Average (GPA) 

 ACT Scores 

 MCA Math Scores 

 MCA Reading Scores 

 Likelihood to have behavioral referrals 

 Number of behavioral referrals for those who had them 

 Likelihood of removals from class 

 Number of suspensions for those who had them in the past 

Center for Applied Research and Educational Improvement, University of Minnesota 

77 
 

 

 

 

 

 

 

 

 

 

 

 

Second design, preliminary evidence: one-group pretest-posttest within participants 

design. 

School outcomes. Participants who completed internships in 2016 were compared with 

themselves on school outcomes. Because outcome data came from Minneapolis Public Schools 

(MPS), the analysis includes students who were enrolled in MPS in both 2015-16 and 2016-17. 

Outcomes examined in this analysis were GPA, attendance, behavior referrals, class removals, 

and suspensions. Because students do not take the same standardized tests every year, test scores 

could not be used for this analysis. In order to answer the evaluation questions, these analyses 

looked at Achieve and Discover students separately. It is important when interpreting these 

analyses that we not attribute causation to the program, whether the effects are positive or 

negative. Other factors (e.g., maturation, other experiences) may have contributed to the results. 

GPA: Achieve. Using a paired-samples t test, we compared the 2015-16 GPAs with the 

2016-17 GPAs for students who were Achieve interns in the summer of 2016. There was not a 

statistically significant difference between GPA in SY16 (N = 270, M = 2.90, SD = .75) and 

SY17, (N =270, M = 2.87, SD =.709); t(269) = .987, p = .325, suggesting that completing an 

Achieve internship had no meaningful effect on GPA for Achieve interns. 

GPA: Discover. Using a paired-samples t test, we compared the 2015-16 GPAs with the 

2016-17 GPAs for students who were Discover interns in the summer of 2016. There was a 

statistically significant difference between GPA in SY16 and SY17, with the SY 17 average (N 

=227, M = 2.79, SD = .052) lower than the SY16 average (N = 227, M = 2.85 SD = .796); t(226) = 

3.138, p = .002. The mean difference in GPA (.0611) decreased from one school year to the next. 

Cohen’s d = 0.078 suggested a minimal effect. It is possible that the difference can be attributed 

to the fact that these students moved to a higher grade-level and may have faced different 

standards for grading. 

Attendance: Achieve. Using a paired-samples t test, we compared the 2015-16 percentage 

of school attendance with the 2016-17 attendance for students who were Achieve interns in the 

summer of 2016. There was a statistically significant difference between percentage attendance in 

SY16 and SY17, with the SY17 average (N = 270, M = .93, SD = .08) lower than the SY16 

average N = 270, M = .95, SD = .07); t (269) = 3.94, p <.001. Cohen’s d =0 .27 suggested a small 

effect. It is noteworthy that these students attended school on average at rates at least 93% of the 

time. Without further research, we are not able to determine what is influencing decreased 

attendance from one year to the next. 

Center for Applied Research and Educational Improvement, University of Minnesota 

78 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Attendance: Discover. Using a paired-samples t test, we compared the 2015-16 

percentage of school attendance with the 2016-17 attendance for students who were Discover 

interns in the summer of 2016. There was a statistically significant difference between percentage 

attendance in SY16 and SY17, with the SY17 average (N = 227, M = .94, SD = .07) lower than 

the SY16 average (N = 227, M = .95, SD = .06); t (226) = 3.618, p < .001. Cohen’s d = 0.15 

suggested a small effect. It is noteworthy that these students attended school on average at rates at 

least 93% of the time. Without further research, we are not able to determine what is influencing 

decreased attendance from one year to the next. 

Behavior: referrals, removals, and suspensions. As shown in Figure 11, the data about 

Achieve and Discover interns show that the majority (and in some cases nearly all) of STEP-UP 

interns had no involvement with the disciplinary system in Minneapolis Public Schools. Even so, 

the percentage of Achieve interns with no referrals, removals or suspensions increased in SY17. 

Among Discover interns, there were fewer students with referrals and removals in SY17 

compared to SY16, but the percentage of students with suspensions increased. Figures 12, 13, and 

14 show comparisons on these outcomes between SY16 and SY17. Findings about changes in the 

number of referrals, removals, and suspensions should be considered in light of the larger finding 

that most STEP-UP interns had no disciplinary records. 

Figure 11. Involvement of STEP-UP Interns in the Minneapolis Public Schools Disciplinary System, 

SY17. 

Behavior referrals: Achieve. Ninety percent of Achieve interns had no behavior referrals 

in the school year prior to the internship (N = 270). Following the 2016 internship, 97% of 

Achieve interns (N = 270) had no referrals. In 2015-16, the number of referrals per student ranged 

from 1 to 23 with the mean = 3.27, median = 1, and mode = 1 for those students who had them. In 

2016-17 the number of referrals ranged from 1 to 5 with the mean = 1.5, median = 1, and mode = 

1 for those students who had them. What is of interest here is that for students who had referrals 

in SY16 (26), 88% fewer had referrals the year after the internship.45 

45 This does not account for the number of referrals, just the number of students who had them. There were 26 

students with referrals in SY16 and of those, only 3 had referrals in SY17. 

Center for Applied Research and Educational Improvement, University of Minnesota 

79 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Behavior referrals: Discover. Eighty-six percent of Discover interns had no behavior 

referrals in the school year prior to the internship (N = 227). Following the 2016 internship, 92% 

of Discover interns (N = 227) had no referrals. In 2015-16, the number of referrals per student 

ranged from 1 to 58 with the mean = 7.32, median = 1, and mode = 1 for those students who had 

them. In 2016-17, the number of referrals ranged from 1 to 4 with the mean = 1.6, median = 1 and 

mode = 1 for those students who had them. For students who had referrals in SY16, 74% fewer 

had referrals the year after the internship.46 

Removals from class: Achieve. The overwhelming majority of Achieve interns had no 

removals during SY16 (97%, N = 261) or SY17 (98%, N = 264). In 2015-16, the number of 

removals per student ranged from 1 to 4, with the mean = 1.4 and the median = 1 and the mode = 

1 for those students who had them. In 2016-17, the number of removals per student ranged from 1 

to 5 with the mean = 1.7, median = 1, and mode = 1 for those students who had them. For 

students who had removals in SY16 (N = 9), none had removals in SY17. Six students who did 

not have removals in SY16 had them in SY17. 

Removals from class: Discover. The vast majority of Discover interns had no removals in 

SY16 (91%, N = 207) or in SY17 (92%, N = 209). In 2015-16, the number of removals per 

student ranged from 1 to 3 with the mean = 1.3, median = 1 mode = 1 for those students who had 

them. In 2016-17, the number of students with removals ranged from 1 to 6 with the mean = 1.9, 

median = 1, and mode = 1 for those students who had them. For students who had removals in 

SY16, 65% fewer had referrals the year after the internship.47 Fourteen students who did not have 

removals in SY16 had them in SY17, and four had the same number of removals in both years. 

Suspensions: Achieve. Just as with removals, very few Achieve interns had suspensions 

in either SY16 (13%, N = 262) or in SY17 (3%, N = 262). In 2015-16, the number of suspensions 

per student ranged from 1 to 3 with the mean = 1.75, median = 1, and mode = 1 for those students 

who had them. In 2016-17, the number of suspensions ranged from 1 to 3 with the mean = 1.25, 

median = 1 and mode = 1 for those students who had them. For students who had suspensions in 

SY16 (n = 8), 7 had fewer suspensions the year after the internship and 3⁄4 of them had no 

suspensions the following year. Six students who had no suspensions in SY16 had them in SY17. 

It is difficult to draw conclusions given the small numbers of students with suspensions. 

Suspensions: Discover. More Discover interns had suspensions in SY17 (14%, N = 31) 

than in SY16 (8%, N = 19). In 2015-16, the number of suspensions per student ranged from 1 to 4 

46 This does not account for the number of referrals, just the number of students who had them. There were 31 

students with referrals in SY16 and of those, only 8 had referrals in SY17. 

47 This does not account for the number of removals, just the number of students who had them. There were 20 

students with removals in SY16 and of those, only 7 had removals in SY17. 

Center for Applied Research and Educational Improvement, University of Minnesota 

80 
 

 

 

 

 

with the mean = 1.5, median = 1, and mode =1 for those students who had them. In 2016-17, the 

number of suspensions ranged from 1 to 3 with the mean = 1.2 median = 1, and mode =1 for those 

students who had them. Twelve students had fewer suspensions in SY17 than in SY16, six had 

the same number, and 21 who had no suspensions in SY16 had them in SY17. For students who 

had suspensions in SY16, 47% fewer had suspensions the year after the internship. 48 

Figure 12: Percentages of Interns with NO Referrals in S16 and SY17. 

Figure 13: Percentages of Interns with NO Removals in SY16 and SY17. 

48 This does not account for the number of suspensions, just the number of students who had them. There were 19 

students with referrals in SY16 and of those, 10 had suspensions in SY17. 

Center for Applied Research and Educational Improvement, University of Minnesota 

81 
 

 

 

 

 

 

Figure 14: Percentages of Interns with NO Suspensions in SY16 and SY17. 

Table 14 presents comparisons of interns between SY16 and SY17 for each of the school 

outcomes. 

Table 14 

Comparison of STEP-UP Interns’ Outcomes Pre (School Year 2015-16) and Post (School Year 2016-17) 

Group N M SD t df p d 

ACHIEVE Grade Point 

Average: GPA 

Pre 270 2.90 .75 .987 269 .325 X Post 270 2.89 .709 

Attendance Pre 270 .95 .07 3.94 269 <.001 0.27 Post 270 .93 .08 

DISCOVER Grade Point 

Average: GPA 

Pre 227 2.85 .796 3.138 226 .002 0.078 Post 227 2.79 .052 

Attendance Pre 227 .95 .06 3.618 226 <.001 0.15 Post 227 .94 .07 

Employability outcomes. Employability outcomes included attitude, occupational or 

professional knowledge, future orientation, 21st Century skills such as teamwork and 

communication, and professional networks. Data were taken from surveys completed by students 

before training and in the end-of-summer survey. The difference scores were symmetrically 

distributed as assessed by a histogram. A Wilcoxon signed-rank test found statistically significant 

differences in median scores from pre- to post using the Wilcoxon signed-rank test on 7 items 

among Achieve interns and on 7 items among Discover interns. Of these, the two items about 

interview preparedness and the one about naming people to ask for references had statistically 

significant differences in median scores for both Achieve and Discover interns. A table of 

analyses by question, including effect sizes, can be found in Appendix O. 

Center for Applied Research and Educational Improvement, University of Minnesota 

82 
 

 

 

 

 

 

 

 

 

 

 

 

Findings. For Achieve interns there were small effects for the following from pre- to post: 

 Staying calm under stress 

 Accepting criticism openly 

 Actively looking for additional tasks when my own work is done 

 Breaking problems into smaller parts to solve them49 

 Knowing what questions to expect in a job interview 

 Knowing what questions to ask in a job interview 

 Naming two or more people to ask for references50 

For Discover interns there were small effects for the following from pre- to post: 

 Knowing what clothes to wear to work 

 Oral communication in a professional setting 

 Written communication in a professional setting 

 Describing skills and strengths on a professional resume51 

 Knowing what questions are appropriate to ask during a job interview 

 Naming two or more people to ask for references 

For Discover interns there was a medium effect on the following from pre- to post: 

 Knowing what questions to expect in a job interview 

The analyses of the employability items showed a few items with higher scores prior to training 

than scores after the internship on the Wilcoxon signed rank test. For Achieve level students, two 

items were related to future orientation (“I intend to continue my education following high 

school” and “I am hopeful about my future”) and one was about perceptions of self (“I bring 

energy and enthusiasm to tasks”). For Discover, students scored higher on items about future 

orientation (“I am hopeful about my future”) and perceptions of self (“I bring energy and 

enthusiasm to tasks”) prior to training. For those items we looked at the frequencies of each of 

the responses. In all cases, the proportion of respondents who selected “agree or strongly agree” 

were high both before and after. This suggests that there was not much room for change over the 

course of the training and internship; they had already reached a ceiling. See Appendix O. 

49 After correcting for multiple comparisons, there were not statistically significant differences pre to post on this
 

item. See Appendix O for details. 

50 After correcting for multiple comparisons, there were not statistically significant differences pre to post on this
 

item. See Appendix O for details.
 51 After correcting for multiple comparisons, there were not statistically significant differences pre to post on this
 

item. See Appendix O for details.
 

Center for Applied Research and Educational Improvement, University of Minnesota 

83 
 

 

 

 

 

 

 

 

 

 

 

 

 

Exploratory Questions on Different Experiences & School Outcomes: Number of Hours. 

Number of hours per week52 worked and GPA changes from SY16 to SY17. Using a oneway ANOVA, we compared the differences in GPA in SY17 and SY16 for 2016 Achieve interns 

grouped by the range of hours per week they worked. There were no statistically significant 

differences in GPA change among the groups of interns who worked different numbers of hours, 

F (5, 148) = 1.159, p =.332. Given that there were outliers and the normality assumption was 

violated for one of the groups, a nonparametric test (Kruskal-Wallis) was also run. There were not 

significant differences between interns who worked different numbers of hours and change in 

GPA, χ2(5) = 3.461, p = .629. This analysis indicates that the number of hours worked per week 

was not related to a change in GPA from the school year before to the school year after the 

internship for Achieve interns. 

Number of hours per week worked and ACT scores. ACT scores were available for 63 

students in our data set. Using a one-way ANOVA, we compared ACT scores for interns grouped 

by the range of hours per week they worked. There were not statistically significant differences in 

ACT scores among the groups of interns who worked different numbers of hours, 

F (5, 57) = .914, p = .479. Given that there were outliers and the normality assumption was 

violated for one of the groups, a nonparametric test (Kruskal-Wallis) was also run. There were not 

significant differences in ACT scored based on number of hours worked χ2

(5) = 5.119, p = .402. 

Number of hours per week worked and MCA Math scores. MCA Math scores were 

available for 18 interns in our data set. Using a one-way ANOVA, we compared MCA math 

scores for interns grouped by the range of hours per week they worked. There were no statistically 

significant differences in MCA Math scores among the groups of interns who worked different 

numbers of hours, F (5, 12) = .663, p = .658. Because of small number of interns in some of the 

groups, a nonparametric test (Kruskal-Wallis) was also run. There were not significant differences 

among the groups, χ2

(5) = 3.608, p = .607. We should be cautious about drawing conclusions 

about this outcome given the small number of cases for which we had data and the number in 

each group. 

Number of hours per week and change in rate of attendance. Using a one-way ANOVA, 

we compared the differences in rate of attendance in SY17 and SY16 for 2016 Achieve interns 

grouped by the range of hours per week they worked. There were no statistically significant 

differences in rate of attendance among the groups of interns who worked different numbers of 

hours, F (5, 148) = .400, p =.848. Given that there were outliers and the normality assumption 

was violated for several of the groups, a nonparametric test (Kruskal-Wallis) was also run. There 

were not significant differences between groups for difference in the rate of attendance between 

52 Groups were less than 10 hours, 10-15 hours, 16-25 hours, 26-34 hours, 35-40 hours, more than 40 hours. 

Center for Applied Research and Educational Improvement, University of Minnesota 

84 
 

 

 

 

 

 

 

 

SY16 and SY17, χ2(5) = 3.367, p = .644. This analysis indicates that the number of hours per 

week worked was not related to a change in rate of attendance from the school year before to the 

school year after the internship for Achieve interns. 

These analyses indicate that the number of hours per week worked was not related to changes in 

changes in GPA or rate of attendance from one school year to the next. Number of hours per week 

work was also not related to differences in ACT scores or MCA math scores for Achieve interns. 

Table 15 presents a summary of the analyses of school outcomes for interns who worked different 

numbers of hours per week. 

Table 15 

Analyses of School Outcomes for Interns Who Worked Different Numbers of Hours per Week 

HOURS PER WEEK 

WORKED 

Group N Test 

Statistic 

df p Test 

Statistic 

df p Page 

in 

report ANOVA Nonparametric: KruskalWallis 

Change in Grade 

Point Average: 

GPA 

<10 9 

1.159 5,148 .332 3.461 5 .629 p. 83 

10-15 28 

16-25 38 

26-34 25 

35-40 48 

> 40 6 

ACT Scores <10 1 

.914 5,57 .479 5.119 5 .402 p. 83 

10-15 11 

16-25 12 

26-34 12 

35-40 23 

> 40 4 

MCA Math Scores <10 1 

.663 5,12 .658 3.608 5 .607 p. 83 

10-15 2 

16-25 3 

26-34 2 

35-40 8 

> 40 2 

Center for Applied Research and Educational Improvement, University of Minnesota 

85 
 

 

 

 

 

 

 

 

HOURS PER WEEK 

WORKED 

Group N Test 

Statistic 

df p Test 

Statistic 

df p Page 

in 

report 

Change in <10 9 

Percentage 

Attendance 

10-15 28 

16-25 38 

.400 5,148 .848 3.367 5 .644 p. 83 

26-34 25 

35-40 48 

> 40 6 

Exploratory Questions on Different Experiences & School Outcomes: Number of Weeks. 

Number of weeks53 worked and GPA changes from SY16 to SY17. Using a one-way 

ANOVA, we compared the differences in GPA in SY17 and SY16 for 2016 Achieve interns 

grouped by the number of weeks they worked. There were no statistically significant differences 

in GPA change among the groups of interns who worked different numbers of weeks, F (5, 146) = 

.262, p =.968. Given that there were outliers and the normality assumption was violated for one 

of the groups, a nonparametric test (Kruskal-Wallis) was also run. There were not significant 

differences between groups of number of hours worked and change in GPA, χ2(7) = 1.797, p = 

.970. This analysis indicates that the number of weeks worked was not related to a change in GPA 

from the school year before to the school year after the internship for Achieve interns. 

Number of weeks worked and ACT scores. ACT scores were available for 63 students in 

our data set. Using a one-way ANOVA, we compared ACT scores for interns grouped by the 

number of weeks they worked they worked. There were no statistically significant differences in 

ACT scores among the groups of interns who worked different numbers of weeks, Welch’s F (6, 

9.081) = 1.10, p = .430. 

Number of weeks worked and MCA Math scores. MCA Math scores were available for 

18 interns in our data set. Using a one-way ANOVA, we compared MCA math scores for interns 

grouped by the number of weeks they worked. There were no statistically significant differences 

in MCA Math scores among the groups of interns who worked different numbers of hours, F 

(4,13) = 1.321, p = .313. Because of small number of interns in some of the groups, a 

53 Groups were 5 weeks, 6 weeks, 7 weeks, 8 weeks, 9 weeks, 10 weeks and more than 10 weeks. 

Center for Applied Research and Educational Improvement, University of Minnesota 

86 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

nonparametric test (Kruskal-Wallis) was also run. There were not significant differences among 

the groups, χ2

(4) = 4.368, p = .358. We should be cautious about drawing conclusions about this 

outcome given the small number of cases for which we had data and the number in each group. 

Number of weeks worked and change in rate of attendance. Using a one-way ANOVA, 

we compared the differences in rate of attendance in SY17 and SY16 for 2016 Achieve interns 

grouped by the range of weeks they worked. There were no statistically significant differences in 

rate of attendance among the groups of interns who worked different numbers of weeks, F (7, 

146) = .429, p =.883. Given that there were outliers and the normality assumption was violated 

for several of the groups, a nonparametric test (Kruskal-Wallis) was also run. There were not 

significant differences between groups for difference in the rate of attendance between SY16 and 

SY17, χ2

(7) = 6.136, p = .524. This analysis indicates that the number of weeks worked was not 

related to a change in rate of attendance from the school year before to the school year after the 

internship for Achieve interns. 

These analyses indicate that the number of weeks worked was not related to changes in GPA or 

rate of attendance from one school year to the next. The number of weeks worked was also 

unrelated to either ACT scores or MCA math scores for Achieve interns. Table 16 presents a 

summary of the analyses of school outcomes for interns who worked different numbers of weeks. 

Table 16 

Analyses of School Outcomes for Interns that Worked Different Numbers of Weeks 

WEEKS WORKED Group N Test 

Statistic 

df p Test 

Statistic 

df p Page in 

report 

ANOVA Nonparametric: Kruskal-Wallis 

Change in Grade 

Point Average: 

GPA 

4 5 

.262 5,146 .968 1.797 7 .970 p. 85 

5 4 

6 19 

7 11 

8 20 

9 65 

10 17 

>10 13 

ACT Scores 4 0 

1.10 6,9.0 

81 .430 p. 85 

5 2 

6 5 

7 6 

8 6 

9 29 

Center for Applied Research and Educational Improvement, University of Minnesota 

87 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

WEEKS WORKED Group N Test 

Statistic 

df p Test 

Statistic 

df p Page in 

report 

ANOVA Nonparametric: Kruskal-Wallis 

10 10 

>10 5 

MCA Math 

Scores 

4 0 

1.321 4,13 .313 4.368 4 .358 p. 85 

5 0 

6 2 

7 0 

8 2 

9 9 

10 4 

>10 1 

Change in 

Percentage 

Attendance 

4 5 

.429 7,146 .883 6.136 7 .524 p. 86 

5 4 

6 19 

7 11 

8 20 

9 65 

10 17 

>10 13 

Exploratory questions on different experiences & school outcomes: number of internships. 

Single or multiple STEP-UP internship54s and GPA. An independent-samples t test was 

conducted to compare differences in GPA between SY16 and SY17 for STEP-UP Achieve interns 

who completed an internship in 2016 only and interns with at least one other summer STEP-UP 

internship. For Achieve, there was not a significant difference in the scores for single year interns 

(N = 137, M = -.0113, SD = .156), and multiple year interns (N = 97, M = .0047, SD = .164), 

t(232) = -.759, p = .449. Because of violations of test assumptions, a nonparametric test (Mann 

Whitney) was also run and there were not significant differences between the two groups on 

differences in GPA, U = 7,007, z = .711, p = .477. 

For Discover interns there was not a significant difference in the scores for single year interns (N 

= 153, M = -.0813 SD = .250), and multiple year interns (N = 33, M = -.0821, SD = .278), t(184) = 

.071, p = .987. Because of violations of test assumptions, a nonparametric test (Mann Whitney) 

54 Groups were based on whether a student had an internship only in 2016 or in 2016 plus 2014 and/or 2015. 

Center for Applied Research and Educational Improvement, University of Minnesota 

88 
 

 

 

 

 

 

 

 

 

 

 

was also run and there were not significant differences between the two groups on differences in 

GPA, U = 3.224, z = .494, p = .621. 

This analysis indicates that doing more than one STEP-UP internship was not related to a change 

in GPA for Achieve or Discover interns. 

Single or multiple STEP-UP internships and ACT scores. An independent-samples t test 

was conducted to compare ACT test scores of STEP-UP Achieve interns who completed an 

internship in 2016 only and interns with at least one other summer STEP-UP internship. For 

Achieve, there was not a significant difference in the scores for single year interns (N = 61, M = 

17.05, SD = 5.503), and multiple year interns (N = 40, M = 16.95, SD = 4.489), t(99) = .095, p = 

.924. Because of violations of test assumptions, a nonparametric test (Mann Whitney) was also 

run and there were not significant differences between the two groups on ACT scores, U = 

1,248.5, z = .199, p = .842. 

An independent-samples t test was conducted to compare ACT test scores of STEP-UP interns 

who completed a Discover internship in 2016 only and interns with at least one other summer 

STEP-UP internship. For Discover there was not a significant difference in the scores for single 

year interns (N = 17, M = 15.65, SD = 5.408), and multiple year interns (N = 5, M = 15.80, SD = 

3.033), t(20) = -.060, p = .953. Because of violations of test assumptions, a nonparametric test 

(Mann Whitney) was also run and there were not significant differences between the two groups 

on ACT scores, U = 49.0, z = .514, p = .649. At the Discover level, we had a small number of 

cases and results should be interpreted with caution. These results suggest doing more than one 

STEP-UP internship did not influence ACT scores. 

Single or multiple STEP-UP internships and MCA Math scores. An independentsamples t test was conducted to compare MCA math test scores of STEP-UP interns who 

completed an internship in 2016 only and interns with at least one other summer STEP-UP 

internship. For Achieve there was not a significant difference in the scores for single year interns 

(N = 20, M =1131.75, SD = 16.157, and multiple year interns (N = 8, M = 1125.50, SD = 9.15), 

t(26) = 1.023, p = .316. At the Discover level we did not have data for enough students to run the 

analysis. These results suggest doing more than one STEP-UP internship did not influence MCA 

Math scores. Caution should be taken, though, in interpreting these results given the small number 

of multiple year interns. 

Single or multiple STEP-UP internships and MCA Reading scores. An independentsamples t test was conducted to compare MCA reading test scores of STEP-UP interns who 

completed an internship in 2016 only and interns with at least one other summer STEP-UP 

internship. For Discover there was not a significant difference in the scores for single year interns 

Center for Applied Research and Educational Improvement, University of Minnesota 

89 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

(N = 49, M =1038.06, SD = 18.1 and multiple year interns (N = 5, M = 1038.6, SD = 15.978), 

t(52) = -.064, p = .949. At the Achieve level we did not have data for enough students to run the 

analysis. These results suggest doing more than one STEP-UP internship did not influence MCA 

Reading scores. Caution should be taken, though, in interpreting these results given the small 

number of multiple year interns. 

Single or multiple STEP-UP internships and change in attendance. An independentsamples t test was conducted to compare differences in attendance rates between SY16 and SY17 

for STEP-UP Achieve interns who completed an internship in 2016 only and interns with at least 

one other summer STEP-UP internship. For Achieve, there was not a significant difference in the 

scores for single year interns (N = 137, M = -.0138, SD = .057), and multiple year interns (N = 97, 

M = -.02, SD = .055), t(232) = .836, p = .404. Because of violations of test assumptions, a 

nonparametric test (Mann Whitney) was also run and there were not significant differences 

between the two groups on differences in GPA, U = 5,980, z = -1.303, p = .193. 

For Discover interns there was not a significant difference in the scores for single year interns (N 

= 153, M = -.0158 SD = .059), and multiple year interns (N = 33, M = -.0218, SD = .059), t(184) = 

.536, p = .593. Because of violations of test assumptions, a nonparametric test (Mann Whitney) 

was also run and there were not significant differences between the two groups on differences in 

GPA, U = 2,316.5, z = -.742, p = .458. This analysis indicates that doing an internship more than 

once was not related to a change in rate of attendance from the school year before to the school 

year after the internship for Achieve or Discover interns. 

Overall, the analyses indicate that doing an internship for more than one summer was not related 

to changes in GPA or rate of attendance from the school year before to the school year after the 

internship, ACT scores, or MCA scores for Achieve or Discover interns. Results for all school 

outcomes are presented in Tables 17 and 18. 

Table 17 

Analyses of School Outcomes for Achieve Interns Who Had 1 Internship or More Than 1 Internship 

ACHIEVE Group: 

Years of 

internships 

N Test 

Statistic 

df p Test 

Statistic 

df p Page in 

report 

Independent samples t test Mann Whitney test 

GPA 1 137 -.759 232 .449 .711 .477 p. 87 >1 97 

ACT 1 61 .095 99 .924 .199 .842 p. 88 >1 40 

MCA Math 1 20 1.023 26 .316 p. 88 >1 8 

Change in % of 1 137 .836 232 .404 -1.303 .193 p. 89 

Center for Applied Research and Educational Improvement, University of Minnesota 

90 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Attendance >1 97 

Table 18 

Analyses of School Outcomes for Discover Interns Who Had 1 Internship or More Than 1 Internship 

DISCOVER Group: 

Years of 

internships 

N Test 

Statistic 

df p Test 

Statistic 

df p Page in 

report 

Independent samples t test Mann Whitney test 

GPA 1 153 .071 184 .987 .494 .621 p. 83 >1 33 

ACT 1 17 -.060 20 .953 U = 49 

z =.514 .649 p. 83 >1 5 

MCA Reading 1 49 -.064 52 .949 p. 84 >1 5 

Change in % of 

Attendance 

1 153 .536 184 .593 -.742 .458 p. 84 >1 33 

Exploratory questions on different experiences & employability outcomes: number of hours. 

A Kruskal-Wallis H test was run to determine if changes in employability self-ratings (decrease, 

same, increase) differed among six groups of participants who worked different numbers of hours 

per week during the summer. There was one employability item (“I get my work done on time”) 

with statistically significant differences in mean ranks55 of changes in self-ratings, χ2(5) = 12.231, 

p = .032. The results for that employability item are shown in Table 19 for groups that worked 

different numbers of hours. We compared the groups that had at least 10 people and found the 

group with the largest proportion reporting an increase pre- to post on this item worked 16-25 

hours/week during the summer. It is difficult to say why that would be the case. 

Table 19 

Analysis of Employability Item with Statistical Significance for Interns that Worked Different Number of 

Hours per Week 

Item Group: 

Number 

of Hours 

N Mean 

Rank 

Test 

Statistic 

df p 

Q12. I get my work done on <10 8 66.50 

12.231 5 .032 

time. 10-15 22 63.84 

16-25 36 80.50 

26-34 24 62.08 

55 Distribution of scores was not similar for all groups as assessed by visual inspection of a boxplot. 

Center for Applied Research and Educational Improvement, University of Minnesota 

91 
 

 

 

 

 

 

 

 

 

 

 

 

 

35-40 35 52.57 

> 40 5 70.10 

Exploratory questions on different experiences & employability outcomes: number of 

weeks. A Kruskal-Wallis H test was run to determine if changes in employability self-ratings 

(decrease, same, increase) among eight groups of participants who worked different numbers of 

weeks during the summer (4, 5, 6, 7, 8, 9, 10, more than 10) differed. There was one 

employability item (“I know the education required for the work I am interested in doing”) where 

the mean ranks56 of changes in self-ratings was statistically significant, χ2(7) = 14.926, p = .037. 

The results are shown in Table 20 shows results for that employability item for groups of 

participants who worked different numbers of weeks. We compared the groups that had at least 

10 people and found the group with the largest proportion reporting an increase pre- to post on 

this item worked 8 weeks during the summer. It is difficult to say why that would be the case. 

Table 20 

Analysis of Employability Item with Statistical Significance for Interns Who Worked Different Number of 

Weeks 

Item Group: 

Number of 

Weeks 

N Test 

Statistic 

df p 

Q25. I know the education required for 

the work I am interested in doing. 

4 5 

14.926 7 .037 

5 3 

6 16 

7 6 

8 17 

9 55 

10 10 

>10 12 

Exploratory questions on different experiences & employability outcomes: number of 

internships. Two sets of Chi square tests (one for Achieve, one for Discover) were run to 

determine if changes in employability self-ratings (decrease, same, increase) between two groups 

of participants who had either one internship in 2016 or an internship in 2016 plus one in 2014 

and/or 2015 differed. 

For Achieve interns, there were statistically significant associations between number of 

internships (1, more than 1) and changes in employability self-ratings for four items. Results are 

summarized in Table 21. 

Distribution of scores was not similar for all groups as assessed by visual inspection of a boxplot. 

Center for Applied Research and Educational Improvement, University of Minnesota 

92 

56 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

A larger proportion of the multiple internships group (27% vs. 8%) reported an increase pre- to 

post on actively looking for additional tasks after completing their own work (χ2

(2) = 8.676, p = 

.013). Of the one-year interns, 60% (57) decreased their self-rating on this item from pre-to post. 

It is possible to imagine that after the first STEP-UP internship experience, interns got more 

realistic or honest about their own tendencies, or the internship gave them context in which to test 

themselves. It is possible that with added experience, interns felt more comfortable or placed 

greater value on taking the initiative to do more than assigned. See Appendix P for details. 

A larger proportion of the multiple internship group (33% vs. 21%) reported an increase pre- to 

post on thinking about several possible solutions to a problem (χ2

(2) =6.6465, p = .039). Close to 

half of both groups decreased their self-ratings pre- to post. Any internship experience may have 

helped youth assess themselves more realistically. It is also possible that with increasing 

experience, interns learned additional problem-solving skills. 

A larger proportion of the multiple internship group (27% vs. 17%) reported an increase pre- to 

post on thinking about arriving on time (χ2

(2) = 7.693, p = .021). More than half of both groups 

decreased their self-ratings pre- to post. Any internship experience may have helped youth assess 

themselves more realistically. It is also possible that with increasing experience, interns improved 

their promptness. 

A larger proportion of the multiple internship group (27% vs. 18%) reported an increase pre- to 

post on knowing what questions to expect in a job interview (χ2(2) = 6.20, p = .045). It is possible 

that with more experience with interviews interns become more familiar with the interview 

process and what to expect from it. 

Table 21 

Analysis of Employability Items with Statistical Significance for Achieve Interns that Had 1 Internship or 

More Than 1 Internship 

Item Group: 

Internships 

N χ

2 df p 

Q6. I actively look for additional tasks 

when my own work in done. 

1 95 8.676 2 .013 >1 44 

Q9. I think of several possible solutions 

to a problem. 

1 95 6.465 2 .039 >1 42 

Q13. I arrive on time. 1 94 7.693 2 .021 >1 44 

Q19 I know what questions to expect in 

a job interview. 

1 95 6.200 2 .045 >1 44 

Center for Applied Research and Educational Improvement, University of Minnesota 

93 
 

 

 

 

 

 

 

 

Table 22 

Analysis of Employability Items with Statistical Significance for Discover Interns Who Had 1 Internship 

or More Than 1 Internship. 

Item Group: 

Internships 

N χ

2 df p 

Q19. I know what questions to expect 1 96 7.182* 2 .028 in a job interview. >1 16 

Q20. I know what questions are 1 97 

appropriate for me to ask during a job 10.866* 2 .004 

interview. 

>1 16 

*One cell had expected count less than 5 

For Discover interns, there were statistically significant associations between number of 

internships (1, more than 1) and changes in employability self-ratings for two items. Caution 

should be taken in interpreting these results given the small size of the group with more than one 

internship (N = 16). Results are summarized in Table 22. 

A larger proportion of the single STEP-UP internship group (54% vs. 19%) reported an increase 

pre- to post on knowing what questions to expect in a job interview (χ2

(2) = 7.182, p = .028). It is 

possible that the first-time interns are exposed to training about interviews, it is new information 

and they learn a great deal; subsequent exposure may add to their knowledge but not to the same 

extent. This finding runs counter to the finding among Achieve interns. 

A larger proportion of the single internship group (41% vs. 6%) reported an increase pre- to post 

on knowing what questions are appropriate to ask during a job interview. (χ2(2) = 10.866, 

p = .004). It is possible that the first-time interns are exposed to training about interviews, it is 

new information and they learn a great deal; subsequent exposure may add to their knowledge but 

not to the same extent. 

Center for Applied Research and Educational Improvement, University of Minnesota 

94 
 

 

 

 

 

 

 

Summary of Findings
 

The following sections are a summary of the findings for each evaluation question addressed in 

the study. 

Impact Evaluation Questions: Confirmatory 

1. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes than comparison group 

students?
 

STEP-UP Achieve participants had better GPAs than comparison group students even when 

controlling for differences the year prior to the internship, but both intern and matched 

student GPA decreased from one school year to the next. The effect size was small. They 

were more likely to be on-track to graduate on time (completing sufficient credits each year) 

even when controlling for prior achievement. Participants were slightly more likely to have 

a better rate of attendance in SY17 than comparison students (minimal effect size), but 

overall attendance rates dropped for both groups from one year to the next. Results should 

be interpreted in light of attendance rates for both groups in both years being at 91% or 

above. There were no effects on test scores (GPA, MCA Math). There were no effects on 

behavior outcomes even when controlling for prior achievement; results need to be 

interpreted in light of the small proportions of participants and matched students who had 

any behavioral incidents—referrals, removals, or suspensions. 

These results are similar to, but not the same as, those reported in the 2015 interim impact 

report. That year there were differences only in participants’ likelihood to be on track to 

graduate compared to matched students. There were no differences for GPA or attendance. 

b. Do the Discover participants achieve better school outcomes than comparison group 

students?
 

STEP-UP Discover participants were more likely to be on track to graduate than matched 

students. Participants were slightly more likely to have a better rate of attendance in SY17 

than comparison students (minimal effect size), but overall attendance rates dropped for 

both groups. Results should be interpreted in light of attendance rates for both groups in 

both years being at 92% or above. For Discover students, there were no effects on GPA, 

ACT scores, MCA Math scores, or MCA reading scores. With the exception of likelihood of 

suspension, there were no effects on behavior outcomes; results need to be interpreted in 

Center for Applied Research and Educational Improvement, University of Minnesota 

95 
 

 

light of the small proportions of participants and matched students who had any behavior 

incidents. Discover interns were more likely to have suspensions, but the effect size was 

minimal. 

These results are similar to, but not the same as, those found in 2015. That year there were 

no differences in any school outcomes for the Discover level. 

c. Do STEP-UP participants have higher postsecondary enrollment than the comparison group? 

There were no differences in postsecondary enrollment rates between participants and 

comparison students for the 2014 and 2015 cohorts. 

Impact Evaluation Questions: Exploratory 

1. Does the STEP-UP summer jobs program improve school outcomes? 

a. Do the Achieve participants achieve better school outcomes pre- to post-program? 

A comparison in Achieve interns’ outcomes before and after the 2016 internship showed no 

effect on GPA and a small negative effect on attendance. Further research is needed to 

determine why attendance went down from one year to the next, but the average both years 

was over 91%. Results were similar to those of 2015 when there were minimal negative 

changes in GPA and attendance. GPA may have gone down as a result of increasing 

demands in higher grades. 

The majority (and in some categories nearly all) of STEP-UP Achieve interns had no 

involvement in the disciplinary system in Minneapolis Public Schools. Even so, the 

percentage of Achieve interns with no referrals, removals, or suspensions increased in SY17 

compared to SY16. The results for behavior outcomes are similar to those in 2015. 

b. Do the Discover students achieve better school outcomes pre- to post-program? 

A comparison of Discover interns’ outcomes before and after the 2016 internship showed a 

minimal negative effect on GPA and a small negative effect on attendance. Without further 

research it is impossible to say why attendance went down from one year to the next, but the 

average both years was over 94%. GPA may have gone down as a result of increasing 

demands in higher grades. Results were similar to 2015, but the effect size for GPA was 

slightly larger that year. 

The majority of Discover interns had no involvement with the disciplinary system in 

Minneapolis Public Schools. Among Discover interns, there were fewer students with 

Center for Applied Research and Educational Improvement, University of Minnesota 

96 
 

 

 

 

 

 

 

 

 

 

 

referrals and removals in SY17 compared to SY16, but the percentage of students with 

suspensions increased. With the exception of suspension results, the results for behavior 

outcomes in 2016 are similar to those for 2015. 

2.	 Does the STEP-UP summer jobs program improve employability outcomes? 

In 2016, both Achieve and Discover interns improved in job-related skills: knowing what 

questions to ask in an interview, what questions to expect in an interview, and identifying 

people to use as professional references.57 These outcomes were consistent with the results of 

2015. 

a.	 Do the Achieve participants achieve better employability outcomes pre- to post-program? 

In addition to the skills mentioned above, in 2016, Achieve interns showed positive 

changes in their self-assessments of other personal skills: staying calm under stress, 

accepting criticism, looking for additional tasks when work is complete, and breaking 

problems into small parts to solve them.58 We were not able to compare their ratings with 

those of their supervisors to see if those perceptions were aligned. 

b. Do the Discover participants achieve better employability outcomes pre- to post-program? 

In addition to the skills mentioned above, in 2016, Discover interns showed positive 

changes in their self-assessments of other employment skills: knowing what clothes to wear 

to work, oral and written communication in a professional setting, and creating a 

professional resume.59 

3.	 Do differential experiences in the STEP-UP summer job program – internship length or 

longitudinal experience– lead to differential outcomes on school and employability measures? 

a.	 Do students who work longer work weeks and/or more weeks in the program have better 

outcomes? 

The number of hours per week worked was not related to changes in GPA or rate of 

attendance from the school year before to the school year after the internship, ACT scores, 

or MCA math scores for Achieve interns. 

57 After correcting for multiple comparisons, the difference on naming references was not statistically significant 

from pre to post.
 58 The difference on breaking problems into small parts to solve them was not statistically significant after correcting
 

for multiple comparisons.
 59 After correcting for multiple comparisons there was not a statistically significant difference pre to post on resume 

writing.
 

Center for Applied Research and Educational Improvement, University of Minnesota 

97 
 

 

 

 

 

Interns who worked more hours did not have better employability outcomes. 

The number of weeks worked was not related to changes in GPA or rate of attendance from 

the school year before to the school year after the internship, ACT scores, or MCA math 

scores for Achieve interns. 

Interns who worked more weeks did not have better employability outcomes. 

b. Do students who work for STEP-UP multiple years in a row have better outcomes? 

Doing an internship for more than one summer was not related to changes in GPA or rate 

of attendance from the school year before to the school year after the internship, ACT 

scores, or MCA scores for Achieve or Discover interns. 

Achieve interns with more than one STEP-UP internship experience had better outcomes 

on: 

 Looking for additional tasks after completing assigned work 

 Thinking of multiple solutions to a problem 

 Arriving on time 

 Knowing what questions to expect in a job interview 

Discover interns with more than one STEP-UP internship experience had better outcomes 

on: 

 Knowing what questions to expect in a job interview 

 Knowing what questions to ask during a job interview 

Center for Applied Research and Educational Improvement, University of Minnesota 

98 
 

 

 

Interpretations and Limitations 

Confirmatory Study Results Targeting Moderate Level of Evidence: School Outcomes 

Focusing on 2016 participants and a comparison group, this confirmatory impact study sought 

moderate evidence to answer the question, “Does the STEP-UP summer jobs program improve 

school outcomes?” The study yielded results on school outcomes similar to those of previous 

years—there were limited positive effects of STEP-UP on school outcomes. 

Summary of findings. At the Achieve level, there were no effects on ACT or MCA scores, 

the likelihood or number of behavioral referrals, the likelihood of removals, the likelihood or 

number of suspensions even when controlling for prior achievement. Attendance rates for both 

groups also declined, but interns’ attendance rates were marginally better. Grade point average 

for both interns and the comparison group declined on average, but interns had higher GPAs 

even when controlling for differences the prior year. Achieve interns were more likely than 

matched students to have the credits needed to graduate in four years even when controlling for 

prior achievement. These results may indicate that stronger students applied to and participated 

in STEP-UP at the Achieve level. 

At the Discover level, we also saw limited improvements in school outcomes in the confirmatory 

study. There were no effects of the STEP-UP program on GPA, ACT scores, MCA scores, 

likelihood or number of behavioral referrals, the likelihood of removals from class, or the 

number of suspensions. Attendance rates for both interns and matched students declined, but 

interns’ rates of attendance were marginally better. Discover interns were actually more likely 

than their peers to be suspended. The results were the same even when we controlled for prior 

achievement. 

Results for both Achieve and Discover were similar to but not exactly the same as those found in 

2015. That year there were differences only in Achieve participants’ likelihood to be on track to 

graduate compared to matched students. On post-secondary enrollment, there were no 

differences between participants and matched students in the 2014 nor the 2015 cohorts. 

Interpretation of findings. One possible explanation for these results is that STEP-UP 

does not specifically target school-related skills in its trainings or internships. Using school 

outcomes, then, may be neither a desirable nor a realistic way of measuring the impact of the 

program on its participants. If changes in school-related metrics were the goal of the program, 

these school outcomes would need to be targeted more explicitly through programming. 

Limitations: School outcomes. The comparisons showed small negative effects on GPA 

and attendance. Without further research it is impossible to say why these decreased from one 

Center for Applied Research and Educational Improvement, University of Minnesota 

99 
 

 

 

 

 

 

 

 

 

 

 

 

year to the next, but the changes could be related to school transitions and increasing difficulty of 

coursework in higher grades. Outcomes on attendance should be interpreted in light of the larger 

finding that rates of attendance for both interns and comparison students before and after the 

internship summer were above 90%. The data about Achieve and Discover interns and matched 

students show that the majority (and in some cases nearly all) had no involvement with the 

disciplinary system in Minneapolis Public Schools. The analyses about likelihood of having 

referrals, removals, and suspensions and those about changes in the number of referrals, 

removals, and suspensions were performed on small subsets of each of the groups. Some 

analyses could not be conducted due to the small sample size. While some results were 

promising, the small number of cases relative to the entire group of interns calls for caution in 

interpreting them and must be interpreted in light of the larger finding that most STEP-UP 

interns and their matches did not have disciplinary incidents. 

There are limitations to the generalizability of these findings. Many interns in the program were 

not enrolled in Minneapolis Public Schools or were not enrolled both before and after the 

internship. Since we only had school data for MPS students, results should not be generalized to 

program participants in other school districts nor to other youth employment programs. 

Exploratory Study Results Targeting Preliminary Level of Evidence: School Outcomes 

Summary of findings. The exploratory impact study sought preliminary evidence about 

whether interns achieved better school outcomes pre- to post-program. In both 2015 and 2016 at 

the Achieve and Discover levels, interns did not achieve better GPAs nor attendance rates; in 

fact, students declined on these measures. In the area of behavior outcomes, as shown in Table 

23, most interns were not involved with the disciplinary system either before or after the 

internship, though in almost all categories a higher percentage avoided the disciplinary system 

the year following the internship than the year before. Of those students who had behavior 

incidents in the year prior to the internship, many had none the year after.60 In contrast, there 

were students who did not have incidents in the year prior to the internship and did have them in 

the year after. The exploratory analyses examining differences for interns who had an internship 

more than one year, who worked more hours per week, or more weeks per year showed no 

differences in school outcomes (GPA, attendance, ACT scores, MCA scores). 

60 At the Achieve level, of students with referrals in SY15, 66% fewer had them in SY16; of students with referrals 

in SY16, 88% fewer had them in SY17. Of students with removals in SY15, 68% fewer had them in SY16; of 

students with removals in SY16, 100% fewer had them in SY17. Of students with suspension in SY15, 79% fewer 

had them in SY16; of students with suspensions in SY16, 75% fewer had them in SY17. 

At the Discover level, of students with referrals in SY15, 68% fewer had them in SY16; of students with referrals in 

SY16, 74% fewer had them in SY17. Of students with removals in SY15, 82% fewer had them in SY16; of students 

with removals in SY16, 65% fewer had them in SY17. Of students with suspension in SY15, 59% fewer had them in 

SY16; of students with suspensions in SY16, 47% fewer had them in SY17. 

Center for Applied Research and Educational Improvement, University of Minnesota 

100 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

Interpretation of findings. Given that matched students also had declining GPAs and 

attendance rates from one year to the next, it seems likely that other factors influenced the 

changes. One possibility for lower GPAs could be increasing course difficulty over time. 

It may be that participation in STEP-UP influenced students whose behavior improved. In light 

of the fact that some students with no behavior incidents before the internship had them the 

following year, we have to consider that factors other than the program had an influence on 

behavior change. 

Table 23 

Comparison of Intern Non-involvement in the Disciplinary System, 2015 and 2016 Cohorts 

2015 Cohort 2016 Cohort 

Achieve % without 

SY15 

% without 

SY16 

% without 

SY16 

% without 

SY17 

Referrals 83 92 90 97 

Removals 99* 96* 97 98 

Suspensions 95 97 87 97 

Discover % without 

SY15 

% without 

SY16 

% without 

SY16 

% without 

SY17 

Referrals 79 83 86 92 

Removals 97* 93* 91 92 

Suspensions 90 90 92* 86* 

*Decline rather than an increase pre-to post 

Exploratory Study Results Targeting Preliminary Level of Evidence: Employability 

Summary of findings. The exploratory impact study sought preliminary evidence about 

whether interns achieved better employability outcomes pre- to post-program. In 2016, both 

Achieve and Discover interns improved in job-related skills: knowing what questions to ask in an 

interview, what questions to expect in an interview, and identifying people to use as professional 

references.61 These outcomes were consistent with the results of 2015. 

In addition to the skills mentioned above, in 2016, Achieve interns showed positive changes in 

self-assessments of a number of personal skills62. Other than the interview and references skills, 

Achieve interns did not show change in employment-specific skills pre- to post. In 2016, 

61 After correcting for multiple comparisons, the difference on naming references was not statistically significant 

from pre to post. 

62 Staying calm under stress, accepting criticism, looking for additional tasks when work is complete, and breaking 

problems into small parts to solve them (the last one not statistically significant after correcting for multiple 

comparisons) 

Center for Applied Research and Educational Improvement, University of Minnesota 

101 
 

 

 

 

Discover interns showed positive changes in their self-assessments of other employment-specific 

skills: knowing what clothes to wear to work, oral and written communication in a professional 

setting, and creating a professional resume63. They did not show positive changes in their selfassessments of personal skills. 

Interpretation of findings. It is possible that Discover students, because they most likely 

had less experience in a work environment prior to their STEP-UP internship, had more to learn 

about workplace decorum and practices. Achieve interns, older and more experienced, may have 

been more open to personal growth. 

The exploratory analyses examining differences for interns who worked more hours per week or 

more weeks per year showed no differences in employability outcomes. There were some 

differences for interns with more than a single year of experience with STEP-UP. Both Achieve 

and Discover interns had better outcomes about interview skills, and two personal skills and one 

employment-specific skill. This suggests that having more than one STEP-UP summer internship 

may improve some employability skills. 

Limitations: employability skills. When interpreting outcomes of the employability 

section of the study, it is important to consider a number of limitations. The instrument used to 

gather data on interns was revamped for the 2016 cohort. Because of changes to many of the 

questions and the rating scale, it is difficult to compare many of the outcomes to previous 

cohorts. The instrument addressed self-ratings by youth, and many of their ratings were generous 

in the pre-test, making it difficult to have much positive change from pre- to post. We were not 

able to compare ratings by youth to ratings given by their supervisors to check for alignment. In 

addition, the kind of personal skills included in the instrument (e.g., problem-solving, accepting 

criticism) take more time to develop than a single summer’s internship. On the employability 

instrument self-ratings, an increase pre- to post is not necessarily an improvement. If a student 

came to assess his/her skills more realistically and gave a lower rating, that might be a positive 

outcome. We cannot be certain from these data what happened to that student. Finally, while 

most students in the program completed the pre-program survey, a more limited number 

completed the post-program survey. Results might have differed if we had data from all interns 

who completed their internship. 

63 After correcting for multiple comparisons there was not a statistically significant difference pre to post on resume 

writing. 

Center for Applied Research and Educational Improvement, University of Minnesota 

102 
Center for Applied Research and Educational Improvement, University of Minnesota 

103 
 

 

 

 

Conclusions and Recommendations 

This impact study targeted a moderate level of evidence using a quasi-experimental matchedgroups design to examine the program’s effect on educational outcomes and post-secondary 

enrollment. It sought to answer the confirmatory question, “Does the STEP-UP summer jobs 

program improve school outcomes?” The evaluation for 2016 for all STEP-UP participants (and 

only for Achieve in 2015) demonstrated moderate evidence of a small positive effect on 

students’ being on track to graduate in four years. The evaluation did not find moderate evidence 

of positive program effects on other school outcomes, academic or behavioral, including 

postsecondary enrollment. 

Recommendations 

While this report concludes the SIF external evaluation of STEP-UP, there are several areas of 

research that would be beneficial to AchieveMpls, the City of Minneapolis, and others in the 

field of youth employment regarding the benefits of internships on youth. 

1.	 Continue to examine longitudinal data to see if there are differences between a 

comparison group and interns. Because of the length of the grant, this evaluation was not 

able to access data on any participants who completed post-secondary education and 

entered the workforce. It is possible that participation in STEP-UP internships will have 

an impact on these outcomes, but we were not yet able to access data to examine this 

possibility. Tracking more youth over a longer trajectory and examining variables such as 

college graduation, type of employment, and wages could demonstrate longer-term 

impacts of the program. 

2.	 Focus on the use of non-academic outcome measures to measure program impact. 

STEP-UP did not show many positive effects on short-term school outcomes, which is 

logical given that the elements of the program (training, internship, career 

enrichments/specialized training) do not specifically target these outcomes. We know 

from survey responses that the program is having an effect on interns—expanding their 

consciousness about what the world of work is like, developing their networks in the 

professional community, helping them imagine what kind of career they might like and 

what path they need to take to achieve it, and becoming more comfortable in the 

workplace. 

Developing an instrument to measure these effects and comparing them to peers over 

time may provide useful information about program effects in the future. We encourage 

STEP-UP to move in the direction they are considering for assessing employability 

Center for Applied Research and Educational Improvement, University of Minnesota 

104 
 

 

outcomes. A retrospective pre-post instrument on which program participants can think 

about their growth rather than simply rating themselves could be helpful. One way to 

develop an instrument is to conduct interviews and/or focus groups to help develop prepost questions. It may also be possible to collaborate with Minneapolis Public Schools on 

its measures of social and emotional learning instruments. The results could also be 

compared to supervisor ratings of students. 

3.	 Consider a rigorous exploration of the dynamic between the supervisor and the intern as a 

contributing factor to internship success. This could draw on and contribute to the 

extensive research base on coaching/mentoring youth (and connects to other programs 

run by AchieveMpls). Interviewing both supervisors and interns could identify key 

factors that influence a productive internship as well as some of the longer-term goals of 

the program. 

Growing out of the implementation and impact studies, there are a number of steps the program 

could consider for strengthening the work of STEP-UP. 

4. As the curriculum for intern training is revamped, align training curriculum for interns 

more closely with desired employability outcomes. Having seen changes in the students’ 

self-ratings on skills taught specifically in the training (e.g., interviewing skills), the 

evidence suggests that even over a short period of time it is possible to influence the 

growth of participants in the program. 

5.	 Better align supervisor orientation and oversight with the desired employability outcomes 

for interns. Provide increased support to supervisors throughout the internship period that 

focuses on these outcomes. In the 2016 and 2017 program seasons, STEP-UP staff 

members took steps to work with the supervisors to encourage their intentional work with 

interns on employability outcomes. This continues to be an area of potential development 

for the program, though not a simple one to achieve given the challenges of finding 

supervisors, constraints on the time supervisors have available for working directly with 

interns, coaching skills of the supervisors, constraints on STEP-UP staff time, and of 

course, the complex nature of many of the employability outcomes. 

6.	 Revamp the logic model to eliminate school outcomes that are not impacted by the 

program and continue to revise it as other outcomes are demonstrated to have positive 

effects. 

Center for Applied Research and Educational Improvement, University of Minnesota 

105 
 

 

 

 

 

References
 

Maruyama, VanBoekel and Cobie (2013). “The STEP-UP Summer Jobs Program: Participants 

and Outcomes [Confidential: For Comment and Discussion].” University of Minnesota. 

Michlin, M. L. & Schultz, A. R. (2016). Summer Youth Employment Programs Effect 

Size Review. Saint Paul, MN: University of Minnesota, College of Education and 

Human Development, Center for Applied Research and Educational 

Improvement. 

Shadish, W. R., Cook, T. D. & Campbell, D. T. (2002). Experimental and quasi-experimental 

designs for generalized causal inference. New York: Houghton Mifflin Co. 

Valentine, E.J., Anderson, C., Hossain, F., & Unterman, R. (2017). An Introduction to the World 

of Work: A Study of the Implementation and Impacts of New York City’s Summer Youth 

Employment Program. Prepared for the U.S. Department of Labor by MDRC. Retrieved 

from 

http://www.mdrc.org/sites/default/files/SYEP_Embedded_Full_Report_508_rev2.pdf. 

Center for Applied Research and Educational Improvement, University of Minnesota 

106 
 

Appendix A
 

STEP-UP Logic Model
 

Center for Applied Research and Educational Improvement, University of Minnesota 

107 
 

 

 

Appendix B: STEP-UP Intern Training Curriculum Topics
 

Discover (High School) Achieve	 Advanced 

 Program information  Program information  Program information 

 Jobs vs. Careers  Jobs vs. Careers  Reflection on previous experience 

 Professional Introductions  Networking  Networking 

 Attitude & Character  Resumes  Resumes Session 1  Attitude & Character	  Marketing yourself as a young 

 Job preferences professional 

 Cover letters 

 Job preferences 

 Learning styles  Communication  Interpersonal communications 

 Communication types  Professionalism  Professionalism 

 Perception vs. Reality  Ethics  Praise, criticism, & feedback 

Session 2  Situational communications  Praise, criticism & feedback  MHA 

 Professionalism  Interviewing  Interviewing 

 Dressing for Success  Dressing for success 

 Starting your job 

Mock Interviews Mock Interviews	 Mock Interviews 

 Praise, criticism & feedback Intake Interviews	 Intake Interviews 

 Building a strong relationship with your 

supervisor 

 Decision-making & workplace ethics 

Session 3  Resumes 

 Interviewing 

 Dress for Success 

 Career interest assessment 

 Job preference for the application 

Session 4	  First day on the job 

 Required work documents 

 I-9 proper identification form 

 Rules for using cell phones at work 

 Email and phone etiquette 

Intake Interviews 

Center for Applied Research and Educational Improvement, University of Minnesota 108 
 

 

 

 

 

APPENDIX C
 

Implementation Data: 2017
 

Program Fidelity: Major components remained the same in 2017 as in 2016. 

Training: 

During training in 2017, Achieve staff members assisted with particular parts of the curriculum 

for the first time. 

In addition to the training prior to the summer, Discover interns participated in classes during the 

internship period. In the summers of 2014-16 interns attended classes once a week for eight 

weeks; this was a vestige in the program from the time when there was not training in the spring 

for Discover interns. In 2017 weekly classes were replaced with a single professional 

development day (7 hours). The change was made in response to youth dissatisfaction with the 

weekly classes. Youth were frustrated by the repetitive content, the weekly time commitment 

getting to and from the class, as well as the class itself. The day long program afforded 

participants the opportunity to talk to each other about their jobs and heard about why the 

program is different for 14-15 years old and how it changes as they turn 16. They also took part 

in activities based on Holland’s code, guiding them into active learning about career pathways. 

Supervisor Orientation: 

In 2017 STEP-UP Achieve made significant changes and improvements to the Supervisor 

Orientation, developed and offered a more advanced class to returning supervisors for the first 

time, and drove participation by communicating to employers that these trainings were 

mandatory. Three hundred and ninety-five (395) supervisors attended a training this year, about a 

third more than last, and AchieveMpls received overwhelmingly positive feedback. During the 

101 sessions, they stressed to new supervisors the importance of their role, that each one of them 

could be the person who helps their intern get their footing and start down a rewarding, life-long 

career pathway. During these classes, they walked through the basic elements of a well-planned 

internship, including a thorough work plan, the importance of the first week, having a colleague 

to serve as a mentor in tandem with the supervisor, and providing continuous feedback 

connected with foundational skills outlined in their handbooks. They also covered cultural 

competency, implicit biases, Imposter Syndrome, and to not assume that interns know what is 

expected of them in the workplace. 

Center for Applied Research and Educational Improvement, University of Minnesota 

109 
 

 

 

The 201 Curriculum built on training from last year, introducing more advanced tools for 

building interns’ skills, such as adopting the Stance of Inquiry and giving an overview of more 

complex topics of cultural competency. The training focused on helping managers appreciate 

diversity in the workplace through the Intercultural Development Continuum (IDC). Participants 

learned to recognize where the workplace and individual falls on the IDC and how to create an 

actionable strategy to achieve an intercultural mindset. They stressed that diversity alone 

produces nothing; people must cultivate their ability to recognize, adapt to, and promote the 

unique skills, experiences, and perspectives of their colleagues. 

In 2017, STEP-UP partnered with The Brand Lab, Right Track, BrookLynk, and Genesys Works 

to put on the Supervisor’s Toolbox on June 27th, hosted by Ameriprise Financial and sponsored 

by US Bank and MHA labs, and 54 attended. The session included performances, a panel of 

alumni, and breakout sessions. 

The Achieve supervisor handbook was reworked for 2017. 

Internships: 

In in 2017 the minimum wage for Achieve interns was established at $.50 above the state 

minimum. 

In 2017 all Discover interns were limited to 20 hours a week – including their one-day PD 

session (they had to work fewer hours at their worksite that week) – due to the large funding cut 

from the state Youth at Work competitive grant. This cut did not account for the funding cut, but 

it helped. 

RESEARCH QUESTIONS: 

The discussion that follows summarizes the findings according to the research questions upon 

which the study was based. 

4. Has the STEP-UP program implemented all of its major components with fidelity? 

b. To what extent have the youth output targets been met? (comparison of current 

year’s outputs to current year’s targets) 

Center for Applied Research and Educational Improvement, University of Minnesota 

110 
 

 

 

 

 

Eligible Applicants: Compared to targets for 2017, the actual number of eligible 

applications fell short of targets by 573 (4,000 vs. 3,427). The number of applicants in 

2017 was just short of the number in 2016 (20 fewer). 

Work-Readiness Training Completers: Compared to targets for 2017, the number of 

students completing work-readiness training fell slightly short of the target (2,250 vs. 

2, 077). More students completed work-readiness training than in 2016 (2,036). 

Internships Completed: In 2017, the number of youth completing internships fell 

short of the target (1,585 vs. 1,378). More youth completed internships in 2017 than 

in any of the years included in the SIF grant (2014, 2015, 2016). 

Extra Enrichments/Specialized Career Training: For activities of 20 or fewer hours, 

656 interns participated in 2017. This exceeded the target of 520 and was more than 

the number in 2016 (424). For activities longer than 20 hours, 66 interns participated 

in 2017, fewer than the target of 90 participated and fewer than the number in 2016 

(71). 

Work-based Learning Credit: Approximately 1,000 credits were earned (still being 

processed at the time of writing). 

Wages: In 2017, STEP-UP interns earned an estimated $3 million in wages). 

Youth output numbers in 2017 were close to the 2016 numbers for training and higher 

for internships and short specialized training. The shorter career enrichment 

opportunities and number of available internships exceeded targets; applications, 

training completers, internship completers, and longer career enrichment numbers in 

2017 were short of targets. 

i. To what extent have the employer output targets been met? 

Number of Available Internships: In 2017, 1,495 internships were available, more 

than the target (1,440). This was an increase over 2016 (1,389). 

Number of Companies: In 2017, 212 companies had STEP-UP interns. This was a 

small decrease over 2016 (256). There were not targets for the number of companies. 

Number of Supervisors: In 2017, 543 supervisors worked with STEP-UP interns, an 

increase of 30 over 2016. No targets were set for the number of supervisors. 

Types of Settings: Achieve interns served in a range of industries as well as non-profit 

and government settings. While Discover interns worked in the nonprofit sector only, 

Center for Applied Research and Educational Improvement, University of Minnesota 

111 
 

 

the types of organizations in which they were placed also varied. This was consistent 

with previous years. 

Center for Applied Research and Educational Improvement, University of Minnesota 

112 
 

 

 

 

 

 

 

 

 

 

 

 

2014 2015 2016 2017 

Target Actual Target Actual Target Actual Target Actual 

Eligible Applications 3,754 3,678 4,000 3,062 4,000 3,447 3,427 

Accepted to program 3,678 3,582 3,800 3,003 3,100 3,447 4,000 3,427 

Completed Work-Readiness 

Training 

2,200 2,076 2,125 1,879 2,200 2,036 2,250 2,077 

Internships started 1,500 1,425 1,400 1,238 1,420 1,389 1,440 1,495 

Completed internships 1,278 1,222 1,300 1,132 1,310 1,272 1,585 1,378 

Companies/Organizations 

participating 

268 212 226 212 

Supervisors 539 505 513 543 

Specialized Training/Career 

Enrichment Activities (20 

hours or fewer) 

250 576* 470 491 500 424 520 656 

Specialized Training/Career 

Enrichment Activities (more 

than 20 hours) 

82 124 85 120 90 66 

Center for Applied Research and Educational Improvement, University of Minnesota 113 
 

APPENDIX D
 

Career Enrichment and Specialized Trainings
 

During the 2016 program cycle, AchieveMpls offered the following events: 

Legal Careers Exploration: A field trip sponsored by the Hennepin County Bar Association 

(HCBA). Students learned about legal careers from a panel of HCBA members, met with 

Hennepin County Judges William Koch and James Moore, and observed a variety of hearings. (4 

hours) 

Financial Services Careers: Financial services company volunteers spent a day with STEP-UP 

Achieve interns to talk about the wide range of careers available in financial services and the 

underlying skills that are essential to success. The event featured a panel representing distinct 

areas of financial services, professional development sessions, and a speed career networking 

exercise with industry representatives. (4.5 hours) 

High Tech Innovation Day: Interns explored new technologies (from Google Glass and Google 

Cardboard, to virtual reality tools, to smart watches and others) and learned about design 

thinking and the process of iteration. The program was led by volunteers from CoCo, a coworking space and innovation hub, and local tech businesses. (8 hours) 

Pre-Employment Training for Healthcare (PETH): This training was designed to help youth 

understand the range of health careers available and the underlying skills essential to success. 

Interns learned about customer service in healthcare, terminology, confidentiality, HIPAA, and 

living the organization’s mission. Volunteers representing nine different industries hosted (8 

hours) panels to introduce interns to different career paths and education requirements. 

Excel Training Program: Excel is an essential skill for the workplace but generally not taught in 

the Minneapolis schools. The purpose of the training series was to fill that gap, allowing interns 

to develop Excel skills for use in their internships and beyond. Over the course of the summer, 

participating interns completed a series of online training modules at different levels to enhance 

their skill development. Interns who were already adept with Excel could prepare for the official 

certification exam. (Units 1-6 is under 20 hours of training; Level 1 (Units 1-7), Level 2 (Units 

7-13, and Level 3 (Units 13-17) are all over 20 hours of training) 

Summer Legal Institute (SLI): This intensive 5-day law immersion program took place in June at 

the University of Minnesota Law School. Interns visited federal courts and law firms, met with 

judges and practitioners, engaged in mock trials and oral argument competitions, participated in 

Center for Applied Research and Educational Improvement, University of Minnesota 

114 
 

 

 

 

 

 

 

 

 

 

professional skills and financial literacy workshops and learned about a wide variety of legal 

careers and required education. SLI was led by Just the Beginning, an organization based in 

Chicago that develops these camps across the nation. (40 hours) 

Outdoor Careers Academy: In partnership with Wilderness Inquiry64 , interns in the program 

learned about “leave no trace” principles, first aid/CPR, workforce options, and earning an 

Outdoor Leadership credential. (24 hours) 

IT Training Credential: This new training and certification program introduced students to 

Scratch (multi-media) programming, Linux, Python, computer security, and careers and 

education in IT. Consisting of ten 45-minute modules (each including lecture and hands-on 

activity) based on Advance IT Minnesota’s IT Exploration Curriculum, it was led by an IT 

instructor from Minneapolis Community and Technical College (MCTC). (21hours) 

Scrubs Camp: This week-long camp in July gave interns the chance to participate in hands-on 

healthcare activities and to learn about health-related careers while experiencing life on the 

Augsburg College campus. (40 hours) 

Camp Explore: This is a week-long residential camp offering high school students hands-on, 

engaging activities to help them explore many different STEM careers in key sectors. 

Financial Education Day: Run by US Bank, this day is an opportunity for interns to learn about 

their personal finances. 

Golden Gopher Day: This program is a day at the University of MN where interns learn more 

about the college experience, connections between degrees and careers, and how to prepare for 

post-secondary education. 

For Discover interns, the following programs were offered in 2016: 

Silicon North Stars: A partnership with the Silicon North Stars program made it possible for 

eight 14-year old interns to travel to Silicon Valley for a week-long technology camp. The trip 

included meeting with internet and technology leaders from top companies, startups, and venture 

capital funds. The camp concluded with a design thinking challenge and demo day project in 

which the students competed in teams to create and pitch their own startup ideas to a crowd of 

tech professionals. Quarterly these youth, plus the youth from past years get together on a 

64 Wilderness Inquiry is a non-profit organization that conducts outdoor adventure experiences and travel with the 

purpose of inspiring personal growth, enhanced awareness of the environment and community integration 

Center for Applied Research and Educational Improvement, University of Minnesota 

115 
 

 

Saturdays to tour tech companies in the Twin Cities and continue to receive mentoring 

throughout high school. 

Camp Sunrise: A partnership with YouthCARE/Camp Sunrise offered STEP-UP Discover 

interns the chance to spend a week at a rustic camp. Many interns have never had a camping 

experience. This experience offered them an opportunity to participate in all the usual camp 

activities and introduced them to outdoor career opportunities. In addition, interns developed 

leadership skills and completed work projects to support the camp. Forty-one (41) interns spent a 

week at Camp Sunrise in the summer of 2016. 

Center for Applied Research and Educational Improvement, University of Minnesota 

116 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

APPENDIX E
 

Survey Questions Used to Assess Program Implementation 

OUTPUTS 

Program component Data source Survey questions (where relevant) 

Training STEP-UP database 

Pipelines/Career 

Enrichment Activities 

STEP-UP database 

Internships: Interns STEP-UP database; 

baseline survey; endof-summer intern 

survey 

End-of-summer intern survey (Achieve): 

Salary: What was your hourly wage? 

Minimum Wage ($9.00, $9.50 after August 1st) 

$9.51--$9.74 per hour 

$9.75--$9.99 per hour 

$10.00--$10.24 per hour 

$10.25--$10.49per hour 

$10.50--$10.74 per hour 

$10.75--$10.99 per hour 

$11.00--$11.49 per hour 

$11.50--$11.99 per hour 

$12.00--$12.49 per hour 

$12.50--$12.99 

$13.00 or more per hour 

Weeks worked: 

The "official" dates for STEP-UP internships for 

this summer were June 20--August 19th, or 9 

weeks. 

Some organizations needed interns for 

different dates. How many weeks did you work 

in your job? 

4 weeks 

5 weeks 

6 weeks 

7 weeks 

8 weeks 

9 weeks 

10 weeks 

More than 10 weeks 

Hours worked per week 

On average, how many hours a week did you 

work at your job this summer? 

Less than 10 hours a week 

Between 10-15 hours a week 

Between 16-25 hours a week 

Between 26-34 hours a week 

Between 35-40 hours a week 

More than 40 hours a week 

Center for Applied Research and Educational Improvement, University of Minnesota 

117 
 

 

 

 

 

Baseline survey: 

Previously held job 

I have had a paid job before. Y/N 

Internships: Supervisors STEP-UP database 

Internships: Placements STEP-UP database 

FIDELITY OF IMPLEMENTATION 

Program component Data source Survey questions (where relevant) 

Training Staff interviews; 

program website; 

program documents 

Pipelines/Career 

Enrichment 

Staff interviews; 

program website; 

program documents 

Internships Staff interviews; 

program website; 

program documents 

PROGRAM DOSAGE 

Program component Data source Survey questions (where relevant) 

Training Program documents 

Internship Program documents 

Pipelines/Career 

Enrichment 

Program documents 

PROGRAM EXPOSURE/SATISFACTION/QUALITY 

Program component Data source Survey questions (where relevant) 

Training Post-training survey; 

end-of-summer intern 

survey 

Post-training survey: 

SD/D/A/SA 

My trainer was knowledgeable about the topics 

covered during training. 

My trainer taught the lessons in a way that was clear. 

My trainer taught the lessons in a way that was 

engaging. 

After today's training, I am ready for a job. 

Training End-of-summer intern 

survey 

End-of-summer intern survey: 

Center for Applied Research and Educational Improvement, University of Minnesota 

118 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

To what extent do you agree with each of the 

following statements about your experience in 

STEP-UP? 

SD/D/A/SA 

The training sessions prepared me for my summer job. 

I used the skills I learned at training in my job. 

Internships: Time 

worked 

End-of-summer intern 

survey (Achieve); 

STEP-UP database 

(Discover) 

End-of-summer intern survey: 

Weeks worked: 

The "official" dates for STEP-UP internships for 

this summer were June 20--August 19th, or 9 

weeks. 

Some organizations needed interns for different 

dates. How many weeks did you work in your 

job? 

4 weeks 

5 weeks 

6 weeks 

7 weeks 

8 weeks 

9 weeks 

10 weeks 

More than 10 weeks 

Hours worked per week 

On average, how many hours a week did you 

work at your job this summer? 

Less than 10 hours a week 

Between 10-15 hours a week 

Between 16-25 hours a week 

Between 26-34 hours a week 

Between 35-40 hours a week 

More than 40 hours a week 

Internships: Activities End-of-summer intern 

survey; End-ofsummer supervisor 

survey; program 

documents 

End-of-summer intern survey: 

How often, if at all, did these things happen at 

your internship? 

Never/Once/A few times over the summer/Once a week/A 

few times a week/Every day 

Someone on the staff talked to me about my future plans. 

My supervisor met with me to track my work progress. 

My supervisor gave me feedback about my performance. 

My supervisor used the MHA Labs 12 Hirability skills when 

talking about my performance. 

I had a chance to learn more about a career in the field 

(examples: job shadow, informational interview). 

Someone reflected with me on my accomplishments and 

what I learned. 

I participated in specialized training (such as Excel, Outlook, 

or database) 

Center for Applied Research and Educational Improvement, University of Minnesota 

119 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

How well does each of the following statements 

describe your internship experience? 

1 (not at all) 2 3 4 5 6 (Completely) 

Someone gave me clear directions on how to complete 

tasks. 

Someone helped me understand what the 

company/organization does. 

Someone explained expectations and policies (such as cell 

phone use and appropriate dress). 

My supervisor provided me with a written work plan with 

clear goals. 

My supervisor gave me input on setting my internship goals. 

My supervisor gave me choices about the projects I would 

work on. 

The tasks I had were challenging. 

I had a mixture of short-term and longer-term projects over 

the summer. 

I got to learn skills I needed to do my job. 

End-of-summer supervisor survey 

How often, if at all, did these things happen at 

your workplace over the summer 

Never/Once/A few times over the summer/Once a week/A 

few times a week/Every day 

My intern had a chance to learn more about a career in the 

field (examples: job shadow, informational interview). 

How well does each of the following statements 

describe the internship experience at your site? 

1 (not at all) 2 3 4 5 6 (Completely) 

My intern(s) had a mixture of short-term and longer-term 

projects over the summer. 

My intern(s) had opportunities to learn job-related skills. 

My intern(s) had a structured work plan with measurable 

goals. 

My intern(s) tasks were appropriately challenging. 

Internships: End-of-summer End-of-summer supervisor survey 

Supervisory Time supervisor survey 

How often, if at all, did these things happen at 

your workplace over the summer 

Never/Once/A few times over the summer/Once a week/A 

few times a week/Every day 

I met with my intern(s) to give feedback. 

I used the MHA Labs 12 hirability skills to give feedback to 

my intern(s). 

In a typical week, how much time did you spend 

on each of these activities? 

Not at all/15 minutes/30 minutes/45 minutes/An hour/90 

Center for Applied Research and Educational Improvement, University of Minnesota 

120 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

minutes/2 hours/More than 2 hours 

a) Teaching my intern job related skills 

b) Giving feedback to my intern 

c) Doing check-ins 

d) Talking about the career field 

e) Developing/reviewing the work plan with my intern 

Internships End-of-summer intern 

survey; end-ofsummer supervisor 

survey 

End-of-summer intern survey: 

To what extent do you agree with each of the 

following statements about your experience in 

STEP-UP? 

SD/D/A/SA 

My summer job was related to my interests. 

My summer job was a valuable learning experience. 

I felt I made a valuable contribution to my workplace. 

My job helped me decide what career to pursue in the future. 

How likely is it that you will stay in touch with 

your supervisor and/or mentor? 

Very likely, we have already made plans to do so/Likely, we 

talked about it/I'm not sure/Not likely/I am not interested in 

keeping in touch with my supervisor/mentor 

STEP-UP has prepared me to become a 

valuable part of the future 

workforce of Minneapolis. 

SD/D/A/SA 

How have you benefited from participating in the 

STEP-UP Summer Jobs Program? (open-ended) 

How would you suggest improving the program 

for next year? 

End-of-summer supervisor survey: 

How well does each of the following statements 

describe the internship experience at your site? 

1 (not at all) 2 3 4 5 6 (Completely) 

The tasks the intern(s) did had a positive impact on our 

workplace. 

My STEP-UP experience helped me feel more culturally 

competent. 

My STEP-UP experience helped me feel better prepared to 

work with the future work force. 

Think about the amount of time you spent weekly 

Center for Applied Research and Educational Improvement, University of Minnesota 

121 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

working with your intern and engaging in 

activities 

related to the internship. How did this compare 

with the amount of time you expected to spend 

on STEPUP? 

Less time than I expected 

About what I expected 

More time than I expected 

Was the student matched to your 

company/organization a good fit for the job? 

No/Yes 

Overall, did your intern(s) make a valuable 

contribution to your organization this summer? 

No/Yes 

To what extent do you feel that STEP-UP 

Program was a success at your organization? 

Not at all successful/Minimally successful/Moderately 

successful/Very successful 

Please answer these questions about the 

supervisor orientation. 

1 (not at all) 2 3 4 5 6 (A lot) 

To what extent did the Orientation help you prepare for the 

internship experience? 

b) To what extent did the Orientation help you supervise 

your intern during the summer? 

What was most helpful about the handbook? 

(open-ended) 

What could we add to the handbook that would 

make it more helpful? (open-ended) 

How helpful did you find each of the following 

parts of the weekly emails from STEP-UP? 

1 (not at all) 2 3 4 5 6 (Completely) 

a) Intern/Supervisor Connection 

b) Information about Professional Development Events 

c) Requests and reminders 

d) Job Coach Corner on Hirability Skills 

Please answer these questions about the 

responses you got from STEP-UP 

Achieve/Discover during the summer. N/Y 

a) Did you receive timely responses when you had a 

question/concern? 

b) Did you receive helpful responses when you had a 

question/concern? 

1 (not at all) 2 3 4 5 6 (Completely) 

To what extent did you get the support you needed from 

STEP-UP Achieve/Discover? 

Please give us your feedback on the MHA Labs 

Center for Applied Research and Educational Improvement, University of Minnesota 

122 
 

 

 

 

 

 

 

 

 

 

Hirability Skills Framework and weekly check-in 

template. 

1 (not at all) 2 3 4 5 6 (Completely) 

How helpful was the MHA Labs 12 Hirability Skills 

Framework to you as a 

supervisor? 

How helpful was the weekly check-in template for 

supporting your intern(s)? 

Please provide feedback on the work plan and 

evaluation template. How helpful was the work plan section of the template? 

How helpful was the intern evaluation section of the 

template? 

What did you enjoy most about being a STEPUP supervisor this summer? (open-ended) 

What suggestions do you have for how the 

program could be improved in the future? 

PERCEPTIONS OF THE PROGRAM 

Program component Data source Survey questions (where relevant) 

Training Group interviews 

conducted by 

AchieveMpls staff 

Internships Group interviews 

conducted by 

AchieveMpls staff 

Center for Applied Research and Educational Improvement, University of Minnesota 

123 
 

 

 

 

APPENDIX F
 

Perceptions Study
 

Group Interview Protocol Spring Interviews: 

Never Participated in STEP-UP 

Thanks for coming today. We represent the STEP-UP program and we’re trying to learn what young 

people think about STEP-UP and how we can improve it. 

1.	 Please tell us your name and what you did last summer. 

2.	 When you hear the name STEP-UP, what are words that come to your mind? 

3.	 What have you heard other students in your school say about STEP-UP? 

4.	 What have you heard adults in your school say about STEP-UP? 

5.	 Did you ever consider applying to STEP-UP? 

a.	 If you did apply, what appealed to you about the program? 

b.	 If you did not apply, what bothered you about the program? 

6.	 Please complete this sentence in writing and then we’ll share answers: STEP-UP is a 

program that is best for . . . (probe responses) 

7.	 What would make you want to do STEP-UP? 

8.	 We’d like your advice; Please complete this sentence: STEP-UP, you’d be smart if you ; ; ; 

9.	 Is there anything I missed today? Is there anything you’d like to say that you didn’t get a 

chance to say? 

Previously participated in Discover but didn’t reapply 

1.	 Please tell us your name and what STEP-UP internship you had. 

2.	 When you hear the name STEP-UP, what are words that come to your mind? 

3.	 Think back to your internship with STEP-UP. What did you like most about STEP-UP? 

(Possibility: sticky notes) 

4.	 What did you not like about STEP-UP? (Possibility: sticky notes) 

5.	 What patterns do we see in the sticky notes? (if that activity is used) 

6.	 What kept you from applying again? 

7.	 What will you do this summer instead of STEP-UP? 

8.	 We’d like your advice; Please complete this sentence: STEP-UP, you’d be smart if you ; ; ; 

9.	 Is there anything I missed today? Is there anything you’d like to say that you didn’t get a 

chance to say? 

Previously participated in !chieve but didn’t reapply 

1.	 Please tell us your name and what STEP-UP internship you had. 

2.	 When you hear the name STEP-UP, what are words that come to your mind? 

3.	 Think back to your internship with STEP-UP. What did you like most about STEP-UP? 

(Possibility: sticky notes) 

4.	 What did you not like about STEP-UP? (Possibility: sticky notes) 

Center for Applied Research and Educational Improvement, University of Minnesota 

124 
 

 

 

 

 

 

5.	 What patterns do we see in the sticky notes? (if that activity is used) 

6.	 What kept you from applying again? 

7.	 What will you do instead of STEP-UP? 

8.	 We’d like your advice; Please complete this sentence: STEP-UP, you’d be smart if you ; ; ; 

9.	 Is there anything I missed today? Is there anything you’d like to say that you didn’t get a 

chance to say? 

!pplied and didn’t go to training 

1.	 Please tell us your name and what you did last summer. 

2.	 When you hear the name STEP-UP, what are words that come to your mind? 

3.	 You applied to STEP-UP. What made you decide to apply? 

4.	 You were accepted to the program but didn’t go to training; What stopped you? 

5.	 Please filling in the blanks in this sentence in writing and then we’ll talk about what you 

wrote: I’d want to go to STEP-UP training if . . . and . . . 

6.	 We’d like your advice. Please complete this sentence: STEP-UP, you’d be smart if you ; ; ; 

7.	 Is there anything I missed today? Is there anything you’d like to say that you didn’t get a 

chance to say? 

Adult Staff 

1.	 Please introduce yourself and tell me your role in the school. 

2.	 When you think about STEP-UP, what is the first thing that comes to mind? 

3.	 What have you heard other adults in your building say about STEP-UP? 

4.	 Please complete this sentence in writing and then we’ll share answers: STEP-UP is a 

program that’s best for ; ; ; (probe responses) 

5.	 To whom, if anyone, would you/do you recommend STEP-UP? What would be your reasons 

for recommending it? 

6.	 This is a scale about being an advocate for STEP-UP. Where would you put yourself on the 

scale? 

1 2 3 4 5 6 7
 

Not at All Very Strong
 

What made you put yourself at that place on the scale? 

7.	 We’d like your advice; Please complete this sentence: STEP-UP, you’d be smart if you ; ; ; 

8.	 Is there anything I missed today? Is there anything you’d like to say that you didn’t get a 

chance to say? 

Summer Interviews: 

SUA Summer Group Interviews: Questions 

 Please briefly tell us your name, where you’re working this summer. 

 What has been a highlight of your internship this summer? 

 Think about you current internship. What do you like best about it? 

 What past experiences have you had with STEP UP? 

Center for Applied Research and Educational Improvement, University of Minnesota 

125 
 

 

 

 

 

 

 What were some of the benefits you got from previous experiences in STEP-UP? 

 Why did you decide to apply to STEP-UP again this year?
 

 Why did you decide to apply to STEP UP the first time?
 

 What did you hope to get out of the program? 

o Have your hopes come true? Or what have you gotten out of the program? 

 What challenges have you had in STEP-UP? This year or previously?
 

 How was the training helpful to you once you got to your internship?
 

 If you did Discover, how would you describe the differences between Achieve and Discover?
 

o Where or how did you learn about the differences? 

 Imagine you are trying to get a friend to apply to STEP UP next summer. What would you say to 

them to encourage them? 

 Anything else we should know about your experience with STEP-UP? 

Interview Groups 

Group Number of Participants 

Roosevelt HS, Never Participated 8 

Roosevelt HS, Applied but did not train 8 

Roosevelt HS, Did Discover, did not reapply 6 

Southwest HS, Applied but did not train 8 

Southwest HS, Did Discover but did not reapply 6 

Southwest HS, Did Achieve but did not reapply 2 

MPS Achieve interns 9 

Excel Achieve interns 4 

Center for Applied Research and Educational Improvement, University of Minnesota 

126 
 

 

 

 

Appendix G 

Propensity Score Match 

Proportion of Cases Matched 

Students included in 

model 

Proportion of Cases 

Matched: Comparison 

Proportion of Cases 

Matched: Treatment 

Model 1 Grade 10 Discover 8% (196/2,374) 100% (209/209) 

Model 2 Grade 11 Discover 6% (33/539) 100% (36/36) 

Model 3 Grade 10 Achieve 3% (7/269) 100% (10/10) 

Model 4 Grade 11 Achieve 6% (118/2,059) 100% (125/125) 

Model 5 Grade 12 Achieve 4% (138/3,142) 100% (147/147) 

Distribution of the Propensity Scores in the Treatment and Comparison 

Groups 

Treatment Groups 

Center for Applied Research and Educational Improvement, University of Minnesota 

127 
 

Comparison Groups 

Center for Applied Research and Educational Improvement, University of Minnesota 

128 
 

 

 

 

Standardized Mean Differences in Baseline Characteristics Between 

Treatment and Comparison Groups 

Model 1: Grade 10 Discover 

Co-variate Before Match After Match 

Distance 0.981747 0 

Race 1 -0.48597 0 

Race 2 0.485968 0 

Gender -0.0928 0.009608 

ELL -0.18661 0 

Special Ed -0.13474 0 

Free/Reduced Lunch 0.467407 -0.01206 

Homeless/Highly Mobile -0.07702 0 

Home Language 1 0.516378 0.015926 

Home Language 2 0.261268 -0.01282 

Home Language 3 -0.10697 0 

Home Language 4 0.086351 0 

Center for Applied Research and Educational Improvement, University of Minnesota 

129 
 

Model 2: Grade 11 Discover 

Co-variate Before Match After Match 

Distance 0.916262 0.004624 

Race 1 -0.54051 0 

Race 2 0.540506 0 

Gender 0.013018 -0.05475 

ELL 0.134715 0 

Special Ed -0.0156 -0.06917 

Free/Reduced Lunch 0.444828 -0.15831 

Homeless/Highly Mobile -0.04408 0 

Home Language 1 0.561213 0 

Home Language 2 0.218605 0.099038 

Home Language 3 -0.25706 0 

Home Language 4 0.152443 0.166567 

School 1 -0.31956 0 

School 2 -0.172 0 

Model 3: Grade 10 Achieve 

Co-variate Before Match After Match 

Distance 4.233933 0.144917 

Race 1 -0.25923 0 

Race 2 0.25923 0 

Gender -0.57421 -0.3703 

ELL -0.36787 0 

Special Ed -0.13661 .1/0 

Free/Reduced Lunch 0.435458 0 

Homeless/Highly Mobile -0.08595 0 

Home Language 1 2.236934 .2/0 

Home Language 2 0.462791 0 

Home Language 3 -0.5976 0 

Home Language 4 0.531762 -0.37793 

School 1 -0.30577 0 

School 2 -0.21567 0 

Center for Applied Research and Educational Improvement, University of Minnesota 

130 
 

Model 4: Grade 11 Achieve 

Co-variate Before Match After Match 

Distance 1.086154 0 

Race 1 -0.46226 0 

Race 2 0.462264 0 

Gender -0.10744 0 

ELL -0.01881 0.020921 

Special Ed 0.016213 0.020921 

Free/Reduced Lunch 0.521431 -0.02173 

Homeless/Highly Mobile 0.079038 0 

Home Language 1 0.536921 0.026093 

Home Language 2 0.37806 -0.01845 

Home Language 3 -0.13624 0 

Home Language 4 0.246421 0 

School 1 -0.3633 -0.05205 

School 2 -0.08752 0 

Model 5: Grade 12 Achieve 

Co-variate Before Match After Match 

Distance 1 0 

Race 1 -0.35059 -0.02238 

Race 2 0.35059 0.02238 

Gender -0.09455 0 

ELL 0.184204 0 

Special Ed -0.08753 0 

Free/Reduced Lunch 0.305687 0 

Homeless/Highly Mobile -0.05922 0 

Home Language 1 0.677619 0 

Home Language 2 0.402678 0 

Home Language 3 -0.11667 0 

Home Language 4 0.239617 0 

School 1 -0.33426 -0.04237 

School 2 -0.27779 0.026909 

Center for Applied Research and Educational Improvement, University of Minnesota 

131 
 

Appendix H
 

Employability Outcomes Survey Instrument
 

Center for Applied Research and Educational Improvement, University of Minnesota 

132 
Center for Applied Research and Educational Improvement, University of Minnesota 

133 
 

 

Appendix I 

Demographic Data on Completing Interns 

Gender 

% N 

Male 43 552 

Female 57 720 

Race/Ethnicity 

% N 

African American 33 424 

American Indian 6 73 

Asian 11 145 

Hispanic 9 110 

White 6 78 

Ethnic African 29 371 

Other/Did Not 

Indicate 

6 71 

Training Level 

% N 

Achieve 34 435 

Advanced 14 182 

HS Discover 30 383 

MS Discover 21 272 

Eligible for Free/Reduced Lunch 

% N 

No 10 126 

Yes 90 1145 

Average Hours Worked per Week: Achieve (n = 437) 

% n 

Less than 10 hours 4 17 

10-15 hours 18 79 

16-25 hours 28 123 

26-34 hours 20 86 

35-40 hours 27 119 

More than 40 hours 3 13 

Center for Applied Research and Educational Improvement, University of Minnesota 

134 
 

 

 

 

Average Number of Weeks Worked: Achieve (n = 437) 

% n 

4 weeks 5 22 

5 weeks 5 21 

6 weeks 11 48 

7 weeks 8 37 

8 weeks 18 80 

9 weeks 38 168 

10 weeks 8 34 

More than 10 weeks 6 27 

Previous Paid Job: Achieve (n = 437) 

% n 

No 32 140 

Yes 68 296 

Previous Paid Job: Discover (n = 479) 

% n 

No 63 362 

Yes 37 217 

Earnings 

Discover $944,886 

Achieve $1,346,340 

Total $2,291,226 

Center for Applied Research and Educational Improvement, University of Minnesota 

135 
 

 

 

 

 

 

Appendix J 

Industries: Intern Placement 

Achieve 

Number Percentage of Total 

Architecture & Construction 3 <1 

Arts, A/V Technology & Communication 56 9 

Business Management & Administration 7 1 

Construction 9 1 

Education & Training 95 15 

Finance 75 12 

Government & Public Administration 105 17 

Health Science 78 13 

Human Services 38 6 

Information Technology 2 <1 

Law, Public Safety & Security 12 2 

Marketing, Sales & Services 26 4 

Outdoor and Natural Resources 30 5 

Science, Technology, Engineering & Mathematics 59 10 

Tourism Services 17 3 

Transportation, Distribution & Logistics 4 1 

TOTAL 617 100 

Discover 

While Discover interns worked in the nonprofit sector only, the types of organizations in which they 

were placed also varied. 

Number Percentage of Total 

Architecture & Construction 16 2 

Arts, A/V Technology & Communication 51 8 

Business Management & Administration 1 <1 

Education & Training 101 23 

Government & Public Administration 17 3 

Health Science 19 3 

Human Services 223 34 

Law, Public Safety & Security 6 1 

Marketing, Sales & Services 2 <1 

Outdoor and Natural Resources 216 33 

Misc. 14 2 

TOTAL 657 100 

Center for Applied Research and Educational Improvement, University of Minnesota 

136 
 

 

 

Appendix K 

Program Exposure 

ACTIVITIES 

Structure/Orientation. One set of questions asked about the extent (Not at all [1] to Completely 

[6]) internships were structured and that the interns were oriented to the work they would be 

doing. As indicated in Figure K1, for 4/5 of Achieve there was strong agreement that someone 

explained the work of their organization and explained its policies and procedures. Three 

quarters of the interns got clear directions on completing tasks. About 3/5 gave a strong rating to 

the use of a structured work plan with measurable goals. For the Discover interns, as can be seen 

in Figure K2 above, there was strong agreement among 4/5 of interns that someone explained the 

work of their organization and explained its policies and procedures. Three quarters of the 

interns got clear directions on completing tasks. About 3/5 gave a strong rating to the use of a 

structured work plan with measurable goals. Discover and Achieve interns responded in similar 

ways to these questions. 

Figure K1. Achieve interns’ responses on a scale of 1-6 on questions about orientation and structure at 

their internships. 

Center for Applied Research and Educational Improvement, University of Minnesota 

137 
 

 

 

Figure K2. Discover interns’ responses on a scale of 1-6 on questions about orientation and structure at 

their internships. 

Interpersonal experiences. Another set of questions asked about the extent (Not at all [1] to 

Completely [6]) to the kind of interpersonal experiences they had as interns. As seen in Figure 

K3 above, Achieve interns had strong responses to being made to feel comfortable (86%), having 

a supervisor who talked to them respectfully (90%), and having a supportive supervisor (85%). 

As seen in Figure K4 above, Discover interns’ responses were similar, with a slightly lower 

proportion of strong responses. Four-fifths of Discover interns had strong responses to being 

made to feel comfortable, having a supervisor who talked to them respectfully, and having a 

supportive supervisor. When asked about the sensitivity to cultural differences of their 

supervisors, 2/3 of Discover and just over 2/3 of Achieve interns responded with the strongest 

responses. 

Center for Applied Research and Educational Improvement, University of Minnesota 

138 
 

 

 

 

Figure K3. Achieve interns’ responses on a scale of 1-6 on questions about interpersonal experiences at 

their internships. 

Figure K4. Discover interns’ responses on a scale of 1-6 on questions about interpersonal experiences at 

their internships. 

Opportunities for learning. A number of questions asked interns about chances they had during 

the internship for learning and growth. As shown in Figures K5 and K6, more than half Achieve 

Center for Applied Research and Educational Improvement, University of Minnesota 

139 
 

 

 

 

 

(57%) and Discover (51%) chose strong responses about having a mix of short and long-term 

projects. More than three-quarters of Achieve (79%) and Discover (77%) interns chose strong 

responses about learning job-related skills in their internships. Responses were less strong to the 

question about how challenging their tasks were. Among Achieve interns 43% gave strong 

responses and 39% gave moderate ones. Among Discover interns 37% gave strong responses and 

42% gave moderate ones. 

Figure K5. Achieve interns’ responses on a scale of 1-6 on questions about opportunities for learning at 

their internships. 

Center for Applied Research and Educational Improvement, University of Minnesota 

140 
 

 

 

 

 

 

Figure K6. Discover interns’ responses on a scale of 1-6 on questions about opportunities for learning at 

their internships. 

Some of the questions about opportunities for learning asked interns how frequently, if at all, 

they had particular experiences. Given the nature of these experiences, one would expect them to 

happen at varying rates. For some of the experiences, more does not necessarily mean better.65 It 

is important for program staff to take note of the proportion of interns who indicated they never 

had particular experiences. Figures K7 and K8 show that four-fifths of Achieve and Discover 

interns reported that someone talked to them about future plans at least a few times over the 

summer. Four-fifths of Achieve interns reported learning about a career in the field at least a 

few times over the summer; under 3/4 of Discover interns reported that experience. Regarding 

specialized training (e.g., Excel) 1/4 of Achieve interns did not have any and another 1/4 had 

such training every day. Among Discover interns, fewer participated in specialized training; 42% 

had none. 

65 Imagine, for example, if a supervisor talked to an intern every day about the intern’s plans for the future. That 

would likely have been considered annoying and inappropriate to the intern rather than helpful. 

Center for Applied Research and Educational Improvement, University of Minnesota 

141 
 

 

 

Figure K7. Frequency of learning experiences reported by Achieve interns. 

Figure K8. Frequency of learning experiences reported by Discover interns. 

Voice and choice. In responses to questions about having choices about projects to work on and 

input on setting goals, 2/3 of Discover and Achieve interns gave strong responses. About 1/4 of 

each group gave moderate responses to those questions. Figures K9 and K10 show the 

distribution of interns’ responses to questions about the amount of choice they had. 

Center for Applied Research and Educational Improvement, University of Minnesota 

142 
 

 

 

 

Figure K9. Achieve interns’ responses on a scale of 1-6 on questions about opportunities for input and 

choice of tasks at their internships. 

Figure K10. Discover interns’ responses on a scale of 1-6 on questions about opportunities for input and 

choice of tasks at their internships. 

Supervision. Questions about supervision asked about the frequency of experiences with their 

supervisor. The amount of time spent on supervisory activities varied across internships in both 

Achieve and Discover. Results are displayed in Figures K11 and K12 above. There was a spread 

in the frequency of meeting to track work progress. Among Achieve interns, a fifth to a quarter 

did so a few times, once a week, a few times a week, and daily. The spread was similar for 

Discover interns. There was also a spread of responses on the frequency of feedback on 

performance. A small percentage of Achieve (13%) and Discover (16%) said they got feedback 

never or once. The rest were fairly evenly distributed from a few times over the course of the 

summer to every day, with about a fifth to a quarter of the respondents getting feedback a few 

times over the summer, weekly, a few times a week or every day. More than four-fifths of 

Achieve interns and just over 3/4 of Discover interns reflected on their accomplishments and 

Center for Applied Research and Educational Improvement, University of Minnesota 

143 
 

learning with their supervisors or mentors at least a few times during the summer. Fewer interns, 

both Achieve and Discover, reported using the hirability skills to talk about their performance. 

Forty-six percent of Achieve interns and 40% of Discover interns never did so. 

Figure K11. Frequency of experiences with supervisors as reported by Achieve interns. 

Figure K12. Frequency of experiences with supervisors as reported by Discover interns. 

Supervisor Perspective. Structure/Orientation. Supervisors also responded to one question about 

structure and orientation—on the use of a structured work plan with measurable goals. Their 

responses showed that there was variation across internships in the extent to which interns had 

structured work plans with measurable goals. As shown in Figure K13 above, about two-fifths of 

Achieve supervisors gave strong ratings to this item and another two-fifths gave moderate 

Center for Applied Research and Educational Improvement, University of Minnesota 

144 
 

 

 

 

 

 

 

ratings. There was some misalignment here with the interns’ responses (Interns—59% strong vs. 

Supervisors—44% strong). Just over half of Discover supervisors gave strong ratings to this 

item, whereas 61% of interns gave it a strong rating. 

Figure K13. Supervisors responses about the extent to which interns had structured work plans with 

measurable goals. 

Opportunities for learning. More than half of Achieve supervisors (57%) and Discover 

supervisors (57%) chose strong responses about having a mix of short and long-term projects. 

This aligned fairly closely with the interns’ responses. Over two-thirds of Achieve supervisors 

and over three-quarters of Discover supervisors chose strong responses about learning jobrelated skills. While Discover interns and supervisors were aligned on their responses, Achieve 

interns gave stronger responses (79%) to this question than did supervisors (68%). As shown in 

Figure K14 below, there was a noteworthy discrepancy between the supervisors and the interns 

about challenging tasks. More than half of Achieve supervisors (54%) gave strong ratings, 

whereas 43% of interns did. The differences between Discover supervisors and interns were 

greater. Sixty percent of Discover supervisors gave strong ratings; only 37% of interns did. On 

opportunities to learn about a career in the field, about 4/5 of Achieve supervisors said interns 

had experiences at least a few times over the summer; this was about the same as interns’ 

responses. About 3/4 of Discover supervisors said interns had such experiences; the interns’ 

responses were fairly close in their responses. 

Center for Applied Research and Educational Improvement, University of Minnesota 

145 
 

 

Figure K14. A comparison of interns’ and supervisors’ responses to how challenging tasks were at their 

internships. 

Center for Applied Research and Educational Improvement, University of Minnesota 

146 
 

 

 

 

 

 

 

 

 

 

 

Supervision. In the 2016 end-of-summer survey, supervisors were asked to estimate the amount 

of time they spent in a typical week on a number of activities: teaching job-related skills, giving 

feedback, doing check-ins, talking about the career field, and developing or reviewing the 

intern’s work plan. More than half of Achieve supervisors and two-thirds of Discover 

supervisors spent an hour or more each week teaching job-related skills to their interns. About 

three-fifths of all supervisors spent 15-45 minutes weekly developing or reviewing work plans, 

15-45 minutes weekly giving feedback to their interns66, 15-45 minutes weekly talking about the 

career field, and 15-45 minutes weekly checking in with their interns. 

As shown in Figure K15, even more supervisors than interns reported not using the hirability 

skills in giving feedback to their interns. More than half of Achieve (55%) and Discover (55%) 

supervisors said they never used the framework with their interns. 

Figure 25. Percentages of interns and supervisors who reported NOT using the Hirability Skills 

framework in feedback to interns. 

66 The majority of Achieve supervisors reported giving feedback to their interns a few times over the summer or 

once a week. Nearly a third gave feedback a few times a week or every day. There was a discrepancy between what 

supervisors and interns reported about frequency of feedback. Achieve interns reported getting feedback more 

frequently; close to half (45%) said they got feedback a few times a week or daily. Half of Discover supervisors also 

reported giving feedback to their interns a few times over the summer or once a week. Nearly half of Discover 

interns (47%) said they got feedback a few times a week or daily. 

Center for Applied Research and Educational Improvement, University of Minnesota 

147 
 

 

Time with Supervisor in a Typical Week (Data source: End-of-Summer Supervisor Survey) 

Center for Applied Research and Educational Improvement, University of Minnesota 

148 
Center for Applied Research and Educational Improvement, University of Minnesota 

149 
 

 

 

 

Appendix L 

Program Satisfaction 

Training: Intern Perspective 

Figure L1 Ratings of training by MS Discover participants. This figure illustrates the percentage of 

students who agreed or strongly agreed with statements about training. 

MS Discover. Approximately 316 MS Discover students completed a post-training survey. 

Figure L1 displays their highly favorable ratings of the training experience. Nearly all 

participants (97%) reported that their trainers were knowledgeable about the topics covered 

during training, that the trainer taught the lessons in a way that was clear (97%), that the trainer 

taught the lessons in a way that was engaging (91%). Eighty-seven percent agreed or strongly 

agreed that after the training they were ready for a job. 

Center for Applied Research and Educational Improvement, University of Minnesota 

150 
 

 

 

 

 

 

 

Figure L2. Ratings of training by Advanced participants. This figure illustrates the percentage of students 

who agreed or strongly agreed with statements about training. 

Advanced. Approximately 135 students completed a post-training survey for the special 

Advanced training. Figure L2 displays their ratings of the training experience. Nearly all 

participants (98%) reported that their trainers were knowledgeable about the topics covered 

during training, that the trainer taught the lessons in a way that was clear (99%), that the trainer 

taught the lessons in a way that was engaging (95%). There was a split among students about the 

novelty of the material presented in training—62% said a lot was new and 39% said it was not. 

Still, 95% found the information taught valuable. 

In response to questions about the relatively new way of delivering training for Advanced 

interns, 98% of participants appreciated learning from professionals from businesses who hire 

STEP-UP interns and 95% found the opportunity to select their training package made training 

more rewarding. Comparing the conference-style training to previous STEP-UP trainings, 95% 

agreed that the conference format was more engaging. In conclusion, 95% said they felt better 

prepared for an internship after the training. 

Center for Applied Research and Educational Improvement, University of Minnesota 

151 
 

 

 

 

Figure L3. Ratings of Training by HS Discover and Achieve participants. This figure illustrates the 

percentage of students who agreed or strongly agreed with statements about training. 

Achieve & HS Discover. Achieve and High School Discover training were conducted by 

AchieveMpls in the same location. The Achieve/HS Discover training had 1,462 students 

respond to the post-training survey. Figure L3 displays their ratings of the training experience. 

Nearly all participants (97%) agreed that their trainers were knowledgeable about the topics 

covered during training, that the trainer taught the lessons in a way that was clear (96%) and 

engaging (94%). Eighty-four percent agreed or strongly agreed that after the training they were 

ready for a job. 

Figure L4. Percentages of interns who agreed or strongly agreed about training. 

Center for Applied Research and Educational Improvement, University of Minnesota 

152 
 

 

As shown in Figure L4 above, at the end of the end of the summer Achieve and Discover interns 

agreed or strongly agreed that training helped them in their internships. As shown in Figure 10 

above, at the end of the end of the summer Achieve and Discover interns agreed or strongly 

agreed that training helped them in their internships. In response to a question about the extent to 

which training prepared them for their summer job, 85% of Achieve interns who responded to 

the survey (n = 469) agreed or strongly agreed that the training prepared them as did about 84% 

of Discover interns who responded (n = 401). In reflecting on the extent to which they used the 

skills learned in training at their jobs, 84% of Achieve interns who responded to the question (n 

= 470) said they used the skills from training in their jobs as did about 88% of Discover interns 

who responded (n = 403). It should be noted that both HS and MS Discover students responded 

to the Discover survey but participated in different trainings. 

Intern Suggestions for Improving Training 

In the end of summer survey, students mentioned training in the open-ended comments, 

especially when asked for their suggestions about improving the program. Most frequently the 

interns recommended that the training be shortened. Among the other suggestions were: 

 Change (or eliminate) requirements for training for youth who have had internships 

before 

 Continue training during the summer so interns can get support they need 

 Provide real scenarios for role play 

 More hands-on experiences, including resume and cover letter writing, other professional 

writing 

 Match interns one-on-one with previous interns 

 Provide more training for specific jobs 

 Increase the number of people who come in to talk about their jobs and careers 

 Extend the training hours over more days 

In many of the open-ended comments from Discover interns, it was difficult to differentiate 

whether these comments applied to the weekly classes during the summer or to the spring 

training sessions. 

Internships: Intern Perspective 

Exposure 

Discover interns who had previous experience with the program were asked about the change in 

the number of hours per week they could work. Of the 90 interns who responded to the question, 

nearly three-quarters preferred last year’s hours (up to 30 per week). 

Center for Applied Research and Educational Improvement, University of Minnesota 

153 
 

 

 

 

Supervisors 

Four hundred forty-five Achieve interns and 358 Discover interns responded to a question about 

the likelihood of staying in touch with their supervisor or mentor after the summer. As shown in 

Figure L5 below, Achieve interns were more likely to stay in touch with their supervisors or 

mentors (47% Achieve vs. 38% Discover were likely or very likely to stay in touch). While quite 

a few interns in both groups were not sure about keeping in touch (45% Discover, 42% Achieve), 

few were not likely to stay in touch (12% Discover, 8% Achieve) or not interested in staying in 

touch (5% Discover, 4% Achieve). 

Figure L5. Likelihood of staying in touch with STEP-UP supervisor. 

Support from STEP-UP 

The one form of support from STEP-UP that interns were asked about on the end-of-summer 

survey was the weekly emails that were sent to Achieve and Discover interns to provide 

important information. Most interns read emails from STEP-UP at least occasionally, defined as 

1-3 times over the course of the summer. A larger proportion of Achieve interns read emails 

weekly (32%) than Discover interns (19%). 

Center for Applied Research and Educational Improvement, University of Minnesota 

154 
 

 

 

 

 

 

Benefits 

Interns reported on learning of various kinds ranging from the personal to the professional. 

On a personal level, interns commented on discoveries they made about themselves. One 

Achieve intern wrote, “I learned that I already had valuable assets.” Another mentioned 

uncovering strengths and weaknesses, or, as one intern called them, “opportunities.” Another 

said that by learning what she enjoyed and what she didn’t, it helped her plan for the future. 

Interns noted the development of character traits and social skills. These included staying calm 

under stress, communication, teamwork, being assertive, taking criticism, time management, 

perseverance, patience, being responsible, talking to people whom you don’t know, and handling 

conflict. One wrote, “. . .I have grown as a person. I am more outspoken and confident. Also a 

much better self-advocate.” Repeatedly, interns commented on learning to work with people 

from different cultural backgrounds. One intern concluded, “I got to learn skills that are useful 

not only at work, but in daily life as well.” Some of the lessons learned were not easy ones. An 

intern wrote, “I have also faced challenges that one could experience within the workplace. 

Through STEP-UP I have also learned that not everyone that you work with will be respectful 

towards nor mature about situations. However, you have to stay open-minded.” 

Another kind of learning involved the acquisition of work-place specific skills like photography, 

conducting a SWOT analysis, and working with children. Their professional learning focused on 

the knowledge, skills, and mindset required in nearly every work place. These included things 

like working as part of a team, working independently, and writing and speaking in a 

professional setting. One intern wrote, “I have benefited by learning how a real workplace works 

together and communicates with each other.” They learned about workplace practices and 

culture: how to interview for a job, creating a resume, what to expect from a job, the value of 

networking, how to dress for work, punctuality. One intern said, “I was able to learn new skills 

such as learning the professional way to talk to people, how to write appropriate emails, and 

make a professional voicemail.” 

A number of respondents mentioned learning about the particulars of different career paths and 

industries. One intern’s comment reflected the kind of insight gained from direct experience, 

writing, “I received an inside look in the hospital, and what goes on behind the scenes.” Another 

said, “I’ve gotten a real look at how a law firm operates, and I’ve gotten experience that a 16-

year old can’t find anywhere else.” They found that their summer jobs opened up their thinking 

about potential careers, and also helped them clarify things they do not want to do. It allowed 

them to uncover their interests and capacities they were not aware of. One intern wrote, “I 

learned new things about myself and I also discovered what I want to do for my future because 

of my job.” Another commented about the way STEP-UP helped test and clarify interest in a 

Center for Applied Research and Educational Improvement, University of Minnesota 

155 
 

 

 

 

 

 

 

 

particular field saying, “This summer job gave me the opportunity to see if teaching is a career 

path I would want to consider and it gave me the opportunity to see how I would be as a teacher. 

I loved working with the children and I love how I was able to be involved in their learning and I 

got to see them make progress each day which was wonderful. It gave me the experience I 

wanted and it helped me a lot.” 

Many wrote about developing relationships during the internship experience with peers as well 

as with adults who served as co-workers and supervisors. They appreciated the social aspects of 

their experience as well as the value of these networks for the future. Interns commented about 

having people who are guiding and supporting them. One said, “participating in the STEP-UP 

program helped me establish a network of people who I can ask any advice about my 

personal/professional life.” Another, commenting about the chance STEP-UP gave to broaden 

one’s professional network, said, “Who would ever [have] thought the CEO of U.S. Bank will 

know you by name!” 

One benefit that ran through many of the interns’ comments was about making money. Beyond 

earning money, interns learned to value it, to budget it, to manage it and to save it. Interns 

reported on using it to help support family and to purchase clothing and other things they needed. 

Comments about money were especially frequent among Discover interns. A summary of 

perceived benefits appears in Figure 32 below. 

Intern Suggestions for Improving Internships 

The end of summer survey asked interns for suggestions on how to improve the STEP-UP 

program for suggestions related to training and classes. Many comments said they like the 

program as it is and didn’t make suggestions. There were numerous interns who did suggest 

ways to improve the program. 

Placement. Regarding the placement process, several interns suggested that there be more varied 

job opportunities and more jobs to accommodate more interns each year, and greater choice for 

students about their work placement. Some wished that they were informed earlier about whether 

or not they got a job. Others wanted jobs better aligned with their interests. Discover interns 

thought clearer job descriptions would be helpful. 

Hours. Many students expressed a desire for more work hours each week and over the course of 

the summer. Many Discover interns, in particular, commented about more hours. Along with 

comments on more hours were requests for better pay. 

Programming. Several interns suggested that the program should organize and provide 

opportunities over the summer for interns to come together with others to talk about their job 

experiences. Others suggested adding social programs for interns. 

Center for Applied Research and Educational Improvement, University of Minnesota 

156 
 

 

 

 

Supervision. A few interns said their supervisors could have been better prepared with work for 

them to do. Others said they would have liked more contact and feedback from supervisors. 

Others talked about having more clarity about what the job would or could involve with greater 

specificity about tasks and responsibilities. 

Internships: Supervisor Perspective 

Time Commitment. In the end-of-summer survey, supervisors were asked how the actual time 

they spent training and supervising interns compared to the time they expected to spend. As 

Figure L6 shows, the majority of supervisors in both Achieve and Discover spent about the same 

amount of time supervising interns as they expected. Among Discover supervisors who 

responded to the survey, 70% said the actual time was about what they expected; among Achieve 

supervisors who responded to the survey, 64% said that the actual time was about what they 

expected. About 20% of Achieve supervisors and 21% of Discover supervisors found the actual 

time they spent working with interns was more time than they expected. 

Figure L6. Percentages of supervisors who spent the amount of time they expected supervising interns. 

Support from STEP-UP. Supervisors responded to questions about various forms of support 

provided by the program staff. 

Supervisor Orientation. One hundred seventy-four Achieve supervisors responded to questions 

on the end-of-summer survey about the supervisor orientation, and the vast majority found the 

supervisor orientation to be helpful in many aspects. Half of them found the orientations 

moderately helpful for preparing for the internship (49%) and another 41% found it very helpful. 

A little over half (53%) found their orientations moderately helpful for supervising their interns 

over the summer and about a third (33%) found it very helpful. 

Center for Applied Research and Educational Improvement, University of Minnesota 

157 
 

 

 

Of the Discover supervisors, 88 responded to questions about their orientation. Most of them 

found the orientations moderately helpful (45%) or very helpful (47%) for preparing for the 

internship. Half the supervisors (49%) found their orientations moderately helpful for 

supervising their interns over the summer and another 40% found it very helpful. 

Job Coaches (Achieve staff members) and Monitors (Discover staff members). During the 

summer job coaches (Achieve) and monitors (Discover) oversaw internships and checked in with 

supervisors. Supervisors from both Discover and Achieve reported high levels of support from 

these staff members during the summer. Among Discover supervisors, 88% of Discover 

supervisors who responded to the question (n = 89) said responses from monitors were timely. 

Among Achieve supervisors who responded to the question (n = 175), 94% said responses from 

coaches were timely. In response to how helpful the coaches and monitors were, 90% of 

Discover supervisors and 98% of Achieve supervisors found them to be helpful. 

Handbook/Tools. More than 4/5 of Discover supervisors (84%) used the handbook provided by 

STEP-UP. When asked what they found helpful, the most frequent answers were dates and 

contact information and information about timesheets and the payroll process. Other mentioned 

expectations, tips for working with interns and preparing for the first day. In response to the 

question about what could be added to the handbook to make it more helpful there was a range of 

suggestions including and Frequently Asked Questions section, more information about the 

classes the interns take, skills to work on with the interns, blank copies of forms, and a list of 

whom to contact for different types of questions/issues. 

Fewer Achieve supervisors reported using the handbook—59%. For those who used it the most 

frequently mentioned components were hirability skills and goal setting, logistics and dates, 

templates, guidelines for coaching and giving feedback, ideas for keeping the intern engaged, 

and work planning information. In response to the question about what could be added to the 

handbook to make it more helpful, suggestions included having a section on working with 

interns with special needs, case examples, more on providing constructive feedback, whom to 

contact for different issues, a printable timesheet, and simplified forms. 

Supervisors were asked about the tools provided by STEP-UP to use with their interns including 

the MHA Labs Hirability framework. As reported above, many supervisors did not use the 

framework. Figure L7 shows that among those who did use the framework, nearly half found the 

tool moderately helpful. 

Center for Applied Research and Educational Improvement, University of Minnesota 

158 
 

 

Figure L7. Supervisors’ responses on a scale of 1-6 to how helpful they found the MHA Labs Hirability 

Skills framework. 

Emails. In 2016 the Achieve and Discover weekly emails to supervisors included different 

material. Both contained a section on requests and reminders; both groups of supervisors found 

that section the most valuable. 

Supervisor Suggestions for Improving Internships 

In response to, “What suggestions do you have for how the program could be improved in the 

future,” 175 supervisors wrote comments. Many said the program was great and they had no 

suggestions. Several themes emerged from the other comments. 

As in the 2015 survey, many supervisors commented on the importance of a good fit of students 

with the skills required in the internship; this was more frequently mentioned by Achieve 

supervisors. Others commented on improving the alignment between the interns’ interests and 

the job assigned to him/her. Supervisors in both programs expressed interest in having some 

involvement in selecting their interns. 

Regarding training, a number of comments addressed how to improve intern preparation. 

Supervisors requested better training in “soft skills” like work ethic and reliability and hard skills 

like professional writing. Other comments suggested that more training and networking 

opportunities be available to supervisors and interns throughout the summer. A few of the 

Achieve supervisors requested site visits from STEP-UP staff during the summer. 

There were comments about improving communication and support from STEP-UP program 

staff—more checking in by the associates to see how things are going and providing more 

Center for Applied Research and Educational Improvement, University of Minnesota 

159 
 

support to supervisors. Discover supervisors requested that clearer information be provided 

earlier about the number of interns assigned to their site. 

Among Achieve supervisors there were comments about streamlining communication (emails) 

and documents (hirability skills materials, check-ins). Someone in a law firm commented that 

these documents were geared to a corporate setting and could be made more flexible for other 

settings. 

Some of the suggestions addressed logistics. Among Discover supervisors there were numerous 

requests for streamlining the process of paying interns by having online submission of 

timecards/payroll. 

Suggestions from Participants for Program Improvement: Interns 

Training 

 Change (or eliminate) requirements for training for youth who have had 

internships before 

 Continue training during the summer so interns can get support they need 

 Real scenarios for role play 

 More hands-on experiences, including resume and cover letter writing, other 

professional writing 

 Match interns one-on-one with previous interns 

 More training for specific jobs 

 Increase the number of people who come in to talk about their jobs and careers 

 Extending the training hours over more days 

Internships 

 More jobs, more varied opportunities, more hours, better pay 

 Greater choice for students in their placements 

 Opportunities over the summer to come together with others to talk about their 

job experiences and/or to socialize 

Center for Applied Research and Educational Improvement, University of Minnesota 

160 
 

Suggestions from Participants for Program Improvement: Supervisors 

Training 

 More training in skills like work ethic and reliability
 

 More training in skills like professional writing
 

Internships 

 Better alignment between interns’ interests and sills and the jobs they get 

 Opportunities for training and networking for supervisors and interns during the 

summer 

 Streamline emails and documentation materials 

 More checking in by program staff during the summer 

Center for Applied Research and Educational Improvement, University of Minnesota 

161 
 

 

 

 

 

 

Appendix M 

ANCOVA: GPA in SY17, Controlling for GPA in SY16 

Testing Assumptions 

There was a linear relationship between SY16 and SY17 GPA’s for both Achieve and Discover 

as assessed by visual inspection of a scatterplot. There was homogeneity of regression slopes as 

the interaction term was not statistically significant for either Achieve (F (1,447) = 0.488, 

p = 0.485) or Discover (F (1,373) = 0.574, p = 0.449). There was a violation of the assumption of 

normality but that can be tolerated due to the sample size. Table 2 presents the adjusted means 

for interns and matched students in SY17 when we controlled for GPA in SY16. 

GPA SY17 (controlling for GPA SY16) 

ACHIEVE Adjusted Mean Standard Error 

Intern 2.805 .010 

Matched 2.756 .012 

DISCOVER Adjusted Mean Standard Error 

Intern 2.732 .016 

Matched 2.727 .019 

Center for Applied Research and Educational Improvement, University of Minnesota 

162 
 

 

 

 

Appendix N
 

Full Model Outputs: First Design (Comparison of Participants & Matches), Controlling for
 

Prior Achievement
 

GPA: Achieve 

Tests of Between-Subjects Effects 

Dependent Variable: GPA_SY17_2dig_trunc 

Type III Sum of Partial Eta 

Source Squares df Mean Square F Sig. Squared 

Corrected Model 249.406b 2 124.703 4540.377 .000 .953 

Intercept .938 1 .938 34.169 .000 .071 

GPA_SY16_2dig_trunc 242.502 1 242.502 8829.388 .000 .952 

PARTICIPANT .258 1 .258 9.391 .002 .021 

Error 12.304 448 .027 

Total 3760.966 451 

Corrected Total 261.710 450 

a. LEVEL = 1 

b. R Squared = .953 (Adjusted R Squared = .953) 

GPA Discover 

Tests of Between-Subjects Effects 

Dependent Variable: GPA_SY17_2dig_trunc 

Type III Sum of Partial Eta 

Source Squares df Mean Square F Sig. Squared 

Corrected Model 269.118b 2 134.559 2379.042 .000 .927 

Intercept .002 1 .002 .031 .860 .000 

GPA_SY16_2dig_trunc 267.188 1 267.188 4723.958 .000 .927 

PARTICIPANT .002 1 .002 .038 .845 .000 

Error 21.154 374 .057 

Total 3100.670 377 

Corrected Total 290.272 376 

a. LEVEL = 2 

Center for Applied Research and Educational Improvement, University of Minnesota 

163 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

b. R Squared = .927 (Adjusted R Squared = .927) 

Attendance 

Comparison of Rate of Attendance for Interns and Matched Students SY16 and SY17 

Mean % 

attendance 

SY16 

SD N Mean % 

attendance 

SY17 

SD N 

ACHIEVE Intern 0.9495 0.0624 225 0.9294 0.0815 225 

Matched 0.9337 0.0721 136 0.9130 0.0946 136 

DISCOVER Intern 0.9508 0.0576 178 0.9370 0.0708 178 

Matched 0.9477 0.0582 111 0.9235 0.0890 111 

Comparison of Attendance for Interns and Matched Students, Controlling for Attendance in SY16 

Estimate Standard Error z-value p 

ACHIEVE Intercept -5.31870 0.14593 -36.447 <2e-16 *** 

Attendance 16 8.41456 0.16073 52.352 <2e-16 *** 

Participant or not 0.07440 0.03162 2.353 0.0186* 

DISCOVER Intercept -6.34490 0.19894 -31.894 <2e-16 *** 

Attendance 16 9.50747 0.21536 44.146 <2e-16 *** 

Participant or not 0.19654 0.03691 5.325 1.01e-07*** 

*Estimate –the estimated parameter is on the log odds scale. 

**z – the Z statistic 

Referrals 

Level 1 = Achieve, Level 2 = Discover 

Variables in the Equation 

LEVEL 

1 Step 1a PARTICIPANT 

Yes or No to 

Referrals in SY16 

Constant 

2 Step 1a PARTICIPANT 

Yes or No to 

Referrals in SY16 

B 

.107 

1.469 

-3.860 

.382 

2.194 

S.E. 

.583 

.623 

.494 

.458 

.432 

Wald 

.034 

5.560 

60.924 

.695 

25.838 

df 

1 

1 

1 

1 

1 

Sig. 

.854 

.018 

.000 

.405 

.000 

Exp(B) 

1.113 

4.346 

.021 

1.465 

8.974 

Center for Applied Research and Educational Improvement, University of Minnesota 

164 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Constant -3.439 .419 67.422 1 .000 .032 

a. Variable(s) entered on step 1: PARTICIPANT, Yes or No to Referrals in SY16. 

Removals 

Level 1 = Achieve, Level 2 = Discover 

Variables in the Equation 

LEVEL B S.E. Wald df Sig. Exp(B) 

1 Step 1a PARTICIPANT -.034 .629 .003 1 .957 .967 

Removals_Binary_ 2.180 .726 9.005 1 .003 8.844 

SY16 

Constant -3.957 .516 58.812 1 .000 .019 

2 Step 1a PARTICIPANT .027 .425 .004 1 .949 1.028 

Removals_Binary_ 2.000 .489 16.698 1 .000 7.385 

SY16 

Constant -2.831 .341 69.138 1 .000 .059 

a. Variable(s) entered on step 1: PARTICIPANT, Removals_Binary_SY16. 

Suspensions 

Level 1 = Achieve, Level 2 = Discover 

Variables in the Equation 

LEVEL B S.E. Wald df Sig. Exp(B) 

1 Step 1a PARTICIPANT -.371 .509 .530 1 .466 .690 

Suspensions_Binar 2.499 .571 19.149 1 .000 12.168 

y_SY16 

Constant -3.352 .394 72.386 1 .000 .035 

2 Step 1a PARTICIPANT 1.079 .426 6.405 1 .011 2.940 

Suspensions_Binar 2.535 .428 35.090 1 .000 12.619 

y_SY16 

Constant -3.314 .397 69.574 1 .000 .036 

a. Variable(s) entered on step 1: PARTICIPANT, Suspensions_Binary_SY16. 

Center for Applied Research and Educational Improvement, University of Minnesota 

165 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

ACT Composite Scores 

Level 1 = Achieve, Level 2 = Discover 

Tests of Between-Subjects Effects 

Dependent Variable: ACT Scale Score - Composite 

Type III Sum of Partial Eta 

LEVEL Source Squares df Mean Square F Sig. Squared 

1 Corrected Model 644.925a 2 322.462 17.239 .000 .165 

Intercept 967.231 1 967.231 51.709 .000 .229 

GPA_SY16_2dig_tr 644.913 1 644.913 34.477 .000 .165 

unc 

PARTICIPANT 17.679 1 17.679 .945 .332 .005 

Error 3254.736 174 18.705 

Total 53033.000 177 

Corrected Total 3899.661 176 

2 Corrected Model 343.545b 2 171.772 10.750 .000 .301 

Intercept 192.173 1 192.173 12.027 .001 .194 

GPA_SY16_2dig_tr 328.253 1 328.253 20.544 .000 .291 

unc 

PARTICIPANT 3.206 1 3.206 .201 .656 .004 

Error 798.908 50 15.978 

Total 15358.000 53 

Corrected Total 1142.453 52 

a. R Squared = .165 (Adjusted R Squared = .156) 

b. R Squared = .301 (Adjusted R Squared = .273) 

MCA Reading 

Level 1 = Achieve, Level 2 = Discover 

Tests of Between-Subjects Effects 

Dependent Variable: READING SCALE SCORE 

Type III Sum of 

LEVEL Source Squares df Mean Square F Sig. 

1 Corrected Model 316.532a 2 158.266 .752 .528 

Intercept 76976.851 1 76976.851 365.917 .000 

Center for Applied Research and Educational Improvement, University of Minnesota 

166 
 

 

 

 

 

 

 

PARTICIPANT 3.341 

GPA_SY16_2dig_trunc 301.949 

Error 841.468 

Total 7369890.000 

Corrected Total 1158.000 

1 

1 

4 

7 

6 

3.341 .016 .906 

301.949 1.435 .297 

210.367 

2 Corrected Model 6677.918b 2 3338.959 14.369 .000 

Intercept 10543164.980 1 10543164.980 45371.700 .000 

PARTICIPANT 188.328 1 188.328 .810 .370 

GPA_SY16_2dig_trunc 6637.865 1 6637.865 28.566 .000 

Error 28814.271 124 232.373 

Total 137180379.000 127 

Corrected Total 35492.189 126 

a. R Squared = .273 (Adjusted R Squared = -.090) 

b. R Squared = .188 (Adjusted R Squared = .175) 

MCA Math 

Level 1 = Achieve, Level 2 = Discover 

Tests of Between-Subjects Effects 

Dependent Variable: MATH SCALE SCORE 

Type III Sum of 

LEVEL Source Squares df Mean Square F Sig. 

1 Corrected Model 1552.970a 2 776.485 3.105 .054 

Intercept 4749828.324 1 4749828.324 18991.557 .000 

PARTICIPANT 38.157 1 38.157 .153 .698 

GPA_SY16_2dig_trunc 1530.608 1 1530.608 6.120 .017 

Error 12505.105 50 250.102 

Total 67315118.000 53 

Corrected Total 14058.075 52 

2 Corrected Model 1667.908b 2 833.954 3.373 .062 

Intercept 1048480.271 1 1048480.271 4240.049 .000 

PARTICIPANT 15.157 1 15.157 .061 .808 

GPA_SY16_2dig_trunc 1207.772 1 1207.772 4.884 .043 

Error 3709.203 15 247.280 

Total 23102716.000 18 

Center for Applied Research and Educational Improvement, University of Minnesota 

167 
 

 

 

Corrected Total 5377.111 17 

a. R Squared = .110 (Adjusted R Squared = .075) 

b. R Squared = .310 (Adjusted R Squared = .218) 

On track to graduate (credit-ready) 

Level 1 = Achieve, Level 2 = Discover 

Variables in the Equation 

LEVEL 

1 

2 

Step 1a 

Step 1a 

credtag_SY16 

PARTICIPANT 

Constant 

credtag_SY16 

PARTICIPANT 

Constant 

B 

4.014 

.988 

-.989 

4.398 

.926 

-1.563 

S.E. 

.446 

.448 

.395 

.489 

.475 

.451 

Wald 

80.840 

4.853 

6.268 

80.766 

3.801 

12.017 

df 

1 

1 

1 

1 

1 

1 

Sig. 

.000 

.028 

.012 

.000 

.051 

.001 

Exp(B) 

55.358 

2.686 

.372 

81.265 

2.525 

.209 

Center for Applied Research and Educational Improvement, University of Minnesota 

168 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Appendix O 

Employability Outcomes Analysis Using Wilcoxon Signed-Rank Test—Items with 

Statistically Significant Difference Pre-and Post 

The first analysis of these data considered each survey item independently; statistical 

significance was determined by p < .05. Then they were corrected for multiple comparisons 

(.05/25 survey items = .002). After corrections for multiple comparisons, some pre to post 

differences were no longer statistically significant. Those are marked with an asterisk (*) 

ACHIEVE 

QUESTION N z p r 

2. I bring energy and enthusiasm to the tasks I 

do. pre>post* 

326 -2.147 .032* .08 

4. I stay calm under stress. 329 -4.022 <.001 .16 

5. I accept criticism openly. 324 -4.114 <.001 .16 

6. I actively look for additional tasks when my 

own work is done. 

333 -3.391 .001 .13 

8. I break problems into smaller parts in order to 

solve them. 

333 -2.501 .012* .10 

17. When writing I can communicate in a 

professional way. 

327 -2.261 .024* .09 

18. I can describe my skills and strengths on a 

professional resume. 

331 -2.155 .031* .08 

19. I know what questions to expect in a job 

interview. 

331 -4.770 <.001 .19 

20. I know what questions are appropriate for 

me to ask during a job interview. 

330 -3.865 <.001 .15 

21. I can name two or more people I can ask for 

a professional reference. 

320 -2.774 .006* .11 

24. I intend to continue my education following 

high school. pre>post* 

317 -2.004 .045* .08 

26. I am hopeful about my future. pre>post* 320 -4.558 <.001 .18 

*See below for comparisons of pre- to post 

DISCOVER 

Center for Applied Research and Educational Improvement, University of Minnesota 

169 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

QUESTION N z p r 

2. I bring energy and enthusiasm to the tasks I 

do. pre>post* 

362 -3.139 .002 .12 

8. I break problems into smaller parts in order 

to solve them. 

360 -2.297 .022* .09 

12. I get my work done on time. 363 -2.317 .020* .09 

14. I know what clothes to wear for work. 361 -3.168 .002 .12 

15. When speaking, I can communicate in a 

professional way. 

362 -3.474 .001 .13 

17. When writing I can communicate in a 

professional way. 

362 -3.210 .001 .12 

18. I can describe my skills and strengths on a 

professional resume. 

362 -2.799 .005* .10 

19. I know what questions to expect in a job 

interview. 

361 -8.566 <.001 .32 

20. I know what questions are appropriate for 

me to ask during a job interview. 

363 -5.577 <.001 .21 

21. I can name two or more people I can ask 

for a professional reference. 

356 -4.937 <.001 .19 

23. I can make a valuable contribution to a 

workplace. 

351 -2.262 .024* .09 

26. I am hopeful about my future. pre>post 354 -4.177 <.001 .16 

*See below for comparisons of pre- to post 

Comparison of Agree/Strongly Agree Responses where pre>post, Achieve 

Item Pre (Agree/Strongly 

Agree) 

Post (Agree/Strongly 

Agree) 

N 

Q2. I bring energy and 

enthusiasm to the 

tasks that I do. 

89.4% 

1.2% disagreed or 

strongly disagreed 

84.3% 

1.2% disagreed 

326 

Q24. I intend to 

continue my education 

following high school. 

91% 

.9% disagree or strongly 

disagreed 

94.1% 

.9% disagree or strongly 

disagreed 

317 

Q26. I am hopeful 

about my future. 

91.7% 

.9% disagree or strongly 

disagreed 

91.9% 

2.2% disagree or 

strongly disagreed 

320 

Comparison of Agree/Strongly Agree Responses where pre>post, Discover 

Question Pre (Agree/Strongly 

Agree) 

Post (Agree/Strongly 

Agree) 

N 

Q2. I bring energy 

and enthusiasm to the 

tasks that I do. 

84.1% 

.3% disagreed or 

strongly disagreed 

75.2% 

.3% disagreed 

362 

Center for Applied Research and Educational Improvement, University of Minnesota 

170 
 

 

Q26. I am hopeful 94.2% 93.4% 354 

about my future. No one disagree or .3% disagree or 

strongly disagreed strongly disagreed 

Center for Applied Research and Educational Improvement, University of Minnesota 

171 
 

 

 

 

 

 

 

 

Appendix P 

Employability Analyses for Differential Experiences 

Outcome variable is Decrease/No Change/Increase in self-rating on item 

X = statistical significant difference among groups 

QUESTION Hours Per Week 

Worked (KruskalWallis) 

Weeks Worked 

(Kruskal-Wallis) 

Multiple Years: 

Achieve (Chisquare) 

Multiple Years: 

Discover (Chisquare) 

Q2. I bring energy and enthusiasm to the tasks that I do. 

Q3. I take responsibility for my actions instead of blaming 

others. 

Q4. I stay calm under stress. 

Q5. I accept criticism openly. 

Q6. I actively look for additional tasks when my own work 

in done. X 

Q7. I actively look for ways to help other people. 

Q8. I break problems into smaller parts in order to solve 

them. 

Q9. I think of several possible solutions to a problem. X 

Q10. I identify better ways to solve problems. 

Q11. I manage my time well. 

Q12. I get my work done on time. X 

Q13. I arrive on time. X 

Q14. I know what clothes to wear for work. 

Q15. When speaking, I can communicate in a professional 

way. 

Q17. When writing, I can communicate in a professional 

way. 

Q18. I can describe my skills and strengths on a 

Center for Applied Research and Educational Improvement, University of Minnesota 172 
 

 

professional resume. 

Q19 I know what questions to expect in a job interview. X X 

Q20. I know what questions are appropriate for me to ask 

during a job interview. X 

Q 21. I can name two or more people I can ask for a 

professional reference. 

Q22. I do not give up on tasks, even when I have trouble 

with them. 

Q23. I make a valuable contribution to a workplace. 

Q24. I intend to continue my education following high 

school. 

Q25. I know the education required for the work I am 

interested in doing. X 

Q26. I am hopeful about my future. 

Center for Applied Research and Educational Improvement, University of Minnesota 173 
 

 

 

Hours per Week/Employability: I get my work done on time
 

Group Decrease No Change Increase 

n % n % n % 

<10 2 25 3 37.5 3 37.5 

10-15 3 14 14 64 5 23 

16-25 2 6 17 47 17 47 

26-34 6 25 11 46 7 29 

35-40 10 29 20 57 5 14 

>40 1 20 2 40 2 40 

Weeks of Work/Employability: I know what education is required for the work I am interested in doing. 

Group Decrease No Change Increase 

n % n % n % 

4 weeks 0 0 3 60 2 40 

5 weeks 1 33 2 67 0 0 

6 weeks 4 22 5 28 7 39 

7 weeks 1 14 1 14 4 57 

8 weeks 3 18 6 35 8 47 

9 weeks 17 30 29 51 9 16 

10 weeks 4 36 5 46 1 9 

>10 weeks 4 33 8 67 0 0 

Center for Applied Research and Educational Improvement, University of Minnesota 174 
 

 

 

 

 

 

One vs. Multiple Internships/Employability--ACHIEVE 

GroupMultipleYears * Direction of Change Pre Post on Employability Q6 Crosstabulation 

Direction of Change Pre Post on Employability Q6 

0 1 2 Total 

GroupMultipleYears 1 Count 57 30 8 95 

% within GroupMultipleYears 60.0% 31.6% 8.4% 100.0% 

2 Count 21 11 12 44 

% within GroupMultipleYears 47.7% 25.0% 27.3% 100.0% 

Total Count 78 41 20 139 

% within GroupMultipleYears 56.1% 29.5% 14.4% 100.0% 

GroupMultipleYears * Direction of Change Pre Post on Employability Q9 Crosstabulation 

Direction of Change Pre Post on Employability Q9 

0 1 2 Total 

GroupMultipleYears 1 Count 42 33 20 95 

% within GroupMultipleYears 44.2% 34.7% 21.1% 100.0% 

2 Count 22 6 14 42 

% within GroupMultipleYears 52.4% 14.3% 33.3% 100.0% 

Total Count 64 39 34 137 

% within GroupMultipleYears 46.7% 28.5% 24.8% 100.0% 

Center for Applied Research and Educational Improvement, University of Minnesota 175 
 

 

 

 

 

 

 

GroupMultipleYears * Direction of Change Pre Post on Employability Q13 Crosstabulation 

Direction of Change Pre Post on Employability Q13 

0 1 2 Total 

GroupMultipleYears 1 Count 52 25 17 94 

% within GroupMultipleYears 55.3% 26.6% 18.1% 100.0% 

2 Count 25 4 15 44 

% within GroupMultipleYears 56.8% 9.1% 34.1% 100.0% 

Total Count 77 29 32 138 

% within GroupMultipleYears 55.8% 21.0% 23.2% 100.0% 

GroupMultipleYears * Direction of Change Pre Post on Employability Q19 Crosstabulation 

Direction of Change Pre Post on Employability Q19 

0 1 2 Total 

GroupMultipleYears 1 Count 37 42 16 95 

% within GroupMultipleYears 38.9% 44.2% 16.8% 100.0% 

2 Count 22 10 12 44 

% within GroupMultipleYears 50.0% 22.7% 27.3% 100.0% 

Total Count 59 52 28 139 

% within GroupMultipleYears 42.4% 37.4% 20.1% 100.0% 

Center for Applied Research and Educational Improvement, University of Minnesota 176 
 

 

 

One vs. Multiple Internships/Employability—DISCOVER 

GroupMultipleYears * Direction of Change Pre Post on Employability Q19 Crosstabulation 

Direction of Change Pre Post on Employability Q19 

Decrease No change Increase Total 

GroupMultipleYears 1 Count 13 31 52 96 

% within GroupMultipleYears 13.5% 32.3% 54.2% 100.0% 

2 Count 3 10 3 16 

% within GroupMultipleYears 18.8% 62.5% 18.8% 100.0% 

Total Count 16 41 55 112 

% within GroupMultipleYears 14.3% 36.6% 49.1% 100.0% 

GroupMultipleYears * Direction of Change Pre Post on Employability Q20 Crosstabulation 

Direction of Change Pre Post on Employability Q20 

Decrease No change Increase Total 

GroupMultipleYears 1 Count 20 37 40 97 

% within GroupMultipleYears 20.6% 38.1% 41.2% 100.0% 

2 Count 2 13 1 16 

% within GroupMultipleYears 12.5% 81.3% 6.3% 100.0% 

Total Count 22 50 41 113 

% within GroupMultipleYears 19.5% 44.2% 36.3% 100.0% 

Center for Applied Research and Educational Improvement, University of Minnesota 177