Evaluation Report Brief

Greater Twin Cities United Way

AchieveMpls: STEP-UP

Office of Research and Evaluation, Corporation for National and Community Service August 2018

1 

What is the community challenge?

Underrepresented populations, particularly youth from 

low income families, youth of color, youth from 

immigrant families, and youth with disabilities, face 

significant barriers to employment. Compared to their 

peers, they are less likely to have the skills, knowledge, 

and social networks vital to obtaining meaningful and 

gainful employment and careers. As the workforce ages 

in the United States, it is vital that subsequent 

generations are well-equipped to take on the careers of 

the 21st century.

Program At-a-Glance

CNCS Program: Social Innovation Fund

Intervention: STEP-UP

Subgrantee: AchieveMpls

Intermediary: Greater Twin Cities United Way

Focus Area(s): Youth Development

Focus Population(s): Youth aged 14-21 years, facing 

barriers to employment

Community(ies) Served: Minneapolis-St. Paul, MN

What is the promising solution?

The STEP-UP program provides underrepresented youth aged 14 to 21 years 

with work readiness training, career-oriented internships, job coaching, and 

industry-focused certifications in a wide variety of fields. Youth in the 

Discover program (for ages 14-15) and the Achieve program (ages 16-21) are 

placed with either a school-year or summer internship. The program helps 

organizations diversify their workforce and build a base of young, skilled 

workers for the entire region. 

What was the purpose of evaluation?

The impact and implementation evaluations of STEP-UP program by Cindy Reich of the Center for Applied 

Research and Educational Improvement (CAREI) at the University of Minnesota began in 2014 and finished 

reporting in 2018. The evaluation team sought to assess the program’s overall impact on school-related 

outcomes (including academic achievement, attendance, and discipline) and employability-related outcomes 

using two different study designs: (1) a quasi-experimental design (QED) involving matching with propensity 

scores; and (2) a single group design comparing outcomes for students at the end of their first and second 

years of participation. The QED study included 497 STEP-UP participants matched with 339 similar youth 

from Minneapolis; the single group design study included 696 youth who completed internships with the 

program. The implementation evaluation, which focused on fidelity of the program, exposure, dosage, and 

participant satisfaction, included approximately 1,900 youth.

What did the evaluation find?

As a subgrantee of the Social Innovation Fund, AchieveMpls program engaged an independent evaluator to 

evaluate STEP-UP. The evaluation found the following significant results:

• Participation in the STEP-UP program had small but positive effects for youth in the program compared 

to similar youth who were not in the program for being on track to graduate. (Achieve: Cramer’s V=.153; 

Discover: Cramer’s V=.181). This effect persisted after controlling for whether a student was on track to 

graduate the previous year for Achieve interns (Odds Ratio = 2.7, p = .028), but moved to marginally 

significant for Discover interns (Odds Ratio = 2.5, p = .051). 

nationalservice.gov/research
Office of Research and Evaluation, Corporation for National and Community Service August 2018

nationalservice.gov/research 2 

• At the end of school year 2017, Achieve participants’ GPAs were slightly higher than the comparison 

group (partial eta-squared = 0.021) after controlling for GPA in SY2016. However, GPAs declined for both 

groups from 2016 to 2017. There was no significant difference in Discover participants’ GPAs compared to 

non-participants, and mean GPAs for both groups also decreased.

• There was a minimal, but statistically significant positive effect on attendance for STEP-UP participants 

compared to non-participants (about a 2% difference in attendance rate, Odds Ratio = 1.07 for Achieve and 

1.27 for Discover). However, attendance for both groups went down from 2016 to 2017.

• In a self-report pre-post survey, STEP-UP youth were more likely to report positive responses related to 

employability after completing their internships, including knowing what questions to ask in an interview 

and what questions to expect. Achieve participants were more likely to report being able to stay calm 

under stress and accept criticism openly. Discover participants were able to name people for job

references and communicate in a professional way.

 

Notes on the evaluation

The study evaluation met the SIF criteria for moderate level of evidence due 

to its rigorous design with a matched comparison group and its ability to 

demonstrate program effectiveness for at least one confirmatory outcome. 

The QED evaluation showed a small impact on the key school outcome of 

being on track to graduate, and minimal positive effects on GPA (for the 

older interns in Achieve) and attendance, although GPA and attendance 

decreased for both treatment and comparison groups overall. Other schoolrelated outcomes, such as how well students did on standardized tests such 

as the ACT were not found to be statistically significantly different between 

the STEP-UP youth and their peers who did not participate in the program. 

The evaluator noted that this might be due to the possibility that 

participation in STEP-UP is more closely associated with employmentoutcomes than school-related outcomes. 

Evaluation At-a-Glance

Evaluation Design(s): QED using 

propensity score matching and a 

single group design using pre-/posttests

Study Population: Underrepresented 

youth aged 14-21 years

(Independent) Evaluator(s): Cindy

Reich, CAREI, University of 

Minnesota 

This Evaluation’s Level of

Evidence*: Moderate

*SIF and AmeriCorps currently use different definitions of 

levels of evidence.

How is GTCUW AchieveMpls using the evaluation findings to improve?

The report offers suggestions for strengthening the STEP-UP program and evaluation:

• Align training curriculum for interns more closely with desired employability outcomes; students’ selfratings indicate that skills taught specifically in the training influence participants’ growth.

• Better align supervisor orientation and oversight with desired employability outcomes for interns and 

provide increased support throughout the internship period.

• Consider revamping the logic model to eliminate school outcomes that are not impacted by the 

program and continue to revise it as other outcomes are demonstrated to have positive effects.

• Focus on the use of non-academic outcome measures to measure program impact. From survey 

responses, the program appears to be expanding participants’ consciousness about the world of work, 

developing their networks in the professional community, and helping them imagine career pathways.

The content of this brief was drawn from the full evaluation report submitted to CNCS by the grantee/subgrantee. The section of the brief that discusses 

evaluation use includes contribution of the grantee/subgrantee. All original content from the report is attributable to its authors.

To access the full evaluation report and learn more about CNCS, please visit http://www.nationalservice.gov/research. 

The Social Innovation Fund (SIF), a program of the Corporation for National and Community Service (CNCS), combines public and private resources to grow the impact of innovative, 

community-based solutions that have compelling evidence of improving the lives of people in low-income communities throughout the U.S. The SIF invests in three priority areas: 

economic opportunity, healthy futures, and youth development.