Understanding the Impact ofSUNY Poly’s
S-STEM Scholar Initiative“Supporting Degree Completion in Engineering and Engineering
Technology Programs through Experiential Learning and SelfDirected Professional Development.”~Year 1 Evaluation Activities~
A Report To:

Prepared By:
Megan Mullins, Ph.D.
Ryan Castillo, Ph.D.(c)Mullins Consulting, Inc.
December 2023

Mullins Consulting
Inspired Social Research & Program Evaluation

2

Table of Contents
Introduction ............................................................................ 3
Year 1 Progress Toward Outcomes .......................................... 3
Extent to Which Program is Being Implemented as Planned ............................................ 3
Satisfaction with Program by Key Partners ....................................................................... 5
Identification of Supports and Challenges Influencing Implementation Plans, Processes,and Outcomes ................................................................................................................. 6

Toward the Future ................................................................... 93Introduction
In Fall 2022 SUNY Polytechnic Institute (SUNY Poly) contracted with Dr. Megan Mullins and herteam of evaluators to conduct evaluation activities for SUNY Poly’s newly awarded National
Science Foundation funded S-STEM Scholarship program titled, “Supporting Degree
Completion in Engineering and Engineering Technology Programs through Experiential
Learning and Self-Directed Professional Development.” The long-term goal of this program isto break down the barriers to degree completion within the programs of Civil Engineering, CivilEngineering Technology, Mechanical Engineering, and Mechanical Engineering Technology(CME&ET). In pursuit of this goal, SUNY Poly will provide a total of 65 one- year scholarships to20 unique students in CME&ET. Students selected as Scholars will include those who enter as
first year students as well as transfer students. Over the program’s duration, this project will
provide:This report presents key findings from evaluation work supporting Year 1 activities conducted
for SUNY Poly. As this is the first year of program implementation, and Scholars are yet to berecruited into the program (anticipated for Spring 2024) results focus on the formative
outcomes that best fit anticipated achievements within the first year: (1) the extent to which the
project is implemented as intended (timeline, participants, activities, cost) and (2) the extent towhich key partners (students, faculty, administration) are satisfied with program components.
To create the formative (progress) evaluation report this year, program evaluator Dr. Megan
Mullins, in partnership with the Principal Investigator, reviewed Y1 activities and the evaluator
conducted key partner interviews as planned to collect feedback on the program from eight
key individuals at SUNY Poly responsible for decision-making on implementation andsustaining program components. Their summary feedback informs a majority of the information
presented in this report.

Year 1 Progress Toward Outcomes
Extent to Which Program is Being Implemented as Planned
The program is being implemented as planned for its first year. Specifically, the program hascompleted the following Y1 goals:• Recruitment and marketing materials (including program website) developed;• Eligibility and admissions protocols established;• Know One Be One (KOBO) event held and evaluation feedback collected from event;• First Year Seminar curriculum developed;• Receipt of seven scholarship applications.

4

Additionally, feedback from the 2023 KOBO event indicates participants at this recruitment
event help students understand what the campus has to offer its students and can seethemselves as potential students at the Institute. All respondents “Strongly Agree” (5) that theirstudent host was welcoming and that they were able to explore several areas of campus, andall agree to some extent that can see themselves as a SUNY Poly student and that they would
recommend the program to others. However, several students express neutrality when asked
whether they plan to apply to the university, and one disagrees that they found course
instructors to be knowledgeable and engaging.

Figure 1: Participant Comfort & Event Outcomes (n=6)
My student host was welcoming (mean=5.00)

100.0%I was able to explore several areas of campus with mystudent host (mean=5.00)

100.0%

I would recommend participation in this program to myfriends and classmates (mean=4.83)

16.7%

83.3%

33.3%

I intend to apply to SUNY Poly (mean=4.33)

66.7%

I could see myself as a SUNY Poly student (mean=4.33)
The instructors in the class were knowledgeable and kept
my attention (mean=4.17)

16.7%
0%

Strongly Disagree (1)Disagree (2)

66.7%
33.3%

33.3%
20%

Neither (3)40%Agree (4)

50.0%
60%

80%

100%

Strongly Agree (5)Participants were also asked a series of open-ended questions regarding interactions with theirstudent host, program highlights, and recommendations for improvement. When asked torelay any comments regarding their student host, respondents describe their hosts asknowledgeable, informative, and friendly.
Select Quotes••••“Arianna was excellent, very knowledgeable and listened to my questions.”“Gabriel was a great host, very informative and friendly.”“Sara and the other students were fantastic.”“They were amazing to talk to.”Respondents were also asked to describe what they consider to be highlights of the event. Inresponse, two participants explain that they enjoyed discussing and asking questions about thecollege experience, two others underscore the campus tour and student shadowing, and one
describes the staff they encountered as friendly.

5

Select Quotes
••
•
•
•

“Getting to ask students at the college questions about the school.”“Getting to chat about and observe classes was really cool.”“The campus tour was excellent.”“Walking around the campus, attending classes and checking out dining hall.”“The friendly staff.”

When asked whether they have any recommendations for improving the KOBO program for
future participants, one respondent suggests more hands-on courses for the student
shadowing activity, and another recommends providing opportunities to tour the freshman
dorms.
Select Quotes• “Maybe a more hands on class to attend.”• “Would have liked to check out a freshman dorm room.”

Satisfaction with Program by Key Partners
SUNY Poly partners are excited for this program and most report that this program brings
together many university strengths into an articulated, cohesive approach to recruiting andretaining students into engineering pathways, whether it be the traditional engineering degree
or a degree in engineering technology. Most expect the ability to offer both programs topromising students in engineering combined with the strengthened grant-provided support ofscholarships and mentoring opportunities to increase retention in both programs. When asked
what, for them, will be some of the highlights of this initiative being a part of their institution,
prominent responses focus on the emphasis that will be placed on providing students withoptions along the engineering pathways and emphasis on first year experiences to boost
retention.
Select Quotes• “While they're both engineering, the engineering tech students are a little more hands-on.
Having those pathways for students to slide back and forth in year one, which wecurrently have, but with the S-STEM proposal, helping to provide more opportunities forenabling the students to realize really what it is that they're doing in engineering andengineering tech from a broader perspective. Because they don't always know what anengineer does, and then they walk in the door and, ‘Gosh, that's hard and I'm not sure I cando this.’ But having that support to enable them to see what they could be in four years. I
think that's exciting about the whole proposal.”• “I do see it as a large benefit to the campus community if it really will help to create aunique scholarship for students who would necessarily not be in a higher education
standpoint without this scholarship.”

6

•

•

“I mentioned to you earlier that everything at SUNY Poly is individualized and personalized.
So a student being able to participate in the S-STEM scholarship and received the oneon-one, mentorship, the financial support, all of the components that are encapsulated
through the grant, I can just see it being an incredible opportunity to relaunch ourretention of first year students. It [retention] is abysmal right now.”“I think this program is really critical for us to overcome some of those barriers probably
down the long road haul. Another is I think this program put a lot of emphasis on both theyear science and math courses. That's how important those first year is, especially in theengineering program for retention purposes. … The third part I want to mention is theeffort they put on low income or, in general, underrepresented group of students that canincrease our diversity background diversity population in the engineering program.”

Identification of Supports and Challenges Influencing Implementation Plans,Processes, and Outcomes
Results for this report section are reported from the preliminary Key Partner interview results.
When asked what institutional strengths participants see SUNY Poly providing the initiative,participants highlight newly hired positions that will strengthen the engineering programs, theopportunity the grant provides to pilot interventions and gather feedback from students on
programming effectiveness, and the commitment faculty have to mentoring students tosuccessful completion of their programs.
Select Quotes• “Here at SUNY Poly, we are a small campus community. We have 2,200 undergraduates,
which means that each one of our students, regardless of what year you're coming from,regardless of starting with us as a first year or transitioning into our programs, you're takencare of as a real person. And that's something that I felt firsthand as a student during mytime as a SUNY Poly student. But our institution really has that infrastructure already inplace to support students. And this would only benefit students further to give them a little
bit more support.”• “We're one of probably five or six campuses in the nation with this slate of engineering andengineering tech. That just really makes us unique. And we're a public institution, so thatalso makes us more affordable. And, again, having those multiple pathways for students toget that training and be either an engineer or engineering tech in a way that enhances theirretention, that's just exciting for us at SUNY Poly. And I think we've got a really strong
backbone of industry support and that's just been continuing to build and build and build.
And with our most recent announcement from the governor of a 44 million investment into
our campus, 18 of which is going to be going to College of Engineering and highly
impacting engineering and engineering tech students - it's exciting times for SUNY Poly. So
this S-STEM grant's going to really serve us well and I think we're providing that framework
to help it succeed as well.”

7•

•

•

“It's a good fit for us because we need those programs to help bolster our retention. That,
first and foremost, for me is a huge piece for us. Secondly, I think the people here who aretaking the lead on implementing have a proven track record of students engagements.
Again, because we're small, when our faculty are hired here, they know that they're going
to be doing academic advising. … that one-on-one personalized attention that we alreadygive to students, I think just makes us a great fit for the grant as well.”“We are finally getting permanent leadership in place. We should have a permanent deanof the College of Engineering here fairly soon - no longer an interim. After four years, wehave people who are wanting to see programs like this be successful.”“I think are our upper administration are very supportive of the program because we arereally trying to identify how we can improve the retention rate for our students and how wecan improve our students' science and math scale in their first one two years. … The grant
here is really a great opportunity for us to collect a lot of data on those aspects to help us
identify what works, what doesn't, and we can dedicate future resource and develop somestrategy that, after the grant is gone, we can still sustain.”

Key partners were also asked to consider and then identify concerns that the PI will need to
pay attention to when planning for program sustainability. The feasibility of maintaining
personnel required for this type of muti-faceted programming is the number one challenge
identified by respondents. Clear communication of program functioning is also identified as a
critical element to sustainability planning.
Select Quotes• “I think one of the major challenges would probably be recruitment. I have seen a lot ofstudents that do come from an academically or financially disadvantaged environment, they
don't feel that they can achieve these more difficult-type programs. … So one of thebiggest hurdles in my experience with this would be providing students with theknowledge that they know they'll succeed in this program because of the support that we'll
be able to provide for them. And that's not to say that they wouldn't be able to succeed
without the support, but providing that will just better them off in the time in higher
education.”• “Sometimes with these five-year programs, and probably it's not going to happen in thiscase because it's got clear year 1, 2, 3, 4 goals, don't lose sight of your deadlines. And soprogram management and people management. She's [PI] got a good team, but they've all
got to be corralled to make sure that they're all meeting their goals, whether it's thesummer program or the folks in the admissions office or the faculty programs that she'sgot. They've all got to be invested and stay invested. So that's always a challenge with
program management. … And then long-term sustainability, what's our next funding
source to help grow this program? If we find out what the magic sauce that works and try to
be actively looking for that, even starting to look at that even in year two or three, because
you can't start that development side of the sustainability equation early enough.”8••“I do think one of the challenges would just be the human capital to run the program andknowing that we have systems already put in place that could automate some of theexperience in terms of the administration, I just know how deeply involved Joyce is in
multiple facets of campus. We just adore her. … There's no better way to say it. And I love
how much she cares for our students. So without a doubt, I know that she is the best
person to be the PI on this grant in place of Carolyn, but I also know how much she has on
her plate and I just don't want her to feel overwhelmed or overworked when we canautomate some of those pieces because it does seem right now that a lot of it's a manualprocess.”“I find that keeping momentum for things going just as a general rule, not necessarily withthis program, but in general, I know it's sometimes challenging because we getcomfortable and we do get distracted with other things. So keeping those metrics
graduation rates or retention rates or whatever it is across your spectrum, just sending out
reports perhaps to faculty quarterly, semi-annually, whatever seems to make sense, I think
is a good way to hold yourself accountable. … It'll help everybody understand that theprogram is on track or maybe isn't on track.”

When asked to identify the impact they would like to see the program have, respondents stressthe importance of seeing Scholars succeed in completing the program and moving into
successful graduate school opportunities or post-graduation employment. Additionally,respondents would like to see best practices in increasing retention and diversifying a student
body developed and sustained after the grant has ended in order to benefit all students.
Select Quotes• “The programs, I think have the broader impact because we'll have these types of events
that a lot of the majority of the students will partake in. So I guess a bump in retention
would be great. And also, can we find any correlation with respect to our demographics?
Because I think they had some details in there with how the S stem might impact
demographics, whether it's underrepresented minorities and or I guess lumped into thatnowadays first in their family to go to college.
• “Seeing that longevity of, I started working with this program before I was even a student
at SUNY Poly. I benefited from this program and now I want to be an ambassador for all of
the really great things that it does. So to bring them on the road with us or to have themfacilitate a webinar or something like that. So I think identifying a strategic funnel forstudents to remain involved and then creating sustainable pathways for that to continue tohappen after people have left their roles.”9••“The first one is our retention rate in the engineering program. Although I know the grant ismore on mechanical and engineer technology, it still gives us the data we want to look at in
the future about how we can improve the retention rate, especially in the first two years.
And another one is [that] I want to see what strategy, based on the strategy that grantedusing, I want to see the improved student skills in math and science in their first two years.
That's somehow tied to the retention already. If they are successful in those, we probablyautomatically have a higher retention rate. And also the graduation rate, the four year
graduation rate down the road. That's bringing in the transfer student as well to see they
have a successful graduation and a successful career down the road. I think the final one I
want to see as successful is whether we can increase or diversify the student population in
our engineer program.”“To get them obviously in the door of Poly. But for the program itself, not only for them tosucceed in higher education, but then come out with a career that they love, come out withinternship opportunities that they've already explored, into connection that they've alreadymade, the ability to interview and the ability to have those skills that you don't necessarily
learn in a classroom, but that they will have access to here at Poly.”

Toward the Future
In this first year of STEM Scholar programming, the team began developing evaluative tools,
tracking program implementation strategies, gathering feedback on the annual KOBO event,and collecting data on key partner perception of the program. The program is on track toachieve its short-term goals, as the first cohort will be selected and welcomed in Spring 2024.
Conversation around key partner and KOBO participant feedback will inform programadjustments and communication strategies moving forward. As the program is still in its early
implementation phase, the evaluation team has no recommendations for program
improvement at this time.