Tomodachi STEM @ Rice University
for Female Students
2017 – 2019 Assessment ReportDescription
This report summarizes the results of three years data collected to assess
student learning outcomes for students participating in the Tomodachi STEM @Rice University for Female Students, between 2017-2019. This assessment wascompleted by Dr. Cheryl Matherly, Vice President and Vice Provost forInternational Affairs at Lehigh University, and Junchi Zheng and Mary Beth Grove,
graduate students enrolled in the College of Education at Lehigh University.

Student OutcomesStudent learning was assessed on the following dimensions:Host country language development;
Students’ comfort level when interacting with people from other cultures;Student knowledge, skills, and understandings relevant to being effective researchers;
Students’ career decision making and planning

Assessment Plan
Five instruments were used to assess student learning. Students were asked to complete each of the assessments at thestart and conclusion of their research internships. Student participation in the assessments was voluntary. Whenpossible, the assessments were made available in Japanese and students were invited to respond to open ended
questions in Japanese.
Host country language development: Attitudes of EFL Secondary Students towards Learning English Language (AtLE)Students’ comfort level when interacting with people from other cultures: Intercultural Effectiveness Scale (IES)Student knowledge, skills, confidence, and understandings relevant to being effective researchers; Undergraduate Scientists – Measuring Outcomes of Research Experiences (USMORE)
 General Self-Efficacy Scale (GSE)Students’ career decision making and planning: Collaboratively developed questionnaire and open ended questions

Reasons for Participation (n=19)
0%

20%

40%

60%

80%

The International Dimension of the project

100%

Explore my interest in science

95%

Gain hands-on experience in research

78.90%Have a good intellectual challenge

78.90%

Clarify which field I wanted to study

63.20%

Clarify whether I wanted to pursue a science research career

52.60%

Clarify whether graduate school would be a good choice for me

42.10%

Participate in a program with a strong reputation

36.80%

I wanted to do research to enhance my resume

36.80%

Get good letters of recommendation
Work more closely with a particular faculty member

100%

31.60%
26.30%

120%

Attitudes toward speaking
and learning English

Attitudes Towards Learning English (AtLE)
The AtLE survey (Jafre Zainol Abidin, Pour-Mohammadi, & Alzwari, 2011) includes 38 items thatmeasure individuals’ overall perceptions about the value, challenges, and skills related to English
language learning; items are rated on a scale from (1) “strongly disagree” to (5) “strongly agree”.
The AtLE items include three subscales:
1.
2.
3.

Behavioral aspect of attitude◦ E.g. “I like to give my opinions when speaking English”

Cognitive aspect of attitude◦ E.g. “I am not satisfied with the way in which I speak English”

Emotional aspect of attitude◦ E.g. “I feel excited when I communicate in English with others.”

Attitudes Towards Learning English (n=29)
4.10

4.08
4.05

4.05

3.98

4.00

3.95

3.95
3.90

3.89
3.86

3.86

3.87

3.85
3.80
3.75
3.70

Overall

Behavioral(p = .02)

Pre

Post

Cognitive(p = .009)

Emotional

Attitudes Towards Learning English (n=29)
Individual Items
5
4.5

4.28

4.59
3.86

4
3.5
3

4.03

4.45

2.86

pre
3.38

post

4.03

3.76

3.97
3.34

2.68

2.5

4.34

2.29

2
1.5
1
0.5
0

Studying English
I don't get anxious
When I meethelps me have goodwhen I have tosomeone who speaks
relationships with answer a question in
English, I like tofriends.
my English class.
practice speaking(p = .026)(p < .001)
with him or her.
(p < .05)

I can summarize
important points
from a scientificarticle written in
English.
(p = .005)

Studying English I am able to think andI am not relaxedwhen I have to speak helps me create new analyze the content
English.
thoughts.
of scientific articles(p = .032)(p = .048)
written in English.
(p < .001)

Summary: AtLE Results
 The scores of Behavioral and Cognitive aspects of attitudes increased
significantly, suggesting that students demonstrated gains in theirconfidence in the use of English in personal and academic situations.
 Students use of English for scientific purposes showed marked
improvement over the course of the program.
 Students reported a decline in the Emotional dimension. This suggests thatby the end of the program, students demonstrated a less positive attitude
about speaking English. This assessment was completed at the end of theinternship, and may reflect that students were simply tired of speaking
English intensively as non-native speakers.

Intercultural Learning

Intercultural Effectiveness Scale (IES)*
 The IES (Mendenhall, Stevens, Bird, Oddou, & Osland, 2011) is a statistically valid and reliableself-assessment to evaluate intercultural competences between pre- and post-experience. It
includes an “Overall Intercultural Effectiveness Score,” and scores on three dimensions andsix subscales:

Continuous Learning – Perception of and willingness to engage intercultural interactions
and knowledge
 Self-Awareness - It assesses how aware the respondent is of personal strengths and how much

she reflects on this knowledge in order to pursue personal development and healthy relationships
with various kinds of people.

 Exploration - Fundamental desire to learn new things and strategically seek out new experiencesthat can cause learning or a change in the respondent’s perspective and behavior.

* The IES was only administered in 2018 and 2019.

Intercultural Effectiveness Scale (IES); cont.
Interpersonal Engagement – Level of dedication to, and interest in, developing positive cross-cultural
knowledge and intercultural relationships
 World Orientation - The degree to which the respondent is interested in other cultures and the people wholive in them

 Relationship Development - Initiating and maintaining relationships with people from other cultures

Hardiness – Level of, and willingness to engage, patience and understanding when engaging in

potentially stressful intercultural interactions
 Positive Regard - The degree to which the respondent naturally assumes people are trustworthy, hardworking
and generally good.

 Emotional Resilience – It measures the respondent’s level of emotional strength and her ability to cope withchallenging emotional experiences. It also assesses her capacity to recover quickly from psychologically andemotionally stressful situations and setbacks. How she manages these kinds of experiences influences hertendency to remain open, develop relationships, and interact effectively with others.

IES Pre & Post results (n=19)
5.00
4.50
4.00

3.92

4.13
3.67

3.90

3.50

4.17

4.36

4.05 4.09
3.55 3.58
3.05 3.06

3.00

3.24 3.35

3.39 3.29

Hardiness

Positive Regard

3.10

3.40

2.50
2.00
1.50
1.00
0.50
0.00

Continuous
Learning

Self-Awareness Exploration ** Interpersonal
Engagement
Continuous Learning** p =.039** p < .05*** p = .0015

World
Orientation

Relationship
Development

Interpersonal Engagement
Pre

Post

Hardiness

Emotional
Resilience ***IES Results, Summary
 Gain on the Continuous Learning dimension is statistically significant, “People who consistently strive to learn new things are more successful working across cultures
or demographic differences.” Suggests that students developed greater confidence in their ability to learn about othercountries and engage intercultural interactions.
 Gain on the sub-dimension Exploration is statistically significant.
 “Being open to ideas, values, norms, situations, and behaviors that are different from your own.”
 Suggests that students perceived that they became more inquisitive, curious, and open to new
ideas and experiences.
 Gain on the sub-dimension Emotional Resilience is statistically significant, “Level of emotional strength and ability to cope with challenging emotional experiences.” Suggests that students perceived that they coped better with challenging emotional situations
and had more energy to continue learning about the foreign culture or diverse groups anddevelop and maintain effective relationships with them.

Research Skills

Undergraduate Scientists- Measuring Outcomes of
Research Experiences Student Survey (USMORE-ss)
The USMORE–ss (Maltese, Harsh, & Jung, 2017) is composed of seven constructs to assess
students’ levels of experience, knowledge, and skills directly related to being a researcher:
1. Reading and understanding research literature (9 items)• Interpreting and critiquing the results presented in literature

2. Collecting research data (5 items)• Troubleshooting theoretical errors in research during data collection

3. Programming skills (3 items)• Computer programming for statistical analysis/modeling of numerical data

4. Analyzing and interpreting research data (6 items)• Interpreting statistical analysis of research in the field

USMORE (Cont.)
5. Scientific communication (5 items)•

Writing up research methods

6. Understanding of the field and the research process (5 items)•

Understanding the social or cultural practices of your field

7. Confidence in your research related abilities (5 items).
• Working independently to complete "basic" research tasks (e.g., data entry, weighing ofsamples.)USMORE-Subscales (n=29)
4.00

3.49

3.50
3.00

3.36

3.31
2.93

2.82
2.52

2.50

2.87

2.83

3.79

3.62

3.60
3.06

2.99

2.30

2.00
1.50
1.00
0.50
0.00

Reading andUnderstanding
Research literature(p < .001)

Collecting Resaerch
Data(p < .001)

Programming Skills(p = .01)

Analyzing andInterpreting Data(p = .01)
Pre

Post

Scientific
Communication(p = .005)

Understanding of the Confidence in yourField and the
Research Related
Research Process
Abilities(p = .001)(p = .009)

USMORE Results, Summary• Students reported statistically significant gains on all the subscales, inparticular in student confidence/understanding of:• Reading and Understanding the Research Literature• Collecting Research Data
• Understanding of the Field & Research Process• Confidence in Your Research Related Abilities• Suggests that they perceived greater confidence and enhanced skills in
conducting research.

Self Efficacy

General Self-Efficacy Scale Beliefs about own ability to effectively cope with and successfully navigate novel,stressful, and/or difficult life events (Schwarzer & Jerusalem, 1995)
 General Self-Efficacy is positively correlated with many other important constructs,
such as Optimism and Internal Control Beliefs, as well as positive attitudes toward
productive activities such as learning a new language
 The instrument consists of 10 items, rated on the scale of 1 = Not at all true; 4 =Exactly true Sample items:“I can always manage to solve difficult problems if I try hard enough”“I can solve most problems if I invest the necessary effort”“I can remain calm when facing difficulties because I can rely on my coping abilities”“If I am in a bind, I can usually think of something to do”

3.15

3.09

3.10

3.05

3.00

2.95

2.90

2.90

2.85

2.80

Pre

Post

Generalized Self-Efficacy Scale, Items (n=29)
GSE
3.50
3.00

3.10

3.28

2.97

3.21

2.972.97

2.72

3.00

2.76

3.103.14

3.10

2.62

2.83

2.86

3.10

2.973.03

2.97

3.24

2.50
2.00
1.50
1.00
0.50
0.00

When I am If I am in a bind, No matter what
If someone It is easy for me I am confident Thanks to my I can solve most I can remain
I can always
calm when confronted with I can usually comes my way, I
manage to solve opposes me, I to stick to my that I could deal resourcefulness, problems if I
am usually able
think ofa problem, I caninvest thefacingefficiently with I know how toaims andcan find meansdifficultto handle it.
something tousually findnecessary effort. difficulties
handleproblems if I try and ways to get accomplish my unexpecteddo.
severalbecause I canunforeseen
events.
goals.
what I want.
hard enough.
solutions.
rely on mysituations.
(p = .043)
coping abilities.
(p = .048)
Pre

Post

GSES Results, Summary
 Students showed gains on all the 10 items; 2 were statistically significant: I am confident that I could deal efficiently with unexpected events.
 Thanks to my resourcefulness, I know how to handle unforeseen situations.

 The gains with these items were consistent with the scores on the IES
Emotional Resilience sub-dimension, suggesting that student could deal
with tough situations with greater confidence.

Career decision making
and planning

Overall Gains (n = 19)
I am confident that I can be avaluable part of an international
research collaboration.
4.25

4.21

4.20

4.10

 100% of students reported the research experience
prepared them for graduate school.
 100% of students reported research experiences which
field of study they wanted to pursue.

4.05
4.00

 Strongly Agree: 26%
 Agree: 74%

3.95
3.90
3.85

 Excellent: 79%
 Good: 21%

 Strongly agree : 53%
 Agree: 47%

4.15

4.00

 100% of students had a positive experience withresearch.

Pre

Post

 100% of students reported research experience
prepared them for advanced coursework or thesis work.
 Strongly Agree: 47%
 Agree: 53%

Changes in Future Plans (n=19)
Thinking about your future, how likely are you to…?
4.50
4.00
3.50

3.58

3.89

3.00
2.00
1.50

2.58

2.50

2.50

1.56

1.53

3.16
2.00

3.53

2.47

3.53 3.63

3.05

3.32

3.42

3.74

3.58

3.89

1.00
0.50
0.00

Enroll in a Ph.D. Enroll in medical
Enroll in a
PursueWork in a
Pursue a career Pursue a career Pursue other Pursue a Ph.D.
program inor dental
program to earn certification as anational
in industry as a in academia as a internationalabroad?
science,school?
a different
teacher?
research/science scientist or faculty member opportunities asmathematics orprofessionallab?
engineer?
or researcher?
anengineering?
degree (i.e., law,undergraduate
veterinary
student?
medicine, etc.)(p = .027)
Pre

Post

Career decision making and planning
Results, Summary The students indicated increased confidence in plans to attend graduate
school, as a result of participation in this program.
 Students also reported high satisfaction with this program in the postsurveys.

Open-ended Questions &Free Responses
Students provided free responses when asked about their experiences andfuture plan such as their gains in the program and post-graduation plans. Pre
and post responses are organized by theme.

Pre-survey responses (n=29)
Students’ indicated the following goals for participation in the program:
Conducting research (n = 18)

 “I wanted to experience scientific research.”

Improve English (n = 14)

 “I hope to gain English skills(especially speaking and listening)”

Make connections/Collaborate with researchers from different backgrounds (n = 11)

 “I would like to acquire scientific knowledge, improve my English skills and build up network connections.”

Students reported less confidence in their decisions about career and research before the program
started(n = 13)
“I am not sure if I have enough knowledge on my research field.”“I have thought about it (post graduation plan) but I am still uncertain.”Students expressed their lack of confidence in conducting research (n = 7)

“I'm not confident to conduct research well.”“I want to go to U.S. PhD program after I graduate, but I do not have enough confidence for reaching my dream.”Post-Survey Responses (n=29)
Students’ reported the following outcomes from participation in the programIncreased confidence in conducting research (n = 15): “It made me more clear what it was like to do research in the lab”
Network with other students and researchers (n = 10):
“Got a strong connection with TOMODACHI stem participants and host professors”
Improved English (n = 8): “I enjoyed talking in English”
Students reported greater self confidence (n = 10)
“Simply I am really confident about English, cultural, specialized skills. I realized I can be an influencer in the future!! and I want to
try tons of things/experiences even after TOMODACHI program ended.”Students reported greater self-awareness (n = 7)

“Exposed to other culture, new places, new people and new experiences, what I learned most was about myself: my interest, mybias, my tendency of behavior, my strength, my weakness, my way of thinking. ”
 “This program motivated me a lot not in a temporary way but in a long-lasting way. Through this program, I come to think about
my future goals in more detail and larger picture and they encourage me to move forward”

Assessments suggest that the program was successful withachieving the stated learning objectives.
Students reported more positive attitudes toward speaking
and learning English.
Students reported a greater interest and confidence in theirability to engage interculturallyStudents reported a greater confidence in theirunderstanding of various components essential to being
effective and knowledgeable researchers.

Conclusions Students indicated greater confidence in their adaptability
and their ability to respond to unexpected events andunforeseen situations.

Students indicated that participation in the program
positively affected their decisions about future graduate
education and/or career plans.

All assessment data is based on student self-reports ofperceived gains.
Research lab hosts did not provide complete post-program
student evaluations, so program was not assessed on direct
measures.

Limitations

Some instruments were not provided in Japanese, and may
have affected student understanding.

Consider conducting post program interviews (individual or
focus groups) to seek more information about programs
aspects that most directly impacted student learning.
Add evaluations of student research to be conducted by US
lab hosts to obtain direct measures of student lab
performance.

FutureAssessment

Track data of students’ post-program plans to determine
student likelihood to pursue stated graduate school andacademic plans.

References
Mendenhall, M. E., Stevens, M. J., Bird, A. L. L. A. N., Oddou, G. R., & Osland, J. S. (2011). Interculturaleffectiveness scale: technical report. Chesterfield: The Kozai Group Inc.
Maltese, A., Harsh, J., & Jung, E. (2017). Evaluating Undergraduate Research Experiences—
Development of a Self-Report Tool. Education Sciences, 7(4), 87.
Abidin, M. J. Z., Pour-Mohammadi, M., & Alzwari, H. (2012). EFL students' attitudes towards learning
English language: The case of Libyan secondary school students. Asian social science, 8(2), 119.
Schwarzer, R., & Jerusalem, M. (1995). Generalized self-efficacy scale. Measures in health psychology:
A user’s portfolio. Causal and control beliefs, 1(1), 35-37.