# Presentation: 950886

## Findings from a National Longitudinal Study of the RISE Program

- P. Wesley Schultz
- California State University, San Marcos

- Presented at the 2009 NIGMS Council Meeting, January 23, 2009, Washington DC.

## A Preliminary Findings from a National Longitudinal Study of the RISE Program

**Research Team:**
- Mica Estrada-Hollenbeck		Richard Serpe
- Anna Woodcock			David Morolla
- Randie Chance				Paul Hernandez
- Maria Aguilar				Perla Rivas
- Victor Rocha				Brian McDonald

## Minority Training Programs

- NIH values diversity
- Underrepresentation is an important issue
- STEM and biomedical sciences
- Women in science
- Racial and ethnic representation (Black, Native American, Hispanic/Latino)
- Many NIH-funded programs promote research careers  among members of underrepresented groups
- Primarily through NIGMS-MORE

## Do These Programs Work?

- 1. National-level Data
- 80% increase in minority PhDs since 1977
- 62% increase in minority population during same period
- 2. Site-specific Data
- Each NIH program submits yearly progress reports, and longer-term evaluation results
- Typically track their student success, but what about a control group?

## Do These Programs Work?

- 3. Large-Scale Evaluation
- Assessment of NIH Minority Research and Training Programs (Phase III). *Science*, 2006.
- Surveyed students from 47 programs (1970 -99)
- Substantial limitations in finding former students
- 5,371 identified participants, of which data was obtained from 739 (only 83 trainees funded as undergraduates--19 U*STAR, 31 Bridges, 33 COR)

**Notes:**
“The outcomes for the MARC program are less than anticipated, with less than 15% of all MARC students completing the Ph.D. in biomedical or behavioral sciences” (NIGMS White Paper, 2006)

## Problems with Existing Data

- 1. Programs can “cherry pick” students who are likely to succeed
- 	- These students are likely to succeed *without* the program (growing new talent versus harvesting abilities)
- 2. No control group
- 3. Long-term evaluations outside of funding scope
- 4. No way to examine the “mechanisms” of success
- 5. Retrospective accounts can be biased

## Overview: The Science Study

- Prospective, propensity matched control
- Longitudinal study of R.I.S.E. and MARC students
- Participants from 25 programs nationwide
- Twice yearly surveys from students
- Matched control group
- For each R.I.S.E. or MARC student, we find a similar student who does not go through the program
- _Matching variables_: ethnicity, gender, major, GPA, intention to become a scientist, enrollment level (LD, UD, Grad)
- _Secondary matching_: age, parental education, community college transfer, English as first language

## Overview: The Science Study

## Longitudinal Panel

- Entering Panel N=1,228 (Fall 2005)
- 457 R.I.S.E./MARC
- 457 Matched Control
- 157 Other funded (and 157 matched)

**Notes:**
63 people entered our panel when they were enrolled in a PhD program,  They were from UT San Antonio, Morehouse School of Medicine, and Florida International University, and University of Miami for the most part

23 were in RISE
3 in MARC
8 in other MTP
..and the rest were in the match pool..

## Longitudinal Panel

## Survey Data Collection

- Data collected through web interface
- _[www.TheScienceStudy.com](http://www.thesciencestudy.com/)_

## Survey Measures

- self-efficacy, scientific identity, task-goal orientation, value orientation, ethnic identity, integrative identity, satisfaction, self- esteem, etc.
- _Accomplishments/milestones_
- publications, presentations, research participation, conference attendance, degree completion, etc.
- _Perceptions of the academic environment _
- mentorship, belonging, financial support and MTP experience

- Survey coverage:
- 93.1%

- Response rates

## Questions we can 

- (ultimately) answer

- 1. Does participating in the RISE/MARC program cause an increase in the likelihood that a minority student will pursue a career in the biomedical sciences? (summative)
- 2. What are the types of activities that students involved in RISE/MARC programs are exposed to?
- 3. Are there some types of students who benefit more from the RISE/MARC program than others?
- 4. Are there elements of the RISE/MARC program that are linked with student success?(Research experience, faculty mentoring, financial support, motivated peers, resume of accomplishments)

**Notes:**
Summative evaluation. Formative evaluation. 
Outcome. Process. 

## Preliminary Findings

- Basic effects (summative outcome)
- Intention to become a scientist
- Graduation rates
- Applications and acceptance to doctoral programs
- 2. Program elements (mediators)
- Financial support, research experience, faculty mentorship
- 3. Reasons for success (process)
- Self-efficacy, identity, values

**Notes:**
Our preliminary findings center around our three overarching research objectives – to describe the impact of the RISE program at each stage in the journey toward a career as a research scientist. The mechanisms via which the RISE program has its impact, and the unique contributions of each element of the RISE program.

## Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

- Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

- Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

**Notes:**
 Analyses are based on students who were undergraduates (jr. or sr.) at W0. Intention to pursue career as biomedical scientist. Propensity score (W0) used as time invariant covariate. RISE (N=120), MATCH (N=294). Analysis limited to campuses with a RISE program. 

## Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

- Q: To what extent do you intend to pursue a career as a biomedical scientist? (0 – 10)

- C=1.58

**Notes:**
 Analyses are based on students who were undergraduates (jr. or sr.) at W0. Intention to pursue career as biomedical scientist. Propensity score (W0) used as time invariant covariate. RISE (N=120), MATCH (N=294). Analysis limited to campuses with a RISE program. 

## Note: Juniors and Seniors at screening (W0). Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. RISE = students continuously funded, and MATCH = students never funded by any program. Dropped = students who were at one time enrolled in RISE but did not complete it (N=55).

**Notes:**
 Analyses are based on students who were undergraduates (jr. or sr.) at W0. Intention to pursue career as biomedical scientist. Propensity score (W0) used as time invariant covariate. RISE (N=120), MATCH (N=294). Analysis limited to campuses with a RISE program. 

## Baccalaureate Graduation

- W0 Juniors and Seniors.
- MTP = Minority Training Program

**Notes:**
Use always groups

## Graduate School:

- Applications

- * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

**Notes:**
Use the always
This is not Med school

## Graduate & Medical School:

- Enrollment

- * W0 Juniors and Seniors who have graduated with B.A./B.S. MTP = Minority Training Program.

- 10%

- 13%

- 1%

**Notes:**
Pipeline

## 2. Program Elements (Mediators) 

- 2. Program Elements (Mediators)

**Support to attend professional conferences**

**Faculty Mentorship**

**Research**
**Experience**

**Paid Tuition **
**& Stipends**

**Tutoring**

**Invited Speakers **
**& Workshops**

## 2. Program Elements (Mediators) 

- Note: Table reported at the 2007 NIH progress report meeting. Results based on survey responses from 25 RISE directors, and 457 RISE students.

## 2. Program Elements (Mediators)

- Mediation analyses of programs
- Focus on three program elements
- Research Experience
- Faculty Mentorship
- Financial Support (hours spent in employment off campus)

**Notes:**
Students who were undergraduates (jr. or sr.) at W0. 
RISE = students continuously funded, and MATCH = students never funded
Mediation analyses based on subsample (N=101 RISE, N=200 Match)

## Mediation: Research Experience

*c*=1.58**

*c’*= .68*

*a*=.36 **

*b*=2.38 **

**Research Experience**

- RISE Status

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001
- Bootstrapped Indirect Effect: Mean = 0.68, *CI*99% = 1.31 to 0.45
- Sobel: Z = 4.67, p < .001

## Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded, and MATCH = students never funded by any program and enrolled on a RISE campus. Research is any research experience ever during undergraduate education. Intention to pursue career as biomedical scientist.

- Note: Change over time analyses conducted as a hierarchical linear model, with both linear and quadratic terms. Analyses are based on students who were undergraduates (jr. or sr.) at W0. Propensity score (W0) used as time invariant covariate. RISE = students continuously funded, and MATCH = students never funded by any program and enrolled on a RISE campus. Research is *any* research experience ever during undergraduate education. Intention to pursue career as biomedical scientist.

## Mediation: Faculty Mentor

*c*=1.58**

*c’*= 1.33**

*a*=.32**

*b*=.78*

**Faculty****Mentor**

- RISE Status

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .001
- Sobel: Z = 2.08, p < .05

## Mediation: Financial Support

*c*=1.60**

*c’*= 1.49**

*a*=-.26**

*b*=-.42ns

**Financial****Support**

- RISE Status

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01
- Sobel: Z = .85, ns
- Financial support is hours spent working off campus

## Slide 28

**Notes:**
Process: When we framed our research program we designed a process model that captures information about program participation, psychological process and academic outcomes over time.

## 3. Process of Influence

**Notes:**
Skip to 33 if time constraints.

## 3. Reasons for Success (process)

*c*=2.55 **

*c’*=2.28**

*a*=.28*

*b*=.98**

**Scientific Self-Efficacy (rule)**

- Research Experience

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01
- Bootstrapped indirect effect: mean = .27, CI99% .06 to .56
- Sobel: Z = 2.57, p<.01

## 3. Reasons for Success (process)

*c*=2.55 **

*c’*=1.80**

*a*=.56**

*b*=1.34**

**Scientific Identity ****(role)**

- Research Experience

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01
- Bootstrapped indirect effect: mean = .75, CI99% .33 to 1.23
- Sobel: Z = 4.37, p<.001

**Notes:**
C difference is .75.  

## 3. Reasons for Success (process)

*c*=2.51**

*c’=2.24***

*a*=.24**

*b*=1.08**

**Scientific Values **
**(value)**

- Research Experience

- Intention to Pursue a Career in the Biomedical Sciences

- Note: a, b and c path’s are unstandardized coefficients. * p < .05, ** p < .01
- Bootstrapped indirect effect: mean = .27, CI99% .01 to .60
- Sobel: Z = 2.43, p<.01

## Key Points

- 1. NIH programs for undergraduates (RISE and MARC) have a strong effect on student intentions to pursue a career in the sciences.
- 2. RISE and MARC undergraduates have higher graduation rates, applications to doctoral programs, and enrollment in graduate programs (preliminary).
- 3. RISE works primarily because it provides students with research experience.
- 4.

## Slide 34