# Presentation: 393756

## New Measures of RCR Mentoring, RCR Department Climate and Graduate Student RCR Preparedness in Psychological ScienceCelia B. Fisher, Ph.D.Adam L. Fried, M.A.Sabrina Goodman, M.A.Kaori K. GermanoFordham University Center for Ethics Education

- NINDS  1 R01 NS052877-01
- 2006 Research Conference on Research Integrity

- F  O   R  D  H  A  M     U  N  I  V  E  R  S  I  T  Y *New York City’s Jesuit University*

## Goals of Project

- Develop and validate measures to assess RCR student socialization and internalization in graduate education for psychological science.
- Administer measures to a large national sample of graduate students in psychology to compare RCR socialization and internalization across a wider range of diverse student, faculty, and departmental characteristics.

**Notes:**
The second point is low-lighted because we are only talking about the first point.
RELATED TO THE SECOND POINT (FROM THE PROPOSAL):  To test the hypothesis that RCR mentoring and RCR department climate are individually and conjointly related to RCR perceived competence. A secondary goal is to explore the extent to which student, mentor, and university characteristics influence student responses and relationships among variables. 

## Study Rationale

- Need for the development of adequate measures to evaluate educational factors that promote integrity in research (NAS, 1997)
- Research suggests that mentoring and departmental climate are important RCR educational influences.
- There have been several multidisciplinary approaches to RCR training (Swazey & Anderson, 1996). However,  measures are needed to empirically evaluate RCR socialization in the social sciences.

**Notes:**
Do we need to define RCR somewhere?
This is from the proposal – we can summarize this in the presentation if we think necessary: “At both the individual and institutional level RCR is embodied by a commitment to intellectual honesty and personal responsibility for one’s actions and to a range of practices including:  (a) intellectual honesty in proposing, performing, and reporting research; (b) accuracy in representing contributions to research proposals and reports; (c) fairness in peer review;  (d) collegiality in scientific interactions, including communications and sharing of resources;  (e) transparency in conflicts of interest or potential conflicts of interest;  (f) protection of human subjects in the conduct of research; (g) humane care of animals in the conduct of research; and (h) adherence to the mutual responsibilities between investigators and their teams (NAS, 1997).” 

## RCR Constructs in Need of Measures

- _RCR Socialization_
- RCR Mentoring
- RCR knowledge transmission
- RCR positive modeling
- RCR Departmental Climate
- RCR expectations and resources
- RCR practices

- _RCR Internalization_
- Student RCR  preparedness
- Student attitudes toward the research integrity of the discipline.
- Student RCR competence

## Overview

- Phase I: Item Development and Construct Validation
- Phase II: Scale Refinement

## Step I: Initial Development of RCR Measures

- 3 distinct scales were developed:
- RCR mentoring
- RCR Department climate
- RCR Graduate student preparedness
- Items were constructed from theoretical and empirical literature include RCR mentoring, departmental and competence from other disciplines and scales designed to evaluate transmission of research competencies in psychology.
- Items and format were refined through evaluation by ethics expert panel and student critique of content and format

## Step 2: Validation of RCR Measures

- Newly constructed scales administered to a national sample of doctoral graduate students
- Construct relevant standardized measures were administered for validation.
- Psychometric analyses:
- Item Analysis
- Reliability (inter-item)
- Construct Validity (with standardized measures)
- Predictive Validity (Multiple Regression)
- Factor Analysis: To identify subscales

## Step 3: Revision of RCR Measures (Phase II)

- Revisions made to original items based on the results of psychometric analyses during Phase I
- Construct 2 additional outcome measures
- Administer revised survey to a second national sample of doctoral graduate students
- Dropped standardized measures because construct validated
- Analyze and confirm psychometric properties of measures.

## Phase I

- Scale Development and Construct Validity

## Mentoring the Responsible Conduct of Research (MRCR)

- Designed to evaluate mentor’s RCR knowledge transmission, modeling, and social support.
- Format adapted from research by Swazey & Anderson (1996), Clark et al. (2000) and Hollingsworth & Fassinger (2002).
- 26 items on a 6-point Likert-type scale
** ****1=Strongly Disagree; 6=Strongly Agree**
- Example Items: “My Mentor...
**	****“****Taught me strategies for avoiding plagiarism in my writing”**
**	****“****Handled data ownership fairly”**
**	****“****Encouraged me to consider ethical issues relevant to my research **

**Notes:**
ADAM TO FINISH – where did you get the items from?  FROM THE PROPOSAL:  “The format of the MRCR was adapted from scales measuring research and professional mentoring conducted by Swazey and Anderson (1996) for Chemistry, Engineering, Microbiology, and Sociology, by Clark et al. (2000) for clinical doctoral training programs, and by Hollingsworth & Fassinger (2002) for research mentoring functions in counseling psychology.”

Items assessed mentoring in core areas of research ethics, such as avoidance of plagiarism, disclosure of confidential information, falsification of data, non-coercive incentives, fair assignment of authorship on publications, and appropriate debriefing procedures.
There were an approximately equal number of positively and negatively worded items; negative items were reverse-scored for analyses.

## Who is a Mentor?Scale Instructions to Students

*“**Research Mentor” refers to the faculty member who has/had the primary responsibility for supervising your master’s, doctoral or other graduate level independent psychology research.*
*IF YOU HAVE HAD MORE THAN ONE RESEARCH MENTOR, FOR THIS QUESTION AND FOR ALL THE QUESTIONS THAT FOLLOW, SELECT THE MENTOR YOU BELIEVE HAD THE GREATEST INFLUENCE (POSITIVE OR NEGATIVE) ON YOUR DEVELOPMENT AS A RESEARCHER.*

## MRCR Validity Assessment

- Advisory Working Alliance Index (AWAI; Schlosser & Gelso, 2001)
- Assesses graduate students’ perceptions of working alliance with advisor
- Three dimensions:
- Rapport
- Apprenticeship
- Identification-Individuation
- Examples:
- “I got the feeling that my mentor did not like me very much”
- “I do not want to be like my mentor”
- “My mentor did not encourage my input into our discussions”

**Notes:**
*the term “mentor” was substituted for “advisor” for this study

## RCR Department Climate Index (RCR-DC)

- Assessed presence and availability of RCR courses and materials, and positive and negative RCR faculty modeling.
- Adapted from questions developed by the Arcadia Institute to evaluate the research ethics environment of different disciplinary college programs (Swazey, Anderson, & Louis, 1993; Wright & Klomparens, 1998).
- 18 items on a 6-point Likert-type scale
- 1=Strongly Disagree; 6=Strongly Agree
- Examples: “In my program...
**Faculty members carefully monitor students’ work for plagiarism**
**Fraduate students are confused about their work roles and responsibilities within the department**
**Faculty members do not exploit graduate students**

**Notes:**
This slide is still too dense!

## RCR-DC Validity Assessment

- The Research Training Environment Scale–Revised–Short Form (RTES-R-S; Kahn & Miller, 2000)
- Evaluates instructional and interpersonal dimensions of the research training environment
- Good psychometric properties
- Utilized across a geographically diverse range of graduate programs
- Examples:
- Many of our faculty do not seem to be very interested in doing research
- There is a sense around here that being on a research team can be fun, as well as intellectually stimulating

## RCR Perceived Preparedness Scale (RCR-P)

- Taps the degree to which students feel their graduate training program prepared them to conduct responsible psychological research.
- Adapted from a questionnaire assessing graduate students’ knowledge of RCR options available to them (Brown & Kalichman, 1998)
- 23 items on a 6-point Likert-type scale
- 1=Strongly Disagree; 6=Strongly Agree
- Examples: “At this point in my research career, I feel my graduate training has prepared me to...”
**Maintain research records in a manner consistent with APA ethical standards**
**Appropriately debrief research participants**

**Notes:**
Items tapped knowledge about core ethical concepts in psychological research, including maintenance of research records, protection of confidentiality and avoiding harmful or exploitative multiple relationships.

## RCR-P Validity Assessment

- The Self-Efficacy in Research Measure
- (SERM; Kahn & Scott, 1997)
- Self-evaluation of students’ confidence in research design, writing, practical and quantitative skills
- Strong psychometric properties
- Examples:
- Keeping records during a research project
- Designing an experiment using traditional methods
- Writing the introduction and literature review for a dissertation

## Phase I: Participants

- 201 psychology graduate students enrolled in 13 geographically diverse psychology training programs.
- Students were mentored in the conduct of basic or applied research with human participants at the masters or doctoral level.
- Reflected the gender and ethnic distribution of the field
- mostly female (71%) and non-Hispanic White (81.1%)

## Phase I: Methods

- Participant Recruitment:
- E-mail announcements, posters and flyers
- Word-of-mouth
- Survey administration:
- Online Survey
- Firewalls to prevent identification of participants’ Internet Protocol addresses
- Study approved by the Fordham University IRB

**Notes:**
Participant Recruitment:
E-mail announcements, posters and flyers distributed by faculty at graduate programs on the east and west coast.
Over time, students at other psychology graduate programs became aware of the study and participated. 
Survey administration:
An online survey was constructed with firewalls to prevent anyone (including the investigators) from identifying participants’ Internet Protocol addresses.  

## Phase I: Procedure

- Procedures:
- Logged on to ‘researchmentor.org’
- Completed on-line survey:
- Basic demographic information
- Six measures (158 items)
- Submitted survey
- Retrieved $30 Barnes & Noble online gift certificate

**Notes:**
Students logged onto our website (researchmentor.org) with a username and password printed on the announcements advertising the study.
They entered basic demographic information and completed the six measures (158 items), which took approximately 30 minutes to complete.  
After they completed the survey, the clicked “submit”, which led them to a website where they were instructed to create their own username and password, where they could log on to a database to retrieve a Barnes & Noble online gift certificate worth $30.  

## Phase I: Results Overview

- Overall, all three RCR measures demonstrated:
- 	High inter-item reliability
- 	Good construct validity
- 	Good predictive validity

## Phase I: Results

- Item Analysis--Items were kept if they yielded acceptable:
- 	Item Discriminability
- 	Item Consistency
- 	Item Difficulty
- Cronbach alphas for full scales with items that were maintained:
- 	MRCR    α=.90
- 	RCR-DC α=.83
- 	RCR-P   α=.94

## Results: Construct Validity

- Correlations between newly constructed instruments and existing research scales:
- MRCR and AWAIS =.73 *
- RCR-DC and RTES-S-R =.65 *
- RCR-P and SERM =.63 *
- * p ≤ .001

## Results: Multiple Regression

- Do new RCR scales predict RCR Preparedness & Research Efficacy?
- RCR Preparedness:
- Adjusted R Square = .53 for predictor scales MRCR, RCR-DC and RTES-S-R, and date of doctoral degree
- Research Efficacy:
- Adjusted R Square = .30% for predictor scales MRCR, RTES-S-R and year of doctoral degree

**Notes:**
For RCR preparedness: MRCR, RCR-DC and RTES-S-R, and date of doctoral degree each independently and significantly contributed to RCR preparedness (cumulatively accounted for a 53% of the variance).

For Research efficacy: MRCR, RTES-S-R and year of doctoral degree each also contributed significantly to research efficacy

## Results: MRCR Factor Analysis

- 1.  RCR Knowledge Subscale
- 	Mentor guidance on specific human subject protection procedures (8 items; α =.81)
- 2.  Responsible Scientific Conduct Subscale
-   Mentor guidance on RCR activities not tied to subject protections (e.g. appropriate data storage, avoidance of plagiarism)
- (10 items α =.84)
- 3.  RCR Misconduct Subscale
- 	Instances when the mentor encouraged, permitted or modeled scientific misconduct (8 items; α =.79)

**Notes:**
Put in examples from previous scale
I don’t have the factor analysis so I just went through the MRCR and added examples that fit under the subscales.  

## FACTOR ANALYSIS RCR-DC

-    RCR-DC  Department Expectations and Resources  9 Items α = .76
-    RCR-DC  Faculty & Student RCR Practices   7 Items α = .70

**Notes:**
CBF TO DO

## Phase II

-   Psychometric Evaluation of Subscales
-   Construction of 2 new Student
-      Internalization Measures

**Notes:**
Drawing upon the psychometric analyses, Phase 2 further refined the instruments by grouping validated subscales into separate scales, adding an attitudes toward the RCR integrity of the discipline scale, and adding an APA Ethics Code knowledge scale.

## Phase II: Participants

- 241 psychology graduate and postgraduate students enrolled in 14 geographically diverse psychology training programs
- Students were mentored in the conduct of basic or applied research with human participants at the masters or doctoral level.
- Reflected the gender and ethnic distribution of the field
- mostly female (77%) and non-Hispanic White (83%)
- Recruitment and methods were similar to Phase 1.

## RCR-M

- RCR Mentoring Subscales

## RCR-M Knowledge Transmission

- Measures extent of research mentor’s RCR direct instructions and practical guidance (22 items)
- 6-point Likert-type Scale
- 1=Extremely Unhelpful; 6 = Extremely Helpful
- “My research mentor gave me helpful training about...:
- ...appropriate informed consent procedures”
- ...avoiding plagiarism in my writing”
- ...preparing an IRB application”
- ...how to protect participant confidentiality”

**Notes:**
Designed to tap the extend of their research mentor’s RCR direct instruction and practical guidance.
Each item was a completion of the phrase, “my research mentor gave me helpful training about,” and included statements such as, (read off slide).  Students were asked to respond to each item on a 6-point Likert-type scale.

## RCR-M Positive Modeling

- Measured extent of research mentor’s RCR modeling and supervisory style (22 items)
- 6 point Likert-type Scale
- 1=Extremely False; 6 = Extremely True
- “My research mentor...:
- ...conducted his/her own research ethically”
- ...monitored the ethical implementation of my informed consent procedures”
- ...initiated ethics discussions”
- ...discussed authorship of publications that might emerge from my research

**Notes:**
Designed to tap the extent of their research mentor’s RCR modeling and supervisory style.

- RCR Departmental Climate Subscales
- RCR-DC

## RCR-DC Expectation and Resources

- Assesses RCR content in departmental courses, course requirements, resources, and procedures. core areas of RCR that students would receive explicit instruction on from their mentor (15 items)
- 6 point Likert-type scale
- 1=Extremely False; 6 = Extremely True
- “In my graduate psychology department....”
- ... concern for the welfare of research participants is stressed in research courses”
- ... a major consideration is whether a research design protects participants’ autonomy”
- ...there is a high level of support for making ethically sound decisions”
- ...students are made aware of specific rules prohibiting research misconduct”

## RCR-DC Practices

- Assesses RCR positive and negative departmental student and faculty practices (10 items)
- 6 point Likert-type scale
- 1=Extremely False; 6 = Extremely True
- “In my graduate psychology department...
- ...faculty and students engage in ethically questionable research practices”
- ...research funds are misused”
- ...graduate research assistants are confused about their roles &

## Student RCR Preparedness Scale

- Assesses degree to which students feel their graduate training program has prepared them to conduct responsible research. (23 items)
- 6 point Likert-type scale
- 1=Extremely False; 6 = Extremely True
- “At this point in my research career I feel by graduate training has prepared me to...
- ...assign appropriate authorship credit for publications”
- ...know when it is ethically appropriate to disclose a research participant’s confidential information”

## New RCR Internalization Scales

- Attitudes toward RCR integrity of discipline of psychology
- Familiarity with American Psychological Association Research Related Ethics Code Standards

## RCR Expectations:Student Attitudes toward the Research Integrity of the Discipline

- Assesses influence of graduate training on student expectations for the responsible conduct of research in the field of psychology (12 items)
- 6 point Likert-type scale
- 1=Extremely False; 6 = Extremely True
- “Based on my psychology graduate research training, I believe...
- ...there are adequate safeguards to ensure psychologists engage in ethical research practices”
- ...conducting research ethically is valued in the field of psychology”

**Notes:**
Where does this come from?  

## Student APA Ethics Code Competence Scale

- Measures familiarity with specific APA Ethical Standards related to research (23 items)
- True-False Items
- Examples:
- Studies judged to present no more than minimal risk are exempt from IRB review.
- The APA Ethics Code permits researchers to disclose confidential information to protect individuals who are not research participants
- Deception in research is permissible if any emotional distress that is severe can be remediated during debriefing

## Phase II Results

- Item analyses, factor analyses and inter-item reliabilities confirmed the psychometric validity of 7 scales:
- _Scale__	__	__	__	__	__# Items__	__	__α__	_
- RCR Mentoring
- 	RCR Knowledge Transmission	    22		.95
-     RCR Positive Mentoring	 	    22		.93
- RCR Departmental Climate
- 	RCR Expectations and Resources	    15		.91
-     RCR Practices			    13		.86
- RCR Internalization
- 	Student RCR preparedness		    23		.95
-     Student RCR Expectations		    12		.89
-     Student Ethics Code competence	    23		.95

## Correlations of RCR-Socialization Scales with RCR Internalization Scales

|  | MRCR-Knowledge Transmission | MRCR-Positive Modeling | RCR-DC Expectations & Resources | RCR-DC Faculty & Student Practices |
| --- | --- | --- | --- | --- |
| RCR-Preparedness | .55
 | .54 | .64 | -.47 |
| RCR-Attitudes toward Discipline | .36 | .37 | .65 | -.56 |

*All correlations significant at p < .001*

## Ethics Code Knowledge Scale

- No correlation with subscales
- APA Ethics Code not memorized by mentors, and thus students are not required to memorize ethical standards.
- The salience of different APA Ethics Code research standards differ depending on
- Research Population
- Research Methods
- Research Setting

**Notes:**
The ethic code knowledge scale although internally consistent and demonstrating good distribution did not meet validity criteria (e.g., did not correlate with any of the scales nor with whether students took an ethics class, years of research experience etc). Our interpretation of this finding is three-fold: First, mentors and faculty do not "memorize" the specific standards of the APA Ethics Code, but rather have absorbed the broader nature of responsible research and only refer to the Ethics Code when they are confronted with a dilemma or are challenged. They therefore socialize their students in these broad principles--obtain informed consent, submit documents to IRB, don't plagiarize, protect confidentiality--but do not requiure students to memorize the ethical standards. Second, the extent to which students will be exposed to ethical decision-making especially at the masters level of mentoring is idiosyncratic. Students who follow an apprenticeship model may be mentored in a portion of a larger project that has already passed IRB review so that informed consent, recruitment, and other human subjects protection knowledge does not need to go beyond the procedures required for the specific study; Finally, the salience of different RCR responsibilities will differ depending on the type and scope of the research project. For example, psychologists rarely encounter the conflict of interest dilemmas faced by other scientists whose work may be funded by industry or who are working on patents that will be lucrative to themselves and their universities.  Psychologists often conduct archival or secondary analyses which do not require directly obtaining informed consent and typically confidentiality protections are dictated by the archive etc.  In psychology, clinical trials almost never use "placebo" groups. Rather, treatment as usual or different treatment "dosages" are used.

## Do RCR-M and RCR-DC Scales Predict Student Preparedness?

- Multiple Regression
- Adjusted R Square = .98   p < .001
- Significant Beta Scores
- RCR-DC Expectations & Resources *B = .638 *(p < .001)
- MRCR-Knowledge Transmission *B* = .258
- (p = .001)
- RCR-DC Faculty & Student Practices *B* = .042 (p = .036)

## Do RCR-M and RCR-DC Scales Predict Student Attitudes Toward the RCR Integrity of Psychological Science?

*Multiple Regression*
- Adjusted R Square = .99   p < .001
- Significant Beta Scores
- RCR-DC Expectations & Resources *B = .781 *(p < .001)
- RCR-DC Faculty & Student Practices *B* = .084 (p < .001)

## Discussion

- Validation of New RCR Instruments for Psychological Science
- Next Steps

## RCR Student Internalization Scales

- Student RCR Preparedness RCR-P
- 	 Measures the degree to which students feel their graduate training program has prepared them to conduct responsible research.
- Student RCR attitudes RCR-A
- 	 Measures the influence of graduate training on students’ views on the responsible conduct of research ethics in the field of psychology.
*	**Both scales have excellent inter-item reliability and construct validity*

## Mentoring the Responsible Conduct of Research [MRCR]

- MRCR Knowledge Transmission
- 	 Measures extent of research mentor’s RCR direct instructions and practical guidance
- MRCR Positive Modeling
- 	Measures extent of research mentor’s RCR modeling and supervisory style
- 	*Both scales have high inter-item reliabilities and significantly associated with student RCR preparedness and attitudes toward discipline*
*	**MRCR-Knowledge Transmission yields significant Beta weights when RCR Departmental Climate Scales are included in predicting RCR preparedness*
- 	*Future questions: Does departmental climate mediate or moderate RCR mentoring influences on students’ attitudes toward the RCR integrity of the discipline of psychology?*

**Notes:**
In summary, the following scales are available

## RCR Department Climate Scales

- RCR-DC Expectations & Resources
- 	 Assesses RCR content in departmental courses, course requirements, resources, and procedures. core areas of RCR that students would receive explicit instruction on from their mentor
- RCR-DC Practices
-     Assesses RCR positive and negative departmental student and faculty practices
*	**Both scales have high inter-item reliabilities and are significantly associated with student RCR preparedness and attitudes toward discipline*
*	**Both scales yield significant Beta weights when RCR Mentoring scales are included in predicting RCR preparedness and RCR attitudes*

## Future Directions

- Validate a test of RCR Competence based more on socialization of RCR values and decision-making than on familiarity with specific standards in the APA Ethics Code
*Adaptation of Social Science Research Ethics Decision-making from:*
- “Validation of ethical decision-making measures:
- Evidence for a new set of measures”
- Authors:  Michael D. Mumford, Lynn D. Devenport, Ryan P. Brown, Shane Connelly, Stephen T. Murphy, Jason H. Hill, and Alison L. Antes

**Notes:**
We are confident in the psychometric validity of our measures and in a final stage have drawn upon the work of Stephen Murphy to attempt to validate a test of RCR competence based more on socialization of RCR values and decision-making then on familiarity with specific sections.

In summary, we have developed psychometrically valid measures that can be used by others studying social science and easily adapted by other disciplines to study RCR mentoring, climate, and preparedness.

## Final Phase

- Apply validated scales to a large national sample of graduate students to evaluate the multi-level effects of RCR mentoring, RCR graduate program ethical climate, and student characteristics on RCR preparedness, attitudes toward the RCR integrity of the discipline, and RCR ethics decision-making.

## For Copies of Scales Contact

- Celia B. Fisher, Ph.D.
- Director, Center for Ethics Education
- Fordham University
- 441 East Fordham Road
- Bronx, NY 10458
- Fisher@fordham.edu
- (718) 817-0927