Clinical trial NCT00667940: Faculty Development to Improve Mini-CEX Ratings
Official title: Effect of Faculty Development on Mini-CEX Score Reliability and Accuracy: A Randomized, Controlled Trial
Study type: INTERVENTIONAL
Overall status: COMPLETED
Phase: NA
Conditions: Faculty Development
Keywords: Medical education
faculty development
rater training
clinical competence
assessment
Lead sponsor: Mayo Clinic
Sponsor class: OTHER
First posted: 2008-04-28
Enrollment: 52
Sex: ALL
Standard ages: CHILD
ADULT
OLDER_ADULT
Accepts healthy volunteers: yes
Allocation: RANDOMIZED
Intervention model: PARALLEL
Primary purpose: TREATMENT

Brief summary: We will conduct a faculty development workshop with intent to improve interrater reliability and rater accuracy. Outcomes will include interrater reliability, accuracy, and workshop evaluations.

Detailed description: Hypothesis: An intervention for training faculty preceptors on the use of the mini-CEX will improve inter-rater reliability and classification accuracy.

Methods. We will develop a faculty development workshop aimed at improving the inter-rater reliability and accuracy of mini-CEX scores. To evaluate the workshop we will conduct a randomized, controlled trial. Preceptors will be randomly assigned to immediate or delayed workshop. The primary outcome will be inter-rater reliability of mini-CEX scores as preceptors rate videotaped resident-patient encounters before and four weeks after the workshop. We will also evaluate the accuracy of performance classifications. We will also collect mini-CEX ratings of actual resident-patient interactions one year preceding and one year following the workshop, and calculate interrater reliability.

Statistical methods. Inter-rater reliability will be determined using the intraclass correlation coefficient (ICC) and corresponding confidence intervals. ICC will be used to determine chance-corrected agreement (accuracy). Discrimination (differences in mean scores among scripted levels of performance) will also be evaluated as a measure of accuracy.

Scientific basis. Clinical performance assessments constitute a key element of resident education. The effect of rater training on the inter-rater reliability of mini-CEX scores is unknown. This workshop will utilize methods (rater error and field of reference training) shown to be effective in improving rater accuracy in non-medical fields.

Inclusion criteria. All internal medicine residency continuity clinic preceptors are eligible.

Eligibility criteria: Inclusion Criteria:

* Internal medicine residency continuity clinic preceptor at Mayo Clinic Rochester Minnesota

Exclusion Criteria:

* None

Primary outcomes:
measure: Interrater reliability (videotaped resident-patient encounters)
timeFrame: April-June 2006
Secondary outcomes:
measure: Rater accuracy (agreement with scripted levels of performance on videotaped resident-patient encounters): percent agreement, chance-corrected agreement, mean scores
timeFrame: April-June 2006
measure: Interrater reliability (actual resident-patient encounters)
timeFrame: March 2005 (historical) to June 2007
Other outcomes:
Locations:
city: Rochester
country: United States
facility: Mayo Clinic
state: Minnesota
Study officials:
affiliation: Mayo Clinic
name: David A Cook, MD, MHPE
role: PRINCIPAL_INVESTIGATOR
References:
citation: Cook DA, Dupras DM, Beckman TJ, Thomas KG, Pankratz VS. Effect of rater training on reliability and accuracy of mini-CEX scores: a randomized, controlled trial. J Gen Intern Med. 2009 Jan;24(1):74-9. doi: 10.1007/s11606-008-0842-3. Epub 2008 Nov 11.