Impact of Assessment

Discussant: Rachel Yudkowsky, MD, MHPE
Facilitator: To Be Determined

Combining Scores Based on Compensatory and Noncompensatory
Scoring Rules to Assess Resident Readiness for Unsupervised
Practice: Implications From a National Primary Care Certification
Examination in Japan
Hirotaka Onishi, MD, MHPE, PhD, Yoon Soo Park, PhD, Ryo Takayanagi, MD, and Yasuki Fujinuma, MD

Downloaded from https://journals.lww.com/academicmedicine by BhDMf5ePHKav1zEoum1tQfN4a+kJLhEZgbsIHo4XMi0hCywCX1AWnYQp/IlQrHD3yRlXg5VZA8tMNNLLXQmCQPN0FFgTZnM/oIucJW706hfZnNtmKIw0fw== on 10/31/2018

Abstract
PurposeCompetence decisions in healthprofessions education require combining
scores from multiple sources andidentifying pass–fail decisions basedon noncompensatory (required to pass
all subcomponents) and compensatory
scoring decisions. This study investigatesconsequences of combining scores,
reliability, and implications for validity
using a national examination with
subcomponent assessments.
MethodNational data were used from three
years (2015, 2016, and 2017) of
the Japan Primary Care Association
Board Certification Examination, withfour subcomponent assessments:
Clinical Skills Assessment–IntegratedIn health professions education, thecompetence of junior physicians and theassociated entrustment decisions leading
to unsupervised practice can be measured
by combining information from multiple
sources.1–3 This is often conducted in thecontext of combining trainee performancefrom related, but unique assessments.4,5For example, performance on multiplechoice tests and standardized patient (SP)encounters can be combined to form
composite scores that reflect competencies
in clinical knowledge and patient care
Please see the end of this article for information
about the authors.
Correspondence should be addressed to Hirotaka
Onishi, International Research Center for Medical
Education, Graduate School of Medicine, The Universityof Tokyo, 2F Igakubu-Sogochuokan, 7-3-1 Hongo,
Bunkyo-ku, Tokyo, Japan 113-0033; telephone: (+81)
3-5841-3534; e-mail: onishi-hirotaka@umin.ac.jp.
Acad Med. 2018;93:S45–S51.
doi: 10.1097/ACM.0000000000002380
Copyright © 2018 by the Association of American
Medical Colleges

Clinical Encounter (CSA-ICE), CSA–
Communication and Interpersonal Skills(CSA-CIS), Multiple-Choice Questions(MCQ), and Portfolio. Generalizability
theory was used to estimate variancecomponents and reliability. Kane’s
composite reliability and kappa decisionconsistency were used to examine theimpact of using compensatory andnoncompensatory scoring.

0.50 and 0.60; 0.65 and 0.76; and0.87 and 0.89. Using a completely
noncompensatory scoring approach on all
four subcomponents, decision-consistency
reliability was 0.33. Fully compensatory
scoring yielded reliability of 0.86.

Results
Mean performance (n = 251) on
the CSA-ICE, CSA-CIS, MCQ, andPortfolio subcomponent assessments
were, respectively, 61% (SD = 11%),67% (SD = 13%), 74% (SD = 8%),
and 65% (SD = 9%); componentspecific Φ-coefficient reliability ranged
between, respectively, 0.57 and 0.67;

Conclusions
Assessing a range of abilities inmaking entrustment decisions requiresconsidering the balance of assessment
tools measuring distinct but relatedcompetencies. These results indicatethat noncompensatory pass–faildecision making, which seems morecongruent with competency-based
education, may lead to much lower
reliability than compensatory decision
making when several assessment
subcomponents are used.

skills from both assessments.6,7 Prior
studies on composite scores haveaddressed issues related to identifying
weights of each assessment, by balancing
their clinical or curricular relevancewith their measurement characteristics
such as reliability and interassessment
correlation. Moreover, studies have also
made progress in discussing the reliability
and validity of composite scores,
identifying factors that contribute to theirvalidity evidence.1,4 However, an areathat still requires much needed attention,
particularly in the context of competencybased medical education (CBME), is
the discussion between compensatory
and noncompensatory scoring ofmulticomponent assessments. This issue
focuses on whether trainees should pass
all subcomponents to pass the entire
assessment (i.e., noncompensatory),
or whether performance on one
subcomponent can compensate for
performance on other sections of theassessment (compensatory).

The issue of compensatory andnoncompensatory scoring requires
discussion of the assessment challenges
embedded in making composite
decisions.8,9 For example, if board
certification examinations consistof multicomponent measures—(1)medical knowledge (MK), (2) patient
history taking and physical examination(H&P), and (3) communicationand interpersonal skills (CIS)—test
developers and professional societies needto decide whether examinees must pass
all individual subcomponents separately
to pass the entire test (e.g., pass all threesubcomponents), or whether parts ofthe examination can be combined tocompensate for performance on othercomponents (e.g., specifying H&P and
CIS as compensatory). Engaging in a
deeper discussion around compensatory
and noncompensatory consequences ofassessment scores and examinee pass–failresults addresses a fundamental aspect ofconsequential validity.10,11

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

S45

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment

Composite and noncompensatory
subcomponent scores are prevalent
in local medical school and residency
assessments and also in high-stakes
licensing examinations and board
certification examinations. For example,
the United States Medical Licensing
Examination Step 2 Clinical Skills (CS)examination requires examinees to passthe Integrated Clinical Encounter (ICE),
Communication and Interpersonal
Skills (CIS), and Spoken English
Proficiency subcomponents separately,
in a noncompensatory manner, to pass
the entire Step 2 CS examination.12 The
ICE subcomponent uses a compensatory
scoring approach, combining scores fromthe physical examination skills basedon the SP encounter and the diagnostic
justification skills based on the examinee’s
patient note assessment.4 As such, theStep 2 CS examination includes bothcompensatory and noncompensatorycomponents in determining the overall
examinee performance. Likewise, board
certification examinations administeredby medical specialty boards in the
United States are also based on theprinciple of noncompensatory scoring.
The American Board of Surgery Initial
Board Certification examination
requires examinees to pass both the MKcomponent (qualifying examination),measured using multiple-choicequestions; and an oral examination
component (certification examination),in a noncompensatory manner, to achieveboard certification.13
There are several reasons that support
noncompensatory scoring, including
considerations for patient safety,
proficiency in distinct competencies(e.g., Accreditation Council for Graduate
Medical Education or CanMEDS),
and curricular relevance, among
others.14–16 However, noncompensatory
scoring also has direct implications
on composite score reliability anddiminishing psychometric properties of
the assessment, for every incrementally
added noncompensatory assessment
measure.17–20 As such, a comprehensive
analysis of these considerations canbenefit how test scores are used andinterpreted.
In this study, we use national data fromthree consecutive years to examineinternal structure and consequential
validity evidence of composite scoresand composite decisions related to

S46

compensatory and noncompensatory
scoring. We use the Board Certification
Examination for family medicine
specialists in Japan to contribute to thisempirical discussion. Implications forweighting of assessment components andpass–fail consequences are discussed.
MethodStudy context and participants: Japan
In Japan, each clinical academic societyor association has been independently
managing board certification for different
physician specialties. For primary carespecialties, including family medicine, theJapan Primary Care Association (JPCA)
manages the certification examination.
The JPCA was established in 2010 by
the merger of three academic societies:
Japanese Academy of Family Medicine,Japanese Society of General Medicine,and Japanese Medical Society of Primary
Care. The JPCA is the administeringbody responsible for developing, scoring,and pass–fail decision making for family
medicine specialists in Japan, who mustcomplete both two-year postgraduate
clinical training and three-year family
medicine training, similar to the systemin the United Kingdom.
Data for this study come from threeconsecutive JPCA examinations,administered nationally to all family
medicine specialists in Japan. From 2015
to 2017, 67, 79 and 105 examinees took
the examination, respectively. The resultsof this study are essential to improving
the quality of examinations, as Japan is
also transforming the board certification
system by centralizing them into theJapanese Medical Specialty Board in
subsequent years to promote increased
public accountability, with 19 specialist
program certifications in fundamental
areas to be managed through a central
board certification system.
Assessment components
The JPCA examination consists offour subcomponents: (1) Clinical
Skills Assessment–Integrated ClinicalEncounter (CSA-ICE), which
comprehensively measures historytaking and physical examination
skills, clinical reasoning, interpersonaland communication skills, andprofessionalism of candidates scored by
two independent physician examiners,
as candidates rotate through six

different SP encounters; (2) Clinical
Skills Assessment–Communication
and Interpersonal Skills (CSA-CIS),
which measures patient-centeredinterpersonal and communication skills
using ratings from SPs; (3) MultipleChoice Question (MCQ) test measuring
clinical knowledge; and (4) Portfolio
assessment, which measures integrative
clinical understanding of patient care,system-based practice, and practicebased learning and improvement. JPCA
discloses the use of noncompensatory
pass–fail decision making.
The CSA-ICE and CSA-CIS are measured
as parts of a six-station CSA. Portfolio
consists of reports for 18 different
areas. Its scoring rubric for each area isdisclosed prior to test administration to
all examination candidates. All candidates
are required to select one case with theirbest performances for each area. Two
independent physician raters assess
reports in each item. For the MCQ, eachtest administration year used slightly
different test development procedures,
and as such, the numbers of MCQ items
increased (2015: 41 items; 2016: 94 items;
2017: 103 items) because part of theformat was a previously modified essay
question. Psychometric analyses of each
assessment component are routinely
conducted, including item analysis andreliability estimation to evaluate theirmeasurement characteristics.
Analysis
Data were compiled across the three
testing years (2015, 2016, and 2017)and analyzed using the following steps.
First, we independently examined thedescriptive statistics (item and total
score distribution) and reliability of
the four components: (1) CSA-ICE, (2)CSA-CIS, (3) MCQ, and (4) Portfolio.
Subsequently, we applied componentspecific weights and reliability using theKane approach to calculate composite
scores and their reliability estimates.4,10
Traditional approaches for estimating
reliability or composite scores may ignore
the reliability of each assessment andtheir corresponding pairwise associations.
For example, if an assessment system
includes components of high and low
reliability that are moderately correlated,
simply taking their average may not take
into account the nuanced measurement
characteristics of each assessment. As
such, under the Kane approach, thereliability of composite scores can be

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment

expressed as a combination of reliability
coefficients of component and correlation
coefficients, thereby generating more
psychometrically sound composite scoresand reliability estimates. Finally, we usedcomponent-specific passing standards
specified by the JPCA committee to derivepass–fail rates by treating each component
as noncompensatory and also using acompensatory approach. The final stepwas conducted to simulate consequencesof reliability and pass–fail results whenvarying scoring approaches were used.

Resultscompensatory and noncompensatory
scoring, we derived reliability estimates
for three scenarios:

Descriptive statistics
A total of 251 examinees took the JPCA,
across the four examination components:
CSA-ICE, CSA-CIS, MCQ, and Portfolio(2015: n = 67; 2016: n = 79; 2017:
n = 105). The number of OCSE stationsin the CSA was fixed to 6 stations; thePortfolio was also fixed to 18 items. The
number of MCQ items increased between
testing years to reflect changes in theexamination format. Table 1 shows thenumbers of items for each test component,
numbers of examinees of each year, andthe total test score distribution. Overall,
the mean scores for CSA-ICE, CSACIS, MCQ, and Portfolio components
across years were 61% (SD = 11%), 67%(SD = 13%), 74% (SD = 8%), and 65%(SD = 9%), respectively.

1. Four noncompensatory
subcomponents: assuming
noncompensatory scoring for all four
test subcomponents (CSA-ICE, CSACIS, MCQ, and Portfolio);
2. Three noncompensatory
subcomponents: assuming
noncompensatory scoring for threetest subcomponents, combining
the CSA-ICE and CSA-CIS
subcomponents (CSA, MCQ, andPortfolio); andGeneralizability study (G study) was used
to estimate variance components and to
estimate reliability for each test component
by testing year.21,22 CSA-ICE used a threefacet design, with person (p: examinees)crossed by raters (r) and items (i) nested
in station (s), p × [(r × i): s]. For the CSACIS and MCQ, a one-facet G study design
was used, p × s and p × i, respectively.
Variance components for Portfolio were
estimated using a two-facet G study design,p × (r: i). Variance components were
estimated using urGENOVA. Reliability
indices for component scores were basedon the Φ-coefficient, as all examinations
were conducted as criterion-referenced
assessments.

3. Fully compensatory: using a fullycompensatory scoring approachcombining scores from all four
subcomponents.

Variance components, interassessment
correlation, and reliability

The weights for CSA-ICE, CSA-CIS,
MCQ, and Portfolio were initially set as
20%, 15%, 30%, and 35%, respectively,
following faculty consensus. Then, theweight for CSA-ICE was altered under
the proportional weights of three otherassessment subcomponents (CSA-CIS,
MCQ, and Portfolio). These scoring
decisions were based on discussion
with the JPCA committee on possible
score combinations. Noncompensatory
score reliability was estimated using
decision consistency and k-way kappa
statistics.23–25 Compensatory composite
score reliability was estimated using theKane method.10

Next, pass–fail rates and reliability were
calculated by both noncompensatory andcompensatory scores for three consecutive
years. This calculation was based on
the actual results but only a simulation
because JPCA does not disclose detailed
information regarding its cut scores andstandard-setting procedures.

Variance components. Table 2 shows
variance components and percent
variance component for each assessment
facet. For the CSA-ICE, person variance
ranged between 5% and 12%, withthe largest variance component due toperson-station of 15% to 20%, indicatingcase specificity. For the CSA-CIS, person
variance ranged between 13% and 20%.
For the MCQ, person variance ranged
between 2% and 3%, with majorityof variance due to items, 21% to 28%.
Finally, for the Portfolio component, theperson variance ranged between 17% and20%; person–item interaction accounted
for 20% to 22% of total variance, also
indicating item specificity.

Data compilation and analyses wereconducted using Stata 14 (Stata Corp,
College Station, Texas). This study wasapproved by the institutional review board
at the University of Illinois at Chicago.

To examine the consequences ofdecision-consistency reliability based on

Interassessment correlation andreliability. Correlations between the

Table 1
Distribution of Component Scores by Year: Descriptive Statistics
CSA-ICEa

CSA-CISb

MCQc

PortfoliodYear

No. ofexaminees

No. ofstations

Mean (SD)No. ofstations

Mean (SD)No. ofitems

Mean (SD)No. ofitems

Mean (SD)

2015
2016

67
79

6
6

64.1 (12.6)
55.9 (11.1)

6
6

64.3 (14.9)
67.6 (12.4)

41
94

81.0 (7.2)
71.9 (7.6)

18
18

71.8 (7.5)
60.4 (10.3)

2017

105

6

62.5 (8.6)

6

69.0 (11.8)

103

67.9 (8.3)

18

63.9 (9.3)

Overalle

251—60.8 (10.7)—

67.3 (13.0)—

73.6 (7.7)—

65.4 (9.0)

CSA-ICE measures integrative skills of history taking, physical examination, clinical reasoning, interpersonal and communication, and professionalism of candidates as theyrotate through different standardized patient encounters. This is measured by two independent physician examiners.
b
CSA-CIS measures interpersonal and communication skills using ratings from standardized patients.
c
MCQ is an assessment of clinical knowledge using multiple-choice questions.
dPortfolio assessment measures integrative clinical understanding including patient care, system-based practice, and practice-based learning and improvement.
eOverall takes the descriptive statistics from accumulated data across the years.
a

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

S47

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment

Table 2
Generalizability Theory: Variance Components by Assessment
2015
Examination

CSA-ICE

Effect

df

VC

% VC

df

VC

% VC

df

VC

% VC

66
5

1.069
0.096

11.9%
1.1%

78
5

0.075
0.043

8.5%
4.9%

104
5

0.039
0.097

4.6%
11.3%

item: station (i: s)

18

0.000

0.0%

33

0.011

1.2%

26

0.000

0.0%

rater (r)

1

0.201

2.2%

1

0.000

0.0%

1

0.000

0.0%
14.6%

p×s

330

1.597

17.8%

390

0.175

19.9%

520

0.125

p × i: s

1,188

1.881

21.0%

2,574

0.059

6.7%

2,704

0.000

0.0%

p×r

66

0.138

1.5%

78

0.000

0.0%

104

0.001

0.1%

s×r

5

0.027

0.3%

5

0.000

0.0%

5

0.000

0.0%

i × r: s

18

0.283

3.2%

33

0.092

10.5%

26

0.099

11.6%

p×s×r

330

0.382

4.3%

390

0.000

0.0%

520

0.000

0.0%

1,188

3.302

36.8%

2,574

0.425

48.3%

2,704

0.497

57.9%

Φ-coefficient reliability

0.68

Portfolio

0.67

0.57

person (p)

66

1.098

13.7%

78

0.884

12.7%

104

0.135

station (s)

5

0.185

2.3%

5

0.938

13.5%

5

0.015

2.2%

p × s, error

330

6.728

84.0%

390

5.154

73.9%

520

0.530

78.0%

person (p)

66

0.003

3.3%

78

0.003

1.5%

104

0.005

2.4%

item (i)

40

0.024

22.9%

93

0.051

28.0%

102

0.046

21.0%

p × i, error

2,640

0.077

73.8%

7,254

0.129

70.5%

10,608

0.167

76.6%

Φ-coefficient reliability

Φ-coefficient reliability

MCQ

2017

person (p)station (s)p × i × r: s, error

CSA-CIS

2016

0.50

0.51

19.9%

0.60

0.65

0.65

person (p)66

48.486

17.5%

78

3.038

0.67
19.9%

104

2.442

0.76

item (i)1724.794

9.0%

17

0.950

6.2%

17

0.483

3.3%

rater: item (r: i)

18

6.735

2.4%

18

0.966

6.3%

18

0.989

6.8%

16.9%

p×i

1,122

60.259

21.8%

1,326

3.249

21.2%

1,768

2.884

19.9%

p × r: i, error

1,188

136.051

49.2%

1,404

7.095

46.4%

1,872

7.676

53.0%

Φ-coefficient reliability

0.87

0.89

0.87

Abbreviations: CSA-CIS indicates Clinical Skills Assessment–Communication and Interpersonal Skills; CSA-ICE,
Clinical Skills Assessment–Integrated Clinical Encounter; MCQ, Multiple-Choice Questions; VC, variance component.

CSA-ICE and CSA-CIS were 0.62.
Correlations between CSA-ICE andother assessments ranged between 0.23
and 0.39. For CSA-CIS, interassessment
correlations with other assessments werelower, at 0.10. Correlations between MCQ
and Portfolio were 0.33. The overall
Φ-coefficients for CSA-ICE, CSA-CIS,
MCQ, and Portfolio ranged between,
respectively, 0.57 and 0.68, 0.50 and 0.60,
0.65 and 0.76, and 0.87 and 0.89.
Compensatory versus noncompensatory
scoring
Impact of composite scores. Figure 1 shows
the plot of composite scores using theKane method, using a fully compensatory
approach combining scores across all four
components. Weights used to combine thescores were, respectively, 20% 15%, 30%,
and 35% for CSA-ICE, CSA-CIS, MCQ,
and Portfolio. These weights were selected

S48

by JPCA members to reflect content andclinical relevance based on their testing
blueprint. The x-axis shows the weighted
mean, simply taking the sum of scores
using the weights (= [20% × CSA-ICE]+ [15% × CSA-CIS] + [30% × MCQ] +[35% × Portfolio]). The Kane composite
scores on the y-axis were based on the Kane
method, which uses the weighted mean but
also takes into consideration the individual
assessment reliability, interassessment
correlation, and weights. The scatter plot
based on the composite scores is not a linear
line, indicating that there are variations inthe score range. For example, for a weighted
mean score of 60%, there are seven different
Kane composite scores (range 23–33). As
such, depending on where individual pass–
fail scores were assigned using a weighted
mean approach, the composite score mean
could yield a different score, leading tovarying pass–fail decisions.

Figure 2 shows the relationship between
varying weights of CSA-ICE and its
impact on overall composite scorereliability. The figure shows a parabolic
relationship, with reliability maximized
when CSA-ICE is weighted at 15%.
Using a completely compensatory scoring
approach (combining scores across thefour subcomponents) yields a composite
score reliability of 0.86. This is basedon individual reliability estimates of
0.64, 0.54, 0.69, and 0.88 for the CSAICE, CSA-CIS, MCQ, and Portfolio,
respectively. Table 3 shows the results.
Noncompensatory scores: Assuming
four subcomponents. Table 3 shows thedecision-consistency kappa reliability
for different noncompensatory scoring
scenarios. When noncompensatory
scoring was used for the four

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment

70
60
50
40
30
0

10

20

Kane Composite Score

80

90 100consequences of using compensatory
and noncompensatory pass–faildecision making. Our study reportsthat using noncompensatory scoringhas direct consequences on composite
reliability and decision consistency,
which may be mitigated by combining
the compensatory approach within
the noncompensatory componentsof the assessment. Prior discussions
in the literature have focused onusing weights to combine scores,
without a deeper discussion on how
compensatory or noncompensatory
scoring could affect the reliability andoverall validity of the assessment.1,4 In
the CBME framework, where learners
are required to master and excel inunique but distinct competencies, ourfindings can form a useful basis forfurther discussion.26

40

50

60
Weighted Mean

70

80

Figure 1 Scatter plot of weighted mean and Kane composite scores (n = 251).

subcomponent assessments (i.e.,
examinees must pass CSA-ICE, CSA-CIS,
MCQ, and Portfolio separately to pass
the entire examination), the decisionconsistency reliability was 0.33. This is incontrast to a composite score reliabilityof 0.86 when all subcomponents wereassumed to be compensatory.

When the three subcomponent scores
were combined to form a compensatorycomposite measure, the composite scorereliability only had a minor change from
0.86 (compensatory composite scorecreated from four subcomponents) to
0.87 (compensatory composite scorecreated from three subcomponents).
Discussion

This study contributes to the broad
discussion on composite scores and

.83
.81
.79
.75

.77

Composite Score Reliability

.85

.87

Noncompensatory scores: Assuming threesubcomponents. In our next simulation,
we combined the CSA assessments (CISICE and CIS-CIS). Then, using the threesubcomponent scores (CSA, MCQ, andPortfolio), we calculated the decisionconsistency reliability, which increased

from 0.33 (four noncompensatory
subcomponents) to 0.50.

5

10

15

20

25
30
35
CSA-ICE Weight %

40

45

50

Figure 2 Relationship between CSA-ICE and CSA-CIS weight and composite score reliability.
Note: Vertical line inserted at 20%, which indicates the initial weight for CSA-ICE.

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

Our findings can be largely divided into
two parts. First, we provide important
implications on compensatory scoring,based on a composite score approach.
Competencies and entrustment decisions
in medical education require combining
information from various sources,
including knowledge assessments,
workplace-based assessments, and skill
assessments such as SP encounters.
Findings from this study provide insights
on how weights can be specified. For
example, our results in Figure 2 show aparabolic relationship between weights
and composite score reliability whenscores are combined. Such information
could be shared with assessment
committees or with clinical competencecommittees (CCCs) in identifying
optimal weights that can maximize
reliability, validity, and the overall
curriculum focus of the educational
program. Such information can also
be used to identify deficiencies in the
program with respect to developing
additional assessment tools that canadvance decisions for promotion.
Our study also provides some empiricalguidance on how noncompensatory
scoring can impact the psychometricconsequences of the assessment.
Prior studies by Hambleton27 haveshown that increasing the number ofnoncompensatory measures can reduce
the overall reliability. A well-known
example used in the measurement
literature is an assessment with five
subcomponent assessments scoredin a noncompensatory manner.

S49

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment

Table 3Pass–Fail Rates and Reliability: Consequences of Noncompensatory andCompensatory Scoring

Pass–fail type

Assessment

Four componentsa

Three componentsb

Fail %

Reliability

Fail %

Reliability

7.57
6.37

0.64
0.54

9.56

0.75

CSA-ICE
CSA-CIS
Noncompensatory

Compensatoryc

MCQ

4.38

0.69

4.38

0.69

Portfolio

6.77

0.88

6.77

0.88

Overall (noncompensatory)

18.73d0.33e

18.73d

0.50e

—

4.78d

0.86e

7.57d

0.87e

 Abbreviations: CSA-CIS indicates Clinical Skills Assessment–Communication and Interpersonal Skills; CSA-ICE,
Clinical Skills Assessment–Integrated Clinical Encounter; MCQ, Multiple-Choice Questions.
aFour components of the assessment are CSA-ICE, CSA-CIS, MCQ, and Portfolio.
bThree components of the assessment are CSA (combining ICE and CIS), MCQ, and Portfolio.
cInterassessment correlations range between 0.10 and 0.62. These were used to estimate the Kane composite
score reliability.
dOverall fail rate based on noncompensatory and compensatory for four- and three-component scoring methods,
respectively.
e
Reliability estimates corresponding to overall fail rates.

national testing population and include
data across three consecutive years, whichshould increase the generalizability of
our findings. Moreover, our data andsimulations are based on expert-defined
weights, which may differ depending on
the assessment context. Yet, we hope thatfindings from this study could shed lighton the overall significance and add tothe discussion on deeper understandingof compensatory and noncompensatory
pass–fail decision making.
Assessments in the CBME era should
consider balancing assessment
tools measuring distinct but relatedcompetencies. We conclude that
educators should investigate theimpact of noncompensatory scoringby examining its measurement
characteristics, in addition to curricular,
clinical, and patient safety considerations.
Funding/Support: None reported.

Even if each subcomponent has areliability of 0.90, the overall decisionconsistency reliability can drop to 0.59(= 0.90 × 0.90 × 0.90 × 0.90 × 0.90).
This example, of course, does not take
into account the interdependence
between the assessments (assuming thateach assessment is fully independent).
However, it conveys the straightforward
message that noncompensatory scoringhas its benefits, but also its costs on
significantly reducing the overall
decision-consistency reliability. In otherwords, even if a well-prepared learnerwere to take the examination, the overall
reliability would be 0.59, adding to
uncertainty of whether the learner failed
the examination because of his or her
ability, or based on the noncompensatory
nature of the test.
In our study, we simulated data toshow that an assessment with four
subcomponent individual reliability
estimates ranging between 0.54 and0.88 would yield a decision-consistency
reliability of 0.33—this is much lower
than a fully compensatory approachcomposite score reliability of 0.86. Wealso show that by simply combining twoof the subcomponents, the reliability
could increase from 0.33 to 0.50. Theseempirical results could shed light into
how programs and testing organizations
may develop test-scoring procedures
and identify pass–fail decisions. In thisprocess, we are not explicitly statingthat noncompensatory scoring should

S50

be avoided. Noncompensatory scoring
does have its benefits to ensure thatlearners are fully competent in specificcontent areas, which can be importantfor ensuring patient safety concerns.
However, our findings do underscore
the need to consider both psychometricconsequences, in addition to patient
safety factors that can lead to making
the compensatory or noncompensatory
scoring decisions.
For a high-stakes test like a specialist
examination, valid pass–fail decision
making demands a certain level ofreliability.27 The JPCA examination usedin our study showed that no reliabilityof each component score exceeded 0.90,
and only Portfolio was acceptable forsummative purpose with a reliability
score between 0.85 and 0.9. Reliabilityof noncompensatory scores (equal tocurrent conjunctive standard) for CSA
and MCQ was below 0.70, a level notallowed for pass/fail decision making.
However, it is acceptable if we use acompensatory standard. Changing fromnoncompensatory to compensatory
pass–fail decisions should be considered,
yet whether JPCA’s leaders accept thechange or not is another issue becausesuch a change in the rule could influence
the mind-set of future candidates.
This study is based on a single board
certification examination, with specialtyspecific assessment content and learner
sample. However, our data come from a

Other disclosures: None reported.
Ethical approval: This study was approved by theinstitutional review board at the University of
Illinois at Chicago.
H. Onishi is assistant professor, International
Research Center for Medical Education, Graduate
School of Medicine, The University of Tokyo, and vicechair, Committee for Specialist Certification, Japan
Primary Care Association, Tokyo, Japan; ORCID:
http://orcid.org/0000-0002-6979-1088.
Y.S. Park is associate professor, Department ofMedical Education, University of Illinois at Chicago
College of Medicine, Chicago, Illinois; ORCID: http://
orcid.org/0000-0001-8583-4335.
R. Takayanagi is director, Gumma Family Medicine
Center, Maebashi Kyoritsu Clinic, and vice chair,Committee for Specialist Certification, Japan Primary
Care Association, Tokyo, Japan.
Y. Fujinuma is director, Centre for Family Medicine
Development, Japanese Health and Welfare
Co-operative Federation, and chair, Committee
for Specialist Certification, Japan Primary Care
Association, Tokyo, Japan.

References
1 Corcoran J, Downing SM, Tekian A, DaRosa
DA. Composite score validity in clerkshipgrading. Acad Med. 2009;84(10 suppl):S120–
S123.
2 Hicks PJ, Margolis M, Poynter SE, et al;
APPD LEARN-NBME Pediatrics Milestones
Assessment Group. The Pediatrics
Milestones Assessment Pilot: Developmentof workplace-based assessment content,
instruments, and processes. Acad Med.
2016;91:701–709.
3 Schwartz A, Margolis MJ, Multerer S, Haftel
HM, Schumacher DJ; APPD LEARN–NBME
Pediatrics Milestones Assessment Group. A
multi-source feedback tool for measuring a
subset of pediatrics milestones. Med Teach.
2016;38:995–1002.

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.

Impact of Assessment
4 Park YS, Lineberry M, Hyderi A, Bordage
G, Xing K, Yudkowsky R. Differential
weighting for subcomponent measures ofintegrated clinical encounter scores basedon the USMLE Step 2 CS examination:
Effects on composite score reliability andpass–fail decisions. Acad Med. 2016;91(11suppl):S24–S30.
5 Margolis MJ, Clauser BE, Swanson DB,
Boulet JR. Analysis of the relationshipbetween score components on astandardized patient clinical skills
examination. Acad Med. 2003;78(10suppl):S68–S71.
6 Harik P, Clauser BE, Grabovsky I, Margolis
MJ, Dillon GF, Boulet JR. Relationships
among subcomponents of the USMLE Step 2
Clinical Skills examination, the Step 1,
and the Step 2 Clinical Knowledge
examinations. Acad Med. 2006;81(10suppl):S21–S24.
7 Clauser BE, Balog K, Harik P, Mee J,
Kahraman N. A multivariate generalizability
analysis of history-taking and physicalexamination scores from the USMLE Step
2 Clinical Skills examination. Acad Med.
2009;84(10 suppl):S86–S89.
8 Baldwin P. Weighting components ofa composite score using naïve expert
judgments about their relative importance.
Appl Psychol Meas. 2015;39:539–550.
9 Feldt LS. Estimating the reliability of a test
battery composite or a test score based onweighted item scoring. Meas Eval Couns Dev.
2004;37:184–189.
10 Kane M, Case SM. The reliability and validityof weighted composite scores. Appl Meas
Educ. 2004;17:221–240.

11 Messick S. Standards of validity and thevalidity of standards in performanceassessment. Educ Meas. 1995;14:5–8.
12 Federation of the State Medical Boards andthe National Board of Medical Examiners.
2017 Step 2 Clinical Skills (CS) content
description and general information. http://
www.usmle.org/pdfs/step-2-cs/cs-infomanual.pdf. Accessed July 20, 2018.
13 American Board of Surgery. Training andcertification: General surgery. http://www.
absurgery.org/default.jsp?examoffered_gs.
Published 2017. Accessed July 20, 2018.
14 Association of American Medical Colleges.
Core entrustable professional activities forentering residency: Curriculum developers’
guide. https://www.mededportal.org/icollabo
rative/resource/887. Accessed July 20, 2018.
15 Nasca TJ, Philibert I, Brigham T, Flynn
TC. The next GME accreditation system—
Rationale and benefits. N Engl J Med.
2012;366:1051–1056.
16 Yudkowsky R, Tumuluru S, Casey P, Herlich
N, Ledonne C. A patient safety approachto setting pass/fail standards for basic
procedural skills checklists. Simul Healthc.
2014;9:277–282.
17 Clauser BE, Harik P, Margolis MJ. A
multivariate generalizability analysis of datafrom a performance assessment of physicians’
clinical skills. J Educ Meas. 2006;43:163–191.
18 Swygert KA, Balog KP, Jobe A. The impact of
repeat information on examinee performancefor a large-scale standardized-patient
examination. Acad Med. 2010;85:1506–1510.
19 Norcini JJ, Swanson DB, Grosso LJ, Webster GD.
A comparison of several methods for scoring
patient management problems. In: Kerby S,

Academic Medicine, Vol. 93, No. 11 / November 2018 Supplement

compiler. Proceedings of the Twenty-Second
Annual Conference on Research in Medical
Education. Washington, DC: Association ofAmerican Medical Colleges; 1983.
20 Webster GD, Shea JA, Norcini JJ, Grosso LJ,
Swanson DB. Strategies in comparison of
methods for scoring patient management
problems: Use of external criteria to validate
scores. Eval Health Prof. 1988;11:231–248.
21 Brennan RL. Generalizability Theory. New
York, NY: Springer-Verlag; 2001.
22 Webb NM, Shavelson RJ, Haertel EH.
Reliability coefficients and generalizability
theory. In: Rao CR, Sinharay S, eds.
Handbook of Statistics. Amsterdam, theNetherlands: Elsevier; 2006:81–124.
23 Livingston SA, Lewis C. Estimating theconsistency and accuracy of classification
based on test scores. J Educ Meas.
1995;32:179–197.
24 Park YS, Hyderi A, Bordage G, Xing K,
Yudkowsky R. Inter-rater reliability andgeneralizability of patient note scores using
a scoring rubric based on the USMLE Step-2
CS format. Adv in Health Sci Educ. doi:
10.1007/s10459-015-9664-3
25 Fleiss JL. Measuring nominal scaleagreement among many raters. Psych Bull.
1971;76:378–382.
26 Gruppen L, Frank JR, Lockyer J, et al; ICBMECollaborators. Toward a research agenda forcompetency-based medical education. Med
Teach. 2017;39:623–630.
27 Hambleton RK, Slater SC. Reliability ofcredentialing examinations and the impact ofscoring models and standard-setting policies.
Appl Meas Educ 1997;10:19–38.

S51

Copyright © by the Association of American Medical Colleges. Unauthorized reproduction of this article is prohibited.