Awareness of Diagnostic Error among Japanese
Residents: a Nationwide StudyYuji Nishizaki, MD, MPH, PhD1, Tomohiro Shinozaki, MPH, PhD2, Kensuke Kinoshita, MD3,Taro Shimizu, MD, MPH, MBA4, and Yasuharu Tokuda, MD, MPH5
1

Medical Technology Innovation Center, Juntendo University, Tokyo, Japan; 2Department of Biostatistics, Graduate School of Medicine, Universityof Tokyo, Tokyo, Japan; 3Department of Medicine, Mito Kyodo General Hospital, University of Tsukuba, Tsukuba, Japan; 4Diagnostic and GeneralistMedicine, Dokkyo Medical University, Mibu, Japan; 5Muribushi Okinawa for Teaching Hospitals, Urasoe City, Okinawa, Japan.

BACKGROUND: Residents’ understanding of diagnosticerror may differ between countries. We sought to explore
the relationship between diagnostic error knowledge andself-study, clinical knowledge, and experience.
METHODS: Our nationwide study involved postgraduateyear 1 and 2 (PGY-1 and -2) Japanese residents. The
Diagnostic Error Knowledge Assessment Test (D-KAT)and General Medicine In-Training Examination (GMITE) were administered at the end of the 2014 academic
year. D-KAT scores were compared with the benchmark
scores of US residents. Associations between D-KAT scoreand gender, PGY, emergency department (ED) rotations
per month, mean number of inpatients handled at any
given time, and mean daily minutes of self-study were also
analyzed, both with and without adjusting for GM-ITE
scores. Student’s t test was used for comparisons withlinear mixed models and structural equation models(SEM) to explore associations with D-KAT or GM-ITE
scores.
RESULTS: The mean D-KAT score among Japanese PGY2 residents was significantly lower than that of their US
PGY-2 counterparts (6.2 vs. 8.3, p < 0.001). GM-ITE
scores correlated with ED rotations (≥6 rotations: 2.14;
0.16–4.13; p = 0.03), inpatient caseloads (5–9 patients:1.79; 0.82–2.76; p < 0.001), and average daily minutes ofself-study (≥91 min: 2.05; 0.56–3.53; p = 0.01). SEM revealed that D-KAT scores were directly associated with
GM-ITE scores (ß = 0.37, 95% CI: 0.34–0.41) and indirectly associated with ED rotations (ß = 0.06, 95% CI: 0.02–0.10), inpatient caseload (ß = 0.04, 95% CI: 0.003–0.08),
and average daily minutes of study (ß = 0.13, 95% CI:0.09–0.17).
CONCLUSIONS: Knowledge regarding diagnostic error
among Japanese residents was poor compared with thatamong US residents. D-KAT scores correlated strongly
with GM-ITE scores, and the latter scores were positively
associated with a greater number of ED rotations, larger
caseload (though only up to 15 patients), and more timespent studying.

J Gen Intern Med 33(4):445–8
DOI: 10.1007/s11606-017-4248-y© Society of General Internal Medicine 2017

KEY WORDS: Diagnostic Error Knowledge Assessment Test (D-KAT);
General Medicine In-Training Examination (GM-ITE); postgraduate
education.

METHODS

Received July 21, 2017
Revised October 24, 2017
Accepted November 27, 2017
Published online December 18, 2017

INTRODUCTION

In 2004, the Japanese Ministry of Health, Labor and Welfare
established a new postgraduate clinical training program, mandating that residents complete a 2-year internship after graduating from medical school and passing the National Examination for Medical Practitioners. In those 2 years, residents areexpected to achieve a basic set of skills for primary carepractice, obtained through rotations in internal medicine, surgery, and emergency medicine during the first year, and pediatrics, obstetrics and gynecology, psychiatry, and community
medicine during the second year. In 2010, the required departments for rotation were reduced from seven to three: internal
medicine, emergency department (ED), and community medicine. Residents are allowed to rotate on surgery, anesthesiology, pediatrics, OB-GYN, or psychiatry as electives. After this2-year residency, they choose their subspecialty program.
Knowledge of diagnostic errors such as cognitive bias orheuristics is an important factor in preventing such errors. Nostudies to date have explored experiential variables, such asworkload, ER rotations, or minutes of study per day, that might
improve D-KAT scores. In a previous study, we found that
GM-ITE scores were improved with optimal resident caseload,
ED rotations, and access to online resources.1 The purpose of
the current study was to assess diagnostic error knowledge
among residents throughout Japan and to compare the resultswith the benchmark scores in a previous US study.2 We also
evaluate the relationships between diagnostic error knowledge
and self-study, clinical knowledge, and experience.

We conducted a nationwide study of Japanese PGY-1 and -2
residents. The Diagnostic Error Knowledge Assessment Test
(D-KAT) and General Medicine In-Training Examination(GM-ITE) were administered at the end of the 2014 academicyear. Immediately after the D-KAT and GM-ITE were conducted, the participants completed a questionnaire regarding
445

446

JGIM

Nishizaki et al.: Diagnostic Error Knowledge Among Residents

modeling. We also conducted the same analyses for the DKAT score as a dependent variable, with and without adjustingfor the GM-ITE score (except for the D-KAT score). A twotailed p value of <0.05 was considered statistically significant.
Analyses were performed using SAS version 9.4 software (SASInstitute Inc., Cary, NC). Structural equation models were built
to test for the direct and indirect determinants of D-KAT performance using standardized estimates of effect and standard
goodness-of-fit measures (STATA 14; StataCorp LP, CollegeStation, TX).7 The study was approved by the Ethics Committeeof the Tsukuba University Mito Medical Center.

their clinical experience and self-study habits, assessing ED
rotations per month (the average frequency of night duty permonth), the mean number of inpatients handled at one time,and mean daily minutes of self-study.
Diagnostic error knowledge was assessed using the D-KAT,
a 13-item multiple-choice test developed and validated byReilly et al. in 2010.2 The GM-ITE is a clinical knowledge
test that uses a methodology similar to that of the US Internal
Medicine In-Training Examination (IM-ITE).3 The purpose of
the IM-ITE is to provide residents and program directors withan objective, reliable, and valid assessment of clinical knowledge through a multiple-choice examination.4,5 Our GM-ITE,
which was designed and written by a committee of experienced physicians assembled by the Japan Institute for Advancement of Medical Education Program (a non-profit organization in Tokyo, Japan), included 87 questions testing a wide
range of clinical knowledge, from clinical skills and practical
medical knowledge to psychosocial care of patients.6
The mean D-KAT scores were compared between Japaneseand US residents using Student’s t test.2 In addition, the associations between D-KAT scores and gender, PGY, ED rotations permonth, mean number of inpatients handled at any given time,and mean daily minutes of self-study were analyzed with orwithout adjusting for GM-ITE scores (calculated by the domains
other than the D-KAT score). Linear mixed models were used toaccount for hospital variability as random intercepts, with HuberWhite (sandwich) variance estimators. Answers of Bunknown^(for ED duty and inpatient caseload) were retained in the analyses, whereas non-responders for either ED duty, inpatient caseload, or self-study (N = 224) were excluded from mixedRESULTS

A total of 2652 residents (1123 PGY-1 and 1529 PGY-2) from
258 teaching hospitals participated in the examination; 777(29.3%) of participants were women. The voluntary survey
response rate was 91.6% (2429/2652). Among respondents,
the weighted average number of ED rotations was 3.94, with
3% having no rotations, 8% with 1–2, 70% with 3–5, and 17%
with 6 or more rotations. By self-report, residents managed anaverage of 6.9 inpatients (weighted average value): 9% reported managing 4 or fewer, 64% reported 5–9, 19% noted 10–14,
and 6% reported 15 or more. Residents reported spending anaverage of 42.2 min studying (weighted average value), with
39% spending less than 30 min, 38% between 31 and 60 min,
13% 61–90 min, and 4% 91 min or more of daily self-study.
The mean (± SD) D-KAT score among Japanese PGY-2
residents (6.2 ± 1.6) was significantly lower than that of their

Table 1 Linear Mixed Model Estimates for D-KAT Scores among Responding Doctors (N = 2428)
Category

Fixed effects
Female (vs. male)
2nd PGY (vs. 1st)GM-ITE score (except D-KAT)
ED rotations (per month)

Inpatient caseload (no. patients)

Self-study (min)Random components
Var (random intercepts)
Var (residual)

0
1–2
3–5
≥6
Unknown
0–4
5–9
10–14
≥15
Unknown
0–30
31–60
61–90
≥91
None

Including GM-ITE score

Excluding GM-ITE score

Estimate

95% LL

95% ULP value

Estimate

95% LL

95% ULP value

−0.05
0.12
0.04
Reference
−0.25
−0.18
−0.26
−0.75
Reference
0.21
0.04
0.150.35
Reference
0.03
−0.10
−0.15
−0.05

−0.20
0.00
0.03

0.09
0.24
0.05

0.48
0.05*
<.0001*

−0.06
0.18

−0.21
0.06

0.09
0.31

0.43
0.00*

−0.60
−0.45
−0.58
−1.32

0.11
0.08
0.06
−0.18

0.17
0.18
0.12
0.01*

−0.62
−0.37
−0.51
−1.34

0.13
0.18
0.17
−0.13

0.19
0.50
0.32
0.02*

−0.03
−0.22
−0.19
−0.27

0.44
0.30
0.49
0.97

0.09
0.76
0.39
0.27

−0.11
−0.30
−0.48
−0.31

0.17
0.10
0.18
0.22

0.04
2.52

0.00
2.37

0.09
2.66

0.05
−0.15
−0.14
−0.26

0.52
0.37
0.56
0.98

0.02*
0.390.23
0.260.71
0.33
0.37
0.72

Reference
−0.25
−0.10
−0.17
−0.73
Reference
0.29
0.11
0.21
0.36
Reference
0.06
−0.03
−0.07
−0.07

−0.08
−0.23
−0.42
−0.35

0.20
0.18
0.27
0.21

0.41
0.80
0.67
0.63

0.06
<.0001*

0.06
2.57

0.00
2.42

0.11
2.72

0.03*
<.0001**Indicates statistical significance at p <0.05. D-KAT = Diagnostic Error Knowledge Assessment Test, GM-ITE = General Medicine In-Training
Examination, 95% LL = 95% lower limit, 95% UL = 95% upper limit, PGY = postgraduate year, ED = emergency department. Linear mixed models
were used to account for hospital variability as random intercepts, with Huber–White (sandwich) variance estimators. Answers of Bunknown^ (for EDrotations and inpatients handled) were retained during analyses, while non-responders for either ED rotations, inpatients handled, or self-study (N =224) were excluded from mixed modeling. All analyses were conducted using SAS version 9.4 (Cary, NC)

JGIM

447

Nishizaki et al.: Diagnostic Error Knowledge Among Residents

Table 2 Linear Mixed Model Estimates for GM-ITE Score (except for D-KAT) among Responding Doctors (N = 2428)
Category

Fixed effects
Female (vs. male)
2nd PGY (vs. 1st)
D-KAT
ED rotations (per month)

Inpatient caseload (no. patients)

Self-study

Random components
Var (random intercepts)
Var (residual)

0
1–2
3–5
≥6
Unknown
0–4
5–9
10–14
≥15
Unknown
0–30 min
31–60 min
61–90 min≥91
None

Including D-KAT score

Excluding D-KAT score

Estimate

95% LL

95% ULP value

Estimate

95% LL

95% ULP value

−0.10
1.43
0.64
Reference0.011.94
2.14
1.14
Reference
1.79
1.68
1.12
0.36
Reference
0.80
1.85
2.05
−0.42

−0.64
0.75
0.45

0.44
2.11
0.83

0.72
<.0001*<.0001*

−0.13
1.55

−0.68
0.87

0.42
2.23

0.65
<.0001*

−1.86
−0.03
0.16
−0.89

1.87
3.90
4.13
3.18

0.99
0.05
0.03*
0.27

−2.09
−0.17
−0.06
−1.48

1.78
3.92
4.10
2.92

0.87
0.07
0.06
0.52

0.82
0.52
−0.65
−1.68

2.76
2.83
2.89
2.41

0.00*
0.00*
0.22
0.73

1.07
0.65
−0.56
−1.43

2.94
2.89
3.07
2.55

<.0001*
0.00*
0.17
0.58

0.17
0.72
0.56
−1.61

1.42
2.99
3.53
0.77

0.01*
0.00*
0.01
0.49

Reference
−0.16
1.87
2.02
0.72
Reference
2.00
1.77
1.26
0.56
Reference
0.83
1.82
1.97
−0.46

0.20
0.68
0.45
−1.68

1.47
2.96
3.50
0.77

0.01*
0.00*
0.01*
0.46

7.07
42.96

4.73
40.44

9.41
45.48

<.0001*<.0001*

7.48
43.88

5.02
41.31

9.94
46.45

<.0001*<.0001**Same as shown in Table 1US PGY-2 counterparts (8.3 ± 1.5; p < 0.001).2 Similar to US
resident scores, GM-ITE scores increased from PGY-1 to PGY-2(6.1 to 6.2, respectively; p = 0.018).8 Using linear mixed models,the D-KAT scores were found to be associated with GM-ITE
scores and inpatient caseload (Table 1). The GM-ITE scores
were associated with D-KAT scores (estimate of change in score,0.64; 95% confidence interval, 0.45–0.83; p < 0.001), ED rotations (≥6 rotations: 2.14; 0.16–4.13; p = 0.03), inpatient caseloads (5–9 patients: 1.79; 0.82–2.76; p < 0.001), and average
daily minutes of self-study (≥91 min: 2.05; 0.56–3.53; p = 0.01;
Table 2).
SEM models indicated a direct relationship between GMITE scores and D-KAT performance (ß = 0.37, 95% CI: 0.34–0.41; Fig. 1), and suggested that D-KAT scores were indirectlyaffected by the number of ED rotations (ß = 00.06, 95% CI:0.02–0.10), inpatient caseload (ß = 00.04, 95% CI: 0.003–0.08), and average daily minutes of study (ß = 00.13, 95% CI:0.09–0.17). Our model was well fitted to our data (root meansquare error of approximation [RMSEA]: 0.03; comparative
fit index [CFI]: 0.98; Tucker-Lewis index [TLI]: 0.96).

DISCUSSION

It is unclear why Japanese residents scored more poorly on
diagnostic error knowledge than US residents. We can infer,
however, that graduate and postgraduate medical education onclinical reasoning has not reached a level similar to that in theFigure 1 Structural equation modeling for D-KAT scores. D-KAT = Diagnostic Error Knowledge Assessment Test.

448

Nishizaki et al.: Diagnostic Error Knowledge Among Residents

US. Medical education in clinical clerkships in Japan has long
been provided during subspecialty rotations that occur in thePGY years following the first 2 years. This long-held educational tradition has brought fewer educational opportunities
for clinical reasoning and even postgraduate clinical training
because of the short training period in the department ofgeneral medicine (GM) or general internal medicine (GIM).6
Compared to subspecialty physicians, teachers of GIM or GM
departments may be better able to provide their students andresidents specific clinical problem-solving knowledge, including that for diagnostic error.
Our study suggests that self-study, time spent in ED rotations, and caseload are all directly associated with GM-ITE
scores, and are indirectly associated with D-KAT scores
through the relationship between GM-ITE and D-KAT scores.
A greater number of ED rotations and minutes of self-study
and a larger inpatient caseload appeared to contribute to general improvement in clinical knowledge and skill. An appropriate caseload in both emergency and inpatient settings may
provide residents with the experience needed to reflect onvarious cognitive biases, leading to improved clinical anddiagnostic reasoning. In addition, the development of independent curricula, such as simulation-based training, reflective
practice, or active metacognitive review, may be useful.9–11
Our SEM models allowed us to explore the potential causal
relationships between our variables, but given the nature of
our cross-sectional data, our results can only be speculative.
Our findings should be validated in a larger longitudinal study.
In conclusion, diagnostic error knowledge among Japanese
residents was poor compared with that among US residents.
GM-ITE performance was related to scores on D-KAT. Diagnostic error knowledge was indirectly related to increased
clinical experience and self-study. Resident diagnostic error
performance may improve with greater opportunities for reflection on inpatient and emergency patient conditions andadditional independent curricula.10,11
Acknowledgements: The authors would like to thank Takeshi
Asakura for help in translating this manuscript into English, and weJGIM

thank all members of the GM-ITE production committee for theirsupport. We would also like to thank Jeffrey Jackson, MD, MPH, for
his editorial work and assistance with the SEM models.
Corresponding Author: Yasuharu Tokuda, MD, MPH; Muribushi
Okinawa for Teaching Hospitals, Urasoe City, Okinawa, Japan(e-mail: yasuharu.tokuda@gmail.com).
Compliance with Ethical Standards:
Conflict of Interest: The authors declare that they do not have aconflict of interest.

REFERENCES
1.

2.

3.

4.

5.

6.

7.
8.

9.

10.
11.

Kinoshita K, Tsugawa Y, Shimizu T, et al. Impact of inpatient caseload,
emergency department duties, and online learning resource on General
Medicine In-Training Examination scores in Japan. Int J Gen Med.
2015;8:355–60.
Reilly JB, Ogdie AR, Von Feldt JM, et al. Teaching about how doctors
think: a longitudinal curriculum in cognitive bias and diagnostic error forresidents. BMJ Qual Saf. 2013;22:1044–50.
McCoy CP, Stenerson MB, Halvorsen AJ, et al. Association of volume ofpatient encounters with residents’ in-training examination performance.
J Gen Intern Med. 2013;28:1035–41.
Kanna B, Gu Y, Akhuetie J, et al. Predicting performance using
background characteristics of international medical graduates in an
inner-city university-affiliated Internal Medicine residency training program. BMC Med Educ. 2009;9:42.
Perez JA, Greer S. Correlation of United States Medical Licensing
Examination and Internal Medicine In-Training Examination performance. Adv Health Sci Educ Theory Pract. 2009;14:753–8.
Shimizu T, Tsugawa Y, Tanoue Y, et al. The hospital educational
environment and performance of residents in the General Medicine InTraining Examination: a multicenter study in Japan. Int J Gen Med.
2013;6:637–40.
Kline RB. Principles and practice of structural equation modeling. New
York: Guilford Press; 1998.
Kay C, Jackson JL, Frank M. The relationship between InternalMedicine Residency Graduate performance on the ABIM certifying
examination, yearly in-service examinations and attending evaluations.
Acad Med. 2015;90(1):100–4.
Schmidt HG, Mamede S. How to improve the teaching of clinicalreasoning: a narrative review and a proposal. Med Educ. 2015;49:961–73.
Graber ML, Kissam S, Payne VL, et al. Cognitive interventions to reduce
diagnostic error: a narrative review. BMJ Qual Saf. 2012;21:535–57.
Singh H, Schiff GD, Graber ML, et al. The global burden of diagnosticerrors in primary care. BMJ Qual Saf. 2017;26:484–94.