JMIR MEDICAL INFORMATICS

Shen et al

Review

Artificial Intelligence Versus Clinicians in Disease Diagnosis:
Systematic Review
Jiayi Shen1,2*, MBBS; Casper J P Zhang3*, MPH, PhD; Bangsheng Jiang4,5, MBBS; Jiebin Chen6, BSc; Jian Song7,
BA; Zherui Liu6, BSc; Zonglin He4,5, MBBS; Sum Yi Wong4,5, MBBS; Po-Han Fang4,5, MBBS; Wai-Kit Ming1,4,8,9,
MPH, MD, MMSc, PhD
1

Department of Obstetrics and Gynecology, The First Affiliated Hospital of Sun Yat-sen University, Guangzhou, China

2

School of Medicine, Jinan University, Guangzhou, China

3

School of Public Health, The University of Hong Kong, Hong Kong, China (Hong Kong)

4

International School, Jinan University, Guangzhou, China

5

Faculty of Medicine, Jinan University, Guangzhou, China

6

College of Information Science and Technology, Jinan University, Guangzhou, China

7

School of International Studies, Sun Yat-sen University, Guangzhou, China

8

Harvard Medical School, Harvard University, Boston, MA, United States

9

Division of Pharmacoepidemiology and Pharmacoeconomics, Brigham and Women’s Hospital, Boston, MA, United States*these authors contributed equally

Corresponding Author:
Wai-Kit Ming, MPH, MD, MMSc, PhD
Department of Obstetrics and Gynecology
The First Affiliated Hospital of Sun Yat-sen University
No 58 Zhongshan Road 2
Guangzhou,
China
Phone: 86 14715485116
Email: wkming@connect.hku.hk

Abstract
Background: Artificial intelligence (AI) has been extensively used in a range of medical fields to promote therapeutic
development. The development of diverse AI techniques has also contributed to early detections, disease diagnoses, and referralmanagement. However, concerns about the value of advanced AI in disease diagnosis have been raised by health care professionals,
medical service providers, and health policy decision makers.
Objective: This review aimed to systematically examine the literature, in particular, focusing on the performance comparison
between advanced AI and human clinicians to provide an up-to-date summary regarding the extent of the application of AI todisease diagnoses. By doing so, this review discussed the relationship between the current advanced AI development and clinicianswith respect to disease diagnosis and thus therapeutic development in the long run.
Methods: We systematically searched articles published between January 2000 and March 2019 following the Preferred
Reporting Items for Systematic reviews and Meta-Analysis in the following databases: Scopus, PubMed, CINAHL, Web of
Science, and the Cochrane Library. According to the preset inclusion and exclusion criteria, only articles comparing the medical
performance between advanced AI and human experts were considered.
Results: A total of 9 articles were identified. A convolutional neural network was the commonly applied advanced AI technology.
Owing to the variation in medical fields, there is a distinction between individual studies in terms of classification, labeling,training process, dataset size, and algorithm validation of AI. Performance indices reported in articles included diagnostic accuracy,
weighted errors, false-positive rate, sensitivity, specificity, and the area under the receiver operating characteristic curve. The
results showed that the performance of AI was at par with that of clinicians and exceeded that of clinicians with less experience.
Conclusions: Current AI development has a diagnostic performance that is comparable with medical experts, especially inimage recognition-related fields. Further studies can be extended to other types of medical imaging such as magnetic resonanceimaging and other medical practices unrelated to images. With the continued development of AI-assisted technologies, the clinical

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 1
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al

implications underpinned by clinicians’ experience and guided by patient-centered health care principle should be constantly
considered in future AI-related and other technology-based medical research.
(JMIR Med Inform 2019;7(3):e10010) doi: 10.2196/10010
KEYWORDS
artificial intelligence; deep learning; diagnosis; diagnostic imaging; image interpretation, computer-assisted; patient-centered
care

Introduction
BackgroundAn aging patient population and a shortage of medical
professionals have led to a worldwide focus on improving theefficiency of clinical services via information technology.
Artificial intelligence (AI) is a field of algorithm-based
applications that can simulate humans’ mental processes andintellectual activity and enable machines to solve problems withknowledge. In the information age, AI is widely used in the
medical field and can promote therapeutic development. AI
may optimize the care trajectory of patients with chronic disease,suggest precision therapies for complex illnesses, and reduce
medical errors [1].
There are currently 2 common types of AI. The first type isexpert systems. An expert system is a computer system thatgenerates predictions under supervision and can outperform
human experts in decision making. It consists of 2interdependent subsystems: a knowledge base and an inferenceengine. Although the knowledge base contains accumulated
experience, the inference engine (a reasoning system) can accessthe current state of the knowledge base and supplement it withnew knowledge. Expert systems can create more explicit criticalinformation for the system, make maintenance easy, and increase
the speed of prototyping [2]. However, expert systems arelimited regarding knowledge acquisition and performance.
Computer-assisted techniques have been introduced in medical
practice for decades but have recently yielded minimal
improvements. The second type is machine learning. This is
the core of AI and is a fundamental approach to making
computers intelligent. Machine learning requires vast amountsof data for training. This systematically improves theirperformance during the process. One of the focuses underlying
machine learning is parameter screening. Too many parameters
can lead to inaccurate entries and calculations; therefore,
reducing the number of parameters can improve the efficiency
of AI, but it may also lower its accuracy. However, 1 of thecritical objectives of AI is to outperform humans via self-study
in challenging fields without any previous knowledge.
AI has been extensively used in a range of medical fields.
Clinical diagnoses of acute and chronic diseases, such as acuteappendicitis [3] and Alzheimer disease [4], have been assisted
via AI technologies (eg, support vector machines, classification
trees, and artificial neural networks). Integrative AI consistingof multiple algorithms rather than a single algorithm
substantially improves its abilities to detect malignant cells,
yielding higher diagnostic accuracy [5].

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

The development of diverse AI techniques also contributes tothe prediction of breast cancer recurrence [6]. In-home AI
systems may potentially oversee patients with insulin
abnormalities and swallowing problems [7] rather than doctors.
Treatment optimization is achievable by AI [8] for patients with
common, but complex diseases characterized as being ascribedto multiple factors (eg, genetic environmental or behavioral)such as cardiovascular diseases are more likely to benefit from
more precise treatments on account of the AI algorithms basedon big data [8]. On the other hand, AI-assisted hospitalmanagement systems could also help minimize
logistics-associated monetary and temporal costs on a larger
scale [9].

Objectives
To our knowledge, there is no published review comparing thediagnostic performance between AI and clinicians. Thus, weaimed to systematically review the literature and provide an
up-to-date summary indicating the extent of application of AI
to disease diagnoses compared with clinicians. We hope thisreview would help foster health care professionals’ awarenessand comprehension of AI-related clinical practices.

Methods
Search Strategy, Selection Criteria, and Study Selection
This search strategy was developed upon consultation with aprofessional librarian. The literature search was conducted in
Scopus (the largest abstract and citation database spanning
multiple disciplines), PubMed, CINAHL, Web of Science, andCochrane Library using the combination of searching terms(see Multimedia Appendix 1). The search was limited to articles
published between January 2000 and March 2019 following thePreferred Reporting Items for Systematic reviews andMeta-Analysis. Additional potentially eligible articles weremanually searched via screening of the reference list of included
articles as well as our personal archives.
We included articles if they (1) focused on advanced AI (defined
as an AI encompassing a training or learning process toautomate expert-comparable sophisticated tasks), (2) enclosed
at least an application to particular disease diagnoses, (3)compared the performance between AI and human experts on
specific clinical tasks, and (4) were written in English. Articles
were excluded if they (1) only described simpler AIs that do
not involve any training or learning process; (2) did not compare
performance of AI with that of medical experts; and (3) wereconference abstracts, book chapters, reviews, or other forms
without detailed empirical data.

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 2
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
On the basis of the above inclusion and exclusion criteria, 2reviewers (JS and BJ) independently screened article titles andabstracts and identified eligible articles. The full text of eligible
articles was retrieved via the institutional access. Any
discrepancy occurred during this process was resolved bydiscussion with 2 senior authors (WKM and CJPZ). The processof systematic search and the identification of reviewed articles
are depicted in Figure 1.

Data Extraction, Data Synthesis, and Quality
Assessment
Characteristics of included studies were extracted independently
by 2 reviewers (JS and BJ) after verification by 2 senior authors(WKM and CJPZ). The characteristics comprised (1) first author
and publication year, (2) AI technology, (3) classification andShen et allabeling, (4) data sources (including the sample size of total
sets, training sets, validation, and/or tuning sets and test sets),(5) training process, (6) internal validation methods, (7) humanclinician reference, and (8) performance assessment.
Study quality was assessed using the Cochrane’s risk-of-bias
tool [10]. This tool provides a domain-based approach to help
reviewers judge the reporting of various types of risk by
scrutinizing information from reviewed articles, and in turn, thejudgment can be made based on these pieces of supporting
information against specific types of risk of interest. The typesof risk assessed in this review include (1) blinding of participants
and personnel (performance bias), (2) blinding of outcome
assessment (detection bias), (3) incomplete outcome data
(attrition bias), and (4) selective reporting (reporting bias).

Figure 1. Flow diagram of study inclusion and exclusion process.

Results
Systematic Search
Following the systematic search process, 41,769 citations wereretrieved from the database and 22,900 articles were excluded
based on their titles and abstracts, resulting in 850 articles tobe reviewed in detail. In addition, 842 articles were further
excluded based on their full text. One article was identified fromthe manual searches. Finally, 9 studies were included for review(Figure 1).

Characteristics of Included Studies
Table 1 summarizes the characteristics of these 9 studies. These
9 included studies were published between 2017 and 2019 andconducted across countries, including China, Germany, South
Korea, the United Kingdom, and the United States. Regarding
their studied medical conditions, 3 studies could be categorizedunder ophthalmology, including diabetic retinopathy [11],macular degeneration [11], and congenital cataracts [12],whereas another 3 studies focused on onychomycosis [13] andhttp://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

skin lesions/cancers [14,15]. The other studies related toradiology were focused on thoracic [16,17] and neurological[18] conditions.
A convolutional neural network (CNN) was the commonly
applied advanced AI technology in all reviewed studies, withthe exception of 1 study: González-Castro et al adopted support
vector machine classifiers in their study [18].
Owing to the difference in study objectives, methodology, andmedical fields, classification type between individual studies
differed correspondingly. For instance, studies related toophthalmological images [11,12] had differences in image
sources (eg, ocular images [12] or optical coherence tomography[OCT]–derived images [11]) and, thus, the classification differedcorrespondingly (Table 1). Another study that was also basedon OCT-derived images [19] focused on the referral suggestion
made between clinical experts and AI, and the classification ofmultiple suggestion decisions was used. With regard toonychomycosis images, 4 and 6 classes were both used fortraining, and binary classification was subsequently used in
testing by Han et al [13].

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 3
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al

Table 1. Characteristics of included studies.
Authors (year) Artificial intelligence technology

Classification/label- Data source; sample Training process
ing
size of total dataset,training sets, validation and/or tuning
sets and test-set

Internal validation

Human clinicians(external validation)

Brinker(2019) [14]A convolutional
neural network;
CNN (trained withenhanced techniques
on dermoscopic images)

All melanomas wereverified byhistopathologicalevaluation of biopsies; the nevi were
declared as benign
via expert consensus

International Skin
Imaging Collaboration (ISIC) image
archive; Total:
13,737; Training:
12,378 (1888melanomas and10,490 atypicalnevi); Validation:
1359 (230melanomas and1129 atypical nevi);
Test: 100 dermoscopic images

A ResNet50 CNN
model (residual
learning) used forthe classification ofmelanomas andatypical nevi.

Not reported

One hundred andforty-five dermatologists from 12 German university hospitals (using 100 images)

De Fauw(2018) [19]A segmentation
CNN model using a
3-dimensional UNet architecture

Referral suggestion:urgent/semi-urgent/routine/observation only (golden
standard labels wereretrospectively obtained by examining
the patient clinicalrecords to determine
the final diagnosis
and optimal referral
pathway in the lightof the subsequently
obtained information)

Clinical OCT scansfrom Topcon 3D
OCT, Topcon,
Japan; Device type
1: Training: segmentation network: 877(segmentation); gold
standard referral decision: 14,884 (classification); Validation: 224 (segmentation); 993 (classification); Test: 997; Device type 2: Training: segmentation
network: Additional
152 with 877 scansfrom device type 1(segmentation); gold
standard referral decision: 0 with 14,884
from device type 1(referral decision);
Validation: 112(classification);
Test: 1161) Deep segmentation network, trainedwith manually segmented OCT scans;
2) Resulting tissue
segmentation map;
3) Deep classification network, trainedwith tissue maps
with confirmed diagnoses and optimalreferral decisions; 4)
Predicted diagnosis
probabilities and referral suggestions.

Manually segmented and graded by 3
trained ophthalmologists, reviewed
and edited by a senior ophthalmologist

Device type 1: 8
clinical experts (4consultant ophthalmologists/retinal
specialists and 4 optometrists trained in
OCT interpretation
and retinal disease);
Device type 2: Fiveconsultant ophthalmologists (4 of themwere participants inthe device type 1
and the other was
new participant)

Esteva (2017)[15]

Deep CNNs (aGoogleNet Inception
v3 CNN architecture
pretrained on the
ImageNet dataset)

Biopsy-proven clinical images with 2
critical binary classification, labeled bydermatologists

Eighteen different
clinician-curated,
open-access online
repositories andclinical data from
Stanford University
Medical Center; Total: 129,405; Training and validation:
127,463 (9-foldcross validation);
Test: 1942

1) Classification ofskin lesions using asingle CNN; 2)
Trained end-to-end
from images directly, using only pixels
and disease labels as
inputsTwo dermatologists (at both 3class and 9-classdisease partitions)using 9-fold crossvalidation

Twenty-one boardcertified dermatologists on epidermal
and melanocytic lesion classification

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 4
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al

Authors (year) Artificial intelligence technology

Classification/label- Data source; sample Training process
ing
size of total dataset,training sets, validation and/or tuning
sets and test-set

Han (2018)[13]

A region-based convolutional deep neural network (RCNN)

Four classes (onychomycosis, naildystrophy, onycholysis, and melanonychia) and 6 classes(onychomycosis,
nail dystrophy, onycholysis, melanonychia, normal, andothers), manuallycategorized by dermatologists

Kermany(2018) [11]Deep CNN (alsoused transfer learning)

Four categories (3labels): choroidal
neovascularization
or diabetic macularedema (labeled asurgent referrals),
drusen (routine referrals), normal (observation); Binary classification also implemented (normal vschoroidal neovascularization/diabetic
macular edema/drusen)

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Internal validation

Human clinicians(external validation)

Four hospitals (Asan
Medical Center, Inje
University, Hallym
University, andSeoul National University); Total:
57,983; Training:
53,308 consist ofdatasets A1 (49,567)
and A2 (3741); Test:
1358 consist ofdatasets B1 (100),
B2 (194), C (125),and D (939)1) Extracted clinical Two classes (onyphotographs automat- chomycosis or not)ically cropped by the
R-CNN; 2) One dermatologist cropped
all of the images
from the A2, RCNN model trainedusing information
about the crop location; 3) fine image
selector trained to
exclude unfocused
photographs;(4)Three dermatologists tagged clinicaldiagnosis to the nailimages generated by
the R-CNN, withreference to the existing diagnosis taggedin the original image; (5) ensemble
model as the output
of both the ResNet152 and VGG-19
systems computedwith the feedforward
neural networks

1) Forty-two dermatologists (16 professors, 13 clinicianswith more than 10
years of experiencein the department of
Dermatology, and 8
residents) and 57 individuals from thegeneral populations(11 general practitioners, 13 medical
students, 15 nurses
in the dermatology
department, and 18
nonmedical persons)in the combined
B1+C dataset; 2)
The best 5 dermatologists among themin the combined
B2+D dataset.

Optical coherence
tomography (OCT)
images selectedfrom retrospective
cohorts of adult patients from the Shiley Eye Institute of
the University ofCalifornia San
Diego, the California Retinal Research
Foundation, Medical
Center Ophthalmology Associates, theShanghai First People’s Hospital, andBeijing Tongren Eye
Center between July
1, 2013 and March
1, 2017. Total:
207,130; Training:
108,312 (passed initial image quality review); Validation:
1000 (randomly selected from the same
patients); Test: 1000
(independent sample
from other patients)

After 100 epochs (iterations through theentire dataset), thetraining was stopped
because of the absence of further improvement in bothaccuracy and crossentropy loss

Six experts with significant clinical experience in an academic ophthalmology
center1000 images randomly selectedfrom the images
used for training(limited model)

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 5
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al

Authors (year) Artificial intelligence technology

Classification/label- Data source; sample Training process
ing
size of total dataset,training sets, validation and/or tuning
sets and test-set

Internal validation

Human clinicians(external validation)Long (2017)[12]

Deep CNN

Binary classification
by an expert panel interms of opacity area
(extensive vs limited), opacity density(dense vs nondense),and opacity location(central vs peripheral)

Childhood Cataract
Program of the Chinese Ministry of
Health (CCPMOH);
Total: 1239; Training: 886; Validation:
5-fold cross validation for in silico test;
57 for multi-ospaital
clinical trial; 53 for
Web sited–based
study; 303 for further validation; Test:
50

The championship
K-fold cross-validamodel from the Ima- tion (K=5)
geNet Large Scale
Visual Recognition
Challenge 2014,
containing 5 convolutional or down-sample layers in addition
to 3 fully connected
layersThree ophthalmologists with varying
expertise (expert,
competent, andnovice)Nam (2018)[16]

Deep learning–based Binary classificaautomatic detection tion: normal or nodalgorithm (DLAD) ule chest radiographs (image-level
labeling); Nodule
chest radiographs
were obtained frompatients with malignant pulmonary nodules proven at
pathologic analysis
and normal chest radiographs on the basis of their radiology
reports. All chest radiographs were carefully reviewed bythoracic radiologists.

Normal and nodule
chest radiographsfrom three Korean
hospitals (Seoul National University
Hospital; Boramae
Hospital; and National Cancer Center)and 1 US hospital(University of California San Francisco
Medical Center).
Total: 43,292;
Training: 42,092(33,467 normal and8625 nodule chest
radiographs); Tuning: 600 (300 normal and 300 nodule
chest radiographs);Internal validation:
600 (300 normal and
300 nodule chest radiographs); External
validation/test: 693

DLAD was trainedin a semisupervisedmanner by using allof the image-level
labels and partiallyannotated by 13
board-certified radiologists, with 25
layers and 8 residual
connections

Radiograph classification and nodule
detection performances of DLAD
were validated byusing 1 internaland 4 external
datasets in terms of
the area under
ROC (AUROC)
and figure of merit(FOM) form jackknife alternative
free-response ROC(JAFROC)18 physicians (including 3 nonradiology physicians, 6 radiology residents, 5board-certified radiologists, and 4 subspecialty trained
thoracic radiologists)

Rajpurkar(2018) [17]Deep CNN with a
121-layer DenseNetarchitecture(CheXNeXt)

ChestX-ray14
dataset; Total:
112,120; Training:
98,637; Tuning:
6351; Validation:
4201) Multiple networks
were trained on the
training set to predict the probability
that each of the 14
pathologies ispresent in the image;2) A subset of those
networks, each chosen based on the average error on thetuning set, constituted an ensemble that
produced predictions
by computing themean over the predictions of each individual network

Comprehensive
comparison of the
CheXNeXt algorithm to practicing
radiologists across
7 performance
metrics (ie, no external validation)

Nine radiologists (6board-certified radiologists and 3 senior
radiology residents
from 3 institutions)

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Binary values (absence/presence) in
14 pathologies: atelectasis, cardiomegaly, consolidation, edema, effusion, emphysema, fibrosis, hernia; Infiltration, mass; nodule, pleural thickening, pneumonia, andpneumothorax, obtained using automatic extraction methods on radiology reports

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 6
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Authors (year) Artificial intelligence technology

Shen et alClassification/label- Data source; sample Training process
ing
size of total dataset,training sets, validation and/or tuning
sets and test-set

González-Cas- Support vector ma- Binary classifier oftro (2017) [18] chine (SVM) classifi- the burden of ener
larged perivascularspaces (PVS) as low
or high

Data from 264 patients in Royal Hallamshire Hospital;
Total: 264 (randomly partitioned into 5
equal-sized subsets);
Training: 4 of the 5subsets (~211); Test:
one of the five subsets (~53)

Similarly, the training processes employed in individual studies
were not identical to each other because of their field-specificnature and classification-peculiar algorithms. For instance,
predictions in 1 ophthalmological study [11] were informed bya model using transfer learning on a Web-based platform on
the basis of training on graded OCT images. The otherophthalmological study [12] focusing on congenital cataracts
employed a 3-stage training procedure (ie, identification,
evaluation, and strategist networks) to establish a collaborative
disease management system beyond only disease identification.
Owing to this, data sources for training were field specific. The
training procedures in the other studies are detailed in Table 1.
Furthermore, 2 studies [11,16] employed both internal andexternal validation methods via training and/or validating theeffectiveness of their AI algorithms using images from theirown datasets and external datasets. Kermany et al investigatedthe effectiveness of their AI systems in the prediction of adiagnosis in their own ophthalmological images as well as thegeneralizability to chest x-ray images [11]. In contrast, Nam etal validated their work using datasets from not only their own
hospital but also other different local or overseas hospitals [16].
The remaining studies did not report both internal or external
validation or differentiate either.
Variation in dataset size was also observed. Specifically, thequantity of training sets, validation (and tuning) sets, and test
sets ranged from 211 to approximately 113,300, from 53 toapproximately 14,163, and from 50 to 1942, respectively.

Performance Indices and Comparison Between
Artificial Intelligence and Clinicians
All studies compared the diagnostic performance between AI
and licensed doctors (see Table 2). Performance indices used
for comparison included diagnostic accuracy, weighted errors,sensitivity, specificity (and/or the area under the receiver
operating characteristic curve [AUC]), and false-positive rate.
A total of 4 articles [11,12,15,17] adopted the accuracy (ie, theproportion of true results [both positives and negatives] among
the total number of cases examined) to compare diagnostic
performance between AI and humans. Long et al observed ahigh accuracy in AI (90%-100%) compared with a panel of
specialty doctors’ predefined diagnostic decision andtranscended the average levels of clinicians in most clinical

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Several combinations of the regularization parameter C
and gamma, wereused and assessedwith all descriptors
to find the optimal
configuration using
the implementation
provided by the libSVM library

Internal validation

Human clinicians(external validation)

A stratified 5-fold Two observers (ancross-validation re- experienced neurorapeating ten times
diologist and a
trained image analyst)

situations except for treatment suggestion. Esteva et al also
found that AI achieved comparable accuracy with oroutperformed their human rivals (AI vs dermatologists: 72.1%(SD 0.9%) vs 65.8% using 3-class disease partition and 55.4%(SD 1.7%) vs 54.2% using 9-class disease partition [15]). The
same was also observed in the study by Rajpurkar et al [17],indicating an agreement in results between AI and radiologists.
Similarly, Kermany et al showed that their AI achieved highaccuracy (96.6%) while acknowledging that their 6 experienced
ophthalmologists still performed well [11]. They also reported
weighted errors in which medical doctors maintained better
accuracy (4.8% vs 6.6%). De Fauw et al [19] reported
unweighted errors by using 2 devices, and the results showed
their AI’s performance commensurate with retina specialists
and generalizable to another OCT device type.
Overall, 7 studies [11,13-18] compared the sensitivity,
specificity, and/or AUC between AI and medical experts.
Overall, the performance of the algorithm was on par with thatin human experts and significantly superior to those experts
with less experience [11,13,16,18] (Table 2).
False-positive rates between AI and clinicians were comparedin 2 studies [12,16]. The number of false discoveries occurringin AI was approximate to that by expert and competent
ophthalmologists with respect to image evaluation (AI vs expert
or competent: 9 vs 5 or 11) and treatment suggestion (AI vsexpert or competent: 5 vs 1 or 3) but was lower than that ofnovice ophthalmologists with 5 versus 12 and 8, regarding image
evaluation and treatment suggestion, respectively [12]. The
other study also found the false-positive rate of their deeplearning algorithm in nodule detection being close to the averagelevel of thoracic radiologists (0.3 vs 0.25) [16].
Other performance indices were compared in single studies.
Apart from false positives, Long et al also compared the numberof missed detections between their AI and ophthalmologists,
and their AI outperformed (ie, fewer missed detections) all
ophthalmologists with varying expertise (expert, competent,
and novice). The time to interpret the tested images between
AI and human radiologists was reported by Rajpurkar et al [16]The authors also compared AI and radiologists with respect topositive and negative predictive values, Cohen kappa, and F1
metrics (Table 2).

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 7
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al

Table 2. Comparison between artificial intelligence and human clinicians.
Authors(year)

Performance index (AIa vs human clinicians)
AUCb

Sensitivity

Specificity

Brinker
N/Ac(2019) [14]Details provided in the article

Sensitivity (at specificity=73.3%):86.1%; versus ;86.7%(among 3 resident
dermatologists)

Specificity (atN/Asensitivity=89.4%):
mean=68.2%(range: 47.5%86.25%) versus
mean=64.4%(all 145 dermatologists, range:
22.5%-92.5%);
Specificity (atsensitivity=92.8%):
mean=61.1%
versusmean=57.7 %(among 16 attending dermatologists)

De Fauw
N/A(2018) [19]No comparison

N/A

N/A

Esteva(Internal validation(2017) [15] with 2 dermatologists);Three-class disease partition: 72.1%(SD 0.9%) versus
65.56% and 66.0%;
Nine-class disease
partition: 55.4%(SD1.7) versus 53.3%
and 55.0%

AUC of AI wasreported but nocomparison
with humanclinicians (Details provided in
the article)

AI outperformed theaverage of dermatologists; (Details provided in the article)

AI outperN/A
formed the average of dermatologists (Details
provided in thearticle)

N/A

N/A

González- N/ACastro(2017) [18]

AUC (model 1): N/A
0.9265 versus
0.9813 and0.9074; AUC(model 2):
0.9041 versus
0.8395 and0.8622; AUC(model 3):
0.9152 versus
0.9411 and
0.8934

N/A

N/A

N/A

Accuracy

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Error/weighted
error

False positives

Other indices

N/A

N/A

Device type 1: N/A
Error rate: 5.5%
versus 2 best
retina specialists: 6.7% and6.8% (performed comparably with 2
best and significantly outperformed the other 6 experts);
Device type 2:
Error rate: 3.4%
versus 2.4%
(average) (Details provided in
the article)

N/A

N/A

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 8
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Authors(year)

Shen et al

Performance index (AIa vs human clinicians)
Accuracy

AUCb

Sensitivity

N/A

Error/weighted
error

False positives

Other indices

Youden index (sensitivity + specificity 1): B1+C dataset:>67.62% (trainedwith A1 dataset) and>63.03% (trainedwith A2 dataset) vs48.39% (99% CI
29.16% (SD67.62%); 95% CI
33.76% (SD63.03%); B2+D
dataset: Only onedermatologist performed better thanthe ensemble modeltrained with the A1
dataset, and only
once in three experiments

N/A

N/A

N/A

Kermany
96.6% versus 95.9% N/A(2018) [11] (mean; range: 92.1%99.7%)

97.8% versus 99.3% 97.4% versus(mean; range:
95.4% (mean;
98.2%-100%)
range: 82%99.8%)

6.6% versus
4.8% (mean;
range: 0.4%10.5%)

N/A

N/A

Long
Accuracy (distinguish- N/A(2017) [12] ing patients andhealthy individuals):
100% versus 98%(expert), 98% (Competent), 96% (novice)[mean=97.33%]; Accuracy (opacity areas):
90% versus 90% (expert), 84% (competent), 78% (novice)[mean=84%]Accuracy(densities): 90% versus 90% (expert),
90% (competent),
86% (novice)[mean=88.7%]; Accuracy (location): 96%
versus 88% (expert),
88% (competent),
86% (novice)[mean=82.7%]; Accuracy (treatment suggestion): 90% versus
92% (expert), 92%(competent), 82%(novice)[mean=88.7%]

N/A

N/A

Number of false
positive in 50
cases; Evaluation network(opacity area,
density and location): 9 versus 5(expert), 11(competent), 12(novice); Strategist network(treatment suggestion): 5 versus 1 (expert), 3(competent), 8(novice)

Missed detections: Evaluation network(opacity area,
density and location): 4 versus
11 (expert), 8(competent), 20(novice) Strategist network(treatment suggestion): 0 versus 3 (expert), 1(competent), 1(novice)Han (2018) N/A[13]

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Specificity

N/A

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 9
(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
Authors(year)

Performance index (AIa vs human clinicians)
AccuracyaShen et al

AUCb

Sensitivity

Specificity

Error/weighted
error

False positives

Other indicesNam
N/A(2018) [16]

AUROC (in ra- 80.7% versus
diograph classi- mean=70.4%
fication): 0.91
versusmean=0.885(DLAD higher
than 16 physicians and significantly higher
than 11);
JAFROC FOM(in nodule detection): 0.885 versus mean=0.794(DLAD higher
than all physicians and significantly higher in
15)

No report ofN/Aphysicians’ performance

0.3 versus
mean=0.25

N/A

Rajpurkar Mean proportion cor(2018) [16] rect value for all
pathologies: 0.828(SD=0.12) versus
0.675 (SD=0.15;
board-certified radiologists) and 0.654(SD=0.16; residents)

AUC (cardiomegaly):
0.831 versus
0.888 (P<.05);
AUC (emphysema): 0.704 versus 0.911(P<.05); AUC(hernia): 0.851
versus 0.985;(P<.05); AUC(atelectasis):0.862 versus
0.808 (P<.05);
No significant
difference forother 10
pathologies

CheXNEXt ver- N/Asus board-certfied radiologists
only; Specificity (masses):
0.911 (95% CI
0.880-0.939)
versus 0.933(95% CI 0.9220.944); Specificity (nodules):
0.900 (95% CI
0.867-0.931)
versus 0.937(95% CI 0.9270.947) Specificity (consolidation): 0.927(95% CI 0.8970.954) versus
0.935 (95% CI
0.924-0.946)
Specificity (effusion); 0.921(95% CI 0.8890.951) versus
0.883 (95% CI
0.868-0.898);(detailed comparison on other10 pathologies
are available in
the original article)

N/A

Time to interpret the 420 images: 1.5 minversus 240 min(range 180-300min); Positive
and negative
predictive values; Cohen’s
kappa F1 metric(Details provided in the Appendices of thearticle)CheXNEXt versus
board-certfied radiologists only; Sensitivity (masses): 0.754(95% CI 0.6440.860) versus 0.495(95% CI 0.4430.546); Sensitivity(nodules): 0.690(95% CI 0.5810.797) vs 0.573(95% CI 0.5250.619); Sensitivity(consolidation):
0.594 (95% CI
0.500-0.688) versus
0.456 (95% CI
0.418-0.495); Sensitivity (effusion);
0.674 (95% CI
0.592-0.754) versus
0.761 (95% CI
0.731-0.790); (detailed comparison onother 10 pathologies
are available in theoriginal article)

AI: artificial intelligence.

b

AUC: area under the receiver operating characteristic curve.

c

Not applicable.

Quality Assessment of Included Studies
The methodological quality of included studies (see Figures 2
and 3) was assessed using the Cochrane’s risk-of-bias tool [10].
This tool was designed to assist the assessment on the risk of
http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

bias in reviewed articles based on their reporting in terms ofspecified domains. The evaluation is grounded on whether
individual articles provided supporting details, and the summaryis presented as high, low, or unclear bias in graphs. Overall,most of reviewed studies had a low risk of bias with respect to
JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 10(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
the specified domains (Figures 2 and 3). A total of 3 studies
were classified as unclear risk in particular domains.
Specifically, there was no report on whether blinding ofparticipants and personnel (related to performance bias) was
observed in the study by De Fauw et al [19]. The study by
González-Castro et al [18] was classified as unclear risk in terms

Shen et alof selective reporting (reporting bias) because of failing to report
all prespecified performance indices. Attrition bias rising fromincomplete outcome data (ie, physicians’ performance) was notassessable based on the reporting by Nam et al [16] (seeMultimedia Appendix 2 for details).

Figure 2. Distribution of bias in the included studies.

Figure 3. Risk of bias in the included studies.

Discussion
Principal Findings
Our systematic review identified 9 articles on advanced AI
applications for disease diagnosis. These spanned multiple
medical subjects, including retinal diseases, skin cancers,
pulmonary nodules, and brain tumors. Although several articles
http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

covered similar medical topics, distinct AI algorithms andtraining processes were employed across articles. The validation
methods of AI algorithm effectiveness also varied between
articles. According to our inclusion criteria, only articles
encompassing comparisons of diagnostic performance between
advanced AI and clinical experts were reviewed.

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 11(page number not for citation purposes)

JMIR MEDICAL INFORMATICSThe literature has shown that AI has comparable performancewith medical experts. Major advanced AI approaches such as
deep learning and CNNs yield significant discriminative
performance upon provision of sufficient training datasets. In
addition to relatively high sensitivity and specificity inobject-identifying tasks [11,15], the advantages of AI have also
been visible in the instantaneity of reporting and consistencyof producing results [17]. Although neural network approaches
generally require substantial data for training, recent researchsuggested that it may be feasible to apply AI to rare diseases[11,12] and, in particular circumstances, to databases where alarge number of examples are not available. The combination
with other technologies such as a cloud-based data-sharing
platform would extend AI’s likely use beyond clinical settingsor spatial limits [20].
Most AI achievements can be observed in image recognition[21]. Object-identification tasks were the main applications inmedical diagnoses across the reviewed articles.
Computer-assisted technologies facilitate the rapid detection ofclinical symptoms of interest (eg, benign and malignant) basedon image features (eg, tone and rim) resulting in consistent
outputs. AI-based classification of physical characteristics via
vast numbers of examples is reinforced during training, and thisability is consolidated and gradually levels the discriminative
academic performance in appearance-based diagnoses such as
skin diseases [15,21]. Such AI-assisted imaging-related clinicaltasks can reduce the cognitive burden on human experts [17]and thus increase the efficiency of health care delivery.
AI performs at par with human experts in terms of image
analysis. Image analysis involves a number ofobject-identification tasks whose outputs rely exclusively on
the detection and interpretation of concrete features such as
shapes and colors. The nonfatigue characteristic of advanced
artificial networking enables constant training and learning untilachieving satisfactory accuracy [17]. This shows marked successin disease diagnoses related to image evaluation. This uniqueadvantage of AI, which humans are biologically unlikely topossess, contributed to its performance exceeding that of clinicalprofessionals, as seen in the reviewed articles.
The literature shows that almost every achievement of AI isestablished based on diagnosis outcomes. However, any
assessment of diagnostic outcomes needs to yield meaningful
implications. The diagnostic criteria are developed based onlong-standing and recursive processes inclusive of real-worldpractice appraised by clinicians, as summarized in Table 1.
Although the recently promising self-learning abilities of AI
may lead to additional prospects [22], the viability of suchdiagnostic processes is inevitably determined by human experts
through cumulative clinical experience [23,24]. In other words,
clinical experts are the go-to persons informing AI of what thedesired predictions are. AI is still incapable of interpreting what
it has obtained from data and of providing telling results.
Therefore, the final success of AI is conditionally restricted by
medical professionals who are the real evaluators of theirdiagnostic performance. This signifies its artificial nature in ahuman-dominated medical environment.

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

Shen et al
Given such a relationship between AI and human users, theapplicability of advanced AI and clinical significance cannot
be isolated. The development of AI technology itself may
provide an encouraging outlook on medicine applications, but
an evaluation conducted by medical specialists plays afundamental role in AI’s continued blooming. In medical
applications, AI cannot exist without human engagement
because the final diagnoses need to have real-world implications.
Patient-oriented medicines specify the essence of patient datain the AI establishment and learning process. Each successful
AI, regardless of whether it is database driven or self-learning,needs to eventually improve patients’ health. The tirelesslearning abilities of AI can complement cognitive fatigue inhumans [17] and can substantially improve clinical efficiency.
Its outstanding performance, comparable with that of experts,
saves huge amounts of time in clinical practice, which, in turn,alleviates the tension in the long-established process of thetransition from novice clinician to expert.
Despite being a propitious moment for AI, there are issues tobe addressed in the coming stages. It remains unclear whether
AI can transform the current clinician-dominant assessment inclinical procedures. It is not surprising that a hybrid system
contributed by both AI and physicians would produce more
effective diagnostic practices, as evidenced by 1 of the reviewed
articles [17]. This could, in turn, bring about improved healthcare. Data interpretation still appears to be a significant
challenge to AI. Future research may focus more on this topic.

Comparison With Previous Work
Before this review, several reviews on general AI application
have been available in the specific fields such as neurosurgery,
digital dermoscopy, and interpretation of intrapartum fetal heart
rate [25-27]. However, most of these reviews did not limit theirscope to advanced AI or deep learning, which is deemed to bean emerging interest to health care professionals in terms of
disease diagnoses. Our review particularly compared the
diagnostic performance of advanced AI with that of clinician
experts, providing an updated summary on latest developmentof AI applications to disease diagnoses. Our findings suggestthat AI’s diagnostic performance is at par with clinical experts,
and the streamlined efficiency of AI transcends human doctors.
Acknowledging the practical value of AI added to current
practice, the underpinning of human clinical experience andpatient-centered principle should remain in the future AI
application to disease diagnoses.

Limitations
Our review systematically searched articles published in selectedmajor databases. According to our preset inclusion and exclusion
criteria, we did not specifically review the conference abstracts
that may contain the most developed AI that can inform
diagnostic practice. Only English articles were included in thisreview, and thus relevant studies published in other languages
may have been missed.

Conclusions
In summary, current AI developments have achieved comparable
performance with medical experts in specific fields. Their
predictive performance and streamlined efficiency pertaining
JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 12(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
to disease diagnoses—particularly in medical imaging
tasks—have transcended that of clinicians because of theirtireless and stable characteristics. Further studies can be focusedon other medical imaging such as magnetic resonance imaging
and other image-unrelated medical practices [28,29]. With theShen et alcontinued development of AI-assisted technologies, the clinicalimplications underpinned by clinicians’ experience and guided
by patient-centered health care principles should be consideredin future AI-related and technology-based medical research.

Acknowledgments
This review did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Conflicts of Interest
None declared.

Multimedia Appendix 1
Search terms used to identify articles related to telemedicine and related technology used in disease diagnoses.
[PDF File (Adobe PDF File), 34KB-Multimedia Appendix 1]

Multimedia Appendix 2
Risk of bias table for individual studies.
[PDF File (Adobe PDF File), 105KB-Multimedia Appendix 2]

References
1.
2.
3.

4.

5.
6.
7.

8.
9.
10.
11.

12.
13.

14.

Miller DD, Brown EW. Artificial intelligence in medical practice: the question to the answer? Am J Med 2018
Feb;131(2):129-133. [doi: 10.1016/j.amjmed.2017.10.035] [Medline: 29126825]
Gill TG. Early expert systems: where are they now? MIS Q 1995 Mar;19(1):51-81. [doi: 10.2307/249711]
Park SY, Seo JS, Lee SC, Kim SM. Application of an artificial intelligence method for diagnosing acute appendicitis: thesupport vector machine. In: Park JJ, Stojmenovic I, Choi M, Xhafa F, editors. Future Information Technology: FutureTech.
Berlin, Heidelberg: Springer; 2013:85-92.
Cascianelli S, Scialpi M, Amici S, Forini N, Minestrini M, Fravolini M, et al. Role of artificial intelligence techniques(automatic classifiers) in molecular imaging modalities in neurodegenerative diseases. Curr Alzheimer Res
2017;14(2):198-207. [doi: 10.2174/1567205013666160620122926] [Medline: 27334942]
Setlak G, Dąbrowski M, Szajnar W, Piróg-Mazur M, Kożak T. Semantic Scholar. 2009. Artificial intelligence approachto diabetes diagnostics URL: https://pdfs.semanticscholar.org/40f3/e4017d497bffe556f882d4f1389462296b59.pdf
Delen D, Walker G, Kadam A. Predicting breast cancer survivability: a comparison of three data mining methods. ArtifIntell Med 2005 Jun;34(2):113-127. [doi: 10.1016/j.artmed.2004.07.002] [Medline: 15894176]
Jayatilake D, Ueno T, Teramoto Y, Nakai K, Hidaka K, Ayuzawa S, et al. Smartphone-based real-time assessment ofswallowing ability from the swallowing sound. IEEE J Transl Eng Health Med 2015;3:2900310 [FREE Full text] [doi:10.1109/JTEHM.2015.2500562] [Medline: 27170905]
Krittanawong C, Zhang H, Wang Z, Aydar M, Kitai T. Artificial intelligence in precision cardiovascular medicine. J Am
Coll Cardiol 2017 May 30;69(21):2657-2664 [FREE Full text] [doi: 10.1016/j.jacc.2017.03.571] [Medline: 28545640]
Chi CL, Street WN, Katz DA. A decision support system for cost-effective diagnosis. Artif Intell Med 2010
Nov;50(3):149-161. [doi: 10.1016/j.artmed.2010.08.001] [Medline: 20933375]
Higgins JP, Green S, editors. Cochrane Handbook for Systematic Reviews of Interventions. Edition 5.1. London, England:
The Cochrane Collaboration; 2011.
Kermany DS, Goldbaum M, Cai W, Valentim CC, Liang H, Baxter SL, et al. Identifying medical diagnoses and treatablediseases by image-based deep learning. Cell 2018 Feb 22;172(5):1122-31.e9 [FREE Full text] [doi:10.1016/j.cell.2018.02.010] [Medline: 29474911]
Long E, Lin H, Liu Z, Wu X, Wang L, Jiang J, et al. An artificial intelligence platform for the multihospital collaborative
management of congenital cataracts. Nat Biomed Eng 2017 Jan 30;1(2):1. [doi: 10.1038/s41551-016-0024]
Han SS, Park GH, Lim W, Kim MS, Na JI, Park I, et al. Deep neural networks show an equivalent and often superior
performance to dermatologists in onychomycosis diagnosis: automatic construction of onychomycosis datasets by region-based
convolutional deep neural network. PLoS One 2018;13(1):e0191493 [FREE Full text] [doi: 10.1371/journal.pone.0191493][Medline: 29352285]
Brinker TJ, Hekler A, Enk AH, Klode J, Hauschild A, Berking C, Collaborators. A convolutional neural network trainedwith dermoscopic images performed on par with 145 dermatologists in a clinical melanoma image classification task. Eur
J Cancer 2019 Apr;111:148-154 [FREE Full text] [doi: 10.1016/j.ejca.2019.02.005] [Medline: 30852421]

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 13(page number not for citation purposes)

JMIR MEDICAL INFORMATICS
15.
16.

17.

18.

19.

20.
21.

22.
23.

24.
25.

26.

27.

28.

29.

Shen et al

Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer withdeep neural networks. Nature 2017 Feb 2;542(7639):115-118. [doi: 10.1038/nature21056] [Medline: 28117445]
Nam JG, Park S, Hwang EJ, Lee JH, Jin KN, Lim KY, et al. Development and validation of deep learning-based automatic
detection algorithm for malignant pulmonary nodules on chest radiographs. Radiology 2019 Jan;290(1):218-228. [doi:10.1148/radiol.2018180237] [Medline: 30251934]
Rajpurkar P, Irvin J, Ball RL, Zhu K, Yang B, Mehta H, et al. Deep learning for chest radiograph diagnosis: a retrospective
comparison of the CheXNeXt algorithm to practicing radiologists. PLoS Med 2018 Nov;15(11):e1002686 [FREE Full text][doi: 10.1371/journal.pmed.1002686] [Medline: 30457988]
González-Castro V, Hernández MD, Chappell F, Armitage P, Makin S, Wardlaw J. Reliability of an automatic classifier
for brain enlarged perivascular spaces burden and comparison with human performance. Clin Sci (Lond) 2017 Jul
1;131(13):1465-1481. [doi: 10.1042/CS20170051] [Medline: 28468952]
de Fauw J, Ledsam JR, Romera-Paredes B, Nikolov S, Tomasev N, Blackwell S, et al. Clinically applicable deep learningfor diagnosis and referral in retinal disease. Nat Med 2018 Sep;24(9):1342-1350. [doi: 10.1038/s41591-018-0107-6][Medline: 30104768]
Lin H, Long E, Chen W, Liu Y. Documenting rare disease data in China. Science 2015 Sep 4;349(6252):1064. [doi:10.1126/science.349.6252.1064-b] [Medline: 26339020]
Brinker TJ, Hekler A, Utikal JS, Grabe N, Schadendorf D, Klode J, et al. Skin cancer classification using convolutional
neural networks: systematic review. J Med Internet Res 2018 Oct 17;20(10):e11936 [FREE Full text] [doi: 10.2196/11936][Medline: 30333097]
Silver D, Schrittwieser J, Simonyan K, Antonoglou I, Huang A, Guez A, et al. Mastering the game of Go without humanknowledge. Nature 2017 Oct 18;550(7676):354-359. [doi: 10.1038/nature24270] [Medline: 29052630]
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learningalgorithm for detection of diabetic retinopathy in retinal fundus photographs. J Am Med Assoc 2016 Dec
13;316(22):2402-2410. [doi: 10.1001/jama.2016.17216] [Medline: 27898976]
Amato F, López A, Peña-Méndez EM, Vaňhara P, Hampl A, Havel J. Artificial neural networks in medical diagnosis. J
Appl Biomed 2013 Jul 31;11(2):47-58. [doi: 10.2478/v10136-012-0031-x]
Senders JT, Arnaout O, Karhade AV, Dasenbrock HH, Gormley WB, Broekman ML, et al. Natural and artificial intelligencein neurosurgery: a systematic review. Neurosurgery 2018 Aug 1;83(2):181-192. [doi: 10.1093/neuros/nyx384] [Medline:
28945910]
Balayla J, Shrem GJ. Use of artificial intelligence (AI) in the interpretation of intrapartum fetal heart rate (FHR) tracings:
a systematic review and meta-analysis. Arch Gynecol Obstet 2019 Jul;300(1):7-14. [doi: 10.1007/s00404-019-05151-7][Medline: 31053949]
Rajpara SM, Botello AP, Townend J, Ormerod AD. Systematic review of dermoscopy and digital dermoscopy/artificial
intelligence for the diagnosis of melanoma. Br J Dermatol 2009 Sep;161(3):591-604. [doi: 10.1111/j.1365-2133.2009.09093.x][Medline: 19302072]
de Langavant LC, Bayen E, Yaffe K. Unsupervised machine learning to identify high likelihood of dementia in
population-based surveys: development and validation study. J Med Internet Res 2018 Jul 9;20(7):e10493 [FREE Full text][doi: 10.2196/10493] [Medline: 29986849]
Gibbons C, Richards S, Valderas JM, Campbell J. Supervised machine learning algorithms can classify open-text feedbackof doctor performance with human-level accuracy. J Med Internet Res 2017 Mar 15;19(3):e65 [FREE Full text] [doi:10.2196/jmir.6533] [Medline: 28298265]

Abbreviations
AI: artificial intelligence
AUC: area under the receiver operating characteristic curve
CNN: convolutional neural network
OCT: optical coherence tomography

Edited by G Eysenbach; submitted 01.02.18; peer-reviewed by C Krittanawong, T Arroyo-Gallego, I Gabashvili, M Mulvenna, YH
Yeo; comments to author 17.08.18; revised version received 31.01.19; accepted 19.07.19; published 16.08.19
Please cite as:
Shen J, Zhang CJP, Jiang B, Chen J, Song J, Liu Z, He Z, Wong SY, Fang PH, Ming WK
Artificial Intelligence Versus Clinicians in Disease Diagnosis: Systematic Review
JMIR Med Inform 2019;7(3):e10010
URL: http://medinform.jmir.org/2019/3/e10010/
doi: 10.2196/10010
PMID: 31420959
http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 14(page number not for citation purposes)

JMIR MEDICAL INFORMATICS

Shen et al©Jiayi Shen, Casper J P Zhang, Bangsheng Jiang, Jiebin Chen, Jian Song, Zherui Liu, Zonglin He, Sum Yi Wong, Po-Han Fang,
Wai-Kit Ming. Originally published in JMIR Medical Informatics (http://medinform.jmir.org), 16.08.2019. This is an open-accessarticle distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/),
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR
Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on
http://medinform.jmir.org/, as well as this copyright and license information must be included.

http://medinform.jmir.org/2019/3/e10010/

XSL• FO
RenderX

JMIR Med Inform 2019 | vol. 7 | iss. 3 | e10010 | p. 15(page number not for citation purposes)