Received: 16 May 2024

| Revised: 8 August 2024 | Accepted: 18 August 2024

DOI: 10.1111/cas.16330

ORIGINAL ARTICLE

High-­quality expert annotations enhance artificial intelligencemodel accuracy for osteosarcoma X-­ray diagnosis
Joe Hasei1| Ryuichi Nakahara2 | Yujiro Otsuka3,4,5 | Yusuke Nakamura3 |Tamiya Hironari6 | Naoaki Kahara7 | Shinji Miwa8| Shusa Ohshika9 |Shunji Nishimura10 | Kunihiro Ikuta11 | Shuhei Osaki12 | Aki Yoshida2 |Tomohiro Fujiwara2| Eiji Nakata2| Toshiyuki Kunisada2 | Toshifumi Ozaki2
1
Department of Medical Information and Assistive Technology Development, Okayama University Graduate School of Medicine, Dentistry and Pharmaceutical
Sciences, Okayama, Japan2

Department of Orthopedic Surgery, Okayama University Graduate School of Medicine, Dentistry and Pharmaceutical Sciences, Okayama, Japan3

Department of Radiology, Juntendo University School of Medicine, Tokyo, Japan

4

Milliman, Inc., Tokyo, Japan

5

Plusman LCC, Tokyo, Japan

6

Department of Musculoskeletal Oncology Service, Osaka International Cancer Institute, Osaka, Japan

7

Department of Orthopedic Surgery, Mizushima Central Hospital, Okayama, Japan

8

Department of Orthopedic Surgery, Kanazawa University Graduate School of Medical Sciences, Kanazawa, Japan

9

Department of Orthopedic Surgery, Hirosaki University Graduate School of Medicine, Aomori, Japan10

Department of Orthopedic Surgery, Kindai University Hospital, Osaka, Japan

11

Department of Orthopedic Surgery, Nagoya University Graduate School of Medicine, Nagoya, Japan

12

Department of Musculoskeletal Oncology, National Cancer Center Hospital, Tokyo, Japan

Correspondence
Joe Hasei, Department of Medical
Information and Assistive Technology
Development, Okayama University
Graduate School of Medicine, Dentistry
and Pharmaceutical Sciences, 2-­5-­1
Shikata-­cho, Kita-­ku, Okayama 700-­8558,
Japan.
Email: py3g9rcw@s.okayama-u.ac.jp
Funding information
Japan Society for the Promotionof Science, Grant/Award Number:
21K09228; Japan Agency for Medical
Research and Development, Grant/Award
Number: 22579674

Abstract
Primary malignant bone tumors, such as osteosarcoma, significantly affect the pediatric and young adult populations, necessitating early diagnosis for effective treatment.
This study developed a high-­performance artificial intelligence (AI) model to detect
osteosarcoma from X-­ray images using highly accurate annotated data to improve diagnostic accuracy at initial consultations. Traditional models trained on unannotateddata have shown limited success, with sensitivities of approximately 60%–70%. In contrast, our model used a data-­centric approach with annotations from an experienced
oncologist, achieving a sensitivity of 95.52%, specificity of 96.21%, and an area under
the curve of 0.989. The model was trained using 468 X-­ray images from 31 osteosarcoma cases and 378 normal knee images with a strategy to maximize diversity in the
training and validation sets. It was evaluated using an independent dataset of 268
osteosarcoma and 554 normal knee images to ensure generalizability. By applying theU-­net architecture and advanced image processing techniques such as renormalization and affine transformations, our AI model outperforms existing models, reducing
missed diagnoses and enhancing patient outcomes by facilitating earlier treatment.

This is an open access article under the terms of the Creative Commons Attribution-NonCommercial License, which permits use, distribution and reproduction
in any medium, provided the original work is properly cited and is not used for commercial purposes.
© 2024 The Author(s). Cancer Science published by John Wiley & Sons Australia, Ltd on behalf of Japanese Cancer Association.
Cancer Science. 2024;00:1–10.	﻿

wileyonlinelibrary.com/journal/cas

| 1

|

HASEI et al.

This study highlights the importance of high-­quality training data and advocates ashift towards data-­centric AI development in medical imaging. These insights can beextended to other rare cancers and diseases, underscoring the potential of AI in transforming diagnostic processes in oncology. The integration of this AI model into clinicalworkflows could support physicians in early osteosarcoma detection, thereby improving diagnostic accuracy and patient care.
KEYWORDS

artificial intelligence, clinical decision support, diagnostic imaging, image annotation,
osteosarcoma detection

1|

I NTRO D U C TI O N

In developing AI for reading X-­ray images of primary malignant
bone tumors, conventional reports have utilized models trained onPrimary malignant bone tumors are considered “rare cancers,” withdata without detailed segmentation annotations, with limited per-osteosarcoma being the most common type with an incidence rate offormance, showing sensitivities in the 60%–70% range, which has

1,2

However, these tumors frequently

not significantly improved in recent years.10–12 This suggests that,

affect the pediatric, adolescent, and young adult populations, and var-when aiming to improve the performance of X-­ray image interpreta-ious support measures tailored to their age and social background aretion for primary malignant bone tumors, handling the training datarequired for their treatment.3–5 Although advances have been mademight be more crucial than the type of base model used. Unlike MRI,in the treatment of bone and soft tissue sarcomas, such as multidisci-abnormal findings of primary malignant bone tumors on X-­ray im-plinary treatment, the most significant prognostic factor for malignant

ages are often extremely subtle, making it difficult for physiciansbone tumors is the presence or absence of metastases.6,7 Therefore,

to notice them without missing any, and rendering the task chal-

early diagnosis and initiation of treatment before the development

lenging. Consequently, although there are various reports on MRIof metastases are of utmost importance. However, due to the rarity

image-­reading AI models,13–15 reports on AI models for reading X-­of these tumors, physicians often lack clinical experience, making di-

ray images are extremely limited. Moreover, while multiple images

agnosis at the initial consultation challenging, especially in patientscan be obtained from a single case using MRI or computed tomog-

presenting with pain associated with exercise or growth. While ab-raphy, the number of images obtained from a single case using X-­

normalities can be relatively easily identified through detailed exam-

ray imaging is overwhelmingly small. Therefore, the development of

inations such as MRI, initial consultations are usually limited to X-­rayhigh-­performance models based on extremely limited training data

examinations, making it crucial to minimize missed diagnoses on X-­

is crucial. We developed an AI model for reading X-­ray images of os-

rays taken during the first visit. However, due to the scarcity of spe-teosarcomas using highly accurate annotated training data, demon-

cialized physicians, characteristic findings such as Codman's triangle

strating superior performance compared to previous reports10–12

only 3–4.5 per million people.

8

on X-­ray images can be overlooked. Consequently, missed diagnoses

and achieving clinical-­grade accuracy. This approach is expected tolead to a prolonged time to diagnosis, allowing metastases to occur.

enhance physicians' ability to detect subtle abnormalities in X-­ray

Patients with metastatic disease have poor prognosis, and in some

images, thereby improving the accuracy and speed of diagnosis. The

cases, this diagnostic delay can be fatal.

main objective of this project was to facilitate earlier diagnosis and

To reduce the number of children missed at the initial diagnosis,

treatment initiation by integrating this AI model into the diagnostic

we are developing a reading tool for X-­ray images taken at initial

workflow with the ultimate aim of improving patient outcomes andoutpatient consultations, with the goal of ensuring a consistent level

prognosis.

of reading proficiency in clinical settings worldwide. Recent studies have explored various approaches to artificial intelligence (AI)-­
assisted diagnosis of bone lesions on radiographs. A recent study

2|

M ATE R I A L S A N D M E TH O DS

developed an AI model that classifies X-­ray images into two categories: normal and bone tumors (including benign and malignant).9X-­ray images of 31 patients (468 images) with osteosarcoma locatedAlthough this binary classification approach provides valuable in-in the distal femur treated at Okayama University Hospital were usedsights, it differs from the specific task of detecting malignant tumors.

in this study. All the patients were pathologically diagnosed with os-

Our study builds upon these efforts by developing a model that spe-teosarcoma. Of these, 328 images were used for the training. The

cifically targets the detection of malignant bone tumors, addressing

dataset also included X-­ray images taken by general practitioners, or-a critical need in early cancer diagnosis. This approach aims to en-

thopedic surgeons who did not specialize in oncology, and other physi-

hance the accuracy of initial consultations and potentially improvecians involved in the initial consultation of patients with orthopedic

patient outcomes through earlier detection and treatment initiation.

conditions, ensuring that the model covered a diverse range of cases

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

2

encountered in real-­world clinical settings. This approach not onlythat of a residual block17 with skip connection. The network output

enhances the diagnostic accuracy of the model, but also enables theuses multiresolution maps in the same manner as the U-­Net. Eachevaluation of its applicability in various clinical environments, further

X-­ray image was first renormalized by the window level within thestrengthening the versatility of the model. For the normal knee im-

1st–99th percentile of the pixel value to exclude outliers of pixel

ages, 265 of the 378 images from 54 cases without degeneration werevalue. The images were then resampled into 1 mm squares in the im-used for training, and the remaining images were used for validation.

ager pixel spacing space. During training, patches of 128 mm square

Because osteosarcoma is a rare disease, it is necessary to develop awere cropped to include a lesion area with a ratio of 0.5, augmented

model using a limited image dataset, and it is crucial to use strategies

in random intensity shift (prob = 0.5, offsets = 0.05) and random affinethat maximize the diversity of images obtained from each case. Giventransform (prob = 0.5) using Project MONAI.18 A stochastic gradient

the limited number of cases available due to the rarity of osteosar-

descent19 optimizer was used to train the model with cross-­entropy

coma, we applied a unique approach to split the dataset into training

loss in an end-­to-­end manner. We used One Cycle LR20 with a max-

and internal validation datasets. To maximize the diversity of images inimum learning rate of 1 × 10−1 and a total steps of 100, then per-both subsets and to ensure that the model's generalization ability andformed 40 cycles. At the end of each epoch, the Dice coefficient21

diagnostic accuracy were optimized, we combined all available cases

for the entire annotated tumor region was calculated to determine

and randomly selected 70% of the images from this pooled dataset tothe best metric epoch. The Dice coefficient is an evaluation metricserve as training data, whereas the remaining 30% were used for inter-that assesses the accuracy of the segmentation region drawn by AI

nal validation. This method allowed for images from the same patient

as a potential lesion area compared with the ground-­truth annotatedto be present in both training and validation sets, which may lead totumor region. To evaluate the performance of the image-­based diag-

overfitting. However, this approach was chosen as a trade-­off to miti-

nostic tool in detecting lesions, the sensitivity, specificity, Dice coef-

gate the limitations imposed by the small number of available cases.

ficient, receiver operating characteristic (ROC) curve, and area underTo address the potential issue of overfitting, we used a separate andthe curve (AUC) were calculated, and the optimal cut-­off value forcompletely independent dataset for the final performance evaluation

lesion detection probability was determined. The program was codedof the model, ensuring that the model's generalization ability was as-

using Python 3.11 with Pytorch 2.0.1 (Python Software Foundation).

sessed on unseen data from different institutions (Table 1).

The Pytorch, sklearn, and scipy packages were used for all analyses.

To ensure high-­quality training data, a two-­step validation process

All patient data used in this study were obtained retrospectively, andfor tumor segmentation was implemented. First, a bone and soft tissue

patient consent was obtained using the opt-­out method. Participants

oncologist with ≥10 years of experience meticulously annotated thewere allowed to withdraw from the study. This study was approved bytumor boundaries on the X-­ray images while referencing the corre-

the institutional ethics committee and carried out in accordance withsponding MRI scans to confirm the extent of the tumor. Subsequently,

all ethical guidelines. This ensured the protection of patient privacy

a second specialist, also with extensive experience in musculoskeletal

and transparent data utilization.

oncology, independently reviewed and validated these segmentations.
This dual-­expert approach was used to minimize potential biases andensure the highest possible accuracy in tumor delineation, particularly

3|

R E S U LT S

in cases where tumor boundaries were subtle or ambiguous on X-­ray
images alone. This dataset included 268 X-­ray images of osteosarcoma

Basic statistical measures were calculated using all AI-­based inter-in the distal femur from 32 cases provided by the Osaka International

pretation data, and the distribution of lesion detection probabilities

Cancer Institute, and 554 X-­ray images of normal knees from 148 cases

was confirmed using histograms and kernel density estimation plotsobtained from Mizushima Central Hospital. These images were not(Figure 2A). A ROC curve was created to maximize the AUC. The

used during model training or internal validation processes. On X-­ray

results revealed that our AI model achieved a remarkably high AUC

images of osteosarcoma, the shadows of periosteal reactions caused

of 0.989 (Figure 2B), demonstrating its excellent ability to detect ab-by the tumor are often faint, making it challenging to accurately anno-

normalities. The optimal cut-­off value was selected by maximizingtate the extent of the tumor.

Youden's J statistic (sensitivity + specificity – 1), resulting in a sen-

The network architecture (Figure 1) was based on U-­Net16 shaped

sitivity of approximately 95.52% and a specificity of approximately

convolutional neural networks. Pixel de-­shuffle and pixel shuffle were

96.21% at this threshold. The false positive and false negative rates

carried out before the first convolution and after the last convolu-

were 3.79% and 4.48%, respectively.

tion, respectively, to reduce the total number of network parameters.

These statistical analyses support the high accuracy and re-

Convolutional operations were conducted in each unit, referred to in

liability of our diagnostic tool for detecting abnormal regions in athis study as a “Res block.” A Res block has an architecture similar toclinical setting. High sensitivity has the potential to significantly

TA B L E 1 Number of radiographs
in the dataset used to develop an
artificial intelligence model to detect
osteosarcoma.

Training set
Number ofradiographs

Internal validation set

External test set

BenignMalignant

BenignMalignant

BenignMalignant

265

328

113

140

148

268

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License| 3

HASEI et al.

|

HASEI et al.

F I G U R E 1 Illustration of the network architecture used in our study, based on U-­Net shaped convolutional neural networks. The
architecture processes a 2D input image through several steps, including pixel de-­shuffle before the first convolution and pixel shuffle after
the last convolution to reduce the total number of network parameters. Convolutional operations are performed in each unit, referred toas a “Res block,” which is similar to a residual block with a skip connection. The network output employs multiresolution maps, akin to theU-­Net architecture.
reduce missed diagnoses during initial consultations, thereby con-However, it is important to note that the interpretation of Dice

tributing to early treatment. Moreover, the high specificity enables

coefficients in osteosarcoma X-­r ay analysis requires careful con-the accurate identification of normal cases, leading to a reduction insideration. Malignant bone tumors often display specific features,

unnecessary treatments and additional examinations. Our segmen-such as periosteal reactions, in very limited areas on X-­r ays. Our

tation model displays lines delineating the boundaries of the lesions

annotation approach, guided by MRI, encompasses a broader area

(Figure 3). The Dice coefficient is an essential indicator of the validity

than the localized characteristic changes visible on X-­r ays alone.

of the segmentation performance of an AI model. The analysis re-Consequently, although the AI may accurately identify these criti-

sults showed that the average Dice coefficient for individual tumor

cal localized changes, resulting in a high detection score, the Dicecases with confirmed lesion detection was 0.66 (calculated as thecoefficient might be lower due to the discrepancy between the AI-­

mean of the Dice coefficient for each tumor), with a maximum value

detected area and the broader annotated region. To illustrate thisof 0.94. Furthermore, we undertook a detailed analysis of the cor-point, we present examples of cases with high and low Dice co-relation between lesion detection probability and the Dice coeffi-

efficients (Figure 5). In the high Dice coefficient case (Figure 5A),

cient. The results yielded a correlation coefficient of 0.68, revealing

the lesion detection score of the AI model was 0.99, and the Dicea trend in which higher lesion detection probabilities corresponded

coefficient was 0.94, showing close agreement between AI seg-

to higher Dice coefficients (Figure 4). This positive correlation was

mentation and expert annotation. In contrast, the low Dice coeffi-

statistically significant (p < 0.001), confirming that our AI model con-cient case (Figure 5B) shows an AI model's lesion detection scoresistently showed high performance in identifying abnormal regions.

of 0.82 and a Dice coefficient of 0.46. Despite the lower DiceThis indicates that the proposed AI model can segment abnormal

coefficients, the AI model still identifies the primary area of con-regions with high accuracy.

cern, demonstrating its ability to detect subtle lesions, even when

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

4

| 5

F I G U R E 2 Evaluation of artificial intelligence (AI)-­based lesion detection in X-­ray images. (A) Score distribution. Histogram showing thefrequency distribution of lesion detection probabilities assigned by the AI model to the test set of X-­ray images, with an overlaid kernel
density estimate indicating the score distribution pattern. (B) Receiver operating characteristic (ROC) curve. The ROC curve for the AI
model, depicting the true positive rate (TPR) against the false positive rate (FPR) at various threshold settings. The area under the curve(AUC) is 0.989, illustrating the diagnostic accuracy of the model.

F I G U R E 3 Osteosarcoma radiographs of the distal femur with artificial intelligence (AI) analysis results. (A) Original radiograph. X-­ray
image of a patient's knee with osteosarcoma in the distal femur, where the tumor may be difficult to detect by untrained observers. (B)AI-­processed radiograph. X-­ray image with superimposed lesion detection; the red outline marks the tumor boundary as determined byexpert human annotation, and the green outline shows the lesion area as identified by the AI model, highlighting the utility of the model indetecting subtle osteosarcoma manifestations. The lesion detection score of the AI model was 0.93, and the Dice coefficient was 0.63.
full segmentation is challenging. This apparent discrepancy does

Finally, we compared the performance of our AI model with that

not necessarily indicate poor diagnostic performance. In fact, theof existing models and alternative architectures. A comparative analy-

AI's ability to identify these subtle, localized changes is clinicallysis determined that the sensitivity and specificity of our U-­Net-­based

significant for differential diagnosis, even when the overall Dice

model were significantly superior to those of the other three models

coefficients are lower. Therefore, although the Dice coefficient(Table 2). To further validate the importance of high-­quality anno-

provides valuable insight into segmentation performance, it does

tations and address concerns regarding model fixation, we also im-

not fully reflect the diagnostic capabilities of the model for malig-

plemented a YOLOv5 model22 using the same dataset. Although thenant bone tumors on X-­r ays.

YOLOv5 model did not achieve the performance of our U-­Net-­based

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

HASEI et al.

|

HASEI et al.

F I G U R E 4 Scatter plot illustrating
the correlation between lesion detection
scores using artificial intelligence (AI) andDice coefficients. The plot maps the lesion
detection score assigned by the AI (x-­axis)
against the Dice coefficient (y-­axis), whichmeasures the agreement between the AI-­
detected lesions and the expert-­annotatedground truth. A significant positive
correlation (Pearson's correlation, 0.68;
p < 0.001) shows that higher detection
probabilities were generally associatedwith higher Dice coefficients, indicating
reliable lesion identification using AI.

F I G U R E 5 Artificial intelligence (AI) segmentation results with high and low Dice coefficients. For each case (A, B): left, original
radiograph of the distal femur showing an osteosarcoma lesion; right, AI-­processed image with superimposed lesion detection. The redoutline indicates the tumor boundary, as determined by expert human annotation, whereas the green outline represents the lesion area
identified by the AI model. (A) High Dice coefficient case. The lesion detection score of the AI model was 0.99, and the Dice coefficient
was 0.94, showing close agreement between AI segmentation and expert annotation. (B) Low Dice coefficient case. The AI model's lesion
detection score was 0.82 and the Dice coefficient was 0.46. Despite the lower Dice coefficients, the AI model still identifies the primaryarea of concern, indicating its ability to detect subtle lesions even when full segmentation is challenging.

approach, it still showed remarkably high scores compared to pre-annotated data. Our model showed excellent performance, with aviously reported models, with a sensitivity of 91.41%, specificity ofsensitivity of 95.52%, specificity of 96.21%, and AUC of 0.989, sig-

89.68%, and an AUC of 0.954 (Figure S1). These results emphasizenificantly outperforming existing models. These results suggest thethat the quality of the dataset is a crucial determinant of model per-potential contribution of our model to the early diagnosis of osteo-

formance, regardless of the specific architectural approach adopted.

sarcoma, a rare cancer. Even powerful machine learning algorithms

This observation highlights the significance of high-­quality annota-

struggle to achieve a satisfactory performance without high-­quality

tions in developing robust AI models for medical image analysis.

data. 23,24 In particular, in the diagnosis of rare diseases, such as osteosarcoma, using X-­ray images, the quality of the limited available

4|

DISCUSSION

data is crucial.

4.1 | Performance of AI and its clinical impact

4.2 | Advantages of data-­centric AI development

In this study, we successfully developed a high-­performance deep

There are two approaches for improving the performance of AI sys-learning model for detecting osteosarcomas in X-­ray images bytems: model-­centric and data-­centric AI development. Model-­centric

adopting a data-­centric AI approach and utilizing high-­quality

approaches have become mainstream over the past three decades

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

6

TA B L E 2 Performance comparison of
the proposed artificial intelligence model
to detect osteosarcoma with previously
reported models.

Sensitivity (%)
10

Specificity (%)

Training images(malignant)aBase model

He et al.

77.7

89.6

417

EfficientNet-­B 0

von Schacky
et al.11

62.9

88.2

189

Mask-­RCNN-­X101

Li et al.12

66.7

98.21

108

YOLO-­v5

Our model

95.5

96.2

328

U-­Neta

Calculation of 417 training images: From 741 total malignant tumor images, 596 were used forthe internal dataset after excluding 145 for external testing. In each fold of 5-­fold cross-­validation,
approximately 70% of 596 images (≈417) were used for training. Note that across all folds, nearly
all 596 images were used for training at some point.

and have been accelerated by the significant success of neural net25

optimization of the learning process itself. Therefore, it is crucial

These approaches focus on model development andto carefully consider and use completely independent external

often overlook data quality. In model-­centric AI, the amount andevaluation data for performance assessments to ensure the gen-type of data collected remain fixed, whereas the model is iteratively

eralizability of the model to unseen cases.

work models.

upgraded. Conversely, data-­centric AI development keeps the model

Although the model was capable of detecting lesions in wide-­

fixed and continuously improves data quality. A paradigm shift fromarea images, such as the entire lower limb, it sometimes responded

model-­centric AI development to data-­centric AI has been proposed

to intestinal shadows, indicating that the current model is special-in recent years to enhance the performance of AI systems. 26–29

ized for knee joint X-­ray images (Figure 6). Our model focuses onknee area osteosarcomas, which account for approximately 70% of

4.3 | Challenges in annotating medical images

all cases, particularly in growing children. This approach considersage-­related changes in bone structure, including the growth plate.
Unlike previous models29 that classify images as normal or patholog-

Primary malignant bone tumors may show specific findings suchical, ours specifically detects malignant findings. Future work should

as periosteal reactions in some areas, but many cases have sub-develop region-­specific models for comprehensive AI-­assisted diag-

tle or invisible findings on X-­r ay images. Consequently, annotat-

nosis of primary bone tumors. As bone shape varies by region, de-ing X-­r ay images can result in inconsistencies and variations in theveloping specialized models for each common site of occurrence will

data created by different annotators, making it difficult to consist-be important for the comprehensive AI-­assisted diagnosis of primary

ently obtain high-­quality data. To address this issue, in our study, abone tumors in the future.

specialist with more than 10 years of experience in bone and soft

It is important to note that tumor boundaries are often difficult

tissue tumor treatment meticulously annotated the entire tumor

to discern on X-­ray images alone. By referencing MRI scans during

region while referring to the MRI images to confirm the tumor

the annotation process, we were able to delineate the tumor re-extent. This approach enabled the creation of consistently high-­

gions more accurately. We believe that the potential inter-­annotator

quality data, which likely contributed to the development of high-­variability in MRI-­guided segmentation is likely to be significantly

performance models.

smaller than the error between true tumor extent and segmentations based solely on X-­ray images. This MRI-­guided approach, while

4.4 | Methodological considerations andlimitations

labor-­intensive, provides a more reliable ground truth for training
our AI model, potentially contributing to its high performance. To
illustrate this point, we have included examples of cases where thetrue tumor extent is difficult to determine from X-­ray images alone

Furthermore, to maximize information extraction from the limited(Figure S2).

data available in rare disease research and improve the generalization ability of the model, we randomly used 70% of the entire
image dataset for training and the remaining 30% for internal
validation. This approach allowed the preservation of image di-4.5 | Implications for future research and clinicalpracticeversity within the dataset and ensured an even distribution acrossboth the training and internal validation datasets. However, dis-

Our data-­centric approach offers a promising strategy for AI devel-tributing images from the same patient into both the training andopment in other rare diseases, potentially overcoming the common

validation sets could lead to overfitting, as the model may learn

challenge of limited data availability. By emphasizing the impor-patient-­specific features rather than the generalizable character-tance of high-­quality, expert-­driven annotations over sheer data

istics of osteosarcoma. Although this risk is mitigated by using ex-

quantity, our research provides a novel framework for addressing

ternal data for performance evaluation, it could still influence thethe persistent challenge of data scarcity in rare disease research.

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License| 7

HASEI et al.

|

HASEI et al.

F I G U R E 6 Osteosarcoma radiographs of the distal femur on whole lower extremity images and artificial intelligence (AI) reading results.
(A) Original radiograph. A comprehensive full-­leg X-­ray depicting the distal femur, where a lesion is present but may be difficult to discern
due to its small size in the context of the whole limb. (B) AI-­enhanced radiograph. X-­rays with AI detection overlays. The red contour
represents the expert-­determined lesion boundary, while the green contour indicates the AI-­identified osteosarcoma lesion. The green
markings in the pelvic area, which do not correspond to the red outline, indicate regions where the AI incorrectly identified intestinal
shadows as potential osteosarcoma lesions.
This study presents the first AI model for interpreting primary ma-

One limitation of this study is the use of data from a limited

lignant bone tumors, developed using detailed segmentation an-

number of institutions for training. In the future, multi-­institutional

notations guided by MRI, demonstrating a significant performancecollaborative research will be required to ensure data diversity andimprovement compared with previous reports. It has become

validate the versatility of the model. Additionally, prospective stud-

evident that high-­performance AI models can be developed evenies are required to evaluate the clinical usefulness of developed AI

with limited data by focusing on improving data quality throughmodels.

expert-­driven, MRI-­guided segmentation rather than modifying
the model architecture. This novel approach, leveraging expert
knowledge and multimodal imaging for precise segmentation, has

4.6 | Summary and future directions

resulted in an AI model showing performance suitable for clinicaluse. The impact of these high-­quality, MRI-­guided segmentations

The AI model developed in this study, using a data-­centric approach

was notable, suggesting that this annotation method and focus onwith high-­quality annotated data, achieved exceptional perfor-

segmentation quality could be valuable for future research in AI-­

mance in diagnosing osteosarcoma using X-­rays. With a sensitivity

assisted diagnosis of malignant bone tumors on X-­r ay images. Ourof 95.52%, specificity of 96.21%, and AUC of 0.989, it significantly

model is expected to reduce the number of missed diagnoses andimproves upon previous models.9–12 This advancement could po-

improve early detection rates for osteosarcoma. The success of ourtentially improve patient prognosis by enabling earlier detection

approach in improving AI performance for osteosarcoma detec-

and reducing missed diagnoses during initial consultations. Futuretion, despite limited data, suggests its potential for other rare dis-

work will include multi-­institutional collaborative research to vali-

eases that face similar challenges. While our study focuses on X-­r ay

date the model's versatility and clinical usefulness across different

diagnosis, AI applications in osteosarcoma extend to histopatho-

regions. Prospective studies are needed to assess its impact in real-­

logical diagnosis, as reported in recent studies. 30 The integrationworld clinical settings. Ethical considerations and a clear role divisionof AI across various diagnostic techniques, including radiology andbetween physicians and AI are essential when implementing AI inpathology, could provide a more comprehensive and accurate ap-

healthcare. We recommend using this model as a secondary reader

proach for osteosarcoma diagnosis in the future, potentially lead-to support physician decision-­making, with final diagnostic decisions

ing to earlier detection and improved treatment outcomes.

made by physicians.

However, when introducing AI-­based diagnostic support tools,

This study identifies the potential of AI integration in medical

it is necessary to clarify the division of roles and responsibilities be-technology, establishing a foundation for future advancements.

tween physicians and patients using AI. Artificial intelligence should

These findings may contribute to improving the diagnostic and treat-

be used to support physicians' diagnoses, and the final diagnosis

ment quality of other rare cancers and diseases in various medical

should be made by a physician.

fields.

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

8

AU T H O R C O N T R I B U T I O N S

Shinji Miwa

https://orcid.org/0000-0002-5962-8287

Joe Hasei: Conceptualization; data curation; formal analysis; inves-

Tomohiro Fujiwara

tigation; methodology; supervision; validation; visualization; writing

Eiji Nakata

https://orcid.org/0000-0003-4257-9180

https://orcid.org/0000-0002-5971-4142– original draft. Ryuichi Nakahara: Conceptualization; data curation; methodology; resources. Yujiro Otsuka: Formal analysis; in-

REFERENCES

vestigation; methodology; software; validation. Yusuke Nakamura:

1. Savage SA, Mirabello L. Using epidemiology and genomics to understand osteosarcoma etiology. Sarcoma. 2011;2011:548151.
2. Stiller CA, Bielack SS, Jundt G, Steliarova-­Foucher E. Bone tumoursin European children and adolescents, 1978–1997. Report fromthe automated childhood cancer information system project. Eur J
Cancer. 2006;42:2124-­2135.
3. Gardner MH, Barnes MJ, Bopanna S, et al. Barriers to the useof psychosocial support services among adolescent and young
adult survivors of pediatric cancer. J Adolesc Young Adult Oncol.
2014;3:112-­116.
4. Bleyer A, Barr R, Ries L, Whelan J, Ferrari A. Cancer in Adolescentsand Young Adults. Springer; 2016.
5. Zebrack BJ, Block R, Hayes-­L attin B, et al. Psychosocial service use
and unmet need among recently diagnosed adolescent and young
adult cancer patients. Cancer. 2013;119:201-­214.
6. Xiao ZW, Zheng K, Jiang YZ, et al. Prevalence, risk factors, andprognostic factors of primary malignant bone neoplasms with bonemetastasis at initial diagnosis: a population-­based study. J Oncol.
2022;2022:9935439.
7. Kager L, Zoubek A, Pötschger U, et al. Primary metastatic osteosarcoma: presentation and outcome of patients treated on neoadjuvant cooperative osteosarcoma study group protocols. J Clin Oncol.
2003;21:2011-­2018.
8. Kundu ZS. Classification, imaging, biopsy and staging of osteosarcoma. Indian J Orthop. 2014;48:238-­246.
9. Breden S, Schulze M, Schwarzenberg A, et al. Deep learning-­based
detection of bone tumors around the knee in X-­rays of children. J
Clin Med. 2023;12:5960. doi:10.3390/jcm12185960
10. He Y, Yu H, Aubry S, et al. Deep learning-­based classificationof primary bone tumors on radiographs: a preliminary study.
EBioMedicine. 2020;62:103121.
11. von Schacky CE, Sohn JH, Liu F, et al. Multitask deep learning forsegmentation and classification of primary bone tumors on radiographs. Radiology. 2021;301:398-­4 06.
12. Li J, Zeng Y, Zhang H, et al. Primary bone tumor detection and classification in full-­field bone radiographs via YOLO deep learningmodel. Eur Radiol. 2023;33:4237-­4248.
13. Ouyang T, Yang S, Gou F, Dai Z, Wu J. Rethinking U-­net from an attention perspective with transformers for osteosarcoma MRI image
segmentation. Comput Intell Neurosci. 2022;2022:7973404.
14. Wu J, Zhou L, Ouyang T, et al. BA-­GCA net: boundary-­aware grid
contextual attention net in osteosarcoma MRI image segmentation.
Comput Intell Neurosci. 2022;2022:3881833.
15. Wu J, Zhou L, Gou F, Tan Y. A residual fusion network for osteosarcoma MRI image segmentation in developing countries. Comput
Intell Neurosci. 2022;2022:7285600.
16. Ronneberger O, Fischer P, Brox T. U-­net: convolutional networks
for biomedical image segmentation. Medical Image Computing andComputer-­A ssisted Intervention–MICCAI 2015. Springer International
Publishing; 2015:234-­241.
17. He K, Zhang X, Ren S, Sun J. Deep residual learning for image recognition. Proc IEEE Comput Soc Conf Comput vis Pattern Recognit;
2015:770-­778.
18. Chlap P, Finnegan RN. PlatiPy: processing library and analysis toolkit for medical imaging in python. J Open Source Softw. 2023;8:5374.
19. Ruder S. An overview of gradient descent optimization algorithms.
arXiv [cs.LG]. 2016. doi:10.48550/arXiv.1609.04747
20. Smith LN, Topin N. Super-­convergence: very fast training of neural networks using large learning rates. In: artificial intelligence and machine

Formal analysis; investigation; methodology; software; validation.
Tamiya Hironari: Conceptualization; resources. Naoaki Kahara:
Conceptualization; resources. Shinji Miwa: Conceptualization;
resources. Shusa Ohshika: Conceptualization; resources. Shunji
Nishimura:Conceptualization;

resources.

Kunihiro

Ikuta:

Conceptualization; resources. Shuhei Osaki: Conceptualization;
resources. Aki Yoshida: Conceptualization; resources. Tomohiro
Fujiwara: Conceptualization; investigation; resources; validation.
Eiji Nakata: Conceptualization; investigation; resources; validation. Toshiyuki Kunisada: Conceptualization; validation. Toshifumi
Ozaki: Conceptualization; supervision; validation.
AC K N OW L E D G M E N T S
We extend our deep gratitude to the Japan Society for the
Promotion of Science and the Japan Agency for Medical Researchand Development, whose financial support was fundamental to our
research.
F U N D I N G I N FO R M AT I O NThis research was supported by the Japan Society for the Promotionof Science (JSPS) as part of the Grants-­in-­Aid for Scientific Research(KAKENHI) program, specifically under the categories of Basic
Research C (Grant Number 21K09228 to JH) and the Japan Agency
for Medical Research and Development (AMED) (Grant Number
22579674 to TO).
C O N F L I C T O F I N T E R E S T S TAT E M E N T
The author, Joe Hasei, is affiliated with an endowed chair fundedby Plusman LLC (Tokyo, Japan). This article was written as part ofhis role at the institution and reflects the academic and scientificstandards and policies of the university. Joe Hasei has no financial
relationships or interests to disclose that are directly relevant to thecontent of this study aside from institutional support. The other authors have no conflicts of interest.
E T H I C S S TAT E M E N T
Approval of the research protocol by an institutional review board:This study was approved by the Research Ethics Committee of
Okayama University (approval no. 2304–004).
Informed consent: Informed consent was obtained using an opt-­out
method, in which patients were informed about the study and given
the opportunity to decline participation.
Registry and the registration no. of the study/trial: N/A.
Animal studies: N/A.
ORCID
Joe Hasei

https://orcid.org/0000-0002-4028-0786

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License| 9

HASEI et al.

21.
22.

23.

24.

25.

26.

27.

28.

|

HASEI et al.

learning for multi-­domain operations applications. SPIE; 2019:369-­
386. doi:10.48550/arXiv.1708.07120
Carass A, Roy S, Jog A, et al. Evaluating white matter lesion segmentations with refined Sørensen-­Dice analysis. Sci Rep. 2020;10:8242.
Liu H, Sun F, Gu J, Deng L. SF-­YOLOv5: a lightweight small object detection algorithm based on improved feature fusion mode.
Sensors. 2022;22:5817.
Vidhya G, Nirmala D, Manju T. Quality challenges in deep learning
data collection in perspective of artificial intelligence. J Info Tech
Comp. 2023;4:46-­58.
Whang SE, Roh Y, Song H, Lee JG. Data collection and quality
challenges in deep learning: a data-­centric AI perspective. VLDB J.
2023;32:791-­813.
Hamid OH. Data-­centric and model-­centric AI: twin drivers of compact and robust industry 4.0 solutions. NATO Adv Sci Inst Ser E Appl
Sci. 2023;13:2753.
Hamid OH. From model-­centric to data-­centric AI: a paradigm shift
or rather a complementary approach? 8th International Conferenceon Information Technology Trends (ITT). IEEE; 2022:196-­199.
Zha D, Lai K-­H, Yang F, Zou N, Gao H, Hu X. Data-­centric AI:
Techniques and future perspectives. Proceedings of the 29th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining.
Association for Computing Machinery; 2023:5839-­5840.
Zha D, Bhat ZP, Lai K-­H, Yang F, Hu X. Data-­centric AI: perspectives and challenges. Proceedings of the 2023 SIAM International

Conference on Data Mining (SDM). Society for Industrial and Applied
Mathematics; 2023:945-­948.
29. Singh P. Systematic review of data-­centric approaches in artificial
intelligence and machine learning. Data Sci Manag. 2023;6:144-­157.
30. Gawade S, Bhansali A, Patil K, Shaikh D. Application of the convolutional neural networks and supervised deep-­learning methods
for osteosarcoma bone cancer detection. Healthcare Analytics.
2023;3:100153.

S U P P O R T I N G I N FO R M AT I O N
Additional supporting information can be found online in the
Supporting Information section at the end of this article.

How to cite this article: Hasei J, Nakahara R, Otsuka Y, et al.
High-­quality expert annotations enhance artificial
intelligence model accuracy for osteosarcoma X-­ray
diagnosis. Cancer Sci. 2024;00:1-10. doi:10.1111/cas.16330

13497006, 0, Downloaded from https://onlinelibrary.wiley.com/doi/10.1111/cas.16330 by Okayama University, Wiley Online Library on [08/09/2024]. See the Terms and Conditions (https://onlinelibrary.wiley.com/terms-and-conditions) on Wiley Online Library for rules of use; OA articles are governed by the applicable Creative Commons License

10