<figure><img src="image_1.png" /></figure>

HAL
open science

# Improving Pneumonia Diagnosis with RadImageNet: A Deep Transfer Learning Approach

Arzu Ahmadova, Huseynov Ismail, Ibrahimov Yusif

► To cite this version:

Arzu Ahmadova, Huseynov Ismail, Ibrahimov Yusif. Improving Pneumonia Diagnosis with RadImageNet: A Deep Transfer Learning Approach. 2023. [https://hal.science/hal-04163800](https://hal.science/hal-04163800)

HAL Id: hal-04163800

https://hal.science/hal-04163800

Preprint submitted on 27 Jul 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Improving Pneumonia Diagnosis with RadImageNet: A Deep Transfer Learning Approach

Arzu Ahmadova¹, Ismail Huseynov², Yusif Ibrahimov³

¹Faculty of Mathematics, University of Duisburg-Essen, Essen, Germany,
e-mail: arzu.ahmadova@uni-due.de

²Faculty of Mathematics and Computer Science, University of Münster, Münster,
Germany,
e-mail: ismailhuseynov6666@gmail.com

³Department of Computer Science, University of York, York, United Kingdom,
e-mail: ioseffibrahimov@gmail.com

July 17, 2023

## Abstract

Pneumonia, a lung infection causing inflammation, remains a leading cause of global mortality, resulting in an estimated annual fatality rate of approximately 4 million individuals according to the World Health Organization. Accurate and timely diagnosis of pneumonia in its early stages is crucial for effective patient care. Given the criticality of this issue, our study introduces an automated approach utilizing convolutional neural networks and transfer learning to detect pneumonia in chest X-ray images of children aged 1 to 5 years. We employed two pretrained ResNet-50 models trained on different datasets: ImageNet, containing 14 million natural images, and RadImageNet, consisting of 1.4 million medical images. Our results show that RadImageNet outperformed ImageNet, demonstrating superior performance in pneumonia detection. Evaluation using six key performance metrics revealed that the ResNet-50 model pretrained on RadImageNet achieved superior performance compared to the ImageNet-based model, further emphasizing the efficacy and interpretability of RadImageNet for binary classification tasks in medical image datasets. These findings underscore the importance of RadImageNet as a valuable source of pretrained models, particularly for small medical image datasets.
# 1 Introduction

Pneumonia is an infection that causes inflammation in one or both lungs. It can be caused by various organisms, including bacteria, viruses, and fungi. According to the World Health Organisation (WHO), pneumonia has an estimated annual fatality rate of approximately 4 million people. Consequently, it is the leading cause of death in both children and elderly individuals worldwide. Therefore, ensuring a timely and accurate diagnosis of pneumonia at an early stage is crucial for optimal patient care.

Several radiological imaging techniques can be utilized for lung disease diagnosis, including chest X-ray, computerized tomography (CT) of the lungs, chest ultrasound (US), and magnetic resonance imaging (MRI) of the chest. Among these techniques, a chest X-ray is the most commonly used modality for detecting pneumonia due to its low cost and widespread availability [1]. However, it should be noted that the use of chest X-rays for pneumonia detection can pose challenges, even for experienced radiologists, due to the similarities between pneumonia-related chest X-ray findings and those associated with other lung conditions such as cancer, pulmonary bleeding, and fluid overload [2]. Consequently, relying solely on chest X-rays to diagnose pneumonia may result in delayed diagnosis and hinder the implementation of appropriate treatment strategies.

Therefore, there is a pressing need for the development of improved pneumonia detection methods that can enhance both the accuracy and speed of diagnosis. Researchers and healthcare professionals are actively exploring alternative approaches, including the utilization of advanced imaging techniques and the development of machine learning algorithms, to assist in the interpretation of radiological images. These advancements hold significant potential to facilitate early and accurate detection of pneumonia, enabling timely treatment interventions and ultimately improving patient outcomes.

One promising area of research is the application of machine learning in medical imaging, particularly in the field of medical image classification. Machine learning, a widely utilized technique within the realm of artificial intelligence (AI), enables computers to learn patterns from data samples [3]. By leveraging machine learning algorithms, meaningful patterns can be extracted from various data types, including medical images. Within the domain of machine learning, deep learning, or deep machine learning, is a subset that involves the application of neural network models with multiple layers [4]. These deep neural networks have the ability to extract
higher-level features from raw input data samples, such as medical images.

In the realm of medical image analysis, convolutional neural network (CNN) models have shown great promise [4]. Specifically designed for processing and extracting relevant information from images, CNNs employ an architecture that includes layers for convolution operations, pooling, and nonlinear activation functions. This design enables CNNs to effectively capture spatial relationships and patterns within images, making them highly suitable for tasks such as pneumonia detection in chest X-rays [5].

Over the past decade, numerous studies have utilized various machine learning models and deep neural networks to automatically identify lung diseases from chest X-ray images. It is important to note that training CNN models requires a large volume of labeled samples. Moreover, training a CNN model can be computationally intensive and demands powerful hardware due to the complex architectures of deep neural networks. To address these challenges, the use of transfer learning technique has been proposed as a deep learning approach [6].

Transfer learning has gained popularity as a solution for limited sample sizes and resource constraints. It is a common method in deep learning where a model designed for one problem is repurposed to tackle a different but related task. This approach is particularly valuable in cases where annotated images are scarce, and there are limited computing resources available for training new models from scratch [7]. By leveraging transfer learning, the training process can be expedited with fewer input data, leading to improved performance and generalizability of deep learning models.

Successful transfer learning relies on several factors, including a large sample size, image diversity, and similarity between the training and target application images [6]. In the field of image analysis AI applications, transfer learning utilizing models trained on the ImageNet dataset has been extensively studied. ImageNet is a comprehensive dataset comprising over 14 million images of the natural world [8]. It has served as a valuable resource for developing sophisticated models in computer vision, particularly for image classification tasks. Architectures of deep neural networks such as ResNet [9], Inception [10], and DenseNet [11] pretrained with ImageNet have gained widespread adoption in medical imaging applications, despite the differences in the tasks at hand. However, recent research has demonstrated that pretrained models derived from medical source databases can outperform pretrained models from ImageNet, showcasing the potential for achieving better performance with domain-specific data [12].

Recently, an open radiologic deep learning research dataset called RadImageNet has been developed to address the challenges mentioned earlier [13]. RadImageNet serves as a valuable basis for transfer learning in medical imaging applications. The
database comprises a vast collection of 1.35 million medical images, covering CT, MRI, and US modalities, across 11 distinct anatomic regions. Importantly, these images have annotated by radiologists who possess fellowship training and board certification, ensuring the accuracy and reliability of the dataset.

To further enhance the capabilities of deep learning models in medical imaging, a study was conducted where four convolutional neural networks (CNNs) were trained from scratch using the RadImageNet dataset [13]. The outcomes of this study have provided compelling evidence for the superiority of RadImageNet pretrained models over ImageNet pretrained models in the classification of eight distinct medical applications. Notably, the RadImageNet models showcased notable improvements in performance, ranging from 0.9% to 9.4% in terms of the area under the receiver operating characteristic curve (AUC).

Inspired by the significant advancements observed in the previous study, the present investigation employed pretrained ResNet-50 models obtained from both RadImageNet and ImageNet datasets. The primary objective was to focus on pneumonia detection by analyzing a medical dataset comprising chest X-ray images.

Evaluation using six key performance metrics demonstrated that the ResNet-50 model pretrained on RadImageNet achieved superior performance compared to the ImageNet-based model. These results further highlight the effectiveness and interpretability of RadImageNet for binary classification tasks in medical image datasets. Consequently, these findings reinforce the significance of RadImageNet as a valuable source of pretrained models, particularly for smaller medical image datasets.

## 2 Related Works

In recent years, CNN models have gained significant popularity for their accurate detection of various types of pneumonia infections. Several studies have proposed different CNN architectures to improve the performance of pneumonia detection systems.

Mao et al. [14] introduced a new and improved approach called RetinaNet and Mask R-CNN models for pneumonia detection. Their ensemble model utilized ResNet-50 and ResNet-101 networks as supporting networks. By combining the strengths of these architectures, they achieved enhanced accuracy in detecting pneumonia disease.

Rifa'i et al. [15] experimented with the ResNet-50 architecture using different epoch numbers, including 20 epochs, 50 epochs, and 100 epochs. They found that the best results were obtained using the ResNet-50 architecture with 100 epochs, showcasing its effectiveness in pneumonia detection.
Another study by Hasan et al. [16] presented a model based on the VGG-16 and VGG-19 architectures to detect pneumonia in chest X-ray images. Their approach demonstrated superior accuracy and reliability compared to the transfer learning model InceptionV3.

Chouhan et al. [17] focused on feature extraction using pretrained CNN models on the ImageNet dataset. They proposed an ensemble model that combined the outputs from AlexNet, DenseNet-121, InceptionV3, ResNet-18, and GoogLeNet. The ensemble model achieved an impressive accuracy of 96.4% on unseen dataset, surpassing the performance of individual models.

From the literature survey conducted above, it can be observed that CNN models such as VGG-16, VGG-19, ResNet-50, and ResNet-101 have shown promise in detecting pneumonia. However, it is crucial to conduct a comparative evaluation and analysis of these models to fully understand their effectiveness and accuracy in detecting pneumonia and to identify the most suitable model for a specific application.

Please note that the high performances mentioned earlier were achieved using pretrained models based on ImageNet, which is a collection of natural images, despite the differences in the specific tasks being addressed. In this research, we showcased that a pretrained ResNet-50 model derived from a medical source database called RadImageNet exhibited superior performance compared to the pretrained ResNet-50 model trained on ImageNet when applied to a medical image dataset [19].

# 3 Methodology

## 3.1 Dataset

We utilized a dataset of chest X-ray images, collected from the Guangzhou Women and Children's Medical Center [19] presented in [18], for training and testing our models. The dataset comprises 5856 images, with 1583 images classified as Normal and 4273 images depicting Pneumonia. Therefore, each X-ray scan is labeled with one of two classes, Normal or Pneumonia, based on radiological records. The original images have varying dimensions. Sample images for each class are shown in Figure 1.

## 3.2 Data Preprocessing and Splitting

Image preprocessing is a crucial step in training deep learning models using specific libraries for deep learning tasks like TensorFlow and Keras. This subsection focuses
<figure><img src="image_3.png" /><figcaption>Figure 1: Sample scans of chest X-ray with pneumonia and normal patients.</figcaption></figure>

on how to preprocess an image dataset using these libraries, particularly through the use of the `ImageDataGenerator` module.

When working with image datasets, the images may have varying sizes and dimensions, which can pose a problem for training a model since the input data must be consistent to be processed effectively. Therefore, a common preprocessing step is to resize all images to a fixed size. In our case, we will resize all images to 224 × 224 pixels.

The `ImageDataGenerator` module is a powerful tool for image preprocessing. This module has the ability to rescale the pixel values of images and split the image dataset into training and validation sets. In many cases, pixel values in images can range from 0 to 255. By using the argument `rescale = 1./255`, we can rescale the pixel values of images to be between 0 and 1. This procedure is called *normalization* of the pixel values, and it can help improve the training of the model, as it makes the training more numerically stable.

To ensure that our model is trained effectively and can generalize well to new, unseen data, we split the dataset into three sets: *training*, *validation*, and *test*. The `validation_split` argument in the `ImageDataGenerator` module splits the data into 85% training data and 15% validation data.

Table 1 shows the exact split of our dataset into the three sets. We use the training set to train our model, the validation set to monitor the performance during training, and the test set to evaluate the final performance of the model on new, unseen data. It is important to keep the test set completely separate from the training and validation sets to avoid overfitting and obtain an unbiased estimate of
the model's performance.

<table><caption>Table 1: Distribution of a dataset of chest X-ray images.</caption><thead><tr><th>Data Splitting</th><th>Normal</th><th>Pneumonia</th><th>Total</th></tr></thead><tbody><tr><td>Training samples</td><td>1147</td><td>3301</td><td>4448 (85 %)</td></tr><tr><td>Validation samples</td><td>202</td><td>582</td><td>784 (15 %)</td></tr><tr><td>Test samples</td><td>234</td><td>390</td><td>624</td></tr><tr><td>Total</td><td>1583 (27 %)</td><td>4273 (73 %)</td><td>5856</td></tr></tbody></table>

As shown in Table 1, the dataset consists of 5856 images, with 4273 (73%) images belonging to the Pneumonia class and 1583 (27%) images belonging to the Normal class. The training set contains 4448 images, the validation set contains 784 images, and the test set contains 624 images.

Therefore, the module reads the images one by one, resizes them to the fixed size, shuffles them, and then splits the dataset into smaller batches using the `batch_size` parameter provided by the `flow_from_directory` function. Batching can help improve the convergence of the model during training, allowing for better generalization and faster training times.

# 4 Convolutional Neural Networks

Convolutional Neural Networks (CNNs) are a powerful tool for classifying image datasets. Specifically designed for image analysis and computer vision tasks, CNNs excel in tasks such as disease detection in medical imaging [20]. A CNN model comprises multiple layers that perform various operations on the input image data. The key layers in a CNN model include:

* **Convolutional layer**: This layer applies a convolution operation to the input data, allowing the network to detect features from the image. Typically, a nonlinear activation function, such as the rectified linear unit (ReLU), follows the convolution operation to introduce nonlinearity and extract more complex features. The ReLU activation function sets the negative values to zero and preserves the positive values and it is defined as $\sigma(z) = max(0, z)$, where z is the input value. By eliminating the negative values, ReLU accelerates the training process and helps CNNs learn discriminative features effectively [21]. Convolutional layers produce feature maps as their output, which are then passed on to the next layer in the network for further processing.
* **Pooling layer:** The pooling layer reduces the number of parameters in the feature maps, thereby decreasing computational complexity. It operates by downsampling the spatial dimensions of the feature maps, effectively summarizing the presence of features in the local neighborhood. Common pooling techniques include *max pooling* and *average pooling*, which retain the *maximum* or *average value* within each pooling region, respectively. By discarding irrelevant details and retaining important features, pooling helps improve translation invariance and reduce overfitting.

* **Fully connected or dense layer:** This layer is used for classification tasks, such as image classification problems. In a dense layer, each neuron is connected to every neuron in the previous layer, receiving inputs from all of them. The dense layer calculates a *dot* or *scalar product* between the inputs and learned weights and applies an activation function to produce the output. Common activation functions used in dense layers include *sigmoid* and *softmax* functions:

- **Sigmoid activation function:** The sigmoid function is often used in binary classification tasks. It maps the output of a dense layer to a probability value between 0 and 1, indicating the *likelihood* of belonging to the positive class. The sigmoid function is given by $\sigma(z) = \frac{1}{1+e^{-z}}$, where z is the input value. The sigmoid function's S-shaped curve (see Figure 2) ensures that the output value is always bounded within the range [0, 1], making it suitable for binary classification.

- **Softmax activation function:** The softmax function is commonly used in multi-class classification tasks. It converts the output of a dense layer into a *probability distribution* over multiple classes. The softmax function exponentiates each element of the input vector and normalizes it by dividing by the sum of all exponentiated values. This ensures that the resulting probabilities sum to 1, making it suitable for mutually exclusive classes. The softmax function is given by $\sigma(z_i) = \frac{e^{z_i}}{\sum_{j=1}^{n} e^{z_j}}$, where $z_i$ is the input value and n is the total number of classes.

Figure 2 illustrates the plots of sigmoid and softmax activation functions which are serving binary and multi-class classification problems, respectively.

By combining these layers with appropriate activation functions (e.g., **ReLU** for convolutional layers, **softmax** or **sigmoid** for classification tasks), CNNs can learn and extract meaningful features from images. Consequently, they achieve accurate image classification and perform exceptionally well in various computer vision tasks.
<figure><img src="image_4.png" /><figcaption>Figure 2: Plots of sigmoid and softmax activation functions.</figcaption></figure>

## 4.1 Residual Networks: ResNet-50

Residual Networks (ResNets) are a revolutionary architecture in deep learning designed to overcome the challenges associated with training very deep neural networks. Introduced by K. He et al. [23] from Microsoft Research in 2015, ResNets have had a profound impact on the field of image analysis.

The core concept behind ResNets is the use of *residual connections*, also known as shortcut connections. In traditional architectures, the aim is to directly learn the underlying mapping $H(x)$ by sequentially applying transformations to the input $x$, resulting in a layer output $F(x)$. However, ResNets take a different approach by introducing the idea of residual learning. Residual learning involves learning a residual mapping $F(x) = H(x) - x$ instead. This means that the network focuses on learning the difference between the output of a layer and its input, rather than directly learning the mapping itself. By doing so, the network can capture the underlying features necessary for improving predictions.

The key advantage of residual connections is that they allow information to skip one or more layers and be directly fed to deeper layers. In other words, the output of a residual block in a ResNet is computed as $H(x) := F(x) + x$, where $x$ represents the input and $F(x)$ represents the learned residual mapping. This addition operation enables the network to learn the residual functions and leverage the identity mapping $(x)$ to preserve important information throughout the layers.

Figure 3 illustrates a residual block within a deep residual network, where the residual connection skips two layers.

One of the significant benefits of ResNets is that they alleviate the vanishing gradient problem commonly encountered in deep neural networks [23]. The vanish-
<figure><img src="image_5.png" /><figcaption>Figure 3: A residual block of the deep residual network.</figcaption></figure>

ing gradient problem occurs when the gradients become exponentially small, making it difficult to update the earlier layers effectively. By using residual connections, ResNets provide a shortcut for the *gradient flow*, allowing the gradients to propagate more easily through the network. This facilitates training deep networks with improved optimization and avoids degradation in performance as the network depth increases.

**ResNet-50:** This model is a deep residual CNN with an architecture comprising 50 layers. Its architecture is composed of a series of *convolutional layers*, including *residual blocks*, followed by a *global average pooling layer* and a *fully connected layer* for classification. The residual blocks are the fundamental building blocks of ResNet-50. Each residual block typically contains multiple convolutional layers. These layers learn to extract increasingly complex features from the image data as the depth of the network increases. The residual connections in each block allow for the direct flow of information, enabling the network to effectively capture and utilize both low-level and high-level features.

ResNet-50 has approximately 23.5 million trainable parameters, which makes it a relatively large model. Despite its depth and parameter count, ResNet-50 achieves outstanding performance on various image classification tasks. Its success can be attributed to the effectiveness of the residual connections in training deep networks and capturing intricate feature representations.

# 5 Image Datasets: ImageNet and RadImageNet

ImageNet is a widely used, publicly available dataset for image classification tasks in computer vision research [8]. It is known for its large-scale collection of labeled images and has become a benchmark dataset for training and evaluating deep learning
models, particularly CNNs, in the field of image classification [12].

The ImageNet dataset consists of over 14 million labeled images that are categorized into more than 20,000 different classes or categories. The dataset covers a wide range of object categories, including natural images such as animals, plants, and everyday objects. It has played a crucial role in advancing the state-of-the-art in image classification and has been instrumental in the development of deep learning algorithms.

On the other hand, the RadImageNet database is a significant addition to the field of medical imaging. It is a large-scale dataset that comprises approximately 1.4 million radiologic images, specifically focusing on CT, MRI, and US modalities [13]. The images in RadImageNet cover 11 anatomic regions and have been meticulously annotated by fellowship-trained and board-certified radiologists.

The annotations provided by the experts in the field of radiology make RadIm-
ageNet a valuable resource for training and evaluating deep learning models specifi-
cally designed for medical image analysis. In [13], the authors have utilized RadIm-
ageNet to train neural networks from scratch, enabling the development of powerful
pretrained models that excel in tasks related to medical image classification and anal-
ysis. Therefore, ImageNet is a widely recognized dataset for general image classifi-
cation tasks, while RadImageNet serves as an essential resource for medical imaging
research, particularly in the development of deep learning models for medical image
analysis.

6 Transfer Learning

Transfer learning has gained popularity in deep learning due to the lack of annotated images and limited computing resources required to train models from scratch. This technique enables faster training with fewer input data and improves the performance and generalizability of deep learning models. In medical imaging applications, transfer learning using models trained on ImageNet has been extensively studied. The ResNet, DenseNet, and Inception architectures have been widely adopted for classifying pneumonia disease.

While medical imaging models pretrained with ImageNet have shown high perfor-
mance, recent studies have demonstrated that models pretrained on medical source
datasets can achieve even better results. However, the effectiveness of transfer learn-
ing depends on several factors, including a large sample size, diverse images, and the
similarity between the training and target application images. Drawing inspiration
from these concepts, we utilized the pretrained ResNet-50 model trained on RadIm-
ageNet, a comprehensive and diverse medical image dataset. We then compared its
performance with the pretrained ResNet-50 model trained on ImageNet. It is im-
portant to note that we employed the same architecture for both models to ensure a
fair comparison.

In our approach, we utilized pretrained ResNet-50 models that were originally
trained on ImageNet and RadImageNet, respectively. To tailor these models to
our specific task, we removed the original output layer. This involved discarding the
final classification layer, which was initially designed to classify images into numerous
categories, and adapting it for our binary classification problem.

Next, we froze all the layers in the networks, which means that we prevented their
weights from being updated during the training process. This approach is commonly
referred to as transfer learning, as we leveraged the knowledge and representations
learned by the models on the respective datasets (ImageNet and RadImageNet) to
benefit our own classification task.

To customize the network for our specific task, we added a **flatten layer** after
the frozen layers. This layer reshapes the output from the previous layers into a
one-dimensional vector, which serves as the input for subsequent layers. In addition,
we included a dense layer with a **sigmoid** activation function. This dense layer has a
number of neurons equal to the desired number of classes in our binary classification
problem. The **sigmoid** activation function is commonly used in binary classification
tasks as it outputs probabilities for the two classes or labels. The output of this layer
represents the predicted probabilities for each class, which can be interpreted as the
likelihood of an input image belonging to a certain class.

Meanwhile, we utilized *class activation maps* to facilitate the debugging of deep
neural networks. To achieve this, we employed the Gradient-weighted Class Activa-
tion Mapping (Grad-CAM) algorithm for both models. By leveraging this technique,
we gained valuable insights into the areas of interest within chest X-ray radiographs
that the proposed network focused on while detecting pneumonia disease. The re-
sult of Grad-CAM is presented as a *heatmap* visualization, allowing for a clearer
understanding of the network's attention regions.

7 Training Phase

In this section, we employed a transfer learning approach to evaluate and compare
pretrained ResNet-50 models on the ImageNet and RadImageNet datasets, respec-
tively. Initially, radiologists were tasked with distinguishing between Pneumonia
chest X-ray images and Normal images. Therefore, our models were designed to ad-
dress a binary classification problem, specifically classifying images as either Pneu-
monia or Normal.
All of the proposed networks were trained using a **binary cross-entropy** loss function. Furthermore, the convergence results for **gradient descent systems** play a crucial role in understanding the behavior and efficiency of optimization algorithms [22]. To optimize our models, we employed the **Adam** optimizer, which is a **stochastic gradient descent** (SGD) method with additional features. The **Adam** optimizer has gained popularity in various deep learning tasks due to its adaptive learning rates and momentum-based updates. We set the **learning rate** to 0.001, the **batch size** to 16, and the number of **epochs** (iterations) to 25. Python was utilized in conjunction with the **TensorFlow** and **Keras** libraries to train, validate, and test our proposed models. The training process was conducted on a Kaggle notebook equipped with two NVIDIA T4 GPUs and 16 GB of allocated RAM. The codes for this research work can be accessed at https://github.com/IsmailHuseynov/RadImageNet.

Figure 4a depicts the training and validation curves for accuracy, while Figure 4b illustrates the training and validation curves for loss of each model. For the ImageNet case, the training data accuracy increases sharply from epoch 0 to epoch 4, and gradually increases until epoch 25, which is equivalent to 0.96. The accuracy graph of the validation dataset is 0.98 for epoch 14, as shown in Figure 4a. In terms of the training results, there is a significant decline in the loss curve from epoch 0 to epoch 25, where the loss is close to 0.1. As for the loss of the validation results, the loss curve declines with fluctuation, and at epoch 16, it equals 0.07.
<figure><img src="image_6.png" /><figcaption>Figure 4: Accuracy and loss curves of the proposed ResNet-50 models</figcaption></figure>

However, the scenario is different for the RadImageNet case. In the RadImageNet case, the training data accuracy increases sharply from epoch 0 to epoch 6, reaching 1 at epoch 6. The accuracy slope of the validation dataset increases with minor fluctuations and reaches 0.985 at epoch 8, as shown in Figure 4b. As for the training results, there is a significant decline in the loss curve from epoch 0 to epoch 5, at which the loss is close to 0.01. The loss curve of the validation dataset shows a smooth decline, with a loss of 0.05 at epoch 7.
<figure><img src="image_7.png" /><figcaption>Figure 5: Comparison of accuracy and loss curves of the proposed ResNet-50 models.</figcaption></figure>

A comparison of the accuracy and loss curves achieved by each proposed model on the training and validation datasets is depicted in Figure 5. These results clearly indicate that the RadImageNet case outperforms the ImageNet case, showcasing superior performance in terms of accuracy score and loss value during 25 epochs. This observation highlights the effectiveness of the RadImageNet model in achieving better results compared to the ImageNet model on the training and validation datasets. Now, it is worth noting that further analysis and evaluation, including testing on an independent dataset, are required to validate these findings and ensure their generalizability.
# 8 Evaluation Criteria

Upon completing the training phase, we proceeded to evaluate the performance of the proposed models by measuring their effectiveness on the test dataset. We employed six key performance metrics to compare and assess the models' capabilities: accuracy, recall, precision, F1-score, area under the curve (AUC), and receiver operating characteristic (ROC) curve:

* **Accuracy** represents the overall correctness of the model's prediction, measuring the proportion of correctly classified instances out of the total number of instances;

* **Recall** indicates the model's ability to correctly identify positive instances among all the actual positive instances in the dataset;

* **Precision** measures the model's ability to correctly classify positive instances among all instances predicted as positive. It focuses on minimizing false positives;

* **F1-score** is a metric that combines both precision and recall into a single value, providing a balanced measure of a model's performance. The F1-score is a *harmonic mean* of the precision and recall. The highest possible value of an F1-score is 1, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero;

* **Area under the curve (AUC)** measures the model's ability to distinguish between positive (Pneumonia) and negative (Normal) classes. The AUC value ranges from 0 to 1, where a value of 0.5 indicates a model that performs no better than random guessing, and a value of 1 indicates a perfect model;

* **Receiver operating characteristic (ROC)** is a graphical representation of the performance of a binary classification problem. It plots the true positive rate (TPR) on the y-axis against the false positive rate (FPR) on the x-axis. The ROC curve demonstrates how well the model can distinguish between positive (Pneumonia) and negative (Normal) classes at different classification thresholds. The closer the curve is to the top-left corner of the plot, the better the model's performance.
# 9 Results and Discussion

Table 2 compares the performance metrics between proposed models on the test dataset. The pretrained ResNet-50 model on ImageNet dataset achieves 84% accuracy score, while ResNet-50 model pretrained on the RadImageNet dataset achieves 90% accuracy on testing dataset. In comparison with these models, the pretrained model on RadImageNet showed better performance than the pretrained model on ImageNet dataset. The output may appear excellent in terms of accuracy score, but it can be misleading since the used dataset of chest X-rays is an *imbalanced* dataset (73 % Pneumonia class and 23 % Normal class). In such scenarios, other metrics such as precision, recall and F1-score should be used for each class in conjunction with the accuracy score to gain better understanding of the model's performance. As observable in Table 2, the precision, recall and F1-score in RadImageNet case shows superior performance than ImageNet case for each class.

<table><caption>Table 2: Performance metrics for the proposed ResNet-50 models.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Precision</th><th colspan="2">Recall</th><th colspan="2">F1-score</th></tr><tr><th>ImageNet</th><th>RadImageNet</th><th>ImageNet</th><th>RadImageNet</th><th>ImageNet</th><th>RadImageNet</th></tr></thead><tbody><tr><td>ResNet-50</td><td></td><td></td><td></td><td></td><td></td><td></td></tr><tr><td>Normal</td><td>97%</td><td>99%</td><td>60%</td><td>73%</td><td>74%</td><td>84%</td></tr><tr><td>Pneumonia</td><td>80%</td><td>86%</td><td>99%</td><td>99%</td><td>89%</td><td>92%</td></tr></tbody></table>

Figure 6 presents a comparison of the *confusion matrices* obtained from each proposed model. The confusion matrix provides valuable insights into the performance of a classification model, illustrating four key parameters: true positive (TP), false positive (FP), true negative (TN), and false negative (FN).

Specifically, Figure 6a illustrates the confusion matrix for the ResNet-50 model pretrained on the ImageNet dataset. This particular model accurately predicts 140 out of 234 Normal chest X-rays, but it misclassifies 94 normal chest X-rays as Pneumonia. Moreover, it correctly predicts 385 out of 390 Pneumonia chest X-rays, while mistakenly classifying 5 Pneumonia X-rays as Normal chest X-rays.

On the other hand, Figure 6b portrays the confusion matrix for the ResNet-50 model pretrained on the RadImageNet dataset. In this case, the model achieves improved performance compared to the ImageNet scenario. It accurately predicts 171 out of 234 Normal chest X-rays, with only 63 Normal chest X-rays being misclassified as Pneumonia. Additionally, it correctly predicts 388 out of 390 Pneumonia chest X-rays, misclassifying only 2 Pneumonia X-rays as Normal chest X-rays.

Therefore, based on the comparison of these confusion matrices, it is evident that the RadImageNet case outperforms the ImageNet case, demonstrating superior performance in terms of classification accuracy.
<figure><img src="image_8.png" /><figcaption>Figure 6: Confusion matrices of the proposed ResNet-50 models.</figcaption></figure>

The AUC of the ROC curve is widely regarded as one of the most crucial evaluation metrics for assessing the performance of classification models. It provides a comprehensive measure of the model's ability to correctly classify instances across all possible thresholds. By analyzing the ROC curves and AUC values, we can determine the effectiveness of the transfer learning models in the specific classification tasks at hand. The receiver operating characteristic (ROC) curves depicting the performance of different transfer learning models on the testing dataset can be observed in Figure 7. The ROC curve illustrates the trade-off between the true positive rate (TPR) and the false positive rate (FPR) for varying classification thresholds.

In the ImageNet case (Figure 7a), the pretrained ResNet-50 model achieved an area under the ROC curve (AUC) of 96.6%. This indicates a high discriminative ability of the model in distinguishing between positive (Pneumonia) and negative (Normal) instances. On the other hand, in the RadImageNet case (Figure 7b), the pretrained ResNet-50 model achieved an AUC of 98.1%. This suggests that RadImageNet case demonstrates an even better performance and higher accuracy in classifying the data from the dataset of chest X-ray images of lungs [19].
<figure><img src="image_9.png" /><figcaption>Figure 7: ROC curves with AUC for the proposed ResNet-50 models.</figcaption></figure>

<figure><img src="image_10.png" /><figcaption>Figure 8: X-ray image and the corresponding heatmaps</figcaption></figure>

Figure 8 presents examples of Grad-CAM activation maps, which are utilized to
identify informative regions in chest X-ray images for pneumonia detection using the proposed ResNet-50 models. As shown in Figure 8, the ResNet-50 model pretrained on the RadImageNet dataset effectively focuses on the lungs to detect pneumonia, surpassing the performance of the ResNet-50 model pretrained on the ImageNet dataset. An intriguing finding from our study is that all the chest X-rays depicted in Figure 8 exhibit *class activation maps* that highlight regions within the chest, indicating that the proposed models exclusively consider areas inside the chest and effectively leverage heat maps to identify lung inflammation and diagnose pneumonia.

# 10 Conclusion and Future Directions

In this study, we employed two CNN-based deep transfer learning algorithms to automatically classify chest X-ray radiographs into Pneumonia and Normal classes, addressing a binary classification problem. We utilized two ResNet-50 models pretrained on the natural image dataset of ImageNet and the medical image collection of RadImageNet, respectively, while maintaining the same architecture. By leveraging transfer learning, we accelerated the training of our proposed models and overcame the challenge of limited data samples.

Based on six key performance metrics, the ResNet-50 model pretrained on RadImageNet showcased superior performance compared to the ImageNet case, demonstrating its higher interpretability and effectiveness in binary classification tasks on medical image datasets. This finding highlights the value of RadImageNet as a pretrained model source, particularly for small medical image datasets.

Moving forward, our research will expand to encompass the detection and differentiation of multi-class X-ray images, specifically targeting bacterial and viral pneumonia cases. Additionally, we plan to create new pretrained VGG-16 and VGG-19 deep neural network models based on RadImageNet, a large-scale medical image dataset. These models will be further explored for disease detection beyond the scope of ImageNet, offering enhanced utility and applicability in the medical domain.

# References

[1] Y. Li, Z. Zhang, C. Dai, Q. Dong, S. Badrigilan, Accuracy of deep learning for automated detection of pneumonia using chest X-ray images: a systematic review and meta-analysis, Computers in Biology and Medicine, 123 (2020) 103898.
[2] G. Liang, L. Zheng, A transfer learning method with deep residual network for pediatric pneumonia diagnosis, Computer Methods and Programs in Biomedicine, 187 (2020) 104964.

[3] B. J. Erickson, P. Korfiatis, Z. Akkus, T. L. Kline, Machine learning for medical imaging, Radiographics, 37 (2017) 505-515.

[4] S. Do, K. D. Song, J. W. Chung, Basics of deep learning: a radiologist's guide to understanding published radiology articles on deep learning, Korean Journal of Radiology, 21 (2020) 33-41.

[5] P. Lakhani, B. Sundaram, Deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks, Radiology, 284 (2017) 574-582.

[6] S. J. Pan, Q. Yang, A survey of transfer learning, IEEE Transactions on Knowledge and Data Engineering, 22 (10) (2010) 1345-1359.

[7] K. Weiss, T. M. Khoshgoftaar, D. Wang, A survey of transfer learning, Journal of Big Data, (2016) 3-9.

[8] O. Russakovsky, J. Deng, H. Su et al., ImageNet large scale visual recognition challenge, International Journal of Computer Vision, 115 (3) (2015) 211-252.

[9] K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, (2016) 770-778.

[10] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, Z. Wojna, Rethinking the Inception Architecture for Computer Vision, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, (2016) 2818-2826.

[11] G. Huang, Z. Liu, L. Van Der Maaten, K. Q. Weinberger, Densely Connected Convolutional Networks, 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, (2017) 2261-2269.

[12] Y. Xie, D. Richmond, Pre-training on grayscale ImageNet improves medical image classification. In: Leal-Taixé, L., Roth, S. (eds) Computer Vision – ECCV 2018 Workshops. ECCV 2018. Lecture Notes in Computer Science, (11134). Springer, Cham, (2019) 476-484.
[13] X. Mei, Z. Liu, P. M. Robson, B. Marinelli, M. Huang, A. Doshi, A. Jacobi, C. Cao, K. E. Link, T. Yang et al., RadImageNet: An Open Radiologic Deep Learning Research Dataset for Effective Transfer Learning, Radiology: Artificial Intelligence, 4 (2022).

[14] L. mao, T. Yumeng, C. Lina, Pneumonia Detection in chest X-rays: a deep learning approach based on ensemble RetinaNet and Mask R-CNN, 2020 Eighth International Conference on Advanced Cloud and Big Data (CBD), Taiyuan, China, (2020) 213-218.

[15] A. M. Rifa'i, E. Utami, D. Ariatmanto, Analysis for Diagnosis of Pneumonia Symptoms Using Chest X-Ray Based on Resnet-50 Models With Different Epoch, 2022 6th International Conference on Information Technology, Information Systems and Electrical Engineering (ICITISEE), Yogyakarta, Indonesia, (2022) 471-476.

[16] M. M. Hasan, M. Md. Jahangir Kabir, M. R. Haque, M. Ahmed, A Combined Approach Using Image Processing and Deep Learning to Detect Pneumonia from Chest X-Ray Image, 2019 3rd International Conference on Electrical, Computer & Telecommunication Engineering (ICECTE), Rajshahi, Bangladesh, (2019) 89-92.

[17] V. Chouhan, S. K. Singh, A. Khamparia, D. Gupta, P. Tiwari, C. Moreira, R. Damaševičius, V. H. C. de Albuquerque, A Novel Transfer Learning Based Approach for Pneumonia Detection in Chest X-ray Images, Applied Sciences, 10(2) (2020) 559.

[18] D. S. Kermany, M. Goldbaum, W. Cai, C. C. S. Valentim, H. Liang, S. L. Baxter, A. McKeown, G. Yang, X. Wu, F. Yan et al., Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning, Cell, 172 (2018) 1122-1131.

[19] D. S. Kermany, K. Zhang, M. Goldbaum, Labeled Optical Coherence Tomography (OCT) and Chest X-Ray Images for Classification, Mendeley Data, V2, (2018) doi: 10.17632/rscbjbr9sj.2.

[20] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition, arXiv preprint arXiv: 1409.1556, (2014).

[21] J. Berner, D. Elbrächter, P. Grohs, A. Jentzen, Towards a regularity theory for ReLU networks – chain rule and global error estimates, 2019 13th International
conference on Sampling Theory and Applications (SampTA), Bordeaux, France,
(2019) 1-5.

[22] A. Ahmadova, Convergence results for gradient flow and gradient descent sys-
tems in the artificial neural network training, arXiv preprint arXiv:2306.13086,
(2023)

[23] K. He, X. Zhang, S. Ren, J. Sun, Deep Residual Learning for Image Recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, (2016) 770-778.