# Attention-Based Transfer Learning for Efficient Pneumonia Detection in Chest X-ray Images
## 1 Introduction
ResNet adds feature maps together, but DenseNet [11] makes concatenation between feature maps. Constructing a CNN model in computer vision, most of them are not trained from scratch(random initialization). They just simply used structures of the pre-trained model for a new task. In the medical field, it is difficult to build sufficient datasets with labels because it is expensive to acquire ground truth labels such as ImageNet [15]. From a machine learning perspective, the ground truth is the representation of the original or actual value of the data we want to train. The ground truth labels of the data we used are normal and pneumonia. Transfer learning refers to applying the knowledge accumulated by Artificial Intelligence to another task. It is a process of adapting the model trained on one data set to another data set [16]. As a result, it is possible to train faster at an early stage than the model trained from scratch [17]. In transfer learning, a CNN model which pre-trained on a large and diverse dataset such as ImageNet can be applied to perform a new classification task [15]. When the dataset is insufficient for building and training a new model from scratch, transfer learning can be applied as the solution. Additionally, the amount of data required for training is not as big as that from the scratch model. Transfer learning makes the model more stable. So, Instead of building a model from scratch, simply adapt the structure of pre-trained models that were trained on the ImageNet [15]. Pre-trained CNN models are used to diagnose disease in fields with small datasets. Fine-tuning is the method of transfer learning that changes the classifier to fit a new task. The classifier removes the last fully connected layer from the pre-trained model and changes it by adding a new fully connected layer for the new task. This method slightly modifies existing weights during the training. Afterwards, many studies have focused on further improvement of the performance of CNN and show significant improvements over the years. However, the CNN model only considers the correlation between spatially adjacent pixels in the receptive field defined by the filter size. Therefore, it is difficult to find correlations with distant pixels. Recently, attempts to apply attention mechanisms are followed to solve this problem. Attention is a mean of finding and focusing on the most informational part of the data. The attention model has shown good performance in various fields of computer vision [18; 19]. With an attention mechanism, integrating channel operations into convolution blocks showed great potential in improving performance [20; 21]. As a result, attention is used as a method to complement or replace the existing CNN structure. The squeeze-and-excitation Network(SENet) [20] was suggested to incorporate global information into the decision process of a network with channel-wise interdependency for each convolution block. The key idea behind the Squeeze-And-Excitation block is to incorporate global information into the network's decision process. SENet learns the interactions between channels of convoluted features. While CNN only sees local information of a certain size, the Squeeze-And-Excitation block collects information from all receptive fields. SENet uses the Squeeze-And-Excitation building block for feature recalibration. This method can flexibly apply to existing CNN models and requires little additional computation. It is performed by adding computational blocks to all or some stages of the network such as ResNet [10], Inception [22] and ResNeXt [23]. The Efficient Channel Attention Model(ECA-Net) [21] used \(1D\) Convolution to demonstrate distinct performance improvements with a small number of parameters through cross-channel interactions without dimension reduction. ECA-Net highlighted the problem of the increasing model complexity of SENet. ECA-Net also uses channel-wise-attention, which is the same as SENet. However, ECA-Net excludes the process of dimensionality reduction in the fully connected layer.