Should we Replace CNNs with Transformers for Medical Images?
Authors: Christos Matsoukas
Johan Fredin Haslum
Moein Sorkhei
Magnus Soderberg
Kevin Smith
Venue: ICLR 2022 Submitted (2022)
Keywords: vision transformers
medical image analysis

Abstract: Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis, pushing the state-of-the-art in classification, detection and segmentation tasks. Recently, vision transformers (ViTs) have appeared as a competitive alternative to CNNs, yielding impressive levels of performance in the natural image domain, while possessing several interesting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is feasible to switch to transformer-based models in the medical imaging domain as well, or if we should keep working with CNNs - can we trivially replace CNNs with transformers? We consider this question in a series of experiments on several standard medical image benchmark datasets and tasks. Our findings show that, while CNNs perform better if trained from scratch, off-the-shelf vision transformers are on par with CNNs when pretrained on ImageNet in both classification and segmentation tasks. Further, ViTs often outperform their CNN counterparts when pretrained using self-supervision.

Under review as a conference paper at ICLR 2022
CAN TRANSFORMERS REPLACE CNNS FOR MEDICAL IMAGE CLASSIFICATION?
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional Neural Networks (CNNs) have reigned for a decade as the de facto approach to automated medical image diagnosis, pushing the state-of-the-ard in classification, detection and segmentation tasks. Recently, vision transformers (ViTs) have appeared levels of performance in the natural image domain, while possessing several in- teresting properties that could prove beneficial for medical imaging tasks. In this work, we explore whether it is feasible to switch to transformer-bast based models for medical image classification as well, or if we should keep working with CNNs- can we trivially replace CNNs with transformers? We consider this question in a series of experiments on several standard medical image benchmark datasets and tasks. Our findings show that, while CNNs perform better if trained from scratch, off-the- off-th vision transformers can perform on par with CNNs when pretrained ImageNet, both in supervised and self supeesueeee setting.
competitive alternative to CNNs, yielding impressive
INTRODUCTION
Breakthroughs in medical image analysis over the past decade have been largely fuelled by convo- lutional neural networks (CNNs). CNN architectures have served as the workhorse for numerous medical image analysis tasks including breast cancer detection (Wu et al.) 2019), ultrasound diag- nosis (Christiansen al. 2021), and diabetic retinopathy (De Fauw al. 2018). Whether applied directly as a plug-and-play solution used as the backbone for bespoke model, CNNs such as ResNet (He et al. 2016) are the dominant model for medical image analysis. Recently, however, vision transformers have gained increased popularity for natural image recognition tasks, possibly signalling a transition from convolution-basee feature extractors to attention- based models (ViTs). This raises the question: can ViTs also replace CNNs in the medical imaging domain?
In the natural image domain, transformers have been shown to outperform CNNs on standard vision tasks such IMAGENET classification (Dosovitskiy et al. 2020), as well in object detection (Car- ion et al. 2020) and semantic segmentation (Ranftl et al. (021). Interestingly, transformers succeed despite their lack of inductive biases tied to the convolution operation. This may be explained, in part, by the attention mechanism central to transformers which offers several key advantages over convolutions: explicitly models and more efficiently captures long-range relationships (Raghu et al. 2021), and it has the capacity for adaptive modeling via dynamically computed self-attention weights that capture relationships between tokens. In addition, it provides a type of built-in saliency, giving insight as to what the model focused on (Caron et al. 202T).
Over the years. we have gained solid understanding of how CNNs perform on medical images, and which techniques benefit them. For example, it is well-known that performance drops dramatically when training data is scarce (Cho et af.) 2015). This problem is particularly acute in the medical imaging domain, where datasets are smaller and often accompanied by less reliable labels due to the inherent ambiguity of medical diagnosis. For CNNs, the standard solution is to employ transfer larger dataset such as IMAGENET
learning (Morid et al. 2020): typically, model is pretrained (Deng et al. 2009) and then fine-tuned for specific tasks using smaller, specialized datasets. For medical imaging, CNNs pre- pre-trained on IMAGENET typically outperform those trained from scratch, both in terms of final performance and reduced training time despite the differences between the two
domains (Morid al. 2020 Raghu al. 2019 Tajbakhsh al., 2016). However, recent studies

---

Under review as a conference paper at ICLR 2022
show that, for CNNs, transfer from IMAGENET to medical tasks is not as useful as previously thought (Raghu et al. 2019 Neyshabur et al. 2020).
Supervised transfer learning requires many annotations. This can be problematic for medical tasks where annotations are costly and require specialized experts. Self-supervised approaches, on the other hand, can learn powerful representations by leveraging the intrinsic structure present in the image, rather than explicit labels (He et al.] 2020; Chen et al.) 2020). Although self-supervision performs best when large amounts of unlabelled data are available, has been shown be effective
with CNNs for certain medical image analysis tasks (Azizi et al. 2021 Sowrirajan et al. (021).
In contrast to CNNs, we know relatively little about how vision transformers perform at medical image classification. Are vanilla ViTs competitive with CNNs? Despite their success in the natural image domain, there are questions that cast doubt as to whether that success will translate to medical tasks. Evidence suggests that ViTs require very large datasets to outperform CNNs - in Dosovitskiy jt al.] (2020), the benefits of ViTs only became evident when Google's private 300 million image dataset, JFT-300M, was used for pretraining, Reliance data of this scale may be a barrier to the application of transformers for medical tasks. As outlined above, CNNs rely on techniques such as transfer learning and self-supervision to perform well on medical datasets. Are these techniques as effective for transformers as they are for CNNs? Finally, the variety and type of patterns and textures in medical images differ significantly from the natural domain. Studies on CNNs indicate that the benefits from transfer diminish with the distance from the source domain (Azizpour et al., 2016 Raghu et al., 2019; Neyshabur al. 2020) Can ViTs cope with this difference?
In this work we explore whether it is feasible for VITs to replace CNNs for medical image classifica- tion, and if there is advantage of doing so. Given the enormous computational cost of considering the countless variations in CNN and ViT architectures, we limit our study to prototypical CNN and ViT models over a representative set of well-known publicly available medical image benchmark datasets. Through these experiments we show that:
ViTs pretrained on IMAGENET perform comparably to CNNs for classification tasks on medical images. While CNNs outperform ViTs when trained from scratch, ViT' receive larger boost in performance from pretraining on IMAGENET.
Despite their reliance on large datasets, self-supervised ViTs perform on par or better than CNNs, if IMAGENET pretraining utilized, albeit by small margin.
Features of off-the-shelf IMAGENET pre-trained ViTs are versatile. k--N tests show self- supervised ViT representations outperform CNNs, indicating their potential for other tasks. We also show they can serve as drop-in replacements for segmentation.
These findings, along with additional ablation studies, suggest that ViTs can be used in medical image analysis, while at the same time potentially gaining from other properties of ViTs such as built-in explainability. The souree code reproduce our work is included in the supplementary material, and we will release it at the time of publication.
2 RELATED WORK
Transformers in vision problems. Following the success of transformers (Vaswani et al. 2017a) in natural language processing (NLP), attention- based models captured the interest of the vision community and inspired numerous improvements to CNNs (Hu et al., 201 Zhang et al. 2020 The
first work to show that vanilla transformers from NLP can be applied with minimal changes to large- scale computer vision tasks and compete with standard CNNs was from Dosovitskiy et al. (2020). which introduced the term vision transformer, or Vi1 Like their NLP counterparts, the original vision transformers required an enormous corpus of training data to perform well. To overcome this problem. Touvron et al. (202T) introduced DEITs, which proposed distillation token addition to the CLS token that allows transformers to learn more efficiently smaller datasets. As the number of applications of vision transformers and architectural variations have exploded (Khan et al. (2021)
While Dosovitskiy's 's use of the term VIT referred to a direct adaptation of the original transformer model from NLP Vaswami et al. 2017h), confusingly ViT has since been adopted as term to refer any transformer- based architecture for vision task. We also use VIT in the latter sense, for brevity.
2

---

Under review as a conference paper at ICLR 2022
provides good review), DeiT and the original VIT (Dosovitskiy et al. 2020) have become the standard benchmark architectures for vision transformers.
Vision transformers in medical imaging. Given the very recent appearance of vision transform- ers, their application in medical image analysis has been limited. Segmentation has seen the most applications of ViTs. Chen et al. (2021a), Chang et al. (202T) and Hatamizadeh et al. 202 I) in- dependently suggested different methods for replacing CNN encoders with transformers in U-Nets (Ronneberger et al. 2015). resulting in improved performance for several medical segmentation tasks. Vanilla transformers were not used in any of these works, however. In each, modifications to the standard vision transformer from Dosovitskiy et al. (2020) are proposed, either to adapt to the U-Net framework (Chen et al., 202 la) or to add components from CNNs (Chang et al. 2021). Other transformer-based adaptations of CNN based-architectures suggested for medical segmenta- tion tasks include Zhang et al. (002I) who combined pyramid networks with transformers, and Lopez Pinaya et al. (202 who used transformers in variational auto encoders (VAEs) for segmen- tation and anomaly detection in brain MR imaging.
Only handful of studies have tackled tasks other than segmentation, such 3-D image registration (Chen et al. 2021b) and detection (Duong et al.] 2021). The only work so far, to our knowledge, applying transformers to medical image classification is CNN/transfommer hybrid model proposed by Dai et al. (202T) applied to a small (344 images) private MRI dataset. Notably, none of these works consider pure, off-the-shelf vision transformers all propose custom architectures combining transformer/attention modules with components from convolutional feature extractors.
Improved initialization with transfer- and self-supervised learning. Medical imaging datasets are typically orders of magnitude smaller than natural image datasets due to cost, privacy concerns, and the rarity of certain diseases. A common strategy to learn good representations for smaller datasets transfer learning. For medical imaging, it is well-known that CNNs usually benefit from transfer learning with IMAGENET (Morid et al., 2020) despite the distinct differences between the domains. However, the question of whether ViT's benefit similarly from transfer learning to medical domains has yet to be explored.
While transfer learning is one option to initialize models with good features, another more recent approach is sulf-suuee pre-training. Recent advances in self-supervised learning have dramat- ically improved performance of label-free learning. State-of-the- methods such as DINO (Caron) al. 2021) and BYOL (Grill et al. 2020) have reached performance on par with supervised learning on IMAGENET and other standard benchmarks. While these -op-performing methods have not yet been proven for medical imaging, there has been some work using earlier self-supervision schemes on medical data. Azzzii et al. (2021) adopted SimCLR (Chen et al.) 2020). superviee con- trastive learning method, to pretrain CNNs. This yielded tfate-of-the-arr results for predictions on chest X-rays and skin lesions. Similarly, (Sowrirajan et al. (2021) employed MoCo (He et al.] (2020) pre-training on a target chest X-ray dataset, demonstrating again the power of this approach. While these works exhibit promising improvements thanks to self-supervision, they have all employed CNN-based encoders. It has yet to be shown how self-supervised learning combined with ViTs performs in medical imaging, and how this combination compares to its CNN counterparts.
3 METHODS
The main question we investigate is whether prototypical vision transformers can be used as a drop- in alternative to CNNs for medical diagnostic tasks. More concretely, we consider how each model type performs on various tasks in different domains, with different types of initialization, and across a range of model capacities. To that end, we conducted series of experiments to compare rep- resentative ViTs and CNNs for various medical image analysis tasks under the same conditions. Hyperparameters such as weight decay and augmentations were optimized for CNNs and used for both CNNs and ViTs with the exception of the initial learning rate, which was determined individ- ually for each model type using a grid search.
To keep our study tractable, selected representative CNN and ViT model types there too many architecture variations to consider each and every one. For CNNs, an obvious choice is the RESNET family (He al. 2016), as it is the most common and highly cited CNN backbone, and
3

---

Under review as a conference paper at ICLR 2022
recent works have shown that RESNETs competitive with more recent CNNs when modern training methods are applied (Bello et al. 2021). The choice for a representative vision transformer is less clear because the field is still developing. We considered several options including the original ViT (Dosovitskiy et al. 2020), SWIN transformers (Liu et al. 2021). Coa'T (Xu et al. (021). D. and focal transformers (Yang et al. 202T), but we selected the DEIT family [Touvron et al. (22TT) for the following reasons: (7) it is one of the earliest and most established vision transformers, with the most citations aside from the original ViT, (2) it is similar in spirit to a pure transformer, (3) was the first to show that vision transformers can compete on mid-sized datasets with short training times, (4) retains the interpretability properties of the original transformer.
As mentioned above, CNNs rely on initialization strategies to improve performance for smaller datasets. Accordingly, we consider three common initialization strategies: randomly initialized weights, transfer learning from IMAGENET, and self-supervised pretraining on the target dataset. Using these models and initialization strategies, we consider the following datasets and tasks:
Medical image classification. Five standard medical image classification datasets were chosen to be representative of a diverse set of target domains. These cover different imaging modalities, color distributions, dataset sizes, and tasks, with expert labels.
APTOS 2019 In this dataset, the task is classification of diabetic retinopathy images into
categories of disease severity (Kaggle 2019). APTOS 2019 contains 3,662 high-resolution retinal images.
CBIS-DDSM - A mammography dataset containing 10,239 images. The task is to detect
the presence of masses in the mammograms (Sawyer- et al. 2016 Lee et al., 2017
Clark et al.) 2013).
ISIC 2019 Here, the task is to classify 25,333 dermoscopic images among nine different diagnostic categories of skin lesions (Tschandl et al. 2018 Codella et al.) 2018 Combalia
ct al.] 2019).
CheXpert This dataset contains 224,316 chest X-rays with labels over 14 categories of diagnostic observations (Irvin al. 2019).
PatchCamelyon Sourced from the Camelyon 16 segmentation challenge (Bejnordi al., 2017), this dataset contains 327 327.680 6,80 patches of H&E stained WSIs of sentinel lymph node sections. The task is to classify each patch as cancerous or normal (Veeling et al. 2018).
Medical image segmentation. Although there has been some recent work applying vision trans- formers to medical image segmentation, all works thus far have proposed hybrid architectures. Here, evaluate the ability of vanilla ViTs to directly replace the encoder of standard segmentation framework designed for CNNs. In particular, we use DEEPLAB3 (Chen et al.) 2017) featuring a RESNET50 encoder. We merely replace the encoder of the segmentation model with DEIT-S and feed the outputs of the CLS token to the decoder of DEEPLAB3. As DEEPLAB3 was designed and optimized for RESNETS, CNNs have clear advantage in this setup. Nevertheless, these experiments can shed some light into the quality and versatility of ViT representations for segmentation. Note that, unlike the classification experiments, all models were initialized with IMAGENET pretrained weights. We consider the following medical image segmentation datasets:
ISIC 2018 This dataset consists of 2,694 dermoscopic images with their corresponding pixel-wise binary annotations. The task to segment skin lesions (Codella et al. 2019
Tschandl et al. (018).
CSAW-S - This dataset contains 338 images of mammographic screenings. The task is pixel-wise segmentation of tumor masses (Matsoukas et al. 2020).
Experimental setup. We selected the RESNET family (He et al.. 2016) as representative CNNs. and 2 (Touvron et al. 2021) as representative VíTs. Specifically, we test RESNET-18. RESNET-50, and RESNET-152 against DEIT-T, DEIT-S, and DEIT-B. These models were cho- sen because they are easily comparable in the number of parameters, memory requirements, and compute. They are also some of simplest, most commonly used, and versatile architectures of their respective type. Note that ViT's have additional parameter, patch size, that directly influences the memory and computational requirement. We use the default 16 x 16 patches to keep the number of parameters comparable the CNN models.
Note that we refer the DEIT architecture pretrained without the distillation token Touvron et al.
2021
4

---

Under review as a conference paper at ICLR 2022
Initialization
Model
APTOS2019, e DDSM, ROC-AUC1
ISIC2019, Recall 1 CheXpert, ROC AUC
Cumelyon, ROC-AUC
3,662
10,299
25,335
224,316
327,000
Kandom
RoNNar50
DelT-S
0.849 + 0.022
+ 0.017
0.06 0.005
0916 = 0.005
0.560 + 0.066
+
0.796
0.762
0.002
0.000
0.921 0.002
01943 + 0.008
ImageNet (supervised)
ResNar50
DefF-S
0.893 + 0.004
0.896 +: 0,005
0954 +
0.949 + 0.006
0.844 + 0.021
0.810 +
0,794 0.000
0.801 0.000
0.964 0.008
0.900 0.004
ImageNet (supervised) welf-spprrviied with DJNO
ResNert0
0.894 + 0.008
0,555 + 0.002
0.833 + 0.007
0.801 0.0000
0.962 0.004
DeTT-5
0.896 +:
0.966
0.853
0.801
0.976 0.002
Table 1: Comparison of vanilla CNNs vs. ViTs with different initialization strategies on medical
image classification tasks.
Using these models, we compare three commonly used initialization strategies:
1. Randomly initialized weights (Kaiming initialization from He et al. (2015),
2. Transfer learning using supervised IMAGENET pretrained weights,
3. Self-supervised pretraining on the target dataset, following IMAGENET initialization.
After initialization using the strategies above, the models are fine-tuned on the target data using the procedure described below. Early experiments indicated that initialization using self-supervision on the target dataset consistently outperforms supervsupoi on IMAGENET. We also determined that DINO performs better than other self-supervision strategies such as BYOL (Grill et al. 2020).
Training procedure. For supervised training, we use the ADAM optimizer (Kigmma & Ba 2014 We performed independent grid searches to find suitable learning rates, and found that 10 works best for both pretrained CNNs and ViTs, while 10-3 is best for random initialization. We used these as base learning rates for the optimizer along with default 1,000 warm-up iterations. When the validation metrics saturated, the learning rate dropped by a factor of 10 until it reached its final value of 10 6, For classification tasks, images were resized to 256 x 256 with the following augmentations applied: normalization; color jitter which consists of brightness, contrast, saturation, hue; horizontal flip: vertical flip; and random resized crops. For segmentation, all images were resized to 512 512 and we use the same augmentation methods as for classification except for CSAW-S, where elastic transforms are also employed. For each run, we select the checkpoint with highest validation performance. For self- ceff-supervision, pretraining starts with IMAGENET initializa- tion. then applies DINO (Caron et (222)) on the target data following the default settings except for three small changes determined to work better on medical data: (1) the base learning rate was set to 10-1, (2) the initial weight decay is set at 10-5 and increased to 10-1 using cosine schedule, and (3) we used an EMA of 0.99. The same settings were used for both CNNs and ViTs; both were described above.
pre-trained for 300 epochs using a batch size of 256, followed by fine-tuming
The classification datasets were divided into rrain/test/aliddtiio splits (80/1 10/0). with the exception of APTOS2019, which was divided 70/15/15 due to its small size. For the segmentation datasets, we split the training set into train/validation with a 90/10 ratio. For ISIC2018 and CSAW-S, we used the provided validation and test sets for model evaluation.
Unless otherwise specified, each experimental condition was repeated five times. We report the median and standard deviation of the appropriate metric for each dataset: Cohen Kappa, Recall, ROC-AUC, and IoU. For classification, we measure performance of the:
Evaluation.
initialized model representations, through a k--N evaluation protocol (Caron et al. 2021
final model after fine-tuning on the target dataset.
The k-NN evaluation protocol is technique measure the usefulness of learned representations for some target task; it is typically applied before fine-tuning. For CNNs, it works by freezing the pretrained encoder, passing test examples through the network, and performing a k-NN classification using the cosine similarity between embeddings of the test images and the k nearest training images (k 200 in our case). For ViTs, the principle is the same except the output of the CLS token is used instead of the CNN's penultimate layer.
DDETT trained with random initialization used AdamW Loshhhhlov & Hutter 2017) instead of Adam.
5

---

Under review as a conference paper at ICLR 2022
APTOS2019,
DDSM, ROC-AUCT
ISIC2019, Recall
CheXpert, ROC-AUC
Camelyon, ROC-AUCT
0.8
0
0.9
0.9
0.9
0.0
0.5
0.7
0.7
0.7
0.4
0.6
0.4
0.6
Ro MANAA Aie oand
0.2
0.2
0
o D
Dalis, AAN
0.0
Handom Imageltet DINO
0.4
Random ImageNul DIN
0.0
Random
Imagefee:
DIND
0.4
Random Imagelee DRO
0.4
Ransom Imagehed DIN
Ietlabzaiin Nathod
Inttal oation Mechod
In tisticaiinn Method
tratiakam. on Mathad
Inital nation Method
Figure 1: Performance comparison of RESNET5O and DEIT-S, two commonly used CNN-based and ViT-based architectures. The comparison covers several standard medical image classification datasets and different types of initialization including random init, IMAGENET pretraining, and self-supervision using DINO (Caron et al., 2021). Performance is measured after fine-tuning on the dataset, as well using k-NN evaluation without fine-tuning. We report the median over repetitions, error bars represent standard deviation. Numeric values appear in 3 (Appendix A)
4 EXPERIMENTAL RESULTS
Are randomly initialized vision transformers useful? Our first experiment compares the perfor- mance of ViT' against CNNs when initialized with random weights (He et al., 2015). The results in aad Figure inndicate that in this setting, CNNs outperform ViT's across the board. This is in line with previous observations in the natural image domain, where ViTs trained on smaller datasets are outperformed by similarly-size lack of inductive bias (Dosovitskiy et al. 2020). Note that the performance gap appears to shrink as the number of training examples n increases. However, since most medical imaging datasets are of modest size, the usefulness of randomly initialized ViT' appears to be limited.
CNNs, a trend that was attributed to the vision transformer's
Does pretraining transformers on IMAGENET work in the medical image domain? To deal with the smaller size of medical imaging datasets, CNNs often initialized with IMAGENET pre- trained weights. This typically results in better performance than random initialization (Morid et al., 2020), while recent works have questioned the usefulness of this approach for CNNs (Raghu al. 2019 Neyshabur et al. 2020). We investigate ViT's benefit from IMAGENET pre-training in the medical domain and which extent. To test this, we initialize all models with supervised IMA- GENET-pretraing weights and then fine-tune on the target data using the procedure described in Section 3 The results in Table D and Figure [ show that both CNNs and ViTs benefit from IMA- GENET initialization. However, ViT's appear to benefit more from transfer learning, as they make up for the gap observed using random initialization, performing on par with their CNN counterparts.
Do translormers benefit from self-supervision in the medical image domain? Recent self- supervised learning schemes such as DINO and BYOL achieve performance near that of supervised learning. When used for pretraining in combination with supervised fine-tuning, they can achieve new state-of-the-art (Caron al.) 2021 Grill et al.] 2020). While this phenomenon has been demonstrated for CNNs and ViTs on big data in the natural image domain, it is not clear whether self-supervised pretraining of VITs helps for medical imaging tasks. To test if ViTs benefit from self- supervision, we adopt the learning scheme of DINO, which can be readily applied to both CNNs and ViTs (Caron et al. 2021). The results reported in Table ! and Figure I show that both ViTs and CNNs perform better with self-supervised pretraining. ViTs appear to perform on par, or better than CNNs this setting, albeit by a small margin.
Are pretrained VIT features useful for medical imaging, even without fine-tuning? When data is particularly scarce, features from pre-trained networks can sometimes prove useful, even without fine-tuning on the target domain. The low dimensional embeddings from a good feature extractor can be used for tasks such as clustering or few-shot classification. We investigate whether different types of pretraining in ViTs yields useful representations. We measure this by applying the k-NN evaluation protocol described in Section 3 The embeddings from the penultimate layer of the CNN and the CLS token of the ViT are used to perform k-NN classification of test images, assigning labels based the most similar examples from the training set. We test whether out- of-domain IMAGENET pretraining or in-domain self-supervised DINO pretraining (Caron et all 2021) yields useful representations when no supervised fine-tuning has been done. The results

---

Under review as a conference paper at ICLR 2022
Model
ISIC2018, loU
CSAW-S, IoU
DEEPLAB3-RESNEIE
0.802 1 0.012
0.320 0.008
DEEPLAB3-DEIT-S
0.845 + 0.014
0.322 0.028
Table 2: Medical image segmentation with DEEPLAB3 comparing CNN vs. VIT encoders.
ISIC2018
CSAW-S
Original
Ground truth
Predictions
Original
Ground truth
Predictions
Figure 2: Medical image segmentation results comparing DEEPLAB3-REE - (blue), DEEPLAB3-DEITH (red). Ground truth mask appears in yellow. Note that the ViT segmentations tend to do a better job of segmenting distant regions.
appear in Figure and RESNET50. Pretraining with IMAGENET yields surprisingly useful features for both CNNs and ViTs, considering that the embeddings were learned on out-of-domain data. We observe even more impressive results with in-domain self-supervised DINO pretraining. VITs appear to benefit more from eeffssupervision than CNNs in fact, ViTs trained without any labels using DINO perform similar better than the same model trained from scratch with full supervision. The results indicate that representations from pretrained vision transformers, both in-domain and out-of domain, can serve solid starting point for transfer learning or k-shot methods.
and Table of the Appendix A] where we compare embeddings for DEIT-S and
Do ViTs learn meaningful representations for other tasks like segmentation? Segmentation is important medical image analysis task. While previous works have shown that custom attention- based segmentation models can outperform CNNs (see Section 2 here we ask: can we achieve comparable performance CNNs by merely replacing the encoder of CNN-based segmentation model with a vision transformer? This would demonstrate versatility of VIT representations for other tasks. We consider DEEPLAB3 (Chen al., 2017). mainstream segmentation model with RESNET50 encoder. We simply replace the RESNET5O with DErT-S, and train the model as prescribed in Section 3 Despite the model being designed for RESNET50 and DEIT-S having fewer parameters connected to the DEEPLAB3 head, we observe that ViT' perform on par or better than CNNs our segmentation tasks. The results Table 2 and Figure 2 clearly indicate that vision transformers are able to produce high quality embeddings for segmentation, even in this disadvantaged setting.
How does capacity of VIT models affect medical image classification? To understand the practi- cal implications of VIT model size for medical image classification, we conducted an ablation study comparing different model capacities of CNNs and ViTs. All models were initialized with IMA- GENET pretrained weights, and we followed the training settings outlined in Section 3 The results are presented Figure and Table Appendix B Both CNNs and ViTs seem benefit similarly from increased model capacity, within the margin of error. For most of the datasets, switching from a small ViT to bigger model results in minor improvements, with the exception of ISIC 2019 where model size appears to be important factor for optimal performance and APTOS2019 where DeiT-T performs better than its larger variants. Perhaps the tiny size of APTOS2019 results in diminishing returns for large models.
Other ablation studies. We also considered the effect of token size ViT performance. Table in Appendix Appendix([sshows [ishows that reducing patch size results in a slight boost in performance, but at the cost of increasing the memory footprint. Finally, medical images. In S Appendix D we compute the attention distance, the average distance in the image across which information is integrated in ViTs. We find that, in early layers, some heads attend to local regions and some to the entire image. Deeper into the network, attention becomes more global though more quickly for some datasets than for others.
investigate how ViTs attend to different regions of

---

Under review as a conference paper at ICLR 2022
APTOS2019,
DDSM, ROC-AUC1
ISIC2019, Recall
CheXpert, ROC-AUCT
Camelyon, ROC-AUCT
0.93
0.69
0.9
0.92
0.97
0.98
0.84
0.80
tets
0.81
0.98
0.9
0.96
0.8
0.0
P
0.95
0.8
Musa
0.79
0.55
0.94
0.93
0.92
0.76
0.78
0.93
0.22
512
25
512
25
60
(M)
(M)
paramelees
Figure 3: Impact of model capacity on performance for the RESNET and DEIT families on standard medical image classification datasets. Both model types seem to perform better with increasing capacity, roughly scaling similarly. Numeric results appear in Table Appendix B
5 DISCUSSION
Our investigation of prototypical CNNs and ViTs indicate that CNNs can readily be replaced with vision transformers in medical imaging tasks without sacrificing performance. The caveat being that employing some form of transfer learning is necessary. Our experiments corroborate previous findings in the natural image domain and provide new insights, which we discuss below.
Discussion and implications of findings. Unsurprisingly, we found that CNNs outperform ViTs when trained from scratch on medical image datasets, which corroborates previous findings in the natural image domain (Dosovitskiy al. (020). This trend appears consistently and fits with the "insufficient data to overcome the lack of inductive bias" argument. Thus, the usefulness of ran- domly initialized ViTs appears to be limited in the medical imaging domain.
When initialized with supervised IMAGENET pretrained weights, the gap between CNN and ViT performance disappears on medical tasks. The benefits of supervised IMAGENET pretraining of CNNs is well-known, but it was unexpected that ViTs would benefit so strongly. To the best of our knowledge, we are the first to confirm that supervised AMAGEN pretraining is so effective for ViTs in the medical domain. This suggests that further improvements could be gained via transfer learning from other domains more closely related to the task, as the case for CNNs (Azizpour et al., 20T6). The benefits of IMAGENET pretraining was not only observed for the fine-tuned classification tasks, but also self-supervised learning, k-nn classification and segmentation.
The best overall performance on medical images is obtained using ViTs with in-domain self- supervision, where small improvements over CNNs and other initialization methods were recorded. Our N-NN evaluation showed that ViT features learned this way even strong enough to outper- form supervised learning with random initialization. Interestingly, we did not observe as strong of an advantage for self-supervised ViTs over IMAGENET pretraining as was previously reported in the natural image domain, e.g. in Caron et al. (202I). We suspect this is due to the limited size of medical datasets, suggesting a tantalizing opportunity apply self-supervision on larger and easier to obtain unlabeled medical image datasets, where greater benefits may appear.
Our other ablation studies showed that VIT performance scales with capacity in a similar manner to CNNs; that ViT performance can be improved by reducing the patch size; and that ViTs attend to information across the whole image even in the earliest layers. Although individually unsurprising, it is worthwhile to confirm these findings for medical images. It should be noted however, that the memory demands of DeiT's increase quadratically with the image and patch size which might limit their application in large medical images. However, newer ViT architectures (e.g. Liu et al. (2021) and Yang et al. (202)) mitigate this issue while demonstrating increased predictive performance.
Interesting properties of ViT's. Our results indicate that switching from CNNs to ViTs can be done without compromising performance, but are there any other advantages to switching to ViTs? There a number of important differences between CNNs and ViTs. We briefly discuss some of these differences and why they may be interesting for medical image analysis.
Lack of inductive bias ViTs do away with convolutional layers. Our experiments indicate their implicit locality bias is only necessary when training from scratch on medical datasets. For larger datasets, evidence suggests removing this bias improves performance (Dosovitskiy et al., (020)).
Global + local features - ViTs, unlike CNNs, can combine information from distant and local regions of the image, even in early layers (see Appendix D). This information can be propagated
8

---

Under review as a conference paper at ICLR 2022
ISIC 2019
APTOS 2019
PatchCamelyon
CheXpert
CRIS-DDSM
Figure 4: Comparing saliency for RESNET5O (2nd row) and DEIT-S (3rd row) on medical classi- fication. Each column contains the original, Grad-CAM visualization visualisation for ResNet50 (Selvrrauu et al. 2017) and the top-50% attention map of the CLS token of DEIT-S.
efficiently to later layers due better utilization of residual connections that are unpolluted by pooling layers. This may prove beneficial for medical modalities which rely on local features. It may be necessary to use transfer learning benefit from local features, as they require huge amounts of data learn (Raghu et al. 2021).
Interpretability - transformers' self-attention mechanism provides, for free, new insight into how the model makes decisions. CNNs do not naturally lend themselves well to visualizing saliency. Popular CNN explainability methods such as class activation maps (Zhou et al.) 2016) and Grad- CAM Selvaraju et al. 2017) provide coarse visualizations because of pooled layers. Transformer tokens give a finer picture of attention, and the self-attention maps explicitly model interactions between every region in the image. In Figure 4 show examples from each dataset along with Grad-CAM visualizations of RESNET-50 and the top-50% self-attention of 16 x 16 DELT-S CLS token heads. The ViTs provide clear, localized picture of attention, e.g. attention the boundary of the skin lesion in ISIC, on hemorrhages and exudates in APTOS, the membrane of the lymphatic nodule in PatchCamelyon, the opacity the Chest X-ray, and dense region in CBIS-DDSM.
CONCLUSION
To answer the question posed in the title: vanilla transformers can reliably replace CNNs medical image classification with little effort. More precisely, ViTs reach the same level of performance as CNNs in an array of medical classification and segmentation tasks, but they require transfer learning to do so. But since IMAGENET pretraining is the standard approach for CNNs, this does not incur any additional cost in practice. The best overall performance on medical imaging tasks is achieved using in-domain self- self-supervised pretraining, where ViTs show small advantage over CNNs. As the data size grows, this advantage expected to grow as well, as observed by Caron et al. (202T) and our own experiments. Additionally, ViT's have number of properties that make them attractive: they scale similarly to CNNs (or better), their lack of inductive bias, global attention and skip connections may improve performance, and their self-attention mechanism provides clearer picture of saliency. From a practitioner's point of view, these benefits are compelling enough to explore the use of ViTs in the medical domain. Finally, modern CNNs have been studied extensively for 15 years, while the first pure vision transformer appeared one year ago the potential for ViT's to improve is considerable.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Shekoofeh Azizi, Basil Mustafa, Fiona Ryan, Zachary Beaver, Jan Freyberg, Jonathan Deaton, Aaron Loh, Alan Karthikesalingam, Simon Kornblith, Ting Chen, al. Big self-supervised models advance medical image classification. arXiv preprint vrXiv:2101.05224 2021.
Hossein Azizpour, Ali Sharif Razavian, Josephine Sullivan, Atsuto Maki, and Stefan Carlsson. Fac- tors of transferability for generic convnet representation. IEEE transactions pattern analysis and machine intelligence, 38(9): 790--182, 2016.
Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine Manson, Maschenka Balkenhol, al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases women with breast cancer. Jama, 318(22):2199 2210, 2017.
Irwan Bello, William Fedus, Xianzhi Du, Ekin D Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. arXiv preprint arXiv: 103.07579 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint a0Xiv:2104. 14294, 2021.
Yao Chang, Hu Menghan, Zhai Guangtao, and Zhang Xiao-Ping. Transclaw u-net: Claw u-net with transformers for medical image segmentation. arXiv preprint arXiv:2107.05188, 2021.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L. Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 2021a.
Junyu Chen, Yufan He, Eric C Frey, Ye Li, and Yong Du. Vit-v-net: Vision transformer for unsu- pervised volumetric medical image registration. arXiv preprint arXiv:2104.06468, 2021b.
Liang-Chies Khang-Chich Chen, George Papandreou. Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint vrXiv:1706.05550 2017.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Junghwan Cho, Kyewook Lee, Ellie Shin, Garry Choy, and Synho Do. How much data is needed to train a medical image deep learning system achieve necessary high accuracy? arXiv preprint orXiv:/511.0634 2015.
Christiansen, EL Epstein, Smedberg, M Åkerlund, Kevin Smith, and Epstein. Ultrasound im- age analysis using deep neural networks for discriminating between benign and malignant ovarian tumors: comparison with expert subjective assessment. Ulrraoonnd in Obstetrics & Gynecology, 57(1):155-163, 2021.
Kenneth Clark, Bruce Vendt, Kirk Smith, John Freymann, Justin Kirby, Paul Koppel, Stephen Moore, Stanley Phillips, David Maffitt, Michael Pringle, et al. The cancer imaging archive (Icia): maintaining and operating a public information repository. Journal of digital imaging, 26(6): 1045-1057, 2013.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, Emre Celebi, Stephen Dusza, David Gut- man, Brian Helba, Aadi Kalloo. Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint 1902v 2019.
10

---

Under review as a conference paper at ICLR 2022
Noel CF Codella, David Gutman, M Emre Celebi, Brian Helba, Michael A Marchetti, Stephen W Dusza, Aadi Kalloo, Konstantinos Liopyris, Nabin Mishra, Harald Kittler, et al. Skin lesion anal- ysis toward melanoma detection: A challenge at the 2017 international symposium on biomedical imaging (isbi), hosted by the international skin imaging collaboration (isic). In 2018 IEEE 15th international symposium on biomedical imaging (ISBI 2018), pp. 168-172. IEEE, 2018
Mare Combalia, Noel CF Codella, Veronica Rotemberg, Brian Helba, Veronica Vilaplana, Ofer Reiter, Cristina Carrera, Alicia Barreiro, Allan C Halpern Susana Puig, et. Ben20000: Dermo- scopic lesions in the wild. arXiv preprint
2019.
Yin Dai, Yifan Gao, and Fayu Liu. Transmed: Transformers advance multi-modal medical image classification. Diagnostics, 118):1384, 2021.
Jeffrey De Fauw Joseph R Ledsam, Bernardino Comera-Paredess Stanislav Nikolov, Nenad Toma- sev, Sam Blackwell, Harry Askham, Xavier Glorot, Brendan O' Donoghuue Daniel Visentin, et al. Clinically applicable deep learning for diagnosis and referral in retinal disease. Nature medicine, 2449):1342-1350, 2018.
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghan,, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition scale. In International Confer- ence on Learning Representations, 2020.
Linh T Duong, Nhi H Le, Toan B Tran, Vuong M Ngo, and Phuong T Nguyen. Detection of tuberculosis from chest x-ray images: Boosting the performance with vision transformer and transfer learning. Expert Systems with Applications, pp. 115519, 2021.
Jean-Base Man--Batienn Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint Ntt1:22006.07733, 2020.
Ali Hatamizadeh, Dong Yang, Holger Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. arXiv preprini arXiv:2103. 100.00 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Jie Hu, Li Shen, and Gang Sun. Squeere-and-exciiation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Jeremy Irvin, Pranay Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-llcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 590-597, 2019.
Kaggle. Aptos 2019 blindness detection, 2019.
URL. https://www. wwwwwwwwwww....c.....
aptos201 htw.119---nnnnesdeeeeeeeee
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. arXiv preprint vrXiv:2/01.00 169, 2021.
11

---

Under review as a conference paper at ICLR 2022
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Rebecca Sawyer Lee, Francisco Gimenez, Assaf Hoogi, Kanae Kawai Miyake, Mia Gorovoy, and Daniel L Rubin. A curated mammography data set for use in computer-aided detection and diagnosis research. Scientific data, 4:170177, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint MrXiv:2103.140500 2021.
Walter Hugo Lopez Pinaya, Petru-Daniel Tudosiu, Robert Gray, Geraint Rees, Parashkev Nachev, Sebastien Ourselin, and M Jorge Cardoso. Unsupervised brain anomaly detection and segmenta- tion with transformers. arXiv e-prints, pp. arXiv-2102, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled arXiv:1711.05101, 2017.
weight decay regularization. arXiv preprint
Christos Matsoukas, Albert Bou Hermandez, Yue Liu, Karin Dembrower, Gisele Miranda, Emir Konuk, Johan Fredin Haslum, Athanasios Zouzos, Peter Lindholm, Fredrik Strand, al. Adding seemingly uninformative labels helps in low data regimes. In International Conference on Ma- chine Learning. pp. 6775 6784. PMLR, 2020.
Mohammad Amin Morid, Alireza Borjali, and Guilherne Del Fiol. A scoping review of transfer learning research on medical image analysis using imagenet. Computers in biology and medicine, pp. 104115, 2020.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What being transferred in transfer learn- ing? arXiv preprint arXiv:2008.1 166877 1687,
2020.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. In Advances in Neural Information Processing Systems, pp. 3342-3352, 2019.
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? arXiv preprint arXiv:2108.088100 2021.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. arXiv preprint 1rXiv:2103./3444 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Rebecca Sawyer-Lee, Francisco Gimenez, Assaf Hoogi, and Daniel Rubin. Curated breast imag- ing subset of ddsm, 2016. URL https: https://wiki. wwncernnncaaam iaa.nerrnnngnaarccieeee.... net/x/
IZNXAQ
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Hari Sowrirajan, Jingbo Yang, Andrew Y. Ng, and Pranav Rajpurkar. Moco pretraining improves representation and transferability of chest x-ray models. In Mattias Heinrich, Qi Dou, Marleen de Bruijne, Jan Lellmann, Alexander Schläfer, and Floris Ernst (eds.), Proceedings of the Fourth Conference Medical Imaging with Deep Learning, volume 143 of Proceedings of Machine Learning Research, pp. 728-744. PMLR, 07 Jul 2021. URL https: //tp::/://..ooeednggg... http//l133///e /owr Jan21a .htll
m Atr.press/w143/sso
12

---

Under review as a conference paper at ICLR 2022
Nima Tajbakhsh, Jae Y Shin, Suryakanth R Gurudu, R Todd Hurst, Christopher B Kendall, Michael B Gotway, and Jianming Liang. Convolutional neural networks for medical image anal- ysis: Full training fine tuning? IEEE transactions medical imaging, 125))1299-1112, 2016.
Hugo Touvron, Matthieu Cord, Mathhis Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham dataset, large collection of multi-source dermatoscopic images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, t ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems. volume 30. Curran Asso- ciates, Inc., 2017a. URL lttpts:: htt.:::://proceedingss. new.iiss...u.ip...... pwtppppee//11///ieee.... $f5ee243547dese 91 fbd053c c4 c4a845aa Paper. .ddf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998 5998-6008, 2017b.
Bastiaan S Veeling, Jasper Linmans, Jim Winkens, Taco Cohen, and Max Welling. Rotation equiv- ariant cnns for digital pathology. In International Conference on Medical image computing and computer-assion asssisted intervention, pp. 210-218. Springer, 2018.
Nan Wu, Jason Phang, Jungkyu Park, Yiqiu Shen, Zhe Huang, Masha Zorin, Stanislaw Jastrzebski, Thibault Fevry, Joe Katsnelson, Eric Kim, et al. Deep neural networks improve radiologists performance breast cancer screening. IEEE transactions on medical imaging, 39(4):1188 1194, 2019.
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transform- ers. arXiv preprint MXXi::214400633 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Xiyang Dai, Bin Xiao, Lu Yuan, and Jianfeng Gao. Focal self-attention for local-global interactions in vision transformers. arXiv preprint arXiv:2107.00641, 2021.
Hang Zhang, Chongruo Wu, Zhongyue Zhang, Yi Zhu, Haibin Lin, Zhi Zhang, Yue Sun, Tong He, Jonas Mueller, R Manmatha, et al. Resnest: Split-attention networks. arXiv preprint arXiv:2004.08955, 2020.
Zhuangzhuang Zhang, Baozhou Sun, and Weixiong Zhang. Pyramid medical transformer for medi- cal image segmentation. arXiv preprins arXiv:2 104. 14702, 2021.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2921-2929, 2016.
13

---

Under review as a conference paper at ICLR 2022
APPENDIX
The Appendix contains several details and additional experimental results that did not fit within the page limit, organized by the order in which they are referenced the main article.
A K-NN EVALUATION OF DEIT-S AND RESNET50
The k-NN evaluation protocol is a technique to measure the usefulness of learned representations for some target task; it is typically applied before fine-tuning. For CNNs, it works by freezing the pretrained encoder, passing test examples through the network, and performing a k-NN classification using the cosine similarity between embeddings of the test images and the & nearest training images (k 200 in our case). For ViTs, the principle is the same except the output of the CLS token used instead of the CNN's penultimate layer.
We applied the k-NN evaluation protocol to test whether out-of-domain IMAGENET pretraining or in-domain self-supervised DINO pretraining yields useful representations when supervised fine- tuning has been done. The results appear in Table Pretraining with IMAGENET yields surprisingly useful features for both CNNs and ViTs, considering that the embeddings were learned out- of-domain data. We observe even more impressive results with in-domain self-supervised DINO pretraining. ViTs appear benefit more from self-supervision than CNNs in fact, ViTs trained with DINO perform similar or better than the same model trained from scratch with full supervision! The results indicate that representations from pretrained vision transformers, both in-domain and out-of domain, can serve as solid starting point for transfer learning or k-shot methods.
Initializzatinn
Model
APTOS2019,
DDSM, ROC-AUCT ISIC2019, Recall ? CheXpert, ROC-AUC
Canselyon, ROC-AUCT
ResNac50
8.000 + 0.000
0.531 + 0.063
+
0.490 0.029
0.306 0.679
Kandom
DelT-S
0,018 1: 0.086
0.596 4: 0.103
0,129 1: 0.026
0,501 0.013
0.443 : 0.073
ImageNet (supervised)
ResNar50
0.676 + 0.000
0874 + 0.000
0.248 + 0.000
0.682 0.000
01806 0.000
DeiT-S
0.000
ImageNet (supervised) ResNar50
+
+
+
0.907
self-uparvised with DINO
DeTT-
0.761 +: 0.000
0.898 :: 0,000
0.637 1 0.000
0.739 = 0.000
0.953 : 0.000
Table 3: The k-NN evaluation of CNNs and ViTs with different initialization methods including random initialization, IMAGENET pretraining, and self-supervision using DINO (Caron et al. 2021 on the target medical dataset. For each task, we use the metrics that are commonly used in the liter- ature, and for random initialization we report the median (# standard deviation) over 5 repetitions. The numbers in this table correspond Figure
B DEIT AND RESNET WITH DIFFERENT CAPACITIES
To understand the practical implications of VIT model size for medical image classification, we compare different model capacities of CNNs and ViTs.
Specifically, we test RESNET-18, R5s and RESNET-152 against DEIT-T, DEIT-S, and DEIT-B. These models were chosen because they are easily comparable in the number of param- eters, memory requirements, and compute. They are also some of simplest, most commonly used, and versatile architectures of their respective type. All models were initialized with IMAGENET pretrained weights, and trained following the settings outlined in Section The results appear in Table
Both CNNs and ViTs seem to benefit similarly from increased model capacity, within the margin of error. For most of the datasets, switching from small ViT to bigger model results in minor improvements, with the exception of ISIC 2019 where model size appears to be an important factor for optimal performance.
14

---

Under review as a conference paper at ICLR 2022
Model
K
DDSM, ROC-AUCT
ISIC2019, Recall 1
CheXpert, ROC-AUC
Camelyon ROC-AUCT AUC1
ReNNet88
0.893 + 0.003
0,950 +: 0.003
0.785 +: 0.015
0.793 + 0.001
0.959 0.012
ResNer50
0.883 + 0.004
0,954 + 0.055
0.800 + 0.088
0.801 = 0.001
0.960 0.044
ResNe1122
0500 0.004
0.965 0.005
Deft:T
0,001 + 0.005
0.946 + 0.006
0.810 + 0.013
0.789 + 0.001
0.961 0.003
DeiT-S
0.896 1: 0.005
0,919 1: 0.006
0.844 1: 0.021
0.794 1: 0.000
0.964 0.008
0.013
Table 4: Effect of model capacity on the performance after fine-tuning, with three different RESNET variants (RESNET18 8. RESNET5O and RESNET152) and DErT variants (DEIT-T, DEIT-S and DEIT-B using IMAGENET pretraining. For each task, we report the median (# standard devi- ation) over repetitions using the metrics that are commonly used in the literature. We can see that both CNNs and ViTs mostly benefit from increased model capacity. Further, model types with similar capacity perform approximately on par in most of the datasets, indicating that ViTs scale similarly to CNNs. The numbers in this table correspond to Figure
C EFFECT OF PATCH SIZE
ViTs have an additional parameter, patch size, that directly influences the memory and computational requirement. We compare the default 16 x 16 patches, used in this work, with 8 x 8 patches, as it has been noted in previous works that reducing the patch size leads increased performance (Caron et al. 2021
The results appear in 5 We observe that smaller patch sizes result in marginally better perfor- mance, though to lesser degree than has been observed in the natural image domain.
Patch size
APTOS2019,
DDSM, ROC-AUC ISIC2019, Recall 1 CheXpert, ROC-AUCT Camelyon. ROC-AUC1
8 x
0.89 0.006
8947 + 0.007
0.847 0.015
0,800 + 0.001
0.964 + 0.002
Table 5: Comparison of performance of DEIT-S using 8 x 8 and 16 x 16 pixel input patches standard medical image classification datasets. Performance is measured after fine-tuning on the dataset. For each task report the median (+ standard deviation) over repetitions using the metrics that are commonly used in the literature. Although, the model using 8 x 8 patch sizes median performance slightly better in the majority of datasets, we see no significant improvements, aside from CheXpert.
D MEAN ATTENDED DISTANCE OF VITS
We compute the attention distance, the average distance in the image across which information is integrated in ViTss As reported previously (e.g. (ooooviskkyy et al. (2020)), in early layers, some heads attend local regions and to the entire image. Deeper into the network, attention
becomes more global - though more quickly for some datasets than for others.
APTOS2019
Wean atturtann catarce (secc)))
DDSM
ISIC2019
CheXpert
Maan altuutar aleurce taeakl)
Camelyon
120
1220 1zo
220
120
300
100
100
120
eo
ag
10
60
663
1hed3
10
Hoin
10
a
3 10
Ncinok depth Bayert
Natwort depti Haperr
casth (layer)
Network depth Dayert
hatwork degth Japar)
Figure 5: Mean attention distance with respect to the attention head and the network depth. Each
point is calculated as the average over 512 test samples as the mean the element-wise wise multiplica- tion of each query token's 's attention and its distance from the other tokens
15


Reviews:
[{"review_id": "yolPwi4JzRP", "reviewer": "Reviewer_taLm", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n-> It asks and answers some important questions in the current research scenario like are vanilla ViTs really competitive with CNNs?; Is transfer learning and self supervision as effective for transformers as they are for CNNs? Etc.\n-> This paper finds out that self-supervised pre-training on domain-specific medical images followed by supervised fine-tuning of ViTs gives the best performance. This could be useful in future works, especially in applications like clinical deployment.\n\nConcerns:\n\n-> The authors choose one representation for both CNN and transformer model to study the difference. It is definitely true that not all the architecture variations could be considered for this study. However, I feel that the authors should have chosen U-Net instead of Resnet family as the representative CNN architecture for segmentation. While Resnet is a strong baseline for classification, almost all the medical segmentation methods are built with U-Net as the backbone (Litjens et al. “A survey on deep learning in medical image analysis” ).  While the authors give 4 reasons on why they chose DeiT as their transformer, there is no reason why resnet was chosen over U-Net.\n\n-> There seems to be a significant difference in performance between DeiT and resnet on the ISIC classification task when compared to the other classification tasks (from Table 1). It would be interesting if the authors could shed some light on this observation. \n\n-> It is shown in Isensee et al. (“nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”) that most of the performance improvement comes for medical imaging in choosing the perfect pre-processing, post-processing, training and optimization strategy of the network. I think it is important to see if nnU-Net outperforms DeiT in similar settings. If not, the authors should give a good reason why that comparison is not necessary.\n\n-> For the segmentation experiments, the authors consider replacing the encoder part of DeepLab with DeiT which makes the case that transformers make strong encoders for segmentation. A more thorough analysis would include experimenting with a transformer based decoder. Is there any specific reason why the authors left that out, if yes, it would be nice to see that information in the paper.\n\n-> In the discussion section “Interesting Properties of ViTs”, it would be nice to see some emphasis on proving it from the medical imaging perspective for the “Global+Local features” section.\n\nMinor Suggestions:\n\n-> If the page limit isn’t exceeded, It might be better to include a separate table on the findings and conclusions of this paper. For example, there could be a table with CNN and transformer as the columns and the rows corresponding to different settings (like random init, self-supervised etc.). That table will help improve the readability of the paper.\n\n-> I think a clear difference between the findings of this paper and findings of Azizi et al. (which was on self-supervision and transfer learning for CNNs would be good to tabulate.\n\nPost rebuttal:\n\nI thank the reviewers for their efforts put in the reply. I think the reviewers have addressed my main concerns well. I am also satisfied with most of their responses to other reviewers as well. I am increasing my score.\n", "summary_of_the_paper": "This paper primarily analyses if we should replace CNNs with transformers for medical imaging. Experiments are conducted on a number of medical image benchmark datasets with both CNNs and transformers for both classification and segmentation. It is concluded that ViTs outperform CNNs when pretrained using self-supervision.", "summary_of_the_review": "I think the paper has some issues regarding representation for both CNN and transformer models to study the difference. As the main contribution of the paper is empirical, this becomes a main issue. The empirical findings of paper like paper self-supervised pre-training on domain-specific medical images followed by supervised fine-tuning of ViTs gives the best performance could be useful to the research community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "kmSxGaW6mJ", "reviewer": "Reviewer_T9mB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1The manuscript is clearly written with well-organized sections to present the main contributions of this work. It sheds the initial light on the feasibility of replacing CNNs with the Vision Transformers for the medical imaging analysis tasks, specifically for the 2D medical imaging classification and segmentation.\n\n2 The experimental designs are reasonable and makes sense to me to explore the feasibility of replacing CNNs with the recent trends of Vision Transformers. Datasets and experimental details are clearly explained and sufficiently provided for the producibility. \n\n3 The discussion of including the self-supervised manner into the pre-training bears some value as the medical imaging is always lacking of annotations due to the requirement of professional knowledge and intensive laboring to obtain high-quality label. The self-supervised way can alleviate the lacking of annotation problem.\n\nWeaknesses:\n1 Medical imaging are often obtained in 3D volumes, not only limited to 2D images. So experiments should include the 3D volume data as well for the general community, rather than all on 2D images. And the lesion detection is another important task for the medical community, which has not been studied in this work. \n\n2 More analysis and comments are recommended on the performance trending of increasing the number of parameters for ViT (DeiT) in the Figure 3. I disagree with authors' viewpoint that \"Both CNNs and ViTs seem to benefit similarly from increased model capacity\". In the Figure 3, the DeiT-B models does not outperform DeiT-T in APTOS2019, and it does not outperform DeiT-S on APTOS2019, ISIC2019 and CheXpert (0.1% won't be significant). However, CNNs can give more almost consistent model improvements as the capacity goes up except on the ISIC2019.\n\n3 On the segmentation mask involved with cancer on CSAW-S, the segmentation results of DEEPLAB3-DEIT-S cannot be concluded as better than DEEPLAB3-RESNET50. The implication that ViTs outperform CNNs in this segmentation task cannot be validly drawn from an 0.2% difference with larger variance.\n\nQuestions:\n1 For the grid search of learning rate, is it done on the validation set?\n\nMinor problems:\n1 The n number for Camelyon dataset in Table 1 is not consistent with the descriptions in the text in Page 4.", "summary_of_the_paper": "This work studies the feasibility of replacing CNNs with ViTs for medical imaging analysis on the task of classification and segmentation. Careful experimental designs are conducted to study the comparison between ViTs and CNNs in the aforementioned two medical imaging tasks. Several research questions are raised and answered from the experiments and results. It sheds light on the feasibility of replacing CNNs by the recent vision trends of using ViTs for medical imaging task. It is shown that ViTs can perform better than their CNN counterparts if pretrained via a self-supervision manner. ", "summary_of_the_review": "Overall, it is a good starting point to explore the vision transformers in the medical imaging task. However, some preliminary conclusions are not well-supported. What's more, analysis on 3D imaging and lesion detection are also missing in this work, which are recommended to cover as they are very common for the medical community. So I would like to recommend a weak reject as of now.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "jCgIRPJ7olz", "reviewer": "Reviewer_LKZi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths:\n   - This paper performs tests in multiple datasets for two common types of tasks in the medical imaging domain: classification and segmentation.  It also evaluates CNNs and ViTs models that are very representative like ResNets and DEIT. \n- The paper is also well written and easy to follow.\n\n# Weaknesses:\n- I feel some simple analyses are missing: For instance, how long take to train each model? What are their inference time? Which one uses less GPU memory during training and inference? How do these models behave as we increase the size of the training dataset? \n- There are no fine-grained qualitative experiments: What kind of mistakes are more common in each model. For instance, CNNs in segmentation are known to not segment well very small structures, does it also happen with ViTs?\n- Use of more challenging datasets for segmentation: brain segmentation would be a good benchmark since it has more challenging structures and a lot more classes than the ones used.\n", "summary_of_the_paper": "This paper describes experiments comparing convolutional neural networks (CNNs) and visual transformers (ViTs) models in classification and segmentation problems in medical imaging. They also evaluate how the initialization of these models is essential to reach good performance in these problems which suffer from a lack of labeled data. In these experiments, they conclude that, ViTs can reach the same level of performance as CNNs, but they require transfer learning to do so, which can be easily accomplished with IMAGENET pertaining (like CNN's usually does) or self-supervised learning.", "summary_of_the_review": "I think this paper is not ready for publication in a top venue like ICLR. This paper does not present any technical novelty since it is an empirical paper. I agree that empirical papers can contribute a lot to the community as well, but it requires deep analysis of the results. The experiments are very coarse only comparing the average performance of two models in classification and segmentation of medical imaging. In my opinion, the findings of these experiments are somewhat expected and even the difference between the two models are too small to conclude something. Due to these reasons, I suggest rejecting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bPbKPEkMx1", "reviewer": "Reviewer_hd46", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. This paper is working on an important question on the capacities of ViT and CNN models for medical image tasks\n2. The experiments are clearly described\n3. The paper is well-written\n\nWeaknesses:\n1. This paper does not specify why they want to focus particularly on medical image tasks. As we all know, there are some major differences between medical images and natural images, in terms of the graphic patterns, scenarios, densities of labels, etc.. As a result, replacing CNNs with ViTs will have different impacts on medical and natural image tasks in different ways. We would like to see more information or rational on this.\n2.  In order to support the claim that the usages of GPUs of ResNet and BEIT models can be treated as counterparts, the authors should show statistical data about the numbers of parameters, volumes of computations, and the usages of GPUs of the two models.\n3. The comparisons were performed only between the ResNet family models and the BEIT family models. However, these evidences are not enough to draw a conclusion for the substitutability of CNNs and ViTs.\n4. The methodology of this paper lacks of novelty.\n", "summary_of_the_paper": "The paper under review is working on an important question: Could vision transformers (ViTs) outperform CNNs on medical imaging when performing tasks such as classification, detection, and segmentation? The authors employ ResNet family and DEIT family as the counterparts of CNNs and ViTs. Then they implement these methods for both classification and segmentation models, with and without the integrations of transfer learning and self-supervised learning. By comparing the results of these schemes on several medical image datasets, the authors draw a conclusion that vanilla transformers can reliably replace CNNs on medical image tasks with help of transfer learning and self-supervised learning.", "summary_of_the_review": "See \"Main Review\" section for more details.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "D1EZQCF4rT1", "reviewer": "Reviewer_aKib", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Unfortunately, I find that the paper oversells and underanalyzes. As I point out below, the authors mischaracterize a few aspects, mostly about the state and interests of medical imaging, including missing important parts of the field. The core of the paper -- a comparison between the two camps -- has very peculiar choices which I believe are inappropriate (not representative of the field or interest), leading to misleading conclusions. Given these points and the results presented, I believe the conclusions are inappropriate. For a paper whose core is experimentation and analysis of that experimentation (rather than, for example, providing a new method or theory), the details of those experiments and the analysis are especially important, (since this is where the proposed insight may come) and in this paper I believe they are not appropriate. I provide suggestions for what might be more appropriate below.\n\nIn addition, I think ICLR is not the most fitting venue for this sort of medical imaging experimentation paper, but that may well be debatable and I am happy to be convinced otherwise. However, I believe the authors would have a much bigger impact with such analyses at CVPR or MICCAI.\n\nOverall, I don't believe the paper is ready for publication and ICLR. However, I strongly encourage the authors to continue the work, but importantly:\n\n -  I encourage them to focus on one specific application, e.g. one of classification or segmentation. Trying to do both leads to the authors cross-motivating (e.g. making segmentation decisions motivated by conclusions in the classification domains) and confusing the insights\n\n - I encourage them to more faithfully represent the results. I believe this paper in its current form is not on whether ViTs should replace CNNs in medical imaging, but rather on an analysis of ResNet with ViT on 2D medical imaging classification and segmentation.\n  \n- Whether ViTs should replace CNNs is a factor of many many aspects, not just accuracy on a few choices. I would encourage the authors to focus more on the fact that they can do a timely and interesting analysis in the future, rather than suggest what the entire field should do based on this narrow analysis. \n\n- I would encourage the authors to focus more on *why* methods perform the way they do. For example, there is a claim that is consistently repeated that ViTs can build larger-space connections and that's why they have a benefit -- but this is simply not shown and theoretical. In practice, these large-space connections can be achieved by the dominant medical image architectures (like the U-Net) thanks to the multi-scale operations. While in theory there is a limit to that, I haven't seen a case in practice where it's relevant.\n\n\n\n\nDetails:\n - In the motivation, the authors conflate conclusions from classification with conclusions about segmentation, and apply that motivation to both domains. For example, in many medical imaging domains, segmentation does *not* requite many images as example. In fact, as the authors say the datasets are small -- very small, some segmentation datasets have ~20-30 images in them, and the algorithms achieve really good results (see Antonelli et al 2021, medical decathlon). In fact, these algorithms can do really well even with one example (because they see a lot of pixels/voxels) -- see for example one-shot segmentation in Chaitanya IPMI 2019 or Zhao, Balakrishan CVPR2019; several recent papers that measure the number of segmentation examples (or atlases) that are required for training to achieve great results (often under 10); and so on.  There is substantial literature that shows that zero or one (or very few) medical image with tons of augmentation can achieve remarkable results. The claim that a huge amount of data is needed depends on what is being done (e.g. classification needs more data), the variability of the data and task, and so on. This holds true for a lot of medical imaging tasks, like segmentation, registration, super-resolution, modality transfer, etc.\n\n - Overall in the motivation several such things are overstated and hand-waived. For example, the authors state that pre-training on imagenet is standard in medical images and outperform those trained from scratch, citing Raghu et al, 2019. The citation, however, concludes the opposite: there's a quote from their abstract \"A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.\". It's possible the authors meant something else than I am understand, or meant to cite a different paper? Nevertheless, I don't think the claim is representative of the medical imaging field.\n\n - I would overall encourage the authors to take out overly aggressive claims, like \"there is little to prevent ViTs from becoming the dominant architecture for medical images\" -- there are a lot of challenges to ViTs (training, data, etc), even if the accuracies were better (which they are, so far, not)\n\n -  As noted, the authors miss a substantial amount of literature. Please see many many papers in the medical imaging community (MICCAI/IPMI/MIDL/etc), or even CVPR, which tackle few-shot segmentation, data augmentation for segmentation, test-time segmentation, registration with limited data, etc. I give two example papers above, but there are even papers with zero-shot segmentation. These methods show that (1) a ton of data is not needed, contradicting a substantial amount of the motivation, (2) the predominant architectures are not ResNet based, but mostly Unet-like, among other lessons. Perhaps it's possible that the authors are not familiar with tehse conferences as most of their influence seems to come from the ML community (I could be wrong!) -- if so, I encourage them to look at these venues. There is a lot of relevant literature with substantial leslsons.\n\n - The authors broadly speak about medical imaging, but only show examples on 2D modalities. As can be seen in papers from the aforementioned venues, from MICCAI/other challenges, etc -- a substantial amount (perhaps the vast majority) or work focuses on 3D data (MRI, CT, etc). While it's perfectly fine to focus on 2D data, this should be clarified in the paper, and not broadly claim that the lessons here apply to all of medical imaging.\n\n - As the authors say, a lot of the work in medical image analysis is on segmentation and similar tasks (registration, super resolution, synthesis, etc), unlike CV where classification is by far the dominant task. Therefore, I encourage the authors to focus on those tasks. Importantly, the authors make choices that are inspired mostly by classification tasks, like using resnet in both classification and the backbone for segmentaiton. However, by far the most popular and fair architecture is the U-Net, -- not a particular variant of it, but unet like architectures. For example, the nnUNet (a unet-derivative that summarizes and applied architectural lessons of the unet) has won a multitude of challenges in the last year from the field (isenee 2020). Similarly, there are a slew of very popular segmentation tasks, like the BRATS challenge (menze 2014) or the Medical Decathelon (Antonelli 2021). Instead, the authors focus on mostly classification and on resnet backbones, and in the segmentation tasks they do not choose among these popular tasks. This seems inappropriate to me for a paper whose goal is to gain general insight of what is appropriate and successful for medical images -- neither the architectures nor the data chosen are representative of that.\n\n - In choosing among methods, the accuracy of a specific metric is not usually the only aspect leading to a choice. Most often, several metrics are used (e.g. see for segmentation the Brats challenge, which uses quite a few metrics like Dice, HD95, Jaccard, etc), as well as aspects like train time, training difficulty, inference time, etc. Unfortunately, none of these aspects are considered in the paper -- there is only one metric in the segmentation experiments (IoU, which is not often used in medical imaging, where we mostly use Dice for better or worse), and runtimes and such are not discussed. This latter discussion (runtimes) is crucial to the question posed in the paper, and more metrics compared to Dice are useful.\n\n - While of course the authros need to choose a training strategy, making very broad claims when there is a very specific training strategy seems misleading to me. How sensitive are the methods to these strategies? Surely as researchers use CNNs vs ViTs they will vary these strategies -- would conclusions carry? \n\n - I believe the main point of the paper is the final results, and I am not sure I understand the conclusions the authors empahsize. In most scenarios, ViTs are inferior -- such as in training from scratch (which the authors do acknowledge) but even with pre-training in classification in many of the datasets. Perhaps I am reading something wrong, but even with pre-training, I see many cases of both ResNet doing better and DeIT doing better. Overall, I am hard pressed to find a conclusion other than 'ResNet and DeIT perform comparably given pre-training'. \n\n- I would encourage the authors to study more about the differences between the two paradigms -- what can one learn? I know that the claim of 'transformers can build connections over large distances' is popular, but I have not seen this to be true in practice. I think if the authors wish to make this claim, they need to compare it to this ability by the U-Net or other hierarchical architectures which absolutely capture large-space dependencies.\n\n- Overall, the authors should re-evaluate their conclusion. As discussed in several aspects here, there is little evidence that ViTs outperform even in these experiments, and I really believe these are not the right expeirments (not the right architectures or data to work with, etc). \n\n\n", "summary_of_the_paper": "In this paper, the authors compare ResNets with ViTs on 2D medical (mostly classification) imaging datasets. I think this is a very interesting and timely topic since the current SOTA methods in medical imaging are dominated by CNN-based architectures, and the computer vision community (and NLP) are widely exploring transformers. I was quite excited to read this work, looking forward to good analysis and insight from the conclusion.", "summary_of_the_review": "The first couple of paragraph (before \"Detail\") of the review above offer a summary.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: I recommend a rejection of this paper.

My overall impression is that this is genuinely an interesting topic and this a good basis for a solid paper, however, as pointed by several reviewers, there are multiple unanswered questions due to a very large scope of this work. It might be that a format of a conference paper is not the most appropriate for this work. The authors should consider instead submitting to some of the leading journals on medical image analysis, e.g. IEEE Transactions on Medical Imaging or Medical Image Analysis. I expect this work, as it is mostly empirical, would be appreciated there and could in fact make a much bigger impact if published there.
Decision: Reject