Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper under review is working on an important question: Could vision transformers (ViTs) outperform CNNs on medical imaging when performing tasks such as classification, detection, and segmentation? The authors employ ResNet family and DEIT family as the counterparts of CNNs and ViTs. Then they implement these methods for both classification and segmentation models, with and without the integrations of transfer learning and self-supervised learning. By comparing the results of these schemes on several medical image datasets, the authors draw a conclusion that vanilla transformers can reliably replace CNNs on medical image tasks with help of transfer learning and self-supervised learning.
main_review: Strengths:
1. This paper is working on an important question on the capacities of ViT and CNN models for medical image tasks
2. The experiments are clearly described
3. The paper is well-written

Weaknesses:
1. This paper does not specify why they want to focus particularly on medical image tasks. As we all know, there are some major differences between medical images and natural images, in terms of the graphic patterns, scenarios, densities of labels, etc.. As a result, replacing CNNs with ViTs will have different impacts on medical and natural image tasks in different ways. We would like to see more information or rational on this.
2.  In order to support the claim that the usages of GPUs of ResNet and BEIT models can be treated as counterparts, the authors should show statistical data about the numbers of parameters, volumes of computations, and the usages of GPUs of the two models.
3. The comparisons were performed only between the ResNet family models and the BEIT family models. However, these evidences are not enough to draw a conclusion for the substitutability of CNNs and ViTs.
4. The methodology of this paper lacks of novelty.

summary_of_the_review: See "Main Review" section for more details.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors compare ResNets with ViTs on 2D medical (mostly classification) imaging datasets. I think this is a very interesting and timely topic since the current SOTA methods in medical imaging are dominated by CNN-based architectures, and the computer vision community (and NLP) are widely exploring transformers. I was quite excited to read this work, looking forward to good analysis and insight from the conclusion.
main_review: Unfortunately, I find that the paper oversells and underanalyzes. As I point out below, the authors mischaracterize a few aspects, mostly about the state and interests of medical imaging, including missing important parts of the field. The core of the paper -- a comparison between the two camps -- has very peculiar choices which I believe are inappropriate (not representative of the field or interest), leading to misleading conclusions. Given these points and the results presented, I believe the conclusions are inappropriate. For a paper whose core is experimentation and analysis of that experimentation (rather than, for example, providing a new method or theory), the details of those experiments and the analysis are especially important, (since this is where the proposed insight may come) and in this paper I believe they are not appropriate. I provide suggestions for what might be more appropriate below.

In addition, I think ICLR is not the most fitting venue for this sort of medical imaging experimentation paper, but that may well be debatable and I am happy to be convinced otherwise. However, I believe the authors would have a much bigger impact with such analyses at CVPR or MICCAI.

Overall, I don't believe the paper is ready for publication and ICLR. However, I strongly encourage the authors to continue the work, but importantly:

 -  I encourage them to focus on one specific application, e.g. one of classification or segmentation. Trying to do both leads to the authors cross-motivating (e.g. making segmentation decisions motivated by conclusions in the classification domains) and confusing the insights

 - I encourage them to more faithfully represent the results. I believe this paper in its current form is not on whether ViTs should replace CNNs in medical imaging, but rather on an analysis of ResNet with ViT on 2D medical imaging classification and segmentation.
  
- Whether ViTs should replace CNNs is a factor of many many aspects, not just accuracy on a few choices. I would encourage the authors to focus more on the fact that they can do a timely and interesting analysis in the future, rather than suggest what the entire field should do based on this narrow analysis. 

- I would encourage the authors to focus more on *why* methods perform the way they do. For example, there is a claim that is consistently repeated that ViTs can build larger-space connections and that's why they have a benefit -- but this is simply not shown and theoretical. In practice, these large-space connections can be achieved by the dominant medical image architectures (like the U-Net) thanks to the multi-scale operations. While in theory there is a limit to that, I haven't seen a case in practice where it's relevant.




Details:
 - In the motivation, the authors conflate conclusions from classification with conclusions about segmentation, and apply that motivation to both domains. For example, in many medical imaging domains, segmentation does *not* requite many images as example. In fact, as the authors say the datasets are small -- very small, some segmentation datasets have ~20-30 images in them, and the algorithms achieve really good results (see Antonelli et al 2021, medical decathlon). In fact, these algorithms can do really well even with one example (because they see a lot of pixels/voxels) -- see for example one-shot segmentation in Chaitanya IPMI 2019 or Zhao, Balakrishan CVPR2019; several recent papers that measure the number of segmentation examples (or atlases) that are required for training to achieve great results (often under 10); and so on.  There is substantial literature that shows that zero or one (or very few) medical image with tons of augmentation can achieve remarkable results. The claim that a huge amount of data is needed depends on what is being done (e.g. classification needs more data), the variability of the data and task, and so on. This holds true for a lot of medical imaging tasks, like segmentation, registration, super-resolution, modality transfer, etc.

 - Overall in the motivation several such things are overstated and hand-waived. For example, the authors state that pre-training on imagenet is standard in medical images and outperform those trained from scratch, citing Raghu et al, 2019. The citation, however, concludes the opposite: there's a quote from their abstract "A performance evaluation on two large scale medical imaging tasks shows that surprisingly, transfer offers little benefit to performance, and simple, lightweight models can perform comparably to ImageNet architectures.". It's possible the authors meant something else than I am understand, or meant to cite a different paper? Nevertheless, I don't think the claim is representative of the medical imaging field.

 - I would overall encourage the authors to take out overly aggressive claims, like "there is little to prevent ViTs from becoming the dominant architecture for medical images" -- there are a lot of challenges to ViTs (training, data, etc), even if the accuracies were better (which they are, so far, not)

 -  As noted, the authors miss a substantial amount of literature. Please see many many papers in the medical imaging community (MICCAI/IPMI/MIDL/etc), or even CVPR, which tackle few-shot segmentation, data augmentation for segmentation, test-time segmentation, registration with limited data, etc. I give two example papers above, but there are even papers with zero-shot segmentation. These methods show that (1) a ton of data is not needed, contradicting a substantial amount of the motivation, (2) the predominant architectures are not ResNet based, but mostly Unet-like, among other lessons. Perhaps it's possible that the authors are not familiar with tehse conferences as most of their influence seems to come from the ML community (I could be wrong!) -- if so, I encourage them to look at these venues. There is a lot of relevant literature with substantial leslsons.

 - The authors broadly speak about medical imaging, but only show examples on 2D modalities. As can be seen in papers from the aforementioned venues, from MICCAI/other challenges, etc -- a substantial amount (perhaps the vast majority) or work focuses on 3D data (MRI, CT, etc). While it's perfectly fine to focus on 2D data, this should be clarified in the paper, and not broadly claim that the lessons here apply to all of medical imaging.

 - As the authors say, a lot of the work in medical image analysis is on segmentation and similar tasks (registration, super resolution, synthesis, etc), unlike CV where classification is by far the dominant task. Therefore, I encourage the authors to focus on those tasks. Importantly, the authors make choices that are inspired mostly by classification tasks, like using resnet in both classification and the backbone for segmentaiton. However, by far the most popular and fair architecture is the U-Net, -- not a particular variant of it, but unet like architectures. For example, the nnUNet (a unet-derivative that summarizes and applied architectural lessons of the unet) has won a multitude of challenges in the last year from the field (isenee 2020). Similarly, there are a slew of very popular segmentation tasks, like the BRATS challenge (menze 2014) or the Medical Decathelon (Antonelli 2021). Instead, the authors focus on mostly classification and on resnet backbones, and in the segmentation tasks they do not choose among these popular tasks. This seems inappropriate to me for a paper whose goal is to gain general insight of what is appropriate and successful for medical images -- neither the architectures nor the data chosen are representative of that.

 - In choosing among methods, the accuracy of a specific metric is not usually the only aspect leading to a choice. Most often, several metrics are used (e.g. see for segmentation the Brats challenge, which uses quite a few metrics like Dice, HD95, Jaccard, etc), as well as aspects like train time, training difficulty, inference time, etc. Unfortunately, none of these aspects are considered in the paper -- there is only one metric in the segmentation experiments (IoU, which is not often used in medical imaging, where we mostly use Dice for better or worse), and runtimes and such are not discussed. This latter discussion (runtimes) is crucial to the question posed in the paper, and more metrics compared to Dice are useful.

 - While of course the authros need to choose a training strategy, making very broad claims when there is a very specific training strategy seems misleading to me. How sensitive are the methods to these strategies? Surely as researchers use CNNs vs ViTs they will vary these strategies -- would conclusions carry? 

 - I believe the main point of the paper is the final results, and I am not sure I understand the conclusions the authors empahsize. In most scenarios, ViTs are inferior -- such as in training from scratch (which the authors do acknowledge) but even with pre-training in classification in many of the datasets. Perhaps I am reading something wrong, but even with pre-training, I see many cases of both ResNet doing better and DeIT doing better. Overall, I am hard pressed to find a conclusion other than 'ResNet and DeIT perform comparably given pre-training'. 

- I would encourage the authors to study more about the differences between the two paradigms -- what can one learn? I know that the claim of 'transformers can build connections over large distances' is popular, but I have not seen this to be true in practice. I think if the authors wish to make this claim, they need to compare it to this ability by the U-Net or other hierarchical architectures which absolutely capture large-space dependencies.

- Overall, the authors should re-evaluate their conclusion. As discussed in several aspects here, there is little evidence that ViTs outperform even in these experiments, and I really believe these are not the right expeirments (not the right architectures or data to work with, etc). 



summary_of_the_review: The first couple of paragraph (before "Detail") of the review above offer a summary.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies the feasibility of replacing CNNs with ViTs for medical imaging analysis on the task of classification and segmentation. Careful experimental designs are conducted to study the comparison between ViTs and CNNs in the aforementioned two medical imaging tasks. Several research questions are raised and answered from the experiments and results. It sheds light on the feasibility of replacing CNNs by the recent vision trends of using ViTs for medical imaging task. It is shown that ViTs can perform better than their CNN counterparts if pretrained via a self-supervision manner. 
main_review: Strengths:
1The manuscript is clearly written with well-organized sections to present the main contributions of this work. It sheds the initial light on the feasibility of replacing CNNs with the Vision Transformers for the medical imaging analysis tasks, specifically for the 2D medical imaging classification and segmentation.

2 The experimental designs are reasonable and makes sense to me to explore the feasibility of replacing CNNs with the recent trends of Vision Transformers. Datasets and experimental details are clearly explained and sufficiently provided for the producibility. 

3 The discussion of including the self-supervised manner into the pre-training bears some value as the medical imaging is always lacking of annotations due to the requirement of professional knowledge and intensive laboring to obtain high-quality label. The self-supervised way can alleviate the lacking of annotation problem.

Weaknesses:
1 Medical imaging are often obtained in 3D volumes, not only limited to 2D images. So experiments should include the 3D volume data as well for the general community, rather than all on 2D images. And the lesion detection is another important task for the medical community, which has not been studied in this work. 

2 More analysis and comments are recommended on the performance trending of increasing the number of parameters for ViT (DeiT) in the Figure 3. I disagree with authors' viewpoint that "Both CNNs and ViTs seem to benefit similarly from increased model capacity". In the Figure 3, the DeiT-B models does not outperform DeiT-T in APTOS2019, and it does not outperform DeiT-S on APTOS2019, ISIC2019 and CheXpert (0.1% won't be significant). However, CNNs can give more almost consistent model improvements as the capacity goes up except on the ISIC2019.

3 On the segmentation mask involved with cancer on CSAW-S, the segmentation results of DEEPLAB3-DEIT-S cannot be concluded as better than DEEPLAB3-RESNET50. The implication that ViTs outperform CNNs in this segmentation task cannot be validly drawn from an 0.2% difference with larger variance.

Questions:
1 For the grid search of learning rate, is it done on the validation set?

Minor problems:
1 The n number for Camelyon dataset in Table 1 is not consistent with the descriptions in the text in Page 4.
summary_of_the_review: Overall, it is a good starting point to explore the vision transformers in the medical imaging task. However, some preliminary conclusions are not well-supported. What's more, analysis on 3D imaging and lesion detection are also missing in this work, which are recommended to cover as they are very common for the medical community. So I would like to recommend a weak reject as of now.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper primarily analyses if we should replace CNNs with transformers for medical imaging. Experiments are conducted on a number of medical image benchmark datasets with both CNNs and transformers for both classification and segmentation. It is concluded that ViTs outperform CNNs when pretrained using self-supervision.
main_review: Strengths:

-> It asks and answers some important questions in the current research scenario like are vanilla ViTs really competitive with CNNs?; Is transfer learning and self supervision as effective for transformers as they are for CNNs? Etc.
-> This paper finds out that self-supervised pre-training on domain-specific medical images followed by supervised fine-tuning of ViTs gives the best performance. This could be useful in future works, especially in applications like clinical deployment.

Concerns:

-> The authors choose one representation for both CNN and transformer model to study the difference. It is definitely true that not all the architecture variations could be considered for this study. However, I feel that the authors should have chosen U-Net instead of Resnet family as the representative CNN architecture for segmentation. While Resnet is a strong baseline for classification, almost all the medical segmentation methods are built with U-Net as the backbone (Litjens et al. “A survey on deep learning in medical image analysis” ).  While the authors give 4 reasons on why they chose DeiT as their transformer, there is no reason why resnet was chosen over U-Net.

-> There seems to be a significant difference in performance between DeiT and resnet on the ISIC classification task when compared to the other classification tasks (from Table 1). It would be interesting if the authors could shed some light on this observation. 

-> It is shown in Isensee et al. (“nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation”) that most of the performance improvement comes for medical imaging in choosing the perfect pre-processing, post-processing, training and optimization strategy of the network. I think it is important to see if nnU-Net outperforms DeiT in similar settings. If not, the authors should give a good reason why that comparison is not necessary.

-> For the segmentation experiments, the authors consider replacing the encoder part of DeepLab with DeiT which makes the case that transformers make strong encoders for segmentation. A more thorough analysis would include experimenting with a transformer based decoder. Is there any specific reason why the authors left that out, if yes, it would be nice to see that information in the paper.

-> In the discussion section “Interesting Properties of ViTs”, it would be nice to see some emphasis on proving it from the medical imaging perspective for the “Global+Local features” section.

Minor Suggestions:

-> If the page limit isn’t exceeded, It might be better to include a separate table on the findings and conclusions of this paper. For example, there could be a table with CNN and transformer as the columns and the rows corresponding to different settings (like random init, self-supervised etc.). That table will help improve the readability of the paper.

-> I think a clear difference between the findings of this paper and findings of Azizi et al. (which was on self-supervision and transfer learning for CNNs would be good to tabulate.

Post rebuttal:

I thank the reviewers for their efforts put in the reply. I think the reviewers have addressed my main concerns well. I am also satisfied with most of their responses to other reviewers as well. I am increasing my score.

summary_of_the_review: I think the paper has some issues regarding representation for both CNN and transformer models to study the difference. As the main contribution of the paper is empirical, this becomes a main issue. The empirical findings of paper like paper self-supervised pre-training on domain-specific medical images followed by supervised fine-tuning of ViTs gives the best performance could be useful to the research community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper describes experiments comparing convolutional neural networks (CNNs) and visual transformers (ViTs) models in classification and segmentation problems in medical imaging. They also evaluate how the initialization of these models is essential to reach good performance in these problems which suffer from a lack of labeled data. In these experiments, they conclude that, ViTs can reach the same level of performance as CNNs, but they require transfer learning to do so, which can be easily accomplished with IMAGENET pertaining (like CNN's usually does) or self-supervised learning.
main_review: # Strengths:
   - This paper performs tests in multiple datasets for two common types of tasks in the medical imaging domain: classification and segmentation.  It also evaluates CNNs and ViTs models that are very representative like ResNets and DEIT. 
- The paper is also well written and easy to follow.

# Weaknesses:
- I feel some simple analyses are missing: For instance, how long take to train each model? What are their inference time? Which one uses less GPU memory during training and inference? How do these models behave as we increase the size of the training dataset? 
- There are no fine-grained qualitative experiments: What kind of mistakes are more common in each model. For instance, CNNs in segmentation are known to not segment well very small structures, does it also happen with ViTs?
- Use of more challenging datasets for segmentation: brain segmentation would be a good benchmark since it has more challenging structures and a lot more classes than the ones used.

summary_of_the_review: I think this paper is not ready for publication in a top venue like ICLR. This paper does not present any technical novelty since it is an empirical paper. I agree that empirical papers can contribute a lot to the community as well, but it requires deep analysis of the results. The experiments are very coarse only comparing the average performance of two models in classification and segmentation of medical imaging. In my opinion, the findings of these experiments are somewhat expected and even the difference between the two models are too small to conclude something. Due to these reasons, I suggest rejecting this paper.
review_title: null

Meta-review: I recommend a rejection of this paper.

My overall impression is that this is genuinely an interesting topic and this a good basis for a solid paper, however, as pointed by several reviewers, there are multiple unanswered questions due to a very large scope of this work. It might be that a format of a conference paper is not the most appropriate for this work. The authors should consider instead submitting to some of the leading journals on medical image analysis, e.g. IEEE Transactions on Medical Imaging or Medical Image Analysis. I expect this work, as it is mostly empirical, would be appreciated there and could in fact make a much bigger impact if published there.
Decision: Reject