Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Volumetric segmentation of medical images is still an open problem. This work proposes a new U-net type segmentation network that uses large receptive field depth-wise convolution operations to emulate vision transformers (ViTs). Additionally, patch-wise feature propagation and 1x1 convolutions are introduced as means of reducing the compute compared to ViTs. Experiments on multiple volumetric datasets and relevant CNN and ViT baseline models are reported with small to no improvements. 
strength_and_weaknesses: **Strengths:**

* The authors propose a volumetric segmentation method, that is evaluated on multiple relevant datasets. The choice of different architectural components in this work are explained clearly. 

* Ablation studies are reported and model hyperparameters are reported clearly.

* Experiments on multiple datasets and several relevant baselines are conducted. 

**Weaknesses:**

* **Motivation for the model**: This work aims to reintroduce CNNs to match the performance of Vision transformers (ViTs) for volumetric medical image segmentation. This is strange to me, as the chronology of how these models were developed is somehow warped. This work claims to revisit CNNs for volumetric segmentation, and improve them so that they can emulate ViTs -- which were already mimicking CNNs. This logic of arguments is circular to me. I found it very hard to subscribe to the primary motivation of this work. Further, and more fundamentally, the claim that ViTs have surpassed CNNs in volumetric segmentation is highly exaggerated (first sentence in abstract). Going by even the results reported in this work this claim does not hold up.

* **Experimental evaluation**: Results in Table 1 which have the main results are reported in single repeats. For the scale of performance improvements that is claimed, this could be just random, as the difference in Dice accuracy is very small (first or second decimal in most cases). Reading them as improvements based on single model runs is highly problematic. These small differences could simply be overcome with different initialisations. Also, the number of parameters between the current model and a CNN baseline like nn-Unet is double. The claims of superior performance under these conditions is not convincing.

* **Additional insights**: Unfortunately, there are no new insights gained from this work. How does the use of the depth-wise convolution with large kernels influence segmentation across different tasks. What is the reason for these improvements? What type of errors that the other ViT or CNN models make are alleviated with this new architecture? If ViTs were capable of modelling longe range interactions and CNNs introduced image-specific inductive biases, how does the new model bridge this gap? How can this be evaluated?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, however as mentioned above is not sufficiently well motivated. The reading of the results is problematic based on single runs and minor improvements. This could have implications on reproducibility. 

summary_of_the_review: This work proposes a volumetric segmentation method with a motivation of bridging the performance gap between ViTs and CNNs, by adding depth-wise convolutions with large kernels and couple of other tricks. The premise of the work that ViTs have surpassed CNNs, and CNNs need to emulate ViTs is flawed. Experimental evaluation shows small or no improvements, even with orders of magnitude more compute/parameters compared to other baseline methods.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors present an extension to convolutional networks (ConvNets) tailored to simulate the behavior of hierarchical transformers, aiming to provide an alternative to visional transformers (ViT) with a significantly reduced number of trainable parameters. The architectural variations in the proposed model focused on volumetric depth-wise convolutions with large kernel size for feature extraction that enables larger global receptive field. They then provide experimental evidence using 3 datasets that demonstrated an overall improvement in performance of 3D semantic segmentation in medical imaging.
strength_and_weaknesses: Strength:
The manuscript is well written and constructed, the figures and tables are coherent.
The provided context in the literature is adequate.
The experimental validation is well applied and provides evidence to support the authors conclusions.

Weaknesses:
While the authors provide an extensive literature review to provide the right context to their research, it would be beneficial if they could explicitly outline in paragraph 2.2 the differences of the proposed approach to the previously published models. Particularly, relating to the main inspiration of this paper, e.g. Liu et al. 2022, in the context of novelty should be better clarified.

clarity,_quality,_novelty_and_reproducibility: The study is well communicated with a high standard of clarity and reproducibility. 
The novelty of the paper may be questioned as many design choices for the proposed model have been examined separately or in part.
summary_of_the_review: In the current stage of scientific research in medical image segmentations, after the immersion of Visual transformers, this study provides a relevant point of view, aiming to simulate a convolutional neural network alternative with comparable performance and simplified architecture. The study is well applied and communicated in this manuscript and would be a good contribution to the field and ICLR.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper explores the necessity of incorporating transformer elements (self-attention, swin module) into 3D segmentation networks. It demonstrates SOTA performance on three challenge datasets (FLARE, FeTA, AMOS) with comparable model size and computational FLOPS. It is heavily inspired Liu's "A ConvNet for the 2020s" CVPR paper, but nonetheless makes a meaningful contribution for practical 3D applications that could be further enhanced by a slightly better ablation study. 
strength_and_weaknesses: **Strengths**
+ the paper is well written, easy to follow and addresses a relevant task in 3D/medical image analysis. 
+ the experimental section contains numerous direct comparisons with different versions of Swin/Vision-Transformers and demonstrates both faster convergence and higher accuracy of the proposed 3D UX-Net
+ the authors also evaluate pre-train/fine-tune scenarios and reach SOTA performance on AMOS22
+ within the method a new module: "depthwise convolutional scaling (DCS)" is introduced that performs slightly better than MLPs

the work gives (yet another) reason to consider claimed advantages of transformers with much care, since e.g. the increased receptive field of using 7x7x7 convolutions together with reasonable practical design choices alone can outperform those in a simpler Conv-only model

**Weaknesses**
- while closely following the narrative of ConvNeXt (which I liked) the authors perform there ablation study in the reverse direction: ie. in Tab 3 they start from the proposed model and **remove** individual parts, while ConvNeXt starts from a plain ResNet and **adds** the transformer-inspired changes. This is unfortunate, because for 3D segmentation there is indeed one prominent framework the nnUNet that uses a plain Conv-only model and has to date won nearly every medical segmentation challenge (incl. AMOS22). This is also evident from Tab. 1 where the nnUNet outperforms every single transformer approach and is only beaten by the proposed method (in light of these facts I would also strongly recommend to rephrase/remove the statement "To our best knowledge, this is the first 3D ConvNet architecture that competes favorably with transformers SOTA in volumetric segmentation tasks.)  A somewhat more elaborate discussion on the differences of the nnUNet (which e.g. also doesn't use BatchNorm) would be of important value. Importantly: the kernel-size ablation should in addition be extended by 3x3x3 and 5x5x5!
- I am also missing a more detailed discussion about runtimes for training and inference, since the raw FLOPs numbers are not always very favourable for UX-Net and depth-separable convolutions are additionally prone to lower efficiency (when comparing similar FLOP numbers). On the other hand the nnUNet is known to be fairly resource intensive for training, so any method that reduces this burden would be of great impact!
As a minor comment (since this was published after the ICLR submission deadline): one could include the nnUNet results for AMOS22 from https://arxiv.org/pdf/2208.10791.pdf in the final version 
clarity,_quality,_novelty_and_reproducibility: As mentioned above the paper is clear, of high quality and reproducible. The novelty is high and comes from a mixture of empirical and methodological findings. The impact and the interest of the community in performant alternatives to both nnUNet and SwinUNETR is arguably high (looking at the large number of medical segmentation challenges e.g. at MICCAI). 
summary_of_the_review: A recommend acceptance, subject to moderate improvements in terms of the comprehensiveness of the ablation study. Both the practical impact and the methodological insights (revisiting larger kernels and depth-separable convolutions in contrast to transformer modules) are relevant.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Inspired by the performance of vision transformers, the authors of this paper re-design convnet architecture for volumetric image analysis leading to improvements in accuracy, parameter count, and computational complexity over SOTA.
justification_for_why_not_higher_score: While two out of three reviewers strongly recommend acceptance, one reviewer still questions the significance of the work to the volumetric biomedical image analysis community.
justification_for_why_not_lower_score: There is good agreement that the method is novel and leads to tangible benefits in terms of accuracy, computational complexity, and parameter count.
Decision: Accept: poster