Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tackles filter pruning on deep convolutional networks for medical image segmentation. Training loss is augmented with a regularization term that penalizes distances between feature maps and forms clusters, and thus, pruning does not introduce extra training parameters or phases such as pre-training. Feature maps whose distances are below a threshold are then dropped, where the thresholds are adaptive based on layer and epoch. This strategy is used for training the state-of-the-art nn-UNet on 3 benchmark medical image segmentation datasets. The resulting pruned models maintain the segmentation performance of unpruned models and outperform other pruning methods in terms of segmentation performance and inference time.   

strengths_and_weaknesses: Strengths: Pruning deep neural networks for medical image segmentation is an important problem due to high dimensionality and long training times. The paper is well-written, method and experiment setup are clearly explained. 

Weaknesses: Following recent and similar works are not mentioned in related works/experiments, hindering the clarity of novelty. Particularly, Zhou et al. (2020), Chen et al. (2020) and Dinsdalea et al. (2022) focus on filter pruning for medical image segmentation; it is suggested that the authors compare/contrast their approach with these works. 

Zhou et al. “Evolutionary Compression of Deep Neural Networks for Biomedical Image Segmentation” (2020)

Chen et al. “α-UNet++: A Data-Driven Neural Network Architecture for Medical Image Segmentation” (2020)

Dinsdale et al. “STAMP: Simultaneous Training and Model Pruning for Low Data Regimes in Medical Image Segmentation” (2022)

He et al. “CAP: Context-Aware Pruning for Semantic Segmentation” (2021)

Ditschuneit et al. “AUTO-COMPRESSING SUBSET PRUNING FOR SEMANTIC IMAGE SEGMENTATION” (2022)

Chen et al. “MTP: MULTI-TASK PRUNING FOR EFFICIENT SEMANTIC SEGMENTATION NETWORKS” (2022)

Sabih et al. “DyFiP: Explainable AI-based Dynamic Filter Pruning of Convolutional Neural Networks” (2022)

questions: N/A
limitations: Limitations and potential impacts are discussed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed a regularization term in the loss for the filter pruning in medical image segmentation tasks. The regularization term minimizes the difference of feature maps from each convolutional layer, which enforces the similarity of all features against the feature map of the first channel. The filter pruning is conducted by removing the filter producing similar features. Three medical segmentation datasets (with 3D volumes) are employed for the experiments, and Superior results of the proposed method are reported in comparison to previous feature pruning methods. I have several concerns about the presented work, mainly regarding the experimental results. 
strengths_and_weaknesses: + The proposed pruning method can achieve better results than prior arts, proving the add-on regularization term effectively promotes feature clustering.
+ The manuscript is overall easy to follow.

Weakness:
- The motivation for including the regularization term is not clear to me. For instance, I am not sure why the similarity of features is always enforced between the feature of the first channel and the rest. I did not see how the statement "... makes those feature maps near the feature map in the first channel O_1 even closer. At the same time, those features that are dissimilar to O_1 become more similar to other feature maps..." is achieved, especially how are the other clusters formed as in the second part?  
- The proposed regularization term is rather straightforward, which is not technically innovative.
- Are the computed FLOPs computed in training or testing? It will be helpful to show the number of filters removed in the model and the exact number of parameters for the adjusted models.
questions: It will be helpful to see the authors' response to my abovementioned concerns.
limitations: Good for the presented application.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed a method for pruning filters in image segmentation networks by removing filters during training that are closely clustered. Unlike prior works, the approach is described as single-phase, meaning it prunes during normal training. To promote smaller networks, a term which promotes feature map clustering is added to the loss. The experiments uses nnUNet as a baseline network and 3 other recent network pruning methods as comparison. The results show better performance (Dice/HD95) with more pruned networks.
strengths_and_weaknesses: + Well motivated.

+ clearly written and easy to follow.

- I find significance is hard to asses, as the results and the novel contributions of the present work is no
t put and discussed in the context of prior work to a high enough degree.

- Having a list of contributions at the end of introduction, I feel is a good way to summarize reasons the a particular manuscript is relevant to read. In this case, however, it repeats too much for my taste. I would
 have preferred a stronger focus on novelty and what separates this work from prior works specifically.

- The section on Previous work hard to follow. I would have preferred more detail and more relevance given the context of the submitted work. As it is, it just reads as a listing of prior approaches, with too little detail to be informative.

- The section on Feature maps interpretation could benefit greatly from having the comparison to the non-pruned nnUnet in the paper and not in the appendix. As it is, it is not possible to assess the claims that the pruning makes the models easier to interpret.

- It is unclear to me if the clustering experiments (Figure 1) were done with multiple training runs. A single experiment where clustering is improved is perhaps not so interesting.

questions: - It seems like the proposed method does have a number of additional hyperparameters. See for instance lambda (l114), the layer-specific thresholds (l137), the patience hyperparameter (l149), the number of pruning steps (l198), etc. Considering that more hyperparameters are mentioned as a weakness of the prior works in contrast to the proposed, the authors should maybe clarify why and if they consider their approach superior in this respect?

limitations: - Aside from the above limitations, I think the work addresses limitations as aspected.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors proposed Sauron, a filter pruning methodology designed for U-Net-like medical image segmentation networks. In comparison to most filter pruning frameworks that consist of more than one distinct phase, Sauron applies filter pruning during optimization in a single phase, making it require fewer hyperparameters and design decisions. To achieve this goal, Sauron facilitates and promotes the formation of feature map clusters by optimizing a regularization term, and does not enforce the number of these clusters. In experiments, Sauron achieves comparable segmentation performance and largely reduced FLOPs on nnUNet.
strengths_and_weaknesses: Strengths:

1. The proposed clustering-based filter pruning method sounds interesting and reasonable. The regularization term $\delta_{opt}$ in Sauron makes those feature maps near the feature map in the first channel even closer. Meanwhile, based on the upper bound in lines 129-130, $\delta_{opt}$ forces those feature maps that are dissimilar to the feature map in the first channel to be more similar to other feature maps from the same cluster. 

2. Sauron performs well on 2D nnUNet, providing comparable performance on Rats and ACDC datasets with largely reduced FLOPs.

Weaknesses:

1. In Tables 1-3, it seems that the regularization term $\delta_{opt}$ does not have a significant impact on the segmentation performance. For example, Sauron $\delta_{opt}=0$ performs comparably with Sauron on Rats, ACDC, and KiTS (Kidney). In my opinion, I think these results cannot fully reflect the significance of the introduced regularization term (the major contribution of Sauron).

2. The results of 3D nnUNet on KiTS are not satisfactory, especially on the Tumor class. Actually, I recommend the authors to lay more emphasize on 3D UNets, which often require many more computational resources than 2D UNets. However, most of the experiments were made on 2D UNets, which are not deep and maintain high inference speed (not really in need of filter pruning).

3. The authors stated that Sauron is a single phase pruning method, which is more advantageous since it requires fewer hyperparameters and design decisions, including the number of epochs for training and fine-tuning, pruning iterations, etc. The authors should verify their statements with either experimental results or appropriate citations.

4. The pruning baselines, i.e., cSGD, FPGM, and Autopruner, are somewhat out-of-dated, which were published about three years ago. The authors should consider including more recent (published within two years) filter pruning approaches for fair comparisons.
questions: 1. Lines 29-30: The statement "models with a few filters can be easier to interpret than large models, which is crucial not only in clinical applications but also in research." needs citations.

2. Table 4 is never referred in the main text.

3. The provided github link does not exist.
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: The paper proposed a method for pruning filters in image segmentation networks by removing filters during training that are closely clustered. Unlike prior works, the approach is described as single-phase, meaning it prunes during normal training. To obtain smaller networks, a term which promotes feature map clustering is added to the loss. The experiments uses nnUNet as a baseline network and 3 other recent network pruning methods as comparison. The results show good performance (Dice/HD95) with more pruned networks on three medical segmentation datasets (with 3D volumes), and largely reduced FLOPs.

Deep neural networks pruning for medical image segmentation is an important problem due to high dimensionality and long training times.  The paper is well-written, method and experiment setup are clearly explained.

The contribution was nevertheless found to be limited among a very fast growing literature. 
The core contribution could be better explained.
The authors mostly proposed a general filter pruning without any specific optimization for U-Net series. It is thus necessary to compare Sauron against more existing filter pruning methods.

The contribution is fair, but that it remains overall a bit limited for NeurIPS, in the sense that it does not offer strong insights. Besides, the effectiveness of Sauron is also questionable on 3D U-Net, where the pruned model fails to provide satisfactory performance.

confidence: Certain
Decision: Reject