Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In order to handle the overfitting issue in few-shot segmentation, this paper proposes to fine-tune a small part of backbone parameters recognized via the singular value decomposition. Precisely, the proposed Singular Value Fine-tuning (SVF) method suggests merely tuning the decomposed singular-value diagonal matrix for each convolutional layer. The experiments on few-shot segmentation among two datasets show the positive effects of fine-tuning the backbone using the SVF approach.
strengths_and_weaknesses: [Strengths]
+ The idea of recognizing the singular-value-related backbone parameters for fine-tuning is interesting.
+ The manuscript is well organized and has several experiments.

[Weaknesses] 
- The motivation is vague since the strong connection between the backbone fine-tuning and few-shot segmentation is unclear. The frozen pretrained-backbone could be treated as a feature extractor for a downstream task; hence the downstream task could focus on the mechanism design for employing the extracted features. For fine-tuning a backbone network as a goal, why not compare with the methods of meta-learning, adaptor, bias tuning, or domain adaption? Therefore, it is curious why it must fine-tune the backbone for tackling the few-shot segmentation. Specifically, the reviewer cannot find any components within the proposed Singular Value Fine-tuning specially designed for the “few-shot” scenario. As a result, it is suggested to deploy the proposed Singular Value Fine-tuning on various tasks, including classification and detection, to demonstrate the overfitting-proof advantages of the proposed SVF method.
- In line 78, the authors claimed that the work [10] is not suitable for few-shot vision tasks, yet there are no related experiments showing this issue.
- Since the proposed SVF is used to decompose the basic convolutional layers within a backbone, it may result in additional computational time while carrying out SVF. Therefore, it is better to include the experiments in discussing the required extra training time.
- The reference is not sufficient. For example, two related methods shown as follows should be compared.
[A] Kui Jia, Dacheng Tao, Shenghua Gao, Xiangmin Xu: Improving Training of Deep Neural Networks via Singular Value Bounding. CVPR 2017: 3994-4002.
[B] Hanie Sedghi, Vineet Gupta, Philip M. Long: The Singular Values of Convolutional Layers. ICLR 2019.

questions: The major concern is the vague motivation since the reviewer cannot find any connection of the proposed Singular Value Fine-tuning is specially designed for the “few-shot” scenario. In addition, the other concern of this paper is lacking compared with previous methods [A, B] related to manipulating the weight matrices of the convolutional layers. Please see [Weaknesses] for reference.  
limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Authors re-examine the idea of fine-tuning the backbone feature extractor during few-shot semantic segmentation, showing that overfitting can be avoided by limiting updates to a small set of parameters. Specifically, authors decompose convolution layers using SVD, and fine tune only the singular values S. Results indicate that this approach increases performance relative to using a frozen backbone, while fine-tuning other groups of parameters consistently decreases performance. 
strengths_and_weaknesses: STRENGTHS:

Motivation is clear, and approach is sensible, straightforward, and highly applicable. Results are convincing and a thorough analysis is provided. Paper is well organized (though with occasional typos and some awkward language, see below). 

WEAKNESSES:

While the paper convincingly demonstrates that SVF is effective, and acts as expected, it does not adequately explain why. Authors imply that fine-tuning in the singular value space is uniquely non-destructive, but it is not intuitively obvious that this should be the case. Indeed, in Fig.4 many values 
switch signs, indicating that nothing is stopping SVF from zeroing out large swaths of the output manifold in practice. Without knowing why the singular value space is so particularly effective, the contribution is limited, as constrained fine-tuning is already a widely known approach in the few-shot regime (see below) and is not novel in and of itself. 

Along these lines, while the analysis is broad and involved, the comparisons are somewhat apples-to-oranges. Rather than singular values being special, it could instead be that SVF simply allows the authors to fine-tune a smaller number of parameters than their comparative baselines. For starters, all fine-tuning comparisons except for BN involve a far larger number of parameters, and so the observed overfitting is not surprising, and even the BN baseline involves twice as many parameters (scale _and_ shift). Additionally, it could be that SVF allows for a smaller number of _effective_ updates compared to BN. For example, if matrix U is highly axis-aligned, changes to S will shrink or vanish in the subsequent BN layer. If U=I, then S should not update at all. 

Further analysis is required. 

Less importantly: while the related work is quite broad, there exist similar approaches in the broader few-shot literature also based on fine-tuning a highly constrained subset of introduced parameters. These may be worth mentioning – e.g. LEO (Meta-Learning with Latent Embedding Optimization, ICLR2019), CNAPS (Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes, NeurIPS2019), or possibly FiLM (FiLM: Visual Reasoning with a General Conditioning Layer, AAAI2018). 
questions: What is truly responsible for the success of SVF? 

I propose the following experiment to start: let R be a random rotation matrix, and set U=R’ and V=RW, where W is the original weight matrix for the given layer. Then attempt SVF fine-tuning. This will show whether SVF depends crucially on the singular value space, or simply on the number of effective updated parameters. Fine tuning only the BN scale terms might also be worth a try, for a true apples-to-apples comparison – it could be that the bias term is uniquely destructive. 

Alternatively, if authors could elaborate on what makes the singular value space so unreasonably amenable to fine-tuning, this may become clearer. Perhaps I am simply failing to understand the intended argument. 

SMALL COMMENTS AND TYPOS:

Pg3 line 100: while does not  while it does not

Pg4 line 103: one need  one needs

Pg4 line 132: parameters, also  parameters, and also

Pg5 line 177: all classes all classes  all classes

Pg5 lines 178-179: are used … are used  used … used

Pg7 line 214: shows that the  shows the

Pg7 lines 227-228: you seem to have a redundant sentence here (Finally… Finally)

Pg8 line 251: without destroy  without destroying

limitations: Discussion of limitations is fair. Societal impacts are not discussed, though do not extend beyond those of few-shot learning in general.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors have proposed a new training scheme for FSS frameworks by only adapting a few parameters of the ImageNet pre-trained backbone to the segmentation task. The core idea is to adjust the singular values of the pre-trained kernel weights of the backbone, bringing considerable improvements to representative FSS methods (PFENet and BAM). Extensive experiments have shown the effectiveness of the proposed method. 
strengths_and_weaknesses: Strength:

+ A good extension of the current training scheme for FSS frameworks.

+ Decent performance gain has been brought to representative FSS methods (PFENet and BAM).

+ Clear motivation and good presentation of the method and discussion. 

+ The overall submission is well-prepared with comprehensive appendix.

Weakness: 

- Unfair comparisons with the previously proposed methods (please see Questions section for details).

- Visualizations in Figure 5 and Figure 6 are confusing. 

If the above weaknesses could be well addressed, I would like to give a higher rating.




questions: This paper does present a good method for boosting existing few-shot segmentation methods whose backbone parameters are fixed, and considerable improvements have been achieved. However, my biggest concern is still about the training setting. 

The authors of submission 1788 follow BAM to remove all training images that contain novel classes, for the purposes of avoiding information leakage, but all previous methods in the community follow Shaban et al[1] to keep those images during the training phase by setting the labels for the novel classes as background, which explains why PFENet without dagger in Table 1 is much worse than the one with the dagger. 

More specifically, in section 3 of the paper [1], the authors of [1] wrote ''In this problem, unlike image classification, examples from L_test might appear in training images. This is handled naturally when an annotator unaware of some object class, labels it as background''. 

Therefore, the reviewer thinks that BAM actually has introduced an unfair comparison in their paper, and it would be better if the authors of submission 1788 could clearly present the results of Baseline + SVF, PFENet + SVF, and BAM + SVF in Table 1 and Table 2, by keeping the novel categories but setting them as the background during the base training phase, for a fair comparison with previous methods that adopt the same setting as described in [1].  

In Table 2 of the appendix, what causes the success on Fold-2 when the novel classes are removed from the training set? If the novel classes are included as [1] and set as the background, whether the proposed method will be negatively affected? 

Minor issues:
1. Are the PFENet and BAM without dagger shown in table 3 trained with novel classes whose labels are set to the background?

2. Visualizations in Figure 5 and Figure 6 are confusing and they could be improved by indicating what the target classes are. For example, in the last examples of Figure 5 and Figrue 6, the 1x1 weights focus on the person more than boat, but contrarily, the 3x3 weights are more curious about the boat, which contradicts the problem setting that only one target class exists in each evaluation episode. 


References
[1]  One-Shot Learning for Semantic Segmentation. BMVC 2017

 
limitations: As described in the Questions section, there might be some unfair comparisons with previous methods, and some critical aspects are not clear enough. The authors are encouraged to show additional results to support their claims.

[updated after rebuttal] My initial concerns regarding the comparison have been well-addressed in the revision. Thus my final rating is ACCEPT (increased from 4 -> 7).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: In this paper, the authors propose a novel SVF to changes the standard paradigm of freeze backbone in few-shot segmentation.The results show that only fine-tune the parameters of the singular value subspace (S) and freeze other subspaces can not only effectively avoid the overfitting problem, but also significantly improve the performance of FSS model. Specifically, the visualization results prove that some semantic cues with high weight in the pretrained weight are not conducive to downstream tasks, thus SVF adjusts the weights of different semantic cues by fine-tuning the parameters of subspace S. They further confirm the effectiveness of their proposed method by running experiments on real datasets and applications and comparing them to other topologies and methods. Moreover, they further confirm the effectiveness and superiority of SVF by comparing them to other representative methods on two common datasets.
strengths_and_weaknesses: - This paper is overall writing is clear and easy to follow. This is a really interesting paper. This paper not only analyzes the reason why the freezing backbone in FSS has become the traditional paradigm, but also proposes a novel fine-tuning method to try to break the traditional paradigm. The novel method SVF design is simple yet effective as proven by extensive experiments and analysis. Meanwhile, this paper analyzes the impact of hidden tricks in FSS on model performance to facilitate fair comparisons in future work. The core of SVF is a novel method of fine-tuning backbone, thus SVF has universality and can be easily used for all FSS models. As a result, this could serve as a new paradigm for few-shot semantic segmentation task. I believe that this paper will be a valuable contribution to the field, and I strongly recommend acceptance.

- The experimental results demonstrate the effectiveness of SVF, but will increase the training cost by a small amount compared to freeze backbone.
questions: - In section 3.3 and appendix, the author details two perspectives of SVF. The two perspectives are implemented differently. The paper does not analyze the impact of SVF from different perspectives on the FSS model.

- As an important part of the backbone, the bias has a huge impact on the model during the finetuning process. The authors lack an analysis of bias among different fine-tuning methods.

- In section 4.4 and apendix, the author only shows the change of the top-30 singular values of different convolutions, and the change results of all the singular values are missing.

limitations: This paper does not reflect any potential negative societal impact.
review_title: null

Meta-review: metareview: This paper presents a solution to overcome the overfitting problem in few-shot segmentation. Specifically, the proposed method decomposes the backbone parameters into three matrices via singular value decomposition (SVD) and fine-tunes only the singular values, while leaving the others frozen. This allows the model to adjust the feature representation in a new class while maintaining the semantic cues in the pre-trained backbone. All reviewers admit that this paper is well written, and the proposed method is applicable and novel. Furthermore, the authors provide great additional experiments and answers to the reviewers’ concerns. These made all reviewers positive for this paper. The AC agreed with the reviewers that the proposed method would make waves in the few-shot learning paradigm where the parameters of the pre-train model should be frozen. The AC recommends including the results described in the rebuttal for the final camera-ready version.

confidence: Certain
Decision: Accept