Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning
Authors: Yanpeng Sun
Qiang Chen
Xiangyu He
Jian Wang
Haocheng Feng
Junyu Han
Errui Ding
Jian Cheng
Zechao Li
Jingdong Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Freezing the pre-trained backbone has become a standard paradigm to avoid overfitting in few-shot segmentation. In this paper, we rethink the paradigm and explore a new regime: {\em fine-tuning a small part of parameters in the backbone}. We present a solution to overcome the overfitting problem, leading to better model generalization on learning novel classes. Our method decomposes backbone parameters into three successive matrices via the Singular Value Decomposition (SVD), then {\em only fine-tunes the singular values} and keeps others frozen. The above design allows the model to adjust feature representations on novel classes while maintaining semantic clues within the pre-trained backbone. We evaluate our {\em Singular Value Fine-tuning (SVF)} approach on various few-shot segmentation methods with different backbones. We achieve state-of-the-art results on both Pascal-5$^i$ and COCO-20$^i$ across 1-shot and 5-shot settings. Hopefully, this simple baseline will encourage researchers to rethink the role of backbone fine-tuning in few-shot settings.

Singular Value Fine-tuning: Few-shot Segmentation
requires Few-parameters Fine-tuning
Yanpeng Sun Qiang C Xiangyu He Jian Wang", Haocheng Feng?
Junyu Han?, Errui Ding", Jian Cheng", Zechao Li' Jingdong Wang* School of Computer Science and Engineering, Nanjing University of Science and Technology Baidd VIS NLPR, Institute of Automation, Chinese Academy of Sciences
Abstract
Freezing the pre-trained backbone has become standard paradigm to avoid over- fitting in few-shot segmentation. this paper, we rethink the paradigm and explore new regime: fine-tuning small part parumeters in the backbone. We present a solution to overcome the overfitting problem, leading to better model generalization on learning novel classes. Our method decomposes backbone pa- rameters into three successive matrices via the Singular Value Decomposition (SVD), then only fine-tunes the singular values and keeps others frozen. The above design allows the model to adjust feature representations on novel classes while maintaining semantic clues within the pre-trained backbone. We evaluate our Singular Value Fine-tuming (SVF) approach on various few-shot segmenta- tion methods with different backbones. We achieve state-of-the-arr results on both Pascal-5' and COCO-20* across 1-shot and 5-shot settings. Hopefully, this simple baseline will encourage researchers to rethink the role of backbone fine- tuning in few-shot settings. The source code and models will be available at https::/gitai htt.bttttu....../////// cww comm/syp2ysy/SV
Introduction
Benefiting from the large amounts of annotated data, deep learning has achieved noticeable im- provements in the field of semantic segmentation [4 17 34]. In contrast, their performances dramatically degrade when novel classes arrive or label data is insufficient. Thus, few-shot segmenta- tion (FSS) 22 49 was proposed address these challengs.
FSS, one needs segment novel samples (i.e., support images and
class objects in guery images given only a few densely-annotar support masks) Due to the extremely limited data in FSS, over-fitting has become a critical problem that needs to be carefully handled.
One feasible solution is restrict the model's 's learning capacity so that it can overfit the small dataset. Most recent works [43 47
30 33 follow this idea by freezing the pre-trained backbone. Then, different feature fusion methods and prototypes are introduced to enhance the generalization ability. Although this paradigm has achieved promising results, it is still suboptimal to directly adopt parameters pre-trained on image classification image segmentation. The semantic clues contained in the pre-trained backbone can be irrelated to objects shown in support images, bringing unexpected obstacles to segmenting novel class objects in FSS.
In this paper, we rethink the paradigm of freezing the pre-trained backbone and show that fine- tuning small part of parameters in the backbone is free from overfitting, leading to better model generalization in learning novel classes, Our method is illustrated in Figure [b). First, to find such
"Equal Contribution.
i Corresponding author.
'In this paper, we consider two settings of few-shot segmentation: 1-shot and 5-shot, which contain only one and five support images and masks, respectively.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Support Mask
Support Mask
W
S
SVD
Support Image
F
Support Image
Pre-trained Backbone
Segmentation Head
Pre-trained Backbone
Segmentation Head
Query
Query
Image
Image
(a) Previous paradigm
(b) SVF paradigm
Freezing weight
finetuning weight
Figure 1: Previous paradigm vs. SVE. (a) Previous paradigm introduces different segmentation heads based the frozen pre-trained backbone. (b) SVF uses SVD decompose the pre-trained parameters into three consecutive matrices, then only fine-tune the singular values and keep others frozen. Compared the previous paradigm, SVF shows that fine-tunie tin-ttning a small part of parameters in the backbone is invulnerable to over-fitting, leading to better model generalization in learning novel classes.
small part of parameters for fine-tuning, decompose pre-trained parameters into three successive matrices via the Singular Value Decomposition (SVD). Second, we then fine-tune the singular value matrices and keep others frozen. The above design, called Singular Value Fine-tuning (SVF), follows two principles: (i) maintaining rich semantic clues in the pre-trained backbone and (ii) adjusting feature map representations when learning to segment novel classes.
We evaluate our SVF on two few-shot segmentation benchmarks, Pascal-5' and COCO-20', Extensive experiments show that SVF is invulnerable to overfitting and works well with various FSS methods using different backbones. It is significantly better than the freezing backbone counterpart, leading to new state-of-the-at results both Pascal-5' and COCO-20'. Moreover, we provide quantitative and qualitative analyses on how singular values change during fine-tuning. Results show that SVF helps models focus highlight that proper backbone fine-tuning consistently outperforms backbone freezing on several leading methods. We hope our simple method will encourage researchers rethink the role of backbone fine-tuning for few-shot segmentation.
the objects to be segmented instead of the noisy background. Our experiments
2 Related Work
2.1
Few-shot Segmentation
The purpose of few-shot segmentation to segment the unseen class in query image with a few densely-annoaate samples. In this task, a semantically rich representation and a nice matching approach have a particularly large impact on the results. Therefore, mainstream methods [43]
40
focus obtaining excellent prototypes from support images, and obtaining accurate segmentation results by improving the quality of prototype features. CANet [45). PFENet (35) and
32
PANet [37 filters class irrelevant information by global average pooling to obtain foreground and background prototypes. ASGNet [16] pointed out that increasing the number prototypes can further improve the segmentation results. CyCTR 47 believes that pixel-level features in support image are important for segmentation tasks, and proposes to use pixel-level prototypes to predict query images. On the other hand, some methods focus designing better matching methods to improve segmentation performance. SG-One
uses cosine similarity to match prototype and query feature for segmentation results. CANet 45 proposes additive alignment module to iteratively refine the network output. HSNet 23 exploits neighborhood consensus to disambiguate semantics by analyzing patterns of local neighborhoods in matching tensors. In addition the above work, BAM [15] utilizes the segmentation results of the base class to guide the generation of unseen classes, and achieves SOTA results. However, the above methods are all based backbone freeze, and freezing backbone not only reduces the representational ability of the model, but also does not fit distribution to data better. Unlike previous work, in this paper we focus on the prospect fine-tuning backbone in FSS. Therefore, instead of proposing a new model, we adopt the classic PFENet [35] and BAM 15] as our baselines. Our SVF enables these methods further improve segmentation results.

---

180
8
Seat 1V.B
121
151
116 18 - Fally fretuo
H9 SCp
Figure 2: The mloU curve of PFENet 35 with different fune-tune strategies on Pascal-5' Fold-O. (a) the result of freezing backbone, (b) and (c) represent the result of fine-tuning the entire backbone and SVE, respectively. Compared with the direct fine-tuning (b), SVF not only avoids the overfitting problem, but also brings positive results.
2.2 Backbone Fine-tuning
Fine-tuning backbone in downstream tasks has become a common approach in deep learning. The
initial breakthroughs in vision tasks 10 13 18 were achieved by fine-tuning classification networks based ImageNet pre-trained weight, such as R-CNN 7 for detection and FCN (20] for segmentation. However, the direct application of fine-tuing the entire backbone in few-shot scenarios will lead to over-fitting of the model [30]. Therefore, fine-tuning part parameters of the backbone in few-shot learning may avoid model over-fitting. P-Transfer (30) utilizes NAS to search parameters of backbone that require fine-tune in few-shot classification tasks. However, this method is very complicated and cannot be directly applied to small sample segmentation. And some works [14] borrow the idea of prompt [36] 31 (39 in NLP to fine-tuning part parameters in the visual transformer [6]. The above methods are proposed in transformer-basee
model. but most few-shot segmentation models use CNN-based backbones. Applying prompt-based methods to various few-shot segmentation methods may need further adjustments. Different from the above methods,
SVF borrows the commonly used SVD [ in model compression and constructs 4 500 a novel part fine-tune method for few-shot segmentation task. In addition, some approaches [28] 26 [25] introduce highly constrained subset of parameters to fine-tuning. However, these methods are not applied on few-shot segmentation task.
3 From Freezing Backbone to Singular Value Fine-tuning
In this section, we start with the preliminaries on the few-shot segmentation (FSS) setting. Then, we revisit the overfitting problem in FSS when fine-tuning the backbone in Section 3.1 In Section 3.2 propose a novel Singular Value Fine-tuning (SVF) method for FSS instead of freezing the pre-trained backbone as proposed in previous methods. Section 3.3 provides discussion on the differences between SVF and other fine-tuning methods.
FSS Setup. Few-shot segmentation (FSS) aims segment novel class objects given only a few densely-annotated samples. In this task, datasets are split into the training set (Derain) with base classes (Ctrain) and the testing set (Dtest) with novel classes (Ctest), where Ctrain O Ctest = 0. Following previous works, we adopt episode training. Each episode consists of k support images and one query image to construct k-shot segmentation task (k FSS methods are trained with episodes to segment novel class objects in the query image given the knowledge of k support images and support masks.
in this paper). Then,
3.1 Revisiting Model Overfitting in FSS
As presented in Section D model overfitting is a critical problem in extremely limited data scenarios (I-shot and 5-shot), especially when the model has large amounts of learnable parameters. We validate this problem and design experiments with typical FSS method, PFENet [35). Figure shows that as model training moves on, fine-tuning backbone leads to better performance on the training dataset while it does not improve results the validation set. It is typical overfitting phenomenon. In contrast, freezing backbone can achieve steady improvements on the validation set during training. Therefore, existing methods in FSS turn to freezing the pre-traine pre-trained backbone avoid the overfitting problem.
Although this strategy has achieved promising results, it is clear that directly adopting an ImageNet pre-trained backbone to image segmentation can suboptimal. One need to extract the most related semantic clues within the backbone instead of involving too much noise coming from the irrelevant
3

---

categories learned from upstream tasks. In light of this, we rethink the paradigm of freezing the pre-trained backbone and try find new solution to the overfitting problem.
3.2 Singular Value Fine-tuning
According to the analysis above. fine-tuning all parameters in the backbone can be unsatisfactory. One feasible solution is to restrict the backbone's learning capacity so that it is suitable for the few-shot circumstance. Instead of freezing the whole pre-trained backbone as in previous works, we consider exploring new regime, which is only fine-tuning a small part of the parameter in the backbone.
However, it is nontrivial to find such a small part of parameters be fine-tuned in the backbone. Simply splitting the backbone's parameters into learnable and freezing ones results in negative results. Table []] and Ta- show the inferior performances no mar- ter splitted by layers or convolution types. We attribute this to the adjustment of the backbone, making the model biased towards base class objects shown in the training set, yet leads to worse model generalization in segmenting novel classes. Figure [3] also pro- vides evidences of the overfitting problem. Given that the backbone is pre-trained on a
Algorithm Pseudocode of SVF in Python style
Input: Conv2d with weight natrix N, Input feature Dutput: Outpot feature
Previous 3x3 Conv Y F. Cono2dd x, kerzel-(3,3))
SVF U. 8, V vvd(U) # decoupose veights by SVD
Falsa # freeze Conv_U V. v.requires.grad
U. u.requtrea_gra
False # freeze Conv_V
CoCon2ddV, x, kermal==3333) # a now 3x3 conv m.mul(S) reconstruct nex affine layer Y F. CoCovdd(U, Y, kermel-(1,1)) # new 1x1 conv
large-scale dataset with a classification task, it contains rich semantic clues but it is suboptimal to adopt for a segmentation task directly. We deliver two principles for finding a small part of fine-tuning parameters in the backbone: (i) maintaining rich semantic clues in the pre-trained backbone and (ii) adjusting feature map representations when learning to segment novel classes.
To fulfill the above goal, we resort to model compression methods in this paper. They are designed to approximate the original pre-trained model with fewer parameters, and also follow the above two principles. Among these methods, low-rank decomposition is a common technique to achieve model compression. first splits the model weights into multiple subspaces and then compresses each subspace by shrinking its rank. We follow this direction and decompose the backbone parameters into subspaces via the Singular Value Decomposition (SVD). However, we do not shrink subspaces' ranks since our target is to find a small part of parameters to be fine-tuned instead of model compression.
In detail, for convolution layer with C; input channels, C. output channels, and kernel size of K> x K in the pre-trained backbone, we first fold its weight tensor W E gC, xX. into a matrix W' E CCC then decompose the obtained matrix by applying SVD with full-rank in subspaces (rank min(C., C2 ))). Thus,
W USVT, where U U.RRR.RR , E R RxR and V E RRRCKK
The obtained pair of matrices VT and U construct two new convolution layers, and S is a diagonal matrix with singular values on the diagonal. Then back to convolutions, the results of the Equation I corresponds to three successive layers: a R x C, X K x K convvluutinn layer, a scaling layer, and followed by C, x R 1 x 1 convolution layer (Algorithm) [ further provides pseudo-code for SVF). We thus split each convolution layer the pre-trained backbone into three functionalities: (i) decouple the semantic clues into subspace with rank It, (ii) re-weight the semantic clues with singular values for the given task, and (iii) project the weighttt clues back to the original space. Based on the interpretation above, propose to fine-tune the scaling layer, which is Singular Value Fine-tuning (SVF). SVF does not crase the semantic clues contained in the pre-traine pre-trained backbone. but it re- weights the representations to help adjust the model for new segmentation tasks. As SVF restricts the learnable parameters to only singular values, which are extremely few (0.25%) compared with the parameters in the whole backbone, SVF is less vulnerable to overfitting and shows better generalization capacity in learning novel classes (shown in Table 3 Table 6
Figure 2 and Figure 3).
3.3 Discussion on Fine-tuning Methods
Fine-tuning pre-trained backbones is promising way to achieve state-of-the-ard results on down- stream vision tasks. Many fine-tuning methods have been introduced to transfer pre-trained back- bone's 's knowledge, such as full-model fine-tuning [10) 46), task-specific fine-tuning (freezing the

---

Table I: Performance on Pascal-5' 29 in terms of mloU for 1-shot and 5-shot segmentation. The best mean results are show in bold. indicates that images from training set containing the novel class on test set were removed.
I-shot
5-shot
Method
backbone
Fold-1
Fold-2
Fold-3
Fold-1
Fold
Fold-3
baseline
57.48
66.72
62.66
53.72
60.15
62.98
70.57
68.62
59.60
65.44
baseline' +VVF
SSVF
63.07
68.40
65.81
54.28
(22.94+2
68.52
72.15
69.08
63.59
PFENet
[S]
61.91
70.34
63.77
57.38
63.35
67.73
72.82
69.31
67.59
69.36
VGG16
PTENee +VVF
63.43
71.40
64.18
58.30
64.33
(40008)
69.11
73.67
69.13
67.30
69.80,
69.80
#0.442
BAM [6]
63.18
70.77
66.14
57.53
64.41
67.36
73.05
70.61
64.00
68.76
BAM +SVF
64.09
71.07
66.79
57.54
64.87
(+.0
67.75
74.11
70.99
63.57
69.11
(0.11,+0.00 35)
baseline'
65.60
70.28
64.12
60.27
65.07
69.89
74.16
67.87
65.73
69.41
haseline SVF
67.42
71.57
67.99
61.57
(422007)
70.37
75.06
71.06 08.08
69.16
(42200))
baseline
66.36
69.22
57.64
58.73
62.99
70.75
72.92
58.86
65.56
67.02
haseline SVF
66.88
70.84
62.33
60.63
65.17
(4.18)
71.49
74.04
59.38
67.43
68.09
PFFNei 35
66.61
72.55
65.33
60.91
66.35
70.93
75.32
69.60
68.96
71.20
PFENet +VFF
ResNet50
69.27
73.55
67.49
62.30
68.15
1.)80)
71.82
74.92
70.97
69.58
71.82
(10.02)
PFENet
67.06
71.61
55.21
59.46
63.34
72,11
73.67
61.61
67.50
68.72
PFENet
71.99
56.25
61.82
64.59
(+1)2))
72.09
73.99
63.58
70.03
(9.92
BAM' [5]
68.97
73.59
67.55
61.13
67.81
70.59
75.05
70.79
67.20
70.91
RAM +VV
69.38
74.51
68.80
63.09
68.95 (+1.14)
72.05
76.17
71.97
68.91
72.28
(+1.37)
72.05
57.55
70.72
74.21
RAM + SVF
68. 17
72.86
57.77
62.04
65.2
(+0.082)
72.30
74.43
65.16
69.43
70.33
(+1106))
backbone 9 41 residual adapter [12 38), and bias tuning 488 8]. We compare our SVF with
these methods. As presented in previous sections, methods like full-model fine-tuning may be suitable with extremely limited data, and task-specific fine-tuning can not provide adjustment to the representations in the backbone for downstream tasks. Moreover, methods like residual adapter and bias tuning need prior knowledge to model structure or weights. SVF does not have the above limitations in the few-shot segmentation task.
Recently, a new fine-tuning method named Vision Prompt Tuning (VPT) 14 has been proposed to fine-tune vision transformers. It introduces small number of trainable parameters in the input space while keeping the backbone frozen. From this perspective, our SVF also introduces a small number of trainable parameters but in the singular value space. In SVF, the learned singular value diagonal matrix can be formulated as a product of a frozen matrix frozen and a trainable matrix Sprainables which is S S frazen S Strainable- We give a detailed explanation of this perspective in the Appendix.
Experiments
We conduct experiments on Pascal-5' 229 and COCO-20' 2 to discuss fine-tuning approach in FSS. In this section, we first introduce the used representative method and implementation details. Then we discuss the impact of different fine-tune methods on the FSS model, and finally verify the effectiveness and versatility of the proposed fine-tune method.
4.1
Setting
Datasets.
Experiments are conducted on Pascal-5' [29 and COCO-20* 24 Following the previous
work 23 35 29]. we separate all classes both datasets into 4 folds. For each fold, Pascal-5' [2 has 15 classes used for training and 5 classes for test, COCO-20* (24) has 60 classes used for training and
20 classes for test. To verify the performance of the model, we randomly sample 1000 query-support pairs in each fold. Following the BAM [15), we remove images from training set containing novel classes of test to prevent potential information Ieakage. We give a detailed explanation of this setting about train sets in the Appendix.
Dataset Tricks: The previously methods annotated novel classes in the training set as background during training step. It become a common paradigm in few-shot segmentation. However, based on BAM [1]] we found a novel dataset trick to improve the performance of FSS models. simply removes images from the training set that contain the novel classes. For fair comparison with BAM, we use this trick our experiments. However, we know that previously methods does not use this trick. Therefore, we present the experimental results with and without dataset trick in Table I and more detailed fair comparison results in Appendix. Here, we hope that researchers can make fair comparison under the same setting.
5

---

Table 2: Performance on COCO-20' best mean results are show in bold. class on test set were removed.
24 in terms of mloU for I-shot and 5-shot segmentation. The indicates that images from training set containing the novel
I-shot
S-ahot
Method
Backbone
Fold-O
Fold-1
Fold-2
Fold-3
Mean
Fold-O
Fold-1
Fold-2
Fold-3
Mean
baseline
37.56
37.73
38,78
35.66
37,43
43.07
49.42
44.38
46,38
45,81
baseline's ++VF
39.32
39.64
38.63
35.45
38.26 ( 0.83)
46.48
50.72
45.79
45.63
_ (1.5))
PFENer 15
35.40
38.10
36,80
34,70
36.30
38.20
42,50
41.80
38.90
40.40
PFENet R51
VGG-16
41.03
44.22
43.74
38.90
41.97
48.66
48.26
45.49
51.02
48.36
PFENet +SVF
42.68
44.90
42.60
38.79
(2 0.27)
49.02
53.71
47.59
49.49; ( (1113)
RAM 15
38.96
47.04
46.41
41.57
43,50
47.02
52.62
48.59
49.
49.34
BAM +SVF
40.21
46.62
46.23
41.97
43.76
(-0 (6)
45.05
53.59
48.35
49.28
49.07 -0.ar)
baseline
38.91
46.07
42.67
39.71
41.84
50.35
56.78
49.61
50.96
51.93
baseline' +VVF
44.22
46.38
42.65
41.65
43.72
(_-
51.47
57.48
50.33
52.29
52.89
(+L
PFENet 3S]
44,93
50.32
44.68
44.26
46.05
52.29
59.34
51.50
53.53
54.17
ResNet-50
PFENer +SVF
46.88
50.86
47.69
46.64
48.02
52.72
58.14
52.52
54.15
54.38
(4.38_+0.21
BAM [15]]
43.41
50.59
47.49
43.42
46.23
29.26
54.20
51.63
49.55
51.16
BAM' +SV
46.87
53.80
48.43
44.78
48.47 (-2.24)
52.25
57.83
51.97
53.41
53,87 (+2.713
Table Performance on Pascal-5' (9 in terms of FB-IoU for 1-shot and 5-shot segmentation.
Table 4: Ablation study of BN on Pascal-5' un- der 1-shot setting. / /represents fine-tuning this feature space. The best mean results are show in bold.
FB-JoU (%)
Method
backbone
1-shox
5-shot
baseline'
77.11
Method
BN S
Mean
baseline' +SVF
78.86
82.66
65.07
PFENet
ResNet-50
77.35
82,30
63. (-.96))))
PFENet'+SVE BAM
baseline
64.20 (-0.8)
81.10
82.18
67.14
BAM'+SVE
80.13
83,17
Methods. To quickly verify the effectiveness of SVF, we propose a simple baseline method. Then, two representative methods are used to verify the generality of SVF. Next, we briefly introduce these three methods. Mcanwhile, for fair comparison, we unify the three methods into the same framework.
Baseline: We replace FEM module on PFENet 35 with ASPP module to get the baseline method. The baseline only sets main loss, without adding auxiliary loss.
PFENet (35]: As classic method in FSS. it proposed the Prior Guided Feature Enrichment Network, which has a huge impact on the subsequent FSS methods.
BAM [5]: As the state-of-the-ard method in FSS, it represents the most cutting-edge results in FSS.
Implementation details. In our experiments, we use VGG-16 3 and ResNet-50 as the backbone network and initialize it with ImageNet (27] pre-trained weights. Due to the particularity of BAM, we use the initialization weights provided by author, and SVF does not finetuning base branch in BAM. We use SGD optimizer with cosine Learning rate decay [21], the learning rate 0.015 and the random seed 321 when fine-tuning backbone. We keep original settings for the model without fine-tuning. All models are trained 200 epochs on Pascal-5' with batch size and trained 50 epochs on COCO-20' with batch size 8. Image is resized to 473 x 473 Pascal-5 and 641 x 641 on COCO-20'. Following 119 41 42 we adopt the mean intersection over union (mloU) and segground--ackggroun
IoU (FB-IoU as our evaluation metric. Since the middle-level features and high-level features are used in all FSS model, we set the SVF to fine-tuning the parameters of layers 2, 3, and 4 only, All model runs on four NVIDIA A100 GPUs.
4.2 Comparison with State-of-the-Art
In this section, the effectiveness of SVF is validated on three most representative methods. For fair comparison, we rerun all methods with unified framework. Then, we compare different methods with singular value fine-tuning and freeze backbone. The experimental results on Pascal-5' are in Table It can be seen that the performance of different methods has been significantly improved after SVR. When the backbone is VGG-16, the SOTA method BAM improves miou by 0.46 and 0.35 in 1-shot

---

18
(U4))
21 01
101 121 140 151 191
21
41 11
(3)
14)
Figure 3: The mloU curve of baseline with different finetune strategies on Pascal-5' Fold-O. (a) represent the results of baseline with freezing backbone, (b) represent directly fine-tuning layers 2, 3 and 4, (c) represent fine-tuning all 1 convolution layer in backbone, and (d) represent the proposed method SVF.
Table 5: Comparative experiment with fine-tuning different layer of backbone on Pascal-5'.
Table 6: Comparative experiment with fine-tuning different convolutional layer of backbone on Pascal-5'.
Method
layer
Mean
Method
layer
3 x 3
1 xx1 1
Mean
haseline"
baseline'
65.07
*fully fine-tune
2,2,3,4
60.90
-4.17)
2,
v
v
(1.15
2,3.4
61.15
+part fine-tune
2, 3,3
(1.86(-3.21)
+ part line-tune
3.4
60.58
4.49)
3,
61.62
+SVF
2.3.4
67.14, (2.07)
+SVF
2,
67.14
and 5-shot respectively after singular value fine-tuning, However. when the backbone is ResNet-50, SVF improves BAM by 1.14 and 1.37 mloU on 1-shot and 5-shot, respectively. It shows that SVF bring better performance in deeper backbone. Meanwhile, Table[]shove shows the effectiveness of SVF on more complex dataset COCO-20'. Exspecially, SVF improves the performance of BAM by 2.24 and 2.71 mIoU on 1-shot and 5-shot. Furthermore, Table shows the comparison results with FB-IoU on Pascal-5', Our SVF can also improve the performance of model. This experiment proves that SVF not only achieve state-of-the-art results, but also is a general method in FSS. In addition, the results (without ") in Table prove that the dataset trick can indeed improve the performance of FSS model. It also shows that whether not the dataset tricks is used does not affect the effectiveness of SVF. And we give more comparative results in the Appendix.
4.3 Ablation Study
To verify the effectiveness of SVF, we conduct series of ablation study in this section. We use the baseline method to conduct ablation study on Pascal-5' 1-shot setting with ResNet-50 as the backbone network. Furthermore, we give the ablation study about hyperparameter in the Appendix.
Batch Normalization (BN): In Table 4 we test the effect of BN on SVF. In the case of only fine- tuning the BN layer, the baseline will greatly reduce the performance. Next, we test SVF (fine-tune subspace S) on the baseline without fine-tuning BN. The results show that SVF achieves the best performance. Finally, we test the performance of the baseline method when fine-tuning subspace S and BN simultaneously. The results show that fine-tuning parameters of BN layer can cause performance of baseline to degrade. Therefore, we freeze the parameters of BN layer when using SVF.
Traditional fine-tune methods: In this part, we conduct experiments verify the impact of traditional fine-tuning methods on the FSS model. Traditional fine-tuning methods can be divided into fully fine-tune and part fine-tune. The fully fine-tune method means to fine-tuning all the parameters in the backbone. The part fine-tune methods means to fine-tuning part parameters in the backbone, which includes baser and convolution-based fine-tune methods. In table we conduct quantitative experiments with fully and layer-based fine-tune on baseline method. The results show that fully fine-tune brings negative results baseline method. Meanwhile, we find that the negative results of fully fine-tuning method are mitigated as the number of fine-tuning layers is reduced. However, these methods do not have a positive impact on the baseline. In table we conduct quantitative experiments on iavolution-based fine-tune tune methods. For fair compari- son, we only fine-tuning convolutions 2, 2, and 4 layers. The results show that only fine-tuning 3 x
7

---

Table 7: Ablation study of SVF fine- tuning different subspace on Pascal-S'.
Table 8: Ablation study of SVF fine-tuning different
layer on Pascal-5'. The best results are show in bold.
Method
S
Mean
61.09
Method
layer
Mean
/
67.14
4
66.21
60.88
3.4
67.20
baseline
61.37
baseline" x + +VF
60.42
2, 4
67.14
60.02
1, 2, 4
67.12
61.24
Table 9: Ablation study of different ways of chang- ing semantic cues in weights on Pascal-5' I-shot.
Table 10: Compare with parameter-efficient tuning methods on Pascal-5' 1-shot.
Mothod
Expression of weight Fine-tune param
Mcan
Method
fine-tune method
Mean
W S'W
65.07
S'
63.52
freeze backbone
65.07
WS'
S'
64.62
fully fine-tune
60.90
baseline
RS RS'R" w USV
S' 43.36
baseline
SVF 67.14
URSR F Yr URSR 2 YT
8 67.14
29.31
Adapter
20.71
30.26
bias tuning
62.93
convolution or 1 convolution can further improve the performance of layer-based fine-tune methods. It show that traditional fine-tune methods cannot bring positive results. However, SVF brings positive results to the baseline method. The success of SVF proves that traditional fine-tune method destroys the rich semantic clues in the pre-trained backbone. In Figure we compare the mloU curves of the training and test sets in different fine-tune methods. The results show that part fine-tune method also produces the over-fitting problem. It proves that disrupting the rich semantic cues in pre-trained backbone will lead to model over-fitting, reducing model generalization. However, SVF solves the over-fitting problem without destroying semantic clues in pre-trained weight. And, it brings a new perspective for fine-tuning backbone.
Fine-tuning which subspace: To verify the influence of different sub-spaces on SVF, we conduct experiments on the subspace after SVD decomposition. The results are shown in Table 7 We find that only fine-tuning S subspace brings positive results. Either fine-tuning the U or V subspace returns negative results. It shows that U and V contain rich semantic information in pre-trained weight after SVF. In other words, directly changing the feature distribution of the U or V subspace reduces the generalization ability of the model. To verify the above point, we test the performance of fine-tuning different subspace combinations. The results confirm that changing the distribution of U or spaces brings negative results. The subspace S represents the weight distribution of different semantic cues. Therefore, fine-tuning the subspace S does not change the semantic cues of pre-trained weights. Meanwhile, adjusting the weights of different semantic cues enables model to better perform downstream tasks.
Since the FSS model directly uses feature maps of layers 2, 3, and 4, we initially fine-tuning the subspace S of layers 2, 3 and 4. However, this setting is unreasonable. To verify which layer S have a greater impact on baseline, we conducted experiments on SVF under different layer combinations. The results are shown in Table 8 It can be seen that fine-tuning layers and achieves the best performance, while only fine-tuning the layer 4 achieves the lowest performance. It shows that semantic clues in layer3 are the most important for FSS. Next we discuss the reasons why SVF can achieve better performance by visualizing semantic cues in layer3.
Fine-tuning which layers:
Compare with other parameter-efficient tuning methods: Unlike SVF, the purpose of parameter- efficient tuning methods to obtain performance similar with fully fine-tune by fine-tuning a small number of parameters. To verify the superiority of SVF over parameter-efficien tuning methods in FSS, we compare SVF with adapter [12] and bias tuning on Pascal-5' with the 1-shot setting. The details for adapter and bias tuning are given below:
Adapter: Adapter is proposed in transformer-basee models. When applying it into CNN-
based backbone (ResNet), we make simple adjustments. We follow [12] to build the adapter structures and add them after the stages in the ResNet.
Bias Tuning: In the ResNet backbone, the convolution layers do not contain bias term. The bias terms that can be used for tuning is the ones BN layers. We fine-tune the bias terms in all BN layers in this method.
8

---

a
(b)
Figure 4: The visualization of segmentation cues with the largest variation singular values from
the last 3 x 3 convolution in layer 3. (a) represents segmentation clues of subspace U with the largest singular value reduction, (b) represents segmentation clues of subspace U with the largest singular value growth.
(8)
(b)
Figure 5: The visualization of segmentation cues with the largest variation in singular values from the last 1 x convolution in layer 3. (a) represents segmentation clues of subspace U with the largest singular value reduction, (b) represents segmentation clues of subspace U with the largest singular value growth.
The experimental results are given in the table T0 shows that SVF outperform Adapter and Bias Tuning by large margins. Moreover, we find that the introduction of Adapter will directly lead to over-fitting while Bias Tuning reduces performance of the baseline model.
4.4 Discussion on Why SVF Works
The larger singular value in subspace S, the more important semantic cues in subspace U and V. We first focus on the changes of singular values during fine-tuning based on the initial distribution of S. In Figure 6 we visualize the variation of Top-30 singular values in pre-trained weights. It can be seen that the singular values of either 1 x 1 or 3 x convolution change dramatically after fine-tuning, Next, we visualize the semantic cues of subspace with the largest variation in singular values. The results are shown in Figure Figure We only visualize the semantic cues where the singular value grows and decreases the most for better view. Notice that semantic cues of decreasing singular values tend to focus on background regions. The semantic cues of increasing singular values always focus on foreground regions. The Beckground-focussee pre-trained backbone will damage the performance of FSS model. Since the original distribution of semantic cues in pre-trained backbone not suitable for downstream tasks, SVF brings positive results to FSS model by increasing the weight of foreground cues and reducing the weight of background cues. It is also important to keep the semantic cues unchanged during fine-tume ain-tuning. Overall, dynamically adjusting the weight of each cue without changing the semantic representation the key to the success of SVE.
Figure Statistics chart about the changes of initial Top-30 largest singular values of the last 1 x 1 and 3 convolution layer in layer3 after fine-tuning.
semantic cues in
To verify that changes in the singular value space do not affect the semantic information in pre- trained weight, we conduct an interesting experiment intentionally changing semantic cues in weights. In Table 9 We compare different approaches, including introducing small number of

---

training parameters S", and introducing a random rotation matrix R. It can be seen that changing the semantic cues the weights negatively affects the FSS model (with or without fine-tuning a small number of parameters). Experimental results demonstrate that fine-tuning the singular value space is non-destructive (without destroy semantic cues). We give a detailed analysis about why SVF work in the Appendix.
4.5 Broader Impact
In this paper, we prove that freeze backbone is not the only paradigm in few-shot segmentation, fine-tune backbone is feasible. Meanwhile, we explore a new mechanism redistribute the weights of different semantic cues without changing the semantic cues. As a new perspective of few-shot segmentation, it exposes the influence of pre-trained backbone on few-shot segmentation. Moreover, this mechanism not only works on few-shot, but also may be effective when fine-tune very large pre-trained models. This greatly reduces the cost of fine-tuning large models on downstream tasks.
4.6 Limitations
Although the above experiments demonstrate the power of SVF, it still has some limitations. For instance, SVF introduces small number of learning parameters, but the occupancy rate of memory resources is high during training process. Using SVF in ResNet-50 will occupy 16G video memory per image in COCO-20' 5-shot setting. Furthermore, SVF increase small amount of training time compared with freeze backbone.
Conclusion
In this paper, rethink the paradigm of freezing backbone in FSS and propose paradigm Singular Value Fine-tuning (SVF) for fine-tuning backbone. Firstly, SVF decompose pre-trained parameters into three subspaces by SVD, and then only fine-tune the singular value. Our SVF dynamically adjusts the weights of different semantic cues without changing the rich semantic cues in pre-trained backbone. We evaluate the effectiveness of SVF on two commonly used benchmarks, Pascal-5' and COCO-20'. Extensive experiments prove that SVF as a new perspective to avoid overfitting and significantly improve the performance of various FSS methods. As new paradigm of finetune, we will extend it to a variety of vision tasks in the future.
Acknowledge This work was partially supported by the National Natural Science Foundation of China (Grant No. U20B2064 and U21 1B2043).
References
[1] H Andrews and CLIII Patterson. Singular value decomposition (svd) image coding. IEEE transactions on Communications, 24(4):425-432. 1976.
[2] Ankur Bapna and Orhan Firat. Simple. scalable adaptation for neural machine translation. In Proceedings of the 20/9 Conference on Empirical Methods in Natural Language Processing, pages 1538--548,, 2019.
[3] Yoshua Bengio and Yann LeCun. Very deep convolutional networks for large-scale image recognition. In International Conference Learning Representations. 2015.
[4] Emily Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances neural information processing systems,
27, 2014.
[5] Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning. In BMVC. volume 3, 2018.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. In International Conference on Learning Representations, 2020.
[7] Ross Girshick. Fast r-cnn. In Proceedings of the IEEE International Conference Computer Vision,
pages 1440-1448, 2015.
[8] Yunhui Guo. Houghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5805-4814, 2019.
[9] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[10] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE International Conference Computer Vision, pages 2961-2969, 2017.
10

---

[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 770-778. 2016. [12] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[13] Zilong Huang. Xinggang Wang. Lichao Huang. Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 603-612, 2019.
[14] Menglin Jia, Luming Tang. Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. arXiv preprint 2203, 19, 2022.
[15] Chunbo Lang, Gong Cheng. Binfei Tu, and Junwei Han. Learning what not to segment: A new perspective few-shot segmentation. arXiv preprint arXiv:2203 07611, 2022.
[16] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE Conference on
Computer Vision and Puttern Recognition, pages 8343. 2021.
(17] Zechao Li, Yanpeng Sun, Liyan Zhang, and Jinhui Tang. Ctnet: context-based tandem network for semantic segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[18] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference on Computer Vision, pages 2980-29888 2017.
[19] Weide Liu, Zhonghua Wu, Henghui Ding, Fayao Liu, Jic Lin, and Guosheng Lin. Few-shot segmentation with global and local contrastive learning. arXiv preprin! arXiv:2 05293, 2021.
[20] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431 -3400, 2015.
[21] Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 160iv:1608.00 03983. 2016.
[22] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In Proceedings of the IEEE International Conference on Computer Vision, pages 8741-8750, 2021.
[23] Juhong Min. Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In 2021.
Proceedings of the IEEE International Conference on Computer Vision, pages
[24] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE International Conference on Cumputer Vision, pages 622-631. 2019.
[25] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. Film: Visual reason- ing with a general conditioning layer. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
[26] James Requeima, Jonathan Gordon, John Bronskill, Sehastian Nowozin, and Richard E. Turner. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in Neural Information Processing Systems, 32, 2019.
[27] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, ct al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3):21 11-22,, 2015.
[28] Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta- leaannng with latent embedding optimization. In International Conference on Learning Representations, 2018
[29] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic
segmentation. arXiv preprint arXiv: 110911741010 2017.
[30] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Ting Cheng. Partial is better than all: Revisiting fine-tuning strategy for few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 9594-9602, 2021.
[31] Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980. 2020.
[32] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing xystems, 30, 2017.
[33] Hao Tang. Zechao Li. Zhimao Peng. and Jinhui Tang. Blockmix: Meta regularization and celf-calibrated inference for metric-based meta-learning. In ACM Multimedia, pages 610 618, 2020.
[34] Hao Tang, Chengcheng Yuan, Zechao Li, and Jinhui Tang. Learning attention-guided pyramidal features for few-shot fine-grained recognition. Pattern Recognition, 130:108792, 2022.
[35] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided
feature cnrichment network for fcw-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[36] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal
few-shot learning with frozen language models. Advances in Neural Informtionn Processing Systems, 1::00-2122 2021.
11

---

[37] Kaixin Wang, Jun Hao Liew, Yingtian Zou. Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with protolype alignment. In Proceedings of the IEEE International Conference on Computer Vision, pages 9197-9206, 2019.
[38] Ruize Wang. Duyu Tang. Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. K-adapter: Infusing knowledge into pre-trained models with adapters. Findings of the Association for Computational Linguistics, 2021.
[39] Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? an analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 34, 2021.
[40] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-aware graph neural network for few-shot semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5475 5484, 2021.
[41] Guo-Sen Xie, Huan Xiong, Jie Liu, Yazhou Yao, and Ling Shao. Few-shot semantic segmentation with cyclic memory network. In Proceedings of the IEFE International Conference on Computer Vision, pages
7293-7302, 2021.
[42] Boyu Yang. Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for fow-shot semantic segmentation. In European Conference on Computer Vision, pages 763-778. Springer, 2020, [43] Lihe Yang. Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. Mining latent classes for few-shot segmentation. In Proceedings of the IEEE International Conference Computer Vision, pages 8721-8730, 2021.
[44] Yuhui Yuan, Xilin Chen. and Jingdong Wang. Object-contextual representations for semantic segmentation. In European Conference
Computer Vision, pages 173-190. Springer, 2020.
[45] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learnin. In Proceedings the IEEE Conference
on Computer Vision and Puttern Recognition, pages 5217-5226, 2019.
[46] Dong Zhang. Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, and Qianru Sun. Feature pyramid transformer. European Conference on Computer Vision, pages 323-339. Springer, 2020.
[47] Gengwei Zhang, Guoliang Kang. Yi Yang. and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34, 2021.
[48] Tianyi Zhang, Felix Wu, Arzoo Katiyar, Kilian Q Weinberger, and Yoav Artzi. Revisiting faw-sample bert
fine-tuning. In International Conference on Learning Representations, 2020.
[49] Xiaolin Zhang. Yunchao Wei, Yi Yang. and Thomas Huang. Sg-one Similarity guidance network for ohe-shot semantic segmentation. IEEE Transactions on Cybernetics, 5019):38553365 2020. [50] Zhuangwei Zhuang, Mingkui Tan, Bohan Zhuang, Jing Liu, Yong Guo, Qingyao Wu, Junzhou Huang, and Jinhui Zhu. Discrimination-wware channel pruning for deep neural networks. Advances in neural information processing systems, 31, 2018.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section 4.1.
Did you include the license to the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the
Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
12

---

(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [N/A]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section 4.1
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [Yes]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "vCQAVh-_rml", "reviewer": "Reviewer_LFht", "summary": "The authors have proposed a new training scheme for FSS frameworks by only adapting a few parameters of the ImageNet pre-trained backbone to the segmentation task. The core idea is to adjust the singular values of the pre-trained kernel weights of the backbone, bringing considerable improvements to representative FSS methods (PFENet and BAM). Extensive experiments have shown the effectiveness of the proposed method. ", "questions": "This paper does present a good method for boosting existing few-shot segmentation methods whose backbone parameters are fixed, and considerable improvements have been achieved. However, my biggest concern is still about the training setting. \n\nThe authors of submission 1788 follow BAM to remove all training images that contain novel classes, for the purposes of avoiding information leakage, but all previous methods in the community follow Shaban et al[1] to keep those images during the training phase by setting the labels for the novel classes as background, which explains why PFENet without dagger in Table 1 is much worse than the one with the dagger. \n\nMore specifically, in section 3 of the paper [1], the authors of [1] wrote ''In this problem, unlike image classification, examples from L_test might appear in training images. This is handled naturally when an annotator unaware of some object class, labels it as background''. \n\nTherefore, the reviewer thinks that BAM actually has introduced an unfair comparison in their paper, and it would be better if the authors of submission 1788 could clearly present the results of Baseline + SVF, PFENet + SVF, and BAM + SVF in Table 1 and Table 2, by keeping the novel categories but setting them as the background during the base training phase, for a fair comparison with previous methods that adopt the same setting as described in [1].  \n\nIn Table 2 of the appendix, what causes the success on Fold-2 when the novel classes are removed from the training set? If the novel classes are included as [1] and set as the background, whether the proposed method will be negatively affected? \n\nMinor issues:\n1. Are the PFENet and BAM without dagger shown in table 3 trained with novel classes whose labels are set to the background?\n\n2. Visualizations in Figure 5 and Figure 6 are confusing and they could be improved by indicating what the target classes are. For example, in the last examples of Figure 5 and Figrue 6, the 1x1 weights focus on the person more than boat, but contrarily, the 3x3 weights are more curious about the boat, which contradicts the problem setting that only one target class exists in each evaluation episode. \n\n\nReferences\n[1]  One-Shot Learning for Semantic Segmentation. BMVC 2017\n\n ", "limitations": "As described in the Questions section, there might be some unfair comparisons with previous methods, and some critical aspects are not clear enough. The authors are encouraged to show additional results to support their claims.\n\n[updated after rebuttal] My initial concerns regarding the comparison have been well-addressed in the revision. Thus my final rating is ACCEPT (increased from 4 -> 7).", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n+ A good extension of the current training scheme for FSS frameworks.\n\n+ Decent performance gain has been brought to representative FSS methods (PFENet and BAM).\n\n+ Clear motivation and good presentation of the method and discussion. \n\n+ The overall submission is well-prepared with comprehensive appendix.\n\nWeakness: \n\n- Unfair comparisons with the previously proposed methods (please see Questions section for details).\n\n- Visualizations in Figure 5 and Figure 6 are confusing. \n\nIf the above weaknesses could be well addressed, I would like to give a higher rating.\n\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dXGQryd7iy1", "reviewer": "Reviewer_xj2U", "summary": "In order to handle the overfitting issue in few-shot segmentation, this paper proposes to fine-tune a small part of backbone parameters recognized via the singular value decomposition. Precisely, the proposed Singular Value Fine-tuning (SVF) method suggests merely tuning the decomposed singular-value diagonal matrix for each convolutional layer. The experiments on few-shot segmentation among two datasets show the positive effects of fine-tuning the backbone using the SVF approach.", "questions": "The major concern is the vague motivation since the reviewer cannot find any connection of the proposed Singular Value Fine-tuning is specially designed for the “few-shot” scenario. In addition, the other concern of this paper is lacking compared with previous methods [A, B] related to manipulating the weight matrices of the convolutional layers. Please see [Weaknesses] for reference.  ", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n+ The idea of recognizing the singular-value-related backbone parameters for fine-tuning is interesting.\n+ The manuscript is well organized and has several experiments.\n\n[Weaknesses] \n- The motivation is vague since the strong connection between the backbone fine-tuning and few-shot segmentation is unclear. The frozen pretrained-backbone could be treated as a feature extractor for a downstream task; hence the downstream task could focus on the mechanism design for employing the extracted features. For fine-tuning a backbone network as a goal, why not compare with the methods of meta-learning, adaptor, bias tuning, or domain adaption? Therefore, it is curious why it must fine-tune the backbone for tackling the few-shot segmentation. Specifically, the reviewer cannot find any components within the proposed Singular Value Fine-tuning specially designed for the “few-shot” scenario. As a result, it is suggested to deploy the proposed Singular Value Fine-tuning on various tasks, including classification and detection, to demonstrate the overfitting-proof advantages of the proposed SVF method.\n- In line 78, the authors claimed that the work [10] is not suitable for few-shot vision tasks, yet there are no related experiments showing this issue.\n- Since the proposed SVF is used to decompose the basic convolutional layers within a backbone, it may result in additional computational time while carrying out SVF. Therefore, it is better to include the experiments in discussing the required extra training time.\n- The reference is not sufficient. For example, two related methods shown as follows should be compared.\n[A] Kui Jia, Dacheng Tao, Shenghua Gao, Xiangmin Xu: Improving Training of Deep Neural Networks via Singular Value Bounding. CVPR 2017: 3994-4002.\n[B] Hanie Sedghi, Vineet Gupta, Philip M. Long: The Singular Values of Convolutional Layers. ICLR 2019.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XxpRT9q6noL", "reviewer": "Reviewer_8FCE", "summary": "Authors re-examine the idea of fine-tuning the backbone feature extractor during few-shot semantic segmentation, showing that overfitting can be avoided by limiting updates to a small set of parameters. Specifically, authors decompose convolution layers using SVD, and fine tune only the singular values S. Results indicate that this approach increases performance relative to using a frozen backbone, while fine-tuning other groups of parameters consistently decreases performance. ", "questions": "What is truly responsible for the success of SVF? \n\nI propose the following experiment to start: let R be a random rotation matrix, and set U=R’ and V=RW, where W is the original weight matrix for the given layer. Then attempt SVF fine-tuning. This will show whether SVF depends crucially on the singular value space, or simply on the number of effective updated parameters. Fine tuning only the BN scale terms might also be worth a try, for a true apples-to-apples comparison – it could be that the bias term is uniquely destructive. \n\nAlternatively, if authors could elaborate on what makes the singular value space so unreasonably amenable to fine-tuning, this may become clearer. Perhaps I am simply failing to understand the intended argument. \n\nSMALL COMMENTS AND TYPOS:\n\nPg3 line 100: while does not  while it does not\n\nPg4 line 103: one need  one needs\n\nPg4 line 132: parameters, also  parameters, and also\n\nPg5 line 177: all classes all classes  all classes\n\nPg5 lines 178-179: are used … are used  used … used\n\nPg7 line 214: shows that the  shows the\n\nPg7 lines 227-228: you seem to have a redundant sentence here (Finally… Finally)\n\nPg8 line 251: without destroy  without destroying\n", "limitations": "Discussion of limitations is fair. Societal impacts are not discussed, though do not extend beyond those of few-shot learning in general. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS:\n\nMotivation is clear, and approach is sensible, straightforward, and highly applicable. Results are convincing and a thorough analysis is provided. Paper is well organized (though with occasional typos and some awkward language, see below). \n\nWEAKNESSES:\n\nWhile the paper convincingly demonstrates that SVF is effective, and acts as expected, it does not adequately explain why. Authors imply that fine-tuning in the singular value space is uniquely non-destructive, but it is not intuitively obvious that this should be the case. Indeed, in Fig.4 many values \nswitch signs, indicating that nothing is stopping SVF from zeroing out large swaths of the output manifold in practice. Without knowing why the singular value space is so particularly effective, the contribution is limited, as constrained fine-tuning is already a widely known approach in the few-shot regime (see below) and is not novel in and of itself. \n\nAlong these lines, while the analysis is broad and involved, the comparisons are somewhat apples-to-oranges. Rather than singular values being special, it could instead be that SVF simply allows the authors to fine-tune a smaller number of parameters than their comparative baselines. For starters, all fine-tuning comparisons except for BN involve a far larger number of parameters, and so the observed overfitting is not surprising, and even the BN baseline involves twice as many parameters (scale _and_ shift). Additionally, it could be that SVF allows for a smaller number of _effective_ updates compared to BN. For example, if matrix U is highly axis-aligned, changes to S will shrink or vanish in the subsequent BN layer. If U=I, then S should not update at all. \n\nFurther analysis is required. \n\nLess importantly: while the related work is quite broad, there exist similar approaches in the broader few-shot literature also based on fine-tuning a highly constrained subset of introduced parameters. These may be worth mentioning – e.g. LEO (Meta-Learning with Latent Embedding Optimization, ICLR2019), CNAPS (Fast and Flexible Multi-Task Classification Using Conditional Neural Adaptive Processes, NeurIPS2019), or possibly FiLM (FiLM: Visual Reasoning with a General Conditioning Layer, AAAI2018). ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8E_wAQvQjNO", "reviewer": "Reviewer_jAEv", "summary": "In this paper, the authors propose a novel SVF to changes the standard paradigm of freeze backbone in few-shot segmentation.The results show that only fine-tune the parameters of the singular value subspace (S) and freeze other subspaces can not only effectively avoid the overfitting problem, but also significantly improve the performance of FSS model. Specifically, the visualization results prove that some semantic cues with high weight in the pretrained weight are not conducive to downstream tasks, thus SVF adjusts the weights of different semantic cues by fine-tuning the parameters of subspace S. They further confirm the effectiveness of their proposed method by running experiments on real datasets and applications and comparing them to other topologies and methods. Moreover, they further confirm the effectiveness and superiority of SVF by comparing them to other representative methods on two common datasets.", "questions": "- In section 3.3 and appendix, the author details two perspectives of SVF. The two perspectives are implemented differently. The paper does not analyze the impact of SVF from different perspectives on the FSS model.\n\n- As an important part of the backbone, the bias has a huge impact on the model during the finetuning process. The authors lack an analysis of bias among different fine-tuning methods.\n\n- In section 4.4 and apendix, the author only shows the change of the top-30 singular values of different convolutions, and the change results of all the singular values are missing.\n", "limitations": "This paper does not reflect any potential negative societal impact.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- This paper is overall writing is clear and easy to follow. This is a really interesting paper. This paper not only analyzes the reason why the freezing backbone in FSS has become the traditional paradigm, but also proposes a novel fine-tuning method to try to break the traditional paradigm. The novel method SVF design is simple yet effective as proven by extensive experiments and analysis. Meanwhile, this paper analyzes the impact of hidden tricks in FSS on model performance to facilitate fair comparisons in future work. The core of SVF is a novel method of fine-tuning backbone, thus SVF has universality and can be easily used for all FSS models. As a result, this could serve as a new paradigm for few-shot semantic segmentation task. I believe that this paper will be a valuable contribution to the field, and I strongly recommend acceptance.\n\n- The experimental results demonstrate the effectiveness of SVF, but will increase the training cost by a small amount compared to freeze backbone.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept