Feature-Proxy Transformer for Few-Shot Segmentation
Authors: Jian-Wei Zhang
Yifan Sun
Yi Yang
Wei Chen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-shot segmentation
vision transformer
prompt learning

Abstract: Few-shot segmentation~(FSS) aims at performing semantic segmentation on novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised segmentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the supervised segmentation methods use a simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of ``feature extractor $+$ linear classification head'' and proposes a novel Feature-Proxy Transformer (FPTrans) method, in which the ``proxy'' is the vector representing a semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from bottom to top layers using a novel prompting strategy. 2) FPTrans uses multiple local background proxies (instead of a single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and proxies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of-the-art decoder-based methods. 

Feature-Proxy Transformer for Few-Shot Segmentation
Yi Yang", Wei Chen 1 State Key Lab of CAD&CG, Zhejiang University, Hangzhou, China
Wei Zhang', Yifan Sun?,
2 Baidu Research
3 CCAL College of Computer Science and Technology, Zhejiang University (zjw.cs, yangyiccssnnssss...s @z (ww.rrss)eeeeeu.......m edu. cn, sunyf15@tsinghua org.
Abstract
Few-shot segmentation (FSS) aims at performing semantic segmentation novel classes given a few annotated support samples. With a rethink of recent advances, we find that the current FSS framework has deviated far from the supervised seg- mentation framework: Given the deep features, FSS methods typically use an intricate decoder to perform sophisticated pixel-wise matching, while the super- vised segmentation methods use simple linear classification head. Due to the intricacy of the decoder and its matching pipeline, it is not easy to follow such an FSS framework. This paper revives the straightforward framework of "fea- extractor + linear classification head" and proposes novel Feature-Prox Transformer (FPTrans) method, in which the "proxy" is the vector representing semantic class in the linear classification head. FPTrans has two keypoints for learning discriminative features and representative proxies: 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from bottom to top layers using novel prompting strategy. 2) FP- Trans uses multiple local background proxies (instead of single one) because the background is not homogeneous and may contain some novel foreground regions. These two keypoints are easily integrated into the vision transformer backbone with the prompting mechanism in the transformer. Given the learned features and prox- ies, FPTrans directly compares their cosine similarity for segmentation. Although the framework is straightforward, we show that FPTrans achieves competitive FSS accuracy on par with state-of- -ftth-of-the-rt decoder-baat based methods.
Introduction
Few-shot learning of significant value for semantic segmentation. is because the semantic segmentation task requires pixel-wise annotation, which is notoriously cumbersome and expensive [10 30 28). Therefore, learning from very few samples for semantic segmentation has attracted significant research interest, yielding popular topic, i.e., few-shot semantic segmentation (FSS). Formally, FSS aims at performing semantic segmentation on novel classes given only a few (e.g., one or five) densely-annose annotated samples (called support images) [5].
With a rethink of recent advances in FSS, we find current FSS methods usually require an intricate decoder, deviating far from the plain supervised segmentation framework. More concretely, -f the-art FSS methods adopt the "feature extractor + (intricate) decoder" framework (Fig. (a). (b) and
Work done during an internship at Baidu Research.
i Cooresnn author.
Code is available at httpps://gkshuub.com//rrrsssss/s
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

OKN RooM Pulaon Transfanee
ED CENNR Tooels
Fazure Ranaa
Decader
Clss forie
Prodes
Milly Uap
Mart
4
4
I
I
I
4
(a)
(b)
(0)
(d)
(e)
Figure Comparison between the "feature extractor + intricate decoder" (a, b and e) and the plain"feature extractor + linear classification head" framework (d and e). (a) The decoder refines the prior maps (raw segmentation) for final prediction (e.g., FWB [38] CWT [36]. SCL. [I].. (b) The decoder concatenates the support prototype and the query features and then further feeds them into the CNN or transformer (e.g., PFENet [46), ASGNet (27]. CyCTR (64)). (c) The decoder conducts pixel-to-pixel matching from query to support and then applies 4D convolution transformer to discover the patterns within the matching score maps (e.g., HSNet (37] VAT [9]). (d) The plain framework of "feature extractor + linear classification head" in supervised segmentation. (e) The proposed FPTrans revives the plain framework and makes only one necessary modification (i.e., the proxies extracted from the support images on the fly).
(c)), while the supervised segmentation methods usually adopt the "feature extractor + (simple) linear classification head" framework (Fig. (d). In FSS frameworks, the decoders perform sophisticated matching and can be summarized into three types (Fig. (a), (b), and (c)), as detailed in the related works in Sectiorf.1]] Arguably, the intricacy of the decoder and its sophisticated matching pipeline makes the FSS framework hard to follow. Under this background, we think it is valuable to explore relatively straightforward FSS framework.
This paper revives the plain framework of "feature extractor + linear classification head" for FSS and proposes a novel Feature-Prox Proxy Transformer (FPTrans) method. The term "proxy" denotes the vector representing a foreground class or the background in the linear classification head. We note that in Fig. (d), given the extracted feature maps, the supervised segmentation methods simply feed them into a linear classification head to perform pixel-wise prediction. FPTTans adapts this straightforward framework to the FSS task with only one modification: instead of fixing the already-learned proxies in the classification head (Fig. (d)), FPTrans uses support feature maps and the support mask to generate the proxies on the fly. This modification necessary for recognizing novel classes and consistent with some earlier FSS methods [4] 500
To tackle two FSS challenges (i.e., generalization to novel classes and very few support samples) under this simple framework, FPTrans has two keypoints for learning discriminative features and rep- resentative proxies, respectively. 1) To better utilize the limited support samples, the feature extractor makes the query interact with the support features from the bottom to top layers. Consequently, the support sample provides extra information/clues for extracting the query features and is thus beneficial. 2) To promote generalization to novel classes, FPTrans uses multiple local background proxies instead of single global proxy for representing the background. This design is important because, during base training. the background is not homogeneous and may contain some novel foreground classes. Consequently, it avoids confusing novel classes with the background of the base data and thus benefits generalization to novel classes.
We implement the above two keypoints with a novel prompting strategy. While using the prompt to condition a transformer for different tasks (29 (3 51 or different domains (13) is a common practice, our prompting strategy is significantly different and has two novel functions, i.e., 1) prompting for different (foreground or background) proxies and, 2) acting as the intermediate for query-support interaction. Specifically, FPTrans simultaneously prepends multiple prompts at the input layer, with one prompt for the foreground and the other prompts for the background. These prompts are fed into the transformer and finally become the foreground proxy and local background proxies, respectively. During their flow from the input layer to the output layer, the hidden states of prompts in all the hidden layers are shared by the query and support for cross-attention (query-promm attention and support-po support-prompt attention). Therefore, it significantly reduces the interaction complexity from
2

---

O(N2) to ((N) (N is the number of pixels on the feature maps). Since the prompting and attention mechanism are critical for these two keypoints, we use the transformer backbone as a natural choice.
We conduct extensive experiments and show that FPTrans achieves accuracy on par with the decoder- based FSS methods. For example, PASCAL-55 [5] with one support sample, FPTrans achieves 68.81% mioU, setting a new state of the art. With its simple framework and competitive accuracy, we hope FPTrans can serve as a strong baseline for FSS.
To sum up, our main contributions are summarized as follows: (i) We revive the plain "feature extractor + linear classification head" framework for FSS. We hope the prrespondinglyyproposee method FPTrans can serve as a simple and strong FSS baseline. (ii) We integrate two keypoints into FPTrans, i.e., learning discriminative features through query-support interaction and learning representative local background proxies. These two keypoints rely on a novel prompting strategy of the transformer backbone and correspondingly tackle two FSS challenges, i.e., very few support samples and generalization to novel classes. (iii) We conduct extensive experiments to validate the effectiveness of FPTrans. Experimental results show that FPTrans with a plain framework achieves competitive accuracy, compared with state-of-the-ar FSS methods with intricate decoders.
2
Related Work
2.1 Recent Progress Few-Shot Segmentation
40 45 or apply Early FSS methods directly generate class-specific classifier weights and biases 41
the prototype learning 44
42 The recent state-of-the-an methods shift to the decoder-based framework and can be categorized into three types according to the decoder structure. 1) Some
50
(Fig. Ta)) generate prior maps based on query features and support features. They further use CNN or transformer to refine the prior maps (which may be viewed as raw segmentation) into the final segmentation output. 2) Some methods [66 63
recent methods 38 32 36
6D
46 27
55
56 62 (Fig. (J)) concatenate the support prototype vector and the query features and then feed the concatenated feature maps into a subsequential CNN or transformer for prediction. 3) Some methods focus exploring fine-grained knowledge and calculating the pixel-to-pixel matching scores between
support and query features to derive affinity maps 22 49 58
64
59
36) (Fig. [.)). Some
further apply 4D convolution or transformer (37 [9] to discover the latent patterns within the affinity
maps. Besides, latent information is also investigated to enhance the model [39 67 25 57 52 34).
In contrast, this paper abandons the intricate decoder and revives the plain framework of "feature extractor + linear classification head". We show that this simple framework can also achieve promising FSS results. We note that recent work [2] also uses the plain framework during training. However, they rely on transductive inference during testing to mitigate the gap between the inconsistent training and testing schemes. Compared with [2), the proposed FPTrans maintains its simplicity for testing and achieves superior FSS accuracy.
2.2 Backbone for Few-Shot Segmentation
Previous FSS methods usually adopt CNNs (e.g., VGG 43]. ResNet
17 as the backbone (i.e.,
feature extractor) and typically fix the pretrained backbone parameters 63
46
64
52 36 27
37 [9]. However, fixing the backbone is prone to a side-effect, i.e., insufficient adaptation to the segmentation training data. Some recent methods finetune the CNN backbone along with FSS training
with sophisticated techniques (e.g., model ensemble transductive inference [26] 2 to tackle the problem of insufficient adaptation.
as the backbone because we rely on the attention mechanism and a novel prompting technique for query-support interaction. Moreover, the proposed FPTrans benefits from tne-tuning the backbone parameters without bells and whistles. We note that using the transformer backbone does NOT necessarily improve FSS, because the ablation studies (in Table show that replacing the CNN backbone with transformer does NOT bring improvements to the recoder-based FSS methods [46 64]. Therefore, we attribute the superiority of our method mainly to the two unique keypoints in FPTrans.
In contrast, this paper adopts the vision transformer 9 18 53
35
3

---

3 Methods
3.1 Problem Formulation
Few-shot segmentation aims at tackling the semantic segmentation problem on novel classes under low data regime, Specifically, FSS usually provides a training set with categories Ctrain and testing set with novel categories Ctest (Ctrain n Ctest 0). The mainstream setting 663 64 46 adopts the episodic training and testing scheme: Each episode corresponds to a single class (c E Ctrain during training and E Ctest during testing), and provides a query sample { Yg and K support
samples (1 y, JKK Jk=1 (1 is the image and Y is the label), The superscript &o will be omitted unless necessary. Within each episode, the model is expected to use {Ls, Ys} and L, to predict the query label. In this paper, we follow this popular episodic training and testing scheme.
3.2 Preliminaries on Vision Transformer and Prompt Learning
The proposed FPTrans uses the vision transformer as its backhone and integrates novel prompting strategy. Therefore, we first revisit the vision transformer and the prompting mechanism.
Vision Transformer (ViT) is designed for computer vision tasks based on transformer [8] which originally designed for sequential data 7 33], It is composed of a patch embedding module, transformer encoder, and an MLP head. Given an RGB image as the input, ViT first reshape it into N patches (a, E 3 Px P pp- = 1, .,, .N) (P the patch size) and then projects the flattened image patches into C-Dimensional embeddings by Xp Embed ap) RC
1,2,..... .,, N. We denote the collection of these embedding tokens X { E NxC VIT has L stacked transformer blocks, each one of which consists of a Multiheaded Self-Attention module and an MLP module (See supplementary Section A for details). Given X (the collection of embedding tokens), ViT concatenates it with classification token x2, E RC and then inputs them into the stacked transformer blocks, which is formulated as:
1
X x|| B: (x 1 1),
1= 1,1, ..,,,
where x/1, and X are the outputs of the 1-th block B,, and
[. is the concatenate operation.
Prompting was first introduced in NLP tasks to identify different tasks by inserting a few hint words into input sentences 112 24 31. More generally, prompting techniques can efficiently
condition the transformer to different tasks [29] 20 16 51 or domains [3] without changing
any other parameters of the transformer. To this end, prompting techniques typically prepend some
prompt tokens pD to the input layer. Correspondingly, Eqn IT) transforms into [x X P']
(1x x p'- It can be seen that changing the prompt simultaneously changes the mapping function of the transformer, even all the transformer blocks B₁ remain unchanged.
Our prompting strategy in FPTrans is significantly different from the prior prompting techniques. In contrast to the popular prompting for different tasks different domains, FPPrrnn prepends multiple prompts to simultaneously activate multiple different proxies (i.e., the foreground and local background proxies), as well as to facilitate efficient query-support interactions. Moreover, in prior works, the prompts in the hidden layers are dependent on single input sample. In contrast, in FPTrans, the prompts in the hidden layers are shared by the query and support images through a synchronization procedure. We will illustrate these points in the following section.
3.3 Feature-Proxy Transformer
3.3.1 Overview
The proposed Feature-Proxy Transformer (FPTrans) consists of three major steps, i.e., 1) prompt generation, 2) feature and proxy extraction, and 3) training the classification head and inference, as illustrated in Figure 2 The prompts are extracted from the support images within a (training or testing) episode and contain foreground prompt and multiple local background prompts (Section 3.3.2). Afterward, FPTrans concatenates these prompts with the image patch tokens and forwards them into the stacked transformer blocks to extract both the features and proxies (Section 3.3.3). Different from the popular prompting technique, FPTrans shares the hidden states of the prompt tokens for the query and its support images, using prompt synchronization operation. It facilitates efficient feature

---

Class Aware Generation
Overall Structure FPTrans
0000
Classfication Head
Classry
Fest un Based Prodes
Prompla
Tikens
Extract Presses
Prompisieee Procee
Classly
Regora
Extract Proxios
Synnc Prompt
XL
Partions
Foatuic Mazs
Preteined
Enbwaing
Promots
Einterkinng
Support Image
(a)
Image
Support Image
Figure 2: Overview of the proposed Feature- Transformer (FPTrans). (a) Given support image, we generate foreground prompt and multiple local background prompts. Each prompt consists of multiple tokens. (b) The feature extractor consists of I. transformer blocks. It takes patch tokens from the query and support images, as well as the prompts as its input. After every transformer block, FPTrans synchronizes the prompt tokens from the query and support branches facilitate efficient query-support interactions. The classification head uses two types of proxies (the feature-based and the prompt-base based proxies) for training and uses the feature- based proxies for inference.
interactions between support and query features. Finally, given the extracted features and proxies, elaborate on how to use them for training the classification head and for inference in Section 3.3.4
3.3.2 Prompt Generation
(a). Within each episode, FPTrans uses the support image(s) to extract class-aware prompts and share them for support and query images. The prompts are class-aware because we respectively extract prompts from the foreground and background.
Prompt generation is illustrated in Fig. 2
To this end, we first use a pretrained plain vision transformer (Eqn. (I) to extract deep features from the support image and get F; E CxC xxW xW in which И and W are the feature height and width, and the subscript s indicates the "support". Afterward, we use the support mask to crop out the foreground regions and background regions from F.. While the foreground regions are cropped out as whole, the background regions are cropped and partitioned into multiple (S) local regions, because the background is likely not to be homogeneous. To this end, we employ a Voronoi- based method [ to partition background into S regions (See supplementary Section B. for details).
According to the background partition results, we generate S local masks for the background, i..., [B,]s- with Bn 10,1)xxW, A partition example of
illustrated in Fig. a). Consequently, we calculate the mean feature of the foreground uy E IRC and the local mean feature of background u RC by masked average pooling:
NW Σ
nW Σ F.
F
= [B
n = 1,...., S,
(2)
i=1
where Y, E {0, 1) XWW is the down-sampled foreground mask of the support image.
Based on the mean features, we further expand each C-dimensional mean feature vector (uf E RC and u, E RC) into corresponding G x C' token and then add it with extra learnable tokens by:
ps E(u7) zz, Pn E(u() + Zno
1, ,,
(3)
5

---

where &l is the expansion from C-dimensional vector to G' x C-dimensional token, and Z,, Zn E RGxC are the learnable tokens. Adding these learnable tokens makes prompts gain extra diversity and become more discriminative [60. See supplementary Section B.2 for more details on this prompt augmentation technique. Consequently, after the expansion and augmentation, we get foreground prompt token Py E RGCC and local background prompt tokens Pn, RGxC (n -.2.. 5).
3.3.3 Feature and Proxy Extraction
Feature Extraction. The process for extracting features is illustrated in Fig. 2b). Without loss of generality, take the 1-shot setting the example for clarity. (See supplementary Section C for K-shot settings) Within each episode, a query image ln and a support image I, are split into N patches and then flattened as (a and {as.p) p=1, respectively. An embedding layer projects these patches into query and support patch tokens, i.e., X E N and X E IR NxC. We recall that in Section 3.3.2 we already get multiple prompt tokens, i.e., p [p Po: P..... p.]. The patch tokens from a single image and the prompt tokens are concatenated as the query input and support input, i.e., [ Po] and [X] Po. The query and support inputs processed by the transformer blocks, which are formulated by:
X Pl = Bi[[X 1-1 '), q. q.
(4)
4 X P = B, 1-1 1),
(5)
P = =P'P' PP)/
where / = 1,2, .... ,L enumerates all the transformer blocks.
There is novel and unique prompt synchronization (Eqn. (6) in FPTrans. Specifically, Eqn. (6 averages the query and support prompt tokens after every transformer block. It makes the synced prompt token P, absorb information from both the query and support patch tokens. Therefore, in the subsequential + 1)-th block, the synced prompt token P/ passes the support information to the query tokens g P and vice versa. In a word, this simple prompt synchronization facilitates efficient interaction between query and support features.
Proxy Extraction. As the results of Eqn. 4 to Eqn. (6), the feature extractor outputs support features X query features X as well as the deep states of prompts P'. FPTrans uses the support features X and the deep states of prompts p4 to extract two types of proxies, i.e., the feature-based proxies and the prompt-based proxies, respectively.
crature-based proxies. FPTrans uses the downsampled foreground mask Y, and the partitioned background masks Bn (which are already provided in Section 3.3.2) to extract the feature-based proxies by: uj | Σ; Fs, Va,is U2 = [B_
[. Σ. Fg, Br, n
S.
Prompt-based proxies. Moreover, FPTrans uses the deep states of prompts p4 to get the prompt- based proxies by: V = t E = p/. and V, & Ej= P
These two types of proxies are both used for training FPTrans. In contrast, during testing, FPTrans only uses feature-basee proxies.
3.3.4 Training and Inference
FPTrans two classification losses (one for the feature-based proxies and the other for the prompt-based proxies) and pairwise loss for training.
Classification losses. Since segmentation can be viewed as pixel-wise classification problem, FPTrans directly compares the similarity of each query feature vector (F,,s) and the proxies for linear classification. We elaborate the classification loss with the process with the prompt-based proxies is similar). The predicted probability of F, belonging to the foreground is formulated as:
proxy (and the
exp(sim(F q.i u/)/r)
P(F q.3)
n = 1,2, ... S
(7)
exp (sim (F q.i, uy \/r) + max, (exp (sim( F q:s; un/r))))
where sim(;, :) is the cosine similarity between two vectors, 7 is a temperate coefficient. The max operation is critical because it facilitates comparing feature to its closest background proxy.

---

Table 1: Comparison with state-of-the-art methods on PASCAL-5'. We report 1-shot and 5-shot results using the mean IoU (%)
I-shot
5-shot
Backbone
Method
SO
S1
S2
S3
Mean
S1
S2
S3
Mean
RPMM
156
55.2
66.9
52.6
50.7
56.3
56.3
67.3
54.5
51.0
57.3
PFENet
46
61.7
69.5
55.4
56.3
60.8
63.1
70.7
55.8
57.9
61.9
Res-50
CyCTR
664
67.8
72.8
58.0
58.0
64.2
71.1
73.2
60.5
57.5
65.6
HSNet
37
64.3
70.7
60.3
60.5
64.0
70.3
73.2
67.4
67.1
69.5
BAM
26
69.0
73.6
67.6
61.1
67.8
70.6
75.1
70.8
67.2
70.9
DAN 49
54.7
68.6
57.8
51.6
58.2
57.9
69.0
60.1
54.9
60.5
RePRI 2
59.6
68.6
62.2
47.2
66.2
71.4
67.0
57.7
65.6
Res-101
PFENet
46
60.5
69.4
54.4
55.9
60.1
62.8
70.4
54.9
57.6
61.4
CyCTR
66
69.3
72.7
56,5
58.6
64.3
73.5
74.0
58.6
60.2
66.6
HSNet
17
67.3
72.3
62.0
63.1
66.2
71.8
74.4
67.0
68.3
70.4
Baseline
62.9
69.1
62.2
53.0
61.8
70.5
76.0
74.2
65.5
71.5
ViT-B/16
FPTrans
67.1
69.8
65.6
56.4
64.7
73.5
75.7
77.4
68.3
73.7
Bascline
68.2
69.4
61.7
60.5
64.9
75.3
78.1 1
76.1
73.7
75.8
DeiT-B/16
FPTrans
72.3
70.6
68.3
64.1
68.8
76.7
79.0
81.0
75.1
78.0
Given the predicted probability and the ground-truth label, FPTrans uses the standard cross-entropy loss Lee for supervising the pixel-wise classification. In parallel to Lees FPTrans uses the prompt- based proxies a similar procedure and derives another classification loss L'
Pairwise loss. In addition to the classification losses, FPTrans further employs a pairwise loss to pull close all the foreground features from the query and support samples, as well to push the foreground and background features far away from each other. The pairwise loss is formulated as:
paii
1
Σ
BCE(c(sim(F)) q.i. 1x))/),,11y, YG5s),,
Z
(Y,,1+Y,,)221
where Z = [Y, f + Ye.) ) 2 1/ is the normalization factor and BCE is the binary cross- cross-entropy loss. is Sigmoid layer and 1[-] is the indicator function. If two features both belong the background (Yzg. +Y,)) = 0), this pairwise loss function will NOT pull them close. Ablation studies show that pulling two foreground features close substantially improves FSS (Table while pulling two background features actually compromises the FSS accuracy, as evidenced in the supplementary Section E.2.
Overall, FPTrans sums all the three losses for training:
L = + f.ee + Xheairs
where À is hyperparameter. For inference, FPTrans simply uses the query feature and the feature- based proxies for the final prediction (Eqn. (7))
4 Experiments
4.1
Implementation Details
Datasets and Metries. We use two popular FSS benchmarks PASCAL-5* 41 and COCO-20* 46
for evaluation. PASCAL-5' combines PASCAL. VOC 2012 10] and SBD [15,, and includes 20 classes. Following prior works 26). we split the dataset into four splits with each split 15 classes
for training and 5 classes for testing. COCO-20* is constructed with COCO 2014 [30] and includes 80 classes. It is divided into 4 splits with each split 60 classes for training and 20 classes for testing. To compare with previous methods, we report mean IoU (mIoU) averaged on test classes 46 64
Training Details. All the images are resized and cropped to 480 480 and augmented follow- ing [46). We evaluate the proposed method on two vision transformer backbones, ViT-B/16 (9) and D/16 47 These two backbones are both pretrained on Imagenet- (6). The cross-entropy
7

---

Table 2: Comparison with state-of-the-art methods on COCO-20', We report I-shot and 5-shot results using the mean IoU (%)
1-shot
5-shot
Backbone
Method
S0
S1
S2
S3
Mean
S1
S2
S3
Mean
RePRI 2
32.0
38.7
32.7
33.1
39.3
45.4
39.7
41.8
41.6
Res-50
HSNet 37
36,3
43.1
38.7
38.7
39.2
43.3
51.3
48.2
45.0
46.9
BAM [5]
43.4
50.6
47.5
43,4
46.2
49.3
54.2
51.6
49.6
51.2
DAN 49
24.4
29.6
Res-101
PFENet 46
34.3
33.0
32.3
30.1
32.4
38.5
38.6
38.2
34.3
37.4
HSNet 37
37.2
44.1
42.4
41.3
41.2
45.9
53.0
51.8
47.1
49.5
Baseline
37.3
39.6
41.5
35.3
38.4
48.2
53.5
52.9
48.8
50.8
ViT-B/16
FPTrans
39.7
44.1
44.4
39.7
42.0
49.9
56.5
55.4
53.2
53.8
Baseline
41.8
45.4
48.8
40.3
44.1
53.9
60.1
58.9
54.4
56.8
DeiT-B/16
FPTrans
44.4
48.9
50.6
44.0
54.2
62.5
61.3
57.6
58.9
Table 3: Evaluation (Mean IoU (%)) under the domain shift from COCO-20' to PASCAL-5'.
Table 4: Ablation studies. "Pair Loss", "Prompts" and "Proxies" control using (or not using) the pairwise loss, the prompts, and the multiple local background proxies, respectively.
COCO-PASCAN
Method
Backbone
1-shot
5-shot
Pair Loss Prompts
Proxies PASCAL
COCO
PFENet 46
61.1
63.4
Res-50
RePRI 2
63.2
67.7
61.8
38.4
38.8
HSNet 37
Res-101
64.1
70.3
63.9
41.5
VIT-B/16
67.6
76.9
v
64.0
40.3
FPTrans
DeiT-B/16
69.7
79.3
64.7
42.0
losses optimized with boundary-enhanced weight maps introduced by (65 We use the SGD optimizer with momentum of 0.9, weight decay of 5e-5, and constant learning rate of le-3. Using A100 GPUs, we train 60 epochs with ViT and 30 epochs with DeiT backbone, using a batch size 4 for PASCAL-5' and 16 for COCO-20* (batch size 8 in 5-shot due to the memory limitation). When we generate the local background prompts (and the crature-based proxies). the background of each support image is partitioned into 5 local parts, i.e., S = 5. Each prompt consists of 12 tokens, i.e. 12. The weight factor for balancing the classification loss and pairwise loss (Eqn. ( set as 2e-2 for PASCAL-5' and le-4 for COCO-20', Our baseline is implemented as the plain vision transformer.
4.2 Comparison with State-of-the-Art Methods
Main results. Table evaluates the proposed FPTrans PASCAL-5', from which draw two observations as follows:
First, FPTrans consistently improves the baseline on two backbones. For example, when using the ViT-B/16 as its baseline, FPTrans surpasses the bascline by +2.9% and +2.2% mloU on 1-shot and 5-shot settings, respectively. We note that compared with the baseline. FPTrans has three major differences, i.e., query-support interactions to better utilize the limited support samples, multiple local background proxies to promote novel-class generalization, and an additional pairwise loss function to pull close foreground features. Ablation studies Table confirm these advantages as the main reasons that improve the baseline.
Second, comparing FPTraas against state-of-the methods, we find that FPTrans achieves competi- tive FSS accuracy. Under the 1-shot setting, FPTrans on DeiT-B/16 surpasses the most competitive BAM 26 by 1. 1.0% mloU. Under the 5-shot setting, the superiority of FPTrans is even larger, iee. +2.8% based on VIT-B/16 and +7.1% based on DeiT-B/16.

---

Table 5: Performance (Mean IoU (%)) of pre- vious methods with transformer backbones. Experiments are conducted on PASCAL-5'.
Table 6: Results (Mean IoU (%)) of the different number of background proxies S with the proposed methods (Ours) or mixture model (Mix.) [66]
Method
Res-50
Res-101
ViT
DeiT
5
PFENet [46
60.8
60.1
58.7
57.7
Ours
63.9
64.4
64.7
64.4
64.0
PASCAL-S'
CyCTR 64
64.2
64.3
60.1
61.0
Mix.
64.2
64.1
Baseline
61.8
64,9
Ours
41.5
41.9
42.0
41.7
41.2
COCO-20'
FPTrans
64.7
68.8
Mix.
40.9
40.6
Table 2 summarizes the results on COCO-20', The major observations are consistent as on PASCAL- 5'. FPTrans on ViT-B/16 and DeiT-B/16 both surpass the prior state of the art by clear margin, setting a new state of the art.
Domain shift scenario. Few-shot learning has been studied under a domain shift scenario [2] Therefore, we also evaluate the proposed FPTrans on the domain shift scenario for semantic seg- mentation, where the base training data and the testing data have significant domain gap. We use COCO-20* for training and use PASCAL-5* for testing, following previous works [2 37 for comparison. The training classes (in COCO-20') and the novel testing classes (in PASCAL-51 are not overlapped. We summarize the average results on 4 COCO-tran trained models in Table FPTrans outperforms HSNet 337 by +5.6% and +9.0% on the 1-shot and 5-shot settings, respectively. confirms the effectiveness of FPTrans under the domain shift scenario.
4.3 Ablation Study
To better understand the proposed methods, we conduct ablation studies on FPTrans components. Experiments are conducted with ViT-B/16 backbone on the 1-shot setting unless specified otherwise.
Ablations on some major components. We recall that FPTrans has multiple keypoints / important designs, i.e., an additional pairwise loss for training. novel prompting strategy, and multiple local proxies. The corresponding ablations are shown in Table 4 from which we draw three observations. First, the pairwise loss improves the baseline by +1.1% % gains on PASCAL-5 and +0.4% on COCO- 20'. Second, adding prompts further brings +1.0% and +2.7% gains while adding multiple background proxies further brings 1.11% % and +1.5% gains on PASCAL-5 and COCO-20* respectively. Third, compared with the baseline, the full FPTrans equipped with all the three components achieves overall improvements of +2.9% and 3.6% on PASCAL-5' and COCO-20', respectively.
Transformer backbone for becoder-based method. Based on two competitive decoder-based methods PFENet [46] and CyCTR [64], we replace their CNN backbones with the ViT-B/16 and DeiT-B/16 backbones, as shown in Table We observe that these two methods undergo considerable accuracy decreases after the backbone replacement. It suggests that replacing the CNN backbone with a transformer does not necessarily improve FSS, although the transformer backhone fits our plain FSS framework.
Investigation of the local background proxies. We investigate the local back- ground proxies by varying their numbers $ Table 6 It observed that using 5 back- ground proxies achieves the highest accu- racy on both PASCAL-5' and COCO-20" for our method. Moreover, we compare our method with another multi-proxy method ("Mix.") proposed in RPMM [56). RPMM online trains mixture models to generate multiple background proxies. In contrast, our approach for achieving multiple back- ground is relatively simple (by partitioning
Support
Query
Map
Map
Map
Map 3
Figure 3: Visualization of predicted score maps. Each background proxy focus on the partial background.

---

the background) and superior (e.g., +0.6% gains on PASCAL-5 and +1.4% gains on COCO-20* when S We infer that our partition- based background proxies are more representative because considers a realistic factor, i.e, the far-away regions of the background are likely to be inhomogeneous to each other. Fig- S visualizes the activation maps of different proxies (one foreground proxy and three local background proxies). It confirms that the local background proxies well accommodate the inhomogeneous background.
5
Conclusion
This paper revives the plain framework "feature extractor + linear classification head" and cor- respondingly proposes a novel Feature-Prox Transformer (FPTrans) for few-shot segmentation. During feature extraction, FPTrans makes the query interact with support features in all the trans- former blocks, therefore well utilizing the limited support samples. During proxy extraction, FPTrans encodes the complex background into multiple local background proxies, therefore improving the generalization towards novel classes. Given the discriminative features and the representative proxies, FPTrans directly uses linear classification head compare their cosine similarity and achieves state-of- that-rr performance. With its simplicity and competitive accuracy, we hope FPTrans can serve strong baseline for few-shot segmentation.
Limitations and future works. Currently, FPTrans relies on a pretrained model for generating prompts, which consumes extra computation resources. However, find that FPTrans is robust to the model for extracting the prompts (see supplementary Section for details) to some extent. Therefore, will seek more lightweight prompt-generating models in future works.
Acknowledgements
The work of the authors was supported by the National Natural Science Foundation of China (No. 62132017), and the Fundamental Research Funds for the Central Universities (No. 226-2022-00087)
References
Franz Aurenhammer. Voronai diagrams- survey of fundamental geometric data structure. ACM Comput. Surv., 23(3):345-40) 405, September 1991.
Malik Boudiaf, Hoel Kervadee, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-Shot Segmentation Without Meta-L.caming: A Good Transductive Inference Is All You Need? In CVPR, pages 13979-13988, 2021.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Noss,, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler. Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 2020.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner. Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei, Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 2020.
[5] Sergi Caelles, Kokitsi Maninis, Jordi Pont- Tuset, Laura Leal-Taixe, Daniel Cremers, Luc Van Gool, One-Shot Video Object Segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 221 230, 2017.
[6] J. Deng, W. Dong, R. Socher, L. Li, Kai Li. and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248 255, June 2009.
10

---

[7] Jacob Devlin, Ming-Wo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume 1 (Long and Short Papers), pages 4171-41866 Association for Computational Linguistics, Junc 2019.
[8] Nanqing Dong and Eric Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 1, page 6, 2018.
[9] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010. 1929 [cx], June 2021.
[10] Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results.
[11] Siddhartha Gairola, Mayur Hemani, Balaji Krishnamurthy, and Ayush Chopra. SimPropNet: Improved Similarity Propagation for Few-shot Image Segmentation. In IJCAI, volume 1, pages 573-579, July 2020.
[12] Tianyu Gao, Adam Fisch, and Danqi Chea. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Papers), pages 3816 -8300, Online, August 2021. Association for Computational Linguistics.
[13] Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain Adaptation via Prompt Learning. arXiv:2202.066 06687 [cs], February 2022.
[14] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlic Huang. PPT: Pre-trained Prompt Tuning for Few-shot Learning. arXiv:2109,043: [cs]. March 2022.
[15] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous Detection and Segmentation. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, Computer Vision -ECCV 2014, Lecture Notes in Computer Science, pages 297-312, Cham, 2014. Springer International Publishing.
[16] Junxian He, Chunting Zhou, Xuezhe Ma. Taylor Berg- Kirkpatrick, and Graham Neubig. Towards Unified View of Parameter-Eiffiient Transfer Learning. arXiv:21 10.04366 [cs], Fehruary 2022.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[18] Olivier J. Hénaff, Aravind Srinivas, Jeffree De Fauw, Ali Razavi, Carl Dnersch, M. Ali Eslami, and Aaron van den Oord. [cs), July 2020.
Image Recognition with Contrastive Predictive Coding. arXiv: arXiv:1905.092 09272
[19] Sunghwan Hong, Seokju Cho, Jisu Nam, and Seungryong Kim. Cost Aggregation Is All You Need for Shot Segmentation. 1..i:22112.11683 [es]. December 2021.
[20] Neil Houlshy, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Ffffiieet Transfer I Learning for NL.P. In Proceedings of the 36th International Conference on Machine Learning, pages 2790-2799. PMLR, May 2019.
[21] Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, and Cees G. M. Snoek. Attention- -Based Multi- Context Guiding for Few-Shot Semantic Segmentation. AAAI, 33:8441 044844118448,, July 2019.
[22] Zhengdong Hu, Yifan Sun, and Yi Yang. Switch to generalize: Domain-switch learning for cross-domain few-shot classification. In International Conference on Learning Representations, 2022.
[23] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ner-Nam Lim. Visual Prompt Tuning. arXiv:2203.121 12119 [cs]. March 2022.
[24] Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. How Can We Know What Language Models Know? Transactions of the Association for Computational Linguistics, 8:423-438, July 2020.
[25] Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal. UniT: Unified Knowledge Transfer for Any-Shot Object Detection and Segmentation. In CVPR, pages 3951-5961. 2021.

---

[26] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning What Not to Segment: A New Perspective on Few-Shot Segmentation. CVPR, March 2022.
[27] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive Prototype Learning and Allocation for Few-Shot Segmentation. In CVPR, pages 8334 8343, 2021.
[28] Xiang Li, Tianhan Wei, Yau Pun Chen, Yu-Wing Tai, and Chi-Keung Tang, FSS-1000: A 1000-Class Dataset for Few-Shov Segmentation. CVPR, April 2020.
[29] Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. arXiv:2101.0019 [cx], January 2021.
[30] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects Context. In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelarrs. editors, Computer Vision - ECCV 2014, Lecture Notes in Computer Science, pages 740-755, Cham, 2014. Springer International Publishing.
[31] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. 107. [cs], July 2021.
[32] Weide Liu, Chi Zhang, Guosheng Lin, and Fayao Liu. CRNet: Cross-Reference Networks for Few-Shot Segmentation. In CVPR, pages 4165-4173, 2020.
[33] Vijing Liu, Qinxian Liu, Jian-Wei Zhang, Haozhe Feng, Zhongwei Wang. Zihan Zhou, and Wei Chen. Multivariate Time-Series Forecasting with Temporal Polynomial Graph Neural Networks. In NeurlPS, 2022.
[34] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-Aware Prototype Network for Few-Shot Semantic Segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision ECCV 2020, Lecture Notes in Computer Science, pages 142 158, Cham, 2020. Springer International Publishing.
[35] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hicrarchical Vision Transformer Using Shifted Windows. CVPR, pages 10012-10022, 2021.
[36] Zhihe La, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler Is Better: Few-Shot Semantic Segmentation With Classifier Weight Transformer. ICCV, pages 8741-8750, 2021.
[37] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation Squeeze Few-Shot Segmentation. In ICCV, pages 194126952, 2021.
[38] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[39] Ayyappa Kumar Pamhala, Titir Dutta, and Soma Biswas. SML:: Semantic Meta-leaming for Few-shot Semantic Segmentation. (cXiv:2009.0669) [cs], Septemher 2020.
[40] Kate Rakelly, Evan Shelhamer, Trevor Darrell, Alyosha Etros, and Sergey Levine. Conditional networks for few-shot semantic segmentation. ICLR 2018 Workshop, 2018
[41] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. BMVC, 2017.
[42] Mennatullah Siam, Boris N. Oreshkin, and Martin Jagersand. AMP: Adaptive masked proxies for few-shot segmentation. In The IEEE International Conference on Computer Vision (ICCV), October 2019.
[43] Karen Simonyan and Andrew Zisserman. Very Deep Convolutional Networks for Large-Scale Image Recognition. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[44] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical Networks Few-shot Learning, In Guyon, U. V. Luxburg. S. Bengio. H. Wallach, R. Fergus. S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 4077-4087. Curran Associates, Inc., 2017.
[45] Pinzhuo Tian, Zhangkai Wu, Lei Qi, Lei Wang. Yinghuan Shi, and Yang Gao. Differentiable Meta-learning Model for Few-shot Semantic Segmentation. Proceedings of the AAAI Conference on Artificial Intelligence. 34(7):12087 - November 2019.
12

---

[46] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior Guided Feature Enrichment Network for Few-Shot Segmentation. IEEE Trans. Pattern Anal. Mach. Intell., pages 1-1, 2020.
[47] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers distillation through attention. In Proceedings of the 38th International Conference on Machine Learning, pages 10347-10357. PMLR, July 2021.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NewrIPS, pages 5998-6998999 6008, 2017.
[49] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang. Xianbin Cao, and Xiantong Zhen. Few-Shot Semantic Segmentation with Democratic Attention Networks. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, ECCV, Lecture Notes in Computer Science, pages 730-746, Cham, 2020. Springer International Publishing.
[50] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. PANet: semantic segmentation with prototype alignment. In The IEFE International Conference on Computer Vision (ICCV), October 2019.
Few-shot image
[51] Zifeng Wang, Zizhao Zhang, Yuen-Yu Lee, Han Zhang, Ruexi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to Prompt for Continual Learning. CVPR, March 2022.
[52] Zhonghua Wa, Xiangxi Shi, Guosheng Lin, and Jianfei Cai. Learning Meta-Class Memory for Few-Shot Semantic Segmentation. In ICCV, pages 517-526, 2021.
[53] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M. Alvarez, and Ping Luo. Seg- Former: Simple and Efficient Design for Semantic Segmentation with Transformers. NeurlPS 2021 arXiv:2105.1520. [cx], June 2021.
[54] Guo-Sen Xie, Jie Liu, Huan Xiong, and Ling Shao. Scale-Aware Graph Neural Network for Few-Shot Semantic Segmentation. CVPR, pages 5475 5484, 2021.
[55] Guo-Sen Xie, Huan Xiong, Jie Liu, Yazhou Yao, and Ling Shao. Few-Shot Semantic Segmentation With Cyclic Memory Network. In ICCV, pages 7293-7302 2021.
[56] Boyu Yang. Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype Mixture Models for SeotShot Semantic Segmentation. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan- MachMec Frahm, editors, ECCV, volume 12353, pages 763-778. Springer International Publishing, Cham, 2020.
[57] Lihe Yang, Wei Zhuo, Lei Qi. Yinghuan Shi, and Yang Gao. Mining Latent Classes for Few-Shot Segmentation. In ICCV, pages 8721-8730, 2021.
[58] Xianghui Yang, Bairun Wang, Kaige Chen, Xinchi Zhou, Shuai Yi, Wanli Ouyang, and Luping Zhou. BriNet: Towards Bridging the Intra-class and Inter-class Gaps in One-Shot Segmentation. arXiv:2008.00 06226 [cx], August 2020.
[59] Yuwei Yang, Fanman Meng, Hongliang Li, Qingbo Wu, Xiaolong Xu, and Shuai Chen. New Local Transformation Madule for Few-shot Segmentation. arXiv: 1910.05866 [cs], November 2019.
[60] Zongxin Yang, Yunchao Wei, and Yi Yang. Associating Objects with Transformers for Video Object Segmentation. In NewrlPS, 2021.
[61] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self- Self-Guided and Cross-Guided Learning for Few-Shot Segmentation. In CVPR, pages 8312-8321, 2021.
[62] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based based one-shot semantic segmentation. In The IEEE International Conference on Computer Vision (ICCV). October 2019.
[63] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. CANet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[64] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. NerulPS, 34, 2021.
[65] Jian- Wei Zhang, Lei Lv, Yawei Luo, Hao-Zhe Feng. Yi Yang, and Wei Chen. Prior-Enhann Enhanced Few-Shot Segmentation with Meta-Prototypes arXiv:2106.005: 00572 [cs], June 2021.
13

---

[66] X. Zhang. Y. Wei, Y. Yang. and T. S. Huang. SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation. IEEE Trunsactions Cybernetics, 5019):385-3365, September 2020.
[67] Yinan Zhao, Brian Price, Scott Cohen, and Danna Gurari. Objectness-Aware One-Shot Semantic Segmen- tation. 20X::2004.02245 [es]. April 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5 and supplementary.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section[5]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplementary.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 4.1 and Section 4.3
(c) Did you report error bars (e-g., with respect the random seed after running experi- ments multiple times)? [Yes] See the supplementary.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work existing assets, did you cite the creators? [Yes] See Section 4.1
(b) Did you mention the license of the assets? [Yes] See the supplementary.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content? [Yes] See the supplementary.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "_KvzerFQR4A", "reviewer": "Reviewer_Qm9S", "summary": "The paper proposes a different approach to FSS, whereby instead of independently processing and matching support and query samples, the support samples are processed to generate prompts, which are then concatenated with both query and support tokens. Importantly these prompts act as message passers between the support and query feature extractors. Finally, proxies are generated and the query tokens classified. The authors claim that this approach has multiple benefits,\n1) Architecture is simplified, compared with decoder architectures\n2) Performance is on par with or greater than previous approaches", "questions": "* Is there a particular reason that you chose not to use a decoder? In my experience, using a simple feature + conditioning based decoder on the predictions can increase segmentation accuracy by quite a large amount. I do not agree that refraining from any upsampling is a simplification of the method, but I'm open to hear your reasoning.\n", "limitations": "The authors addressed the issue of having to use a separate network for the prompt generation.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Major Strengths\n* Novelty, specifically the prompting approach and message passing for conditioning has to my knowledge not previously been done in FSS. I think the approach is interesting and may yield further insights.\n\n## Minor Strengths\n* Paper is clearly written and easy to follow.\n* Extensive experiments, in particular ablations.\n\n## Major Weaknesses\n* It is not easy to judge whether the improvements come from the proposed method, as the backbones have been changed. The authors have provided reimplementations of previous methods using the ViT backbone, however one has to assume that such a change may require finetuning (e.g. learning rates) for those previous methods and hence the comparison is not simple to make. Judging from vision transformers success in other related tasks, such as segmentation and detection, I believe there is a large risk that most (if not all) gains can be attributed to the change of backbone.\n\n## Minor Weaknesses\n* While the authors argue that their approach is simpler than previous decoder-based methods, I do not personally agree. To me, the architecture seems quite involved, in particular the need for a first proxy generation step, separate from the rest.\n* Code not provided, however, promised to be released.\n* Performance gains relative to recent work are not significant, in particular, compared to BAM.\n\n## Conclusions\nWhile there are potential issues with where the performance gains come from, the authors also do not claim this as their main contribution. Instead, it is to be viewed as an alternate way to perform FSS. I think the paper is interesting enough to warrant acceptance, despite my concerns. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bxb7krUJkcQ", "reviewer": "Reviewer_d8mB", "summary": "Unlike mainstream methods that involve intricate decoder structure, this paper proposes to revive the plain framework of feature extractor + linear classification head for few-shot segmentation (FSS). Concretely, this paper designs a framework called Feature-proxy Transformer (FPTrans), where it utilizes proxy, a vector to represent the foreground as well as the background classes. It makes queries interact with support features and use multiple background proxies through a novel prompting strategy. Extensive experiment results verify that the proposed method achieves competitive performance, which shows that it can serve as a simple yet strong baseline for FSS. ", "questions": "My questions and suggestions are listed in weaknesses. My main concern lies in the experiment results. ", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "##### Strengths #####\n1) This paper is well-organized and easy to read. Figure 2 is very clear to illustrate the whole framework. \n\n2) This paper incorporates prompt learning to few-shot segmentation in a clever way. The framework provides dedicated modifications to make it more suitable for FSS, for instance, multiple background prompts. \n\n3) The visualization results in the paper are good. \n\n##### Weaknesses #####\nI have some questions about the experiment results. \n1) Network backbone. The comparison is conducted with different network backbones, i.e., the previous methods take ResNet-50 / 101, while the proposed method takes ViT / DeiT. I notice that the author compares PFENet and CyCTR on ViT. However, the experiments on DeiT are not included. And the comparsions on more SOTA methods, e.g., BAM, HSNet are also omitted. In addition, I think it is better to add the parameter size and FLOPS of different backbones for a clearer comparison. \n2) 1-shot performance. I notice that the proposed FPTrans shows inferior performance than BAM when taking ViT as the backbone. Can the authors have more discussions or explanations about this phenomenon? In addition, I am curious about the performance of BAM with ResNet-101, which is also an important baseline. \n\nMinor issues:\nThe text in the figures can be larger to make it clearer. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4evA427VNxj", "reviewer": "Reviewer_Ro1k", "summary": "This paper proposes the Feature-Proxy Transformer for few-shot segmentation, which utilizes the ViT as the backbone and interacts the query features with support features through prompts. Besides, this paper introduces to obtain multiple background proxies via Voronoi-based method. Experiments are conducted on Pascal-$5^i$ and COCO-$20^i$ datasets. ", "questions": "In the rebuttal, I would like to see more discussion about the main contribution of this work and the analysis of the prompts. From my perspective, the Pair Loss and Multiple Proxies are minor contributions that are effective but lack novelty. The Prompts for FSS is novel, but the absolute gain (considering the computational cost) is relatively small. It would be better to include the mentioned ablation study, and also the quantitative computational cost comparison (*eg.*, inference time, flops et. al.) in the rebuttal.", "limitations": "Limitations are disscussed in the paper. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n**Strengths**\n1. This paper is easy to follow and may be easy to implement.\n2. This paper demonstrates that the framework with ViT backbone could achieve decent performance on both benchmarks.\n3. The proposed operations are effective according to the experiments.\n4. Interacting query and support features through prompts is interesting.\n5. It shows an excellent transferable performance (from COCO to Pascal).\n\n**Weaknesses**  \nIt is hard to tell the main contribution of this work.  \nFirst, the idea of extracting multiple proxies (prototypes) from support features is not novel since it has been discussed in [30,52]. The only difference is that this paper applies the Voronoi-based method instead of the EM algorithm [30,52]. \n\nBesides, although this paper claims that the FPTrans is a plain framework, the prompt generation process is too heavy. It requires to use the entire ViT backbone to produce the prompts, which largely increases computational cost. Specifically, this additionally adds an approximate 50% computational overhead compared with the baseline, which may be unacceptable since the ViT-Base backbone is already a much larger model compared with Resnet-50/101, but the overall framework can only obtain comparable results with previous approaches with ResNet-50 backbone. Besides, this paper misses the ablation of only using the learnable prompts.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept