Few-Shot Segmentation via Cycle-Consistent Transformer
Authors: Gengwei Zhang
Guoliang Kang
Yi Yang
Yunchao Wei
Venue: NeurIPS 2021 Poster (2021)
Keywords: Few-Shot Segmentation
Few-Shot Learning
Transformer

Abstract: Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as the conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and target images to facilitate the few-shot semantic segmentation task. We design a novel Cycle-Consistent Transformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-5^i and COCO-20^i datasets, we achieve 66.6% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art by 4.6% and 7.1% respectively.

Few-Shot Segmentation via Cycle-Consistent Transformer
Gengwei Zhang!?2, Guoliang Kang", Yi Yang", Yunchao We
Baidu Research
2 ReLER, Centre for Artificial Intelligence, University of Technology Sydney
University of Texas, Austin
CCAL, College of Computer Science and Technology, Zhejiang University 5 Institute of Information Science, Beijing Jiaotong University
Beijing Key Laboratory of Advanced Information Science and Network (zgwdavid, kgl.prml,, wychao 1987, yee.i.yang) @ @gmail.com
Abstract
Few-shot segmentation aims train a segmentation model that can fast adapt novel classes with few exemplars. The conventional training paradigm is to learn make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as the conditional information. These methods cannot utilize all pixel-wiss wiiee support information for the query predictions, which is however critical for the segmentation task. this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Sycle-Consisteen TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared previous state-of-the-art methods. Specifically, on Pascal-5' and COCO-20 datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-an method by 5.6% and 7.1 % respectively.
Introduction
Recent years have witnessed great progress in semantic segmentation [19, 4, 47]. The success can be largely attributed to large amounts of annotated data [48, 17]. However, labeling dense segmentation masks are very time-consuming [45]. Semi-supervised segmentation [15, 39, 38] has been broadly explored to alleviate this problem, which assumes a large amount of unlabeled data is accessible. However, semi-supervised approaches may fail to generalize to novel classes with very few exemplars. In the extreme low data regime, few-shot segmentation [26, 35] is introduced train a segmentation model that can quickly adapt to novel categories.
" Part of this work was done when Gengwei Zhang was an intern at Baidu Research. Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Translame Enecda
Goople
Figure 1: Different learning frameworks for few-shot segmentation, from the perspective of ways to utilize support information. (a) Class-wise mean pooling based method. (b) Clustering based method. (c) Foreground pixel attention method. (d) Our Cycle-Consistent TRansformer (CyCTR) framework that enables all beneficial support pixel-level features (foreground and background) to be considered.
Most few-shot segmentation methods follow a learning-to-leam paradigm where predictions of query images are made conditioned on the features and annotations of support images. The key to the success of this training paradigm lies in how to effectively utilize the information provided by support images. Previous approaches extract semantic-level prototypes from support features and follow a metric learning [29, 7, 35] pipeline extending from PrototypicalNet [28]. According to the granularity of utilizing support features, these methods can be categorized into two groups, as illustrated in Figure 1: 1) Class wise mean pooling [35, 46, 44] (Figure (a)).. Support features within regions of different categories are averaged to serve as prototypes to facilitate the classification of query pixels. 2) Clustering [18, 41] (Figure 1(b)). Recent works attempt to generate multiple prototypes via EM algorithm or K-means clustering [41, 18], in order to extract more abundant information from support images. These prototype- based methods need to "compress" support information into different prototypes (i.e. class-wise wise or cluster-wise) which may lead to various degrees of loss of beneficial support information and thus harm segmentation query image. Rather than using prototypes to abstract the support information, [43, 34] (Figure I(c) propose to employ the attention mechanism extract information from support foreground pixels for segmenting query. However, such methods ignore all the background support pixels that can be beneficial for segmenting query image, and incorrectly consider partial foreground support pixels that quite different from the query ones, leading to sub-optimal results.
In this paper, we focus equipping each query pixel with relevant information from support im- ages to facilitate the query pixel classification. Inspired by the transformer architecture [32] which performs feature aggregation through at- tention, we design a novel Cycle-Consistee Transformer (CyCTR) module (Figure (d)) to aggregate pixel-wise support features into query ones. Specifically, our CyCTR consists of two types of transformer blocks: the self-alignment block and the cross-alignment block. The self- alignment block is employed to encode the query image features by aggregating its relevant con- text information, while the aross-alignment aims to aggregate the pixel-wise features of support images into the pixel-wise features of query image. Different from self-alignment where Query", Key and Value come from the same embedding, cross-alignment takes features from query images as Query, and those from support images as Key and Value. In this way, CyCTR provides abundant pixel-wise support information for pixel-wise features of query images to make predictions.
support image Foreground points
query image
Background points
Figure 2:
The motivation of our proposed method. Many pixel-level support features are quite differ- ent from the query ones, and thus may confuse the attention. We incorporate cycle-consistency into attention to filter such confusing support features. Note that the confusing support features may come from foreground and background.
Moreover, we observe that due to the differences between support and query images, e.g., scale, color and scene, only a small proportion of support pixels can be beneficial for the segmentation of query image. In other words, in the support image, some pixel-level information may confuse the attention in the transformer. Figure provides a visual example of a support-query pair together with the label
TTo distinguish from the phrase "query" in few-shot segmentation, we use "Query" with capitalization to note the query sequence in the transformer.

---

masks. The confusing support pixels may come from both foreground pixels and background pixels. For instance. point p1 in the support image located in the plane afar, which is indicated as foreground by the support mask. However, the nearest point p2 in the query image (i.e. p2 has the largest feature similarity with pı) belongs to a different category, i.e. background. That means, there exists no query pixel which has both high similarity and the same semantic label with p1- Thus, P1 is likely to be harmful for segmenting "plane" and should be ignored when performing the attention. To overcome this issue, in CyCTR. we propose to equip the wross-alignment block with a novel cycle-consistent attention operation. Specifically, as shown in Figure 2, starting from the feature of one support pixel, we find its nearest neighbor in the query features. In turn, this nearest neighbor finds the most similar support feature. If the starting and the end support features come from the same category, a vecle-consisteeyy relationship is established. We incorporate such an operation into attention to force query features only attend to cycle-consistent support features to extract information. In this way, the support pixels that are far away from query ones are not considered. Meanwhile, cycle-consistent attention enables us to more safely utilize the information from background support pixels, without introducing much bias into the query features.
In a nutshell, our contributions are summarized as follows: (1) We tackle few-shot segmentation from the perspective of providing each query pixel with relevant information from support images through pixel-wise alignment. (2) We propose a novel Cycle- Consistent TRansformer (CyCTR) to aggregate the pixel-wise support features into the query ones. In CyCTR, we observe that many support features may confuse the attention and bias pixel-level feature aggregation, and propose incorporating cycle-consistent operation into the attention to deal with this issue. (3) Our CyCTR achieves state-of- the-art results on two few-shot segmentation benchmarks, i.e., Pascal-5' and COCO-20'. Extensive experiments validate the effectiveness of each component in our CyCTR.
2 Related Work
2.1 Few-Shot Segmentation
Few-shot segmentation [26] is established to perform segmentation with very few exemplars. Recent approaches formulate few-shot segmentation from the view of metric learning [29, 7, 35]. For instance, [7] first extends PrototypicalNet [28] to perform few-shot segmentation. PANet [35] simplifies the framework with an efficient prototype learning framework. SG-One [46] leverage the cosine similarity map between the single support prototype and query features to guide the prediction. CANet [44] replaces the cosine similarity with an additive aligmment module and iteratively refines the network output. PFENet [30] further designs an effective feature pyramid module and leverages a prior map to achieve better segmentation performance. Recently, [41, 18, 43] point out that only single support prototype is insufficient represent given category. Therefore, they attempt to obtain multiple prototypes via EM algorithm to represent the support objects and the prototypes are compared with query image based on cosine similarity [18, 41]. Besides, [43, 34] attempt to use graph attention networks [33, 40] to utilize all foreground support pixel features. However, they ignore all pixels the background region by default. Besides, due the large difference between support and query images, not all support pixels will benefit final query segmentation. Recently, some concurrent works propose learn dense matching through Hypercorrelation Squeeze Networks [22] or mining latent classes [42] from the background region. Our work aims at mining information from the whole support image, but exploring to use the transformer architecture and from a different perspective, i.e., reducing the noise in the support pixel-level features.
2.2 Transformer
Transformer and self-attention were firstly introduced in the fields of machine translation and natural language processing [6, 32], and are receiving increasing interests recently in the computer vision area. Previous works utilize self-attention as additional module on top of existing convolutional networks, e.g., Nonlocal [36] and CCNet [14]. ViT [8] and its following work [31] demonstrate the pure transformer architecture can achieve sfate-of-the-arr for image recognition. On the other hand, DETR [3] builds up an end-to-end framework with transformer encoder-decoder on top of backbone networks for object detection. And its deformable vairents [51] improves the performance and training efficiency. Besides, in natural language processing, few works [2, 5, 27] have been
3

---

introduced for long documents processing with sparse transformers. In these works, each Query token only attends to a pre-defined subset of Key positions.
2.3 Cycle-consistency Learning
Our work is partially inspired by cycle-consistency learning [50, 9] that is explored in various computer vision areas. For instance, in image translation, CycleGAN [50] uses cycle-consistency to align image pairs. It is also effective in learning 3D corresoondence [49], consistency between video frames [37] and association between different domains [16]. These works typically constructs cycle-consisteccy loss between aligned targets (e-g. images). However, the simple training loss cannot be directly applied to few-shot segmentation because the test categories are unseen from the training process and no finetuning is involved during testing. In this work, we incorporate the idea of cycle-consistency into transformer eliminate the negative effect of confusing or irrelevant support pixels.
3 Methodology
3.1 Problem Setting
Few-shot segmentation aims at training segmentation model that can segment novel objects with very few annotated samples. Specifically, given dataset Dtrain and Drest with category set Ctrain and Ctest respectively, where Crrain n Ctest 0. the model trained on Dtrain is directly used to test on Dtest- In line with previous works [30, 35, 44], episode training is adopted this work for few-shot segmentation. Each episode is composed of k support images I, and query image Ig to form s--hot episode {{I,}* Ig). in which all {I,]* and Ig contain objects from the same category. Then the training set and test set are represented by Dtrain = {{I.}* t 1,g)N Nirsin and Drest = [[T,)* Neeot, where Ntrain and Ntett is the number of episodes for training and test set. During training, both support masks M, and query masks Mg are available for training images,
and only support masks are accessible during testing.
3.2 Revisiting of Transformer
Following the general form in [32], transformer block is composed of alternating layers of multi-head attention (MHA) and multi-layer perceptron (MLP). LayerNorm (LN) [1] and residual connection [12] are applied at the end of each block. Specially, an attention layer is formulated as
QKT
AQten(Q, K,K) V) softmax
Vd VV,
where [Q [W, Zw W Z...e.. in which Z, is the input Query sequence, Zeo is the input Key/Value sequence, WG, Ws, W, E Rddd denote the learnable parameters, d is the hidden dimension of the input sequences and we assume all sequences have the same dimension d by default. For each Query element, the attention layer computes its similarities with all Key elements. Then the computed similarities are normalized via softmax, which are used to multiply the Value elements to achieve the aggregated outputs. When Zy Zo.. it functions as self-attention mechanism.
The attention layer is an extention of attention layer, which performs h attention operations and concatenates consequences together. Specifically,
MHA(Q,K,V) [head,, ... headh))
where headm = (Q, K, V] with dimension d/h.
(Qten(Qm. Km, Vmn) and the inputs (Qm, Km, Vmn are the mith group from
3.3 Cycle-Consistee Transformer
Our framework illustrated Figure 3(a). Generally, an encoder of our Cycle-Consistent TRans- former (CyCTR) consists of self-alignment transformer block for encoding the query features and cross-alignment transformer block to enable the query features to attend to the informative support features. The whole CyCTR module stacks L encoders.

---

Layer Tormlizatoon
MLF Nubllayer Perceptron
Weementstos Addion
MIA Murtheed Altention
supo mask
flstten & sample [N,)
Cycle-Consistent Mull-head Attention [h=1]
M.
Encoders
fltttnn & sample IN
Cross- Block
ouuput
W2
supp-
Cyc-
MHA
+
MLP
query
feat.
7
W
CyC
M2
Idoiing
CynhaConntttntty
Jagmax
query %g
Fatton TH W
X
MHA
N
MLP
argmas
Self-a ignment Block
Mr
Figure 3: Framework of our proposed Cycle-Consisteet TRansformer (CyCTR). Each encoder of CyCTR consists of two transformers blocks, i.e., the self-alignment block for utilizing global context within the query feature map and the cross-alignment block for aggregate information from support images. the cross-alignment block, we introduce the multi-head cycle-consistent attention (shown on the right, with the number of heads = for simplicity). The attention operation is guided by the cycle-consistency among query and support features.
Specifically, for the given query feature X, IB',,WW,, xd and support feature X, R.H.W.W.. xd first flatten them into ID sequences (with shape HW x d) as inputs for transformer, in which a token pixel location. The self-alignment block only takes the flattened query feature as input. As context information of each pixel has been proved beneficial for segmentation [4, 47J, we adopt the self-alignment block pixel-wise features of query image to aggregate their global context information. We don't pass support images through the self-alignment block, we mainly focus the segmentation performance of query images. Passing through the support images which don't 't coordinate with the query mask may do harm to the self-alignment on query images.
represented by the feature
Rd
In contrast, the cross-aen tross-alignment block performs attention between query and support pixel-wise features to aggregate relevant support features into query ones. It takes the flattened query feature and a subset of support feature (the sampling procedure is discussed latter) with size N, I H,W, as Key/Value sequence Z
With these two blocks, it is expected to better encoder the query features to facilitate the subsequent pixel-wise classification. When stacking L, encoders, the output of the previous encoder is fed into the self-alignment block. The outputs self-alignment block and the sampled support features are then fed into the eross-alignment block.
3.3.1 Cycle-Consistent Attention
According to the aforementioned discussion, the pure pixel-level attention may be confused by excessive irrelevant support features. To alleviate this issue, shown in Figure 3(b), a cycle- consistent attention operation is proposed. We first go through the proposed approach for 1-shot case for presentation simplicity and then discuss it in the multiple shot setting.
OKT Formally, an affinity map A = Vd spondence between all query and support pixels. Then, for an arbitrary support pixel/token (j {0, N., 1}. N₂ is the number of support pixels), its most similar query pixel/token i' is obtained by
H.,WW.N. is first calculated to measure the corre-
= argmax A
(3)
where i E (0, 1,..,, H,W, 1} denotes the spatial index of query pixels. Since the query mask is accessible, the label of query pixel i' is unknown. However, can in turn find its most similar support pixel j" in the same way:
j argmax A(r
(4)
5

---

Given the sampled support label M, RN,, cycle-consistency is satisfied if Mag) = Ma Previous work [16] attempts encourage the feature similarity between cycle-consistent pixels to improve the model's generalization ability within the same set of categories. However, in few-shot segmentation, the goal is to enable the model to fast adapt to novel categories rather than making the model fit better to training categories. Thus, we incorporate the cycle-consistency into the attention operation to encourage the cycle-consistent cross-attention. First, by traversing all support tokens, an additive bias B E IR is obtained by
O,
ilM,(s) = Ma(s')
B₁ = 00, ifMa(() # Mats*)
where j 1,.,,..NsJ, Then, for single query token Zo(t) Rd at location i, the support information is aggregated by
CyCAlten(Qi: K2, Vi) = sollmax(A(s) + B)V,
'va" where i E (0,1.... W,) and A is obtained by 2 In the forward process, B is element-wise
added with the affinity AGa for ()(() aggregate support features. In this way, the attention weight for the gycle-inconsistent support features become zero, implying that these irrelevant information will not be considered. Besides, the cycle-consistent attention implicitly encourages the consistency between the most relevant query and support pixel-wise features through backpropagation. Note that our method aims at removing support pixels with certain inconsistency, rather than ensuring all support pixels to form truth labels.
which is impossible without knowing the query ground
When performing self celf-attention in the self-alignment block, there may also exist the same issue, i.e. the query token may attend to irrelevant or even harmful features (especially when background is complex). According to our cycle-consistent attention, each query token should receive information from more consistent pixels than aggregating from all pixels. Due to the lack of query mask Mg. it is impossible to establish the cycle-consistency among query pixels/tokens. Inspired by DeformableAAteeniion [51], the consistent pixels can be obtained via learnable way as A = S(Q Coord) and A gQQ + Coord), where A TR. H, W,PPP xP is the predicted consistent pixels, in which each element E IR P in A represents the relative offset from each pixel and P represents the number of pixels to aggregate. And A' E HRH,W,,PP is the attention weights. Coord E IR H Wxxd is the positional encoding [24] to make the prediction be aware of absolute position, and f(-) and g(-) are two fully connected layers that predict the offsets' and attention weights. Therefore, the self-attention within the aligmmenttt transformer block is represented as
PredAtten V7) = Σ softmax(A ((ra)) Vr+
where r E {0, 1,.., W,), } is the index of the flattened query feature, both Q and V are obtained by multiplying the flattened query feature with the learnable parameter.
Generally speaking, the cycle-consistent transformer effectively avoids the attention being biased by irrelevant features to benefit the training of few-shot segmentation.
Mask-guided sparse sampling and K-shot Setting: Our proposed cycle-consistency transformer can be easily extended to K-shot setting where K > 1. When multiple support feature maps are provided, all support features are flattened and concatenated together as input. As the attention is performed at the pixel-level, the computation load will be high if the number of support pixels/tokens is large, which is usually the case under K -sott setting. In this work, we apply a simple mask busk-g sampling strategy to reduce the computation complexity and make our method more scalable. Concretely, given the k-shhot support sequence Z, E R&H, W, xd and the flattened support masks M, E IR RR,,W. the support pixels/tokens are obtained by uniformly sampling Ni, tokens (N,g <= N where N, < &HH W.) from the foreground regions and N, Nr, tokens from the background regions in all support images. With a proper N., the sampling operation reduces the computational complexity, and makes algorithm more scalable with the increase of spatial size of support images. Additionally, this strategy helps balance the foreground- -bckkroond ratio and also implicitly considers different sizes of various object regions in support images.
'The offiets are predicted 2d coordinates and transformed into 1d coordinates.

---

Table 1: Comparison with other state-of-the- methods for 1-shot and 5-shot segmentation on PASCAL-5' using the mloU (%) evaluation metric. Best results are shown in bold.
I-shot
5-shot
Method
Backbone
50
5
5²
5
Mean
5
5'
5²
5
Mean
PANet [35]
42.3
58.0
51.1
41.2
48.1
51.8
64.6
59.8
46.5
55.7
FWB [23]
47.0
59.6
52.6
48.3
51.9
50,9
62.9
56.5
50.1
55.1
Vgg-16
SG-One [46]
40.2
58.4
48.4
38.4
46.3
41.9
58.6
48.6
39.4
47.1
RPMM [41]
47.1
65.8
50.6
48.5
53.0
50.0
66.5
51.9
47.6
54.0
CANet [44]
52.5
65.9
51.3
51.9
55.4
55.5
67.8
51.9
53.2
57.1
PGNet [43]
56.0
66.9
50.6
50.4
56.0
57.7
68.7
52,9
54.6
58.5
RPMM [41]
Res-50
55.2
66.9
52.6
50.7
56.3
56.3
67.3
54.5
51.0
57.3
PPNet [18]
47.8
58.8
53.8
45.6
51.5
58.4
67.8
64.9
56.7
62.0
PFENet [30]
61.7
69.5
55.4
56.3
60.8
63.1
70.7
55.8
57.9
61.9
CyCTR (Ours)
Res-50
65.7
71.0
59.7
64.0
69.3
73.5
63.8
63.5
67.5
FWB [23]
51.3
64.5
56.7
52.2
56.2
54,9
67.4
62.2
55.3
59.9
DAN [34]
Res-101
54.7
68.6
57.8
51.6
58.2
57.9
69.0
60.1
54.9
60.5
PFENet [30]
60.5
69.4
54.4
55.9
60.1
62.8
70.4
54.9
57.6
61.4
CyCTR (Ours)
R01
67.2
71.1
57.6
59.0
63.7
71.0
75.0
58.5
65.0
67.4
3.4 Overall Framework
Following previous works [30, 35, 44], both query and support images are first feed into a shared backbone (e.g., ResNet [12)) which is initialized with weights pretrained from ImageNet [25] to obtain general image features. Similar to [30], middle-level query features (the concatenation of query features from the 3rd and the 4th blocks of ResNet) are processed by a convolution to reduce the hidden dimension. The high-level query features (from the 5th block) are used to generate prior map (the prior map is generated by calculating the pixel-wise similarity between query and support features, details can be found in the supplementary materials) and then are concatenated with the middle-level query features. The average masked support feature is also concatenated to provide global support information. The concatenated features are processed by a 1x1 convolution. The output query features are then fed into our proposed CyCTR encoders. The output of CyCTR encoders is fed into a classifier obtain the final segmentation results. The classifier consists of a 3 convolutional layer, ReI.U layer and a x convolutional layer. More details about our network structure can be found in the supplementary materials.
4 Experiments
4.1 Dataset and Evaluation Metric
We conduct experiments two commonly used few-shot segmentation datasets, Pascal-5' [10] (which is combined with SBD [11] dataset) and COCO-20' [17], to evaluate our method. For Pascal- 5', 20 classes are separated into splits. For each split, 15 classes are used for training and 5 classes for test. At the test time, 1,000 pairs that belong to the testing classes are sampled from the validation for evaluation. In COCO-20', we follow the data split settings in FWB [23] divide 80 classes evenly into 4 splits, 60 classes for training and test on 20 classes, and 5,000 validation pairs from the 20 classes are sampled for evaluation. Detailed data split settings can be found in the supplementary materials. Following common practice [30, 35, 46]. the mean intersection over union (mloU) is adopted the evaluation metric, which is the averaged value of IoU of all test classes. We also report the foreground-bacckrroun loU (FB-IoU) for comparison.
4.2 Implementation Details
In our experiments, the training strategies follow the same setting [30]: training for 50 epochs on COCO-20' and 200 epochs on Pascal-5'. Images are resized and cropped to 473 473 for both datasets and we use random rotation from -100 to 10 as data augmentation. Besides, we use ImageNet [25] pretrained ResNet [12] as the backbone network and its parameters (including BatchNorms) are freezed. For the parameters except those in the transformer layers, we use the initial learning rate 2.5 10-3, momentum 0.9, weight decay 1 10-1 and SGD optimizer with
7

---

Table 2: Comparison with other state-of-the methods for 1-shot and 5-shot segmentation on COCO-20' using the mloU (%) evaluation metric. Best results are shown in bold.
1-shot
5-shot
Method
Backbone
20
20'
20
20
Mean
20
20
20
20
Mean
FWB [23]
Res-101
19.9
18.0
21.0
28.9
21.2
19.1
21.5
23.9
30.1
23.7
PPNet [18]
Res-50
28.1
30.8
29.5
27.7
29.0
39.0
40.8
37.1
37.3
38.5
RPMM [41]
Res-50
29.5
36.8
29.0
27.0
30.6
33.8
42.0
33.0
33.3
35.5
PFENet [30]
Res-101
34.3
33.0
32.3
30.1
32.4
38.5
38.6
38.2
34.3
37.4
CyCTR (Ours)
Res-50
38,9
43.0
39.6
39.8
40.3
41.1
48.9
45.2
47.0
45.6
poly learning rate decay [4]. The mini batch size on each gpu is set to 4. Experiments are carried out on Tesla V100 GPUs. For Pascal-5', one model is trained on single GPU, while for COCO-20' one model is trained with 4 GPUs. We construct our baseline as follows: middle-level query features from backbone network are concatenated and merged with the global support feature and the prior map. This feature is processed by two residule blocks and input to the same classifier as our method. Dice loss [21] is used as the training objective. Besides, the middle-level query feature is averaged using the ground truth and concatenated with support feature to predict the support segmentation map, which produces an auxiliary loss for aligning features. The same settings are also used our method except that we use our cycle-consistent transformer to process features rather than the residule blocks. For the proposed cycle-consistent transformer, we set the number of sampled support tokens N, to 600 for 1-shot and 5 x 600 for 5-shot setting. The number of sampled tokens is obtained according to the averaged number of foreground pixels among Pascal-5' training set. For the self-attention block, the number of points P is set to For other syper-parameters in transformer blocks,
stated in Section 3.4, the
use = transformer encoders. We set the hidden dimension of MLP layer to 3 256 and that of input to 256. The number of heads for all attention layers is set to for Pascal-5' and I for CO'O-20'. Parameters in the transformer blocks optimized with Adam [20] optimizer following other transformer works [3, 8, 31], with learning Dropout with the probability 0.1 in all
rate 1 10-1 and weight decay 10-2 Besides, attention layers.
4.3 Comparisons with State-of-the-Art Methods
In Table 1 and Table we compare our method with other state-of-the-art few-shot segmentation approaches on Pascal-5' and COCO-20* respectively. It can be seen that our approach achieves new state-of-the-art performance on both Pascal-5'
and COCO-20* Specifically, on Pascal-5', to make fair comparisons with other methods, we report results with both ResNet-50 and ResNet- 101. Our CyCTR achieves 64.0% mloU with ResNet-50 backbone and 63.7% mloU with ResNet-101 backbone for 1-shot segmentation, significantly outperforming previous state-of- the-art results by 3.2% and 3.6%, respectively. For 5-shot segmentation, our CyCTR can even surpass state-of-the art methods by 5.6% and 6.0% mloU when using ResNet-50 and ResNet- 101 backbones respectively. For COCO-20* results in Table 2, our method also outperforms other methods by large margin due the capability of the transformer fit more complex data. Besides, Table 3 shows the comparison using FB-loU on PASCAL-5* for 1-shot and 5-shot segmentation, our method also obtains the state-of-the siite--f-thh-eee af- performance.
Table 3: Comparison with other methods using FB-IoU (%) on Pascal-5' for 1-shot and 5-shot segmentation.
FB-IoU (%)
Method
Backbone
1-shot
5-shot
A-MCG [13]
Res-101
61.2
62.2
DAN [34]
Res-101
71.9
72.3
PFENet [30]
Res-101
72.9
73.5
CyCTR (Ours)
Res-101
73.0
75.4
4.4 Ablation Studies
To provide deeper understanding of our proposed method, we show ablation studies in this section. The experiments are performed on Pascal-5* 1-shot setting with ResNet-50 as the backbone network, and results are reported in terms of mloU.
8

---

Table Ablation studies that validate the effectiveness of each component in our Cycle-Consisie TRansformer. The first result is obtained by our baseline (see Section 4.2 for details).
self-aligmment cross-alignment
CyCTR (pred) CyCTR (fg. only) CyCTR
mloU (%)
59.3
62.5
62.9
62,6
63.0
V
63.5
4.4.1 Component- Wise Ablations
We perform ablation studies regarding each component of our CyCTR in Table 4. The first line is the result of our baseline, where we use two residual blocks to merge features as stated in Section 4.2. For all ablations in Table 4, the hidden dimension The mloU results are averaged over four splits. Firstly, we only use the self-alignment block that only encodes query features. The support information in this case comes from the concatenated global support feature and the prior map used in [44]. It can already bring decent results, showing that the transformer encoder is effective for modeling context for few-shot segmentation. Then, we utilize the cross- aossmmaligmmnt block but only with the vanilla attention operation in Equation 1. The mIoU increases by 0.4%, indicating that pixel-level features from support can provide additional performance gain. By using our proposed sycle-consistie by large margin, i.e. 0.6% mIoU compared to the vanilla attention. This result demonstrates our cycle-consistent attention's ' capability to suppress possible harmful information from support. Besides, we assume some background support features may also benefit the query segmentation and therefore use the cycle-consistent transformer to aggregate pixel-level information from background support features as well. Comparing the last two lines in Table 4, we show that our way utilizing beneficial background pixel-level support information brings 0.5% mIoU improvement, validating our assumption and the effecttivnnss of our proposed cycle-consistent attention operation.
to 128 and two transformer encoders are used.
attention module, the performance can be further improved
Besides, one may be curious about whether the noise can also be removed by predicting the ag- gregation position like the way in Equation for aggregating support features to query. Therefore, we use predicted aggregation instead of the cycle-consistent attention in the cross-alignment block, as denoted by CyCTR(pred) Table 4. does benefit the few-shot segmentation by aggregating useful information from support but is 0.9% worse than the proposed cycle-consistent attention. The reason lies the dramatically changing support images under few-shot segmentation testing. The cycle-consistency is better than the learnable way as it can globally consider the varying conditional information from both query and support.
4.4.2 Effeet of Model Capacity
We stack more encoders increase the hid- den dimension of encoders to increase its capac- ity and validate the effectiveness of our CyCTR. The results with different numbers of encoders (denoted as L) or hidden dimensions (denoted as d) are shown in Table 5a and 5b. While in- within a certain range, CyCTR achieves better results. We chose L = 2 as our default choice for accuracy-efficieng trade-off.
Table Effect of varying (a) number of encoders
L and (b) hidden dimensions d. When varying L, is fixed to 128; while varying d, L is fixed to 2.
#Encoder
mloU (%)
#Dim
mloU (%)
62.4
128
63.5
creasing
63.5
256
64.0
63.7
384
63.9
(a)
(b)
4.5 Qualitative results
In Figure 4, we show some qualitative results generated by our model on Pascal-5', Our cycle- consistent attention can improve the segmentation quality by suppressing possible harmful information from support. For instance, without cycle-consistency, the model misclassifies trousers as "cow" in the first row, baby's 's hair as "cat" in the second row, and a fraction of mountain as "car" the third row, while our model rectifies these part background. However, in the first row, CyCTR still

---

Ours without Cycle-consistency
Support
Query
Ours
Figure 4: Qualitative results Pascal-5' From left to right, each column shows the examples of: Support image with mask region in red; Query image with ground truth mask region in blue; Result produced by the model without cycle-consistnccy in CyCTR; Result produced by our method.
segments part of the trousers as "cow" and the right boundary of the segmentation mask is slightly worse than the model without cycle-consistency. The reason comes from the extreme differences between query and support, í.e. the support image shows "cattle" but the query image contains milk cow. The sycle-consistency may over-suppress the positive region in support images. Solving such issue may be potential direction to investigate to improve method further.
5
Conclusion
In this paper, we design a CyCTR module to deal with the few-shot segmentation problem. Different from previous practices that either adopt semantic-level prototype(s) from support images only use foreground support features to encode query features, our CyCTR utilizes all pixel-level support features and can effectively climinate aggregating confusing and harmful support features with the proposed novel cycle-consistency attention. We conduct extensive experiments on two popular benchmarks, and our CyCTR outperforms previous Mate-of-the-art methods by significant margin. We hope this work can motivate researchers to utilize pixel-level support features to design more effective algorithms to advance the few-shot segmentation research.
References
[1] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 1rXiv:1607.06450. 2016.
[2] Iz Beltagy, Matthew E Peters, and Arman Cohan. Longformer: The long- dongmetemment transformer. arXiv preprini arXiv:2004.051500 2020.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, pages 213-229. Springer, 2020.
[4] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence. 40(4):834-848, 2017.
[5] Rewon Child, Scott Gray, Alec Radford, and Ilya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv:/904. 10509, 2019.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 04805, 2018.
10

---

[7] Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning, In BMVC, volume 3-4, 2018.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint
arXiv:2010.11929, 2020.
[9] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle consisteent learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1801-1810, 2019.
[10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[11] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision, pages 297-312. Springer, 2014.
[12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition (CVPR), pages 770-778, 2016.
[13] Tao Hu, Pengwan Yang, Chiliang Zhang, Gang Yu, Yadong Mu, and Cees GM Snoek. Attention- based multi-context guiding for few-shot semantic segmentation. In Association for the Ad- vancement of Artificial Intelligence (AAAI), volume 33, pages pp. 1441-8488, 2019.
[14] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 603-612, 2019.
[15] Zilong Huang, Xinggang Wang, Jiasi Wang. Wenyu Liu, and Jingdong Wang. Weakly- supervised semantic segmentation network with deep seeded region growing. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7014-7023, 2018.
[16] Guoliang Kang, Yunchao Wei, Yi Yang, Yueting Zhuang, and Alexander G Hauptmann. Pixel- level cycle association: A new perspective for domain adaptive semantic segmentation. In NeurlPS, 2020.
[17] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference computer vision (ECCV), 2014.
[18] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision, pages 142-158. Springer, 2020.
[19] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431-3440, 2015.
[20] Iyy Loshchilov and Frank Hutter. Decoupled weight decay regularization, arXiv preprint arXiv:1711.05000 2017.
[21] Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. V-net: Fully convolutional neural networks for volumetric medical image segmentation. In 2016 fourth international conference on 3D vision (3DV), pages 565-571 IEEE, 2016.
[22] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmenta- tion. arXiv preprint arXiv:2104.015:
2021.
[23] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 622 631,, 2019.
[24] Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In International Conference on Machine Learning, pages 4055-4064. PMLR, 2018
11

---

[25] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115())2111255)) 2015.
[26] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference (BMVC), 2018.
[27] Han Shi, Jiahui Gao, Xiaozhe Ren, Hang Xu, Xiaodan Liang, Zhenguo Li, and James T Kwok. Sparsebert: Rethinking the importance analysis in self-attention. arXiv preprint
arXiv:2102.1287,, 2021.
[28] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NeurIPS), 2017.
[29] Flood Sung, Yongxin Yang. Li Zhang. Tao Xiang, Philip HS Tomr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1199-1208, 2018.
[30] Tian, H Zhao, M Shu, Yang, R Li, and J Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[31] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv preprint arXiv:2012. 2020.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the s1 International Conference on Neural Information Processing Systems, pages 6000-6010, 2017.
[33] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv: 1710. 1710.100100 2017.
[34] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision (ECCV), 2020.
[35] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9197-9206, 2019.
[36] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 7794-7803, 2018
[37] Xiaolong Wang, Allan Jabri, and Alexei A Efros. Learning correspondence from the cycle- consistency of time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2566-2576, 2019.
[38] Yunchao Wei, Jiashi Feng, Xiaodan Liang, Ming-Ming Cheng, Yao Zhao, and Shuicheng Yan. Object region mining with adversarial erasing: A simple classification to semantic segmentation approach. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1568- 1568-157,, 1576, 2017.
[39] Yunchao Wei, Huaxin Xiao, Honghui Shi, Zequn Jie, Jiashi Feng, and Thomas S Huang. Revisiting dilated convolution: A simple approach for weakly-and supi-super seman- tic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 2268-7277,22
[40] Yangxin Wu, Gengwei Zhang, Yiming Gao, Xiajun Deng, Ke Gong, Xiaodan Liang, and Liang Lin. Bidirectional graph reasoning network for panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9080-9089 2020.
[41] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision pages 763-778. Springer, 2020.
[42] Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. Mining latent classes for few-shot segmentation. arXiv preprint a1X1v:2103.15402, 2021.
12

---

[43] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region- based one-shot semantic segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 9587-9595, 2019.
[44] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5217-5226,
2019.
[45] Shiyin Zhang, Jun Hao Liew, Yunchao Wei, Shikui Wei, and Yao Zhao. Interactive object segmentation with inside-outside guidance. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12234-12244, 2020.
[46] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions Cybernetics, 50(9):3855- 3865, 2020.
[47] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2017.
[48] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[49] Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d- -uidedddd cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 117-126, 2016.
[50] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference computer vision, pages 2223-2232, 2017.
[51] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr. Deformable transformers for end-to-end object detection. arXiv preprint 4rX1v:2010.04159. 2020.
13


Reviews:
[{"review_id": "xFX6jSdjL1", "reviewer": "Reviewer_1f9d", "summary": "The paper addresses the few-shot semantic segmentation problem with cycle-consistent transformer (CyCTR) that utilizes pixel-wise support information for query prediction. The CytCTR estimate the cross-attention between support and query images with the cross-consistent attention mechanism. The proposed method achieves the start-of-the-art performance on multiple datasets.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea of pixel-level cycle consistency shares similarities with [1], nonetheless, it is still interesting to see it is utilized in the attention map of the transformer modules.\n\n1) From reviewer's understanding, when training, the attention weights of those pixels that are not “cycle-consistent” are added with a minus infinity and can be seen as disabled as they are normalized as zero after the softmax. As I do not see any explicit measures like loss function that guide the pixels to be “consistent”, this reminds me of dropout, which randomly disables a part of the network. Therefore, it is desirable to see more training details, e.g., is the percentage of pixels that are “inconsistent” decreases during the training process and reaches a low level in the late stage of the training? Any visualizations about the attention maps that can illustrate how pixels are “consistent”?\n\n2) There may exist \"misplaced mapping\". For example, the nearest point of \"cow\" in query belongs to \"horse\" in support and then the nearest point of \"horse\" in support belongs to \"cow\" in query, which is actually inconsistent but will be viewed as \"consistent\" in this work.\n\n3) Though the part of “Mask guided sparse sampling” shown effectiveness in the results, it lacks ablation study or comparison with other methods.\n\n4) Some technical details are missing: e.g. when stacking L encoders, is the output of the previous encoder serves as the Z_kv of the next layer? Is X_s is inputted into each layer of the encoders? And some parts of the paper are not easy to follow: e.g. in Fig. 3, same symbols (Z_kv, Z_q) are used for inputs to two blocks (Cross-alignment, Self-alignment), which sounds like they were identical.\n\n5) Another concern is the parameter size. The ResNet-101 is already a big model, and the transformer is also known for its resource requirements. Authors trained this few-shot model with 4 V100s, using much more resources than previous few-shot segmentation methods.\n\n[1] Kang, Guoliang, et al. “Pixel-Level Cycle Association: A New Perspective for Domain Adaptive Semantic Segmentation.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 3569–3580.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Good for me.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "jOEJnNZZVbr", "reviewer": "Reviewer_asGa", "summary": "The paper introduces a novel module called Cycle-Consistent Transformer (CyCTR) that aggregates pixel-level support features into query features via cross-attention. Cycle consistency filters out irrelevant pixel-level support features. CyCTR achieves SOTA results on Pascal-5 and COCO-20 by an appreciable margin.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper contains novel and interesting approach that beats existing SOTA approaches for few shot segmentation on Pascal-5 and COCO-20 by an appreciable margin. Parts of the paper have clarity while some lack clarity.\n\nCyCTR consists of Self-alignment block and Cross-alignment block. In the former, Query, Key and Value come from the same embedding. In the latter, the Query vector comes from the query images while Key and Value come from the support image. While CyCTR uses pixel-level support features, some of its competing approaches use semantic level prototypical features from support image or the foreground region in support image to encode query features.\n\n### Recommendation:\nSection 3 (specifically its second half) has a lot of typos, poorly written and lacks clarity. Likewise, for the bottom paragraph on page 8. There is a stark difference between other sections. Recommend authors to review and improve clarity. Otherwise, the mathematical expressions are sound.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper discusses limitations at the end of the experiments section. It does not include any comments on societal impact, other than the response to check list at the very end of the submission.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "XmmDE3WA3lX", "reviewer": "Reviewer_AxLz", "summary": "The paper introduces a new transformer architecture designed to tackle image segmentation in the few-shot setting. The new architecture seems to make it easier for the pre-trained model to generalise to new classes and produce more accurate segmentation masks. The comparison with benchmarks shows incremental improvements on PASCAL-5i and COCO-20i in 1-shot and 5-shot setting, with two different convolutional backbones.\n\nDue to minor yet numerous issues the paper is currently below the acceptance bar, but I am willing to improve my score if these are addressed.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is novel and is targeted at improving the cross-attention mechanism so that inadequate keys are not attended to. Few-shot segmentation is an important problem in machine vision both application- and research-wise as fully labelled data is scarce and quick generalisation to new classes in general remains difficult for machine learning systems.\n\nDetails:\n1. The proposed cycle-consistent transformer, for a given query image token, attends only to those support tokens that satisfy the cycle-consistency rule: support token i and query token j are in cycle consistency if j has the highest affinity value (QK^T attention matrix) with i among all query tokens and vice versa. This seems to be a 'hard' attention, in contrast to the softmax attention in popular transformers. Authors could potentially provide more intuition on that, for example, what happens if for a given query token there is no support one that satisfies cycle-consistency - in such a case, aren't all attention weights equal to 0, due to additive bias?\n2. The method requires a vision-backbone model such as ResNet. It is mentioned that this was pre-trained, but it is unclear if that part was also fine-tuned together with CyCTR, or remained fixed? Also, the scores obtained with ResNet-101 are often worse than those with ResNet-50, in particular for 1-shot segmentation, what could that be attributed to?\n3. It is unclear which 'middle-level features' and 'high-level features' (lines 233-234) are used in case of each backbone. Also it is unclear what the 'prior map' refers to.\n4.  line 239 is unclear: 'As for the baseline of our method, we use two residual blocks [10] instead of the transformer to encode query feature' -  How residual blocks replace cross-attention? It should be marked in Table 4 that the first line refers to that benchmark model. Also, language errors. \n5. (246) 'during test, 1000 episodes' - what are episodes here? unclear\n6. Figure 4 - if the 'query' column shows the ground truth segmentation, maybe it should be marked.\n7. What are the different stacked encoders that are mentioned in 4.4.2?\n8. There are multiple typos, language errors and unclear formulations:\n- 59: to encoder -> to encode\n- 105: point -> pointed\n- 113: are -> were\n- 128: ideology -> idea\n- 234: 'high-level feature are used to generate a prior map that concatenated with the middle level feature' - unclear\n- 237: restuls -> result\n- 238: convolution -> convolutional\n- 238: is consisted by -> consists of\n- 245: classes is used -> classes are used\n- 246: during test -> at test time\n- 253 training *for*  50 epochs\n- 263: the number of head for all attention layers are set -> the number of heads for all attention layers is set\n- 306: one may curious\n\n\n\n\n\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does not address those issues. Negative societal impact of this work is probably similar to that of any work in machine vision - authors may want to discuss the impact of digital surveillance since improvements in image segmentation might results in greater capacity of those systems.\n\nAs for the limitations, it would be interesting to see how the number of training classes affects the scores.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "5Yag4aYld9K", "reviewer": "Reviewer_JH49", "summary": "This paper proposes a model to facilitate the few-shot semantic segmentation task by utilizing pixel-wise relationships between support and query images. The authors use a cycle-consistent attention mechanism to perform cross-attention between features from support and query images. Experiments on the two commonly used benchmarks validate the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n* The idea of considering pixel-wise relationships between support and query images is interesting.\n* The paper is well-written, easy to follow.\n* Extensive experiments with adequate discussions.\n* The proposed method achieves competitive results on two benchmarks.\n\nCons/Questions:\n\n* I'm confused why the model is designed for the pixel-wise relationship. From the network structure in the supplementary, the input of the cycle-consistent transformer is the features combined with the class prototype, which is obtained by the masked average pooling method. The prototype is class-level, right? Why called the cycle-consistent transformer considers the pixel-wise relationship?\n* The whole CyCTR module consists of stacking L encoders. What's the setting of L in the reported experiments? \n* The method seems to be computationally a little expensive. Did the authors analyze the efficiency of the proposed method?\n* Current benchmark is established on the one-way task, which is a binary mask prediction. Can the proposed method be applied to multi-way semantic segmentation tasks?\n\ntypo: L210 \"... offeset from each piexl and P represents\" should be pixel", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Based on my knowledge, this paper poses no negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes to do few-shot segmentation by applying a transformer to the few-shot support examples. The transformer’s attention is constrained so that only pixels that meet the proposed “cycle-consistency” constraint may be used for the prediction task. While the submission isn’t particularly groundbreaking methodologically, the approach seems to be novel overall and the quantitative results are quite strong across two standard benchmarks: for example on COCO, the proposed approach outperforms previous SotA by 40% to 32% for 1-shot, or 45% to 37% for 5-shot. The method is ablated to demonstrate the benefit of various parts of the model, e.g. the use of the cycle-consistency constraint vs. a baseline transformer with no cycle-consistency masking.

Reviewers pointed out a number of clarity and presentation issues with the submission. Please take these suggestions into account and revise the text accordingly for the camera-ready. Overall the text can be a bit difficult to parse at many points, and this caused confusion among reviewers and myself -- please try to revise and clarify wherever possible to phrase things more straightforwardly, especially in the core method section.

Other few-shot segmentation papers reported similar results to those in the submission [1, 2], but as they were roughly concurrent with this submission, the submission can’t be penalized for them. However, the authors are strongly encouraged to include comparisons with these recent works in the camera-ready version, and provide context comparing their proposed approach to these methods.

Given the novelty of the approach and the strength of the results, I recommend accepting the paper to NeurIPS.

[1] https://arxiv.org/abs/2104.01538
[2] https://arxiv.org/abs/2103.15402
Decision: Accept (Poster)