Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation
Authors: Yuan Wang
Naisong Luo
Tianzhu Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: Computer Vision
Few-shot Segmentation
TL;DR: We proposed a novel few-shot segmentation method that focus on the query image instead of exhaustively explore support information.

Abstract: Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention to the mining of the critical query branch. In this paper, we rethink the importance of support information and propose a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train G and D via an adversarial process, where G is optimized to generate more accurate masks approaching ground truth to fool D. We conduct extensive experiments on commonly used Pascal-5i and COCO-20i benchmarks and achieve state-of-the-art results across all settings. In addition, the decent performance with weak support labels in our query-centric paradigm may inspire the development of more general FSS models.

Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation
Yuan Wang'; Naisong Luo' Tianzhu Zhang
Deep Space Exploration Laboratory/School of Information Science and Technology, University of Science and Technology China, Hefei 230026, China. (wy2016, Lns6]-0mail ustc. edu. cn, tzzhangdustc. edu.
Abstract
Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention the mining of the critical query branch. In this paper, we rethink the importance of support information and propose new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train and via an adversarial process, where G is optimized generate more accurate masks approaching ground truth to fool D. We conduct extensive experiments on commonly used Pascal-5' and COCO-20* benchmarks and achieve state-of-the In addition, the decent performance with weak support labels our query-centric paradigm may inspire the development of more general FSS models. Code will be available at lhttps:/ //gi thub. com/ hoon//ydm////// Syyxdm/MNeet. AMNet
results across all settings.
1 Introduction
As fundamental computer vision task, semantic segmentation [1] has drawn decades of research interest in both academia and industry. However, the underlying driving force of fully supervised segmentation models is massive, densely labeled data. The inherent category sensitivity of these models leads to severe performance degradation when dealing with previously unseen categories. To equip segmentation models with generalization to novel classes, few-shot segmentation (FSS) has been proposed [2]. which achieves segmentation new category images (called query images) with only handful of annotated samples (called support images) without retraining the model.
The current top-performing FSS methods focus adequately exploring the information contained in the support samples to guide the segmentation of the query images. Among them, prototypical learning methods (3-6] concentrate support features into or several representative prototypes to direct the classification of query pixels, while the affinity learning approaches (7-9) try to equip the query features with support semantic clue of pixel granularity. Although considerable progress has been made in the direction of probing support information, we noticed that significant intra-class diversities (as illustrated Figure 2 between support and query images are common occurrences. In these cases, the target feature from the query image may be quite distinct from the support one.
"Equal contribution. 'Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Ps
Intra-class
Fq
Fq
F psd
Intra-object Similarity
Fg
Protatype Extraclion
Protetypes
Pseudo Support
Fusion or Comparison
Object Mining
Rough Guidance
Alignment
Clue
Flatten
features
M2
Prediction
M
Prediction
(a) Support-centric approaches
Our query-centric approach
Figure 1: Comparison between the previous methods and ours. (a) Most of the previous methods focus their efforts on extracting more support information. They condense support information into prototypes 3 4 [2] or directly explore pixel-level support features 7 [3]. (b) The proposed
AMFormer focus on the query features and exploits intra-object similarity to mining the complete
target, forming a query-centric FSS method. Only rough category guidance from support needed in
our approach.
Hence there raises natural question: exhaustive support information extraction indispensable for query image segmentation?
We revisit the role of support information in few-shot segmentation through a pilot study. As shown in Table we randomly crode support foreground features with varying proportions to explore how the support information completeness affects query segmentation accuracy. It can be observed that the performances do not degrade significantly or even surpass the complete one. We deem two main reasons contribute to the phenomenon: Rough category guidance from support is sufficient to guide the accurate query image segmentation. 2) Forcing fine-grained support information may introduce redundancy, even bias, which confuses the prediction of query images, particularly when intra-class diversities are prevalent. Motivated by the above insights, we adjust the direction and focus on the query branch. Building the observation of intra-object similarity, i.e., pixels within the same object are more similar than those across different objects [1O M
(as shown in Figure 2 and Table 8] we argue that partially activated object regions can serve cues infer whole objects. In this paper, we construct a pseudo support image that has low intra-class variation with the query image by activating query image with rough calegory-level support clue. Then we exploit the pseudo support feature to guide the query segmentation, forming a novel query-centric exploration strategy as illustrated in Figure
We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. The holistic semantic clue from the labeled support image is uti- lized roughly locate the most discriminative target area on the query image. (2) Local to global expansion. We utilize intra-object feature similarity explore contextual informa- tion and then highlight the less discriminative object parts. (3) Coarse to fine alignment. The coarse activated region is further refined by eliminating the detailed difference with ground truth. Different from previous methods that focus on extracting fine-grained support information, Our method less susceptible intra-class diversity as only a rough holistic support guidance is needed in the first procedure to generate the pseudo support. More attention is paid to the last two query-focused procedures that enable growing incomplete local regions into accurate segmentations.
part. oblett
part oblets Sparts
cdjjett
object'
t-SNE visualization of the intra-class diversity and the intra- object similarity. As shown by the blue and green circles, different instances of the same category may be far apart in the feature space, e.g., dog. While pixels from the same object share high feature similarity, as shown by the three circles within the blue one.
Figure 2:
We propose an end-to-end Adversarial Mining Trans- former().Fooner) to couple procedures (2) and (3) and mutually enhance each other, which optimizes an object mining transformer G and a detail mining transformer via an adversarial process. AMFormer aims at fully excavating the target area and aligning it in detail under the guidance of the pseudo support features. To dig out the entire object

---

region, in the G, we conduct multi-scale pixel-level correlation between the query features and pseudo support features. The exploration of the intra- and inter-scale contextual information in G progressively highlights the latent target parts that share high intra-object similarities. To effectively align the prediction details, egg., object boundaries, / models multiple detail proxies to accurately distinguish the subtle local differences between the ground truth and the prediction generated by G. By means of adversarial training, G is optimized to predict more accurate segmentations to fool D and only the well-learned G is required to produce accurate predictions in the inference stage.
We evaluate our AMFormer on two widely used benchmarks COCO-20' and Pascal-5' with dif- ferent backbones and the AMFormer consis- tently sets new state-of of-the-art on different set- tings. Moreover, our query-centric approach does not reckon on elaborate support annota- tions. achieves remarkable performance even with weak support labels such scribbles or bounding boxes, formulating a more practical FSS model. We also hope that our work can inspire more research on query-centre FSS methods. To recap, our contributions are concluded as follows: (i) We re-evaluate the importance of support information in FSS and demonstrate that a coarse category hint suffices for accurate query segmentation. This motivated us to put forward novel query-centric FSS method. (ii) We propose novel Adversarial Mining Transformer (AM- Former) that optimizes an object mining transformer G and a detail mining transformer /) for region expansion and detail alignment, respectively. (iii) Extensive experiments show that AMFormer significantly outperforms previous SOTAs. The conspicuous performance with weak support label also sheds light for future research of more general FSS models.
Table 1:
The performance (mloU) under different support feature erosion ratios. We conduct this pilot study on 1st split of Pascal-5' using ResNet- 50 and I-shot setting.
means reproduced results.
Frasion Ratio (%)
80
65
50
35
20
0"
PFFNer 14
60.6
61.0
61.5
61.2
61.0
61.2
HDMNer
69.3
70.0
70.5
70.4
69.9
70.2
2 Related Work
2.1 Semantic Segmentation
Semantic segmentation has been widely applied to autonomous driving, medical image processing [15] 17) and so on. The aim of semantic segmentation is to assign each pixel within the given image to a predefined category label. The seminal FCN []] sparked a wave of remarkable advances in semantic segmentation [18-20) based on convolutional neural networks (CNN). Various networks focus on better context exploration by enlarging the receptive field of CNN via dilated convolutions 211 22).
global pooling 23 and pyramid pooling [21 24 In addition to CNN- -ased models, the success of Vision Transformer (ViT) (25 encourages a series of transformer- based segmentation models 22 29 For instance, maskformer
30 adopts the transformer decoder [1 to conduct the mask classification based on a set prediction objective. Many subsequent works improve this framework [32 B5 and
gradually formed a unified segmentation model that can address different image segmentation tasks. Despite the success, these methods cannot generalize to novel classes in the low-data regime.
2.2 Few-Shot Semantic Segmentation
Few-shot segmentation (FSS) 2 is established to segment new category images (query images) with only a few labeled samples (support images). Owing the reliable annotations in the support set, current FSS methods mainly focus on effectively excavating support information, which can be roughly divided into two categories: prototypical learning methods and affinity learning methods. Motivated by PrototypicalNet (36). many previous works condense the support information into single (37 39 6 40 or multiple prototypes [41 5 46) and then conduct feature comparison or aggregation. For example, ASGNet [3] adopts superpixel-guided clustering to adaptively construct multiple prototypes, which are concatenated with the most relevant query features guide the pixel classification. Differently, for the seek of fine fine-grained support guidance, affinity learning 50) are designed to establish pixel-level associations between support and
methods [ 47 g
query features via attention mechanism 7 9 or cost volume aggregation [8] 48). For instance,
CyCTR 7 introduces a cycle-consistent attention mechanism to equip query features with relevant support information. Though achieving promising results, these methods depend heavily on support information and are prone to segmentation errors in the presence of large intra-class variations. Some methods
[1 6 try to solve this problem by mining the class-specifer representation from the
3

---

query branch but remain at relatively coarse prototype granularity. In this paper, propose a novel query-centric approach that exploits intra-object similarity probe query targets with only overall category-level guidance from support.
3 Method
3.1
Problem Definition
Few-shot segmentation (FSS) tackles novel class object segmentation with only few densely- annotated samples. Episodic meta-training [51] is widely used to enhance the generalization of FSS models. Specifically, the dataset is divided into the training set Dtrain and the testing set Drest- The category sets of "trrii and Diest are disjoint, i.e., Ctrain n Ctest = 0. A series of episodes are sampled from Drrain to train the model, each of which composed of support set S = J kk=1 and a query set Q = {4g, M, in the K-shot setting, where 1 and M denote the RGB image and corresponding binary masks, respectively. Under the supervision of My. the model trained to predict the query mask conditioned on the S and 14. After that, the trained model is evaluated on the episodes sampled from Drest without further optimization.
3.2 Revisiting of Transformer-basee based Feature Aggregation
Transformer layers 52 are widely used in computer vision tasks for feature aggregation. The critical component of transformer layer is the attention mechanism that enables the long-range modeling capability. Specifically, the attention layer is first applied to compute the attention weight of the source feature sequence R xC and the target sequence T E RN where the N1, N2 denote the length of sequences and C' is the embedding dimension, formulaic as:
Q(K) Vd
S Softmax(
)
Q QW K = SW
in which Wº and W E RCxd are learnable linear projections and vd is the scaling factor. The source information is adaptively transported to the target sequence according to attention weight, and a feed-forward network (FFN) is further applied to transform the fused features:
= FFN(SV),
V=SWV,
where the W the linear linear projection and T is the enhanced target feature sequence. We can abbreviate the feature aggregation process as:
T FeatAgg|T S.
Note that when T = S. the Feat explores contextual information within the feature sequence, ie., acts as the self-attention mechanism. The above processes are implemented with the multi-head mechanism to enhance performance further.
3.3
Adversarial Mining Transformer
3.3.1
Overview
The proposed framework tailored for query-centric FSS consists of three procedures. i.e. discrimi- native region localization, 2) local to global region expansion, 3) coarse to fine mask alignment. We roughly locate the object of novel class under a rough support guidance via nonparametric cosine similarity in procedure 1). The resulting incompleted region is expanded by the object miner G to cover the whole target in procedure 2). The detail miner D in procedure 3) is tasked with identifying local discrepancies between the ground truth and the expanded mask generated by G, thereby aligning and D via an adversarial process and forming the end-to-end AMNet as illustrated in Fig. 3 The details are as follows.
the coarse prediction detail. We optimize
3.3.2 Discriminative region localization
Considering the significant intra-class diversity between samples, we contend that support information more suitable as overall guidance indicating the novel class rather than detailed reference. Given

---

Object Mining Transformer 6
Detail Mining Transformer D
Query Imaoge
Ground Truth Ma:
Query Features F4
Creek attention
01-1
Shared Backbone
Real Object Fenture
Real Local Features A
Aggregated feature F
00n11
Support Fexture Y>
Criss
Crrsss
Cross
Query Fenture F
RealiFake
Festure F
17
Local Prusies
Select FC
pooling
Codine
Conv Head
00-0 Cross altention
Pseudo
Support pod
Support coliseo
Take Object Feature
Fale' ama Features
Maa Mar
Prediction M
Discriminative Region Localization
Local Global Expansion
Aignment
Figure 3: Illustration of the proposed AMFormer. We approach the query-centri FSS with three procedures, i.e., discriminative region localization, local to global region expansion, and coarse to fine mask alignment. AMFormer optimizes an object mining transformer and detail mining transformer via an adversarial process couple these procedures.
the support image features F, B and corresponding mask M, E HHW where H and W describe the feature size and C' is the feature dim, we apply mask average pooling (MAP) to obtain the target feature Fù, RIxC that serves as the holistic support guidance, formally:
F2 MAP(F M.).
We exploit the FA, to indicate the target region in the F, based on the cosine similarity:
Myr 1 1"(ooftmax(cos( Fi,, F. F,))), TT - 1, 0,
x > T
(5)
otherwisc
relatively high threshold r set to locate the most class-relevant regions and suppress activation to backgrounds. In experiment, 7 is set 0.7.
3.3.3 Local to global region expansion
Adaptive feature aggregation. The high-confidence object regions Mgr E H.HW obtained by Eqn (5) are incomplete. We introduce the object mining transformer G to expand the local region to the entire target area based on the intra-object similarity. Specifically, G aims at aggregating the representative target features from the pseudo support to less discriminative object parts in the query, formally,
F, =,= (Feat Agg [ (F( F(F F(F psd)]),
where the F HX + HW the spatial flatten operation. Fpeds which serve as the source sequence in Eqn I essentially query features filtered by Mşr:
ped Fg C Mgr.
Homogencous feature guidance enables our aggregation to circumvent the effects intra-class variance. Note that the original attention mechanism applies Softmax activation along the source sequence dimension. Directly adopting this scheme our aggregation makes the query background features inevitably attend foreground features since all the non-zero features in the pseudo support belong to the foreground. Inspired by [3), we adjust to implement Softmax along the axis of Fy- In this way, the discriminative object features are more likely to be aggregated target areas that share high intra-object similarities with F pad-
Cross-scale information transportation. We observe that in many cases, the query image contains different objects with varying scales, which are to be segmented in other episodes. To handle this spatial inconsistency, construct the G in multi-scale form. Concretely, follow [3] to establish hicrarchical query features {F F with down nown-sampling and self-attention layers, in which
F R -
W XC 1= 1=1,..,4
(8)
5

---

The corresponding pseudo support features p=d. is obtained by the Hadamard product of {F, 11 and downsampled Mgr. Then Eqn (6) is implemented independently in each scale to obtain the aggregated query features {F a Finally, we fuse the multi-scale aggregated query features in top-down manner as down in [4. specifically,
F Convaxa CanvixxaCoovv.i (F.,, + R(F + R(F,,+1)))
The fused features from the last stage F' q.1 E I XX are exploited to predict the expanded target area M. IR Nx via small convolution head. Benefiting from the adaptive feature aggregation and cross-scale information transportation in the object mining transformer G, less discriminative target parts can be highlighted in the query image, forming the expanded object mask M
3.3.4 Coarse to fine mask alignment
Detail mining transformer D is designed to discriminate subtle differences between the ground truth Mgt and the expanded object mask M. generated by G, since M. adequately covers the object but still exhibits misalignments in fine-grained details, e.g., boundaries. By training G and D via an adversarial process, is optimized to generate more accurate target masks approaching ground truth to fool D, thus achieving coarse to fine prediction alignment.
To capture comprehensive details, D) models multiple local proxies, each of which is tasked with exploring the object features respectively specified by M. and M9t to obtain a pair of local features. The most different pair is selected to output real/fake results. Concretely, we concatenate the learnable proxies as a feature sequence P E RxxC. xC, where N denotes the number proxies. We perform feature adaptive aggregation through the attention mechanism construct local features, In specific,
f2 gg F(Fa @ Me)]:
(10)
where the "fake" object features specified by predicted mask (F.)M.) serves as the source sequence of aggregation process. Note that the M. is not binarized. Similarly, local features originate from "real" object features are obtained by:
Ω2 Aggt P, F(F @MM MMg))
(11)
Finally, we calculate the cosine similarity among the real local features S2 { and the fake ones S2y = {w> ) The most different pair (wy, w is selected and fed into a fully-connected layer predict the fake/real results for adversarial training. By this way, D is optimized to discriminate detailed local differences, while the G will generate more precise masks to fool D by adjusting itself.
Since there is explicit supervision, different proxies may focus the same object part. We impose a diversity loss to avoid this degradation by expanding the discrepancy among local features:
N
N
(W) w/
(wit wiي wf |لell w/ lll/
Laii N(N 1)
Σ
Σ
+
(12)
[
||2||
l12
1= j=1,1/
The intuition of Eqn 12 is trivial. If different proxies focus on the same object region, Lati will be large and adjust the learning of proxies.
3.3.5 Training and Inference
Training loss. Our architecture consists of generative part, i.e., object mining transformer G and discriminative part, i.e., detail mining transformer D. The D judges whether the mask is real (ground truth) or fake (generated by G) by mining the detailed features of the object framed by the masks. To fool D, is supposed to predict a more accurate mask approaching the ground truth. We alternately train two parts to achieve mutual promotion. When training D, the parameters of G are frozen, and the loss function for D is formulated as:
La log( (FC(w)))) - log(1 FC(w))) Adix Ldios
(13)
where the FC: Ω + the wi w/ Z denote the most different pair of local features from the object specified by ground truth Mgt and the predicted mask Me, respectively. Adio denotes the weight of diversity loss and we
[0, 1] denotes the fully-connected layers that output the real/fake results, and

---

Table 2: Comparison with other state-of-the the-----tte-rrrt methods for 1-shot and 5-shot segmentation on PASCAL-5' using the mloU (%) evaluation metric. Best results are shown in bold.
I-shot
5-sbot
Method
Backbone
Toldo
Told1
Told2
Told3
Mean
Told0
lold]
[old2
told3
Mean
PFENSU TPAM12020)
61.7
69.5
55.4
60.8
63.1
70.7
55.8
61.9
RePRICCVSO
59.8
68.3
62.1
48.5
59.7
64.6
71.4
71.1
59.3
66.6
DSNNUIICVIRII
64.3
70.7
60.3
60.5
64.0
70.3
73.2
67.4
67.1
69.5
CyCTR
65.7
71.0
59.5
64.0
69.3
73.5
63.8
63.5
67.5
NERTNeU/CVPRI002
45
65.4
72.3
59.4
59.8
64.2
66.2
72.8
61.7
62.2
65.7
Res-50
70.5
63.7
SSP|ECCV202]
1
60.5
67.8
66.4
51.0
61.4
67.5
72.3
75.2
62.1
69.3
NIPS202
72.8
73.7
59.2
61.6
66.8
73.1
74.7
61.6
63.4
68.2
VAT]SCCV21221
48
67.6
72.0
62.3
60.1
65.5
72.4
73.6
68.66
65.7
70.1
BAM(CVPR3302
[55
69.0
73.6
67.6
61.1
67.8
70.6
75.
70.8
67.2
70.9
HDMNatjCVPR.000
71.0
75.4
68.9
69.4
71.3
76.2
71.3
71.8
AMFomer (ours)
71.1
75.9
69.7
63.7
70.1
73.2
77.8
73.2
68.7
73.2
PFENUU 1PAM120201
14
60.5
69.4
54.4
55.9
60.1
62.8
70.4
54.9
57.6
61.4
CyCTR
67.2
71.1
57.6
59.0
63.7
71.0
75.0
58.5
65.0
67.4
NERTNeI/CVFR2200
65.5
71.8
59.1
58.3
63.7
67.9
73.2
60.1
66.8
67.0
DCAMAIECCV202)
65.4
71.4
63.2
64.6
70.7
66.8
68.3
Res-101
SSP/ECCV2022]
1
63.2
70.4
68.5
56.3
64.6
70.5
76.4
79.0
66.4
73.1
NIPS20221
[SCVV2C22
70.0
72.5
64.8
64.2
67.9
75.0
75.2
68.4
69.5
72.0
AMFommer (ours)
71.3
76.7
70.7
63.9
70.7
74.4
78.5
74.3
67.2
73,6
Table 3: Comparison with other state-of-the-art methods for 1-shot and 5-shot segmentation on COCO-20' using the mloU (%) evaluation metric. Best results shown in bold.
I-shot Told2
5-sbot fold2
Method
Backbone
FoldU
Fold1
1old3
Mean
foldO
fold1
Fold3
Mean
NERTNNCICVPRR0021
38.3
40.4
39.3
38.
39.1
42.3
44,4
44.2
41.7
43.2
45.1
42.7
47.4
50.2
HSNetTOCV2022
M
Res-101
37.2
44.
42.4
41.3
41.2
45.9
53.0
51.8
47.1
49.5
DCAMAIECCV2003
41.5
46.2
45.2
41.3
43.5
48.0
58.0
54.3
47.1
51.9
IPMT) NIPS2021)))
6
40.5
45.7
44.8
39.3
42.6
45.1
50.3
49.3
46.8
47.9
PFENCU TPAMM2220
14
34.3
33.0
32.3
30.1
32.4
38.5
38.6
38.2
34.3
37.4
RePRRICNPEE0021
34.1
HSSNet (CC/V2021)
8
36.3
43.1
38.7
38.7
39.2
43.3
51.3
48.2
45.0
46.9
CyCTR(NIP5202)
38.9
39.6
BAMICNPR.00Z)
15
Res 50
43.4
50.6
47.5
43.4
46.2
49.3
54.2
51.6
49.6
51.2
IPM'T NIPSS0003
41.4
45.1
45.6
40.0
43.0
43.5
49.7
48.7
47.9
47.5
DCAMATECCVI022
41.9
45.1
44.4
43.3
45.9
50.7
46.0
48.3
ECCV2222
4
39.0
43.8
42.6
39.7
41.3
44.1
51.1
50.2
46.1
47.9
HDMNeeJCNPRa002
50.6
AMFomer (ours)
44.9
55.8
52.7
50.6
51.0
52.0
61.9
57.4
57.9
57.3
set it to 0.1 in experiments. Similarly, D is not optimized when training G. Following 13 also include a KI. (Kullback-Leible divergence loss between the correlation maps of adjacent stages to distill informative semantic cues of earlier stages refine the segmentation quality. Please refer to [13 for more details. The overall loss function for is:
bg = log log(FC(w))) + LKL BCE(M,, Mgt).
(14)
Where the BCE is the Cross-entropy loss. After training, only the well-learned G is needed in the inference stage.
M K-shot inference. When K support samples {I+ P2 lk=1 available, different discriminative be obtained from different support images via Eqn 4 and Eqn ( We adopt the union of all the activated regions the initial object region, i.e., Mgr M M2 4T The comprehensive Myr originating from multiple supports more robust to pose
target regions (M
n M differences, occlusion, etc., than that from single support image, thereby boosting the performance.
Experiments
4.1 Datasets and Evaluation Metrics
We evaluate the proposed AMFormer on two commonly used few-shot segmentation benchmarks, Pascal-5' [2] and COCO-20* [56]. Pascal-5' is constructed based the PASCAL VOC 2012
7

---

Table 4: Comparison of FB-IoU and the number Table Ablation studies for different compo-
of learnable parameters on COCO-20* 20'
nents and architecture design in AMFormer.
Backbone Methods
FB-IoU (%)
#leamable params
OM
DM
1-shot
5-shot
Single
Multi
w/o Lave
W/ Las
HSNet
10.4M
65.0
BAM 5S
68.2
70.7
2.6M
66.5
+1.5
Res-50
DCAMA 9
69.5
71.7
47.7M
69.5
+4.5
HDMNet 6
72.2
77.7
4.2M
/
AMFormer (ours)
72.9
78.8
5.1M
70.7
+5.7
Table 8: Quantitative mea- surement intra- and inter- object similarity.
Table 6: Performance comparison on 1st split of varying the number of local proxies.
Table 7: mloU of different kinds labels (1 split).
#Prr
10
14
mask
bbox 70.5
scribbles 70.0
mloU
70.7
71.0
71.2
71.
71.2
71.2
71.3
Intra-object Inter-object 0.015 0.546
dataset [57] and additional annotations from SBD 158 equally divide the 20 categories into four splits, three which for training and the rest one for testing. COCO-20' is a larger benchmark based MSCOCO dataset [59], the 80 categories of which are partitioned into four splits for cros-vvaldattion as down in [14. We randomly sampled 1000
Following previous works 14 55
we
episodes from the testing split for evaluation. Following common prectices 37 [4 55 38), adopt mean interseciinn-over-nnioo (mloU) and foreground- evaluation metries.
intersection-over-umme (FBIoU)
4.2
Implementation Details
We adopt ResNet-50 and ResNet-101 [60] as the backbone network in our experiment. Following concatenate the features from the 3rd and 4th blocks of backbone and exploit a X 1 convolution layer to generate F, and F, of middle-level to avoid overfitting. The number attention layers in the and D are set and 2, respectively. We employ the same data augmentation setting as [1]. Since the G and D of AMFormer are trained alternately, we increase the number of training epochs to 300 for Pascal-5' and 75 for COCO-20* and set the batch sizes as and 4, respectively. Adam W [6] optimizer with poly learning rate decay is used to train both the G and D. The initial learning rate is set 1 and the weight decay le It should be noted that we adopt a base learner to filter the categories that appear during the training process for a fair comparison with previous works [55 13). For more details please refer to the Supplementary Material. Our approach is implemented using PyTorch and all experiments are conducted on 4 NVIDIA GeForce RTX 3090 GPUs.
previous works 14 3
4.3 Comparison with State-of-the-Art Methods
We present the comparison of our method with previous FSS methods Pascal-5' and COCO-204 datasets in Table 2 and Table 3 It can be observed that the proposed AMFormer significantly outperforms previous advanced approaches and achieves new state-of-the-art results under all settings. Specifically, on Pascal-5', our AMFormer achieves 70.4% and 73.2% mloU when using ResNer-50 as the backbone for 1-shot and 5-shot settings, surpassing the most competitive HDMNet [13 by 1.0% and 1.4%, respectively. With ResNet-101 backbone, our method outperforms the previous best results 6 by 2.8% (1-shot) and 1.6% (5-shot). We can obtain additional improvement when using larger backbone network, which demonstrates the scalability of the AMFormer. We attribute this performance gain better intra-object similarity within the more informative features. As for the more complicated COCO-20' our approach also exhibits superior performances compared to other methods, demonstrating its competitiveness on complex data. Besides, Table gives the comparison with previous methods in terms of FBloU on Pascal-5' using ResNet-50 backbone. AMFormer also outperforms all of the previous works by a considerable margin. Qualitative results are shown in Figure 4 please refer to Supplementary Material for analysis.
8

---

support
Mar
M, wio
M, w/
GT
image
part
part
part
part
bostt
hird
Figure 4: Qualitative results from different
Figure 5: Visualization of regions activated by different local proxies.
stages.
4.4 Ablation Study
As shown Table 5 a series of ablation studies are conducted on the first split of Pascal-5' with ResNet-100 101 backbone analyze each component of the proposed AMFormer. Note that the first line of Table 5 the result our ablation baseline. The baseline adopts self-attention within the query features for feature parsing, and cross-attention to aggregate support information into query features as done in 77.
Effectiveness of object mining transformer G. We first construct a naive single-scale G as the 2nd row of T We can observe significant performance lift, i.e., 1.5% in mloU. This improvement demonstrates the effectiveness of the proposed query-centric strategy, which is built upon intra-object similarity and thus less affected by intra-class diversity. The multi-scale implementation of further brings 3.4% improvement in mloU, and it already achieves a decent performance. It shows the importance multi-scale feature aggregation in dealing with objects with dramatically changing scales in the query image.
Effectiveness of detail mining transformer D. The comparison between the 3rd and the 5th row of Table 5) shows that the combination of D improves the performance by 1.2% mloU on the basis of G. We attribute this performance gain to the exploration and distinction of local features in the D, which encourages G to pay more attention to these ambiguous regions, e.g., object boundaries. Compared with the pixel-level supervision given by Cross-entropy loss, the proposed part-level adversarial training incorporate local region context information to guide accurate segmentation. Note that without Ldive the performance improvement brought by drops to 0.2%. This phenomenon reasonable because in the absence of Laiv. local proxies tend to degrade to focus the same region, leaving some details unexplored.
support image
query image
activation (baseline)
activation (ours)
Investigation of the local proxies. We first visualize the region activated by the learnable proxies to analyze what they mainly focus on. As shown in Figure we observe that the highlighted area mainly lies on the boundary of the target and different proxies correspond to different directions. It confirms that the local proxies can well capture the local misalignment of the relatively coarse prediction from G, and then G optimized to pay more attention to these ambiguous areas under the direction of D. In addition, investigate the impact of the number of proxies on performance. As reported in Table 6 using more proxies can gradually im- prove the performance until the number achieves 10. More proxies longer bring additional improvement. This re- sult is expected, as too many proxies would learn redundant patterns.
Figure Visualization of the activa- tion of query images..
Discussion on the query-centric FSS paradigm. Since the FSS models are trained in the extremely low data regime

---

with the backbone not optimized, the feature space is not well aligned for novel classes. In this situation, the novel class data distribution usually has low coverage the feature space, i.e., object features of the same category but different image instances may be far apart. On the contrary, as we can observe Table [] pixel features within the same object are closer to each other than those across objects. In nutshell, intra-object similarity is more reliable than intra-category but inter- object similarity, As shown Figure we visualize the attention weight query features between support target and pseudo support, respectively. It shows that support targets tend to mistakenly activate background categories, while the pseudo support can well excavate the full object attribute to intra-object similarity.
Owing the reduced reliance on support information, query-centric FSS methods could achieve remarkable performance with more weak support annotations. Table 7 shows the results of the proposed AMFormer on the first split of Paseal-5' with bounding boxes or scribbles as support labels. The promising results demonstrate the feasibility of more general FSS segmentation models.
4.5 Broader Impact and Limitations.
We proposed novel query-centric FSS paradigm that shifts the research focus from support to query features. This is new perspective that may inspire the development of more general FSS models that can be adopted in different tasks such as video object segmentation [62H64] or open-vocabulary segmentation [63] [66]. Although our AMFormer achieves remarkable performance, the number of training epochs is larger than some of previous approaches since the and D in the AMFormer are trained alternately.
Conclusion
In this paper, propose a novel query-centric FSS method, i.e., Adverasial Mining Transformer (AMFormer), which can achieve accurate query segmentation with only rough support guidance. Extensive experimental results demonstrate the superiority of our method. The decent performance with weak support labels also demonstrates the potential of the query-centric FSS paradigm.
6 Acknowledgments
This work was partially supported by the National Defense Basic Scientific Research Program of China (Grant JCKY2021601B013)
References
[1] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3431-3440, 2015.
[2] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.0344 2017.
[3] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8334-8343, 2021.
[4] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 8312-8321, 2021.
[5] Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Efstratios Gavves. Dynamic prototype convolution network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11553-11562, 2022.
[6] Yuanwei Liu, Nian Liu, Xiwen Yao, and Junwei Han. Intermediate prototype mining transformer for fho-shot semantic segmentation. arXiv preprint 21Xiv:2210006780, 2022.
10

---

[7] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle- consistent transformer. Advances in Neural Information Processing Systems, 34:21984-21996 2021.
[8] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmen- tation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6941 -6952, 2021.
[9] Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense cross-query-and support attention weighted mask aggregation for few- shot segmentation. In European Conference on Computer Vision, pages 151-168. Springer, 2022.
[10] Robert M. Haralick and Linda G. Shapiro. Image segmentation techniques. Computer Vision, Graphics, and Image Processing. (9)1):000-132 1985.
[11] Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support few-shot semantic segmentation. In Computer Fision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX, pages 701-719. Springer, 2022.
[12] Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision, pages 142-158. Springer, 2020.
[13] Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chenyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. arXiv preprint arXiv:2303. /4652, 2023.
[14] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence, (01):1-1, 2020.
[15] Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, and Yongdong Zhang. Lesion- aware transformers for diabetic retinopathy grading. In Proceedings the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 19938-10947, 2021.
[16] Li Wangkai, Li Zhaoyang. Sun Rui, Mai Huayu, Luo Naisong, Yuan Wang, Pan Yuwen, Xiong Guoxin, Lai Huakai, Xiong Zhiwei, et al. Maunet: Modality-aware anti-ambiguity u-net for multi-modality cell segmentation. In Competitions in Neural Information Processing Systems, pages 1-12. PMLR, 2023.
[17] Rui Sun, Huayu Mai, Naisong Luo, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Structure- decoupled adaptive part alignment network for domain adaptive mitochondria segmentation. In International Conference Medical Image Computing and Computer- Assistee Intervention. pages 233..33.600 Springer, 2023.
[18] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2017.
[19] Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV), pages 418-434, 434, 2018
[20] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-ussisted intervention, pages 234-241, 234iier,,pringeer 2015.
[21] Liang-Chied Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834-844 848, 2017.

---

[22] Liang- Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pages 801-818, 2018.
Liang-Chich Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.
[23] Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. arXiv preprini arXiv: 050794 2015.
[24] Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmentation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3684-3692 2018.
[25] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[26] Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7262-7272, 2021.
[27] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a
perspective with transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 6881 1881-, 2021.
[28] Yifan Zhang, Bo Pang, and Cewu Lu. Semantic segmentation by early region proxy. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1258-1268, 2022.
[29] Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 34:10326-10338, 2021.
[30] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems, 34, 2021.
[31] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[32] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Gird- har. Masked-attention mask transformer for universal image segmentation. arXiv preprint 12.01221 2021.
[33] Hao Zhang. Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mp- former: Mask-piloted transformer for image segmentation. arXiv preprint arXiv:2303.07336. 2023.
[34] Naisong Luo, Yuwen Pan, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Camou- flaged instance segmentation via explicit de-camouflaging. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 17918-17927, 2023.
[35] Rui Sun, Huayu Mai, Tianzhu Zhang, and Feng Wu. Daw: Exploring the better weighting function for supi-super semantic segmentation. In Advances in Neural Information Processing Systems, 2023.
[36] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[37] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 3855) 3865, 2020.
12

---

[38] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9197-9206, 2019.
[39] Leilei Cao, Yibo Guo, Ye Yuan, and Qiangguo Jin. Prototype as query for few shot semantic segmentation. arXiv preprint 2Xiv:2211.14764, 2022.
[40] Siyu Jiao, Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, and Humphrey Shi. Mask matching transformer for few-shot segmentation. arXiv preprint arXiv:230.001008 2022.
[41] Chunbo Lang, Binfei Tu, Gong Cheng, and Junwei Han. Beyond the prototype: Divide-and- conquer proxies for few-shot segmentation. arXiv preprint arXiv:2204. 09903, 2022.
[42] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. European Conference on Computer Vision, pages 763-778. Springer, 2020.
[43] Yuanwei Liu, Nian Liu, Qinglong Cao, Xiwen Yao, Junwei Han, and Ling Shao. Learning non- non-target knowledge for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11573-11582, 2022.
[44] Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot segmentation. arXiv preprint arXivv2210.06908 2022.
[45] Atsuro Okazawa. Interclass prototype relation for few-shot segmentation. Computer Vision- ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 362-378. Springer, 2022.
[46] Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In European Conference on Computer Vision, pages 36-52. Springer, 2022.
[47] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16, pages 730-746. Springer, 2020.
[48] Sunghwan Hong, Seokju Cho, Jisu Nam. Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 108-126. Springer, 2022.
[49] Zhitong Xiong, Haopeng Li, and Xiao Xiang Zhu. Doubly deformable aggregation of covari- ance matrices for few-shot segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX, pages 133-150. Springer, 2022.
[50] Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7183-7192, 2023.
[51] Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Daan Wierstra, et al. Matching networks for one shot learning. Advances neural information processing systems, 29, 2016.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilin- gual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint 172.v:170..03859, 2017.
13

---

[54] Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Fhw- segmentation without meta-learning A good transductive inference is all you need? In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 13979-13988, 2021.
[55] Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning what not to segment: A new perspective on few-shot segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8057-8067, 2022.
[56] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 622-631, 2019.
[57] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[58] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultancous detection and segmentation. In European conference on computer vision, pages 297-312. Springer, 2014.
[59] Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
[61] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
[62] Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34:11781-11794, 2021.
[63] Rui Sun, Yuan Wang. Huayu Mai, Tianzhu Zhang, and Feng Wu. Alignment before aggregation: trajectory memory retrieval network for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1218-1228. 2023.
[64] Rui Sun, Naisong Luo, Yuan Wang, Yuwen Pan, Huayu Mai, Zhe Zhang, and Tianzhu Zhang. 1st place solution for youtubevos challenge 2022: Video object segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops, 2022.
[65] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pages 2002-2010, 2017.
[66] Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for opca-voca semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2945-2954 2023.
14

---

7 More Details for Multi-scale Object Mining Transformer.
In the object mining transformer G, we alternately down-sampling layers and self-attention layers to construct hierarchical query features. And the corresponding pseudo support features are obtained by the Hadamard product of the downsampled Myr. specifically.
q.1 where the F Hx H IR. #W is the spatial flatten operation and Down denotes the down- sampling layers, which is implemented with convolutional layers of double strides. In this way, we obtain multi-scale query features Fg.. 1 R2 x 2 xc 1....... L. and pseudo support features F ped. F, q. @ ((Mar) (p is bilinear interpolation operation). We perform feature aggregation within each scale to enable contextual information exploration, thus avoiding the spatial inconsistency in the query image. L is set to 3 in our experiments. The embedding dim is set to 64, and the number of head is set in all the attention layers.
(FFentt ((U
(15)
Detailed Experimental Settings
To achieve a fair comparison with previous methods [55 13
we adopt the ensemble strategy following BAM (55) to filter the base categories seen during training. Specifically, base learner is trained using the training splits in fully supervised manner, and the learned base learner is used to explicitly predict the targets base classes. PSPNet (18] is adopted as the base learner in all of our experiments, and we follow the BAM [55] to ensemble the prediction of the base learner and the proposed AMFormer. Our code is available https::/ //tg://bbbtbt com/Wyxdm/ AMNet
We also conduct an additional ablation experiment to evaluate the influence of the ensemble strategy as shown in Table We can observe that the ensemble strategy can incrementally improve performance. It should be noted that our AMFormer can also surpass previous state-of-the-ar methods (IPMT (6) without the ensemble strategy.
Table 9: Ablation of ensemble strategy on Pascal-5' with ResNet- backbone and 1-shot setting.
fold0
fold1
fold2
fold3
mean
IPMT 6
71.6
73.5
58.0
61.2
66.1
w/o ensemble
69.7
75,2
69.5
62.9
69.3
w/ ensemble
71.3
76.7
70.7
63.9
70.7
9 Dataset Settings
We respectively divided Pascal-5' and COCO-20' into four splits following we provide the detailed split settings.
14
for cross-validation.
In Table I0 and Table II
Table 10: Detailed splits setting of PASCAL-5'
Fold
Test classes
PASCAL-50
aeroplane, bicycle, bird, boat, bottle bus, car, cat, chair, cow diningtable, dog, horse, motorbike, person potted plant, sheep, sofa, train, tv/monitor
PASCAL-51
PASCAL-5"
PASCAL-5'
10 More Experimental Results
10.1
Quantitative analysis of intra-object similarity.
We compute the average pairwise pixel similarity from the same object (intra-object) and different objects from the support and query images of the same category (inter-object) using the cosine
15

---

Table 11: Detailed splits setting of COCO-20*
Fold
Test classes
Person, Airplane, Boat, Park meter, Dog, Elephant, Backpack, Suitease, Sports ball, Skateboard, W. glass, Spoon, Sandwich, Hot dog, Chair, D. table, Mouse, Microwave, Fridge, Scissors
COCO-200
Bicycle, Bus, T.light, Bench, Horse, Bear, Umbrella, Frishee, Kite, Surfboard, Cup, Bowl, Orange, Pizza, Couch, Toilet, Remote, Oven, Book, Teddy
COCO-20
Car, Train, Fire H., Bird, Sheep, Zebra, Handbag. Skis, B. hat, T. racket, Fork, Banana, Broccoli, Donut, P. plant, TV, Keyboard, Toaster. Clock, Hairdrier
COCO-202
Motorcycle, Truck, Stop, Cat, Cow, Giraffe, Tie, Snowboard, B. glove, Bottle, Knife, Apple, Carrot, Cake, Bed, Laptop, Cellphone, Sink, Vase, Toothbrush
COCO-203
similarity. Note that the pixel features that we used compute the similarity are middle-level features F, and F,, as described in the L255- in the original manuscript. The quantitative results of different categories are provided in Table 12 and Table 13 From the tables, we can observe that the intra-object similarity is at least one order of magnitude higher than the inter-object similarity. This demonstrates the superiority of the query-centric approach relying on intra-object similarity over support-centric methods that rely on inter-object similarity.
Table 12: Intra- and inter-object similarity of each class within Pascal-5',
Pascal-50
Pascal-5
Pascal-52
Pascal-5
class
Intra-
Inter-
class
Intra-
Inter-
class
Intra-
Inter-
class
Intra-
Inter-
aeroplane
0.449
0.008
bus
0,453 0,011
diningtable
0.496
0,010
potted plant
0,515
0.012
bicycle
0.460
0.009 bus
0.453 0.011
dog
0.571
0.009
sheep
0.536
0.007
bird
0.493
0.010
cat 0.643 0.021
horse
0.489
0.012
sofa
0.535
0.006
boat
0.483
0.009
chair 0.519
0.015
motorbike
0.445
0.008
train
0.509
0.008
bottle
0.504 0.120
cow
0.573 0.016
person
0.529
0.039
tv/monitor
0.513
0.024
10.2 More visualization results.
Visualization of segmentation at different stages. We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. (2) Local to global expansion. (3) Coarse to fine alignment. To illustrate the effects of the above three steps, in Figure 7 we visualize the outcomes of different stages. Procedure (I) can only roughly local the discriminative region of the target (2nd column of Figure 7). In procedure (2), the object mining transformer G exploits the intra-object similarity to explore multi-scale contextual information, thus highlighting the whole object (3nd column of Figure 7}. Segmentation from G can roughly cover the entire target but there still exist misalignments as shown in the yellow boxes in the 3rd column of Figure 7 The detail mining transformer D is responsible for discriminating those detailed misalignments, i.e., procedure (3). The proposed AMFormer couples procedures (2) and (3) via adversarial training. this way, the G can be optimized to generate more accurate segmentations(4" column of Figure] to fool 1).
Visualization of activation maps. We visualize the attention weight of query features between the
support target and pscudo support. Specifically, the attention matrix S E IR H, W,W,W,H... H.WW, is computed according to the Eqn (1) of the original manuscript. Then we compute the average activation of each query pixel over all support foreground pixels:
Act(i) D'WW. H. S(i,)) (F(M) > 0]
(16)
WHW > 0]
where the M, is the (pseudo) support mask. the baseline, the S oriented from the cross attention between the query features and the support features (support-centric). While in our query-centric
16

---

Table 13: Intra- and inter-object similarity of each class within COCO-20*
COCO-20"
COCO-20
COCO-20"
COCO-20"
class
Intra-
Inter-
class
Intra-
Inter-
class
Intra-
Inter-
class
Intra-
Inter-
Person
0.553 0.002
Bicycle
0.398 0.017
Car
0.514 0.012
Motorcycle
0.485 0.007
Airplane
0.582 0.020
Bus
0.440 0.019
Train
0.544 0.017
Truck
0.552 0.021
Boat
0.585 0.013
T.light
0.474 0.013
Fire H.
0.467 0.007
Stop
0.544 0.008
Park meter
0.476 0.014
Bench
0.459 0.044
Bird
0.530 0.016
Cat
0.549 0.010
Dog
0.482 0.010
Horse
0.574 0.018
Sheep
0.487 0.010
Cow
0.584 0.015
Elephant
0.467 0.012
Bear
0.460 0.012
Zebra
0.510 0.012
Giraffe
0.556 0.013
Backpack
0.479 0.013
Umbrella
0.571 0.020
Handbag
0.519 0.032
Tic
0.559 0.006
Suitcase
0.542 0.007
Frisbee
0.384 0.005
Skis
0.432 0.007
Snowboard
0,509 0.014
Sports ball
0.538 0.009
Kite
0.470 0.016
B.bat
0.532 0.023
B.glove
0.576 0.006
Skateboard
0.537 0.018
Surfboard
0.589 0.024
Tracket
0.373 0.012
Bottle
0.545 0.007
W. Wlglass
0.434 0.011
Cup
0.606 0.009
Fork
0.451 0.017
Knife
0.589 0.013
Spoon
0.562 0.027
Bowl
0.563 0.010
Banana
0.511 0.015
Apple
0.656 0.010
Sandwich
0.469 0.013
Orange
0.474 0.028
Broccoli
0.471 0.026
Carrot
0.539 0.007
Hot dog
0.558 0.008
Pizza
0.515 0.015
Donut
0.477 0.300
Cake
0.597 0.006
Chair
0.442 0.015
Couch
0.542 0.013
P.plant
0.433 0.032
Bed
0.560 0.007
D.table
0.474 0.046
Toilet
0.503 0.006
TV
0.456 0.014
Laptop
0.480 0,021
Mouse
0.447 0.021
Remote
0.583 0.011
Keyboard
0.542 0.007
Cellphone
0.498 0.018
Microwave
0.454 0.019
Oven
0.567 0,011
Toaster
0.433 0.007
Sink
0.375 0,027
Fridge
0.476 0.023
Book
0.586 0.025
Clock
0.526 0.018
Vase
0.567 0.013
Scissors
0.456 0.027
Teddy
0.583 0.016 Hairdrier
0.373 0.007 Toothbrush
0.479 0.013
AMFormer, the S is computed from the pseudo support and the query features. As shown in Figure 8 the support targets not only cannot fully activate the target in the query image, but also frequently activates the background categories. While the pseudo support can well excavate the full object attribute to intra-object similarity.
Visualization of local proxies. visualizes the activation maps of a part of proxies. It can be observed that different proxies tend to focus different local regions, and most proxies attend to the boundaries of the object, which is usually the most ambiguous region. In addition, a particular proxy consistently focuses on the boundaries in specific direction, e.g., proxy always activates the right border (5th column). Through the cooperation of multiple proxies, our detail mining transformer G can effectively detect the detailed local differences between the prediction of the object mining transformer G and the ground truth. By means of adversarial training. G will produce more accurate segmentations, especially in ambiguous regions, by adjusting itself fool D.
To explore the regions of interest for learnable local proxies, Figure
17

---

support
M gr
Me w/o
M w/ D
GT
767-300
767-300
767-300
767-300
Local to global expansion
Coarse to fine alignment
Figure 7: Visualization the segmentation the proposed AMFormer at different stages
18

---

suppor image
query image
activation
(baseline)
activation
(ours)
Figure 8: Visualizations of the attention weight of query features between the support target (support-
centric baseline) and pseudo support (ours).
monitor
potted plant
dog
motorbike
aeroplane
train
cow
image
proxy
proxy
proxy 3
proxy
Figure 9: Visualizations of the activated regions of local proxies.
19


Reviews:
[{"review_id": "WUCIVTHM8L", "reviewer": "Reviewer_WjSB", "summary": "In this paper, the authors introduce a concept for solving the few-shot segmentation problem using weakly supervised learning concept. The method relies on weak support labels like scribbles or bounding boxes that highlight the object regions in the query. The segmentation is achieved through two blocks - the object mining transformer and detail mining transformer - during training, while only the object mining transformer is needed during inference. The proposed approach outperforms other methods, as demonstrated by the results and ablation studies. The writing is clear and helps readers understand the proposed concept effectively.", "questions": "1. Incorporating adversarial training may affect the convergence and stability of the learning process. Can you provide more information on this?\n2. Do you notice any signs of over-fitting in the suggested method?\n3. How does the model complexity compare to the competing approaches in Table 2 and Table 3?", "limitations": "Have you considered how well the query-centric method will perform when dealing with small targets, like in satellite images? Specifically, I am wondering about its generalizability for different datasets, ex: satellite images or medical images.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have developed a solution that can solve the few-shot segmentation problem under more practical scenarios. This means that only partial annotation on the support images is required, making it a practical tool. Additionally, the query-centric concept can lessen the reliance on support images and overcome the differences between support and target images of the same class, making the solution more applicable. Although the detail mining transformer approach proposed with part-level adversarial training can slow down training, it is not necessary during inference. Lastly, the solution works well as AMFormer outperforms previous works significantly.", "weaknesses": "1. It seems that the model complexity is quite high due to the large-scale backbone, transformer-based object mining model, and multi-scale structure.\n2. I am unsure about the task division among the proxies, and it seems that the number of required proxies might depend on the complexity of the data. I hope that the design of this part can be made more clear and understandable so that it would be easier to debug in case of any issues.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer WjSB\nThanks for your valuable comments. We will explain your concerns point by point. \n\n---\n**Q1. About the model complexity.**      \n**A1:** In Table 4 of the paper we compare the number of learnable parameters of our model with several state-of-the-art FSS models. Following your advice, we conduct an additional quantitative comparison of the model complexity in terms of the inference speed and the FLOPs. All models are based on the ResNet-50 backbone and tested on the Pascal-20i dataset. Inference speed for all models is tested with one single RTX 3090 GPU for a fair comparison. \n\n|Methods| mIoU | Learnable Params(M)| FLOPs(G)  | FPS   |\n|:-----:|:----:| :----------------: | :-----:   |:---:  |\n| CyCTR | 64.0 | 5.6                | 96.7G     | 15.1  |\n| HSNet | 64.0 | 2.6                | 20.6G     | 35.2  |\n| BAM   | 67.8 | 4.9                | 26.6G     | 86.0  |\n| HDMNet| 69.4 | 4.2                | 10.6G     | 36.4  |\n| Ours  | 70.7 | 5.1                | 8.6G      | 42.0  |\n\nAs shown in the table, the model complexity of the proposed AMFormer is comparable to existing methods, and it achieves faster inference speed and fewer FLOPs than some of them. The relatively low model complexity can be attributed to the following:    \n(1) We employ the same backbone as most of the current FSS models and keep the parameters frozen throughout the training and testing process.    \n(2) We adopt low-dimensional features (dim=64) in all the transformer layers, reducing the computational cost and memory consumption.    \n(3) We further downsample the features extracted from the backbone to low resolution before feeding them into the multi-scale structure. Therefore, our multi-scale structure does not incur significant additional computational costs.   \n(4) The detail mining transformer D is not needed in the inference stage.\n\n**Q2: The task division among the proxies.**   \n**A2:** In L301 of the paper we briefly analyzed the task division of different proxies. As shown in Figure 6 of our paper, we visualize the region activated by different proxies to analyze what they mainly focus on. It can be observed that most of the proxies concentrate on the object boundaries of various orientations, and there is a clear correspondence between proxies and directions. For example, proxy 1 always focuses on the right boundary of the object, while proxy 3 always focuses on the bottom when processing different images. In a nutshell, the task division among the proxies is clear and consistent.    \nWe conducted hyperparameter experiments on the number of proxies on Pascal-5i and COCO-20i to investigate how the datasets' complexities affect the optimal proxies number, and we obtained a consistent optimal proxies number (num=10). This indicates that the optimal setting of proxies is not sensitive to the data complexity. We also took into account different data complexities when organizing Figure 6: we present images including large objects, small objects, multi-categories, and multi-instances, respectively. It is recommended to refer to Figure 3 of our supplementary material for more examples.\n\n**Q3: Convergence and stability of the learning process.**   \n**A3:** To address your concern about the convergence of the learning process, we provide the convergence curves of the training process. As shown in Figure R1 and R2, our G and D can achieve a good Nash equilibrium and converge stably. To verify the stability of our method, we repeated the training of the model 5 times under the same settings (split0 of Pascal-5i with ResNet50 backbone)  and recorded its performances. The table below shows that our model is stable across different runs.\n|     | exp1 | exp2 | exp3 | exp4 | exp5 |\n|:--: |:---: |:---: |:---: |:---: |:---: |\n|mIoU | 71.4 | 71.6 | 71.2 | 71.1 | 71.4 |\n\n**Q4: Signs of over-fitting.**   \n**A4:** Overfitting is a common issue in FSS tasks. Following previous methods, we adopted a series of measures to avoid overfitting, such as freezing the backbone during training, using intermediate-level features for subsequent processing, and minimizing the number of learnable parameters. To verify this visually, we provide the mIoU curve on the validation set in Figure R3, from which we can observe that our method is invulnerable to over-fitting.\n\n**Q5: Generalization to other datasets.**   \n**A5:** We design experiments on cerebral neuron images from Electron Microscopy to validate the generalization of our method as well as its ability to handle small targets. We use Cremi-A and Cremi-B datasets [1], which contain neurons from two different brain regions. Because of their large morphological differences, we adopt neurons from Cremi-A as the base class for episodic training and neurons from Cremi-B as the novel class for testing. The model predicts the foreground region where the neurons are located, and we adopt the commonly used watershed post-processing to generate different instances of neurons.   \nThe segmentation results are shown in Figure R4. Our model is able to segment the dense and small neuron targets accurately (yellow box), proving that our method is equally effective for small targets.   \n|Method |\tF1 Score |\n|:-----:|:-----:|\n|SuperHuman [2] |\t0.756 |\n| Mala [3] |\t0.783|\n|Ours |\t0.851 |   \n\nWe also compare other neuron segmentation methods [2,3], which are trained on Cremi-A and fine-tuned with a small number of samples from Cremi-B. As shown in the table, the results show that the AMFormer outperforms these specific models, demonstrating the generalization ability of our method on other datasets (e.g., Medical images).\n\n[1] *https://cremi.org/data/*   \n[2] *Kisuk Lee et al. Superhuman accuracy on the snemi3d connectomics challenge.*   \n[3] *Jan Funke et al. Large scale image segmentation with structured loss based deep learning for connectome reconstruction.*"}, {"review_id": "GKmS2eV1oR", "reviewer": "Reviewer_ugVV", "summary": "This paper solves the problem of few shot segmentation. It differs from the existing view of solving this problem which leverages heavily on the exploration of the support samples. The core idea is to shift the framework from the support-centric to the query-centric. The problem method contains an object mining transformer and detail mining transformer. Through adversarial training, the method supress existing methods on two benchmarks.", "questions": "The paper is overall sound. Please see the weaknesses for questions.", "limitations": "The paper discusses the impact and limitations in the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+: The idea is new for few shot segmentation. Paying more attention to the query has been less inverstigated in the literature. The paper demonstrates the benefit of query-centric solution.\n\n+: The results are encouraging. \n\n+: The writing is fluent and easy to follow.", "weaknesses": "Table 1 is important for rethinking about the existing supprt-centric methods, more details about how the experimental detials to obtain Table 1 are necessary to include in the paper.\n\nHow about the convergence of the adversarial training?\n\nDoes the discriminative region localization module only deal with one class in the support label? How about the support image has more than one segmentation classes?\n\nIt is unclear why should we have to use the specific architectures in the proposed G and D, is there any evidence or insight for the designs of such networks?\n\nThe method achieves SOTA on most cases, not all the settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer ugVV   \nThanks for your valuable comments. We will explain your concerns point by point.   \n\n---\n**Q1:Experimental details about Table 1.**\nPrevious methods aimed to comprehensively exploit the support features by adopting precise support masks to generate prototypes or to constrain the matching between support and query pixels. We investigated the necessity of exhaustive support information exploration through our foreground erosion experiments on two baselines, i.e., the prototypical learning based PFENet and the affinity learning based HDMNet. Specifically, we randomly convert some parts of the ground truth mask of the support image that were 1 (foreground) to 0 (background) with specific proportions. This operation randomly discards some target features in the support image.\nThe results in Table 1 of our paper show that the mining of support information has reached saturation to some extent, which inspires us to focus on the query branch. We will include more details in the revised version following your advice.\n\n**Q2: Convergence of the adversarial training.**   \n**A2:**  To address your concern about the convergence of the learning process, we provide the convergence curves of the training process. As shown in Figure R1 and R2, our G and D can achieve a good Nash equilibrium and converge stably. \n\n**Q3: Whether discriminative region localization module can deal with multiple classes.**   \n**A3:** The AMFormer is designed for 1-way few-shot segmentation and all methods compared in our paper follow the same standard 1-way setting, i.e., there is only one category object mask in the support label. But the discriminative region localization module can well deal with multi-class support labels with only a simple adjustment. We can obtain discriminative regions of different categories $ \\{ M_{qτ}^{c}, c=1,2,... \\}$ according to Eq.(5), respectively, and then select the category with the largest value in the overlapping area of different $M_{qτ}^{c}$. This is a meaningful setting, and we will continue to explore it in future work.\n\n**Q4: Evidence or insight for the designs of our networks.**   \n**A4:** **For the design of G:** G is responsible for completely mining the foreground area under pseudo-support guidance, which can also be implemented with other FSS model structures. We adopt the multi-scale attention layers to deal with spatial-inconsistent and effectively utilize the intra-object similarity to equip potential foreground pixels with beneficial information. The effectiveness of multi-scale structure and attention layers has been proven in many semantic segmentation models such as [1]. Our ablation experiments also effectively prove the effectiveness of the design of G. **For the design of D:** We design D to mine the local object features corresponding to the predicted mask $M_e$ from G and the ground truth mask. Driven by adversarial training and diversity loss, D can adaptively capture the pattern of error-prone local features (such as boundaries) as shown in Figure 6 of the paper. These local features are targeted constrained, which enables the model to focus more on error-prone regions. However, only using the cross-entropy loss to constrain the prediction of $M_e$ will evenly back-propagate the loss gradient to all feature pixels, making it more difficult for the model to achieve fine-grained alignment.\n\n**Q5: Not SOTA on all settings**   \n**A5:** The most reliable metric for performance comparison is the average mIoU over 4 folds, on which we have achieved state-of-the-art results. As suggested, we will reconsider the wording in the revised version.\n\n[1] *SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers.*\n"}, {"review_id": "nBmiw6ngAO", "reviewer": "Reviewer_gxBQ", "summary": "This paper studies few-shot segmentation (FSS). It proposes a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The core idea is to have a object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth.\nThe proposed method outperforms SOTA on PASCAL-5 and COCO-20 benchmark.\n", "questions": "see above", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The proposed method records strong performance over SOTAs.", "weaknesses": "This is not a weakness but a question arises from not fully understanding the exp setting:\n1. When you evaluate and compare against SOTAs in the experiments presented in Table 2 and 3, are the annotations, i.e. the masks used in training the the few shot learning, the same across all methods?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer gxBQ\n\nThanks for your positive comments on the performance of AMFormer.    \n\n---\n\n**Q1: About the experiment setting.**\n**A1:** Yes, all the methods in Table 2 and Table 3 use the ground truth masks of both support and query images during training, and our method does not require any additional information for both training and testing phases, ensuring a fair comparison. "}, {"review_id": "IS7e8upLIl", "reviewer": "Reviewer_5HhF", "summary": "This paper proposes a query-centric FSS method, which first performs rough segmentation based on the support features, then performs mask propagation based on the intra-semantic similarities, which is supervised by an adversarial learning process. ", "questions": "1. More analysis of the physical rationale of the learned local proxies should be presented.\n\n2. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.", "limitations": "Check the weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method is technically sound. \n2. The experiments validates the effectiveness of the proposed method.\n3. The paper is easy to follow.", "weaknesses": "1. The novelty is relatively limited. \nThe idea of mask expansion based on the intra-semantic similarities is very similar to the SSP (Self-Support Few Shot segmentation): both methods first predicts a rough mask by identifying high-confidence matching regions and then refines the mask leveraging the rough mask as pseudo support features.  Thus, in my view, the novelty of this paper is to supervise such refining process in an adversarial manner.\n\n2. What is the physical rationale of the learned local proxies?\nThere is no explicit explication of the physical rationale of learning the local proxies. In my view, the learnable local proxies acts similarly to the learnable queries in DETR model. In  this sense, each proxy aims to capture some sort of local features in an object. I suggest presenting more theoretical analysis on the physical rationale of the learned local proxies.\n\n3. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 5HhF\n\nThanks for your valuable comments. We will explain your concerns point by point.\n\n---\n**Q1: The idea of mask expansion is similar to the SSP.**   \n**A1:** There are some previous methods, such as IPMT[1], that try to exploit the information of the query branch to mine the target, but the query guidance in both SSP and those approaches remains at a relatively coarse prototype granularity. We have cited and talked about SSP in **L44** and **L127**. We further evaluated the potential of the query branch for FSS by re-examining the role of support information and conducting a quantitative analysis of the similarities within and between objects. Then we reformulate it into a query-centric paradigm and hope to inspire more research.   \nIt should be noted that the part of our method that resembles SSP, i.e., using MAP to derive the support prototype and obtaining a coarse query mask via cosine similarity, is commonly used in FSS models like PANet[2] and PPNet[3], which is not the core of our method. As you mentioned, the main part of our method is totally different from SSP, specifically:   \n(1) **Granularity of query feature utilization:** SSP utilizes the holistic prototype-level query feature to match with query pixels, while our AMFormer further exploits pixel-level query features (pseudo support) to guide the classification of query pixels.   \n(2) **Manner of mining query mask:** SSP adopts cosine similarity to obtain the query mask and supervises it with binary cross-entropy loss, but we designed a multi-scale object mining transformer G to fully explore the target and a detail mining transformer D for details alignment. Besides, we supervise such refining processes in an adversarial manner.   \n(3) **The dependence on support information:** SSP uses the support prototype more than once when predicting the final query mask (Eq.(7) of SSP), whereas AMFormer merely requires the support prototype as the category guidance initially, and does not utilize any support information afterward, i.e., a real query-centric method.\n\n**Q2: Physical rationale of the learned local proxies.**   \n**A2:** Your analysis is reasonable, and similar to the decoder of DETR, our local feature construction process based on the detail mining transformer can also be interpreted as adaptively pooling features based on the matching of proxies and object features. However, due to the differences in source features, loss, and training paradigms, our proxies act differently from the object queries in DETR. Specifically, the object queries in DETR correspond to the absolute positional information of the whole image, while our proxies are focused on the foreground region specified by the mask. The constraints of diversity loss and the adversarial training make the proxies learn to capture specific feature gradient patterns (such as boundaries in different directions in Figure 6) in the foreground object features. The learned proxies enable our model to focus more on these error-prone regions, thus achieving fine-grained alignment of predictions with ground truth.\n\n**Q3: Ablation study on the impact of the adversarial learning process.**   \n**A3:**  We appreciate this insightful suggestion and we conduct the ablation experiment to explore the impact of the adversarial learning process. Specifically, We employ L2 loss to reduce the gap between true and false local features and train the model in a supervised learning manner. The experiment is conducted on Pascal-20i dataset using the ResNet-101 as the backbone.   \n|    | Supervised learning | Adversarial learning |\n|:--:| :------------------:| :------------------: |\n|mIoU|  69.4               |70.7                  |\n\nFrom the table, we can observe that adversarial learning significantly outperforms supervised learning in our context. We deem the reason is that supervised learning tends to focus proxies to areas covered by the predicted mask and the ground truth, resulting in the degradation of local features. Differently, adversarial training will drive detail mining transformer D to explore more diverse and differentiated regions when building local features from real and fake object features, so as to realize the mining and alignment of different details. Thank you for your advice, we will add this discussion in our revised version.\n\n[1] *Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation*.   \n[2] *PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment*.   \n[3] *Part-aware Prototype Network for Few-shot Segmentation*.   "}, {"review_id": "eSsFFNgSdg", "reviewer": "Reviewer_YwZn", "summary": "This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation.\nHowever, the idea is exactly same as SSP (ECCV2022). ", "questions": "See the weakness.", "limitations": "See the weakness.", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation.\n", "weaknesses": "The idea is exactly the same as the self-support idea of SSP (ECCV2022).\nAs shown in Figure 1, this work proposes to first generate pseudo masks on the query image to mine query objects and then use the query object feature to generate the pseudo support (self-support) to match the query feature.\nThe self-support of SSP performs the same operation to leverage the query prototype to match the query feature.\nThe novelty and contribution of this work is extremely limited.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 5HhF\n\nThanks for your comments.\n\n---\n**Q1: The idea of mask expansion is similar to the SSP.**   \n**A1:** There may be some misunderstandings about the contributions of our method. There are some previous methods, such as IPMT[1], that try to exploit the information of the query branch to mine the target, but the query guidance in both SSP and those approaches remains at a relatively coarse prototype granularity. We have cited and discussed SSP in **L44** and **L127**. We further evaluated the potential of the query branch for FSS by re-examining the role of support information and conducting a quantitative analysis of the similarities within and between objects. Then we reformulate it into a query-centric paradigm and hope to inspire more research.   \n\nIt should be noted that the part of our method that resembles SSP, i.e., using MAP to derive the support prototype and obtaining a coarse query mask via cosine similarity, is commonly used in FSS models like PANet[2] and PPNet[3], which is not the core of our method. The main part of our method is **totally different** from SSP, specifically:   \n(1) **Granularity of query feature utilization is different:** SSP utilizes the holistic prototype-level query feature to match with query pixels, while our AMFormer further exploits pixel-level query features (pseudo support) to guide the classification of query pixels.   \n(2) **Manner of mining query mask is different:** SSP adopts cosine similarity to obtain the query mask and supervises it with binary cross-entropy loss, but we designed a multi-scale object mining transformer G to fully explore the target and a detail mining transformer D for details alignment. Besides, we supervise such refining processes in an adversarial manner.   \n(3) **The dependence on support information is different:** SSP uses the support prototype more than once when predicting the final query mask (Eq.(7) of SSP), whereas AMFormer merely requires the support prototype as the category guidance initially, and does not utilize any support information afterward, i.e., a real query-centric method.\n\n\n\n[1] *Intermediate Prototype Mining Transformer for Few-Shot Semantic Segmentation*.   \n[2] *PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment*.   \n[3] *Part-aware Prototype Network for Few-shot Segmentation*.   "}, {"review_id": "CHYPwmRVqS", "reviewer": "Reviewer_7rkU", "summary": "The paper argues that mining information from the support set does not effectively improve Few-shot Segmentation (FSS) results. Instead, the paper proposes an query-centric FSS model called AMFormer, which uses adversarial learning to generate segmentation results with only rough support guidance.", "questions": "Although AMFormer achieves SOTA performance, I am confused about the results, so I give a rating of 4. I will increase the rating if the authors address all my concerns and questions.", "limitations": "More training epochs comparing with some previous methods.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The motivation in this paper is both convincing and novel.\n2) The paper is well-organized and easy to follow.\n3) The proposed AMFormer achieves SOTA performance on the two datasets.\n", "weaknesses": "Major Weaknesses\n1) Overpowered baseline model\nIn Tab5, the results of baseline model have surpassed most previous works, and the main module of this paper improves on the baseline by 5.7%. The authors should provide more details on the baseline model and explain why it achieves such good performance. (Earlier works use a similar ‘’MAP + cosine similarity’’ design [1*, 2*] and only achieve 45%~55% mIoU on Pascal)\nIn addition, in L282, the authors seem to add the cycle-consistant-transformer to the baseline model, which is not introduced in Sec. 3 and Fig. 3. I suggest that the authors use only the Discriminative Region Localization model (introduced in Sec. 3.3.2) as their baseline model.\n\nMinor Weaknesses\n1) Running time & inference speed\nThe proposed AMFormer exploits multi-scale transformers in G and D, which may increase the computational costs. Please discuss the impacts of different components brought to the model efficiency.\n2) Learnable parameters in Tab. 4\nPlease check the learnable parameters in Tab. 4, e.g. HSNet, which is inconsistent with the corresponding paper.\n3) More recent works\nMore recent works [3*, 4*, 5*] should include in Sec. 2.2 and Tab2, 3.\n4)  Generalization study of G and D\nG and D should be easily applied to other methods, e.g., using an advanced method to generate the High-Confidence Map (Mqt), rather than Discriminative Region Localization model. Therefore, it is suggested the authors to investigate more about the G and D in different methods (HDMNet, BAM) to prove its effectiveness is generalizable enough.\n\n[1*] PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment, ICCV 2019\n[2*] SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation, Transactions on Cybernetics 2020\n[3*] Mask Matching Transformer for Few-Shot Segmentation, NeurIPS 2022.\n[4*] Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning, NeurIPS 2022.\n[5*] Feature-Proxy Transformer for Few-Shot Segmentation, NeurIPS 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 7rkU\n\nThank you for acknowledging the strength of our paper. We have carefully considered your insightful and constructive comments and here are the answers to your concerns.\n\n**Q1: Performance of our baseline:**   \n**A1:**  Our baseline can attain a high performance mainly for the following two aspects of reasons:   \n(1) As discribed in L282, we add the self- and cross-attention layers into our baseline model, which can bring a significant performance gain (the cycle-consistent constraint is not included). This is because we aim to investigate the performance improvement brought by our proposed multi-scale and query-centric strategy within the object mining transformer in the ablation study, and it is unreasonable to attribute the performance improvement from ordinary attention layers to our design.   \n(2) For a fair comparison, we follow BAM and HDMNet to ensemble the prediction of a base learner, which can improve the performance. We have conducted an ablation experiment to evaluate the influence of the ensemble strategy, please refer to Table 1 of our supplementary material.   \nAs suggested, we will add a more concise baseline with only the Discriminative Region Localization module for reference in our revised version.\n\n**Q2: Running time & inference speed of the AMFormer:**   \n**A2:** Following your advice, we conduct a quantitative comparison of the efficiency of the model with different components in terms of the inference speed and the FLOPs. All models are based on the ResNet-50 backbone and tested on the Pascal-20i dataset. Inference speed for all models is tested with one single RTX 3090 GPU for a fair comparison.   \n\n|                     | OMT-1 | OMT-2 | OMT-3 | OMT-3+DMT-1 | OMT-3+DMT-2 (AMFormer) |CyCTR| HSNet|  BAM| HDMNet|\n| :---:               | :---: | :---: | :---: | :---------: | :---------:               |:---:|:----:|:---:|:-----:|\n| FLOPs(G)            | 6.7   | 8.1   | 8.6   |9.4          | 10.2                      |96.7 | 20.6 | 26.6| 10.6  |\n| FPS                 | 66.2  |53.3   | 42.0  | 42.0        | 42.0                      |15.1 |35.2  | 86.0| 36.4  |\n\n Note that OMT and DMT mean the object mining transformer and detail mining transformer, respectively, and the number suffix represents the number of attention layers. As shown in the table, the AMFormer achieves faster inference speed and fewer FLOPs than some of the methods. The high efficiency of  our model can be attributed to:    \n (1) We adopt low-dimensional features (dim=64) in all the transformer layers, reducing the computational cost and memory consumption.      \n (2) We further downsample the features extracted from the backbone before feeding them into the multi-scale structure, and these features have a low spatial resolution. Therefore, our multi-scale structure does not incur significant additional computational costs and memory overheads.    \n (3) The detail mining transformer D is not needed in the inference stage.    \n\n**Q3: Table filling error:**   \n**A3:** Thanks, we will fix it and check the manuscript carefully.\n\n**Q4: Missing related works:**   \n**A4:** Thanks, we will certainly cite and discuss these excellent works in the revised version.\n\n**Q5: Generalization study of G and D:**   \n**A5:** We appreciate your insightful suggestion. To verify the generalization of AMFormer, we integrate G and D into two advanced methods, i.e., BAM and HDMNet by taking their mask prediction as the discriminative target region $M_{qτ}$. The new models are trained in an adversarial manner.\n|model       |mIoU |   model       |mIoU | \n| :---:      |:---:|  :---:        |:---:|\n| BAM        |67.8 |  HDMNet       | 69.4|\n| BAM + G+D  |71.1 | HDMNet + G+D  | 71.5|\n\nSurprisingly, the integration of the G and D brings significant performance gains on both BAM and HDMNet. This result shows the generalizability of the query-centric AMFormer and its great potential of enhancing the performance of existing FSS models. Thanks for your advice and we will continue to explore this in future works.\n"}]

Author rebuttal:
Dear Reviewers,  

We thank all of you for your valuable insightful comments. We have carefully responded to your questions accordingly with the necessary additional experiments and analyses. We hope our responses could address all your concerns. Please let us know if you have any further advice, and we are happy to respond.

Paper457 authors

Meta-review: The paper received 6 reviews, and the authors did a serious job in addressing all the reviewer comments and suggestions. This includes running new experiments to analyze performance and run-time. Overall, the general view of the reviewers was positive and leaning to accept. The rebuttal addressed the majority of their concerns. 

Reviewer YwZn took the stance of a Strong Reject for the paper under the premise that the paper has very limited technical novelty in comparison to the recent SSP method that was evaluated against in the paper itself. The authors did provide a convincing argument delineating the differences and improvements comparing the paper with SSP.
Decision: Accept (poster)