Title: Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Computer Vision

Few-shot Segmentation
Abstract: Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention to the mining of the critical query branch. In this paper, we rethink the importance of support information and propose a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train G and D via an adversarial process, where G is optimized to generate more accurate masks approaching ground truth to fool D. We conduct extensive experiments on commonly used Pascal-5i and COCO-20i benchmarks and achieve state-of-the-art results across all settings. In addition, the decent performance with weak support labels in our query-centric paradigm may inspire the development of more general FSS models.

Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation

Abstract

Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention to the mining of the critical query branch. In this paper, we rethink the importance of support information and propose a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer ( G ) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer ( D ) to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train G and D via an adversarial process, where G is optimized to generate more accurate masks approaching ground truth to fool D . We conduct extensive experiments on commonly used Pascal5 i and COCO20 i benchmarks and achieve state-of-the-art results across all settings. In addition, the decent performance with weak support labels in our query-centric paradigm may inspire the development of more general FSS models. Code will be available at https://github.com/Wyxdm/AMNet

1 Introduction

As a fundamental computer vision task, semantic segmentation [1] has drawn decades of research interest in both academia and industry. However, the underlying driving force of fully supervised segmentation models is massive, densely labeled data. The inherent category sensitivity of these models leads to severe performance degradation when dealing with previously unseen categories. To equip segmentation models with generalization to novel classes, few-shot segmentation (FSS) has been proposed [2], which achieves segmentation of new category images (called query images) with only a handful of annotated samples (called support images) without retraining the model.

The current top-performing FSS methods focus on adequately exploring the information contained in the support samples to guide the segmentation of the query images. Among them, prototypical learning methods [3-6] concentrate support features into one or several representative prototypes to direct the classification of query pixels, while the affinity learning approaches [7-9] try to equip the query features with support semantic clue of pixel granularity. Although considerable progress has been made in the direction of probing support information, we noticed that significant intra-class diversities (as illustrated in Figure 2) between support and query images are common occurrences. In these cases, the target feature from the query image may be quite distinct from the support one.

𝐹 𝑠

𝑀𝑠

Fusion

or

Comparison

𝐹 𝑞

Prototype

Extraction

Intra-class Diversity

Prototypes

Pixel features

Prediction

𝐹 𝑠

𝑀𝑠

Holistic

Clue

𝐹 𝑞

Object Mining &

Prediction

𝐹𝑝𝑠𝑑

𝐹 𝑞

Intra-object Similarity

Rough

Guidance

Pseudo

Support

(a) Support-centric approaches

(b) Our query-centric approach

Hence there raises a natural question: is exhaustive support information extraction indispensable for query image segmentation?

We revisit the role of support information in few-shot segmentation through a pilot study. As shown in Table 1, we randomly erode support foreground features with varying proportions to explore how the support information completeness affects query segmentation accuracy. It can be observed that the performances do not degrade significantly or even surpass the complete one. We deem two main reasons contribute to the phenomenon: 1) Rough category guidance from support is sufficient to guide the accurate query image segmentation. 2) Forcing fine-grained support information may introduce redundancy, or even bias, which confuses the prediction of query images, particularly when intra-class diversities are prevalent. Motivated by the above insights, we adjust the direction and focus on the query branch. Building on the observation of intra-object similarity, i.e. , pixels within the same object are more similar than those across different objects [10, 11] (as shown in Figure 2 and Table 8, we argue that partially activated object regions can serve as cues to infer whole objects. In this paper, we construct a pseudo support image that has low intra-class variation with the query image by activating query image with a rough category-level support clue. Then we exploit the pseudo support feature to guide the query segmentation, forming a novel query-centric exploration strategy as illustrated in Figure 1.

We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. The holistic semantic clue from the labeled support image is utilized to roughly locate the most discriminative target area on the query image. (2) Local to global expansion. We utilize intra-object feature similarity to explore contextual information and then highlight the less discriminative object parts. (3) Coarse to fine alignment. The coarse activated region is further refined by eliminating the detailed difference with ground truth. Different from previous methods that focus on extracting fine-grained support information, Our method is less susceptible to intra-class diversity as only a rough holistic support guidance is needed in the first procedure to generate the pseudo support. More attention is paid to the last two query-focused procedures that enable growing incomplete local regions into accurate segmentations.

We propose an end-to-end Adversarial Mining Transformer(AMFormer) to couple procedures (2) and (3) and mutually enhance each other, which optimizes an object mining transformer G and a detail mining transformer D via an adversarial process. AMFormer aims at fully excavating the target area and aligning it in detail under the guidance of the pseudo support features. To dig out the entire object

object 2

3 parts

of object 1

object1

等

part1of object2

part2of object2

part3 of object2

region , in the G , we conduct multi-scale pixel-level correlation between the query features and pseudo support features. The exploration of the intra- and inter-scale contextual information in G progressively highlights the latent target parts that share high intra-object similarities. To effectively align the prediction details , e.g. , object boundaries, D models multiple detail proxies to accurately distinguish the subtle local differences between the ground truth and the prediction generated by G . By means of adversarial training, G is optimized to predict more accurate segmentations to fool D and only the well-learned G is required to produce accurate predictions in the inference stage.

We evaluate our AMFormer on two widely used benchmarks COCO20 i and Pascal5 i with different backbones and the AMFormer consistently sets new state-of-the-art on different settings. Moreover, our query-centric approach does not reckon on elaborate support annotations. It achieves remarkable performance even with weak support labels such as scribbles or bounding boxes, formulating a more practical FSS model. We also hope that our work can

inspire more research on query-centric FSS methods. To recap , our contributions are concluded as follows: (i) We re-evaluate the importance of support information in FSS and demonstrate that a coarse category hint suffices for accurate query segmentation. This motivated us to put forward a novel query-centric FSS method. (ii) We propose a novel Adversarial Mining Transformer (AMFormer) that optimizes an object mining transformer G and a detail mining transformer D for region expansion and detail alignment, respectively. (iii) Extensive experiments show that AMFormer significantly outperforms previous SOTAs. The conspicuous performance with weak support label also sheds light for future research of more general FSS models.

2 Related Work

2.1 Semantic Segmentation

Semantic segmentation has been widely applied to autonomous driving, medical image processing [1517] and so on. The aim of semantic segmentation is to assign each pixel within the given image to a predefined category label. The seminal FCN [1] sparked a wave of remarkable advances in semantic segmentation [18-20] based on convolutional neural networks (CNN). Various networks focus on better context exploration by enlarging the receptive field of CNN via dilated convolutions [21, 22], global pooling [23] and pyramid pooling [21, 24]. In addition to CNN-based models, the success of Vision Transformer (ViT) [25] encourages a series of transformer-based segmentation models [26-29]. For instance, maskformer [30] adopts the transformer decoder [31] to conduct the mask classification based on a set prediction objective. Many subsequent works improve this framework [32-35] and gradually formed a unified segmentation model that can address different image segmentation tasks. Despite the success, these methods cannot generalize to novel classes in the low-data regime.

2.2 Few-Shot Semantic Segmentation

Few-shot segmentation (FSS) [2] is established to segment new category images (query images) with only a few labeled samples (support images). Owing to the reliable annotations in the support set, current FSS methods mainly focus on effectively excavating support information, which can be roughly divided into two categories: prototypical learning methods and affinity learning methods. Motivated by PrototypicalNet [36], many previous works condense the support information into single [37-39, 6, 40] or multiple prototypes [41-43, 5, 4, 44-46] and then conduct feature comparison or aggregation. For example, ASGNet [3] adopts superpixel-guided clustering to adaptively construct multiple prototypes, which are concatenated with the most relevant query features to guide the pixel classification. Differently, for the seek of fine-grained support guidance, affinity learning methods [13, 47, 8, 48-50] are designed to establish pixel-level associations between support and query features via attention mechanism [7, 9] or cost volume aggregation [8, 48]. For instance, CyCTR [7] introduces a cycle-consistent attention mechanism to equip query features with relevant support information. Though achieving promising results, these methods depend heavily on support information and are prone to segmentation errors in the presence of large intra-class variations. Some methods[11, 6] try to solve this problem by mining the class-specific representation from the

query branch but remain at relatively coarse prototype granularity. In this paper, we propose a novel query-centric approach that exploits intra-object similarity to probe query targets with only overall category-level guidance from support.

3 Method

3.1 Problem Definition

Few-shot segmentation (FSS) tackles novel class object segmentation with only a few denselyannotated samples. Episodic meta-training [51] is widely used to enhance the generalization of FSS models. Specifically, the dataset is divided into the training set D train and the testing set D test . The category sets of D train and D test are disjoint, i.e. , C train ∩ C test = ∅ . A series of episodes are sampled from D train to train the model, each of which is composed of a support set S = { I k s , M k s } K k =1 and a query set Q = { I q , M q } in the K -shot setting, where I and M denote the RGB image and corresponding binary masks, respectively. Under the supervision of M q , the model is trained to predict the query mask conditioned on the S and I q . After that, the trained model is evaluated on the episodes sampled from D test without further optimization.

3.2 Revisiting of Transformer-based Feature Aggregation

Transformer layers [52] are widely used in computer vision tasks for feature aggregation. The critical component of transformer layer is the attention mechanism that enables the long-range modeling capability. Specifically, the attention layer is first applied to compute the attention weight of the source feature sequence S ∈ R N 1 × C and the target sequence T ∈ R N 2 × C , where the N 1 , N 2 denote the length of sequences and C is the embedding dimension, formulaic as:

in which W Q and W K ∈ R C × d are learnable linear projections and √ d is the scaling factor. The source information is adaptively transported to the target sequence according to attention weight, and a feed-forward network ( FFN ) is further applied to transform the fused features:

where the W V is the linear linear projection and ̂ T is the enhanced target feature sequence. We can abbreviate the feature aggregation process as:

Note that when T = S , the FeatAgg ( , ) explores contextual information within the feature sequence, i.e. , acts as the self-attention mechanism. The above processes are implemented with the multi-head mechanism to enhance performance further.

3.3 Adversarial Mining Transformer

3.3.1 Overview

The proposed framework tailored for query-centric FSS consists of three procedures, i.e. , 1) discriminative region localization, 2) local to global region expansion, 3) coarse to fine mask alignment. We roughly locate the object of novel class under a rough support guidance via nonparametric cosine similarity in procedure 1). The resulting incompleted region is expanded by the object miner G to cover the whole target in procedure 2). The detail miner D in procedure 3) is tasked with identifying local discrepancies between the ground truth and the expanded mask generated by G , thereby aligning the coarse prediction in detail. We optimize the G and D via an adversarial process and forming the end-to-end AMNet as illustrated in Fig. 3. The details are as follows.

3.3.2 Discriminative region localization

Considering the significant intra-class diversity between samples, we contend that support information is more suitable as overall guidance indicating the novel class rather than detailed reference. Given

pooling

Cosine

…

Cross

attention

…

…

Real/Fake ?

Cross

Cross

attention

Shared

Backbone

Support

Feature

𝑭𝒔

Query

Feature

𝑭𝒒

Support Label

𝑴𝒔

Support Image

Conv

Head

Query Features

𝑭𝒒

Pseudo

Support Features

𝑭𝒑𝒔𝒅

Ground Truth

Prediction

𝑴𝒆

Aggregated

feature

෡

𝑭𝒒

Discriminative Region Localization

Local to Global Expansion

Coarse to Fine Alignment

High-Confidence

𝑴𝒒𝝉

Query

Feature

𝑭𝒒

Real

Object Feature

Fake

Object Feature

Local

Proxies

𝑷

Real Local

Features

𝜴𝒓

Fake Local

Features

𝜴𝒇

fuse

𝝉

Cross

Cross

𝒒

𝒌, 𝒗

Select   FC

Select   FC

Object Mining Transformer G

Detail Mining Transformer D

the support image features F s ∈ R H × W × C and corresponding mask M s ∈ R H × W , where H and W describe the feature size and C is the feature dim, we apply mask average pooling (MAP) to obtain the mean target feature F h ∈ R 1 × C that serves as the holistic support guidance, formally:

We exploit the F h to indicate the target region in the F q based on the cosine similarity:

A relatively high threshold τ is set to locate the most class-relevant regions and suppress activation to backgrounds. In our experiment, τ is set to 0.7.

3.3.3 Local to global region expansion

Adaptive feature aggregation. The high-confidence object regions M qτ ∈ R H × W obtained by Eqn (5) are incomplete. We introduce the object mining transformer G to expand the local region to the entire target area based on the intra-object similarity. Specifically, G aims at aggregating the representative target features from the pseudo support to less discriminative object parts in the query, formally,

where the F : R H × W × C ↦→ R HW × C is the spatial flatten operation. F psd , which serve as the source sequence in Eqn (1), are essentially query features filtered by M qτ :

Homogeneous feature guidance enables our aggregation to circumvent the effects of intra-class variance. Note that the original attention mechanism applies Softmax activation along the source sequence dimension. Directly adopting this scheme in our aggregation makes the query background features inevitably attend to foreground features since all the non-zero features in the pseudo support belong to the foreground. Inspired by [53, 13], we adjust to implement Softmax along the axis of F q . In this way, the discriminative object features are more likely to be aggregated target areas that share high intra-object similarities with F psd .

Cross-scale information transportation. We observe that in many cases, the query image contains different objects with varying scales, which are to be segmented in other episodes. To handle this spatial inconsistency, we construct the G in a multi-scale form. Concretely, we follow [13] to establish the hierarchical query features { F q,l } L l =1 with down-sampling and self-attention layers, in which

The corresponding pseudo support features { F psd,l } L l =1 is obtained by the Hadamard product of { F q,l } L l =1 and downsampled M qτ . Then Eqn (6) is implemented independently in each scale to obtain the aggregated query features { ̂ F q,l } L l =1 . Finally, we fuse the multi-scale aggregated query features in a top-down manner as down in [14], specifically,

The fused features from the last stage ̂ F ′ q, 1 ∈ R H × W × C are exploited to predict the expanded target area M e ∈ R H × W via a small convolution head. Benefiting from the adaptive feature aggregation and cross-scale information transportation in the object mining transformer G , less discriminative target parts can be highlighted in the query image, forming the expanded object mask M e .

3.3.4 Coarse to fine mask alignment

Detail mining transformer D is designed to discriminate subtle differences between the ground truth M gt and the expanded object mask M e generated by G , since M e adequately covers the object but still exhibits misalignments in fine-grained details, e.g., boundaries. By training G and D via an adversarial process, G is optimized to generate more accurate target masks approaching ground truth to fool D , thus achieving coarse to fine prediction alignment.

To capture comprehensive details, D models multiple local proxies, each of which is tasked with exploring the object features respectively specified by M e and M gt to obtain a pair of local features. The most different pair is selected to output real/fake results. Concretely, we concatenate the learnable proxies as a feature sequence P ∈ R N × C , where N denotes the number of proxies. We perform feature adaptive aggregation through the attention mechanism to construct local features, In specific,

where the 'fake' object features specified by predicted mask ( F q ⊙ M e ) serves as the source sequence of aggregation process. Note that the M e is not binarized. Similarly, local features originate from 'real' object features are obtained by:

Finally, we calculate the cosine similarity among the real local features Ω r = { ω i f } N i =1 and the fake ones Ω f = { ω i f } N i =1 . The most different pair ( ω k f , ω k r ) is selected and fed into a fully-connected layer to predict the fake/real results for adversarial training. By this way, D is optimized to discriminate detailed local differences, while the G will generate more precise masks to fool D by adjusting itself.

Since there is no explicit supervision, different proxies may focus on the same object part. We impose a diversity loss to avoid this degradation by expanding the discrepancy among local features:

̸

The intuition of Eqn (12) is trivial. If different proxies focus on the same object region, L div will be large and adjust the learning of proxies.

3.3.5 Training and Inference

Training loss. Our architecture consists of a generative part, i.e. , object mining transformer G and a discriminative part, i.e. , detail mining transformer D . The D judges whether the mask is real (ground truth) or fake (generated by G ) by mining the detailed features of the object framed by the masks. To fool D , G is supposed to predict a more accurate mask approaching the ground truth. We alternately train two parts to achieve mutual promotion. When training D , the parameters of G are frozen, and the loss function for D is formulated as:

where the FC : Ω → [0 , 1] denotes the fully-connected layers that output the real/fake results, and the ω k r , ω k f ∈ Ω denote the most different pair of local features from the object specified by ground truth M gt and the predicted mask M e , respectively. λ div denotes the weight of diversity loss and we

set it to 0.1 in experiments. Similarly, D is not optimized when training G . Following [13], we also include a KL (Kullback-Leibler) divergence loss between the correlation maps of adjacent stages to distill informative semantic cues of earlier stages to refine the segmentation quality. Please refer to [13] for more details. The overall loss function for G is:

Where the BCE is the Cross-entropy loss. After training, only the well-learned G is needed in the inference stage.

K-shot inference. When K support samples { I k s , M k s } K k =1 are available, different discriminative target regions { M i qτ } K i =1 can be obtained from different support images via Eqn (4) and Eqn (5). We adopt the union of all the activated regions as the initial object region, i.e. , M qτ = M 1 qτ ∩ M 2 qτ ∩ · · · ∩ M K qτ . The comprehensive M qτ originating from multiple supports is more robust to pose differences, occlusion, etc., than that from a single support image, thereby boosting the performance.

4 Experiments

4.1 Datasets and Evaluation Metrics

We evaluate the proposed AMFormer on two commonly used few-shot segmentation benchmarks, Pascal5 i [2] and COCO20 i [56]. Pascal5 i is constructed based on the PASCAL VOC 2012

dataset [57] and additional annotations from SBD [58]. Following previous works [14, 55], we equally divide the 20 categories into four splits, three of which for training and the rest one for testing. COCO20 i is a larger benchmark based on MSCOCO dataset [59], the 80 categories of which are partitioned into four splits for cross-validation as down in [14]. We randomly sampled 1000 episodes from the testing split for evaluation. Following common prectices [37, 14, 55, 38], we adopt mean intersection-over-union (mIoU) and foreground-background intersection-over-union (FBIoU) as evaluation metrics.

4.2 Implementation Details

We adopt ResNet-50 and ResNet-101 [60] as the backbone network in our experiment. Following previous works [14, 7, 13], we concatenate the features from the 3 rd and 4 th blocks of backbone and exploit a 1 × 1 convolution layer to generate F s and F q of middle-level to avoid overfitting. The number of attention layers in the G and D are set to 1 and 2, respectively. We employ the same data augmentation setting as [14]. Since the G and D of AMFormer are trained alternately, we increase the number of training epochs to 300 for Pascal5 i and 75 for COCO20 i , and set the batch sizes as 8 and 4, respectively. AdamW [61] optimizer with poly learning rate decay is used to train both the G and D . The initial learning rate is set to 1 e -4 and the weight decay is 1 e -2 . It should be noted that we adopt a base learner to filter the categories that appear during the training process for a fair comparison with previous works [55, 13]. For more details please refer to the Supplementary Material . Our approach is implemented using PyTorch and all experiments are conducted on 4 NVIDIA GeForce RTX 3090 GPUs.

4.3 Comparison with State-of-the-Art Methods

We present the comparison of our method with previous FSS methods on Pascal5 i and COCO20 i datasets in Table 2 and Table 3. It can be observed that the proposed AMFormer significantly outperforms previous advanced approaches and achieves new state-of-the-art results under all settings. Specifically, on Pascal5 i , our AMFormer achieves 70.4% and 73.2% mIoU when using ResNet-50 as the backbone for 1-shot and 5-shot settings, surpassing the most competitive HDMNet [13] by 1.0% and 1.4%, respectively. With ResNet-101 backbone, our method outperforms the previous best results [6] by 2.8% (1-shot) and 1.6% (5-shot). We can obtain additional improvement when using larger backbone network, which demonstrates the scalability of the AMFormer. We attribute this performance gain to better intra-object similarity within the more informative features. As for the more complicated COCO20 i , our approach also exhibits superior performances compared to other methods, demonstrating its competitiveness on complex data. Besides, Table 4 gives the comparison with previous methods in terms of FBIoU on Pascal5 i using ResNet-50 backbone. AMFormer also outperforms all of the previous works by a considerable margin. Qualitative results are shown in Figure 4, please refer to Supplementary Material for analysis.

support

GT

𝑴𝒒𝝉

𝑴𝒆

w/o

D

𝑴𝒆

w/

D

4.4 Ablation Study

As shown in Table 5, a series of ablation studies are conducted on the first split of Pascal5 i with ResNet-101 backbone to analyze each component of the proposed AMFormer. Note that the first line of Table 5 is the result of our ablation baseline. The baseline adopts self-attention within the query features for feature parsing, and cross-attention to aggregate support information into query features as done in [7].

Effectiveness of object mining transformer G . We first construct a naive single-scale G as the 2 n d row of Table 5. We can observe a significant performance lift, i.e. , 1.5% in mIoU. This improvement demonstrates the effectiveness of the proposed query-centric strategy, which is built upon intra-object similarity and thus is less affected by intra-class diversity. The multi-scale implementation of G further brings a 3.4% improvement in mIoU, and it already achieves a decent performance. It shows the importance of multi-scale feature aggregation in dealing with objects with dramatically changing scales in the query image.

Effectiveness of detail mining transformer D . The comparison between the 3 rd and the 5 th row of Table 5 shows that the combination of D improves the performance by 1.2% mIoU on the basis of G . We attribute this performance gain to the exploration and distinction of local features in the D , which encourages G to pay more attention to these ambiguous regions, e.g. , object boundaries. Compared with the pixel-level supervision given by Cross-entropy loss, the proposed part-level adversarial training can incorporate local region context information to guide accurate segmentation. Note that without L div , the performance improvement brought by D drops to 0.2%. This phenomenon is reasonable because in the absence of L div , local proxies tend to degrade to focus on the same region, leaving some details unexplored. support query activation activation

Investigation of the local proxies. We first visualize the region activated by the learnable proxies to analyze what they mainly focus on. As shown in Figure 5, we observe that the highlighted area mainly lies on the boundary of the target and different proxies correspond to different directions. It confirms that the local proxies can well capture the local misalignment of the relatively coarse prediction from G , and then G is optimized to pay more attention to these ambiguous areas under the direction of D . In addition, we investigate the impact of the number of proxies on performance. As reported in Table 6, using more proxies can gradually improve the performance until the number achieves 10. More proxies no longer bring additional improvement. This result is expected, as too many proxies would learn redundant patterns.

Discussion on the query-centric FSS paradigm. Since the FSS models are trained in the extremely low data regime

image

image

(baseline)

(ours)

image

part 1

part 2

part 3

part 4

boat

bird

sofa

person

with the backbone not optimized, the feature space is not well aligned for novel classes. In this situation, the novel class data distribution usually has low coverage in the feature space, i.e. , object features of the same category but different image instances may be far apart. On the contrary, as we can observe in Table 8, pixel features within the same object are closer to each other than those across objects. In a nutshell, intra-object similarity is more reliable than intra-category but interobject similarity. As shown in Figure 6, we visualize the attention weight of query features between support target and pseudo support, respectively. It shows that support targets tend to mistakenly activate background categories, while the pseudo support can well excavate the full object attribute to intra-object similarity.

Owing to the reduced reliance on support information, query-centric FSS methods could achieve remarkable performance with more weak support annotations. Table 7 shows the results of the proposed AMFormer on the first split of Pascal5 i with bounding boxes or scribbles as support labels. The promising results demonstrate the feasibility of more general FSS segmentation models.

4.5 Broader Impact and Limitations.

We proposed a novel query-centric FSS paradigm that shifts the research focus from support to query features. This is a new perspective that may inspire the development of more general FSS models that can be adopted in different tasks such as video object segmentation [62-64] or open-vocabulary segmentation [65, 66]. Although our AMFormer achieves remarkable performance, the number of training epochs is larger than some of previous approaches since the G and D in the AMFormer are trained alternately.

5 Conclusion

In this paper, we propose a novel query-centric FSS method, i.e. , Adverasial Mining Transformer (AMFormer), which can achieve accurate query segmentation with only rough support guidance. Extensive experimental results demonstrate the superiority of our method. The decent performance with weak support labels also demonstrates the potential of the query-centric FSS paradigm.

6 Acknowledgments

This work was partially supported by the National Defense Basic Scientific Research Program of China (Grant JCKY2021601B013).

References

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. arXiv preprint arXiv:1709.03410 , 2017.

Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8334-8343, 2021.

Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross-guided learning for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8312-8321, 2021.

Jie Liu, Yanqi Bao, Guo-Sen Xie, Huan Xiong, Jan-Jakob Sonke, and Efstratios Gavves. Dynamic prototype convolution network for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11553-11562, 2022.

Yuanwei Liu, Nian Liu, Xiwen Yao, and Junwei Han. Intermediate prototype mining transformer for few-shot semantic segmentation. arXiv preprint arXiv:2210.06780 , 2022.

Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycleconsistent transformer. Advances in Neural Information Processing Systems , 34:21984-21996, 2021.

Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE/CVF international conference on computer vision , pages 6941-6952, 2021.

Xinyu Shi, Dong Wei, Yu Zhang, Donghuan Lu, Munan Ning, Jiashun Chen, Kai Ma, and Yefeng Zheng. Dense cross-query-and-support attention weighted mask aggregation for fewshot segmentation. In European Conference on Computer Vision , pages 151-168. Springer, 2022.

Robert M. Haralick and Linda G. Shapiro. Image segmentation techniques. Computer Vision, Graphics, and Image Processing , 29(1):100-132, 1985.

Qi Fan, Wenjie Pei, Yu-Wing Tai, and Chi-Keung Tang. Self-support few-shot semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XIX , pages 701-719. Springer, 2022.

Yongfei Liu, Xiangyi Zhang, Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In European Conference on Computer Vision , pages 142-158. Springer, 2020.

Bohao Peng, Zhuotao Tian, Xiaoyang Wu, Chenyao Wang, Shu Liu, Jingyong Su, and Jiaya Jia. Hierarchical dense correlation distillation for few-shot segmentation. arXiv preprint arXiv:2303.14652 , 2023.

Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence , (01):1-1, 2020.

Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, and Yongdong Zhang. Lesionaware transformers for diabetic retinopathy grading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10938-10947, 2021.

Li Wangkai, Li Zhaoyang, Sun Rui, Mai Huayu, Luo Naisong, Yuan Wang, Pan Yuwen, Xiong Guoxin, Lai Huakai, Xiong Zhiwei, et al. Maunet: Modality-aware anti-ambiguity u-net for multi-modality cell segmentation. In Competitions in Neural Information Processing Systems , pages 1-12. PMLR, 2023.

Rui Sun, Huayu Mai, Naisong Luo, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Structuredecoupled adaptive part alignment network for domain adaptive mitochondria segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 523-533. Springer, 2023.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2881-2890, 2017.

Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In Proceedings of the European conference on computer vision (ECCV) , pages 418-434, 2018.

Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention , pages 234-241. Springer, 2015.

Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence , 40(4):834-848, 2017.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 801-818, 2018.

Wei Liu, Andrew Rabinovich, and Alexander C Berg. Parsenet: Looking wider to see better. arXiv preprint arXiv:1506.04579 , 2015.

Maoke Yang, Kun Yu, Chi Zhang, Zhiwei Li, and Kuiyuan Yang. Denseaspp for semantic segmentation in street scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3684-3692, 2018.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7262-7272, 2021.

Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6881-6890, 2021.

Yifan Zhang, Bo Pang, and Cewu Lu. Semantic segmentation by early region proxy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1258-1268, 2022.

Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems , 34:10326-10338, 2021.

Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances in Neural Information Processing Systems , 34, 2021.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision , pages 213-229. Springer, 2020.

Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. arXiv preprint arXiv:2112.01527 , 2021.

Hao Zhang, Feng Li, Huaizhe Xu, Shijia Huang, Shilong Liu, Lionel M Ni, and Lei Zhang. Mpformer: Mask-piloted transformer for image segmentation. arXiv preprint arXiv:2303.07336 , 2023.

Naisong Luo, Yuwen Pan, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Camouflaged instance segmentation via explicit de-camouflaging. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17918-17927, 2023.

Rui Sun, Huayu Mai, Tianzhu Zhang, and Feng Wu. Daw: Exploring the better weighting function for semi-supervised semantic segmentation. In Advances in Neural Information Processing Systems , 2023.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics , 50(9):38553865, 2020.

Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9197-9206, 2019.

Leilei Cao, Yibo Guo, Ye Yuan, and Qiangguo Jin. Prototype as query for few shot semantic segmentation. arXiv preprint arXiv:2211.14764 , 2022.

Siyu Jiao, Gengwei Zhang, Shant Navasardyan, Ling Chen, Yao Zhao, Yunchao Wei, and Humphrey Shi. Mask matching transformer for few-shot segmentation. arXiv preprint arXiv:2301.01208 , 2022.

Chunbo Lang, Binfei Tu, Gong Cheng, and Junwei Han. Beyond the prototype: Divide-andconquer proxies for few-shot segmentation. arXiv preprint arXiv:2204.09903 , 2022.

Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. In European Conference on Computer Vision , pages 763-778. Springer, 2020.

Yuanwei Liu, Nian Liu, Qinglong Cao, Xiwen Yao, Junwei Han, and Ling Shao. Learning non-target knowledge for few-shot semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11573-11582, 2022.

Jian-Wei Zhang, Yifan Sun, Yi Yang, and Wei Chen. Feature-proxy transformer for few-shot segmentation. arXiv preprint arXiv:2210.06908 , 2022.

Atsuro Okazawa. Interclass prototype relation for few-shot segmentation. In Computer VisionECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 362-378. Springer, 2022.

Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In European Conference on Computer Vision , pages 36-52. Springer, 2022.

Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang, Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In Computer VisionECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIII 16 , pages 730-746. Springer, 2020.

Sunghwan Hong, Seokju Cho, Jisu Nam, Stephen Lin, and Seungryong Kim. Cost aggregation with 4d convolutional swin transformer for few-shot segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX , pages 108-126. Springer, 2022.

Zhitong Xiong, Haopeng Li, and Xiao Xiang Zhu. Doubly deformable aggregation of covariance matrices for few-shot segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XX , pages 133-150. Springer, 2022.

Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7183-7192, 2023.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Samuel L Smith, David HP Turban, Steven Hamblin, and Nils Y Hammerla. Offline bilingual word vectors, orthogonal transformations and the inverted softmax. arXiv preprint arXiv:1702.03859 , 2017.

Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13979-13988, 2021.

Chunbo Lang, Gong Cheng, Binfei Tu, and Junwei Han. Learning what not to segment: A new perspective on few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8057-8067, 2022.

Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 622-631, 2019.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88(2):303-338, 2010.

Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European conference on computer vision , pages 297-312. Springer, 2014.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision , pages 740-755. Springer, 2014.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Ho Kei Cheng, Yu-Wing Tai, and Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems , 34:11781-11794, 2021.

Rui Sun, Yuan Wang, Huayu Mai, Tianzhu Zhang, and Feng Wu. Alignment before aggregation: trajectory memory retrieval network for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1218-1228, 2023.

Rui Sun, Naisong Luo, Yuan Wang, Yuwen Pan, Huayu Mai, Zhe Zhang, and Tianzhu Zhang. 1st place solution for youtubevos challenge 2022: Video object segmentation. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshops , 2022.

Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision , pages 2002-2010, 2017.

Mengde Xu, Zheng Zhang, Fangyun Wei, Han Hu, and Xiang Bai. Side adapter network for open-vocabulary semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2945-2954, 2023.

7 More Details for Multi-scale Object Mining Transformer.

In the object mining transformer G , we alternately use down-sampling layers and self-attention layers to construct hierarchical query features. And the corresponding pseudo support features are obtained by the Hadamard product of the downsampled M qτ , specifically,

where the F : R H × W × C ↦→ R HW × C is the spatial flatten operation and Down denotes the downsampling layers, which is implemented with convolutional layers of double strides. In this way, we obtain multi-scale query features F q,l ∈ R H 2 l -1 × W 2 l -1 × C , l = 1 , . . . , L and pseudo support features F psd,l = F q,l ⊙ φ ( M qτ ) . ( φ is bilinear interpolation operation). We perform feature aggregation within each scale to enable contextual information exploration, thus avoiding the spatial inconsistency in the query image. L is set to 3 in our experiments. The embedding dim is set to 64, and the number of head is set to 4 in all the attention layers.

8 Detailed Experimental Settings

To achieve a fair comparison with previous methods [55, 13], we adopt the ensemble strategy following BAM [55] to filter the base categories seen during training. Specifically, a base learner is trained using the training splits in a fully supervised manner, and the learned base learner is used to explicitly predict the targets of base classes. PSPNet [18] is adopted as the base learner in all of our experiments, and we follow the BAM [55] to ensemble the prediction of the base learner and the proposed AMFormer. Our code is available at https://github.com/Wyxdm/AMNet

Wealso conduct an additional ablation experiment to evaluate the influence of the ensemble strategy as shown in Table 9. We can observe that the ensemble strategy can incrementally improve performance. It should be noted that our AMFormer can also surpass previous state-of-the-art methods (IPMT [6]) without the ensemble strategy.

9 Dataset Settings

We respectively divided Pascal-5 i and COCO-20 i into four splits following [14] for cross-validation. In Table 10 and Table 11, we provide the detailed split settings.

10 More Experimental Results

10.1 Quantitative analysis of intra-object similarity.

We compute the average pairwise pixel similarity from the same object (intra-object) and different objects from the support and query images of the same category (inter-object) using the cosine

similarity. Note that the pixel features that we used to compute the similarity are middle-level features F s and F q as described in the L255-L256 in the original manuscript. The quantitative results of different categories are provided in Table 12 and Table 13. From the tables, we can observe that the intra-object similarity is at least one order of magnitude higher than the inter-object similarity. This demonstrates the superiority of the query-centric approach relying on intra-object similarity over support-centric methods that rely on inter-object similarity.

10.2 More visualization results.

Visualization of segmentation at different stages. We tackle query-centric FSS by applying three intuitive procedures. (1) Discriminative region localization. (2) Local to global expansion. (3) Coarse to fine alignment. To illustrate the effects of the above three steps, in Figure 7, we visualize the outcomes of different stages. Procedure (1) can only roughly local the discriminative region of the target ( 2 nd column of Figure 7). In procedure (2), the object mining transformer G exploits the intra-object similarity to explore multi-scale contextual information, thus highlighting the whole object ( 3 rd column of Figure 7). Segmentation from G can roughly cover the entire target but there still exist misalignments as shown in the yellow boxes in the 3 rd column of Figure 7. The detail mining transformer D is responsible for discriminating those detailed misalignments, i.e. , procedure (3). The proposed AMFormer couples procedures (2) and (3) via adversarial training. In this way, the G can be optimized to generate more accurate segmentations( 4 th column of Figure 7) to fool D .

Visualization of activation maps. We visualize the attention weight of query features between the support target and pseudo support. Specifically, the attention matrix S ∈ R H q W q × H s W s is computed according to the Eqn (1) of the original manuscript. Then we compute the average activation of each query pixel over all support foreground pixels:

where the M s is the (pseudo) support mask. In the baseline, the S is oriented from the cross attention between the query features and the support features (support-centric). While in our query-centric

AMFormer, the S is computed from the pseudo support and the query features. As shown in Figure 8, the support targets not only cannot fully activate the target in the query image, but also frequently activates the background categories. While the pseudo support can well excavate the full object attribute to intra-object similarity.

Visualization of local proxies. To explore the regions of interest for learnable local proxies, Figure 9 visualizes the activation maps of a part of proxies. It can be observed that different proxies tend to focus on different local regions, and most proxies attend to the boundaries of the object, which is usually the most ambiguous region. In addition, a particular proxy consistently focuses on the boundaries in a specific direction, e.g. , proxy 1 always activates the right border ( 5 th column). Through the cooperation of multiple proxies, our detail mining transformer G can effectively detect the detailed local differences between the prediction of the object mining transformer G and the ground truth. By means of adversarial training, G will produce more accurate segmentations, especially in ambiguous regions, by adjusting itself to fool D.

support

GT

𝑴𝒒𝝉

𝑴𝒆

w/o

D

𝑴𝒆

w/

D

Local to global expansion

Coarse to fine alignment

767-300

767-300

767-300

767-300

support

image

query

image

activation

(baseline)

activation

(ours)

image

proxy 1

proxy 2

proxy 3

proxy 4

monitor

potted plant

dog

motorbike

aeroplane

train

cow

国

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper argues that mining information from the support set does not effectively improve Few-shot Segmentation (FSS) results. Instead, the paper proposes an query-centric FSS model called AMFormer, which uses adversarial learning to generate segmentation results with only rough support guidance.\n\nSTRENGTHS:\n1) The motivation in this paper is both convincing and novel.\n2) The paper is well-organized and easy to follow.\n3) The proposed AMFormer achieves SOTA performance on the two datasets.\n\nWEAKNESSES:\nMajor Weaknesses\n1) Overpowered baseline model\nIn Tab5, the results of baseline model have surpassed most previous works, and the main module of this paper improves on the baseline by 5.7%. The authors should provide more details on the baseline model and explain why it achieves such good performance. (Earlier works use a similar ‘’MAP + cosine similarity’’ design [1*, 2*] and only achieve 45%~55% mIoU on Pascal)\nIn addition, in L282, the authors seem to add the cycle-consistant-transformer to the baseline model, which is not introduced in Sec. 3 and Fig. 3. I suggest that the authors use only the Discriminative Region Localization model (introduced in Sec. 3.3.2) as their baseline model.\n\nMinor Weaknesses\n1) Running time & inference speed\nThe proposed AMFormer exploits multi-scale transformers in G and D, which may increase the computational costs. Please discuss the impacts of different components brought to the model efficiency.\n2) Learnable parameters in Tab. 4\nPlease check the learnable parameters in Tab. 4, e.g. HSNet, which is inconsistent with the corresponding paper.\n3) More recent works\nMore recent works [3*, 4*, 5*] should include in Sec. 2.2 and Tab2, 3.\n4)  Generalization study of G and D\nG and D should be easily applied to other methods, e.g., using an advanced method to generate the High-Confidence Map (Mqt), rather than Discriminative Region Localization model. Therefore, it is suggested the authors to investigate more about the G and D in different methods (HDMNet, BAM) to prove its effectiveness is generalizable enough.\n\n[1*] PANet: Few-Shot Image Semantic Segmentation with Prototype Alignment, ICCV 2019\n[2*] SG-One: Similarity Guidance Network for One-Shot Semantic Segmentation, Transactions on Cybernetics 2020\n[3*] Mask Matching Transformer for Few-Shot Segmentation, NeurIPS 2022.\n[4*] Singular Value Fine-tuning: Few-shot Segmentation requires Few-parameters Fine-tuning, NeurIPS 2022.\n[5*] Feature-Proxy Transformer for Few-Shot Segmentation, NeurIPS 2022.\n\nQUESTIONS:\nAlthough AMFormer achieves SOTA performance, I am confused about the results, so I give a rating of 4. I will increase the rating if the authors address all my concerns and questions.\n\nLIMITATIONS:\nMore training epochs comparing with some previous methods.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

comments: SUMMARY:
This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation.
However, the idea is exactly same as SSP (ECCV2022).

STRENGTHS:
This work proposes a new query-centric FSS model Adversarial Mining Transformer for few-shot segmentation.

WEAKNESSES:
The idea is exactly the same as the self-support idea of SSP (ECCV2022).
As shown in Figure 1, this work proposes to first generate pseudo masks on the query image to mine query objects and then use the query object feature to generate the pseudo support (self-support) to match the query feature.
The self-support of SSP performs the same operation to leverage the query prototype to match the query feature.
The novelty and contribution of this work is extremely limited.

QUESTIONS:
See the weakness.

LIMITATIONS:
See the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a query-centric FSS method, which first performs rough segmentation based on the support features, then performs mask propagation based on the intra-semantic similarities, which is supervised by an adversarial learning process.\n\nSTRENGTHS:\n1. The proposed method is technically sound. \n2. The experiments validates the effectiveness of the proposed method.\n3. The paper is easy to follow.\n\nWEAKNESSES:\n1. The novelty is relatively limited. \nThe idea of mask expansion based on the intra-semantic similarities is very similar to the SSP (Self-Support Few Shot segmentation): both methods first predicts a rough mask by identifying high-confidence matching regions and then refines the mask leveraging the rough mask as pseudo support features.  Thus, in my view, the novelty of this paper is to supervise such refining process in an adversarial manner.\n\n2. What is the physical rationale of the learned local proxies?\nThere is no explicit explication of the physical rationale of learning the local proxies. In my view, the learnable local proxies acts similarly to the learnable queries in DETR model. In  this sense, each proxy aims to capture some sort of local features in an object. I suggest presenting more theoretical analysis on the physical rationale of the learned local proxies.\n\n3. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.\n\nQUESTIONS:\n1. More analysis of the physical rationale of the learned local proxies should be presented.\n\n2. Ablation study on the impact of the adversarial learning process, adversarial learning vs direct supervised learning by comparison between the real and fake local features.\n\nLIMITATIONS:\nCheck the weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

comments: SUMMARY:
This paper studies few-shot segmentation (FSS). It proposes a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The core idea is to have a object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth.
The proposed method outperforms SOTA on PASCAL-5 and COCO-20 benchmark.

STRENGTHS:
+ The proposed method records strong performance over SOTAs.

WEAKNESSES:
This is not a weakness but a question arises from not fully understanding the exp setting:
1. When you evaluate and compare against SOTAs in the experiments presented in Table 2 and 3, are the annotations, i.e. the masks used in training the the few shot learning, the same across all methods?

QUESTIONS:
see above

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25

comments: SUMMARY:
This paper solves the problem of few shot segmentation. It differs from the existing view of solving this problem which leverages heavily on the exploration of the support samples. The core idea is to shift the framework from the support-centric to the query-centric. The problem method contains an object mining transformer and detail mining transformer. Through adversarial training, the method supress existing methods on two benchmarks.

STRENGTHS:
+: The idea is new for few shot segmentation. Paying more attention to the query has been less inverstigated in the literature. The paper demonstrates the benefit of query-centric solution.

+: The results are encouraging. 

+: The writing is fluent and easy to follow.

WEAKNESSES:
Table 1 is important for rethinking about the existing supprt-centric methods, more details about how the experimental detials to obtain Table 1 are necessary to include in the paper.

How about the convergence of the adversarial training?

Does the discriminative region localization module only deal with one class in the support label? How about the support image has more than one segmentation classes?

It is unclear why should we have to use the specific architectures in the proposed G and D, is there any evidence or insight for the designs of such networks?

The method achieves SOTA on most cases, not all the settings.

QUESTIONS:
The paper is overall sound. Please see the weaknesses for questions.

LIMITATIONS:
The paper discusses the impact and limitations in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a concept for solving the few-shot segmentation problem using weakly supervised learning concept. The method relies on weak support labels like scribbles or bounding boxes that highlight the object regions in the query. The segmentation is achieved through two blocks - the object mining transformer and detail mining transformer - during training, while only the object mining transformer is needed during inference. The proposed approach outperforms other methods, as demonstrated by the results and ablation studies. The writing is clear and helps readers understand the proposed concept effectively.\n\nSTRENGTHS:\nThe authors have developed a solution that can solve the few-shot segmentation problem under more practical scenarios. This means that only partial annotation on the support images is required, making it a practical tool. Additionally, the query-centric concept can lessen the reliance on support images and overcome the differences between support and target images of the same class, making the solution more applicable. Although the detail mining transformer approach proposed with part-level adversarial training can slow down training, it is not necessary during inference. Lastly, the solution works well as AMFormer outperforms previous works significantly.\n\nWEAKNESSES:\n1. It seems that the model complexity is quite high due to the large-scale backbone, transformer-based object mining model, and multi-scale structure.\n2. I am unsure about the task division among the proxies, and it seems that the number of required proxies might depend on the complexity of the data. I hope that the design of this part can be made more clear and understandable so that it would be easier to debug in case of any issues.\n\nQUESTIONS:\n1. Incorporating adversarial training may affect the convergence and stability of the learning process. Can you provide more information on this?\n2. Do you notice any signs of over-fitting in the suggested method?\n3. How does the model complexity compare to the competing approaches in Table 2 and Table 3?\n\nLIMITATIONS:\nHave you considered how well the query-centric method will perform when dealing with small targets, like in satellite images? Specifically, I am wondering about its generalizability for different datasets, ex: satellite images or medical images.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}