Mask Matching Transformer for Few-Shot Segmentation
Authors: Siyu Jiao
Gengwei Zhang
Shant Navasardyan
Ling Chen
Yao Zhao
Yunchao Wei
Humphrey Shi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Few-Shot Segmentation
Transformer

Abstract: In this paper, we aim to tackle the challenging few-shot segmentation task from a new perspective. Typical methods follow the paradigm to firstly learn prototypical features from support images and then match query features in pixel-level to obtain segmentation results. However, to obtain satisfactory segments, such a paradigm needs to couple the learning of the matching operations with heavy segmentation modules, limiting the flexibility of design and increasing the learning complexity. To alleviate this issue, we propose Mask Matching Transformer (MM-Former), a new paradigm for the few-shot segmentation task. Specifically, MM-Former first uses a class-agnostic segmenter to decompose the query image into multiple segment proposals. Then, a simple matching mechanism is applied to merge the related segment proposals into the final mask guided by the support images. The advantages of our MM-Former are two-fold. First, the MM-Former follows the paradigm of 'decompose first and then blend', allowing our method to benefit from the advanced potential objects segmenter to produce high-quality mask proposals for query images. Second, the mission of prototypical features is relaxed to learn coefficients to fuse correct ones within a proposal pool, making the MM-Former be well generalized to complex scenarios or cases. We conduct extensive experiments on the popular COCO-$20^i$ and Pascal-$5^i$ benchmarks. Competitive results well demonstrate the effectiveness and the generalization ability of our MM-Former. Code is available at https://github.com/Picsart-AI-Research/Mask-Matching-Transformer.

Mask Matching Transformer for Few-Shot Segmentation
Siyu Jiao 12; Gengwei Zhang?, Shant Navasardyan', Ling Chen²,
Yao Zhao 1.2
Yunchao Wei Humphrey Shi
Institute of Information Science, Beijing Jiaotong University
2 Beijing Key Laboratory of Advanced Information Science and Network AAII, University of Technology Sydney Picsart AI Research (PAIR)
jiaosiyuebj wt..uujjjuueee tu..
Abstract
In this paper, aim tackle the challenging few-shot segmentation task from new perspective. Typical methods follow the paradigm to firstly learn prototypical features from support images and then match query features pixel-level to obtain segmentation results. However, to obtain satisfactory segments, such a paradigm needs couple the learning the matching operations with heavy segmentation modules, limiting the flexibility of design and increasing the learning complexity. To alleviate this issue, we propose Mask Matching Transformer FMMFFFrmme), a new paradigm for the few-shot segmentation task. Specifically, MM-Former first uses a class-agnostic segmenter to decompose the query image into multiple segment proposals. Then, simple matching mechanism is applied to merge the related segment proposals into the final mask guided by the support images. The advantages of our MM-Former are two-fold. First, the MM-Former follows the paradigm of decompose first and then blend, allowing our method to benefit from the advanced potential objects segmenter to produce high-quality mask proposals for query images. Second, the mission of prototypical features is relaxed to learn coefficients fuse correct ones within a proposal pool, making the MM-Former be well generalized to complex scenarios or cases. We conduct extensive experiments the popular COCO-20* and Pascal-5' benchmarks. Competitive results well demonstrate the effectiveness and the generalization ability of our MM-Former. Matthhnthiigg------ Traniing-TTnffommrr...
Code is available hetiaub.coo//ccaat----eeeeeeee
Introduction
Semantic segmentation, of the fundamental tasks in computer vision, has achieved grand success [3, 5, 37, 12] recent years with the advantages of deep learning techniques [11] and large-scale annotated datasets [14, 6]. However, the presence of data samples naturally abides by a long-tailed distribution where the overwhelming majority categories have very few samples. Therefore, few-shot segmentation [21, 27, 35] is introduced to segment objects of the tail categories only according to minimal number of labels.
Mainstream few-shot segmentation approaches typically follow the Saaming-to-lcammen fashion, where a network is trained with episodic training to segment objects conditioned on a handful of labeled samples. The fundamental idea behind it is how effectively use the information provided by the labeled samples (called support) to segment the test (referred to as the query) image. Early
Work done during an internship at Picsart Al Research (PAIR).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Segnorker
YAAR
Quary
Qvery Feature
Query image Quary Fostare
Mosi 1p0020
Query Predition
Mask Matching
Enaget Suppathaxaee
qanry
lecel supper featrr
Supper Image
Suppert Praxlopet))
Paak
Suppa Mask
(3)
(b)
Figure 1: Comparison between the existing few-shot segmentation framework and our MM-Former. (a) Previous works first match support prototype (s) [33] or pixel-level support feature [34, 32] with the pixel-level query feature, then pass the matched feature through the segmentation module to obtain the query prediction. (b) Our MM-Former decouples the segmentation and the matching problems for the few-shot segmentation task. The query segmentation result is acquired with simple Mask Matching module, which operates on the support samples and a set of query mask proposals.
works [27, 36, 33, 24, 30] achieve this by first extracting one or few semantic-level prototypes from features of support images, and then pixels in the query feature map are matched (activated) by the support prototypes to obtain the segmentation results. We refer to this kind of method as "few-to-many" matching paradigm since the number of support prototypes is typically much less (e.g., two to three orders of magnitude less) than the number of pixels in the query feature map. While acceptable results were obtained, this few-to-many matching paradigm turns out to be restricted in segmentation performance due to the information loss in extracting prototypes. Therefore recent advances [34, 26, 19] proceed to a "many-to-many" matching fashion. Concretely, pixel- level relationships are modeled between the support and query feature maps either by attention machanism [34, 26] or 4D convolutions [19]. Benefiting from these advanced techniques, the many-to-many matching approaches exhibit excellent performance over the few-to-many matching counterparts. Overall, the aforementioned approaches construct modules combining the matching operation with segmentation modules and optimizing them jointly. For the sake of improving the segmentation quality, techniques of context modeling module such as atrous spatial pooling pyramid [3]. self-attention [39] or multi-scale feature fusion [24] are integrated with the matching operations [33] and then are simultancously learned via the episodic training [30, 34, 24J. However, this joint learning fashion not only vastly increases the learning complexity, but also makes it hard to distinguish the effect of matching modules in few-shot segmentation.
Therefore, in this work, steer toward different perspective for few-shot segmentation: decoupling the learning of segmentation and matching modules as illustrated in Fig. 1. Rather than being matched with the pixel-level query features, the support samples are directly matched with a few class-agnostic query mask proposals, forming a "few-to-few" matching paradigm. By performing matching in the mask level, several advantages are provided: 1) Such a few-to-few matching paradigm releases matching from the segmentation module and focuses on the matching problem itself. 2) It reduces the training complexity, thus a simple few-to-few matching enough for solving the few- segmentation problem. 3) While previous works turned out to be overfitting when using high-level features for matching and predicting the segmentation mask [24, 27, 34], the learning of our matching and segmentation module would not affect each other and hence avoids this daunting problem.
To achieve this few-to-few matching paradigm, we introduce a two-stage framework, named Mask Matching Transforner (dubbed as MM-Former), that generates mask proposals for the query image in the first stage and then matches the support samples with the mask proposals in the second stage. Recently, MaskFormer [4, 5] formulates semantic segmentation as a mask classification problem, which obtains semantic segmentation results by combining the predictions of binary masks and the corresponding classification scores, where the masks and the scores are both obtained by using a transformer decoder. provides the flexibility for segmenting an image of high quality without knowing the categories of the objects in advance. Inspired by this, we also use the same transformer decoder as in [4] to predict a set of class- aggnssic masks based on the query image only. To further determine the target objects indicated by the support annotation, a simple Mask Matching Module is constructed. Given the features extracted from both support and query samples, the Mask Matching Module obtains prototypes from both support and query features through masked global average pooling [27]. Further, matching operation applied to match the supports with all query proposals and produces set of coefficients for each query candidate. The final segmentation result for a given

---

Pixed Backhone
Potential Objects Segmenter
FG (ggen
Transformer
Decoder
Query Image
Proposals
Fiinal mask
Encuder
Self Align
GAP
Coo Strllatty
MILP
Supprt Image
(
Self Align
Pg
GAP
FedzmAllgrrwei Bick
Laeradbe Meeching hiick
Mask Matching Modalle
Note in (Pi) (P)) meuns the 1t layer oulpuls of buckbone
Support
Figure 2: An overview of the proposed MM-Former. The support and query images are passed through an weight-shared encoder. The support and query features from the Encoder donate as Fs and Fq, respectively. We first train a Potential Objects Segmenter, which can predict all the proposal objects in one image (color in blue). Then we use Fs in stage two as guidance information to collaboratively mine the final segmentation by Mask Matching Module (color in grey).
support(s) pair is acquired by combining the mask proposals according to the coefficients. In addition, to resolve the problem of representation misalignment caused by the differences between query and support images, a Feature Alignment Block is integrated into the Matching Module. Concretely, since the features for all images are extracted with fixed network, they may not be aligned well in the feature space, especially for testing images with novel classes. A Self Aligmment block and Cross Alignment block are introduced to consist of the Feature Alignment Block to align the query and support samples in the feature space so that the matching operation can be safely applied to the extracted features.
We evaluate our MMMForrmeraeo on two commonly used few-shot segmentation benchmarks: COCO-20' and Pascal-5' Our model stands out from previous works on the challenging COCO-20' dataset. While our MM-Former only performs comparably with previous state-of-the-art methods due to the limited scale of the Pascal dataset, our MM-Former exhibits strong transferable ability across different datasets (i.e., COCO-20* + Pascal-5'), owing to our superior mask-matching design. In a nutshell, our contributions can be summarized as follows: (1) We put forward a new perspective for few-shot segmentation, which decouples the learning of matching and segmentation modules, allowing more flexibility and lower training complexity.
(2) We introduce simple two-stage framework named MM-Former that efficiently matches the support samples with set of query mask proposals to obtain segmentation results. (3) Extensive evaluations on COCO-20* and Pascal-5' demonstrate the potential of the method to be robust baseline in the few-to-few matching paradigm.
2 Methodology
Few-shot segmentation aims at training a segmentation model that can segment novel objects with very few labeled samples. Specifically, given two image sets Dtrain and Drest with category set Ctrain and Ctest respectively, where Ctrain and Ctest are disjoint in terms of object categories (Ctruin nCtest = 0). The model trained on Dtrain is directly applied to test on Drest- The episodic paradigm was adopted in [24, 38] to train and evaluate few-shot models. A k-shot episode {[L.), k 1 is composed objects from the same category. We estimate the number of episodes for training and testing set are Ntrain and Ntest, the training set and test set can be represented by Dirain = ((1a), I1) Naravs and
Problem Setting:
& support images Is and a query image In. all {1,}* k and 1g contain
Diest = {{I,} } } Newst, Note that both support masks M, and query masks M, are available for training. and only support masks M, are accessible during testing.
Overview: The proposed architecture can be divided into three parts, i.e., Backbone Network, Potential Objects Segmenter and Mask Matching Module. Specifically, the Backbone Network is used extract features only, whose parameters are fixed during the training. The Potential Objects Segmenter (dubbed as POS) is applied to produce multiple mask proposals that may contain potential object regions within the given image. The Mask Matching Module (dubbed as MM module) takes support cues as guidance to choose the most likely ones from the mask proposals. The selected masks are finally merged into the target output. The complete diagram of the architecture shown in Fig. 2. Each of the modules will be explained in detail in the following subsections.
3

---

2.1 Feature Extraction Module
We adopt a ResNet [11] to extract features for input images. Unlike previous few shot segmentation methods [24, 38, 31] using Atrous Convolution to replace strides in convolutions for keeping larger resolutions, we keep the original structure of ResNet following [5]. We use the outputs from the last three layers in the following modules and named them as Fg and Fq for Is and Tq. where F = { P } i E [3, 4, 5] and F is the features of Is or Iq. i is the layer index of backbone. We further extract the output of query layer2 to obtain the segmentation mask (named as Fÿ). F2, F F and Fe have strides of (4, 8, 16, 32) with respect the input image.
2.2 Potential Objects Segmenter
The POS aims segment all the objects in one image. Follow Mask2Former [4], standard transformer decoder [25] is used to compute cross attention between FQ and N learnable embeddings. The transformer decoder consists of 3 consecutive transformer layers. each of which takes the corresponding F as an input. Each layer in the transformer decoder can be formulated as
E++1
TLayer (E',, Fr),
(1)
where E" and E represent the N learnable embeddings before and after applying the transformer
layer respectively. TLayer denote a transformer decoder layer. We simplify the representation of transformer decoder, whereas we conduct the same pipeline proposed by Mask2Former. The output of the transformer decoder is multiplied with F3 to get N mask proposals M E IR TR H//XW// Note that Sigmoid is applied to normalize all mask proposals to [0, 1J. Besides, our POS abandons the classifier of Mask2Former since we don't need to classify the mask proposals.
2.3 Mask Matching (MM) Module
In MM, our goal is to use support cues as guidance to match the relevant masks. The building blocks of MM are a Feature Alignment block and Leamaale Matching block. We first apply Feature Alignment block to align FQ and Fs from the pixel level. Then, the Learnable Matching block matches appropriate query masks correspondence the support images.
Feature Alignment Block: We achieve the alignment using two types of building blocks: a Self- Alignment block and a Cross-Alignment block. The complete architecture is shown in Fig. 3.
We adopt the Self elig-Aligmeen block to align features in each channel. Inspired from Po- larized Self-Attenin Atff-Attention [16], we design a non-parametric block to normalize represen- tations. Specifically, the input feature map F Rexhw is first averaged at the channel dimension to obtain Foeg E IR gxxww Favg is regarded as anchor to obtain the attention weight E Rex1 by matrix multiplication: A == FFT avg- which represents the weights of different channels. A is used to activate the Average means channel- wise average. feature by position-wwe ceattion-wise-nee vise-mulliplication (i.e., expand at the spatial dimension and perform point- wise multiplication with the feature). this way, the input feature is adjusted across the channel dimension, and the outliers are expected be smoothed. Note that the Self Alignment block processes Fs and FQ individually and does not involve interactions across images.
e Softres
K Ments-Maltileatt
Beatto Wise Multiliaaio
Avenge
1xb2gg
Alquaneet
Transfomer Decodec
Reshape
P (Wiggh-shaatt
Sell Allgament
V.
Figure 3: Details of Feature Alignment Block. Note
The Cross-Alignes Alogs-Aligmeent block is introduced to mitigate divergence across different images. Fs and Fq are fed into two weight-shared transformer decoders in parallel. We take ith layer as an example in Fig. 3, which can be formulated as
F = MLP MLP (MHAttee) F5,
(2)
F3 MLP ALP(MMAALen(F's F2 Fg)),
where F2j and F'S represent the 7th layer feature FQ and Fs. F represents the alignment features.
(f { },¹ , [3,4, 5)). MLP denote MultiLayer Perceptron and MHAtten represents multi-

---

head attention [25]
gh
MH k u) = softmax
),,
(3)
Vdl
where q, kt mean three matrices, and dg is the dimension of and k elements. k, downsampled to 32 of the original resolution to save computation. To distinguish from the phrase "Query Image" in few-shot segmentation and "matrix Q" in Transformer, as q. We simplify the representation of MHAtten and omit some Shortcut-Connection Normalizations in transformer decoder, whereas we conduct the same pipeline with the standard transformer.
name "Query Image" as and "matrix Q"
and Layer
Learnable Matching Block: After acquiring Fs and Fq. first apply masked global average pooling (GAP) [38, 27, 24, 31] on each f and concat them together to generate prototypes for support ground-truth and query mask proposals. Named as {PS"). ). {Pa} PS Po R and
n E [1, N. Here d represents the dimension of Fi, 3d is achieved by concatenating fs F4, Fs
together. We use cosine distance measure the similarity between the prototypes of Ps and Po.
In some cases, the mask corresponding the prototype with the highest similarity may not be complete (e.g., the support image has only parts of the object). So, we further use an MLP layer to merge corresponding masks. The detailed diagram can be formulated as
PQ). [L,2...N], S cos( P
M = M x MLP(S), R
where M is our final result, ML.P and cos indicate the fully connected operation and the cosine similarity. We take similarities (S) as the input of ML.,, and use the output to perform a weighted average of N mask proposals. Note that we do not select the mask with the highest similarity directly, our ablation studies prove that using this block can help improve the performance.
2.4 Objective
In POS, we adopt segmentation loss functions proposed by Mask2Former (denote as L.).. We apply Hungarian algorithm match mask proposals with groud-truth and only conduct Dice Loss to supervise on masks with the best matching.
In MM module, we conduct Dice Loss on M (denote as LMr and design a contrastive loss to constrain the cross-alignment module. Our goal is make prototypes for the same class more similar while different classes less similar by constraining S. We first normalize to [0, 1] by min-max miixs))minnS)ee) Then we calculate IoU between N mask proposals and Query ground-truth. We assume that mask proposals contain various objects in query images. It unrealistic to constrain the corresponding prototypes across different categories since it is hard to acquire the proper similarity among them. Therefore. we apply criterion the location of max(loU) and denote the point as positive point Spos- Only constrain on Spos may lead all outputs of Cross Aligmment block tend to be same. Thus, we add a constraint on the point Š corresponding to the lowest IoU, and denote it as negative point Sneg- We assign Upos and yneg O to Spms and Sneg during the optimization, respectively. Therefore, the cross-alignment loss Lco can be defined as
normalization Š = S-min(S)
1 (Upos log Spos + (1 Yneey) log -Sney))
Lco -
(5)
Thus, the final loss function can be formulated as L Lr + XI + Xeheo, where a and X2 are constants and set to 10 and in experiments.
2.5 Training Strategy
In order to avoid the mutual influence of POS and MM module during training, we propose a two-stage training strategy of first training POS and then training MM. In addition, by decoupling POS and MM, the network can share the same POS under 1-shot and K-shot settings, which greatly improves the training efficiency.
5

---

Table 1: Comparison with state-of-ther art methods on COCO-20', Best results are shown bold.
I-shot
5-shot
Method
Backbone
5
5'
5*
5'
Mean
5"
5'
5
5
Mean
PFENet TPAM120) [24]
34.3
33.0
32.3
33.1
32.4
38.5
38.6
38.2
34.3
37.4
SCLICVPR211 [31]
36.4
38.6
37.5
35.4
37.0
38.9
40.5
41.5
38.7
39.9
ASGNe((CVPR21] [13]
34.6
REPRI(CVRR21] [1]
31.2
38.1
33.3
33.0
34.0
38.5
46.2
40.0
43.6
42.1
Res-50
MM-NetjicCV21] [28]
34.9
41.0
37.8
35.2
37.2
38.5
39.6
38.4
35.5
38.0
CCT1iC [18]
32.3
36.0
31.6
31.6
32.9
40.1
43.8
39.0
42.4
41.3
HSNet11cc221 [19]
36.3
43.1
38.7
38.7
39.2
43.3
51.3
48.2
45.0
46.9
CyCTR NNurIPS21] [38]
38.9
43.0
39.6
39.8
40.3
41.1
48.9
45.2
47.0
45.6
MM-Former (Ours)
40.5
47.7
45,2
43.3
44,2
44.0
52.4
47.4
50,0
48,4
Res-50
Oracle
66.1
74.3
64.8
70.4
68.9
66.1
74.3
64.8
70.4
68.9
Table 2: Comparison with state-of ofte-of--the-rr the-hhhrrr art methods on PASCAL-5) Best results are shown in bold.
PASCAL PASCAL
COCO PASCAL
Method
Backbone
shot
5 shot
1 shot
5 shot
PFENet/TPAMIE [24]
60.8
61.9
61.1
63.4
SCL/CVPR21] [31]
61.8
62.9
REPRI/CVPR211 [1]
Res-50
59.1
66.8
63.2
67.7
HSNet(1CC221 [19]
64.0
61.6
68.7
CyCTR[NeurPPSS [38]
64.0
67.5
MM-Former (Ours)
63.3
64.9
67.7
70.4
Res-50
Oracle
82.5
82.5
85.8
85.8
K-shot Setting: Based on the two stages training strategy, MM-Former can easily extend to the K-shot setting by averaging knowledge from K samples, i.e., Pg. Note that after pre-trained the POS, MM-Former can be applied to 1- 1-shot/ K-shot tasks with only very small amount of training.
3 Experiments
3.1 Dataset and Evaluation Metric
We conduct experiments on two popular few-shot segmentation benchmarks, Pascal-5' [9] and COCO-20' [14], to evaluate our method. Pascal-5' with extra mask annotations SBD [10] consisting of 20 classes are separated into 4 splits. For each split, 15 classes are used for training and 5 classes for testing. COCO-20' consists of annotated images from 80 classes. We follow the common data split settings in [20, 38. 19] to divide 80 classes evenly into 4 splits, 60 classes for training and test on 20 classes. 1,000 episodes from the testing split are randomly sampled for evaluation. To quantitatively evaluate the performance, we follow common practice [24, 27, 36, 19, 38], and adopt mean intersection-over-niiil (mloU) as the evaluation metrics for experiments.
3.2 Implementation details
The training process of our MM-Former is divided into two stages. For the first stage, freeze the ImageNet [6] pre-trained backbone. The POS is trained on Pascal-5* for 20,000 iterations and 60,000 iterations COCO-20' respectively. Learning rate second stage, we freeze the parameters of the backbone and the POS, and only train the MM module for 10,000/20,000 000 iterations on Pascal-5' COCO-20', respectively. Learning rate is set l batch size is set to 4. For both stages, we use AdamW [17] optimizer with weight decay of 5e -2 The learning rate is decreased using the poly schedule with factor of 0.9, All images are resized and cropped into 480 x 480 for training. We also employ random horizontal flipping and random crop techniques for data augmentation. All the experiments are conducted on single RTX A6000 GPU. The standard ResNet-50 [11] is adopted as the backbone network.
-4, batch size is set to 8. For the
3.3 Comparison with State-of-the-art Methods
We compare the proposed approach with state-of-the-art methods [24, 31, 13, 28, 19, 18, 1, 15, on Pascal-5' and COCO-20' datasets. The results are shown in Tab. I and Tab. 2.

---

Table 3: Ablations Mask Matching Module (Stage-2). In (a), the result in first row is obtained by our baseline. means non-parameters design (Details in Sec. 2.3). We denote cross-alignment supervised with Lco by i'nn the last row (Sec. 2.4). SA, CA, LM represent Self Alignment block, Cross Alignment block and Learnable Matching block, respectively.
Ablation the components of Mask Matching Module.
(b) Ablation Feature Extraction.
Components
Pascal-5'
COCO-20'
Pascal-5'
COCO-20
Feature Extraction
SA
CA
LM
mean ToU
mean IoU
mean ToU
mean ToU
22.4
After MSDeefommAtn
60.3
42.8
46.4
23.6
Before MSDeformAttn
63.3
44.2
38.0
16.7
(c) Ahlation on Training Strategy.
35.3
56.4
37.9
Pascal-5'
COCO-20*
61.9
43.0
Training Strategy
mean IoU
mean loU
62.1
43.2
End-to-end
60.6
40.3
63.3
44.2
2-stage-training
63.3
44.2
Table 4: Ablations on Potential Objects Segmenter (Stage-1).
(a) Ablation on Mask classification.
(b) Ablation on Number Segmenters.
Mask classification
mean IoU
Pascal-5'
COCO-20'
mean IoU
Num Segmenters
Oracle
Pascal-5'
Final
10
70.4
57.9
78.9
68.4
50
78.2
61.0
X
82.5
68.9
100
82.5
63.3
Results on COCO-20', Tab. our MM-Former performs remarkably well in COCO for both 1-shot and 5-shot setting. Specifically, achieve 3.9% improvement 1-shot compared with CyCTR [34] and outperform HSNet [19] by 1.5% mloU
Results on Pascal-5', Due to the limited number of training samples in the Pascal dataset, the POS may easily overfit during the first training stage. Therefore, following recent works [24, 1, 19], we include the transferring results that transfer the -acnnera model to be tested on Pascal. Note that when training on the COCO dataset, the testing classes shared with the Pascal dataset are removed, so to avoid the category information leakage.
According to Tab. although our MM-Former is slightly inferior some competitive results when training on Pascal dataset, we find MM-Former exhibits remarkable transferability when training on COCO and testing on Pascal. Specifically, the previous state-of-the-art method HSNet shows powerful results on Pascal-Pascal but degrades when transferring from COCO to Pascal. Instead, our MM-Former further enhances the performance of 1-shot and 5-shot by 4.4% and 5.5%, outperforming HSNet by 6.1% and 1.7%, respectively.
Oracle Analysis. We also explore the room for further improvement of this few-shot segmenta- tion paradigm by using query ground truth (GT) during inference. The results refer to the last rows in Tab. 1, 2. In detail, after the POS generates N mask proposals, one proposal mask with the highest loU, and regard this result as the segmentation result. Note that this natural selection is not the optimal solution because there may be other masks complementary to the selected one, but it is still good oracle to show the potential of the new learning paradigm. According the results, there is still large gap between current performance and the oracle (s= 20% mloU), which suggests that our model has enormous potential for improvement whereas we have achieved ofte-oo-thhe-e performance.
use the GT mask select
3.4 Ablation Studies
We conduct ablation studies on various choices of designs of our MM-Former to show their con- tribution to the final results. Component-wise ablations, including the MM Module and the POS, are shown in Sec. 3.4.1. The experiments are performed with the shohot setting on Pascal-5' and COCO-20'. We further experimentally demonstrate the benefits of the two-stage training strategy in Sec. 3.4.2.
7

---

trage
Fare
patth ligher scome It
Query imsaae
Suppont Image
Porp add harn Pig
Figure 4: Qualitative results on Pascal-5'. B, M, F mean Baseline network, learnable matching block and feature alignment block.
Figure Visual results
in Self Alignment block.
3.4.1 Component-wise Ablations
To understand the effect of each component in the MM module, we start the ablations with a heuristic matching baseline and progressively add each building block.
Baseline. The result the heuristic matching baseline shown in the first row of Tab. 3a, which directly selects the mask corresponding to the highest cosine similarity with the support prototype. Note that when not using the learnable matching block, the results in Tab. 3a are all obtained in the same way the heuristic matching baseline. We observe that the heuristic matching strategy does not provide strong performance, which is caused by the feature misalignment problem and fails to fuse multiple mask proposals.
Self-Alignment Block. With the self-alig self-alignment block, the performance improved by 4.3% on Pascal and 1.2% on COCO, shown by the 2nd result in Tab. 3a, demonstrating that channel-wise attention does help normalize the features for comparison. However, the performance is still inferior, encouraging to further align the support and query features with the cross-alignment block.
Pross-Alignment Block. In the third result of Tab. 3a, we experiment with non-parametric variant of the cross-alignment block that removes all learnable parameters the cross-alignment block. significant performance drop observed. This not surprising because the attention in the cross-alignment block cannot attend to proper areas due to the fcature misalignment. When learning the cross-alignen alissmatig block, indicated by the 4th result of Tab. 3a, the performance remarkably improved by 14% on Pascal and 12.9% on COCO, manifesting the necessity of learning the feature alignment for further matching.
Learnable Matching Block. Surprisingly, simply using our learnable matching block can already achieve decent performance (the 5thh result in Tab. 3a) compared with the baseline, thanks to its capability to adaptively match and merge multiple mask proposals.
Mask Matching Module. By applying all components, art performance on COCO to 43.2% (the 7th result in Tab. 3a). addition, to encourage the alignment of query and support in the feature space, add the auxiliary loss Leo to the output of the cross-alignment block, which additionally enhances the performance by more than 1%.
pushes the state-of-th
Potential Objects Segmenter Although we follow Mask2former to build our POS, several differences are made and we evaluate the choice of design as follows. Mask Classification. In Mask2Former [4], linear classifier is trained with cross-entropy loss for categorizing each mask proposal, while in our avoid learning class-specific representations
MM-Former for few-shot segmentation, we remove in the first training stage. The result in Tab. 4a shows that the classifier harms the performance due to that the linear classifier would make the network the "seen" classes in the training set. Since this change only affects the first stage, Proposals. Tab. 4b, we try to vary the numbers of the mask proposals N. Increasing the number of N will significantly improve the oracle result and our result. Thus we chose 100 as the default value in all other experiments. is worth noting that, when varying the number from 10 to 100, our result is improved by 5.4%, but the oracle result is improved by 12.1%, indicating the large room for improvement with our new mask matching paradigm.
use the oracle results to demonstrate the effect. Numbers of
Effect of Different Feature Extraction. Previous few-shot segmentation works [34, 24, 30] typ-
ically integrate matching operations with segmentation-speein techniques [39, 24, 3]. Following Former, POS also includes multi-scale deformable attention (MSDeformAttn) [39]. In
8

---

Table 5: Analysis of the model efficiency, training time for MM-Former shown in terms of 1st
2nd stage.
I-shot
5-shot
mloU
training time
GPUs
memory
mloU
training time
GPUs
memory
HSNet
39.2
168h
27.5G
46.9
168h
4
27.5G
CyCTR
40.3
32.6h
115.7G
45.4
56.6h
4
136.6G
MM-Former
44.2
7.1h A 4.0h
10.7G 6.0G
48.4
7.1h 8.6h
I
10.7G / 11.3G
Tab. 3b, we investigate using the features from the MSDeformAttn instead of the backbone feature for the MM module. Interestingly, although the feature after the context modeling is essential for segmentation, it is not suitable for the matching problem and impairs the matching performance.
3.4.2 Analysis of Training Strategy
Effect of Two-stage Training. One may wonder what if we couple the training of POS and MM modules. Tab. 3c experiments this point, the joint optimization is inferior to the two-stage training strategy, since POS and MM have different convergence rates.
Efficiency of Two-stage Training. We analyze the efficiency of our method and provide comparisons of training time, and training memory consumption (Tab. 5). All models are based on the ResNet-50 backbone and tested on the COCO benchmark. All models are tested with RTX A6000 GPU(s) for a fair comparison. Training times for CyCTR and HSNet are reported according to the official implementation. We report the training time for the first and second stages separately. It is worth noting that for the same test split, our method can share the same stage-1 I model across 1-shot and 5-shot. The training time of stage-1 for 5-shot can be ignored if 1-shot models already exist.
3.5 Analysis model transferability
Our MM-Former shows better transfer performance when trained on COCO but relatively lower performance when trained on Pascal. We make an in-depth study of this phenomenon.
Effect of the number of training samples: We
use all training samples belonging to 15 Pas-
Table 6: Transfer study (testing on Pascal).
cal training classes from COCO to train MM-
Training Set
I-shot
5-shot
Oracle
Former. In this case, training samples are times larger than the number in Pascal but the categories are the same, dubbed COCO-15 in Tab. 6. When the number of classes is limited, more training data would worsen the matching performance (60.7% vs. 63.3% for 1-shot and 64.8% vs. 64.9% for 5-shot), though better POS
Pascal
63.3
64.9
82.5
COCO-15
60.7 (-2.6)
64.8 (-0.1)
86.3
cOCO-75-sub
66.8 (+3.5)
68.9 (+4.0)
85.3
COCO
67.7 (+4.4)
70.4 (+5.5)
85.8
could be obtained, as indicated by the oracle result (86.3% vs. 82.5%).
Effect of the number of training classes: We randomly sample an equal number of training images ( 6000 images averaged across splits) in Pascal training set from 75 classes (excluding test classes) in COCO to train our MM-Former, dubbed COCO-75-suh in Tab. 6. When training with the same amount of data, more classes lead to better matching performance (66.8% vs. 63.3% for 1-shot and 68.9% vs. 64.9% for 5-shot).
In a word, the number of classes determines the quality of the matching module. This finding is reasonable and inline with the motivation of few-shot segmentation and meta-lcarning: learning to learn by observing wide range of tasks and fast adapting new tasks, When the number of classes is limited, the variety of tasks and meta- weta-knowledde are restricted, therefore influencing the learning of the matching module.
3.6 Qualitative Analysis
Visual examples: We show some visual examples in Fig, 4. Without loss of generality, some support images may only contain part of the object (e.g., the 2nd row). Directly selecting the mask with the highest cosine similarity can not obtain the anticipated result. Using learnable mask matching block to fuse multiple masks can solve the problem to a large extent, the proposed feature alignment block can further improve our model by alleviating the misalignment problem, e.g. the results in the last row.

---

Robustness analysis: We also provide robust- ness analysis in Fig. 6, which uses an anomaly support sample for segmenting the query im- age. Compared with the previous state-of-the- art method, HSNet, which tends to segment the salient object in the image, our model more robust to the anomaly inputs.
FENSE Dus
Figure 6: Robustness Analysis. We use different classes of support and query images to verify the robustness of the network.
Explanation of SA: SA is proposed to align the features at the channel dimension so that outliers at the channel dimension could be smoothed and features would be more robust by aligning with the attended global (specifically, channel-wise weighed average) features. Fig. 5 proves this point. It can be seen that Fang (row 2th) has response to general foreground regions. Important channels (row 3th) are emphasized, and outliers are suppressed (row 4th)
Related Work
Few-Shot Segmentation [21] is established to perform segmentation with very few labeled images. Many recent approaches formulate few-shot segmentation from the view of metric learning [23, 7, 27]. PrototypicalNet [22] is the first to perform metric learning few-shot segmentation. PFENet [24] further designs an feature pyramid module to extract features from multi-levels. Many recent methods point out that only single support prototype is insufficient to represent a given category. To address this problem, [32] attempt to obtain multiple prototypes via EM algorithm. [15] utilized super-pixel segmentation technique generate multiple prototypes. Another way to solve the above problem to apply pixel-level attention mechanism. [32, 26] attempt to use graph attention networks to utilize all foreground support pixel features. HSNet [19] propose learn dense matching through 4D Convolution. CyCTR [38] points out that not all foreground pixels are conducive to segmentation and adopt yycle-consistency technology to filter out proper pixels guide segmentation.
Transformers originally proposed for NLP [25] are being rapidly adapted in computer vision task [8, 2, 5, 4]. The major benefit of transformers is the ability to capture global information using self-attention module. DETR [2] the first work applying Transformers on object detection task. Mask2Former [4] using Transformers to unify semantic segmentation and instance segmentation. Motivated by the design of Mask Former, we apply transformers to segment all potential objects in one image, align support features and query features in pixel-level within our MM-Former.
Conclusion
In the paper, we present Mask Matching Transformer (MM-Former), a new perspective to tackle the challenging few-shot segmentation task. Different from the previous practice, MM-Former is a two-stage framework, which adopts a Potential Objects Segmentor and Mask Matching Module to first produce high-quality mask proposals and then blend them into the final segmentation result. Extensive experiments on COCO-20' and Pascal-5 well demonstrate the effectiveness and the generalization advantage of the proposed MM-Former. We hope MM-Former can serve as solid baseline and help advance the future research of few-shot segmentation.
Limitations and societal impact. Our MM-Former introduces the paradigm of decompose first and then blend to the research of few-shot segmentation, which is a totally new perspective and may inspire future researchers to develop more advanced versions. However, there is still large gap between the current results and the oracle (=s 20% mloU). How to further narrow this gap is our future research focus.
Acknowledgment. This work was supported part by the National Key & Program of
China 2No.20212D0112 12100), the National NSF of China (No.U1936212, No.621 20 201060099, the Fundamental Research Funds for the Central Universities (No. K22RC00010), Yao Zhao and Yunchao Wei are the corresponding authors.
10

---

References
[1] Malik Boudiaf, Hoel Kervadec, Ziko Imtiaz Masud, Pablo Piantanida, Ismail Ben Ayed, and Jose Dolz. Few-shot segmentation without meta-learning: A good transductive inference is all you need? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13979-13988, 2021.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[3] Liang-Chiel Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence. 40(4):834-84) 848, 2017.
[4] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[5] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. Advances Neural Information Processing Systems, 34, 2021.
[6] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hicrarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 Icce, 2009.
[7] Nanqing Dong and Eric P Xing. Few-shot semantic segmentation with prototype learning. In BMVC, volume 3-4, 2018
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprins arXiv:2010.11992 19929, 2020.
[9] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, (8(2):303-338 2010.
[10] Bharath Hariharan, Pablo Arbeláez, Ross Girshick, and Jitendra Malik. Simultaneous detection and segmentation. In European Conference on Computer Vision, pages 297-312. Springer, 2014.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[12] Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 603 612. 2019.
[13] Gen Li, Varun Jampani, Laura devilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8334 0043-833,,444 2021.
[14] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision (ECCV), 2014.
[15] Binghao Liu, Yao Ding, Jianbin Jiao, Xiangyang Ji, and Qixiang Ye. Anti-aliasing semantic reconstruction for few-shot semantic segmentation. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 9747-9756, 2021.

---

[16] Huajun Liu, Fuqiang Liu, Xinyi Fan, and Dong Huang. Polarized self-attention: Towards high-quality pixel-wise regression. arXiv preprint arXiv:2107.00782. 2021.
[17] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
[18] Zhihe Lu, Sen He, Xiatian Zhu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Simpler is better: Few-shot semantic segmentation with classifier weight transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8741-8750, 2021.
[19] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmenta- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2021.
[20] Khoi Nguyen and Sinisa Todorovic. Feature weighting and boosting for few-shot segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 622-631, 2019.
[21] Amirreza Shaban, Shray Bansal, Zhen Liu, Irfan Essa, and Byron Boots. One-shot learning for semantic segmentation. In British Machine Vision Conference (BMVC), 2018.
[22] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems (NeurIPS), 2017.
[23] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Tomr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1199-1208, 2018.
[24] Tian, H Zhao, M Shu, Yang, Li, and J Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
(25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[26] Haochen Wang, Xudong Zhang, Yutao Hu, Yandan Yang. Xianbin Cao, and Xiantong Zhen. Few-shot semantic segmentation with democratic attention networks. In European Conference on Computer Vision (ECCV), 2020.
[27] Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. Panet: Few-shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9197-9206, 2019.
[28] Zhonghua Wu, Xiangxi Shi, Guosheng Lin, and Jianfei Cai. Learning meta-class memory for few-shot semantic segmentation. In Proceedings of the IEEE/VVF International Conference on Computer Vision, pages 517-526, 2021.
[29] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34, 2021.
[30] Boyu Yang, Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. European Conference on Computer Vision, pages 763-778. Springer, 2020.
[31] Bingfeng Zhang, Jimin Xiao, and Terry Qin. Self-guided and cross- cross-guided learning for few-shor segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8312-8322 2021.
[32] Chi Zhang, Guosheng Lin, Fayao Liu, Jiushuang Guo, Qingyao Wu, and Rui Yao. Pyramid graph networks with connection attentions for region-based one-shot semantic segmentation. In Proceedings of the IEFE/CVE International Conference on Computer Vision, pages 9587-9595 2019.
12

---

[33] Chi Zhang, Guosheng Lin, Fayao Liu, Rui Yao, and Chunhua Shen. Canet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning. In Proceedings of the IEEF/CVE Conference Computer Vision and Pattern Recognition, pages 5217-5226,
2019.
[34] Gengwei Zhang, Guoliang Kang, Yi Yang, and Yunchao Wei. Few-shot segmentation via cycle-consistent transformer. Advances in Neural Information Processing Systems, 34, 2021.
[35] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. SG-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 2020.
[36] Xiaolin Zhang, Yunchao Wei, Yi Yang, and Thomas S Huang. Sg-one: Similarity guidance network for one-shot semantic segmentation. IEEE Transactions on Cybernetics, 50(9))3855- 3865, 2020.
[37] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jía. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890, 2017.
[38] Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang, and Alexei A Efros. Learning dense correspondence via 3d 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 117-126, 2016.
[39] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010004/555 2020.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]. See Section 5. Limitations and societal impact.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 5. Limitations and societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]. We have read the ethics review guidelines carefully.
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes]. Code is available in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]. See Section 4-2.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] All models are trained on RTX A6000 GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] Our method is implemented based on MaskFormer and Mask2Former code bases.
(b) Did you mention the license of the assets? [N/A]
13

---

(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]. The datasets we used in the paper are all open source.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "_0fbLaCXFbD", "reviewer": "Reviewer_Ytwe", "summary": "The authors propose an alternate way of performing FSS by a 2-stage training procedure, where a network is first trained to generate a set of potential masks for the query image, followed by a second stage where another network is trained to generate features for both query and support, which are then pooled using the support masks, and the candidate query masks. From this, they find the best matching candidate masks through correlation followed by an MLP. The authors claim that the decoupling of segmentation and matching is beneficial for learning, and to back it up they show strong experimental results.", "questions": "## Questions regarding configs and code\nFollowing the authors instructions in the code readme:\n\n> We conduct a 2-stage-training strategy. \nThe Potential Objects Segmenter (POS) is trained in stage 1. The config files for PASCAL and COCO are provided at \n\"configs/pascal/step1.yaml\" and \"configs/coco/step1.yaml\" \nThe Mask Matching Module (MM) is trained in stage 2. The config files are provided at \n\"configs/{*datasets*}/step2_{*split*}.yaml\" \nExecute this command at the root directory: \n    python train.py --config-file {*model_config*}\nNote: We recorded the random_seed during training, using this random seed can fully reproduce the results in the paper.\n\nNotably the authors only provide a single config for the POS network. In the configs it seems like there should be one POS model per fold, so this might have been a mistake.\n\nIf the stage 1 training involves a POS model which sees classes belonging to the test set of other folds, there is leakage.\n\n## Question regarding loss for POS during stage 1\nIn MaskFormer, the loss is divided into classification and segmentation of all K classes. Typically in episodic FSS training the given label is only of the single mask of the target class. Is this also the case in your method, or is your segmentation loss based on all classes that appear in the image? This was not obvious to me from the paper, and I could also not really understand it from reading the code, as the code is quite complex.\n\n## Question regarding generalization of POS\nIf there is no class leakage between base and novel classes during training of POS, how come the N proposals don't overfit to the base classes? In the paper it is claimed that the frozen backbone keeps the POS from overfitting, but one could easily see the proposals closely matching the base classes in feature space since the queries are learnable. Is there any mechanism to prevent this from happening?\n\n## Question regarding motivation for 2-stage training\nIn the paper you claim that training end-to-end:\n> When training in an end-to-end fashion, POS would be influenced by the training of\nthe MM module, degrading the quality of mask proposals\n\nI'm not sure I follow the reasoning behind this. Could you provide a more thorough explanation why one would expect the masks to become worse in this case? From what I gather you seem to be saying that focusing only on the segmentation loss is beneficial, since propagating other losses could lead to worse segmentations. However, why not make a seperate mask prediction for the matching and segmentation tasks, but still corresponding to the same proposal?\n\n## Anonymization\nThe provided code is not sufficiently anonymized, I found references to author home folders in the code. (I did not google the names to respect reviewing anonymity). For future submissions I recommend the authors to more thoroughly anonymize the code.\n\n", "limitations": "The authors' usage of an oracle model to see the performance gap was an interesting and useful way to gauge the current limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Major Strengths\n* Novel approach\n* Simple architecture (although 2-stage training makes it more complicated)\n\n## Minor Strengths\n* Good results\n* Good ablations\n* Code provided\n\n## Major Weaknesses\n* Concerns for test-time class leakage into the first stage of training.\n\n## Minor Weaknesses\n* Somewhat underwhelming results on PASCAL\n* Even without class leakage, I'm somewhat unconvinced of the reasoning behind the improvement from 2-stage training.\n* The paper is somewhat difficult to follow\n* The code is hard to read and seems to include multiple mistakes\n\n## Conclusion\nWithout further clarification by the authors I can not recommend the paper in its current form for acceptance, hence I've selected a rating of 4. However, I am open to increasing the rating to a 6 or 7 given that the authors address my concerns and questions.\n\n### EDIT\nAfter the response of the authors I am increasing my rating from a 4->6, and confidence from 3->4.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ot7c6apDE8", "reviewer": "Reviewer_gY9J", "summary": "This paper proposes a new two-stage framework that decouples the matching and segmentation modules for few-shot segmentation. Extensive experiments and ablation studies on COCO and Pascal datasets also verify the algorithm's effectiveness.", "questions": "\n-  In #40,  \" Concretely, pixel-level relationships are modeled between the support and query feature maps either by attention mechanism or 4D convolutions.\" Using \"either...or...\" is too arbitrary. There are also other methods to deal with the relationship between the query and support sets, like HSNet and PGNet.\n\n\n\n- Poor performance on the Pascal dataset even compared with the methods in the community proposed years ago. In #85, the author attributed the reason to the limited scale of the dataset, which does not convince the reviewer.\n\n\n\n- Adding more visual illustrations will make it clearer while introducing the network structure in subsection 2.2.\n\n\n- In the Introduction and Abstract sections, the author emphasizes the advantages of the algorithm in complexity. Relevant comparative experiments and explanations need to be supplemented to support the claims.\n\n\n-  In #115, \" We use the outputs from the last three layers in the following modules\", \"the last three layers\" is ambiguous. Please further elaborate on the details.\n\n\n- The novelty is marginal since the core idea is borrowed from MaskFormer.\n\n\n- The impacts of different components brought to the model efficiency should be discussed, including the overall final model efficiency in terms of fps and model size.\n\n\n- There are quite a lot of grammatical errors and ambiguous expressions. The reviewer did not try to point them all out together. Please check your manuscript carefully and correct all mistakes before submission. Typos include but are not limited to:\n\na. In #22, \"one of a fundamental tasks\" -> \"one of the fundamental tasks\"\n\nb. In #22, “ has achieved grand success” -> \" has achieved a grand success\"\n\nc. In #29, \" segment any objects\" remove \"any\"\n\nd. A large number of articles are missing or used incorrectly. For example, \"We refer this kind of method as 'few-to-many' matching approach.\" -> \"We refer this kind of method as a 'few-to-many' matching approach.\" I won't list all since there are too many.\n\ne. In #37, \"While acceptable results obtained\" -> \"While acceptable results were obtained\". Lack of the predicate.\n\nf. In #51, \"Rather than matching\" -> \"Rather than being matched\"\n\ng. In #300, \", it is not suitable for the matching problem and impair the matching performance.\" -> \"impairs\"", "limitations": "The stated limitation, i.e., a large gap between the oracle case, indicates room for future improvements, the reviewer still believes that broader negative impacts and limitations should be discussed.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The writing of this paper is redundant and ambiguous, and the logic is not clear.\n\n2. There are lots of grammatical errors which need to be corrected.\n\n3. Lack of innovation and low novelty.\n\n4. Poor performance on the Pascal dataset even compared with the methods proposed years ago. \n\n5. Proposed two-stage strategy introduces a new perspective focusing on mask-level segmentation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "N-G1GCteqcM", "reviewer": "Reviewer_2Z5i", "summary": "This paper tackles the few-shot segmentation task through a proposed mask matching transformer (MM-Former). The MM-Former contains two parts. The first part decomposes query images into multiple segmentation proposals with a class-agnostic segmenter. The second part merges related segment proposals into final masks guided by support images. With the ResNet-50 backbone, the proposed method get competitive results on the popular COCO and PASCAL benchmarks.", "questions": "I am curious about whether the number of meaningful object proposals affects the final performance. There are also many other issues discussed in the weakness section.", "limitations": "I didn't see any obvious negative societal limitations of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "***Strengths***\n1. This paper is well-organized and can be easily understood by readers. The technical details are introduced clearly.\n2. The authors conducted extensive experiments on multiple benchmarks to investigate the effectiveness of different modules and designs in this paper.\n\n***Weakness***\n1. In the potential objects segmenter section, the authors just borrow the ideas from Mask2Former to generate object proposals. But in few-shot learning, the number of meaningful object parts is unknown both in the base training set and the novel test set. How does the network learn these object segmenter without any guidance? If the object segmenter is inaccurate, will it affect the subsequent query-related part merge?\n2. The proposed MM-Former exploits transformers as decoders for few-shot semantic segmentation. As is known a transformer may need a lot of learnable parameters. It will be better to list the parameters of different modules in MM-Former to see how the number of learnable parameters of transformers affects the final performance.\n3. In Table 2, we can find that the performance of MM-Former on PASCAL is not good as other state-of-the-art algorithms. I want to see a thorough analysis of such a phenomenon. Maybe we can know more about the characteristics of MM-Former.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "Ae6QIcIO1K7", "reviewer": "Reviewer_Ecav", "summary": "This paper proposes a few-shot semantic segmentation (FSSS) network based on Mask2Former [1]. By formulating the semantic segmentation as a mask classification problem like [1], the authors introduced a new matching mechanism in mask-level instead of pixel-level matching in previous FSSS methods. Additionally, they propose a feature-alignment block based on the attention mechanism to align both support and query features individually and cross-align between them. They conducted experiments on two FSSS datasets (COCO-20i and PASCAL-5i) and achieve the SOTA results.", "questions": "+ Can the author add the running time, training time, and the memory consumed by the model? As the Mask Matching Module is implemented by standard transformer decoder layers, does it requires large memory and time?\n+ What is the detailed implementation of the Mask Matching Module?\nIn Fig. 2, the final mask has the same resolution as the F2 feature map, how to upsample the mask to the original resolution of the input image?\n+ Explain the self-alignment module and intuition of this. Can we replace it with the self-attention module (compare the performance and runtime)\n+ Explain the use of the MLP in Eq.(4): describe the input, and output range, and how it can correct the cosine similarity vector of N potential masks. Visualize the M and M^ before and after. It helps the reader what is happening inside the block.\nWhy do not use dice loss in L_m (L173)\n+ L180: “Meanwhile, only constrain on Sˆ pos may lead all outputs of Cross Alignment block to tend to be same”: why? explain in more detail?\n+ L182: how to find the lowest IoU (ex: 2 potential masks with no overlap with GT)\n+ Can the authors add more visualization of the failure cases?\n+ In Eq. (4), redundant closing parenthesis ')'.\n\nI will increase my score if the authors address all of my concerns.", "limitations": "No", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n+ The structure of this paper is well-organized and easy to follow the ideas of the authors. \n+ The proposed method achieves significant improvements on the two datasets compared with other recent methods.\n\n**Weaknesses**\n+ The only novel contribution of this paper is the Feature Alignment Block (FAB) in Mask Matching Module and it significantly improves the results, the other blocks are incremental. The authors simply adapt it to Mask2Former architecture. It would be better to investigate more about the FAB in different methods (CyCTR, HSNet) to prove its effectiveness is generalizable enough.\n+ The “few-to-few” matching (L53) is not new. Previous approaches ([2], [3]) have applied it to the few-shot instance segmentation task. \nThe proposed method (the mask2former architecture) tends to overfit small datasets (L225), then it is not suitable for the few-shot setting.\n+ Related work is not well presented. They do not provide any analysis or comparison of previous work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept