MMC Transformer: Multiscale Multigrid Comparator Transformer for Few-Shot Video Segmentation
Authors: Mennatullah Siam
Konstantinos G. Derpanis
Richard Wildes
Venue: NeurIPS 2022 Submitted (2022)
Keywords: few-shot video segmentation
video object segmentation
few-shot learning
actor/action segmentation

Abstract: Learning to compare support and query feature sets for few-shot image and video understanding has been shown to be a powerful approach. Typically, methods limit feature comparisons to a single feature layer and thus ignore potentially valuable information. In particular, comparators that operate with early network layer features support precise localization, but lack sufficient semantic abstraction. At the other extreme, operating with deeper layer features provide richer descriptors, but sacrifice localization. In this paper, we address this scale selection challenge with a meta-learned Multiscale Multigrid Comparator (MMC) transformer that combines information across scales. The multiscale, multigrid operations encompassed by our architecture provide bidirectional information transfer between deep and shallow features (i.e. coarse-to-fine and fine-to-coarse). Thus, the overall comparisons among query and support features benefit from both rich semantics and precise localization. Additionally, we present a novel multiscale memory learning in the decoder within a meta-learning framework. This augmented memory preserves the detailed feature maps during the information exchange across scales and reduces confusion among the background and novel class. To demonstrate the efficacy of our approach, we consider two related tasks, few-shot video object and actor/action segmentation. Empirically, our model outperforms state-of-the-art approaches on both tasks. 

MMC Transformer: Multiscale Multigrid Comparator Transformer for Few-Shot Video Segmentation
Anonymous Author(s)
Affiliation
Address
email
Abstract
Learning to compare support and query feature sets for few-shot image and video understanding has been shown to be a powerful approach. Typically, methods limit feature comparisons
2
single feature layer and thus ignore potentially valuable information. In particular, comparators that operate with early network layer features support precise localization, but lack sufficient semantic abstraction. At the other extreme, operating with deeper layer features provide richer descriptors, but sacrifice localization. In this paper, we address this scale selection challenge with meta-learned Multiscale Multigrid Comparator (MMC) transformer that combines information across scales. The multiscale, multigrid operations encompassed by our architecture provide bidirectional information transfer between deep and shallow features (i.e. coarse-to-fine and fine-to-coarse). Thus, the overall comparisons among query and support features benefit from both rich semantics and precise localization. Additionally, we present novel multiscale memory learning in the decoder within meta-learning framework. This augmented memory preserves the detailed feature maps during the information exchange across scales and reduces confusion among the background and novel class. To demonstrate the efficacy of our approach, we consider two related tasks, few-shot video object and actor/action segmentation. Empirically, our model outperforms state-of-ther thettherrt approaches on both tasks.
4
10
11
13
14
15
17
18
19
Introduction
20
21 Guided by a few labelled examples (i.e. the support set), few-shot learning is focused on improving the generalization ability of models to novel classes unseen during the initial training to classify the query 23 images. this paper, our focus on metric learning for few-shot video segmentation, where methods learn to compare features between the support and query sets (i.e. learning comparators), Previous 24
25 dense prediction work has documented that pixel-to-pixel comparisons between the query and support sets better capture fine details compared to working with global average pooled representations [19
27 21. A key question arises: Which features should be compared? Limiting comparisons between the support and query sets at the finest scales (i.e. shallow network layers) capture only primitive semantics (e.g. local orientation) and thus are error prone, while the coarser scales (i.e. deeper network layers) capture abstract semantics but sacrifice detailed information that support precise localization.
29
31 To address this scale selection challenge, we present novel Multiscale Multigrid Comparator (MMC) transformer that takes as input set of correlation tensors that encompass comparisons 33 between support-query features at multiple abstraction levels. Our transformer incorporates multigrid processing 3 2 that allows bidirectional information to be exchanged across scales (i.e. coarse-to-fie -ine 35 and fine-to-coarse). Critically, this multiscale information exchange reduces the impact of erroneous correlations at the finer scales by incorporating feedback from coarser scales and allows finer scale
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Query
Meta-learning MMC Transformer
Multiscale Mulliggrid Comperator Transformer
Query Video Predictions
Query Key Value
Support-tn-Cue Querry
Correlation
Crass Attention
Nodule
Bilrectional Information Scales
Support
Trimmed Support Video
Image
Baab Hisling
Query
Untrimmed
Query
Few-Shot Video Object Segmentation
Few-Shot Video Actor/Action Segmentation
Figure 1: Overview of our Multiscale Multigrid Comparator (MMC) transformer. Top: We take as input correlations between backbone features at different scales for both support and query sets. We meta-learn MMC transformer that encompasses multiscale the spatial/spatiotemprraa features at different levels and a memory module that helps separate the background and novel class. Our MMC transformer allows bidirectional information exchange across scales following the multigrid formulation between fine-mid-coarse scale features in the multiscale cross attention blocks. Additionally, information exchange across scales through multiscale memory learning (i.e. and V) that help separates the background and novel class. Bottom: We demonstrate approach two video tasks that require dense predictions: few-shot video object and action segmentation. The support set groundtruth and query predictions are highlighted red.
attention between
design preserves the spatiotemporal dimension during the
information to feedback to the coarser scales, allowing more detailed information to modulate the coarse-gres coarse-grained information. A key enabler of MMC allowing the multiscale processing within the transformer decoder to preserve the spatiotemporal dimension, rather than pooling the information 40 into a compact vector 7 (). Finally, to address the issue confusions between the novel class and background we use novel multiscale memory learning module. As two illustrative video tasks, we instantiate our model for few-shot video object (FSVOS) and actor/action segmentation. Fig. provides overview of overall approach. Since our paper explores both few-shot learning and transformers, to reduce ambiguities between the term query used in both, we use the term target query when referring to its usage in few-shot throughout the rest of the paper.
38
42
44
46
Few-shot learning. Metric learning (i.e. learning to compare) a widely adopted approach in few- shot classification (e.g. (00 22
segmentation (e.g. 26 21 19), video object segmentation [5] and action localization in video 29 Multiscale processing often is not exploited this paradigm 29) even though it has the potential enrich the representations over which support-to-taree query comparators operate. While work few-shot segmentation has considered multiscale processing in comparators [19]. the model can be confused by erroneous correlations, especially the finest scales. In our work, we investigate meta-learning a multiscale transformer with a memory module that better separates the background from the novel class by allowing bidirectional information exchange across scales.
24
48
50
52
54
Multiscale transformers. Incorporating multiscale processing in transformers is emerging topic (e.g. action recognition [T0 15I and panoptic segmentation (7]). Previous work has mainly explored multiscale information on the encoder side [10] [5] 27], which is not sufficient when computing
56

---

dense predictions. In recent work 17 61, a multiscale transformer decoder was proposed, but its design exchanged information across scales on the compressed learnable queries. In our approach we preserve the spatiotemporal dimension in the multiscale processing to enable multigrid formulation. Thus, information exchange across scales on the detailed feature maps rather than compressed representation. We also operate on correlation tensors rather than directly on features maps to induce a stronger bias towards learning a comparator. Closely related to our work is the task of finding dense correspondences between images, where both global and local information is combined using transformers [8]; however, they have no notion of memory or meta-learning In contrast, our work explores multiscale memory learning in the transformer decoder within meta-learning framework. There has been previous works on memory augmented transformers 30 18 however, our approach is the first to explore multiscale memory learning in the transformer decoder. This novelty is crucial 69 to preserve the spatiotemporal dimension during the cross scale information exchange and to better separate the background and novel class. Additionally, we establish connections to classical multiscale processing methods [ which has not been explored in the context of transformers.
58
59
60
61
62
63
64
65
67
71
Multigrid methods. Multigrid methods 3 2 were initially developed to accurately solve large systems of partial differential equations in a computationally efficient manner and with reduced 74 residual error. They operate on multiple discretization levels, where interactions among fine and
73
75 coarse grids occur when deriving the approximate solution. The V-cycle correction scheme B is one form of a recursive solution that allows bidirectional information flow across the different
77 scales to ensure smooth solutions with low error as well as efficient computation. This approach inspires multigrid formulation that reduces the effect of erroneous correlations across the scales via bidirectional information exchange. While previous work has integrated multigrid- operations in convolutional architectures [14), we the first to explore multigrid processing within multiscale transformers. Additionally, our formulation is cast within a meta-learning framework that targets few-shot learning tasks associated with dense predictions in videos.
79
80
81
82
85
Contributions. In this paper, we present a novel comparator for few-shot learning tasks associated with dense predictions in videos. Our main contributions are threefold: (i) We present the first attempt to meta-learn a multiscale comparator between the support and target query sets in few-shot video dense prediction tasks. (ii) Our comparator encompasses a multiscale, multigrid transformer decoder that operates on correlation tensors between the support and target query set features with bidirectional multiscale information exchange. (iii) We present multiscale memory learning in the transformer decoder within a meta-learning framework that operates on top of the correlation feature pyramid to better separate the novel class from the background. We demonstrate our MMC transformer on two few-shot video tasks, few-shot video object and action segmentation, where our method outperforms the state of the art on both tasks. Our code will be publicly released upon acceptance.
84
85
87
88
89
81
92
2 Multiscale multigrid transformer comparator
88
94 In this section, we detail our multiscale comparator design. Inspired by classical multigrid methods 3.
95 we develop formulation that allows bidirectional information exchange across scales. Additionally,
96 to allow for information exchange across scales using detailed feature maps we perform cross attention
97 with learnable memory module and preserve the spatiotemporal dimension of the feature maps within our decoder. The proposed memory module helps to distinguish the novel class from the background and enhances support-to-tapee toarggettttttttte query correlation features.
96
99
100
2.1
Multiscale comparator transformer
Since we operating on multiple levels of feature abstractions and resolutions, we the subscript -p to denote the features from scale level E 1,,,.... P. which are extracted from late (coarse), p- 1, intermediate or early (fine), p P. stages. The input features for level p after flattening are Zp E ,,' xC,, where Hp, W, are spatial dimensions, T' is the clip length and Cp are
102
103
104
105
the channels for the corresponding scale. The input features are constructed on the support-query correlation tensors, as detailed in Sec. 3 We further project it with 1 x 1 convolutional layer, to reduce the dimensionality for a memory efficient solution, and end up with Z, E
108
107
108
We start by defining (multihead) attention 25 as,
An (X9,'* X²) = Softmax
X'
(I)
VD
3

---

MMC Tranaformer
Support
Multigrid Comparator Transformer
StekoDmaanc Companter
40 Convai Squaneze Operation
OP
Static
Concatinaton-
TIK-Wy- D.D
Mareory
78 K
7R
Backsonn
Steni Condittion
Wy
Qusty
TR
ConTra
Support
Query
Oynan Conmesion
Perture
Hay
Allention
Durest
Seymonattion
Dynamic
Backbonn
Figure 2: Overview of our architecture for few-shot video object and action segmentation. Features from convolutional backbone that capture information conveyed from single static images (a.k.a. static backbone) are extracted. can optionally (*) be combined with features from convolutional backbone that capture dynamics conveyed from a set of frames (a.k.a. dynamic backbone). 4D correlation tensors are computed among spatial/spatiotemporal features from support and target query sets using Φb. Subsequently, the tensors go through a 4D convolution that yields 2D feature maps for memory efficiency. A multigrid multiscale comparator transformer is used on the input pyramid, { Cross attention among the feature pyramid and the learnable memory, B, is performed to generate attention maps. {ay)p= P
The cross attention module encompasses key, Wk, and value, W... weight matrices. The attention maps are used to re- re-weight the memory, B, in each spatiotemporal position to enhance query features and separate the background from the novel class in the output. {Op) Eq. Information is transferred bidirectionally between scales (denoted by red arrows) and the final output at the finest scale is used as input to the segmentation decoder.
109 where An represents the attention per head, ht and the full multihead attention is, A, that corresponds to the concatenation of each head's 's output. The inputs X X X represent the query, key, value,
111 resp., and Wo, W W E RDD are the query, key and value weight matrices, resp.. for feature dimensions. We use fixed spatiotemporal positional embeddings, E TH,,,W,* corresponding to every scale level, p. and learnable scale embeddings, F.J E IID following 7 We repeat the scale embedding at all spatiotemporal positions, T. Hp. Wp, resulting in E R,,,W,
118
115
Importantly, we seek a formulation that preserves the spatiotemporal dimension during the information exchange across scales in the multiscale transformer decoder and cross attention. This ensures the multiscale processing and information transfer can capture detailed information in the feature maps rather than compressed representation. Therefore, in contrast to previous work that instantiate
116
117
119 a set of learnable queries 77. we meta-learn a memory module that has D dimensional vectors, B E RNXDD with N memory entries that are shared across all decoding layers and scales. This multiscale memory learning allows the per-scale decoded output to preserve the spatiotemporal dimension, unlike (7]. We perform cross attention per resolution and feature abstraction level, p. where we instantiate the multihead attention, (T) as
121
123
A( F. B B),
where E, XD are learnable memory positional embeddings. Thus, we perform multiscale processing while maintaining the spatiotemporal dimension for the output, Op E RH W D as illustrated in the MMC transformer block of Fig. 2 This mode of operation ensures cross- scale communcation with the detailed feature maps and maintains the spatiotemporal dimension output
124
125
126
127
128
from our decoder. For every scale level p, applying A will learn to attend among the different set of learnable memory features based on their relevance to the support-target query correlation features. It then aggregates the learned memory based on these attention maps to better separate the novel class and the background. Since want to allow for information exchange across scales, we
130

---

Zp-= 2pr 11'00
(3)
where I performs bilinear interpolation match the size from level pl. The cross attention operations, (2). are performed consecutively on all levels and are repeated Ni times, with Ni a hyperparameter denoting the number of decoder layers for each level. The final output from multiscale comparator is Op that will be used later for the final segmentation prediction. Since our design does collapse the spatiotemporal dimension, the output, O,, for every level, p, maintains detailed information necessary for the final segmentation task, unlike previous work used in a non-meta-leamiin framework for panoptic segmentation [6].
133
135
136
137
138
139
Typically, previous work has focused on multiscale processing in the transformer decoder with multiscale query learning that contextualizes set of Icarnable features, Q N 6 The multiscale query learning output can be seen as,
141
Oauery A(Q Es. Zp
E 2p),
(4)
where Oquery E R are a set of compressed learnable queries that are exchanged across scales. This formulation uses learnable features. Q, as queries and the multiscale feature maps, Zp. as keys and values resulting in outputs per scale of dimension, N x D. In contrast, our formulation of multiscale processing with memory module uses the learnable features, B, keys and values; hence, the queries are the detailed feature maps, Zp. which yields per-scale output of dimension, T D. Thus, the detailed feature maps necessary for segmentation are lost during the communication across scales in the former, but are preserved in ours. We empirically validate this distinction in Sec. 4.2
142
143
144
145
146
147
148
149
150
2.2
Multiscale multigrid attention (MMA)
Now that we have presented means to leverage multiscale processing in the transformer com- parator, we further explore the different forms of information transfer across different scales. across scales can be conducted in (i) coarse-to-fine process- ing that is performed a stacked manner for multiple layers, as shown in Fig. 3 (left) (ii) multigrid
152
Communication
154
Late/Coarse
I
I
I
I
156
158
MidiMid
I
I
I
I
160
formulation that allows for
Early/Fine
Ip.
I
I2
Ip
bidirectional information
162
transfer between the coarse and fine scales, as shown in Fig. (right). Bidirectional exchange ensures that the coarse-grained smoothed
Bidirectional (Multigrid)
Stacked
163
184
165
Figure 3: Two variants of information exchange across scales with dif- ferent feature abstractioon/resoluiio levels: stacked vs. multigrid. Fea- ture abstraction levels indicate carly-to- stage features and different resolution levels indicate coarse-to- fine levels. I, denotes multihead attention, (T). for level p followed by bilinear interpolation, (3). In the stacked variant, I is for the jth iteration.
186
167
correlation features with
168
high level semantics can modulate information
189
in the fine-grained ones, while allowing fine-gre -raiind detailed information to affect the coarse-grained.
171
173 Inspired by classical multigrid methods, we present multiscale, multigrid attention module that allows for bidirectional information transfer across different scales, as shown in Fig. 3 (right), similar the V-cycle correction scheme [3]. In our case, since the multiscale, multigrid attention operates on correlation tensors from different scales, it ensures smooth solutions through bidirectional information
174
175
176
177
exchange between coarse and fine scales. It thereby avoids erroneous correlations that might exist at the different scales, where the fine scale can exhibit erroneous correlations as it only captures low level semantics and the coarse scale can exhibit noisy correlations from being a subsampled signal.
178
179
Unlike classical multigrid methods our multiscale input exhibits both different levels of resolution as
181
well as different feature abstraction levels (i.e. early, mid and late stage features), instead of solely
using different sampling rates the same information as classical multigrid. Another perspective
5

---

183
that motivates the Searse-to-fine and fine-to-coars communication is inspired from earlier work that has shown convolutional layers mainly act band pass filtering (12). which indicates that features from every layer capture certain set of frequency components. Thus, the bidirectional information transfer can enrich the features through exchanging information across the different set of frequencies captured at each layer.
184
185
186
187
188
Consider the stacked multihead attention, which takes as input multiscale pyramids. It is composed of a series of multihead cross attention (2). followed by merging the two consecutive levels and using bilinear interpolation to match the scale, (3). The combined aforementioned operations are denoted as, Ip for level E {1, ........P............. P}. Thus, we formulate stacked multihead attention (SMA)
189
190
191
192
in coarse-to-fin processing per iteration as
(5) where o denotes function composition and I. corresponds to the jth iteration as the operations in the stacked multihead attention are repeated N₁ times. In comparison, multigrid multihead attention (MMA) processing is defined
SMA = =Ia 0
198
194
195
MMA Iι o oI 1.......0.. op o II 02..... Ip.
196
It is seen that the multigrid approach, (6), encompasses bidirectional information exchange across scales similar to classical recursive methods [3) whereas the stacked approach is strictly coarse-to-fie fine.
197
198
Sec. 4.2 provides empirical support for the superiority of the multigrid formulation.
3
Learning scheme
199
200
In this section, we summarize the few-shot video setup, and our scheme for meta-learning the multiscale comparator with multigrid, multiscale attention for an improved few-shot video tasks. Then we describe two case studies for few-shot video object and actor/action segmentation.
201
202
203
Fhot setup. We formulate the few-shot video object or actor/action segmentation task as follows 29). Let Dtrain and Drest be training and testing data, resp. For every dataset, split the C categories into O folds, each fold will have E novel categories, Crest, and C - E as base classes, Cirain- Both the training and test classes do not intersect, Cirain N Crest 0. In the meta-training phase, we sample No tasks from the corresponding dataset with support and target query set pairs {St, N for classes in Ctrain- Similarly in meta-testing we sample support and target query sets but for classes in Crest. The target query set contains video frames = {xx(") N. Jt=1.
205
206
207
208
209
where N₂ is the number of frames and superscript (9) denotes the target query set. In the case of video object segmentation, the support set in one-way K-shot task has K image-label pairs k== for class to be separated from the background. The superseript (e) denotes support set and Mr (s) is a binary segmentation mask for the class considered. The image-label pairs X E "xxx33 X3 and M2 E "xW, with H x W spatial dimensions. In the case of video actor/action
210
211
212 S (x(*), Mr (*)
213
215 segmentation the one-way K-shot task has K trimmed video-label pairs S = (x!") M (*) Jk=1'
216
The binary segmentation mask Mg
(4) is for one frame in the trimmed video for an actor/action class. Thus, the video-label pairs are X₁ E R xxx3 x3 and M₁ E HxW.
217
218
Meta-learning a multiscale comparator. We start with introducing an overview of the full architecture of our multiscale comparator, as shown in Fig. 2 We initially assume one-shot setting, then discuss the K-shot extension later. We use a pretrained convolutional backbone with fixed weights, F. that are not updated during the meta-training process, to compute the support and target query set features. The features for the one-shot support and target query sets are extracted for layer, I, as, f((a) (x(())), {(") (X(())),, resp., for the set of L layers. Let <(-,-) denote the (hyper)correlation encompassing the comparisons between its arguments, both of which are tensors, and + be concatenation on the channel dimension for m consecutive layers with the same spatial dimensions. Then, we define a 4D hypercorrelation tensors pyramid with P levels as
219
220
221
222
224
225
226
H, + o({('), f/("), cf- [19]. We use the hypercorrelation squeeze network [19, Dnypercont which represents one form of performing efficient 4D convolution and generates a 2D feature pyramid
228
that is further flattened, { Drypercorr (1 {()))_
229
230
Our multiscale comparator transformer uses the features extracted on different levels by perform- ing cross attention to learnable memory, B, to yield the final output feature maps, Op

---

Dmultscate D1ultscatee122ppppp (1Zp)p=1: B). The cross attention re-weights the memory features to enhance the query feature maps and separate the background from the novel class based on their correlation to the support set. Attention and feature aggregation subsequently is computed in pixel-wise manner across all levels to produce the final output features, Op. Our multiscale multigrid transformer decoder enriches the features and allows bidirectional exchange of information across scales. The
232
233
235
236
237
output features, Op, from the multiscale comparator are used in a segmentation decoder, Dseg to
compute the final predictions M = Deeg (ieg(O)), (Op). The predictions, M, are for Ne classes that include the background class. The hypereorrelation squeeze network, Dhypercorr. our multiscale comparator, D multisalee and segmentation decoder, Dxeg- are meta-trained with simple binary cross entropy that encourages the model to segment the class of interest guided by the support set. During few-shot
238
239
240
241
242
inference when operating with K-shot support set, we follow the setup from [19] which infers the target query prediction with every example in the support set separately, sums all predictions and divides by the maximum score.
243
245
Static/Dynamic comparator transformer. In this section, we introduce two related tasks (i.e. few-shot video object segmentation and few-shot video actor/action segmentation) and show how our multiscale multigrid comparator operates on both. The term static factor indicates information learned from a single frame (e.g. texture and colour), while the dynamic factor indicates information extracted from a consecutive set of frames (e.g. motion). We meta-learn static comparator in the case
248
247
248
249
250
of few-shot video object segmentation, while in the case of few-shot video actor/action segmentation
251
we find beneficial to meta-learn both a static/dynamic comparator, as shown Fig. 2 We describe
252
each setting in turn next.
253
In the case of few-shot video object segmentation, which is simpler task, the goal is to segment the novel class per target query frame in the video. Since dynamics might not have a significant effect on identifying the semantic categories (e.g. person moving or standing is class person) we design a static comparator. We use per-frame features extracted from a 2D backbone (e.g. ResNets [13). The support set in this setup a set of single images that can already describe the semantic category. Thus, the fea-
254
255
256
257
tures extracted for support and target query are f( E R XCi and f E x
258
resp. The corresponding hypercorrelation pyramid is computed as the set of correlation tensors for each target query frame in the video and the support set features, {H, E R H, x W x X x W, CCp } Consequently, the extracted features from the hypereorrelation squeeze network that is applied in- dividually per frame and flattened, &zp E TH,,,, XCp pp=1. are used in our MMC transformer to generate enhanced query features, Op RT Finally, the segmentation predictions from the decoder is given by M RTXH*WXN.
259
280
261
282
264
265
In the case of few-shot actor/action segmentation, the dynamic factor is important identifying the action while the static factor delineates the object/actor boundaries. Correspondingly, we meta-learn a static/dynamic comparator that fuses the information from both. Toward this end we use a 3D backbone (e.g. X3D (II) to extract spatiotemporal features that we refer to as dynamic features. In complement, we use a 2D backbone to extract the first frame features from the current input clip, we 269
267
refer to these as static features. We use superscript (dy), _(*t) to denote the corresponding dynamic and static tensors, resp. Additionally, the support sets are trimmed videos instead of a single image.
270
271
Thus, the dynamic features extracted are given as f(() (/(())))) < T H,x XW, The corresponding dynamic hypercorrelation pyramid is computed after averaging along the temporal dimension in each
272
273
274
pp- Similarly, the static hypercorrelation pyramid is built on top of the features extracted for the first frame in
layer, then computing the correlation tensors, {Er,[dh)
R.H,, X,,,,W,,
275
the current clip as (Hr((t) E HR, x xW, xWxxf"0) }p-1- Then, we combine the correlations
276
from both static and dynamic features to yield our final pyramid, {Hl, H1 + (((a)) HH,, Rd ff, x W, x H2 xW,, XC(04) + C(dn)) Ip= The extracted features from the hypercorrelation squeeze are p= and used as inputs by our MMC transformer to generate the final features, Op E H Finally, the segmentation prediction for an input clip from the decoder is given by M E " We use a temporal sliding window over the untrimmed target
277
flattened as 12p E HR JP
279
280
281
282
query video and generate clips that are used to predict the segmentation.
7

---

4
Experimental results
283
4.1
Experiment design
Datasets and evaluation protocol. We evaluate on two standard benchmarks, YouTube- VIS FS- VOS [3] and Common A2D (29]. to facilitate comparison to the state of the art in few-shot video object and action segmentation. We follow their standard evaluation protocol and describe the details
286
287
288
in the supplement.
289
Implementation details. For few-shot video object segmentation we follow the same architectural
290
choices as state-of-the-art approaches [5] to facilitate comparison, where we build on a ResNet-
50 [3] backbone pretrained on ImageNet [2]. For few-shot actor/action segmentation, we use both
292
ResNet-50 pretrained on ImageNet [ and X3D [I] pretrained on Kinetics 4 following [29] for
the static and dynamic backbones, resp. In our MMC transformer, the number of decoder layers per
294
scale is set to Ni- = and the number of entries in our learned memory is N = 20. We meta-train model and baseline [9] on the base classes for given fold using cross entropy, with 50 epochs on YouTube- VIs,, while we use 70 epochs for A2D. We use the same hyperparameters for both our approach and the baseline, where we use AdamW [17] with learning rate of 1 x 10-3 3 and weight decay of x 10-4, Random rotations and flipping data augmentation is used for both. Additional implementation details are provided in the supplement.
296
298
299
300
4.2
Ablation study
301
We start an ablation study on our two main contributions: multigrid (i.e. bidirectional) information exchange across scales and multiscale memory learning which preserves the spatiotemporal dimension
302
303
during multiscale processing. In Table [ we compare four variants: (i) our baseline without
304
transformer decoder [19]. (ii) the multiscale transformer with learnable queries that pools the spatiotemporal dimension following 4 which we call Query, (iii) our multiscale transformer that preserves the spatiotemporal dimensions, (2). and follows a stacked information flow across scales as shown Fig. 3 (left), which we call Stacked and (iv) our MMC transformer with bidirectional information flow as in Fig. 3 (right), which we call Multigrid. We can see that across all folds the mloU for Query is lower than any of the variants (i.e. Stacked and Multigrid) and is even lower than the baseline on average. It shows our approach of multiscale memory learning to preserve the spatiotemporal dimension in the multiscale transformer comparator yields more accurate segmentations. Additionally, we ablate both forms of information flow across scales, Stacked, following (5) and Multigrid following (6). Here, it is seen that the bidirectional information flow in the multigrid approach improves over the stacked coarse-to tone across three of the four folds. It is also seen that our multigrid approach improves over the baseline [19 on average and especially on the first two folds. We hypothesize that the failures in the last two folds were due to under-segmmenattion exhibited by our comparator that made the model more restrictive on what is considered part of the novel class, qualitative examples presented in the supplement. For example, in split two class Hand, although the target query video should have the entire arm segmented the support set mainly has the hand without the arm.
305
307
309
311
312
313
315
316
317
318
319
320
321
we compare our baseline 19,, the improved baseline (i.e. baseline++) that combines correlation tensors from static and dynamic factors and our full approach MMC transformer with static and dynamic features on Common A2D. A greater improvement is seen with respect to the baseline than in object segmentation ablation, with up 7% gain in the five-shot scenario. Additionally, these results demonstrate the flexibility of our multiscale comparator, as it can operate with any backbone network (e.g. ResNet-50 133, X3D (11 the combination of the two) and is able to operate beyond few-shot video object segmentation to actor/action segmentation.
In Table 3
322
323
325
326
327
328
4.3
Comparison to state-of-the-art approaches
329
We provide comparison with existing approaches on YouTube- VIS FS-VOS in Table where our method shows a notable gain of 4.4% with respect to the recently presented many-to-many attention comparator [5] This result demonstrates that our multigrid, multiscale comparator helps separate
330
332
the novel class with respect to the background better than previous approaches. Table 3 shows comparisons with the state of the art for sho-shot video actor/action segmentation, with focus on the
8

---

(a)
(c)
Figure 4: Qualitative results showing better separation of background vs. novel class for MMC transformer. (a) Five-shot support set. (b) Baseline [9]. (c) MMC transformer (ours). The support
groundtruth and query predictions are marked in red. Video results are provided in the supplement.
mioU
Method
mloU
Method
2
3
4
Mcan
2
3
4
Mean
PMMs 28
32.9
61.1
56.8
55.9
51.7
Baseline
49.5
69.5
63.8
65.2
62.0
PFENet 23
37.8
64.4
56.3
56.4
53.7
Query
49.4
70.5
62.9
64.3
61.8
PPNet
45.5
63.8
60.4
58.9
57.1
Stacked
50.7
69.8
63.2
64.4
62.0
DANet w/o OL [S]
41.5
64.8
61.3
61.4
57.2
Multigrid
51.5
70.6
63.0
64.6
62.4
DANet [S]
43.2
65.0
62.0
61.8
58.0
MMC transformer
51.5
70.6
63.0
64.6
62.4
Table 1: Ablation study on YouTube VIS FS-
VOS folds 1, 2, 3 and 4 using our MMC trans-
Table Comparison of mloU for few-shot video object segmentation to the state of the art on YouTube- VIS folds I, 2, 3 and with a five-shot support set. DANet w/o OL indicates the variant without online learning.
former with five shot support set.
mloU
Method
Static
Dynamic
1-shot
5-shot
Co-attention [21]
43.3
44.8
Single-scale Transformer 29
50.6
52.5
Baseline
x
18.1
21.9
Baseline++
47.7
MMC transformer (Ours)
51.9
54.5
Table 3: Comparisons of mloU for few-shot actor/action segmentation with respect to the state of the art on Common A2D and our basclines, with one-shot and five-shot support sets. Static/Dynamic indicates the use of the corresponding factor, where the static compares ResNet-50 features and the dynamic compares X3D [ spatiotemporal features.
[ spatial
334
segmentation task 29 21. seen that our approach consistently outperforms the others both the one-shot and five-shot scenarios on Common A2D.
Fig. shows qualitative results YouTube-VIK VIS FS-VOS, where method improves the baseline with better separation of the novel class with respect to the background and better delineated boundaries. We provide Common A2D qualitative results the supplement.
337
338
Discussions and Conclusion
339
340
We presented a novel MMC transformer that exchanges bidirectional information across scales for comparing between the support and query sets and reducing the impact from erroneous correlations. Our transformer decoder is designed to preserve the spatiotemporal dimension in our bidirectional multiscale processing, unlike previous methods. We meta-learn our multiscale comparator trans- former along with a memory module that better separate the background from the novel class. We showcased the MMC transformer tor/action segmentation. Our method outperforms the state of the art on both tasks. A limitation of our method for few-shot video object segmentation is its reliance on 2D backbone features, whereas in the video domain 3D spatiotemporal features may improve discriminability. We leave it for future
342
343
344
345
two use cases, few-shot video object segmentation and ac-
348
347
348
349
work to explore spatiotemporal models as backbones and investigate FSVOS benchmarks that reflect
350
the fact that certain semantic categories can exhibit different motion patterns (e.g. four legged vs. two
351
legged mammals vs. reptile motion).

---

References
352
353
[1] Kelsey Allen, Evan Shelhamer, Hanul Shin, and Joshua Tenenbaum. Infinite mixture prototypes for few-shot learning. In International Conference on Machine Learning, pages 232-241, 2019.
355
[2] Achi Brandt. Multi-level adaptive solutions to boundary-value problems. Mathematics of Computation, 31(138):333-390 1977.
356
357
[3] William L Briggs, Van Emden Henson, and Steve F McCormick. A multigrid tutorial. SIAM, 2000.
358
359
[4] Joao Carreira and Andrew Zisserman, Quo vadis, action recognition? A new model and the kinetics dataset. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
381
382
[5] Haoxin Chen, Hanjie Wu, Nanxuan Zhao, Sucheng Ren, and Shengfeng He. Delving deep into many-to-many attention for few-shot video object segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 14040-14045 2021.
383
384
365
[6] Bowen Cheng, Anwesa Choudhuri, Ishan Misra, Alexander Kirillov, Rohit Girdhar, and Alexan- der G Schwing. Mask2former for video instance segmentation. arXiv preprint arXiv:21 107644
386
2021.
368
77] Bowen Cheng, Ishan Mísra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
369
370
371
[8] Scokju Cho. Sunghwan Hong, Sangryul Jeon, Yunsung Lee, Kwanghoon Sohn, and Seungryong Kim. CATS: Cost aggregation transformers for visual correspondence. Advances in Neural Information Processing Systems, 34, 2021.
372
373
374
99] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hicrarchical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
376
377
[10] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE International Conference on Computer Vision, pages 6824-6835, 2021.
378
379
380
[11] Christoph Feichtenhofer. X3D: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
381
[12] Isma Hadji and Richard P Wildes. Why convolutional networks learn oriented bandpass filters: Theory and empirical support. arXiv preprint arXiv:2011. 146655 2020.
384
385
[13] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
386
387
388
[14] Tsung- Wei Ke, Michael Maire, and Stella X Yu. Multigrid neural architectures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages
309
2017.
390
[15] Yanghao Li, Chao- Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
392
394
[16] Yongfei Liu, Xiangyi Zhang. Songyang Zhang, and Xuming He. Part-aware prototype network for few-shot semantic segmentation. In Proceedings of the European Conference on Computer Vision, 2020.
385
396
397
[17] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. 2019.
10

---

[18] Francesco Marchetti, Federico Becattini, Lorenzo Seidenari, and Alberto Del Bimbo. MANTra: Memory augmented networks for multiple trajectory prediction. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7143-7152, 2020.
400
402
[19] Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot seg- mentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 6941 16452, 2021.
403
404
405
[20]
Hang Qi, Matthew Brown, and David G Lowe. Low-shot learning with imprinted weights. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5822-5830, 2018.
408
407
408
[21] Mennatullah Siam, Naren Doraiswamy, Boris N. Oreshkin, Hengshuai Yao, and Martin Jager- sand. Weakly supervised few-shot object segmentation using co-attention with visual and semantic embeddings. In Proceedings of the International Joint Conference on Artificial Intelligence, pages 860-867, 7 2020.
409
410
412
[22] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems, 2017.
413
[23] Zhuotao Tian, Hengshuang Zhao, Michelle Shu, Zhicheng Yang, Ruiyu Li, and Jiaya Jia. Prior guided feature enrichment network for few-shot segmentation. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2020.
415
418
417
[24] Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pierre-Antoine Manzagol, and Hugo Larochelle. Meta-dataset: A dataset of datasets for learning to learn from few examples. In International Conference on Learning Representations, 2020.
419
420
421
[25]
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems, 30, 2017.
422
423
[26]
Kaixin Wang, Jun Hao Liew, Yingtian Zou, Daquan Zhou, and Jiashi Feng. PANet: Few- shot image semantic segmentation with prototype alignment. In Proceedings of the IEEE International Conference on Computer Vision, pages 9197-9206, 2019.
425
426
427
[27] Wenxiao Wang, Lu Yao, Long Chen, Binbin Lin, Deng Cai, Xiaofei He, and Wei Liu. Cross- former: A versatile vision transformer hinging cross-scale attention. In International Conference on Learning Representations, 2022.
429
[28] Boyu Yang. Chang Liu, Bohao Li, Jianbin Jiao, and Qixiang Ye. Prototype mixture models for few-shot semantic segmentation. Proceedings of the European Conference on Computer Vision, pages 763-778. 2020.
431
433
[29] Pengwan Yang, Pascal Mettes, and Cees GM Snoek. Few-shot transformation of common actions into time and space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16031-16040) 2021.
435
[30] He Zhao, Isma Hadji, Nikita Dvomik, Konstantinos G Derpanis, Richard P Wildes, and Allan D Jepson. P3IV: Probabilistic procedure planning from instructional videos with weak supervision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
437
439
Checklist
440
1. For all authors...
442
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
443
(b) Did you describe the limitations of your work? [Yes] See Sec.
11

---

445
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplement
448
447
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
449
2. If you are including theoretical results...
450
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
451
(b) Did you include complete proofs of all theoretical results? [N/A]
452
3. If you ran experiments..
453
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplement
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See supplement.
456
457
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report average over multiple runs to provide direct comparisons with the literature as their standard protocol.
458
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplement
461
462
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
463
(a) If your work uses existing assets, did you cite the creators? [Yes] We used publicly available research benchmarks and cited them
464
(b) Did you mention the license of the assets? [Yes] See supplement
(c) Did you include any new assets either in the supplemental material URL? [No]
487
(d) Did you discuss whether and how consent was obtained from people whose data you't 're using/curating? [N/A]
488
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
470
471
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
473
474
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
475
476
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "xPGI67I6LDe", "reviewer": "Reviewer_ZBpu", "summary": "This paper presents a multi-scale transformer as a comparator in few-shots video segmentation tasks. Compared with prior works that mostly use one feature resolution for comparison, they leverage multi-scale comparison to obtain both coarse and fine associations. A multi-grid mechanism is used for communication between different scales. The authors use a pretrained memory module to perform cross attention with the correlation volume. The resultant method outperforms current state-of-the-art methods in few-shot video object segmentation and few-shot actor segmentation.", "questions": "Is there any running time analysis? See also weaknesses.", "limitations": "The authors mention the use of 2D features as a limitation. ", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper is well-motivated. Having multi-scale correlation is a sensible solution to few-shots video segmentation problems. I also like the idea of keeping the spatial resolution in the transformer decoder which is again important to segmentation tasks. \n- The proposed method achieves good results on benchmark datasets, which further support the previous point.\n\n\nWeaknesses:\n- I think the writing can be improved. The authors rely heavily on abstract notation and procedural manipulation of these symbols without a high-level picture (in section 3). Most symbols are also not semantic so readers have to go back and forth many times to recall the meaning of a symbol. Yet, section 2.2 is too high-level and does not connect with the rest of the paper. The exact mechanism of multigrid bidirectional communication is not well explained. Adding some figures or rewriting sections 2 and 3 might help. This obstructs my understanding of this paper so I am putting a low confidence.\n- There should be multiple transformer decoder layers (L134). This is not shown in Figure 1 or 2 (like with a *n). \n- A “memory” module should carry information across different timesteps, iterations or scales. Currently I find that the memory module is meta-learned and is not updated during inference time. If this is the case, it should not be called a memory module as it does not dynamically store information. It is more like a set of queries.\n- The features for the support and target query are extracted from the same feature extractor (L223). This means the mask is not an input to the target query and the correlation volume does not carry mask information. I do not find this mask used as input in any other places. This does not make sense as the output should depend on the support mask.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "b4RXCULAnbZ", "reviewer": "Reviewer_DSqR", "summary": "This paper proposes a multiscale multigrid comparator transformer to address the few-shot video segmentation task. Experiments on few-shot video object segmentation and actor/action segmentation show effectiveness.", "questions": "Please refer to the Weaknesses above for the details.", "limitations": "Yes. There's no suggestions.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pos: A novel comparator for few-shot learning tasks associated with dense predictions in videos is presented.\n\nNeg:\n1. Please explain what is static backbone and dynamic backbone.\n2. In Equation 1, what are X_q, X_k and X_v respectively?\n3. Equation 3 seems to be only a Unidirectional cross-scale information exchange. Where does the bidirectional information exchange mentioned by the author reflect?\n4. This paper is hard to follow.\n5. Writing needs to be improved.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "JvWOM-eW5K3", "reviewer": "Reviewer_973M", "summary": "The paper proposes a multi-scale multi-grid comparator transformer for Few-shot video segmentation that sims to overcome limitations of prior Few-Shot segmentation methods that limit feature comparisons to only a single feature layer thus ignoring information that might be valuable. Typically features in the early net work layers aid in precise localization and deeper layer features give higher level semantic information. The work aims to combine both coarse and fine-grained features via multi-scale and multi-grid operations to improve few shot segmentation. In addition, the work also introduces a novel multiscale memory learning in the decoder which helps preserve the details within feature maps across scales \n\nContributions of the work are stated as:\n+ a meta-learning method for multiscale comparison of query and support features is introduced\n+ this comparator is a multiscale, multigrid transformer decoder that allows bidirectional multiscale information flow\n+ finally the work also proposes a multiscale memory learning module within the transformer decoder", "questions": "Questions:\n\nQ) how does this method work on standard image few-shot segmentation tasks? since reference [19] in the paper was referred to, it would be nice to see results on COCO 20i, Pascal 5i and FSS 1000 datasets.", "limitations": "The work only talks about limitations in terms of 2D backbone features. But it would be important to examine failure cases and see why that is happening.\n\nAdditionally, the impact of pre-trained features can be examined to further understand how important such features are?\n\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ well written paper\n+ tackles an important problem \n+ proposes improvements and novel architectural additions that improves the prior state-of-the-art results by a large margin\n\nWeakness\n+ evaluation can be more rigorous\n+ how does this method work on standard image few-shot segmentation tasks? since reference [19] in the paper was referred to, it would be nice to see results on COCO 20i, Pascal 5i and FSS 1000 datasets.\n\n\n\nReference [19] in paper: Juhong Min, Dahyun Kang, and Minsu Cho. Hypercorrelation squeeze for few-shot segmentation. In Proceedings of the IEEE International Conference on Computer Vision, pages 6941–6952, 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ImFt3l-J3d1", "reviewer": "Reviewer_HsqG", "summary": "**Problem**: This work addresses the problem of identifying correspondences between a given query feature map and a \"support\"/reference feature map. The goal for the correspondences is to be able to perform few-shot object segmentation or actor segmentation in videos.\n\n**Solution**: The paper proposes a novel architecture (MMC) that takes as input dense feature similarity tensors and outputs frame-wise segmentation maps for the videos. The input similarity tensors measure the similarity between the support image features and features of the query video frames, at multiple levels of a CNN feature pyramid. The proposed MMC model comprises of a transformer-based encoder that reasons about the similarities using information from all levels of the similarity tensor pyramid and a decoder to output the frame-wise segmentation masks. \n", "questions": "- See comment on \"novelty\" in weaknesses above. \n- For the task of Few Shot Video Object Segmentation, it looks like the convention is to report two metrics: mIoU and contour accuracy (as in DANet [5]). Is there a reason the contour accuracy is not reported here?", "limitations": "Discussion on limitations of the presented model is very limited in the paper. \nDiscussion on societal impact is not covered in the paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n### Intuitive Architecture \nThe proposed architecture is fairly straightforward to follow. The model involves a transformer architecture that uses learned key/value features to compute the attention and output features. These output features are pooled across the feature pyramid using a bidirectional pooling method. Finally the pooled features are passed to a decoder model to output the segmentation maps.\n\n### Results \nThe comparison to state-of-the-art approaches demonstrates that the presented model can produce improved segmentation maps. The quantitative results show improved mIoU values across the two tasks: object segmentation and actor segmentation.\n\n---\n\n## Weaknesses \n\n### Novelty \nWhile the results are impressive, in the current form of the text, it is difficult to recognize the novelty of the work. This is partly because the novelty or importance of some of the design choices seems overstated in the text. \n- The idea of reasoning across multiple feature levels of a CNN/similarity tensors is not novel (as pointed out by the authors too). This is the key idea of the \"Multiscale Multigrid Attention\". However, the specific implementation of this idea could be unique/novel. But it is not clear whether this is the case here. Section 2.2 seems to be inflating the contribution here. It is not clear why the stacked idea is presented since it is not the obvious/conventional method of performing feature pyramid pooling (to my best knowledge). And in addition to that, the stacked method actually performs almost equally well (in Table 1). Furthermore, the relationship to V-cycle correction is extremely loose. These explanations make it a bit harder to discern what the contribution is here. \n- One of the contributions is the idea of using a memory module. The memory module is essentially learned keys (and values) which are shared across different levels. This idea seems novel, but as a core contribution this has not been investigated enough experimentally. For example, is it necessary to share this across the different levels? Do the same features need to be used as keys and values? \n- The text seems to be using the term meta-learning (and meta-training, meta-testing) a lot. Is this one of the contributions? It is not clear why this is referred to as meta-learning. As far as I understand, meta-learning is the idea of optimizing through gradient descent update steps (or other optimization processes) to facilitate few-shot transfer learning - possibly introduced here first: \n>\"Finn, C., Abbeel, P., & Levine, S. (2017, July). Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning (pp. 1126-1135). PMLR.\" \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject