Mask Propagation for Efficient Video Semantic Segmentation
Authors: Yuetian Weng
Mingfei Han
Haoyu He
Mingjie Li
Lina Yao
Xiaojun Chang
Bohan Zhuang
Venue: NeurIPS 2023 poster (2023)
Keywords: Video Semantic Segmentation; Inference Efficiency
TL;DR: We present an efficient mask propagation framework with a novel query-based flow estimation module for video semantic segmentation.

Abstract: Video Semantic Segmentation (VSS) involves assigning a semantic label to each pixel in a video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS. Our approach first employs a strong query-based image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design a flow estimation module utilizing the learned queries to generate a set of segment-aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped to serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by 4.0% mIoU, requiring only 26% FLOPs on the VSPW dataset. Moreover, our framework reduces up to 4× FLOPs compared to the per-frame Mask2Former baseline with only up to 2% mIoU degradation on the Cityscapes validation set. Code is available at https://github.com/ziplab/MPVSS.

Mask Propagation for Efficient Video Semantic Segmentation
Yuetian Weng 1.2
Mingfei Han 3,4
Haoyu He'
Mingjie Li
Lina Yaol
Xiaojun Chang
Bohan Zhuang"
ZII Lab, Monash University Baidu Inc.
ReLERR, AAII, UTS
1Data61, CSIRO
Mohaaeed bin Zayed University Al
Abstract
Video Semantic Segmentation (VSS) involves assigning semantic label to each pixel in video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS. Our approach first employs strong query- based image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design flow estimation module utilizing the learned queries to generate set of segment-awer aware aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intense and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by 4.0% mloU, requiring only 26% FLOPs on the VSPW dataset. Moreover, our framework reduces up 4x FLOPs compared to the per-frame Mask2Former baseline with only up to 2% mloU degradation on the Cityscapes validation set. Code available at /ttps://githut com/ziplab/MPVSS
segmentors, alleviating temporal redundancy
Introduction
Video Semantic Segmentation (VSS), fundamental task in computer vision, seeks to assign semantic category label to each pixel in a video sequence. Previous research on VSS has leveraged developments in image semantic segmentation models, e.g., FCN [40] and Deeplab [82, 4], which made tremendous progress in the field. However, adapting image semantic segmentation models to VSS remains challenging. On one hand, sophisticated temporal modeling is required to capture the intricate dynamics among video frames. On the other hand, videos contain a significantly larger volume of data compared images. Processing every frame with strong image segmentor can incur significant computational costs.
Prior work in VSS mainly focuses on leveraging both temporal and spatial information to enhance the accuracy of pixel-wise labeling on each video frame, building upon the pixel-wise classification paradigm of traditional image semantic segmentors. Specifically, these methods exploit temporal
Work done during an intersship at Baidu Inc. Corresponding author. Email: bohan. zhuang waagg.@maaa.oooo
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

GT
Query based based
Optical
Figure I: Motivation of the proposed MPVSS. Three consecutive frames with ground truth category labels are sampled from a video in VSPW [42], illustrating a strong correlation between video frames along the temporal dimension. This observation underscores the significant temporal redundancy present in videos. The remaining columns contrast the proposed query-based flow and traditional optical flow, each showing a normalized flow map, a mask prediction, and the final predicted semantic map, respectively.
information by integrating visual features from previous frames into the target frame, employing techniques such as recurrent units [26, 43), temporal shift [33, 1J, or spatial-temporal attention modules [19, 44, 28, 16, 53], among others.
More recently, with the prevalence of the DETR [2] framework, DETR-like segmentors [8, 515 have dominated the latest advances and achieved -tate-of-the-art performance in semantic segmentation. Notably, MaskFormer series [8, 7] learn a set of queries representing target segments, and employ bipartite matching with ground truth segments as the training objective. Each query is learned to predict a binary mask and its associated class prediction. The final result is obtained by combining the binary masks and class predictions of all queries via matrix multiplication. However, these models require computing high-resolution per-pixel embedding for each video frame using a powerful image encoder, leading to significant computational costs. For instance, processing an input video clip with 30x 1024x 2048 RGB frames using the strong Mask2Former [7] requires more than 20.46T Floating-poing Operations (FLOPs) in total. Such computational demands render impractical to deploy these models in real-world VSS systems.
A typical solution is to incorporate temporal information guided by optical flow [50, 82, 22, 66) to reduce the redundancy. The related literature proposes to propagate feature maps from the key frame to other key frames using optical flow. The motivation mainly comes from two aspects. First, videos exhibit high temporal redundancy due to the similar appearance of objects or scenes in consecutive frames [13], especially in videos with a high frame rate. As shown in Fig. 1, empirically observe that semantic information in consecutive video frames is highly correlated. Additionally, previous methods [50, 82, 22, 30, 25, 45] highlight the gradual changes semantic features within deep neural networks. They attempt to reduce the computational costs by reusing feature maps from preceding frames for the target frame, facilitated by the estimated optical flow between the neighboring frames. However, these methods may still suffer from degraded performance due to inaccurate and noisy optical flow estimation, which only measures pixel-to- to-e-el- correspondence between adjacent frames. Recent advancements in Transformer-basee optical flow methods [58, 20, 63, 64] demonstrate the importance of incorporating global correspondence information for each pixel, resulting in more precise per-pixel optical flow estimation. Nevertheless, these methods still focus solely on pixel-level correspondence and do not account for segment-level information among video frames, as illustrated in Fig. 1, which may be insufficient for the task of VSS.
In this paper, we present an efficient mask propagation framework for VSS, namely MPVSS, as shown in Fig. 2. This framework relies on two key components: a strong query-based image segmentor for the key frame (upper part of Fig. 2), and a powerful query-based flow estimator for the non- key frames (lower part of Fig. 2). Specifically, we employ the state-of-the-ar query-based image segmentation model, i.e., Mask2Former [7] to process sparse key frames and generate accurate binary masks and class predictions. We then propose to estimate individual flow map corresponding to each segment-level mask prediction of the key frame. To achieve this, we leverage the learned queries from the key frame to aggregate segment-level motion information between adjacent frame pairs and capture specific movements along the temporal dimension. These queries are fed to a flow head to predict a set of segment- aware flow maps, followed by refinement step. The mask

---

predictions from key frames are subsequently warped to other non-key frames through the generated query-based flow maps, where each flow map is tailored the corresponding mask predicted by the associated query in the key frame. Finally, we derive semantic maps for the non-key frames by aggregating the warped mask predictions with the class probabilities predicted from the key frame. By warping the predicted segmentation masks from key frames, our model avoids processing each individual video frame using computationally intensive semantic segmentation models. This not only reduces temporal redundancy but also significantly lowers the computational costs involved.
In summary, our main contributions are in three folds:
We propose MPVSS, novel mask propagation framework for efficient VSS, which reduces com- putational costs and redundancy by propagating accurate mask predictions from key frames to non-key frames.
We devise novel query-based flow estimation module that leverages the learned queries from the key frame to model motion cues for each segment-level mask prediction, yielding a collection of accurate flow maps. These flow maps are utilized to propagate mask predictions from the key frame to other non-key frames.
Extensive experiments on standard benchmarks, VSPW [42] and Cityscapes [9], demonstrate that our MPVSS achieves SOTA accuracy and efficiency trade-offs.
2
Related Work
Image semantic segmentation. As fundamental task in computer vision, image semantic seg- mentation seeks to assign a semantic label to each pixel in an image. The pioneering work of Fully Convolutional Networks (FCNs) [40] first adopts fully convolutional networks to perform pixel-wise classification in an end-to-end manner and apply a classification loss to each output pixel, which naturally groups pixels in image into regions of different categories. Building upon the per-pixel classification formulation, various segmentation methods have been proposed in the past few years. Some works aim at learning representative features, which propose to use atrous convolutional layers to enlarge the receptive field [68, 4, 5, 46, 67]. aggregate contextual information from multi-scale scale feature maps via pyramid architecture [17, 36), encoder-decoder architecture [49, 6]. or utilize attention modules to capture the global dependencies [27, 76, 14, 79, 78, 61]. Another line of work fo- cuses introducing boundary information to improve prediction accuracy for details [10, 29, 70, 77]. More recent methods demonstrate the effectiveness of Transformer- based architectures for semantic segmentation. SegFormer [62], Segmentor [51], SETR [78], MaskFormer [8, 7] replace traditional convolutional backbones [18] with Vision Transformers [1 [11, 59, 39, 34J and/or implement the head with Transformers following the DETR-like framework [2]. Basically, we propose an efficient frame- work for video semantic segmentation that leverages the mask predictions generated by DETR-like image segmentors. Our franework focuses on propagating these mask predictions across video frames, enabling accurate and consistent segmentation results throughout the entire video sequence. By incorporating the strengths of DETR-like models in image segmentation, we achieve high-quality and efficient video semantic segmentation.
Video semantic segmentation. Video semantic segmentation has recently captured the attention of researchers due to its dynamic nature and practical significance in real-world scenarios. Compared with image semantic segmentation, most existing methods of VSS pay attention to exploiting temporal information, which can be divided into two groups. The first group of approaches exploits temporal relationships improve prediction accuracy and consistency. For example, some works utilize recurrent units [26, 43] or design an attention propagation module [19] to warp the features or results of several reference frames [81, 15], or aggregate neighbouring frames based on optical flow estimation [15, 37J. In particular, ETC [38] proposes temporal consistency regularization during training. [73] instead exploits perceptual consistency in videos, MRCFA [54] mines relationship of cross-frame affinities, STT [28] employs
transformer, CFFM- [53] disentangles static and dynamic context for video frames. The second line of methods infers to alleviate the huge temporal redundancy among video frames and thus reduce the computational cost. Observing the high similarity of the semantic representation of deep layers, ClockNet [50] designs a pipeline schedule to adaptively reuse feature maps from previous frames in certain network layers. DFF [82] utilizes optical flow to propagate feature maps from key frames to non-key frames, achieving remarkable computational efficiency, Accel [22] further proposes to execute large model on key frames and
3

---

Q gM
Mask2Former Segmentation Path
Class head
queries
Transformer
decoder
Mask head
mask predictions N Rrsioo
Key frame result Pe geshow
backbone
Keyframe 1
Pixel decader
Motion encoder
y
Pi 200oow
Motion decader
JFOF
Warping function
[P
query-based flow maps
Flow-baa based Warping Path
Figure 2: Overall architecture of the proposed MPVSS.
compact non-key frames, while DVN [66] proposes region- based execution scheme non-key frames, as well as an dynamic key-frame scheduling policy. LLVS [30] develops an efficient framework involving both adaptive key frame selection and selective feature propagation. Unlike methods that primarily rely on optical flow, which models dense pixel-to-pixel correspondence, we introduce novel query-base baared flow module that focuses segment-level flow fields between adjacent frames. By operating at the segment level, our method captures higher-level motion information that more accurate for VSS.
by optical flow. Optical flow estimation a fundamental module video analysis, which estimates a dense field of pixel displacement between adjacent frames. In deep learning era, FlowNet [12] first utilizes convolutional neural networks learn from labeled data. Most follow-up methods propose employ spatial pyramid networks [52], utilize recurrent layers [58] to apply iterative refinement for the predicted flow in a coarse-to so-fine manner [52, 47, 24, 55]. More recent methods pioneer Transformers capture the global dependencies of cost volume [58, 20], or extract representative feature maps for global matching [63], addressing the challenge of large displacement. As optical flow represents the pixel-to-pixel correspondence in adjacent frames, it is often utilized in a wide range of video analysis tasks, i.e., optical flow as motion information for modeling action motion in the tasks of action recognition [3, 56], temporal action detection [2], or improving the feature representation in video segmentation [74], object detection [8 I] optical flow estimation simultaneous task to model pixel correspondence for video interpolation [23, 48, 21], video impainting [71, 72, 31]. etc.
Motion estimation
3 Mask Propagation via Query-based Flow
In Sec. 3.1, we introduce the overall pipeline of the proposed MPVSS. In Sec. 3.2, we simply revisit Mask2Former that MPVSS built upon. Finally, in Sec. 3.3, we introduce our query-based flow estimation method, which yields a set of flow maps for efficiently propagating mask predictions from key frames other non-key frames.
3.1 Overview
The overall architecture is shown in Fig. 2. Let (I' 3XxWx33T frames, where H and W denote the height and width of frames in the video. Without loss of generality, we select key frames from the video clip at fixed intervals, e.g., e.g. every 5 frames, considering other frames within these intervals as non-key frames. For simplicity, we denote each key frame as I and non-key frame P', To reduce redundant computation, only run the heavy image segmentation model on the sampled key frames and propagate the predictions from key frames to non-key frames.
]t=1 be input video clip with
Specifically, for a key frame 1s we adopt Forkerormer [7] (upper part of Fig. 2) to generate N mask predictions R and class predictions. Then, use the flow estimation module (bottom part

---

of Fig. 2) to predict a set of flow maps Fi E N indicating the motion changes from P to IL, which will be discussed in Sec. 3.3. Subsequently, we apply the bilinear warping function on all locations to propagate per-segment mask predictions M to obtain M of non-key frames, ie., M W(M ). Consequently, we segment the non-key frame F by propagating the predicted masks from its preceding key frame 1s, avoiding the necessity of computationally intensive image segmentation models for every video frame.
During training, we apply the loss function in Mask2Former [7] on the warped mask predictions M' and perform bipartite matching with the ground truth masks for the current frame. The training approach is similar the process described in [82], where the loss gradients are back-propagated throughout the model to update the proposed flow module.
3.2 Fask2Former for Segmenting Key Frames
Mask2Former consists of three components: a backbone, a pixel decoder and transformer decoder, as depicted in the upper part of Fig. 2. The backbone extracts feature maps from the key frame Ist #x The pixel decoder builds multi-scale feature pyramid as well as generates high- resolution per-pixel embedding, following FPN [35] and Deformable DETR [80]. In transformer decoder, the target segments are represented as set of learned queries 2 E Rx where C' is the channel dimension. After gradually enriching their representations with stacked decoder blocks, 2 is fed into a class and a mask head to yield N class probabilities and N mask emheddings, respectively. After decoding the binary mask predictions M E Nx x with the class predictions and the per- pixel embeddings, we can finally derive the semantic maps for the key frame by aggregating N binary mask predictions with their corresponding predicted class probabilities via matrix multiplication. We refer readers to [8, 7] for more details.
3.3 Query-based Flow Estimation
To generate the segmentation masks for the non-key frames efficiently, we propose to estimate set of flow maps, each corresponding to a mask prediction of the key frame. We start with introducing the process of generating N query-based flow maps between the current non-key frame I' and its preceding key frame Is, where 0 < k < T. The overall flow module comprises three key elements: motion encoder to encode the motion feature pyramid between the pair of video frames [T*, [), ). motion information from the motion feature maps for each segment, and a flow head responsible for predicting N query-based flow maps Fii*k, where each flow map is associated with warping a segment (mask prediction) of the key frame to the current non-key frame.
motion decoder that leverages the learned queries Oh from the key frame to extract
Motion encoder. To obtain motion features between the paired frames (Iè D)), we concatenate the two frames along the channel dimension as the input of the motion encoder. The motion encoder, utilizing the same architecture as the flow encoder in FlowNet [12], is employed to extract motion features at each location in a downsampling manner. Following [12, 7J. we reverse the order of the original feature maps, generating a motion feature pyramid (b,, b2, b3, byj. each at resolutions 1/32, 1/16, 1/8 and 1/4 of the original video frame, respectively. Subsequently, the feature pyramid is fed to the model decoder for the decoding of flow maps from the lowest to the highest resolution.
Motion decoder. We then devise a motion decoder to aggregate motion information from the motion feature pyramid using N flow queries 2,F.
First, the motion decoder takes input the first three feature maps (b), b2, by) from the motion feature pyramid and the flow queries 2,. We initialize flow queries with the N learned queries 25 of the preceding key frame, i.e Ø Og. where O indicates the input flow queries to the motion decoder. Following [7], feature map in each layer is projected into C' channels in the channel dimension using a linear layer, which is then added with sinusoidal positional embedding and a learnable level embedding. This results in three scales of motion feature maps in total, i.e.,
(B), B,, B}}, ) as the input of the following layers.
Next, the motion decoder can divided into S stages, each indexed by 8, with each stage involving L blocks, each indexed by L. We adopt S = 3 and L. 3 for our model. Inspired by [7, 65], starting from and = 1, for each block each stage, we first concatenate B; and O together and then feed them into number of Transformer layers, each of which performs information propagation
5

---

between Bi and 2
Z = (05; Bi], 7 LN(MSA (Zi) Zi);
(1)
Q B = LN(FFNN (i) + 27),
where [:] denotes the concatenation operation. LN(-), MSA(-) and FFN(-) denote LayerNorm layer, Mutli- Self-Attention and stage, we learn flow queries and update motion feature maps via self-attention. The self attention mechanism aggregates motion information globally from both the motion feature maps and the flow query features simultaneously. Finally, the learned flow queries 2, and updated motion feature maps (Bi, B3} are fed to the flow head for flow prediction.
Neural Network, respectively. Therefore, in each
As each query of 2 encodes global information for each segment in the key frame, using 2ó as initial flow queries enables each flow query to extract motion information for each segment present in the key frame, which facilitates query-based motion modeling.
Flow head. The flow head generate the final flow maps Fiik E R NxxxXxX by combining query-based flow JQP and pixel-wise flow JPP.
To obtain the query-based flow FQF, an MLP with 2 hidden layers is employed to convert the learnt flow queries 2 to flow embeddings E E N each with C' channels. Inspired by the pixel decoder design Mask2Former, we project the motion feature map b4 from the motion encoder to EB E 2Cx # x % using another MLP with 2 hidden layers. Let E and Eg denote the first and last C' channels of Eg, and they are used to obtain the query-based flow predictions in terms of I and y direction, respectively. Then, for the th query, we get the flow map via a dot product between the n2th flow embedding Ex, and the projected motion map E i.e., F (x) Ey, Eg and FFF(2)
(n) Ex, EE, in terms of x and y direction, respectively. Then we can obtain the predicted
query-based flow FQP = = (R) (y)) E R2x # x tf for the nh query. The dot product of the flow embeddings and projected motion maps quantifies the similarity between the segment-level motion vector and the encoded motion information at each location on the feature maps, which reflects the movement for each query the spatial dimension.
Additionally, we use the updated motion feature maps (B1, BN, Bs} to predict pixel-wise flow JPF E 22 4 W in coarse-to-fin manner, following [12], which is used refine the query-based flow.
To obtain the final flow map Fiik E x for the th query, we apply a 2D convolutional layer to the concatenated FQF and PPF, and then upsample the flow map to the original resolution:
Fi Upsample (Conv2D (FFPFFPPT)
Remark. We use N flow queries, which are initialized with the learned per-segment queries 2 from the key frame, to capture the motion for each mask predicted on the key frame. In contrast to existing optical flow estimation methods, which model pixel-to-po pixpieel dense correspondence, we propose novel query-baser flow that incorporates segment-level motion information between adjacent frames, assisting more accurate mask propagation in VSS.
Experiments
4.1
Experimental Setup
Datasets. We evaluate our method on two benchmark datasets: VSPW [42] and Cityscapes [9]. VSPW the largest video semantic segmentation benchmark, consisting of 2,806 training clips (198,244 frames), 343 validation clips (24,502 frames), and 387 test clips (28,887 frames). Each video in VSPW has densely annotated frames at a rate of 15 frames per second, covering 124 semantic categories across both indoor and outdoor scenes. Additionally, we present results on the Cityscapes dataset, which provides annotations every 30 frames. This dataset serves as an additional evaluation to showcase the effectiveness of our proposed method.

---

Table Performance comparisons with state-of-the methods on VSPW dataset. We report mean IoU (mloU) and Weighted IoU (WIoU) for the performance evaluation and Video Consistency (VC) for the temporal consistency comparison. We measure FLOPs (G) for computational costs, averaging over 15-frame clip with resolution of 480 x 853.
(FPS) is measured on single
NVIDIA V100 GPU with 3 repeated runs.
Methods
Backbone
mloU
WloU*
mVCa f
mVC16 f
GFLOPs
Params(M))
FPST
DeepLab3+ [4]
R101
34.7
58.8
83.2
78.2
379.0
62.7
9.25
UpenNet [60]
R101
36.5
58.6
82.6
76.1
403.6
83.2
16.05
PSPNet [75]
R101
36.5
58.1
84.2
79.6
401.8
70.5
13.84
OCRNet [69]
R101
36.7
59.2
84.0
79.0
361.7
58.1
14.39
ETC [38]
OCRNet
37.5
59.1
84.1
79.1
361.7
NetWarp [15]
OCRNet
37.5
58.9
84.0
79.0
1207
TCB [42]
R101
37.8
59.5
87.9
84.0
1692
Segformer [62]
MIT-B2
43.9
63.7
86.0
81.2
100.8
24.8
16.16
Segformer
MIT-B5
48.9
65.1
87.8
83.7
185.0
82.1
9,48
CFFM-VSS [53]
MiT-B2
44.9
64.9
89.8
85.8
143.2
26.5
10.08
CFFM-VSS
MiT-B5
49.3
65.8
90.8
87.1
413.5
85.5
4.58
MRCFA [54]
MiT-R2
45.3
64.7
90.3
86.2
127.9
27.3
10.7
MRCFA
MiT-B5
49.9
66.0
90.9
87.4
373.0
84.5
5.02
Mask2Former [7]
R50
38.5
60.2
81.3
76.4
110.6
44.0
19.4
Mask2Former
R101
39.3
60.1
82.5
77.6
141.3
63.0
16.90
Mask2Former
Swin-T
41.2
62.6
84.5
80.0
114.4
47.4
17.13
Mask2Former
Swin-S
42.1
63.1
84.7
79.3
152.2
68.9
14.52
Mask2Former
Swin-B
54.1
70.3
86.6
82.9
223.5
107.1
11.45
Mask2Former
Swin-L
56.1
70.8
87.6
84.1
402.7
215.1
8.41
MPVSS
R50
37.5
59.0
84.1
77.2
38.9
84.1
33.93
MPVSS
R101
38.8
59.0
84.8
79.6
45.1
103.1
32.38
MPVSS
Swin-T
39.9
62.0
85.9
80.4
39.7
114.0
32.86
MPVSS
Swin-S
40.4
62.0
86.0
80.7
47.3
108.0
30.61
MPVSS
Swin-B
52.6
68.4
89.5
85.9
61.5
147.0
27.38
MPVSS
Swin-L
53.9
69.1
89.6
85.8
97.3
255.4
23.22
Implementation details. By default, experiments are trained with a batch size of 16 on NVIDIA GPUs. Note that pre-training is important for the DETR-like segmentors. For VSPW dataset, we pretrain Mask2Former on the image semantic segmentation task, serving as the per-frame baseline. For Cityscapes, adopt pretrained Fask2Former with ResNet [18] and Swin Transformer [39] backbones from [7]. For the motion encoder, we utilized the weights from FlowNet [12] encoder which is pre-trained on the synthetic Flying Chairs dataset; the motion decoder and flow head are randomly initialized. We freeze most parameters the pretrained Mask2Former, fine-tuning only its classification and mask head, as well as the proposed flow module. We apply loss functions in Mask2Former, including classification loss and a binary mask loss on the class embeddings and warped mask embeddings, respectively. All the models trained with the AdamW optimizer [41] for a maximum 90k iterations and the polynomial learning rate decay schedule [4] with an initial learning rate of 5e-5.
Compared methods. To validate the effectiveness of our proposed model, we include the following methods for study: Per-frame: Employing Mask2Former [7] to independently predict masks for each video frame. Optical flow: Warping mask predictions from key frames to non-key frames using optical flow estimated by lightweight optical flow model, i.e., FlowNet [12]. Query-random: Implementing the proposed flow estimation method with randomly initialized flow queries. Query- learned: Utilizing the proposed flow estimation method using the learned queries Q as initialization. Query-for-PF: Using pixel-wise flow JPF which incorporates segment-level information from the learned flow queries for warping predictions.
Evaluation metrics. Following previous works, we use mean Intersection over Union (mloU) single-scale inference, and Weighted loU (WIoU) evaluate the segmentation performance. We also compare models in terms of their computational complexity with FLOPs to evaluate the efficiency of these VSS models. We calculate FLOPs each single frame by averaging on video clip with 15 frames and fix the image resolution of 480 x 853 and 1024 x 2048 for VSPW and Cityscapes, respectively. For VSPW dataset, we adopt video consistency (VC) [42] to evaluate the visual smoothness of the predicted segmentation maps across the temporal dimension.

---

Table 2: Performance comparisons with the state-of-the-a VSS methods on Cityscapes. We report mloU for the semantic segmentation performance and FLOPs (G) for the computational cost comparison, which is averaged by a video clip of 15 images with a resolution of 1024 2048.
Frame-per-second (FPS) is measured on single NVIDIA V100 GPU with repeated runs.
Methods
Backbone
mloU
GFLOPs
Params(M)
FPS
FCN [40]
R101
76.6
2203.3
68.5
PSPNet [75]
R101
78.5
2048.9
67.9
DFF [82]
R101
68.7
DVSN [66]
R101
70.3
978.4
Accel [22]
R101
72.1
ETC [38]
71.1
434.1
SegFormer [62]
MiT-B1
78.5
13.8
SegFormer
MiT-B5
82.4
1460.4
84.7
CFFM-VSS [53]
MIT-BO
74.0
80.7
4.6
15.79
CFFM-VSS VSS
MiT-B1
75.1
158.7
15,4
11.71
MRCFA [54]
MiT-BO
72.8
77.5
4.2
16.55
MRCFA
MiT-B1
75.1
145
14.9
12.97
Mask2Former
R50
79.4
529.9
44.0
6.58
Mask2Former
R101
80.1
685.5
63.0
5.68
Mask2Former
Swin-T
82.1
543.6
47.4
5.41
Mask2Former
Swin-S
82.6
730.1
68.7
4.31
Mask2Former
Swin-B
83.3
1057.0
107.0
3.26
Mask2Former
Swin-L
83.3
1911.3
215.0
2.11
Fask2FFmmer-DFF
R101
77.1
457.4
101.7
7.14
DFFF
Swin-B
79.9
525.3
145.7
6.09
Marce
R101+R50
78.9
594.8
145.7
5.78
Mask2Forme-Accell
Swin-B+Swin-1
81.4
680.1
193.1
4,40
MPVSS
R50
78.4
173.2
84.1
13.43
MPVSS
R101
78.2
204.3
103.1
12.55
MPVSS
Swin-T
80.7
175.9
114.0
12.33
MPVSS
Swin-S
81.3
213.2
108.0
10.98
MPVSS
Swin-B
81.7
278.6
147.0
9.54
MPVSS
Swin-L
81.6
449.5
255.4
7.24
4.2 Comparison with State-of-the-art Methods
The comparisons on VSPW dataset are shown in Tab. 1. We also report the backbone used by each method and the computational complexity (GFOLPs), averaged with each frame of resolution 480x853. For our proposed models, we use 5 as the default key frame interval for comparison. We have the following analysis: 1) The proposed MPVSS achieves comparable accuracy with significantly reduced computational cost when compared to the strong baseline method, Mask2Former [7]. Specifically, compared Mask2Former, our MPVSS reduces the computational cost in terms of FLOPs by 71.7G, 96.2G, 74.7G. 104.9G, 162.0G, and 305.4G on R50, R101, Swin-T. T, Swin-S, Swin-B, and Swin-L backbones. respectively, while only experiencing 1.0%, 0.5%, 1.3%, 1.7%, 1.5%, and 2.2% degradation in the mloU score. These results demonstrate promising trade-off between accuracy and efficiency our framework. 2) Our models with different backbones perform on with the state-of-the-ard VSS methods the VSPW dataset with much fewer FLOPs. Specifically, our MPVSS model with the Swin-L backbone achieves an impressive mloU of 53.9%, surpassing CFFM-VSS with MiT-B5 backbone by 4.6%. Notably, our approach achieves this performance with only 24% of the computational cost in terms FLOPs. 3) When considering the comparable computational cost, model consistently outperforms other state-of-the-att methods. For instance, MPVSS with Swin-L backbone surpasses Segformer with MiT-B2 backbone by 10% mloU, under the FLOPs around 3.5G. These compelling results highlight the exceptional performance and efficiency of our MPVSS against the compared VSS approaches. In terms temporal consistency, proposed MPVSS achieves comparable mVC scores when compared approaches that explicitly exploit cross-frame temporal context [54] or incorporate temporal consistency constraints during training [38]. We provide an explanation of VC score and a discussion in the supplementary material.
video clip of 15 frames,
further evaluate our MPVSS on the Cityscapes dataset and achieve state-of-the-art results with relatively low computational complexity. Notably, the Cityscapes dataset only provides sparse anno-

---

Table 3: Ablation study on VSPW dataset.
(a) Fffeec of the proposed flow estimation method.
(b) Effect of cach component.
Backhone
Per frame
Copy
Optical Flow
Ours
Flow design
Params(M)
Swin-T
Swin-B
R50
38.5
35.7
36.1
37.5
Optical Flow
38.68
38.4
52.1
R101
39.3
37.1
38.2
38.8
Query Query-random
37.56
38.1
50.9
Swin-T
Swin-S
42.1
41.2
37.5
38.7
40.1
38.4
40.4
39,9
Query-learned
37.56
39.2
52.4
Swin B
54.1
49.3
52.1
52.9
Query-for-Pe
40.08
38.5
52.3
Swin-L
56.1
50.4
53.2
53,9
Query-based flow maps
40.08
39.9
$2.9
tations for one out of every 30 frames in each video. Therefore, for a fair comparison with previous efficient VSS methods, we report the results following the same evaluation process and compute their average FLOPs per-frame over all video frames. Specifically, compared Mask2Former, our MPVSS reduces the computational cost in terms of FLOPs by 0.36T, 0.48T, 0.37T, 0.52T, 0.78T and 1.5T on R50, R R101, Swin-T, Swin-S, Swin-B, and Swin-L backbones, respectively, with only 1.0%, 1 1.9%, 1.4%, 1.3%, 1.6% and 1.7% degradation in the mloU score. The strong accuracy- aecuracy-effciency trade-offs demonstrate that our proposed framework also works on sparse annotated video data.
4.3 Ablation Study
Effects of the query-based flow. We compare the performance between the proposed query-based flow and the traditional optical flow in Table 3a. Overall, approach demonstrates superior performance compared with the models relying solely on optical flow for mask propagation from key frames to non-key frames. To be specific, the utilization of the proposed query-based flow for mask warping consistently boosts the performance by 1.4%, 0.6%, 1.5%, 0.3%, 0.8% and 0.7% in terms of mloU R50, R101, Swin-T, Swin-S, Swin-B and Swin-L, respectively, which leads less degradation to the per- framer baseline compared to models using pixel-wise optical flow. Moreover, directly copying the mask predictions from the key frame leads to clear performance drop.
Effects of each component. In Table 3, we verify the effects of each component design in our query-based flow. We conduct experiments VSPW using Swin-T and Swin-B. The performance of Query-random on the two backbones is slightly inferior to using optical flow. This suggests that without per-segment query initialization, the query-based flow struggles to retrieve specific motion information from motion features for one-to-one mask warping from the key frame, resulting in suboptimal performance. Then, by using key-frame queries as the initialization for flow queries, we observe an improvement of 1.1% and 1.5% mloU scores for Query-learned on Swin-T and Swin-B respectively, compared Query-random. Furthermore, we develop variant design Query- for-PF, which introduces additional performance gain by 0.1% and 0.2% compared to optical flow respectively. These results underscore two findings: 1) learning query-based flow for each segment requires the learned queries from the key frame, for extracting meaningful motion features for each segment; 2) integrating segment-level information benefits the learning of pixel-wise flow. Therefore, we incorporate a refinement step using the enhanced pixel-wise flow for producing each query-based flow, which results in the best performance on both backbones. The outcomes highlight that warping mask predictions using our query-based flow consistently outperforms the methods using traditional optical flow on VSS.
Effects of the key frame interval. In Fig. 3a and Fig. 3b, we show the mloU scores and TFLOPs versus key frame interval for our MPVSS with different backbones. respectively. Overall, all the results achieve significant FLOPs reduction with decent accuracy drop, which smoothly trade in accuracy for computational resource and fit different application needs flexibly. In Fig. 3c, we present the performance of warping mask predictions using our query-based flow and traditional optical flow, as well as direct copying version, on Swin-T. We observe that, when the key frame interval increases from 2 to 10, the performance of the proposed gusry-based flow only slightly degraded by 1. % and 0.7% in terms of mloU and WloU score, while the scores are decreased by 2.63% and 2.96% using optical flow. These results indicates that the proposed query-baser flow is more robust to capture long-term temporal changing for videos. We provide comparisons on other backbones in the supplementary material.

---

Swir
5851
44
Qucyy noso2 Ton
26500
Capy Optical now
S
265-5
50
S
56311
TRLON
45
15
10 15
20
10
35
20
10 Key frame Interval
20
Key frame interval
Key frame Interval
(a) mloU vs. Key frame interval with different backbones.
(b) TFLOPs vs. Key frame inter- val with different backhones.
(c) mloU vs. Key frame interval of different warping method.
Figure Effects of the key frame interval. We compare the trade-off between accuracy and efficiency on variant backbones and warping strategies with respect to key frame interval. The number of FLOPPs (T) calculated based on a clip of 15 frames with resolution of 480 x 853.
4.4 Qualitative Results
In Fig. 4, present two examples from VSPW dataset visually compare the prediction quality of the per-frame baseline, the method using optical flow, and the proposed MPVSS. We observe that the proposed query-based flow exhibits superior performance in capturing the movement of small objects when compared to optical flow. Additionally, the proposed mask propagation framework demonstrates enhanced category consistency compared the per-frame baseline. For instance, the per-frame baseline exhibits category inconsistency as each frame is independently predicted. This highlights the advantage of our proposed upproach maintaining category coherence across consecutive frames.
(b)
(d)
Figure 4: Qualitative results. We present two examples from VSPW dataset. In each example, display series of consecutive frames from left to right. From top to bottom: (a) the ground truth; (b) the predictions of per-frame baseline, (c) the predictions of method using optical flow; (d) the predictions of the proposed MPVSS.
Conclusion and Future Work
In this paper, we have presented a simple yet effective mask propagation framework, dubbed MPVSS, for efficient VSS. Specifically, we have employed a strong query-based image segmentor to process key frames and generate accurate binary masks and class predictions. Then we have proposed to estimate specific flow maps for each segment-level mask prediction the key frame. Finally, the mask predictions from key frames were subsequently warped to other non-key frames via the proposed query-basee flow maps. Extensive experiments on VSPW and Cityscapes have demonstrated that our MPVSS achieves SOTA accuracy and efficiency trade-off. Future work could explore extending MPVSS to other video dense prediction tasks, e.g., video object and instance segmentation.
Limitations and broader impacts. Although our framework significantly reduces computational costs for processing videos, the new module leads to an increase in the number parameters in the models. Additionally, there might be need for fine-tuning the new module or optimizing hyperparameters to achieve the best performance. can lead to increased carbon emissions, indicating the potential negative societal impact.
10

---

References
[1] A. Bulat, B. Martinez, and G. Tzimiropoulos. Efficient attention-frr free video shift transformers. arXiv preprint arXiv:2208.11108, 2022.
[2] N. Carion, Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV, pages 213-229. Springer, 2020.
[3] J. Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. CVPR, pages 6299-63900 6308, 2017.
[4] L.-C. Chen, Papandreou, Kokkinos, Murphy, and L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI, 4344))834-8 2017.
[5] L.-C. Chen, G. Papandreou, Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: (.X6,0/70000587,, 2017.
[6] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV, pages 801-818, 2018.
[7] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov. and R. Girdhar. transformer for universal image segmentation. In CVPR, pages 1290-12999 1299, 2022.
-aekeed-ttention mask
[8] B. Cheng, A. Schwing, and A. Kirillov. Per-pixel elassification not all you need for semantic segmentation. NIPS, 144117644-7877 2021.
[9] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pages 3213 3223, 2016.
[10] H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, and G. Wang. Boundary-aware feature propagation for scene segmentation. In ICCV, pages 681 6819-6829, 2019.
[11] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, Uszkoreit, and N. Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[12] A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and Brox. Flownet: Learning optical flow with convolutional networks. In CVPR, pages 2758-2582202 27682776,.22 2015.
[13] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In ICCV, pages 6202-6211 6211, 2019.
[14] J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu. Dual attention network for scene segmentation. In CVPR, pages 3146-3154, 2019.
[15] R. Gadde, Jampani, and P. V. Gehler. Semantic video enns through representation warping. In ICCV, pages 4453-4462. 2017.
[16] M. Han, J. Zhang, Y. Wang, R. Yan, L. Yao, X. Chang, and Y. Qiao. Dual-ai: Dual-path actor interaction learning for group activity recognition. In CVPR. IEEE, 2022.
[17] K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. TPAMI, 194-11990-- 2015.
[18] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[19] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi. Temporally distributed networks for fast video semantic segmentation. In CVPR, pages 8818-8827, 2020.
[20] Huang, Shi, C. Zhang, Q. Wang, C. Cheung, Qin, J. Dai, and H. Li. Flowformer: transformer architecture for optical flow. In ECCV, pages 668-685. Springer, 2022.

---

[21] Z. Huang. T. Zhang, W. Heng, B. Shi, and S. Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV, pages 642. Springer, 2022.
[22] Jain, X. Wang, and E. Gonzalez. Accel: corrective fusion network for efficient semantic segmentation on video. In CVPR, pages 8866-8875, 2019.
[23] L. Kong, Jiang, Luo, W. Chu, X. Huang, Tai, C. Wang, and Yang. Iffmet: Intermediate feature refine network for efficient frame interpolation. In CVPR, pages 1969-1978, 2022.
[24] L. Kong, C. Shen, and J. Yang. Fastflownet: A lightweight network for fast optical flow estimation. In ICRA, pages 10310-10316. IEEE, 2021.
[25] S.-P. Lee, S.-C. Chen, and W.-H. Peng. Gsvnet: Guided apatiall--varyiie convolution for fast semantic segmentation on video. In ICME, pages 1-6. IEEE, 2021.
[26] P. Lei and S. Todorovic. Recurrent temporal deep field for semantic video labeling. In ECCV, pages 302-317. Springer, 2016.
[27] H. Li, P. Xiong, J. An, and L. Wang. Pyramid attention network for semantic segmentation. arXiv preprint 1rXhv:1805.101100 2018.
[28] Li, W. Wang, J. Chen, L. Niu, J. Si, C. Qian, and L. Zhang. Video semantic segmentation via sparse temporal transformer. In ACMMM, pages 59-68, 2021.
[29] X. Li, X. Li, L. Zhang, G. Cheng, J. Shi, Z. Lin, S. Tan, and Y. Tong. Improving semantic segmentation via decoupled body and edge supervision. In ECCV, pages 435-453 Springer, 2020.
[30] Y. Li, J. Shi, and D. Lin. Low-latency video semantic segmentation. CVPR, pages 5997-6005, 2018.
[31] Z. Li, C.-Z. Lu, J. Qin, C.-L. Guo, and M.-M. Cheng. Towards an end-to-end framework for flow-guides -ww-guided video inpainting. In CVPR, pages 17571,17571, 2022.
[32] C. Lin, C. Xu, D. Luo, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Fu. Learning salient boundary feature for anchor-free temporal action localization. CVPR, pages 3320-3329, 2021.
[33] Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding, In ICCV, pages 7083-7093, 2019.
[34] S. Lin, H. Xie, B. Wang, K. Yu, X. Chang, X. Liang. and G. Wang. Knowledge distillation via the target-aware transformer. In CVPR, pages 10915-10924, 2022.
[35] T.-Y. Lin, P. Dollár, R. Girshick. K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, pages 2117-2125, 2017.
[36] J. Liu, J. He, J. Zhang. J. S. Ren, and H. Li. Efficientfen: Holistically-gut guided decoding for semantic segmentation. In ECCV. pages 1-17. Springer, 2020.
[37] S. Liu, Wang. Qian, Yu, Bao, and Y. Sun. Surveillance video parsing with single frame supervision. In CVPR, pages 413 421, 2017.
[38] Y. Liu, C. Shen, C. Yu, and J. Wang. Efficient semantic video segmentation with per-frame inference. In ECCV, pages 352-368, 2020.
[39] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, pages 9992-10002, 2021.
[40] J. Long, F. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR, pages 3431-3440, 2015.
[41] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint axXvv:111.05101,, 2017. 2017.
12

---

[42] J. Miao, Y. Wei, Y. Wu, C. Liang, G. Li, and Y. Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In CVPR. pages 4133 533-4143, 2021.
[43] D. Nilsson and C. Sminchisescu. Semantic video segmentation by gated recurrent flow propaga- tion. In CVPR, pages 6819-6828, 2018.
[44] M. Paul, M. Danelljan, L. Van Gool, and R. Timofte. Local memory attention for fast video semantic segmentation. In IROS, pages 100--1009. IEEE, 2021.
[45] M. Paul, C. Mayer, L. V. Gool, and R. Timofte. Efficient video semantic segmentation with labels propagation and refinement. In WACV. pages 2873-2882, 2020.
[46] C. Peng. X. Zhang, G. Yu, G. Luo, and Sun. Large kernel matters-improve semantic segmentation by global convolutional network. In CVPR, pages 4353-4361, 2017.
[47] A. Ranjan and M. J. Black. Optical flow estimation using spatial pyramid network. In CVPR, 2017.
pages
[48] F. Reda, J. Kontkanen, E. Tabellion, D. Sun, C. Pantofaru, and B. Curless. Film: Frame interpolation for large motion. In ECCV, pages 250-266. Springer, 2022.
[49] O. Ronneberger, Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, pages 234-241 241. Springer, 2015.
[50] E. Shelhamer, K. Rakelly, J. Hoffman, and T. Darrell. Clockwork convnets for video semantic segmentation. In ECCV, pages 852-868, 2016.
[51] R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmen- tation. In ICCV, pages 7262-7272, 2021.
[52] D. Sun, X. Yang, M.-Y. Liu, and Kautz. Pwc-net: Cans for optical flow using pyramid, warping, and cost volume. In CVPR, pages 8934 8943, 2018.
[53] G. Sun, Y. Liu, H. Ding, T. Probst, and L. Van Gool. Coarse-to-fine feature mining for video semantic segmentation. In CVPR, pages 3126-3137, 206--317,,2222 2022.
[54] G. Sun, Y. Liu, H. Tang, A. Chhatkuli, L. Zhang, and I.. Van Gool. Mining relations among pross-frame affinities for video semantic segmentation. In ECCV, pages 522-539, 2022.
[55] S. Sun, Y. Chen, Y. Zhu, G. Guo, and G. Li. Skflow: Learning optical flow with super kernels. arXiv preprint arXiv:2205. 14623, 2022.
[56] S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In CVPR, pages 1390-1399, 2018.
[57] J. Tan, J. Tang, L. Wang, and G. Wu. Relaxed transformer decoders for direct action proposal generation. In ICCV, pages 25526-13535, 2021.
[58] 7. Teed and J. Deng. Raft: Recurrent pai-s field transforms for optical flow. In ECCV, pages 402-419. Springer, 2020.
[59] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data efffcceffic image transformers distillation through attention. ICMI., pages 10347-10357, 2021.
[60] Xiao, Y. Liu, B. Zhou, Y. Jiang, and Sun. Unified perceptual parsing for scene understanding. In ECCV, pages 418 434, 2018.
[61] E. Xie, W. Wang, W. Wang, P. Sun, H. Xu, D. Liang, and P. Luo. Segmenting transparent object in the wild with transformer. arXiv preprint Sili::2001.04660 2021.
[62] E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NIPS, 34: -2077 2021.
[63] H. Xu, J. Zhang, Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In CVPR, pages 1121-813,, 2022.
13

---

[64] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger. Unifying flow, stereo and depth estimation. arXiv preprint arXiv:2211.05783, 2022.
[65] Xu, S. De Mello, Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pages 18134-18144, 2022.
[66] Y.-S. Xu, T.-J. Fu, H.-K. Yang. and C.-Y. Lee. Dynamic video segmentation network. In CVPR, pages 6556-6565, 2018.
[67] M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang. Denseaspp for semantic segmentation in street scenes. In CVPR, pages 3684-3692, 2018.
[68] Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint 1rXiv:1511.07122, 2015.
[69] Y. Yuan, X. Chen, and J. Wang. Object-contextual representations for semantic segmentation. ECCV. Springer, 2020.
[70] Y. Yuan, J. Xie, X. Chen, and J. Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV, pages 489-506. Springer, 2020.
[71] K. Zhang, Fu, and D. Liu. Flow-guided transformer for video inpainting. In ECCV, pages 74-90. Springer, 2022.
[72] K. Zhang, J. Fu, and D. Liu. Inertia guiiid flow completion and style fusion for video inpainting. In CVPR. pages 5982-5991, 2022.
[73] Y. Zhang, S. Borse, H. Cai, Wang, N. Bi, X. Jiang, and E. Porikli. Perceptual consistency in video segmentation. In WACV, pages 2564-2573, 2022.
[74] Y. Zhang, A. Robinson, M. Magnusson, and M. Felsberg. Flow-guided semi-supervised video object segmentation. arXiv preprint arXiv:2301.10492, 2023.
[75] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In CVPR, pages 2881-2890, 2017.
[76] H. Zhao, Zhang, Liu, J. Shi, C. C. Loy, D. Lin, and Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV, pages 267-283, 2018.
[77] M. Zhen, J. Wang, L. Zhou, S. Li, T. Shen, J. Shang, T. Fang, and L. Quan, Joint semantic segmentation and boundary detection using iterative pyramid contexts. CVPR, pages 13666- 13675, 2020.
[78] S. Zheneng J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. Rethinking semantic segmentation from a sequence-to-sequens perspective with transformers. In CVPR, pages 6881 6881-6890, 2021.
[79] Z. Zhong, Z. Q. Lin, Bidart, X. Hu, I. B. Daya, Z. Li, W.-S Zheng, Li, and A. Wong. Squeeze-and-attente networks for semantic segmentation. In CVPR, pages 13065-13074, 2020.
[80] X. Zhu, W. Su, L. Lu, B. Li, Wang, and J. Dai. Deformable transformers for end-to-end object detection. In /CLR, pages 3-7, 2021.
[81] X. Zhu, Y. Wang. J. Dai, L. Yuan, and Y. Wei. Flow-guided feature aggregation for video object detection. ICCV, pages 408-417, 2017.
[82] X. Zhu, Y. Xiong. J. Dai, L. Yuan, and Y. Wei. Deep feature flow for video recognition. In CVPR, pages 2349-2358, 2017.
14


Reviews:
[{"review_id": "KqeckvmrJ9", "reviewer": "Reviewer_oxr6", "summary": "This paper investigates an important and fundamental task: video semantic segmentation (VSS). While image semantic segmentation has received significant research attention, VSS has been relatively overlooked due to limited datasets and computational resources. This paper makes a valuable contribution to the field by proposing a method that combines a segmentation network with a flow network. Unlike traditional flow networks, the proposed approach employs query-based flow maps that correspond to each segment. Experimental results demonstrate the effectiveness of the method on two widely-used datasets, namely vspw and cityscape.", "questions": "Please see the weakness", "limitations": "Please see the weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important but relatively underexplored task.\n2. The combination of segmentation and optical flow in a single model is a novel approach, particularly considering the use of query-based flow maps.\n3. The proposed method achieves commendable performance on standard datasets such as vspw and cityscape.", "weaknesses": "1. The paper lacks specific details regarding the training of the flow modules. For example, what is the loss function used to train query-based flow? How are the flow networks (encoder/decoder) initilized? Do you use trained weights for these modules? Also, more visual examples about the query-based flow maps should be given.\n2. The rationale behind utilizing both pixel-wise flow (F^{PF}) and query-based flow (F^{QF}) to generate query-based flow maps is not well-justified. In my opinion, pixel-wise flow is the flow map for every pixel while the query-based flow is only for specific segments. Equation 4, which concatenates both flows, may make the query-based flow not focus on the segments, but on all the pixels.\n3. It would be helpful to include information on the number of parameters for all the methods listed in Table 1. Additionally, it would be beneficial to specify the frames per second (fps) achieved by the mask2former and mpvss methods using a single GPU for inference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We address all the questions below: \n\n**Q1.** Training details for the proposed flow module, e.g., loss function used to train query-based flow, initialization of the flow network (encoder/decoder), utilization of pretrained weights. More visual examples of query-based flow maps should be provided. \n\n**A1.** We have mentioned the training details for the proposed MPVSS which involves the training process for the proposed flow module in Lines 168-171 and Lines 263-264. (1) Loss function: During the training phase, the classification loss and binary mask loss are employed on the class embeddings and warped mask embeddings, respectively, after performing bipartite matching between queries and the ground truth masks, as demonstrated in Mask2Former. Subsequently, the loss gradients are propagated backward across the model to update the proposed flow module. (2) Initialization of the flow network: for the motion encoder, we utilized the weights from FlowNet encoder which is pre-trained on the synthetic Flying Chairs dataset; the motion decoder and flow head are randomly initialized. We will make the training details clearer in the revised version. \n\nWe provide more visualization of query-based flow maps in **Figure A** in the general response PDF. \n\n\n**Q2.** Justification of the rationale behind utilizing both pixel-wise flow and query-based flow to generate flow maps. \n\n**A2.** First, as mentioned in lines 237-239, we use the pixel-wise flow to refine the query-based flow and generate the final flow maps for each segment. Second, to better understand the rationale of utilizing pixel-wise flow and query-based flow, we provide visualization for pixel-wise flow and query-based flow separately in **Figure A**. The query-based flow and pixel-wise flow are compensated. The query-based flow captures the overall motion for the specific segment, while the pixel-wise flow adds more detailed movement at the fine-grained level. By fusing the two kinds of flow, the final flow maps are more stable for mask propagation. \n\n**Q3.** Information on the number of parameters and FPS. \n\n**A3.** We add number of parameters and FPS for each method in Table A and Table B in general response. Compared with other methods, the proposed MPVSS achieves both higher FPS and mIoU scores with different backbones on both datasets, which demonstrates the favorable accuracy and efficiency trade-off. We will include Table A and Table B with number of parameters and FPS in the revised version. \n"}, {"review_id": "uBSh5f5SH9", "reviewer": "Reviewer_8jSp", "summary": "The paper is to develop an approach to video semantic segmentation via propagating the segmented mask in key frames to non-key frames. Experiments were conducted on several databases with various comparisons. ", "questions": "See my comments in the Weakness part. ", "limitations": "limitation is talked in the paper. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Focusing on improving the computational efficiency for video semantic segmentation;\n\nThe paper is well-written.", "weaknesses": "It seems that the work lacks of novelty. In my understanding, the key idea is to compute the mask in key frames, and then propagate the segmentation result to non-keyframes, in order to reduce the computation cost. However, in many previous video analysis works, doing many computation in key frames, and then applying the result to non-keyframes, is quite natural. \n\nThe segmentation in key frames is performed by using an existing method, Mask2Former [7].\n\nThe flow estimation between frames is quite normal: using FlowNet [11] for motion encoding, and transformer based approach [7] for decoding. Not presenting a new method. \n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for taking the time to review our paper and we address your questions as follows.\n\n**Q1.** Questions about the novelty of the proposed method. \n\n**A1.** As recognized by Reviewers oxr6, 41Ay, and cyWY, the primary innovation of this research lies in a **novel and efficient mask propagation framework**. This framework encompasses a *potent query-based image segmentor* and a *highly capable query-based flow estimator*, where the whole framework allows for the accurate propagation of mask predictions from key frames to non-key frames. \n\nAs discussed in Lines 43-47 and Lines 57-64, applying a strong query-based image segmentor to the task of VSS is highly computational expensive. To reduce the computational cost, current approaches rely on optical flow to propagate features from the key frame to other non-key frames, but they still suffer from performance degradation due to the limitations of pixel-to-pixel optical flow estimation. In this context, the proposed mask propagation framework and the query-based flow module design is non-trivial. \n\nMoreover, the motivation and design of the proposed flow module differ from established models like FlowNet or Transformer-based optical flow models. As discussed in Lines 57-74, traditional optical flow estimation focuses on pixel-level correspondences between adjacent frames. In contrast, our flow module employs learned queries from the key frame to aggregate motion information **at the segment level** for each pair of adjacent frames. This approach captures **segment-specific movement** over time, resulting in enhanced accuracy and consistency for mask propagation in the task of VSS. "}, {"review_id": "f8B6yQIhGj", "reviewer": "Reviewer_41Ay", "summary": "This paper presents an approach for video semantic segmentation (VSS) by focusing on the aspect of efficiency. The authors propose a novel mask propagation framework that is built upon a computationally intensive query-based image segmentor called Mask2Former[1]. Instead of processing every frame, the framework leverages the image segmentor to process only the key frames. To generate masks for non-key frames, a novel query-based flow map estimation module is introduced, which predicts optical flows to warp the masks from key frames. The experimental results demonstrate that the proposed framework achieves competitive performance, with only a minimal drop of approximately 1% to 2% when compared to the baseline [1]. Notably, the proposed method achieves these results with significantly reduced Floating Point Operations (FLOPs).\n\n[1] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022.", "questions": "1. It would be inappropriate to claim that the proposed method (MPVSS) \"achieves SOTA performance\". From Table 1 and Table 2, it becomes evident that MPVSS yields slightly lower mIoU scores compared to its baseline counterpart (Mask2Former). Furthermore, Mask2Former already attains SOTA performance when compared to other methods presented in the table, so the primary contribution of MPVSS lies in its efficiency. Please kindly rectify these statements to accurately reflect the contributions of MPVSS.\n\n2. It would be insightful to delve further into the reasons why the learned query from the mask generation branch (Q^k_O) enhances flow estimation. While the paper briefly touches upon this topic in a single sentence (line 220 - line 222), it would be great to include visualizations or attention maps that demonstrate the gain of the learned query on flow estimation.\n\n3. In line 240 - line 242, two flows (query-based flow and pixel-based flow) are stacked to generate the final flow predictions. To gain a better understanding of the effect of the query-based flow, it would be helpful to visualize the query-based flow and pixel-based flow separately. Additionally, it would be beneficial to clarify whether the first row (i.e., Optical flow) in Table 3 (b) refers to the \"pixel-based flow\".", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed method demonstrates competitive performance on two widely recognized benchmarks, showcasing only a slight decline in the mIoU score, ranging from 1% to 2%. These results are achieved while significantly reducing computation costs (FLOPs).\n\n2. The novel query-based flow estimation module introduced in this paper surpasses traditional pixel-wise optical estimation methods by producing better flow maps. This advancement holds great promise for the field of transformer-based flow estimation.", "weaknesses": "Please see my comments in below Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the thorough review and constructive questions. \n\n**Q1**. It would be inappropriate to claim that the proposed method (MPVSS) \"achieves SOTA performance\". \n\n**A1**. Thanks for your advice. We will demonstrate our contribution as \"achieve SOTA accuracy and efficiency trade-offs\". \n\n**Q2.** Why the learned query from the mask generation branch ($Q^k_O$) enhances flow estimation. \n\n**A2.** (1) Previous studies in DETR-like framework explore the initialization of decoder query and demonstrate that providing good prior of decoder query leads to faster convergence and better performance. (2) Additionally, as mentioned in Lines 220-222, we utilize the learned queries to enable each flow query to extract motion information for each segment appeared in the key frame. (3) We offer a visual comparison between utilizing the learned queries $Q^k_O$ through the flow maps and the attention maps derived from the motion decoder. The visualization results provided in **Figure A** in the general response PDF. As shown in row (f)(g) and row (c)(h) in **Figure A**, with $Q^k_O$ initialization, the associated attention maps focus on particular regions of the motion map, leading to improved flow maps for the corresponding segments. \n\n**Q3.** Visualization of the query-based flow and pixel-wise flow. \n\nWe provide the visualization in **Figure A** in the general response PDF. The query-based flow and pixel-wise flow complement each other. The query-based flow captures the overall motion for the specific segment, while the pixel-wise flow adds more detailed movement at a fine-grained level, especially for the edge of the segment. This combination enhances the stability of the final flow maps for mask propagation. \n\n**Q4.** Whether the first row (i.e., Optical flow) in Table 3 (b) refers to the \"pixel-based flow\". \n\n**A4.** Yes, \"Optical flow\" in the first row of Table 3(b) refers to the method warping mask predictions using pixel-based flow estimated by an optical flow model. "}, {"review_id": "0E470Q0XiU", "reviewer": "Reviewer_LSXE", "summary": "An efficient mask propagation framework for VSS, called MPVSS, is proposed in this paper.\n\nMPVSS adopts a strong query-based image segmentation based on sparse key frames and warp prediction to non-key frames by generating a segment-aware flow from a newly designed flow estimation module.\n\nWith the proposed query-based flow, MPVSS achieves the performance of SOTA with a better efficiency.", "questions": "Please refer to the Weaknesses section.", "limitations": "The authors adequately addressed limitation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. MPSVSS captures segment-level information between the video frames, which provides a better propagation result than the pixel-level correspondence information used in the previous work.\n\n2. Compared with vanilla optical flow, query-based flow provides smooth and clear compensation for non-keyframes.\n\n3. The proposed method achieves SOTA performance with significantly reduced computational cost through extensive experiments on standard benchmarks.\n\n4. MPVSS delivers a consistently higher level of video consistency than its base network.", "weaknesses": "1. typo: line 10: wrapped -> warped.\n\n2. The proposed mask propagation framework is identical to that of DFF [1], Accel [2], and CoVOS [3], while the proposed query-based flow requires further experimentation to demonstrate its effectiveness. More specifically:\n- In Table 2, it is difficult to say whether the query-based flow is better than Accel [2] or DFF [1], because the MPVSS uses Mask2Former for key-frames segmentation. In order to prove the advantage of propagation using query-based flow over the propagation module in Accel and DFF, the keyframe segmentation network should be unified.\n- In Table 3a, the query-based flow does bring improvement, but it also introduces additional network parameters. It's hard to say whether the improvement is due to the increase in parameters. \n3. Since propagation relies only on the previous keyframe, I wonder how occlusion will have a negative effect on the propagation results. There is literature using bi-directional motion vector for mask propagation [3] and shows that occlusion has a strong effect on the propagation results, especially causing ghosting effect, however this point is not discussed in the paper.\n4. A measurement on the FPS should be provided.\n\n[1] Zhu, Xizhou, et al. \"Deep feature flow for video recognition.\" CVPR. 2017.\n\n[2] Jain, Samvit, Xin Wang, and Joseph E. Gonzalez. \"Accel: A corrective fusion network for efficient semantic segmentation on video.\" CVPR. 2019.\n\n[3] Xu, Kai, and Angela Yao. \"Accelerating video object segmentation with compressed video.\" CVPR. 2022.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank you for your valuable feedback and address your questions as follows. \n\n**Q1.** Fair comparison between DFF and Accel by unifying the key frame segmentation network. \n\n**A1.** Thanks for your advice. We integrate DFF[78] and Accel[20] with Mask2Former to conduct fair comparisons on Cityscapes dataset. Compared with Mask2Former-DFF, we achieve higher mIoU scores and lower GFLOPs. And the proposed MPVSS surpasses Mask2Former-Accel on Swin-B by 0.3% with only 1/3 GFLOPs. \n\n\nTable C. Performance comparisons between DFF and Accel. \n\n|      Methods      |   Backbone    | mIoU | GFLOPs | #Params (M) |  FPS  |\n| :---------------: | :-----------: | :--: | :----: | :---------: | :---: |\n|  Mask2Former-DFF  |     R101      | 77.1 | 457.4  |    101.7    | 7.14  |\n|  Mask2Former-DFF  |    Swin-B     | 79.9 | 525.3  |    145.7    | 6.09  |\n| Mask2Former-Accel |   R101+R50    | 78.9 | 594.8  |    145.7    | 5.78  |\n| Mask2Former-Accel | Swin-B+Swin-T | 81.4 | 680.1  |    193.1    | 4.40  |\n|       MPVSS       |     R101      | 78.2 | 204.3  |    103.1    | 12.55 |\n|       MPVSS       |    Swin-B     | 81.7 | 278.6  |    147.0    | 9.54  |\n\n\n**Q2.** It's hard to say whether the improvement is due to the increase in parameters. \n\n**A2.** Compared with methods that warp masks by optical flow, our *Query-based* flow introduces additional ~1.4M parameters, which can be considered negligible. Additionally, in Table D, we measure the number of parameters for the flow module listed in Table 3(b) of the main paper. Using *Query-learned flow* already achieves better performance than optical flow with a lower number of parameters, which also proves the improvement mainly comes from the effectiveness of the module design. \n\nTable D. Performance comparisons for different flow designs. We report the number of parameters of different flow modules, and mIoU scores. \n\n|   Flow design    | #Params (M) | Swin-T | Swin-B |\n| :--------------: | :---------: | :----: | :----: |\n|   Optical flow   |    38.68    |  38.4  |  52.1  |\n|   Query-random   |    37.56    |  38.1  |  50.9  |\n|  Query-learned   |    37.56    |  39.2  |  52.4  |\n|   Query-for-OF   |    40.08    |  38.5  |  52.3  |\n| Query-based flow |    40.08    |  39.9  |  52.9  |\n\n\n\n**Q3.**  I wonder how occlusion will have a negative effect on the propagation results, especially the ghosting effect on VSS. \n\n**A3.** Thanks for your valuable feedback. Indeed, we have observed instances of failure due to occlusion challenges when applying our proposed mask propagation framework to VSS datasets. For example, scenarios in which a car moves behind a tree can lead to inaccuracies in mask propagation, primarily due to the reliance on mask predictions from preceding key frames.\n\nNevertheless, the occurrence of ghosting effects seems to be less pronounced when our network is applied to VSS datasets in comparison to VOS. One potential explanation for this could be the smoother transitions between scenes that are commonly observed in VSS datasets. In contrast, the ghosting effects encountered in VOS datasets often stem from delayed responses to rapidly moving objects. Additionally, our query-based flow, which integrates segment-level information, might offer more stable and robust mask propagation compared to using uni-directional optical flow. \n\nMoreover, we acknowledge that the integration of bi-directional motion vectors encoded within compressed videos in method [A] currently exceeds the scope of our work. Implementing bi-directional mask propagation appears promising, even though it might introduce additional model parameters and computational costs. This is deserving of further exploration, and we regard it as a potential area for future research. We will add a comprehensive discussion of the occlusion challenges in the revised version of our work. \n\n\n**Q4.** Information on FPS should be provided. \n\n**A4.** We measure the FPS of different methods on a single NVIDIA V100 GPU. The results are provided in Table A and Table B in the general response. We will update the corresponding tables in the revised version. \n\nWe will fix the typos in the revised version. \n\n**Reference:** \n\n[A] Xu, Kai, and Angela Yao. \"Accelerating video object segmentation with compressed video.\" CVPR. 2022.\n\n"}, {"review_id": "blCO8NutFa", "reviewer": "Reviewer_cyWY", "summary": "This paper presents a mask propagation method, MPVSS, for video semantic segmentation (VSS). MPVSS uses Mask2Former to obtain mask predictions and queries from the key frame. Then, a motion encoder extracts pixel-wise motion from the key frame and its adjacent frame. The queries from the key frame are used to decode query-wise motion features. A flow head generates query-based flow maps, and binary mask predictions for the adjacent frame are generated by applying these flow maps to the binary mask predictions of the key frame. Semantic segmentations of non-key frames are produced by matrix multiplication between the key frame's query classification output and the binary mask prediction for the adjacent frame. A detailed ablation study validates each component. Comparisons between MPVSS and state-of-the-art methods on VSPW and Cityscapes demonstrate good performance and efficiency of MPVSS.", "questions": "Please see Weaknesses", "limitations": "The paper adequately addresses limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper is well-written and easy to follow.\nMotivation leveraging motion estimation to reduce the computation of VSS is sound. \nThe proposed method of propagating the key frame’s query to the motion features is interesting and novel. \nThe experiments are well-designed, and the results are convincing. \n\n", "weaknesses": "The key frames are selected at fixed intervals. As a result, the method may not fully address the mentioned redundancy problem.\n\nEfficiency of the proposed method should be discussed in more detail. For instance, MPVSS uses FlowNet to extract motion features between two frames instead of relying on a backbone network to extract frame-wise features.\n\nFig3 (b), y axis label, “TLOPs” seems a typo.\n\nThere is not much information about training time, and any particular training strategies used in the experiments.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThanks for your constructive comments and we address your questions as follows.\n\n**Q1.** The method may not fully address the mentioned redundancy problem as the key frames are selected at fixed intervals. \n\n**A1.** First, we acknowledge that employing fixed key-frame intervals only results in a partial reduction of redundancy, which is a limitation of the current mask propagation framework. Secondly, a more effective approach would involve utilizing dynamic key-frame selection. However, this may introduce additional model parameters and potentially elevate computational costs due to the decision-making process for key-frame selection. In the future, we will consider exploring the implementation of dynamic key-frame selection to address the temporal redundancy challenge more comprehensively. \n\n**Q2.** Efficiency of the proposed method should be discussed in more detail. \n\n**A2.** We provide detailed computational costs for a single key frame and a non-key frame of the proposed MPVSS in Table E. As mentioned in Lines 43-44, the main computational demands lie in the computation of a high-resolution pixel-embedding. Utilizing the strong semantic correlation in consecutive video frames, we propose to propagate accurately predicted masks from a key-frame to its subsequent non-key frames. It only takes the cost of the flow module and mask propagation process, leading to the accuracy-efficiency trade-off. \n\nTable E. Computational cost in terms of FLOPs (G) for a single key frame and a non-key frame that uses different backbones. We use input resolutions of 480x853 and 1024x2048 for VSPW and Cityscapes, respectively. \n\n| Dataset    |               | R50   | R101  | Swin-T | Swin-S | Swin-B | Swin-L |\n| ---------- | ------------- | ----- | ----- | ------ | ------ | ------ | ------ |\n| VSPW       | Key-frame     | 110.6 | 141.3 | 114.4  | 152.2  | 223.5  | 402.7  |\n|  | Non-key frame | 21.0 |  21.0     |   21.0     |  21.0      |21.0|21.0|\n| Cityscapes | Key-frame     | 529.9 | 685.5 | 543.6  | 730.1  | 1057.0 | 1911.3 |\n|  | Non-key frame | 84.0 |84.0|84.0|84.0|84.0|84.0|\n\n**Q3.** Training time and training strategies. \n\n**A3.** We list the training hours in Table F. The training strategies for the proposed MPVSS have been outlined in Lines 168-171 and Lines 263-264. During training, we sample a pair of frames, i.e., a key frame and a non-key frame. We run the segmentation network on the key frame, and a set of query-based flow maps are estimated by the proposed flow module, which are then used to obtain the warped mask embedding for the non-key frame. We apply loss functions in Mask2Former, including a classification loss and a binary mask loss on the class embeddings and warped mask embeddings, respectively. Bipartite matching is performed between the warped mask embeddings and the ground truth masks. Subsequently, the loss gradients are propagated backward across the model to update the proposed flow module. \n\nTable F. Training time (Hours) on VSPW dataset. Experiments are conducted with a batch size of 16 on 8 NVIDIA V100 GPUs.  \n\n| Backbone    | R50  | R101 | Swin-T | Swin-S | Swin-B | Swin-L |\n| ----------- | ---- | ---- | ------ | ------ | ------ | ------ |\n| Mask2Former | 11.4 | 12.3 | 10.3   | 12.0   | 12.3   | 15.0   |\n| MPVSS       | 9.3  | 11.5 | 10.2   | 11.1   | 12.2   | 13.4   |\n\n"}]

Author rebuttal:
We sincerely thank all reviewers for their valuable comments. 

### Novelty

Most reviewers recognize the novelty of our method.

*"The combination of segmentation and optical flow in a single model is a novel approach, particularly considering the use of query-based flow maps."* (Reviewer oxr6)

*"The novel query-based flow estimation module introduced in this paper surpasses traditional pixel-wise optical estimation methods..."* (Reviewer 41Ay)

*" The proposed method of propagating the key frame’s query to the motion features is interesting and novel. "* (Reviewer cyWY)

### Promising results

All reviewers agree that our method provides a competitive accuracy and efficiency trade-off. 

*"The proposed method achieves commendable performance on standard datasets such as vspw and cityscape."* (Reviewer oxr6)

*"The proposed method demonstrates competitive performance on two widely recognized benchmarks... These results are achieved while significantly reducing computation costs (FLOPs)"* (Reviewer 41Ay)

*"The proposed method achieves SOTA performance with significantly reduced computational cost through extensive experiments on standard benchmarks."* (Reviewer LSXE)

### Accuracy and efficiency trade-off 

In Table A and Table B, we add the number of parameters for methods listed in Table 1 and Table 2 of the main paper. FPS is measured on a single NVIDIA V100 GPU with 3 repeated runs. Compared with other methods, the proposed MPVSS achieves favorable accuracy and efficiency trade-off. We will include Table A and Table B with the number of parameters and FPS in the revised version. 

Table A. Performance comparisons with state-of-the-art methods on VSPW dataset. 

|    Methods     | Backbone | mIoU | WIoU | VC$_8$ | VC$_{16}$ | GFLOPs | #Params |  FPS  |
| :------------: | :------: | ---- | ---- | ---- | ----- | ----- | :-----: | :---: |
| Deeplabv3+[4]  |   R101   | 34.7     | 58.8     | 83.2     | 78.2      |  379.0     |  62.7   | 9.25  |
|  UperNet[56]   |   R101   | 36.5     | 58.6     |  82.6    | 76.1      |  403.6     |  83.2   | 16.05 |
|   PSPNet[71]   |   R101   |  36.5    | 58.1     |  84.2    |  79.6     | 401.8      |  70.5   | 13.84 |
|   OCRNet[65]   |   R101   | 36.7     | 59.2     | 84.0     | 79.0      |  361.7    |  58.1   | 14.39 |
| SegFormer[58]  |  MiT-B2  | 43.9     |  63.7    | 86.0     | 81.2      |  100.8     |  24.8   | 16.16 |
|   SegFormer    |  MiT-B5  | 48.9     | 65.1     |  87.8    | 83.7      | 185.0      |  82.1   | 9.48  |
|  CFFM-VSS[50]  |  MiT-B2  |  44.9    | 64.9     |  89.8    |  85.8     | 143.2      |  26.5   | 10.08 |
|    CFFM-VSS    |  MiT-B5  | 49.3     | 65.8     |  90.8    | 87.1      |  413.5     |  85.5   | 4.58  |
| MRCFA[51]  |  MiT-B2  | 45.3     | 64.7     | 90.3     |  86.2     | 127.9      |  27.3   | 10.7  |
|  MRCFA    |  MiT-B5  | 49.9     | 66.0     |  90.9    |  87.4     |  373.0     |  84.5   | 5.02  |
| Mask2Former[7] |   R50    | 38.5     |  60.2    |  81.3    |  76.4     | 110.6      |  44.0   | 19.44 |
|                |   R101   |  39.3    | 60.2     | 81.3     |  76.4     | 141.3      |  63.0   | 16.90 |
|                |  Swin-T  | 41.2     |  60.1    | 82.5     |  77.6     |  114.4     |  47.4   | 17.13 |
|                |  Swin-S  |  42.1    | 62.6     | 84.5     |  80.0     |  152.2     |  68.9   | 14.52 |
|                |  Swin-B  |  54.1    | 63.1      | 84.7     |  79.3     |  223.5     |  107.1  | 11.45 |
|                |  Swin-L  | 56.1     | 70.3     | 86.6     |  82.9     |  402.7     |  215.1  | 8.41  |
|     **MPVSS**     |   R50    |  37.5    | 59.0  |  84.1  |  77.2     |  38.9     |  84.1   | 33.93 |
|                |   R101   | 38.8     |  59.0    |  84.8    | 79.6      |  45.1     |  103.1  | 32.38 |
|                |  Swin-T  |  39.9    |  62.0    |  85.9    |  80.4     | 39.7     |  114.0  | 32.86 |
|                |  Swin-S  |  40.4    |  62.0    |  86.0    |  80.7     |  47.3     |  108.0  | 30.61 |
|                |  Swin-B  |  52.6    |  68.4    |  89.5    |  85.9     |  61.5     |  147.0  | 27.38 |
|                |  Swin-L  |  53.9    |  69.1    | 89.6     |  85.8     |  97.3     |  255.4  | 23.22 |

Table B. Performance comparisons with the VSS methods on Cityscapes.  

| Methods     | Backbone | mIoU | GFLOPs | #Params (M) | FPS   |
| :---------: | :------: | :--: | :----: | :---------: | :---: |
| FCN[37]         | R101     |  76.6    |  2203.3      | 68.5        | 2.83  |
| PSPNet[71]      | R101     | 78.5     |  2048.9      | 67.9        | 2.88  |
| SegFormer[58]   | MiT-B1   | 78.5   |  243.7   | 13.8        | 20.7  |
| SegFormer   | MiT-B5   |  82.4|  1460.4   | 84.7        | 7.20  |
| CFFM-VSS[50]   | MiT-B0   |   74.0   |  80.7      | 4.6         | 15.79 |
| CFFM-VSS    | MiT-B1   |  75.1    |   158.7     | 15.4        | 11.71 |
| MRCFA[51]   | MiT-B0   |   72.8   |  77.5     | 4.2         | 16.55 |
| MRCFA    | MiT-B1   |  75.1    |  145     | 14.9        | 12.97 |
| Mask2Former[7] | R50      | 79.4     |  529.9      | 44.0        | 6.58  |
|             | R101     |  80.1    |   685.5     | 63.0        | 5.68  |
|             | Swin-T   |  82.1    |   543.6     | 47.4        | 5.41  |
|             | Swin-S   |  82.6    |   730.1     | 68.7        | 4.31  |
|             | Swin-B   |  83.3    |   1057.0     | 107.0       | 3.26  |
|             | Swin-L   |  83.3    |   1911.3     | 215.0       | 2.11  |
| **MPVSS**   | R50      | 78.4     |   173.2     | 84.1        | 13.43 |
|             | R101     |  78.2    |  204.3      | 103.1       | 12.55 |
|             | Swin-T   |  80.7    |  175.9      | 114.0       | 12.33 |
|             | Swin-S   |  81.3    |  213.2      | 108.0       | 10.98 |
|             | Swin-B   | 81.7     |   278.6     | 147.0       | 9.54  |
|             | Swin-L   |  81.6    |     449.5   | 255.4       | 7.24  |



Meta-review: This paper proposes a mask propagation framework for efficient video semantic segmentation. The paper received 3 weak accepts, 1 reject, and 1 borderline accept recommendations from reviewers. Positive points included the novelty of the approach, and competitive accuracy and efficiency trade-off. Negative points included lack of motivation/details for some approach components, lacking novelty (especially by one reviewer), unclear performance improvements compared to similar related works, and missing FPS + computational costs. Most of these concerns were adequately addressed by the rebuttal. The reviewer who gave the reject rating did not participate in the post-rebuttal discussions; however, the ACs felt that the reviewer’s major concerns were addressed by the rebuttal. Overall, after carefully considering the paper, rebuttal, and discussions, the ACs feel that the paper makes a good contribution and recommend accept. It is recommended that the authors incorporate the visualizations of the query-based flow and pixel-wise flow into the final supplementary material.
Decision: Accept (poster)