Title: Mask Propagation for Efficient Video Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Video Semantic Segmentation; Inference Efficiency
Abstract: Video Semantic Segmentation (VSS) involves assigning a semantic label to each pixel in a video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS. Our approach first employs a strong query-based image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design a flow estimation module utilizing the learned queries to generate a set of segment-aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped to serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by 4.0% mIoU, requiring only 26% FLOPs on the VSPW dataset. Moreover, our framework reduces up to 4× FLOPs compared to the per-frame Mask2Former baseline with only up to 2% mIoU degradation on the Cityscapes validation set. Code is available at https://github.com/ziplab/MPVSS.

Mask Propagation for Efficient Video Semantic Segmentation

Abstract

Video Semantic Segmentation (VSS) involves assigning a semantic label to each pixel in a video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS . Our approach first employs a strong querybased image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design a flow estimation module utilizing the learned queries to generate a set of segment-aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped to serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by 4.0% mIoU, requiring only 26% FLOPs on the VSPW dataset. Moreover, our framework reduces up to 4 × FLOPs compared to the per-frame Mask2Former baseline with only up to 2% mIoU degradation on the Cityscapes validation set. Code is available at https://github.com/ziplab/MPVSS .

1 Introduction

Video Semantic Segmentation (VSS), a fundamental task in computer vision, seeks to assign a semantic category label to each pixel in a video sequence. Previous research on VSS has leveraged developments in image semantic segmentation models, e.g ., FCN [40] and Deeplab [82, 4], which made tremendous progress in the field. However, adapting image semantic segmentation models to VSS remains challenging. On one hand, sophisticated temporal modeling is required to capture the intricate dynamics among video frames. On the other hand, videos contain a significantly larger volume of data compared to images. Processing every frame with a strong image segmentor can incur significant computational costs.

Prior work in VSS mainly focuses on leveraging both temporal and spatial information to enhance the accuracy of pixel-wise labeling on each video frame, building upon the pixel-wise classification paradigm of traditional image semantic segmentors. Specifically, these methods exploit temporal

GT

Query-based flow

Op4cal flow

information by integrating visual features from previous frames into the target frame, employing techniques such as recurrent units [26, 43], temporal shift [33, 1], or spatial-temporal attention modules [19, 44, 28, 16, 53], among others.

More recently, with the prevalence of the DETR [2] framework, DETR-like segmentors [8, 7, 51] have dominated the latest advances and achieved state-of-the-art performance in semantic segmentation. Notably, MaskFormer series [8, 7] learn a set of queries representing target segments, and employ bipartite matching with ground truth segments as the training objective. Each query is learned to predict a binary mask and its associated class prediction. The final result is obtained by combining the binary masks and class predictions of all queries via matrix multiplication. However, these models require computing a high-resolution per-pixel embedding for each video frame using a powerful image encoder, leading to significant computational costs. For instance, processing an input video clip with 30 × 1024 × 2048 RGB frames using the strong Mask2Former [7] requires more than 20.46T Floating-point Operations (FLOPs) in total. Such computational demands render it impractical to deploy these models in real-world VSS systems.

A typical solution is to incorporate temporal information guided by optical flow [50, 82, 22, 66] to reduce the redundancy. The related literature proposes to propagate feature maps from the key frame to other non-key frames using optical flow. The motivation mainly comes from two aspects. First, videos exhibit high temporal redundancy due to the similar appearance of objects or scenes in consecutive frames [13], especially in videos with a high frame rate. As shown in Fig. 1, we empirically observe that semantic information in consecutive video frames is highly correlated. Additionally, previous methods [50, 82, 22, 30, 25, 45] highlight the gradual changes in semantic features within deep neural networks. They attempt to reduce the computational costs by reusing feature maps from preceding frames for the target frame, facilitated by the estimated optical flow between the neighboring frames. However, these methods may still suffer from degraded performance due to inaccurate and noisy optical flow estimation, which only measures pixel-to-pixel correspondence between adjacent frames. Recent advancements in Transformer-based optical flow methods [58, 20, 63, 64] demonstrate the importance of incorporating global correspondence information for each pixel, resulting in more precise per-pixel optical flow estimation. Nevertheless, these methods still focus solely on pixel-level correspondence and do not account for segment-level information among video frames, as illustrated in Fig. 1, which may be insufficient for the task of VSS.

In this paper, we present an efficient mask propagation framework for VSS, namely MPVSS , as shown in Fig. 2. This framework relies on two key components: a strong query-based image segmentor for the key frame (upper part of Fig. 2), and a powerful query-based flow estimator for the nonkey frames (lower part of Fig. 2). Specifically, we employ the state-of-the-art query-based image segmentation model, i.e ., Mask2Former [7] to process sparse key frames and generate accurate binary masks and class predictions. We then propose to estimate individual flow map corresponding to each segment-level mask prediction of the key frame. To achieve this, we leverage the learned queries from the key frame to aggregate segment-level motion information between adjacent frame pairs and capture segment-specific movements along the temporal dimension. These queries are fed to a flow head to predict a set of segment-aware flow maps, followed by a refinement step. The mask

predictions from key frames are subsequently warped to other non-key frames through the generated query-based flow maps, where each flow map is tailored to the corresponding mask predicted by the associated query in the key frame. Finally, we derive semantic maps for the non-key frames by aggregating the warped mask predictions with the class probabilities predicted from the key frame. By warping the predicted segmentation masks from key frames, our model avoids processing each individual video frame using computationally intensive semantic segmentation models. This not only reduces temporal redundancy but also significantly lowers the computational costs involved.

In summary, our main contributions are in three folds:

We propose MPVSS , a novel mask propagation framework for efficient VSS, which reduces computational costs and redundancy by propagating accurate mask predictions from key frames to non-key frames.

We devise a novel query-based flow estimation module that leverages the learned queries from the key frame to model motion cues for each segment-level mask prediction, yielding a collection of accurate flow maps. These flow maps are utilized to propagate mask predictions from the key frame to other non-key frames.

Extensive experiments on standard benchmarks, VSPW [42] and Cityscapes [9], demonstrate that our MPVSS achieves SOTA accuracy and efficiency trade-offs.

2 Related Work

Image semantic segmentation. As a fundamental task in computer vision, image semantic segmentation seeks to assign a semantic label to each pixel in an image. The pioneering work of Fully Convolutional Networks (FCNs) [40] first adopts fully convolutional networks to perform pixel-wise classification in an end-to-end manner and apply a classification loss to each output pixel, which naturally groups pixels in an image into regions of different categories. Building upon the per-pixel classification formulation, various segmentation methods have been proposed in the past few years. Some works aim at learning representative features, which propose to use atrous convolutional layers to enlarge the receptive field [68, 4, 5, 46, 67], aggregate contextual information from multi-scale feature maps via a pyramid architecture [17, 36], encoder-decoder architecture [49, 6], or utilize attention modules to capture the global dependencies [27, 76, 14, 79, 78, 61]. Another line of work focuses on introducing boundary information to improve prediction accuracy for details [10, 29, 70, 77]. More recent methods demonstrate the effectiveness of Transformer-based architectures for semantic segmentation. SegFormer [62], Segmentor [51], SETR [78], MaskFormer [8, 7] replace traditional convolutional backbones [18] with Vision Transformers [11, 59, 39, 34] and/or implement the head with Transformers following the DETR-like framework [2]. Basically, we propose an efficient framework for video semantic segmentation that leverages the mask predictions generated by DETR-like image segmentors. Our framework focuses on propagating these mask predictions across video frames, enabling accurate and consistent segmentation results throughout the entire video sequence. By incorporating the strengths of DETR-like models in image segmentation, we achieve high-quality and efficient video semantic segmentation.

Video semantic segmentation. Video semantic segmentation has recently captured the attention of researchers due to its dynamic nature and practical significance in real-world scenarios. Compared with image semantic segmentation, most existing methods of VSS pay attention to exploiting temporal information, which can be divided into two groups. The first group of approaches exploits temporal relationships to improve prediction accuracy and consistency. For example, some works utilize recurrent units [26, 43] or design an attention propagation module [19] to warp the features or results of several reference frames [81, 15], or aggregate neighbouring frames based on optical flow estimation [15, 37]. In particular, ETC [38] proposes temporal consistency regularization during training, [73] instead exploits perceptual consistency in videos, MRCFA [54] mines relationship of cross-frame affinities, STT [28] employs spatial-temporal transformer, CFFM-VSS [53] disentangles static and dynamic context for video frames. The second line of methods infers to alleviate the huge temporal redundancy among video frames and thus reduce the computational cost. Observing the high similarity of the semantic representation of deep layers, ClockNet [50] designs a pipeline schedule to adaptively reuse feature maps from previous frames in certain network layers. DFF [82] utilizes optical flow to propagate feature maps from key frames to non-key frames, achieving remarkable computational efficiency. Accel [22] further proposes to execute a large model on key frames and

Frame pair

[𝐼

!

; 𝐼

"

]

⨂

Current frame result

𝑃

"

∈ ℝ

#×%×&

backbone

Pixel decoder

Transformer

decoder

N queries

Class head

⨂

⨂

Keyframe

𝐼

!

N mask predictions

Key frame result

𝑃

!

∈ ℝ

#×%×&

𝑀

!

∈ ℝ

'×%×&

Warping

function

MLP

ℱ

"→!

∈ ℝ

'×)×%×&

Motion encoder

Motion

decoder

Flow-based Warping Path

N query-based flow maps

Mask head

𝒬

𝒪

!

∈ ℝ

'×#

!

𝒬

ℱ

∈ ℝ

"×$

MLP

Conv2d

Flow head

ℱ

+,

ℱ

-,

ℰ

.

ℰ

ℱ

a compact one on non-key frames, while DVN [66] proposes a region-based execution scheme on non-key frames, as well as an dynamic key-frame scheduling policy. LLVS [30] develops an efficient framework involving both adaptive key frame selection and selective feature propagation. Unlike methods that primarily rely on optical flow, which models dense pixel-to-pixel correspondence, we introduce a novel query-based flow module that focuses on segment-level flow fields between adjacent frames. By operating at the segment level, our method captures higher-level motion information that is more accurate for VSS.

Motion estimation by optical flow. Optical flow estimation is a fundamental module in video analysis, which estimates a dense field of pixel displacement between adjacent frames. In deep learning era, FlowNet [12] first utilizes convolutional neural networks to learn from labeled data. Most follow-up methods propose to employ spatial pyramid networks [52], or utilize recurrent layers [58] to apply iterative refinement for the predicted flow in a coarse-to-fine manner [52, 47, 24, 55]. More recent methods pioneer Transformers to capture the global dependencies of cost volume [58, 20], or extract representative feature maps for global matching [63], addressing the challenge of large displacement. As optical flow represents the pixel-to-pixel correspondence in adjacent frames, it is often utilized in a wide range of video analysis tasks, i.e ., optical flow as motion information for modeling action motion in the tasks of action recognition [3, 56], temporal action detection [57, 32], or improving the feature representation in video segmentation [74], object detection [81]; optical flow estimation as a simultaneous task to model pixel correspondence for video interpolation [23, 48, 21], video inpainting [71, 72, 31], etc.

3 Mask Propagation via Query-based Flow

In Sec. 3.1, we introduce the overall pipeline of the proposed MPVSS . In Sec. 3.2, we simply revisit Mask2Former that MPVSS is built upon. Finally, in Sec. 3.3, we introduce our query-based flow estimation method, which yields a set of flow maps for efficiently propagating mask predictions from key frames to other non-key frames.

3.1 Overview

The overall architecture is shown in Fig. 2. Let { I t ∈ R H × W × 3 } T t =1 be an input video clip with T frames, where H and W denote the height and width of frames in the video. Without loss of generality, we select key frames from the video clip at fixed intervals, e.g ., every 5 frames, considering other frames within these intervals as non-key frames. For simplicity, we denote each key frame as I k and non-key frame as I j . To reduce redundant computation, we only run the heavy image segmentation model on the sampled key frames and propagate the predictions from key frames to non-key frames.

Specifically, for a key frame I k , we adopt Mask2Former [7] (upper part of Fig. 2) to generate N mask predictions M k and class predictions. Then, we use the flow estimation module (bottom part

of Fig. 2) to predict a set of flow maps F j → k ∈ R N × 2 × H × W indicating the motion changes from I j to I k , which will be discussed in Sec. 3.3. Subsequently, we apply the bilinear warping function on all locations to propagate per-segment mask predictions M k to obtain M j of non-key frames, i.e ., M j = W ( M k , F j → k ) . Consequently, we segment the non-key frame I j by propagating the predicted masks from its preceding key frame I k , avoiding the necessity of computationally intensive image segmentation models for every video frame.

During training, we apply the loss function in Mask2Former [7] on the warped mask predictions M j and perform bipartite matching with the ground truth masks for the current frame. The training approach is similar to the process described in [82], where the loss gradients are back-propagated throughout the model to update the proposed flow module.

3.2 Mask2Former for Segmenting Key Frames

Mask2Former consists of three components: a backbone, a pixel decoder and a transformer decoder, as depicted in the upper part of Fig. 2. The backbone extracts feature maps from the key frame I k ∈ R H × W × 3 . The pixel decoder builds a multi-scale feature pyramid as well as generates a highresolution per-pixel embedding, following FPN [35] and Deformable DETR [80]. In transformer decoder, the target segments are represented as a set of learned queries Q k O ∈ R N × C where C is the channel dimension. After gradually enriching their representations with stacked decoder blocks, Q k O is fed into a class and a mask head to yield N class probabilities and N mask embeddings, respectively. After decoding the binary mask predictions M k ∈ R N × H × W with the class predictions and the perpixel embeddings, we can finally derive the semantic maps for the key frame by aggregating N binary mask predictions with their corresponding predicted class probabilities via matrix multiplication. We refer readers to [8, 7] for more details.

3.3 Query-based Flow Estimation

To generate the segmentation masks for the non-key frames efficiently, we propose to estimate a set of flow maps, each corresponding to a mask prediction of the key frame. We start with introducing the process of generating N query-based flow maps between the current non-key frame I j and its preceding key frame I k , where 0 < j -k ≤ T . The overall flow module comprises three key elements: a motion encoder to encode the motion feature pyramid between the pair of video frames { I k , I j } , a motion decoder that leverages the learned queries Q k O from the key frame to extract motion information from the motion feature maps for each segment, and a flow head responsible for predicting N query-based flow maps F j → k , where each flow map is associated with warping a segment (mask prediction) of the key frame to the current non-key frame.

Motion encoder. To obtain motion features between the paired frames { I k , I j } , we concatenate the two frames along the channel dimension as the input of the motion encoder. The motion encoder, utilizing the same architecture as the flow encoder in FlowNet [12], is employed to extract motion features at each location in a downsampling manner. Following [12, 7], we reverse the order of the original feature maps, generating a motion feature pyramid { b 1 , b 2 , b 3 , b 4 } , each at resolutions 1 / 32 , 1 / 16 , 1 / 8 and 1 / 4 of the original video frame, respectively. Subsequently, the feature pyramid is fed to the model decoder for the decoding of flow maps from the lowest to the highest resolution.

Motion decoder. We then devise a motion decoder to aggregate motion information from the motion feature pyramid using N flow queries Q F .

First, the motion decoder takes as input the first three feature maps { b 1 , b 2 , b 3 } from the motion feature pyramid and the flow queries Q F . We initialize flow queries with the N learned queries Q k O of the preceding key frame, i.e ., Q 1 F = Q k O , where Q 1 F indicates the input flow queries to the motion decoder. Following [7], feature map in each layer is projected into C channels in the channel dimension using a linear layer, which is then added with a sinusoidal positional embedding and a learnable level embedding. This results in three scales of motion feature maps in total, i.e ., { B 1 1 , B 1 2 , B 1 3 } , as the input of the following layers.

Next, the motion decoder can be divided into S stages, each indexed by s , with each stage involving L blocks, each indexed by l . We adopt S = 3 and L = 3 for our model. Inspired by [7, 65], starting from s = 1 and l = 1 , for each block in each stage, we first concatenate B s l and Q s F together and then feed them into a number of Transformer layers, each of which performs information propagation

between B s l and Q s F .

where [; ] denotes the concatenation operation. LN( · ) , MSA( · ) and FFN( · ) denote LayerNorm layer, Mutli-head Self-Attention and Feed-forward Neural Network, respectively. Therefore, in each stage, we learn flow queries and update motion feature maps via self-attention. The self-attention mechanism aggregates motion information globally from both the motion feature maps and the flow query features simultaneously. Finally, the learned flow queries ˆ Q F and updated motion feature maps { ˆ B 1 , ˆ B 2 , ˆ B 3 } are fed to the flow head for flow prediction.

As each query of Q k O encodes global information for each segment in the key frame, using Q k O as initial flow queries enables each flow query to extract motion information for each segment present in the key frame, which facilitates query-based motion modeling.

Flow head. The flow head generate the final flow maps F j → k ∈ R N × 2 × H × W by combining query-based flow F QF and pixel-wise flow F PF .

To obtain the query-based flow F QF , an MLP with 2 hidden layers is employed to convert the learnt flow queries ˆ Q F to flow embeddings E F ∈ R N × C , each with C channels. Inspired by the pixel decoder design in Mask2Former, we project the motion feature map b 4 from the motion encoder to E B ∈ R 2 C × H 4 × W 4 using another MLP with 2 hidden layers. Let E x B and E y B denote the first and last C channels of E B , and they are used to obtain the query-based flow predictions in terms of x and y direction, respectively. Then, for the n th query, we get the flow map via a dot product between the n th flow embedding E F n and the projected motion map E B , i.e ., F QF n ( x ) = E F n · E x B and F QF n ( y ) = E F n · E y B in terms of x and y direction, respectively. Then we can obtain the predicted query-based flow F QF n = [ F QF n ( x ); F QF n ( y )] ∈ R 2 × H 4 × W 4 for the n th query. The dot product of the flow embeddings and projected motion maps quantifies the similarity between the segment-level motion vector and the encoded motion information at each location on the feature maps, which reflects the movement for each query on the spatial dimension.

Additionally, we use the updated motion feature maps { ˆ B 1 , ˆ B 2 , ˆ B 3 } to predict a pixel-wise flow F PF ∈ R 2 × H 4 × W 4 in a coarse-to-fine manner, following [12], which is used to refine the query-based flow.

To obtain the final flow map F j → k n ∈ R 2 × H × W for the n th query, we apply a 2D convolutional layer to the concatenated F QF n and F PF , and then upsample the flow map to the original resolution:

Remark. We use N flow queries, which are initialized with the learned per-segment queries Q k O from the key frame, to capture the motion for each mask predicted on the key frame. In contrast to existing optical flow estimation methods, which model pixel-to-pixel dense correspondence, we propose a novel query-based flow that incorporates segment-level motion information between adjacent frames, assisting more accurate mask propagation in VSS.

4 Experiments

4.1 Experimental Setup

Datasets. We evaluate our method on two benchmark datasets: VSPW [42] and Cityscapes [9]. VSPW is the largest video semantic segmentation benchmark, consisting of 2,806 training clips (198,244 frames), 343 validation clips (24,502 frames), and 387 test clips (28,887 frames). Each video in VSPW has densely annotated frames at a rate of 15 frames per second, covering 124 semantic categories across both indoor and outdoor scenes. Additionally, we present results on the Cityscapes dataset, which provides annotations every 30 frames. This dataset serves as an additional evaluation to showcase the effectiveness of our proposed method.

Implementation details. By default, all experiments are trained with a batch size of 16 on 8 NVIDIA GPUs. Note that pre-training is important for the DETR-like segmentors. For VSPW dataset, we pretrain Mask2Former on the image semantic segmentation task, serving as the per-frame baseline. For Cityscapes, we adopt pretrained Mask2Former with ResNet [18] and Swin Transformer [39] backbones from [7]. For the motion encoder, we utilized the weights from FlowNet [12] encoder which is pre-trained on the synthetic Flying Chairs dataset; the motion decoder and flow head are randomly initialized. We freeze most parameters of the pretrained Mask2Former, fine-tuning only its classification and mask head, as well as the proposed flow module. We apply loss functions in Mask2Former, including a classification loss and a binary mask loss on the class embeddings and warped mask embeddings, respectively. All the models are trained with the AdamW optimizer [41] for a maximum of 90k iterations and the polynomial learning rate decay schedule [4] with an initial learning rate of 5e-5.

Compared methods. To validate the effectiveness of our proposed model, we include the following methods for study: Per-frame : Employing Mask2Former [7] to independently predict masks for each video frame. Optical flow : Warping mask predictions from key frames to non-key frames using optical flow estimated by a lightweight optical flow model, i.e ., FlowNet [12]. Query-random : Implementing the proposed flow estimation method with randomly initialized flow queries. Querylearned : Utilizing the proposed flow estimation method using the learned queries Q k O as initialization. Query-for-PF : Using pixel-wise flow F PF which incorporates segment-level information from the learned flow queries for warping predictions.

Evaluation metrics. Following previous works, we use mean Intersection over Union (mIoU) at single-scale inference, and Weighted IoU (WIoU) to evaluate the segmentation performance. We also compare models in terms of their computational complexity with FLOPs to evaluate the efficiency of these VSS models. We calculate FLOPs for each single frame by averaging on a video clip with 15 frames and fix the image resolution of 480 × 853 and 1024 × 2048 for VSPW and Cityscapes, respectively. For VSPW dataset, we adopt video consistency (VC) [42] to evaluate the visual smoothness of the predicted segmentation maps across the temporal dimension.

4.2 Comparison with State-of-the-art Methods

The comparisons on VSPW dataset are shown in Tab. 1. We also report the backbone used by each method and the computational complexity (GFOLPs), averaged over a video clip of 15 frames, with each frame of resolution 480 × 853. For our proposed models, we use 5 as the default key frame interval for comparison. We have the following analysis: 1) The proposed MPVSS achieves comparable accuracy with significantly reduced computational cost when compared to the strong baseline method, Mask2Former [7]. Specifically, compared to Mask2Former, our MPVSS reduces the computational cost in terms of FLOPs by 71.7G, 96.2G, 74.7G, 104.9G, 162.0G, and 305.4G on R50, R101, Swin-T, Swin-S, Swin-B, and Swin-L backbones, respectively, while only experiencing 1.0%, 0.5%, 1.3%, 1.7%, 1.5%, and 2.2% degradation in the mIoU score. These results demonstrate a promising trade-off between accuracy and efficiency of our framework. 2) Our models with different backbones perform on par with the state-of-the-art VSS methods on the VSPW dataset with much fewer FLOPs. Specifically, our MPVSS model with the Swin-L backbone achieves an impressive mIoU of 53.9%, surpassing CFFM-VSS with MiT-B5 backbone by 4.6%. Notably, our approach achieves this performance with only 24% of the computational cost in terms of FLOPs. 3) When considering the comparable computational cost, our model consistently outperforms other state-of-the-art methods. For instance, MPVSS with Swin-L backbone surpasses Segformer with MiT-B2 backbone by 10% mIoU, under the FLOPs around 3.5G. These compelling results highlight the exceptional performance and efficiency of our MPVSS against the compared VSS approaches. In terms of temporal consistency, our proposed MPVSS achieves comparable mVC scores when compared to approaches that explicitly exploit cross-frame temporal context [54] or incorporate temporal consistency constraints during training [38]. We provide an explanation of VC score and a discussion in the supplementary material.

We further evaluate our MPVSS on the Cityscapes dataset and achieve state-of-the-art results with relatively low computational complexity. Notably, the Cityscapes dataset only provides sparse anno-

(a) Effect of the proposed flow estimation method.

(b) Effect of each component.

tations for one out of every 30 frames in each video. Therefore, for a fair comparison with previous efficient VSS methods, we report the results following the same evaluation process and compute their average FLOPs per-frame over all video frames. Specifically, compared to Mask2Former, our MPVSS reduces the computational cost in terms of FLOPs by 0.36T, 0.48T, 0.37T, 0.52T, 0.78T and 1.5T on R50, R101, Swin-T, Swin-S, Swin-B, and Swin-L backbones, respectively, with only 1.0%, 1.9%, 1.4%, 1.3%, 1.6% and 1.7% degradation in the mIoU score. The strong accuracy-efficiency trade-offs demonstrate that our proposed framework also works on sparse annotated video data.

4.3 Ablation Study

Effects of the query-based flow. We compare the performance between the proposed query-based flow and the traditional optical flow in Table 3a. Overall, our approach demonstrates superior performance compared with the models relying solely on optical flow for mask propagation from key frames to non-key frames. To be specific, the utilization of the proposed query-based flow for mask warping consistently boosts the performance by 1.4%, 0.6%, 1.5%, 0.3%, 0.8% and 0.7% in terms of mIoU on R50, R101, Swin-T, Swin-S, Swin-B and Swin-L, respectively, which leads to less degradation to the per-frame baseline compared to models using pixel-wise optical flow. Moreover, directly copying the mask predictions from the key frame leads to a clear performance drop.

Effects of each component. In Table 3, we verify the effects of each component design in our query-based flow. We conduct experiments on VSPW using Swin-T and Swin-B. The performance of Query-random on the two backbones is slightly inferior to using optical flow. This suggests that without per-segment query initialization, the query-based flow struggles to retrieve specific motion information from motion features for one-to-one mask warping from the key frame, resulting in suboptimal performance. Then, by using key-frame queries as the initialization for flow queries, we observe an improvement of 1.1% and 1.5% mIoU scores for Query-learned on Swin-T and Swin-B respectively, compared to Query-random . Furthermore, we develop a variant design Queryfor-PF , which introduces additional performance gain by 0.1% and 0.2% compared to optical flow respectively. These results underscore two findings: 1) learning query-based flow for each segment requires the learned queries from the key frame, for extracting meaningful motion features for each segment; 2) integrating segment-level information benefits the learning of pixel-wise flow. Therefore, we incorporate a refinement step using the enhanced pixel-wise flow for producing each query-based flow, which results in the best performance on both backbones. The outcomes highlight that warping mask predictions using our query-based flow consistently outperforms the methods using traditional optical flow on VSS.

Effects of the key frame interval. In Fig. 3a and Fig. 3b, we show the mIoU scores and TFLOPs versus key frame interval for our MPVSS with different backbones, respectively. Overall, all the results achieve significant FLOPs reduction with decent accuracy drop, which smoothly trade in accuracy for computational resource and fit different application needs flexibly. In Fig. 3c, we present the performance of warping mask predictions using our query-based flow and traditional optical flow, as well as direct copying version, on Swin-T. We observe that, when the key frame interval increases from 2 to 10, the performance of the proposed query-based flow only slightly degraded by 1.1% and 0.7% in terms of mIoU and WIoU score, while the scores are decreased by 2.63% and 2.96% using optical flow. These results indicates that the proposed query-based flow is more robust to capture long-term temporal changing for videos. We provide comparisons on other backbones in the supplementary material.

1

5

10

15

20

Key frame interval

30

35

40

45

50

55

60

mIoU(%)

Swin-T

Swin-B

Swin-S

Swin-L

R50

R101

(a) mIoU vs. Key frame interval

with different backbones.

1

5

10

15

20

Key frame interval

2

4

6

8

TFLOPs

Swin-L

Swin-B

Swin-S

Swin-T

R101

R50

(b) TFLOPs vs. Key frame inter- val with different backbones.

1

5

10

15

20

Key frame interval

30

32

34

36

38

40

42

44

mIoU(%)

Query-based flow

Copy

Optical flow

(c) mIoU vs. Key frame interval of different warping method.

4.4 Qualitative Results

In Fig. 4, we present two examples from VSPW dataset to visually compare the prediction quality of the per-frame baseline, the method using optical flow, and the proposed MPVSS . We observe that the proposed query-based flow exhibits superior performance in capturing the movement of small objects when compared to optical flow. Additionally, the proposed mask propagation framework demonstrates enhanced category consistency compared to the per-frame baseline. For instance, the per-frame baseline exhibits category inconsistency as each frame is independently predicted. This highlights the advantage of our proposed approach in maintaining category coherence across consecutive frames.

(a)

(b)

(c)

(d)

5 Conclusion and Future Work

In this paper, we have presented a simple yet effective mask propagation framework, dubbed MPVSS , for efficient VSS. Specifically, we have employed a strong query-based image segmentor to process key frames and generate accurate binary masks and class predictions. Then we have proposed to estimate specific flow maps for each segment-level mask prediction of the key frame. Finally, the mask predictions from key frames were subsequently warped to other non-key frames via the proposed query-based flow maps. Extensive experiments on VSPW and Cityscapes have demonstrated that our MPVSS achieves SOTA accuracy and efficiency trade-off. Future work could explore extending MPVSS to other video dense prediction tasks, e.g ., video object and instance segmentation.

Limitations and broader impacts. Although our framework significantly reduces computational costs for processing videos, the new module leads to an increase in the number of parameters in the models. Additionally, there might be a need for fine-tuning the new module or optimizing hyperparameters to achieve the best performance. It can lead to increased carbon emissions, indicating the potential negative societal impact.

References

A. Bulat, B. Martinez, and G. Tzimiropoulos. Efficient attention-free video shift transformers. arXiv preprint arXiv:2208.11108 , 2022.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV , pages 213-229. Springer, 2020.

J. Carreira and A. Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , pages 6299-6308, 2017.

L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. TPAMI , 40(4):834-848, 2017.

L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 , 2017.

L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In ECCV , pages 801-818, 2018.

B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR , pages 1290-1299, 2022.

B. Cheng, A. Schwing, and A. Kirillov. Per-pixel classification is not all you need for semantic segmentation. NIPS , 34:17864-17875, 2021.

M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR , pages 3213-3223, 2016.

H. Ding, X. Jiang, A. Q. Liu, N. M. Thalmann, and G. Wang. Boundary-aware feature propagation for scene segmentation. In ICCV , pages 6819-6829, 2019.

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

A. Dosovitskiy, P. Fischer, E. Ilg, P. Hausser, C. Hazirbas, V. Golkov, P. Van Der Smagt, D. Cremers, and T. Brox. Flownet: Learning optical flow with convolutional networks. In CVPR , pages 2758-2766, 2015.

C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In ICCV , pages 6202-6211, 2019.

J. Fu, J. Liu, H. Tian, Y. Li, Y. Bao, Z. Fang, and H. Lu. Dual attention network for scene segmentation. In CVPR , pages 3146-3154, 2019.

R. Gadde, V. Jampani, and P. V. Gehler. Semantic video cnns through representation warping. In ICCV , pages 4453-4462, 2017.

M. Han, D. J. Zhang, Y. Wang, R. Yan, L. Yao, X. Chang, and Y. Qiao. Dual-ai: Dual-path actor interaction learning for group activity recognition. In CVPR . IEEE, 2022.

K. He, X. Zhang, S. Ren, and J. Sun. Spatial pyramid pooling in deep convolutional networks for visual recognition. TPAMI , 37(9):1904-1916, 2015.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi. Temporally distributed networks for fast video semantic segmentation. In CVPR , pages 8818-8827, 2020.

Z. Huang, X. Shi, C. Zhang, Q. Wang, K. C. Cheung, H. Qin, J. Dai, and H. Li. Flowformer: A transformer architecture for optical flow. In ECCV , pages 668-685. Springer, 2022.

Z. Huang, T. Zhang, W. Heng, B. Shi, and S. Zhou. Real-time intermediate flow estimation for video frame interpolation. In ECCV , pages 624-642. Springer, 2022.

S. Jain, X. Wang, and J. E. Gonzalez. Accel: A corrective fusion network for efficient semantic segmentation on video. In CVPR , pages 8866-8875, 2019.

L. Kong, B. Jiang, D. Luo, W. Chu, X. Huang, Y. Tai, C. Wang, and J. Yang. Ifrnet: Intermediate feature refine network for efficient frame interpolation. In CVPR , pages 1969-1978, 2022.

L. Kong, C. Shen, and J. Yang. Fastflownet: A lightweight network for fast optical flow estimation. In ICRA , pages 10310-10316. IEEE, 2021.

S.-P. Lee, S.-C. Chen, and W.-H. Peng. Gsvnet: Guided spatially-varying convolution for fast semantic segmentation on video. In ICME , pages 1-6. IEEE, 2021.

P. Lei and S. Todorovic. Recurrent temporal deep field for semantic video labeling. In ECCV , pages 302-317. Springer, 2016.

H. Li, P. Xiong, J. An, and L. Wang. Pyramid attention network for semantic segmentation. arXiv preprint arXiv:1805.10180 , 2018.

J. Li, W. Wang, J. Chen, L. Niu, J. Si, C. Qian, and L. Zhang. Video semantic segmentation via sparse temporal transformer. In ACMMM , pages 59-68, 2021.

X. Li, X. Li, L. Zhang, G. Cheng, J. Shi, Z. Lin, S. Tan, and Y. Tong. Improving semantic segmentation via decoupled body and edge supervision. In ECCV , pages 435-452. Springer, 2020.

Y. Li, J. Shi, and D. Lin. Low-latency video semantic segmentation. In CVPR , pages 5997-6005, 2018.

Z. Li, C.-Z. Lu, J. Qin, C.-L. Guo, and M.-M. Cheng. Towards an end-to-end framework for flow-guided video inpainting. In CVPR , pages 17562-17571, 2022.

C. Lin, C. Xu, D. Luo, Y. Wang, Y. Tai, C. Wang, J. Li, F. Huang, and Y. Fu. Learning salient boundary feature for anchor-free temporal action localization. In CVPR , pages 3320-3329, 2021.

J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In ICCV , pages 7083-7093, 2019.

S. Lin, H. Xie, B. Wang, K. Yu, X. Chang, X. Liang, and G. Wang. Knowledge distillation via the target-aware transformer. In CVPR , pages 10915-10924, 2022.

T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR , pages 2117-2125, 2017.

J. Liu, J. He, J. Zhang, J. S. Ren, and H. Li. Efficientfcn: Holistically-guided decoding for semantic segmentation. In ECCV , pages 1-17. Springer, 2020.

S. Liu, C. Wang, R. Qian, H. Yu, R. Bao, and Y. Sun. Surveillance video parsing with single frame supervision. In CVPR , pages 413-421, 2017.

Y. Liu, C. Shen, C. Yu, and J. Wang. Efficient semantic video segmentation with per-frame inference. In ECCV , pages 352-368, 2020.

Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , pages 9992-10002, 2021.

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In CVPR , pages 3431-3440, 2015.

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

J. Miao, Y. Wei, Y. Wu, C. Liang, G. Li, and Y. Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In CVPR , pages 4133-4143, 2021.

D. Nilsson and C. Sminchisescu. Semantic video segmentation by gated recurrent flow propagation. In CVPR , pages 6819-6828, 2018.

M. Paul, M. Danelljan, L. Van Gool, and R. Timofte. Local memory attention for fast video semantic segmentation. In IROS , pages 1102-1109. IEEE, 2021.

M. Paul, C. Mayer, L. V. Gool, and R. Timofte. Efficient video semantic segmentation with labels propagation and refinement. In WACV , pages 2873-2882, 2020.

C. Peng, X. Zhang, G. Yu, G. Luo, and J. Sun. Large kernel matters-improve semantic segmentation by global convolutional network. In CVPR , pages 4353-4361, 2017.

A. Ranjan and M. J. Black. Optical flow estimation using a spatial pyramid network. In CVPR , pages 4161-4170, 2017.

F. Reda, J. Kontkanen, E. Tabellion, D. Sun, C. Pantofaru, and B. Curless. Film: Frame interpolation for large motion. In ECCV , pages 250-266. Springer, 2022.

O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI , pages 234-241. Springer, 2015.

E. Shelhamer, K. Rakelly, J. Hoffman, and T. Darrell. Clockwork convnets for video semantic segmentation. In ECCV , pages 852-868, 2016.

R. Strudel, R. Garcia, I. Laptev, and C. Schmid. Segmenter: Transformer for semantic segmentation. In ICCV , pages 7262-7272, 2021.

D. Sun, X. Yang, M.-Y. Liu, and J. Kautz. Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In CVPR , pages 8934-8943, 2018.

G. Sun, Y. Liu, H. Ding, T. Probst, and L. Van Gool. Coarse-to-fine feature mining for video semantic segmentation. In CVPR , pages 3126-3137, 2022.

G. Sun, Y. Liu, H. Tang, A. Chhatkuli, L. Zhang, and L. Van Gool. Mining relations among cross-frame affinities for video semantic segmentation. In ECCV , pages 522-539, 2022.

S. Sun, Y. Chen, Y. Zhu, G. Guo, and G. Li. Skflow: Learning optical flow with super kernels. arXiv preprint arXiv:2205.14623 , 2022.

S. Sun, Z. Kuang, L. Sheng, W. Ouyang, and W. Zhang. Optical flow guided feature: A fast and robust motion representation for video action recognition. In CVPR , pages 1390-1399, 2018.

J. Tan, J. Tang, L. Wang, and G. Wu. Relaxed transformer decoders for direct action proposal generation. In ICCV , pages 13526-13535, 2021.

Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV , pages 402-419. Springer, 2020.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML , pages 10347-10357, 2021.

T. Xiao, Y. Liu, B. Zhou, Y . Jiang, and J. Sun. Unified perceptual parsing for scene understanding. In ECCV , pages 418-434, 2018.

E. Xie, W. Wang, W. Wang, P. Sun, H. Xu, D. Liang, and P. Luo. Segmenting transparent object in the wild with transformer. arXiv preprint arXiv:2101.08461 , 2021.

E. Xie, W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. NIPS , 34:12077-12090, 2021.

H. Xu, J. Zhang, J. Cai, H. Rezatofighi, and D. Tao. Gmflow: Learning optical flow via global matching. In CVPR , pages 8121-8130, 2022.

H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger. Unifying flow, stereo and depth estimation. arXiv preprint arXiv:2211.05783 , 2022.

J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR , pages 18134-18144, 2022.

Y.-S. Xu, T.-J. Fu, H.-K. Yang, and C.-Y. Lee. Dynamic video segmentation network. In CVPR , pages 6556-6565, 2018.

M. Yang, K. Yu, C. Zhang, Z. Li, and K. Yang. Denseaspp for semantic segmentation in street scenes. In CVPR , pages 3684-3692, 2018.

F. Yu and V. Koltun. Multi-scale context aggregation by dilated convolutions. arXiv preprint arXiv:1511.07122 , 2015.

Y. Yuan, X. Chen, and J. Wang. Object-contextual representations for semantic segmentation. In ECCV . Springer, 2020.

Y. Yuan, J. Xie, X. Chen, and J. Wang. Segfix: Model-agnostic boundary refinement for segmentation. In ECCV , pages 489-506. Springer, 2020.

K. Zhang, J. Fu, and D. Liu. Flow-guided transformer for video inpainting. In ECCV , pages 74-90. Springer, 2022.

K. Zhang, J. Fu, and D. Liu. Inertia-guided flow completion and style fusion for video inpainting. In CVPR , pages 5982-5991, 2022.

Y. Zhang, S. Borse, H. Cai, Y. Wang, N. Bi, X. Jiang, and F. Porikli. Perceptual consistency in video segmentation. In WACV , pages 2564-2573, 2022.

Y. Zhang, A. Robinson, M. Magnusson, and M. Felsberg. Flow-guided semi-supervised video object segmentation. arXiv preprint arXiv:2301.10492 , 2023.

H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia. Pyramid scene parsing network. In CVPR , pages 2881-2890, 2017.

H. Zhao, Y. Zhang, S. Liu, J. Shi, C. C. Loy, D. Lin, and J. Jia. Psanet: Point-wise spatial attention network for scene parsing. In ECCV , pages 267-283, 2018.

M. Zhen, J. Wang, L. Zhou, S. Li, T. Shen, J. Shang, T. Fang, and L. Quan. Joint semantic segmentation and boundary detection using iterative pyramid contexts. In CVPR , pages 1366613675, 2020.

S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In CVPR , pages 6881-6890, 2021.

Z. Zhong, Z. Q. Lin, R. Bidart, X. Hu, I. B. Daya, Z. Li, W.-S. Zheng, J. Li, and A. Wong. Squeeze-and-attention networks for semantic segmentation. In CVPR , pages 13065-13074, 2020.

X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable transformers for end-to-end object detection. In ICLR , pages 3-7, 2021.

X. Zhu, Y. Wang, J. Dai, L. Yuan, and Y. Wei. Flow-guided feature aggregation for video object detection. In ICCV , pages 408-417, 2017.

X. Zhu, Y. Xiong, J. Dai, L. Yuan, and Y. Wei. Deep feature flow for video recognition. In CVPR , pages 2349-2358, 2017.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a mask propagation method, MPVSS, for video semantic segmentation (VSS). MPVSS uses Mask2Former to obtain mask predictions and queries from the key frame. Then, a motion encoder extracts pixel-wise motion from the key frame and its adjacent frame. The queries from the key frame are used to decode query-wise motion features. A flow head generates query-based flow maps, and binary mask predictions for the adjacent frame are generated by applying these flow maps to the binary mask predictions of the key frame. Semantic segmentations of non-key frames are produced by matrix multiplication between the key frame's query classification output and the binary mask prediction for the adjacent frame. A detailed ablation study validates each component. Comparisons between MPVSS and state-of-the-art methods on VSPW and Cityscapes demonstrate good performance and efficiency of MPVSS.\n\nSTRENGTHS:\nThis paper is well-written and easy to follow.\nMotivation leveraging motion estimation to reduce the computation of VSS is sound. \nThe proposed method of propagating the key frame’s query to the motion features is interesting and novel. \nThe experiments are well-designed, and the results are convincing.\n\nWEAKNESSES:\nThe key frames are selected at fixed intervals. As a result, the method may not fully address the mentioned redundancy problem.\n\nEfficiency of the proposed method should be discussed in more detail. For instance, MPVSS uses FlowNet to extract motion features between two frames instead of relying on a backbone network to extract frame-wise features.\n\nFig3 (b), y axis label, “TLOPs” seems a typo.\n\nThere is not much information about training time, and any particular training strategies used in the experiments.\n\nQUESTIONS:\nPlease see Weaknesses\n\nLIMITATIONS:\nThe paper adequately addresses limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAn efficient mask propagation framework for VSS, called MPVSS, is proposed in this paper.\n\nMPVSS adopts a strong query-based image segmentation based on sparse key frames and warp prediction to non-key frames by generating a segment-aware flow from a newly designed flow estimation module.\n\nWith the proposed query-based flow, MPVSS achieves the performance of SOTA with a better efficiency.\n\nSTRENGTHS:\n1. MPSVSS captures segment-level information between the video frames, which provides a better propagation result than the pixel-level correspondence information used in the previous work.\n\n2. Compared with vanilla optical flow, query-based flow provides smooth and clear compensation for non-keyframes.\n\n3. The proposed method achieves SOTA performance with significantly reduced computational cost through extensive experiments on standard benchmarks.\n\n4. MPVSS delivers a consistently higher level of video consistency than its base network.\n\nWEAKNESSES:\n1. typo: line 10: wrapped -> warped.\n\n2. The proposed mask propagation framework is identical to that of DFF [1], Accel [2], and CoVOS [3], while the proposed query-based flow requires further experimentation to demonstrate its effectiveness. More specifically:\n- In Table 2, it is difficult to say whether the query-based flow is better than Accel [2] or DFF [1], because the MPVSS uses Mask2Former for key-frames segmentation. In order to prove the advantage of propagation using query-based flow over the propagation module in Accel and DFF, the keyframe segmentation network should be unified.\n- In Table 3a, the query-based flow does bring improvement, but it also introduces additional network parameters. It's hard to say whether the improvement is due to the increase in parameters. \n3. Since propagation relies only on the previous keyframe, I wonder how occlusion will have a negative effect on the propagation results. There is literature using bi-directional motion vector for mask propagation [3] and shows that occlusion has a strong effect on the propagation results, especially causing ghosting effect, however this point is not discussed in the paper.\n4. A measurement on the FPS should be provided.\n\n[1] Zhu, Xizhou, et al. \"Deep feature flow for video recognition.\" CVPR. 2017.\n\n[2] Jain, Samvit, Xin Wang, and Joseph E. Gonzalez. \"Accel: A corrective fusion network for efficient semantic segmentation on video.\" CVPR. 2019.\n\n[3] Xu, Kai, and Angela Yao. \"Accelerating video object segmentation with compressed video.\" CVPR. 2022.\n\nQUESTIONS:\nPlease refer to the Weaknesses section.\n\nLIMITATIONS:\nThe authors adequately addressed limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an approach for video semantic segmentation (VSS) by focusing on the aspect of efficiency. The authors propose a novel mask propagation framework that is built upon a computationally intensive query-based image segmentor called Mask2Former[1]. Instead of processing every frame, the framework leverages the image segmentor to process only the key frames. To generate masks for non-key frames, a novel query-based flow map estimation module is introduced, which predicts optical flows to warp the masks from key frames. The experimental results demonstrate that the proposed framework achieves competitive performance, with only a minimal drop of approximately 1% to 2% when compared to the baseline [1]. Notably, the proposed method achieves these results with significantly reduced Floating Point Operations (FLOPs).\n\n[1] B. Cheng, I. Misra, A. G. Schwing, A. Kirillov, and R. Girdhar. Masked-attention mask transformer for universal image segmentation. In CVPR, pages 1290–1299, 2022.\n\nSTRENGTHS:\n1. The proposed method demonstrates competitive performance on two widely recognized benchmarks, showcasing only a slight decline in the mIoU score, ranging from 1% to 2%. These results are achieved while significantly reducing computation costs (FLOPs).\n\n2. The novel query-based flow estimation module introduced in this paper surpasses traditional pixel-wise optical estimation methods by producing better flow maps. This advancement holds great promise for the field of transformer-based flow estimation.\n\nWEAKNESSES:\nPlease see my comments in below Questions section.\n\nQUESTIONS:\n1. It would be inappropriate to claim that the proposed method (MPVSS) \"achieves SOTA performance\". From Table 1 and Table 2, it becomes evident that MPVSS yields slightly lower mIoU scores compared to its baseline counterpart (Mask2Former). Furthermore, Mask2Former already attains SOTA performance when compared to other methods presented in the table, so the primary contribution of MPVSS lies in its efficiency. Please kindly rectify these statements to accurately reflect the contributions of MPVSS.\n\n2. It would be insightful to delve further into the reasons why the learned query from the mask generation branch (Q^k_O) enhances flow estimation. While the paper briefly touches upon this topic in a single sentence (line 220 - line 222), it would be great to include visualizations or attention maps that demonstrate the gain of the learned query on flow estimation.\n\n3. In line 240 - line 242, two flows (query-based flow and pixel-based flow) are stacked to generate the final flow predictions. To gain a better understanding of the effect of the query-based flow, it would be helpful to visualize the query-based flow and pixel-based flow separately. Additionally, it would be beneficial to clarify whether the first row (i.e., Optical flow) in Table 3 (b) refers to the \"pixel-based flow\".\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper is to develop an approach to video semantic segmentation via propagating the segmented mask in key frames to non-key frames. Experiments were conducted on several databases with various comparisons.

STRENGTHS:
Focusing on improving the computational efficiency for video semantic segmentation;

The paper is well-written.

WEAKNESSES:
It seems that the work lacks of novelty. In my understanding, the key idea is to compute the mask in key frames, and then propagate the segmentation result to non-keyframes, in order to reduce the computation cost. However, in many previous video analysis works, doing many computation in key frames, and then applying the result to non-keyframes, is quite natural. 

The segmentation in key frames is performed by using an existing method, Mask2Former [7].

The flow estimation between frames is quite normal: using FlowNet [11] for motion encoding, and transformer based approach [7] for decoding. Not presenting a new method.

QUESTIONS:
See my comments in the Weakness part.

LIMITATIONS:
limitation is talked in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates an important and fundamental task: video semantic segmentation (VSS). While image semantic segmentation has received significant research attention, VSS has been relatively overlooked due to limited datasets and computational resources. This paper makes a valuable contribution to the field by proposing a method that combines a segmentation network with a flow network. Unlike traditional flow networks, the proposed approach employs query-based flow maps that correspond to each segment. Experimental results demonstrate the effectiveness of the method on two widely-used datasets, namely vspw and cityscape.\n\nSTRENGTHS:\n1. The paper addresses an important but relatively underexplored task.\n2. The combination of segmentation and optical flow in a single model is a novel approach, particularly considering the use of query-based flow maps.\n3. The proposed method achieves commendable performance on standard datasets such as vspw and cityscape.\n\nWEAKNESSES:\n1. The paper lacks specific details regarding the training of the flow modules. For example, what is the loss function used to train query-based flow? How are the flow networks (encoder/decoder) initilized? Do you use trained weights for these modules? Also, more visual examples about the query-based flow maps should be given.\n2. The rationale behind utilizing both pixel-wise flow (F^{PF}) and query-based flow (F^{QF}) to generate query-based flow maps is not well-justified. In my opinion, pixel-wise flow is the flow map for every pixel while the query-based flow is only for specific segments. Equation 4, which concatenates both flows, may make the query-based flow not focus on the segments, but on all the pixels.\n3. It would be helpful to include information on the number of parameters for all the methods listed in Table 1. Additionally, it would be beneficial to specify the frames per second (fps) achieved by the mask2former and mpvss methods using a single GPU for inference.\n\nQUESTIONS:\nPlease see the weakness\n\nLIMITATIONS:\nPlease see the weakness", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}