SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuetian Weng $^{1,2*}$  Mingfei Han $^{3,4}$  Haoyu He $^{1}$  Mingjie Li $^{3}$
Lina Yao $^{4}$  Xiaojun Chang $^{3,5}$  Bohan Zhuang $^{1\dagger}$
<sup>1</sup>ZIP Lab, Monash University <sup>2</sup>Baidu Inc. <sup>3</sup>ReLU, AAII, UTS
$^{4}$ Data61, CSIRO  $^{5}$ Mohamed bin Zayed University of AI
Video Semantic Segmentation (VSS) involves assigning a semantic label to each pixel in a video sequence. Prior work in this field has demonstrated promising results by extending image semantic segmentation models to exploit temporal relationships across video frames; however, these approaches often incur significant computational costs. In this paper, we propose an efficient mask propagation framework for VSS, called MPVSS. Our approach first employs a strong query-based image segmentor on sparse key frames to generate accurate binary masks and class predictions. We then design a flow estimation module utilizing the learned queries to generate a set of segment-aware flow maps, each associated with a mask prediction from the key frame. Finally, the mask-flow pairs are warped to serve as the mask predictions for the non-key frames. By reusing predictions from key frames, we circumvent the need to process a large volume of video frames individually with resource-intensive segmentors, alleviating temporal redundancy and significantly reducing computational costs. Extensive experiments on VSPW and Cityscapes demonstrate that our mask propagation framework achieves SOTA accuracy and efficiency trade-offs. For instance, our best model with Swin-L backbone outperforms the SOTA MRCFA using MiT-B5 by  $4.0\%$  mIoU, requiring only  $26\%$  FLOPs on the VSPW dataset. Moreover, our framework reduces up to  $4\times$  FLOPs compared to the per-frame Mask2Former baseline with only up to  $2\%$  mIoU degradation on the Cityscapes validation set. Code is available at https://github.com/ziplab/MPVSS.
Video Semantic Segmentation (VSS), a fundamental task in computer vision, seeks to assign a semantic category label to each pixel in a video sequence. Previous research on VSS has leveraged developments in image semantic segmentation models, e.g., FCN [40] and Deeplab [82, 4], which made tremendous progress in the field. However, adapting image semantic segmentation models to VSS remains challenging. On one hand, sophisticated temporal modeling is required to capture the intricate dynamics among video frames. On the other hand, videos contain a significantly larger volume of data compared to images. Processing every frame with a strong image segmentor can incur significant computational costs.
Prior work in VSS mainly focuses on leveraging both temporal and spatial information to enhance the accuracy of pixel-wise labeling on each video frame, building upon the pixel-wise classification paradigm of traditional image semantic segmentors. Specifically, these methods exploit temporal
*Work done during an internship at Baidu Inc.
†Corresponding author. Email: bohan.zhuang@gmail.com
Figure 1: Motivation of the proposed MPVSS. Three consecutive frames with ground truth category labels are sampled from a video in VSPW [42], illustrating a strong correlation between video frames along the temporal dimension. This observation underscores the significant temporal redundancy present in videos. The remaining columns contrast the proposed query-based flow and traditional optical flow, each showing a normalized flow map, a mask prediction, and the final predicted semantic map, respectively.
information by integrating visual features from previous frames into the target frame, employing techniques such as recurrent units [26, 43], temporal shift [33, 1], or spatial-temporal attention modules [19, 44, 28, 16, 53], among others.
More recently, with the prevalence of the DETR [2] framework, DETR-like segmentors [8, 7, 51] have dominated the latest advances and achieved state-of-the-art performance in semantic segmentation. Notably, MaskFormer series [8, 7] learn a set of queries representing target segments, and employ bipartite matching with ground truth segments as the training objective. Each query is learned to predict a binary mask and its associated class prediction. The final result is obtained by combining the binary masks and class predictions of all queries via matrix multiplication. However, these models require computing a high-resolution per-pixel embedding for each video frame using a powerful image encoder, leading to significant computational costs. For instance, processing an input video clip with  $30 \times 1024 \times 2048$  RGB frames using the strong Mask2Former [7] requires more than 20.46T Floating-point Operations (FLOPs) in total. Such computational demands render it impractical to deploy these models in real-world VSS systems.
A typical solution is to incorporate temporal information guided by optical flow [50, 82, 22, 66] to reduce the redundancy. The related literature proposes to propagate feature maps from the key frame to other non-key frames using optical flow. The motivation mainly comes from two aspects. First, videos exhibit high temporal redundancy due to the similar appearance of objects or scenes in consecutive frames [13], especially in videos with a high frame rate. As shown in Fig. 1, we empirically observe that semantic information in consecutive video frames is highly correlated. Additionally, previous methods [50, 82, 22, 30, 25, 45] highlight the gradual changes in semantic features within deep neural networks. They attempt to reduce the computational costs by reusing feature maps from preceding frames for the target frame, facilitated by the estimated optical flow between the neighboring frames. However, these methods may still suffer from degraded performance due to inaccurate and noisy optical flow estimation, which only measures pixel-to-pixel correspondence between adjacent frames. Recent advancements in Transformer-based optical flow methods [58, 20, 63, 64] demonstrate the importance of incorporating global correspondence information for each pixel, resulting in more precise per-pixel optical flow estimation. Nevertheless, these methods still focus solely on pixel-level correspondence and do not account for segment-level information among video frames, as illustrated in Fig. 1, which may be insufficient for the task of VSS.
In this paper, we present an efficient mask propagation framework for VSS, namely MPVSS, as shown in Fig. 2. This framework relies on two key components: a strong query-based image segmentor for the key frame (upper part of Fig. 2), and a powerful query-based flow estimator for the nonkey frames (lower part of Fig. 2). Specifically, we employ the state-of-the-art query-based image segmentation model, i.e., Mask2Former [7] to process sparse key frames and generate accurate binary masks and class predictions. We then propose to estimate individual flow map corresponding to each segment-level mask prediction of the key frame. To achieve this, we leverage the learned queries from the key frame to aggregate segment-level motion information between adjacent frame pairs and capture segment-specific movements along the temporal dimension. These queries are fed to a flow head to predict a set of segment-aware flow maps, followed by a refinement step. The mask
predictions from key frames are subsequently warped to other non-key frames through the generated query-based flow maps, where each flow map is tailored to the corresponding mask predicted by the associated query in the key frame. Finally, we derive semantic maps for the non-key frames by aggregating the warped mask predictions with the class probabilities predicted from the key frame. By warping the predicted segmentation masks from key frames, our model avoids processing each individual video frame using computationally intensive semantic segmentation models. This not only reduces temporal redundancy but also significantly lowers the computational costs involved.
In summary, our main contributions are in three folds:
- We propose MPVSS, a novel mask propagation framework for efficient VSS, which reduces computational costs and redundancy by propagating accurate mask predictions from key frames to non-key frames.
- We devise a novel query-based flow estimation module that leverages the learned queries from the key frame to model motion cues for each segment-level mask prediction, yielding a collection of accurate flow maps. These flow maps are utilized to propagate mask predictions from the key frame to other non-key frames.
- Extensive experiments on standard benchmarks, VSPW [42] and Cityscapes [9], demonstrate that our MPVSS achieves SOTA accuracy and efficiency trade-offs.
Image semantic segmentation. As a fundamental task in computer vision, image semantic segmentation seeks to assign a semantic label to each pixel in an image. The pioneering work of Fully Convolutional Networks (FCNs) [40] first adopts fully convolutional networks to perform pixel-wise classification in an end-to-end manner and apply a classification loss to each output pixel, which naturally groups pixels in an image into regions of different categories. Building upon the per-pixel classification formulation, various segmentation methods have been proposed in the past few years. Some works aim at learning representative features, which propose to use atrous convolutional layers to enlarge the receptive field [68, 4, 5, 46, 67], aggregate contextual information from multi-scale feature maps via a pyramid architecture [17, 36], encoder-decoder architecture [49, 6], or utilize attention modules to capture the global dependencies [27, 76, 14, 79, 78, 61]. Another line of work focuses on introducing boundary information to improve prediction accuracy for details [10, 29, 70, 77]. More recent methods demonstrate the effectiveness of Transformer-based architectures for semantic segmentation. SegFormer [62], Segmentor [51], SETR [78], MaskFormer [8, 7] replace traditional convolutional backbones [18] with Vision Transformers [11, 59, 39, 34] and/or implement the head with Transformers following the DETR-like framework [2]. Basically, we propose an efficient framework for video semantic segmentation that leverages the mask predictions generated by DETR-like image segmentors. Our framework focuses on propagating these mask predictions across video frames, enabling accurate and consistent segmentation results throughout the entire video sequence. By incorporating the strengths of DETR-like models in image segmentation, we achieve high-quality and efficient video semantic segmentation.
Video semantic segmentation. Video semantic segmentation has recently captured the attention of researchers due to its dynamic nature and practical significance in real-world scenarios. Compared with image semantic segmentation, most existing methods of VSS pay attention to exploiting temporal information, which can be divided into two groups. The first group of approaches exploits temporal relationships to improve prediction accuracy and consistency. For example, some works utilize recurrent units [26, 43] or design an attention propagation module [19] to warp the features or results of several reference frames [81, 15], or aggregate neighbouring frames based on optical flow estimation [15, 37]. In particular, ETC [38] proposes temporal consistency regularization during training, [73] instead exploits perceptual consistency in videos, MRCFA [54] mines relationship of cross-frame affinities, STT [28] employs spatial-temporal transformer, CFFM-VSS [53] disentangles static and dynamic context for video frames. The second line of methods infers to alleviate the huge temporal redundancy among video frames and thus reduce the computational cost. Observing the high similarity of the semantic representation of deep layers, ClockNet [50] designs a pipeline schedule to adaptively reuse feature maps from previous frames in certain network layers. DFF [82] utilizes optical flow to propagate feature maps from key frames to non-key frames, achieving remarkable computational efficiency. Accel [22] further proposes to execute a large model on key frames and
Figure 2: Overall architecture of the proposed MPVSS.
a compact one on non-key frames, while DVN [66] proposes a region-based execution scheme on non-key frames, as well as an dynamic key-frame scheduling policy. LLVS [30] develops an efficient framework involving both adaptive key frame selection and selective feature propagation. Unlike methods that primarily rely on optical flow, which models dense pixel-to-pixel correspondence, we introduce a novel query-based flow module that focuses on segment-level flow fields between adjacent frames. By operating at the segment level, our method captures higher-level motion information that is more accurate for VSS.
Motion estimation by optical flow. Optical flow estimation is a fundamental module in video analysis, which estimates a dense field of pixel displacement between adjacent frames. In deep learning era, FlowNet [12] first utilizes convolutional neural networks to learn from labeled data. Most follow-up methods propose to employ spatial pyramid networks [52], or utilize recurrent layers [58] to apply iterative refinement for the predicted flow in a coarse-to-fine manner [52, 47, 24, 55]. More recent methods pioneer Transformers to capture the global dependencies of cost volume [58, 20], or extract representative feature maps for global matching [63], addressing the challenge of large displacement. As optical flow represents the pixel-to-pixel correspondence in adjacent frames, it is often utilized in a wide range of video analysis tasks, i.e., optical flow as motion information for modeling action motion in the tasks of action recognition [3, 56], temporal action detection [57, 32], or improving the feature representation in video segmentation [74], object detection [81]; optical flow estimation as a simultaneous task to model pixel correspondence for video interpolation [23, 48, 21], video inpainting [71, 72, 31], etc.
3 Mask Propagation via Query-based Flow
In Sec. 3.1, we introduce the overall pipeline of the proposed MPVSS. In Sec. 3.2, we simply revisit Mask2Former that MPVSS is built upon. Finally, in Sec. 3.3, we introduce our query-based flow estimation method, which yields a set of flow maps for efficiently propagating mask predictions from key frames to other non-key frames.
The overall architecture is shown in Fig. 2. Let  $\{\mathbf{I}^t\in \mathbb{R}^{H\times W\times 3}\}_{t = 1}^T$  be an input video clip with  $T$  frames, where  $H$  and  $W$  denote the height and width of frames in the video. Without loss of generality, we select key frames from the video clip at fixed intervals, e.g., every 5 frames, considering other frames within these intervals as non-key frames. For simplicity, we denote each key frame as  $\mathbf{I}^k$  and non-key frame as  $\mathbf{I}^j$ . To reduce redundant computation, we only run the heavy image segmentation model on the sampled key frames and propagate the predictions from key frames to non-key frames.
Specifically, for a key frame  $\mathbf{I}^k$ , we adopt Mask2Former [7] (upper part of Fig. 2) to generate  $N$  mask predictions  $\mathbf{M}^k$  and class predictions. Then, we use the flow estimation module (bottom part
of Fig. 2) to predict a set of flow maps  $\mathcal{F}^{j\rightarrow k}\in \mathbb{R}^{N\times 2\times H\times W}$  indicating the motion changes from  $\mathbf{I}^j$  to  $\mathbf{I}^k$ , which will be discussed in Sec. 3.3. Subsequently, we apply the bilinear warping function on all locations to propagate per-segment mask predictions  $\mathbf{M}^k$  to obtain  $\mathbf{M}^j$  of non-key frames, i.e.,  $\mathbf{M}^j = \mathcal{W}(\mathbf{M}^k,\mathcal{F}^{j\rightarrow k})$ . Consequently, we segment the non-key frame  $\mathbf{I}^j$  by propagating the predicted masks from its preceding key frame  $\mathbf{I}^k$ , avoiding the necessity of computationally intensive image segmentation models for every video frame.
During training, we apply the loss function in Mask2Former [7] on the warped mask predictions  $\mathbf{M}^j$  and perform bipartite matching with the ground truth masks for the current frame. The training approach is similar to the process described in [82], where the loss gradients are back-propagated throughout the model to update the proposed flow module.
3.2 Mask2Former for Segmenting Key Frames
Mask2Former consists of three components: a backbone, a pixel decoder and a transformer decoder, as depicted in the upper part of Fig. 2. The backbone extracts feature maps from the key frame  $\mathbf{I}^k\in \mathbb{R}^{H\times W\times 3}$ . The pixel decoder builds a multi-scale feature pyramid as well as generates a high-resolution per-pixel embedding, following FPN [35] and Deformable DETR [80]. In transformer decoder, the target segments are represented as a set of learned queries  $\mathcal{Q}_\mathcal{O}^k\in \mathbb{R}^{N\times C}$  where  $C$  is the channel dimension. After gradually enriching their representations with stacked decoder blocks,  $\mathcal{Q}_\mathcal{O}^k$  is fed into a class and a mask head to yield  $N$  class probabilities and  $N$  mask embeddings, respectively. After decoding the binary mask predictions  $\mathbf{M}^k\in \mathbb{R}^{N\times H\times W}$  with the class predictions and the per-pixel embeddings, we can finally derive the semantic maps for the key frame by aggregating  $N$  binary mask predictions with their corresponding predicted class probabilities via matrix multiplication. We refer readers to [8, 7] for more details.
3.3 Query-based Flow Estimation
To generate the segmentation masks for the non-key frames efficiently, we propose to estimate a set of flow maps, each corresponding to a mask prediction of the key frame. We start with introducing the process of generating  $N$  query-based flow maps between the current non-key frame  $\mathbf{I}^j$  and its preceding key frame  $\mathbf{I}^k$ , where  $0 < j - k \leq T$ . The overall flow module comprises three key elements: a motion encoder to encode the motion feature pyramid between the pair of video frames  $\{\mathbf{I}^k, \mathbf{I}^j\}$ , a motion decoder that leverages the learned queries  $\mathcal{Q}_{\mathcal{O}}^k$  from the key frame to extract motion information from the motion feature maps for each segment, and a flow head responsible for predicting  $N$  query-based flow maps  $\mathcal{F}^{j \to k}$ , where each flow map is associated with warping a segment (mask prediction) of the key frame to the current non-key frame.
Motion encoder. To obtain motion features between the paired frames  $\{\mathbf{I}^k,\mathbf{I}^j\}$ , we concatenate the two frames along the channel dimension as the input of the motion encoder. The motion encoder, utilizing the same architecture as the flow encoder in FlowNet [12], is employed to extract motion features at each location in a downsampling manner. Following [12, 7], we reverse the order of the original feature maps, generating a motion feature pyramid  $\{\mathbf{b}_1,\mathbf{b}_2,\mathbf{b}_3,\mathbf{b}_4\}$ , each at resolutions  $1/32, 1/16, 1/8$  and  $1/4$  of the original video frame, respectively. Subsequently, the feature pyramid is fed to the model decoder for the decoding of flow maps from the lowest to the highest resolution.
Motion decoder. We then devise a motion decoder to aggregate motion information from the motion feature pyramid using  $N$  flow queries  $\mathcal{Q}_{\mathcal{F}}$ .
First, the motion decoder takes as input the first three feature maps  $\{\mathbf{b}_1,\mathbf{b}_2,\mathbf{b}_3\}$  from the motion feature pyramid and the flow queries  $\mathcal{Q}_{\mathcal{F}}$ . We initialize flow queries with the  $N$  learned queries  $\mathcal{Q}_{\mathcal{O}}^k$  of the preceding key frame, i.e.,  $\mathcal{Q}_{\mathcal{F}}^{1} = \mathcal{Q}_{\mathcal{O}}^{k}$ , where  $\mathcal{Q}_{\mathcal{F}}^{1}$  indicates the input flow queries to the motion decoder. Following [7], feature map in each layer is projected into  $C$  channels in the channel dimension using a linear layer, which is then added with a sinusoidal positional embedding and a learnable level embedding. This results in three scales of motion feature maps in total, i.e.,  $\{\mathbf{B}_1^1,\mathbf{B}_2^1,\mathbf{B}_3^1\}$ , as the input of the following layers.
Next, the motion decoder can be divided into  $S$  stages, each indexed by  $s$ , with each stage involving  $L$  blocks, each indexed by  $l$ . We adopt  $S = 3$  and  $L = 3$  for our model. Inspired by [7, 65], starting from  $s = 1$  and  $l = 1$ , for each block in each stage, we first concatenate  $\mathbf{B}_l^s$  and  $\mathcal{Q}_{\mathcal{F}}^s$  together and then feed them into a number of Transformer layers, each of which performs information propagation
between  $\mathbf{B}_l^s$  and  $\mathcal{Q}_{\mathcal{F}}^s$
$$
\mathbf {Z} _ {l} ^ {s} = \left[ \mathcal {Q} _ {\mathcal {F}} ^ {s}; \mathbf {B} _ {l} ^ {s} \right], \tag {1}
$$
$$
\hat {\mathbf {Z}} _ {l} ^ {s} = \operatorname {L N} \left(\operatorname {M S A} \left(\mathbf {Z} _ {l} ^ {s}\right) + \mathbf {Z} _ {l} ^ {s}\right), \tag {2}
$$
$$
\hat {\mathcal {Q}} _ {\mathcal {F}} ^ {s}, \hat {\mathbf {B}} _ {l} ^ {s} = \mathrm {L N} \left(\operatorname {F F N} \left(\hat {\mathbf {Z}} _ {l} ^ {s}\right) + \hat {\mathbf {Z}} _ {l} ^ {s}\right), \tag {3}
$$
where  $[\cdot ]$  denotes the concatenation operation.  $\mathrm{LN}(\cdot)$  , MSA(·) and  $\mathrm{FFN}(\cdot)$  denote LayerNorm layer, Mutli-head Self-Attention and Feed-forward Neural Network, respectively. Therefore, in each stage, we learn flow queries and update motion feature maps via self-attention. The self-attention mechanism aggregates motion information globally from both the motion feature maps and the flow query features simultaneously. Finally, the learned flow queries  $\hat{\mathcal{Q}}_{\mathcal{F}}$  and updated motion feature maps  $\{\hat{\mathbf{B}}_1,\hat{\mathbf{B}}_2,\hat{\mathbf{B}}_3\}$  are fed to the flow head for flow prediction.
As each query of  $\mathcal{Q}_{\mathcal{O}}^{k}$  encodes global information for each segment in the key frame, using  $\mathcal{Q}_{\mathcal{O}}^{k}$  as initial flow queries enables each flow query to extract motion information for each segment present in the key frame, which facilitates query-based motion modeling.
Flow head. The flow head generate the final flow maps  $\mathcal{F}^{j\rightarrow k}\in \mathbb{R}^{N\times 2\times H\times W}$  by combining query-based flow  $\mathcal{F}^{QF}$  and pixel-wise flow  $\mathcal{F}^{PF}$ .
To obtain the query-based flow  $\mathcal{F}^{QF}$ , an MLP with 2 hidden layers is employed to convert the learnt flow queries  $\hat{\mathcal{Q}}_{\mathcal{F}}$  to flow embeddings  $\mathcal{E}_{\mathcal{F}} \in \mathbb{R}^{N \times C}$ , each with  $C$  channels. Inspired by the pixel decoder design in Mask2Former, we project the motion feature map  $\mathbf{b}_4$  from the motion encoder to  $\mathcal{E}_B \in \mathbb{R}^{2C \times \frac{H}{4} \times \frac{W}{4}}$  using another MLP with 2 hidden layers. Let  $\mathcal{E}_B^x$  and  $\mathcal{E}_B^y$  denote the first and last  $C$  channels of  $\mathcal{E}_B$ , and they are used to obtain the query-based flow predictions in terms of  $x$  and  $y$  direction, respectively. Then, for the  $n^{th}$  query, we get the flow map via a dot product between the  $n^{th}$  flow embedding  $\mathcal{E}_{\mathcal{F}_n}$  and the projected motion map  $\mathcal{E}_B$ , i.e.,  $\mathcal{F}_n^{QF}(x) = \mathcal{E}_{\mathcal{F}_n} \cdot \mathcal{E}_B^x$  and  $\mathcal{F}_n^{QF}(y) = \mathcal{E}_{\mathcal{F}_n} \cdot \mathcal{E}_B^y$  in terms of  $x$  and  $y$  direction, respectively. Then we can obtain the predicted query-based flow  $\mathcal{F}_n^{QF} = [\mathcal{F}_n^{QF}(x); \mathcal{F}_n^{QF}(y)] \in \mathbb{R}^{2 \times \frac{H}{4} \times \frac{W}{4}}$  for the  $n^{th}$  query. The dot product of the flow embeddings and projected motion maps quantifies the similarity between the segment-level motion vector and the encoded motion information at each location on the feature maps, which reflects the movement for each query on the spatial dimension.