Inter-BMV: Interpolation with Block Motion Vectors for Fast Semantic Segmentation on Video

1 INTRODUCTION . Semantic segmentation , the task of assigning each pixel in an image to a semantic object class , is a problem of long-standing interest in computer vision . Like models for other image recognition tasks ( e.g . classification , detection , instance segmentation ) , semantic segmentation networks have grown drastically in both layer depth and parameter count in recent years , in the race to segment more complex images , from larger , more realistic datasets , at higher accuracy . As a result , state-of-the-art segmentation networks today require between 0.5 to 3.0 seconds to segment a single , high-resolution image ( e.g . 2048× 1024 pixels ) at competitive accuracy ( Zhu et al . ( 2017 ) ; Gadde et al . ( 2017 ) ) . Meanwhile , a new target data format for segmentation has emerged : video . The motivating use cases include both batch applications , where video is segmented in bulk to generate training data for other models ( e.g . autonomous control systems ) , and streaming settings , where high-throughput video segmentation enables interactive analysis of live footage ( e.g . at surveillance sites ) . Video in these contexts consists of long image sequences , shot at high frame rates ( e.g . 30 fps ) in complex environments ( e.g . urban cityscapes ) on modern , high-definition cameras . Segmenting individual frames at high accuracy still calls for the use of competitive image models , but their inference cost precludes their naı̈ve deployment on every frame in a raw multi-hour video stream . A defining characteristic of realistic video is its high level of temporal continuity . Consecutive frames demonstrate significant spatial similarity , which suggests the potential to reuse computation across frames . Building on prior work , we exploit two observations : 1 ) higher-level features evolve more slowly than raw pixel content in video , and 2 ) feature computation tends to be much more expensive than task-specific computation across a range of vision tasks ( e.g . detection , segmentation ) ( Shelhamer et al . ( 2016 ) ; Zhu et al . ( 2017 ) ) . Accordingly , we divide our semantic segmentation model into a deep feature network and a cheap , shallow task network ( Zhu et al . ( 2017 ) ) . We compute features only on designated keyframes , and propagate them to intermediate frames , by warping the feature maps with frame-to-frame motion estimates . The task network is executed on all frames . Nfeat Ntask Ntask Nfeat Ntask Ntask Ntask F F F keyframe k keyframe k+nintermediate frames … … W W W WWW block motion vectors … … Given that feature warping and task computation is much cheaper than feature extraction , a key parameter we aim to optimize is the interval between designated keyframes . Here we make two key contributions . First , noting the high level of data redundancy in video , we successfully utilize an artifact of compressed video , block motion vectors ( BMV ) , to cheaply propagate features from frame to frame . Unlike other motion estimation techniques , which require specialized convolutional networks , block motion vectors are freely available in modern video formats , making for a simple , fast design . Second , we propose a novel feature estimation technique that enables the features for a large fraction of video frames to be inferred accurately and efficiently ( see Fig . 1 ) . In particular , when computing the segmentation for a keyframe , we also precompute the features for the next designated keyframe . Features for all subsequent intermediate frames are then computed as a fusion of features warped forward from the last visited keyframe , and features warped backward from the incoming keyframe . This procedure implements an interpolation of the features of the two closest keyframes . We then combine the two ideas , using block motion vectors to perform the feature warping in feature interpolation . The result is a scheme we call interpolation-BMV . We evaluate our framework on the CamVid and Cityscapes datasets . Our baseline consists of running a competitive segmentation network , DeepLab ( Chen et al . ( 2017 ) ) , on every frame , a setup that achieves published accuracy ( Dai et al . ( 2017 ) ) , and throughput of 3.6 frames per second ( fps ) on CamVid and 1.3 fps on Cityscapes . Our improvements come in two phases . First , our use of motion vectors for feature propagation allow us to cut inference time on intermediate frames by 53 % , compared to approaches based on optical-flow , such as Zhu et al . ( 2017 ) . Second , our bi-directional feature warping and fusion scheme achieves substantial accuracy improvements , especially at high keyframe intervals . Together , the two techniques allow us to operate at over twice the average inference speed as the fastest prior work , at any target level of accuracy . For example , if we are willing to tolerate no worse than 65 mIoU on our CamVid video stream , we are able to operate at a throughput of 20.1 fps , compared to the 8.0 fps achieved by the forward flow-based propagation from Zhu et al . ( 2017 ) . Overall , even when operating in high accuracy regimes ( e.g . within 3 % mIoU of the baseline ) , we are able to accelerate segmentation on video by a factor of 2-6× . 2 RELATED WORK . 2.1 IMAGE SEMANTIC SEGMENTATION . Semantic segmentation is a classical image recognition task in computer vision , originally studied in the context of statistical inference . The approach of choice was to propagate evidence about pixel class assignments through a probabilistic graphical model ( Felzenszwalb & Huttenlocher ( 2004 ) ; Shotton et al . ( 2009 ) ) , a technique that scaled poorly to large images with numerous object classes ( Krähenbühl & Koltun ( 2011 ) ) . In 2014 , Long et al . ( 2015 ) proposed the use of fully convolutional neural networks ( FCNs ) to segment images , demonstrating significant accuracy gains on several key datasets . Subsequent work embraced the FCN architecture , proposing augmentations such as dilated ( atrous ) convolutions ( Yu & Koltun ( 2016 ) ) , post-processing CRFs ( Chen et al . ( 2016 ) ) , and pyramid spatial pooling ( Zhao et al . ( 2017 ) ) to further improve accuracy on large , complex images . 2.2 EFFICIENT VIDEO SEMANTIC SEGMENTATION . The recent rise of applications such as autonomous driving , industrial robotics , and automated video surveillance , where agents must perceive and understand the visual world as it evolves , has triggered substantial interest in the problem of efficient video semantic segmentation . Shelhamer et al . ( 2016 ) and Zhu et al . ( 2017 ) proposed basic feature reuse and optical flow-based feature warping , respectively , to reduce the inference cost of running expensive image segmentation models on video . Recent work explores adaptive feature propagation , partial feature updating , and adaptive keyframe selection as techniques to further optimize the scheduling and execution of optical-flow based warping ( Zhu et al . ( 2018 ) ; Li et al . ( 2018 ) ; Xu et al . ( 2018 ) ) . In general , these techniques fall short in two respects : ( 1 ) optical flow computation remains a computational bottleneck , especially as other network components become cheaper , and ( 2 ) forward feature propagation fails to account for other forms of temporal change , besides spatial displacement , such as new scene content ( e.g . new objects ) , perspective changes ( e.g . camera pans ) , and observer movement ( e.g . in driving footage ) . As a result , full frame features must still be recomputed frequently to maintain accuracy , especially in video footage with complex dynamics , fundamentally limiting the attainable speedup . 2.3 MOTION AND COMPRESSED VIDEO . Wu et al . ( 2018 ) train a network directly on compressed video to improve both accuracy and performance on video action recognition . Zhang et al . ( 2016 ) replace the optical flow network in the classical two-stream architecture ( Simonyan & Zisserman ( 2014 ) ) with a “ motion vector CNN ” , but encounter accuracy challenges , which they address with various transfer learning schemes . Unlike these works , our main focus is not efficient training , nor reducing the physical size of input data to strengthen the underlying signal for video-level tasks , such as action recognition . We instead focus on a class of dense prediction tasks , notably semantic segmentation , that involve high-dimensional output ( e.g . a class prediction for every pixel in an image ) generated on the original uncompressed frames of a video . This means that we must still process each frame in isolation . To the best of our knowledge , we are the first to propose the use of compressed video artifacts to warp deep neural representations , with the goal of drastically improved inference throughput on realistic video . 3 SYSTEM OVERVIEW . 3.1 NETWORK ARCHITECTURE . We follow the common practice of adapting a competitive image classification model ( e.g . ResNet101 ) into a fully convolutional network trained on the semantic segmentation task ( Long et al . ( 2015 ) ; Yu et al . ( 2017 ) ; Chen et al . ( 2017 ) ) . We identify two logical components in our final model : a feature network , which takes as input an image i ∈ R1×3×h×w and outputs a representation fi ∈ R1×A× h 16× w 16 , and a task network , which given the representation , computes class predictions for each pixel in the image , pi ∈ R1×C×h×w . The task network Ntask is built by concatenating three blocks : ( 1 ) a feature projection block , which reduces the feature channel dimensionality to A 2 , ( 2 ) a scoring block , which predicts scores for each of the C segmentation classes , and ( 3 ) an upsampling block , which bilinearly upsamples the score maps to the resolution of the input image . 3.2 BLOCK MOTION VECTORS . MPEG-compressed video consists of two logical components : reference frames , called I-frames , and delta frames , called P-frames . Reference frames are still RGB frames from the video , usually represented as spatially-compressed JPEG images . Delta frames , which introduce temporal compression to video , consist of two subcomponents : block motion vectors and residuals . Block motion vectors , the artifact of interest in our current work , define a correspondence between pixels in the current frame and pixels in the previous frame . They are generated using block motion compensation , a standard procedure in video compression algorithms ( Richardson ( 2008 ) ) : 1 . Divide the current frame into a non-overlapping grid of 16x16 pixel blocks . 2 . For each block in the current frame , determine the “ best matching ” block in the previous frame . A common matching metric is to minimize mean squared error between the blocks . 3 . For each block in the current frame , represent the pixel offset to the best matching block in the previous frame as an ( x , y ) coordinate pair , or motion vector . The resulting grid of ( x , y ) offsets forms the block motion vector map for the current frame . For a 16M × 16N frame , this map has dimensions M ×N . The residuals then consist of the pixel-level difference between the current frame , and the previous frame transformed by the motion vectors .

Summary: This paper advances a method for accelerating semantic segmentation on video content at higher resolutions. Semantic segmentation is typically performed over single images, while there is un-used redundancy between neighbouring frames. The authors propose exploiting this redundancy and leverage block motion vectors from MPEG H.264 video codec which encodes residual content between keyframes. The block motion vectors from H264 are here used to propagate feature maps from keyframes to neighbouring non-keyframe frames (in both temporal directions) avoiding thus an additional full forward pass through the network and integrate this in the training pipeline. Experimental results on CamVid and Cityscapes show that the proposed method gets competitive results while saving computational time.