A NOVEL NEURAL NETWORK BASED MOTION ESTIMATION AND COMPENSATION BASED VIDEO ENHANCEMENT

Motion estimation (ME) and motion compensation (MC) have been widely used for classical video frame interpolation systems over the past decades. Recently, a number of data-driven frame interpolation methods based on convolutional neural networks have been proposed. However, existing learning based methods typically estimate either flow or compensation kernels, thereby limiting performance on both computational efficiency and interpolation accuracy. In this work, we propose a motion estimation and compensation driven neural network for video frame interpolation. A novel adaptive warping layer is developed to integrate both optical flow and interpolation kernels to synthesize target frame pixels. This layer is fully differentiable such that both the flow and kernel estimation networks can be optimized jointly. The proposed model benefits from the advantages of motion estimation and compensation methods without using hand-crafted features. Compared to existing methods, our approach is computationally efficient and able to generate more visually appealing results. Furthermore, the proposed MEMC-Net architecture can be seamlessly adapted to several video enhancement tasks, e.g., superresolution, denoising, and deblocking. Extensive quantitative and qualitative evaluations demonstrate that the proposed method performs favorably against the state-of-the-art video frame interpolation and enhancement algorithms on a wide range of datasets. INTRODUCTION Video frame interpolation aims to synthesize nonexistent frames between original input frames, which has been applied to numerous applications such as video frame rate conversion [1], novel view synthesis [2], and frame recovery in video streaming [3], to name a few. Conventional approaches [4], [5] are generally based on motion estimation and motion compensation (MEMC), and have been widely used in various display devices [6]. A few deep learning based frame interpolation approaches [7], [8] have been developed to address this classical topic. In this paper, we analyze the MEMC-based and learning-based approaches of video frame interpolation and exploit the merits of both paradigms to propose a high-quality frame interpolation processing algorithm. Conventional MEMC-based approaches entail both motion estimation [13] and motion compensation [14] for video interpolation. Motion estimation is used to determine the block-wise or pixel-wise motion vectors between two frames. The block-based methods [4] assume that the pixels within a block share the same motion and use search strategies [15], [16] and selection criteria [4], [17] to obtain the optimal motion vector. On the other hand, the methods based on pixelbased motion, i.e., optical flow, estimate a motion/ flow vector for each pixel of the frames and thus entail heavy computational loads. The recent years have witnessed Motion compensation (MC) is very useful in video filtering to remove noise and enhance signal. It is useful since it allows the filter or coder to process through the video on a path of near-maximum correlation based on following motion trajectories across the frames making up the image sequence or video. Motion compensation is also employed in all distribution-quality video coding formats, since it is able to achieve the smallest prediction error, which is then easier to code. Motion can be characterized in terms of either a velocity vector v or displacement vector d and is used to warp a reference frame onto a target frame. Motion estimation is used to obtain these displacements, one for each pixel in the target frame. Several methods of motion estimation are commonly used: • Block matching • Hierarchical block matching • Pel-recursive motion estimation • Direct optical flow methods • Mesh-matching methods Optical flow is the apparent displacement vector field d D . d1,d2/ we get from setting (i.e., forcing) equality in the so-called constraint equation All five approaches start from this basic equation, which is really just an idealization. Departures from the ideal are caused by the covering and uncovering of objects in the viewed scene, lighting variation both in time and across the objects in the scene, movement toward or away from the camera, as well as rotation about an axis (i.e., 3-D motion). Often the constraint equation is only solved approximately in the least-squares sense. Also, the displacement is not expected to be an integer as assumed in (11.2–1), often necessitating some type of interpolation to be used. Science, Technology and Development Volume IX Issue X OCTOBER 2020 ISSN : 0950-0707