# Object-Wise Video Editing
## 1 Introduction
Over the past few decades, several Novel View Synthesis (NVS) methods have been introduced such as view warping [15] and interpolation [16]. An image-based rendering (IBR) by sampling a plenoptic function from a dense matrix of camera view was introduced in References [17; 18; 19]. These methods work well when input views are separated by larger baseline. Novel view synthesis becomes more challenging task with wider baselines due to scale change, 3D rotation and foreshortening effect of the objects. The idea of IBR based on the 3D modeling is introduced in Reference [20] using multi-view stereo. Because of impressive development of 3D reconstruction techniques from multi-view images [21], we can obtain a novel view image based on the target pose of camera as input. Rather than reconstructing dense point, Reference [22] takes advantage of the sparse point obtained from the Structure-from-motion combined with the segmented region where each segment is considered as a planar associated with homography transformation. Reference [23] proposes an improved pipeline of better representation of 3D scene for NVS. In this paper, we propose a novel and robust video editing method under several challenging background situations like complex visual pattern, occlusion, overlaid object clutter, and depth variation observed from a moving camera which is the extension of previous work [24]. First, our method obtains 3D point cloud and camera translation matrix of video frames using a Structure From Motion (SfM) method. Based on the obtained 3D location of the camera for each frame, our method computes a confidence score relying on the normalized difference between the depth (observed distance) and 3D distance to the ray segment in 3D space (predicted distance). The difference between observed depth and prediction distance quantifies the confidence of the potential point. We collect all the points from the truncated epipolar line from neighbor frames and those points are called as potential points in our inpainting step. After collecting all potential points, we select a few highly qualified potential background points based on the confidence score. Then we apply weight and ranking on the selected highly qualified potential points to fill missing regions in video editing. Finally, we correct the noisy pixel by calculating the temporal coherence between inpainted frames. The sample result of our method is shown in Figure 1 compared with the state of the art method [14]. The contributions of our work include (1) a novel confidence score based on the normalized difference between depth and 3D distance to ray segment in 3D space (NDD) enabling simple but effective video editing in 2D image space, (2) a unified framework of optimal corresponding point collection for video editing that addresses occlusion, complex visual pattern, overlaid object clutter and depth variation problems in the background, (3) depth videos are calculated using photo-consistency over epipolar line of neighbour frames, (4) Our new view video synthesis using NDD produces the scene behind the foreground object to see background, (5) outperforming video editing results over state-of-the-art methods on both the public data set and our own video clips with various challenging conditions in novel view video synthesis.