# Object-Wise Video Editing
## 2 Object Removal and Background Inpainting  
Object removal and background inpainting in video are essential tasks in video editing. Our proposed background inpainting method is inspired by Reference [14]. The workflow of our object removal and background inpainting is shown in Figure 2.  
To inpaint background of each foreground object pixel of a current video frame, a pixel ray (dotted red line in Figure 3) is drawn in 3D space from the camera location which intersects both foreground object and background behind the object. We observes only the foreground object and the intersecting point with background (point \(b\) in Figure 3) is hided behind the foreground object. Background point \(b\) can be observed from some neighboring frames with a condition of a moving camera or moving objects. We truncate a segment of the pixel ray around the background surface which is shown in Figure 3 with a solid red line. As the background surface is unknown in the current frame, obtaining a good ray segment is very important in our method to include background information and avoid foreground information. We prefer the ray segment does not have any intersection with a foreground object that has to be removed. When the ray segment intersects with the foreground, our inpainting results will be corrupted by the foreground information. With other conditions, if the ray segment does not have any intersection with the background, our method fails to gather background information. Therefore, initially, we put the range of ray segments from a certain distance behind the foreground object to be removed to the farthest distance of the scene which makes sure of intersecting background. Figure 1: Our video inpainting result compared to state of the art [14] method. More comparison results are demonstrated in Figures 13–15. Even though the initial ray segment is long, we optimize the range of the ray segment with the calculated normalized difference of depth-3D distance (NDD) in Section 2.2.1. We project the pixel ray shown in Figure 3 to 2D image plane using a transformation matrix calculated from SfM step and 2D plane observe a line called epipolar line. Each frame obtain a truncated epipolar line after projecting the ray segment to the neighbouring frame. Each 2D point on the truncated epipolar line is considered as a potential point for the background information extraction for the current frame. The background intersecting point exist on the truncated epipolar line unless the point is occluded by the foreground object which is shown in Figure 3.  
Our method tries to find highly qualified points among the potential points on the truncated epipolar line. For each \(i_{th}\) frame from input video \(F_{i}\), the binary mask \(M_{i}\) is given to removed the object region from all frames from the input video. If the video of the mask was not given, we create the mask \(M_{i}\) of the object from the video using method [25] where only 1 st frame annotate mask is considered as an input. Point \(p\) represents the pixel of the foreground region that has to be removed which is decided from mask \(M_{i}\). Since the binary mask \(M_{i}\) does not have to be so precise, we dilate the calculated mask \(M_{i}\) to ensure every pixel of the foreground object should be considered for background inpaintng in our method. \(C_{rgb}(p)\) (\(C_{rgb}(p)\in\mathbb{R}^{3}\)) represents the color that is used to fill the point \(p\) after the foreground object removed. The main goal of our method is to estimate \(C_{rgb}(p)\) from the observation of neighboring frames. A potential set of points \(S(p)\) are used to calculate the \(C_{rgb}(p)\) for point \(p\) in current frame where \(S(p)=\{s|\) points of all truncated epipolar line projected from pixel ray of point p\(\}\). \(C_{rgb}(s)\) represents color information of a  
Figure 3: Ray segment in 3D space intersect with background surface and corresponding truncated epipolar lines for potential background point collection from 2D reference frames of input video.