# Object-Wise Video Editing
## 2 Object Removal and Background Inpainting
### Prepossessing
#### 2.1.2 Depth Extraction  
The depth map of each frame is obtained by projecting the 3D points onto the 2-dimensional image space of each frame. The depth value at each pixel is calculated by interpolating the depth of projected 3D points falling within a patch of the fixed size described in Reference [24]. We simply perform a depth calculation for the points from overlaid multiple surfaces by selecting the closest group of points within the patch. If we do not have a large enough number of 3D points within the patch, we increase the patch size making a smoother depth map. In this case, sparse 3D points cause a big hole in the depth map and larger patch size causes poor depth near the edge of the object that has a noticeable difference in depth. On the other hand, the wrong 3D points cause noisy depth. In order to find the depth \(\delta(x)\) of each pixel \(x\), we draw a pixel ray that intersects the scene at a certain depth. The similar color observation of certain depth on the pixel ray from different given views can be potential depth \(\delta(x)\) of the output pixel of current frames \(F_{i}\). The ray that would be measured by the pixel denoted \(x\) in Figure 5 is marked by the red line passing through that pixel. To find the depth along the red line, we have to locate the position of the intersecting point between the red line and the scene on the basis of similar observation with pixel \(x\) from neighboring frames of the input video. In this context, the ray segment should cover the whole range of the scene to collect both the foreground or background information. We collect all observations of each point in the red line from different neighboring frames, which is shown on the right side of Figure 5. Along the red line, the highest photo-consistency observed from neighboring frames can be a potential location to measure \(\delta(x)\) of pixel \(x\). However, it is a challenging task to find the depth location along the red line due to the presence of uniform area and occlusion in the scene when all neighbor frames are used to calculate the score of photo-consistency. To avoid any occlusion, we divide the neighbouring frames into two groups (Left side frames and Right side frames) based on the location of the current frame. In Figure 6,  
Figure 5: Calculate the depth of pixel \(x\) from the similar observation from neighbouring frames along the epipolar line. Top right of the figure showing the color observed at a range of depth along the pixel ray across the neighbouring frames where depth is along the vertical axis and input frame along the horizontal axis. The minimum scores along the depth represent highest photo-consistency which represent the depth location for pixel \(x\). the pixel \(x\) can be occluded either from the left-side or right-side neighbouring frames of input video from the current frame. In Figure 6, we carefully select four pixel \(x_{1}\), \(x_{2}\), \(x_{3}\) and \(x_{4}\) around the edge of an foreground object where \(x_{2}\) and \(x_{4}\) are on the foreground object and \(x_{1}\) and \(x_{3}\) are on the background of the scene. Pixel \(x_{1}\) and \(x_{3}\) is occluded from right and left side frames respectively from the current frame. Considering this fact, we calculate the photo-consistency score \(L_{x}(z)\) from the left frames and \(R_{x}(z)\) from the right frames shown in Equations (2) and (3) along the direction of depth \(z\). We notice that, the highest photo-consistency is found for point \(x_{1}\) from \(L_{x}(z)\) and point \(x_{3}\) from \(R_{x}(z)\) along the depth \(z\). The right and left side neighbouring frames are selected based on the 3D location of the neighbouring camera from current frames camera position. The pixel \(x_{2}\) and \(x_{4}\) can be observed from both left and right side frame and show the highest photo-consistency at same depth in \(L_{x}(z)\) and \(R_{x}(z)\) scores shown in Figure 6. To calculate \(\delta(x)\), we use minimum scores from both \(L_{x}(z)\) and \(R_{x}(z)\) along the depth \(z\) which is shown in Equation (4).