# Object-Wise Video Editing
## 2 Object Removal and Background Inpainting
### Potential Point Set Collection
#### 2.2.1 Normalized Difference of Depth-3D Distance  
After collecting all potential points set \(S(p)\) from all truncated epipolar lines, we measure the confidence of each potential point by calculating the normalized difference between depth and 3D distance to the ray segment in 3D space (NDD). As the ray segment in 3D space is a parameterized line and the location of the current frame and neighboring frames are known, 3-dimensional Euclidean distance to the ray segment can simply calculate between two 3D points. A difference of depth-3D distance quantifies the discrepancy between the predicted distance (assuming that the current observed point is from the 3d position where background surface intersects ray segment) and depth (observed distance). In order to find the missing point behind the foreground object correctly, we have to know the corresponding position \(b\) of the background shown in Figure 7a which is the intersecting point between the ray segment and background surface. The depth-3D distance becomes zero at the position \(b\) where the ray segment lies as the red line and background are intersect each other. In other words, if any potential point \(s\) is a correct observation of point \(b\) from the neighboring frame, the corresponding depth of point \(s\) and 3-D Euclidean distance between 3D point \(b\) and camera should be identical. However, the depth-3D distance of the same observed background from different view changes along with viewpoint changes shown on the right side of Figure 7a. So only depth-3D distance provides different confidence scores for the same point on background in neighboring frames with a different view. Therefore, we consider the normalized difference of depth-3D distance (NDD) as confidence scores shown in Figure 7b as a blue line and the calculation. NDD is the 3D distance along the perpendicular direction from the observed background to the ray segment. NDD still provides a linear measurement of the discrepancy between the depth (observed distance) and 3D distance (predicted distance) assigning consistent confidence scores for identical points. Figure 8a and Equation (5) demonstrate how NDD is really calculated in method [24]. \[NDD(s)=abs(\gamma\times\frac{\alpha}{\beta})=abs(\gamma\times\frac{\delta(s)- \beta}{\beta})\]  
\[=abs(\gamma\times(\frac{\delta(s)\times(q1+q2)}{d1\times q2+d2\times q1}-1)), \tag{5}\]  
where \(\delta(s)=\alpha+\beta\) is depth of potential point \(s\) and \(\beta\approx\frac{d1\times q2+d2\times q1}{q1+q2}\). \(\beta\) is approximated from the ratio between \(q1\) and \(q2\) (image space distance) and \(d1\) and \(d1\) (3d distance in camera space) as illustrated in Figure 8a. \(\gamma\) is 3d distance from \(s\) to light ray of current frame along the perpendicular line. However, in this method, the approximate NND distance is calculated from the ratio between \(q1\) and \(q2\) image space distance which is not compatible with the distance ratio in the 3D space. Instead of approximating the NDD distance from 2D space from Reference [24], we calculate NDD in 3D space which is illustrated in Figure 8b make our method more robust video processing having a different scale of depth in the scene. \(\beta\) is calculate from the triangle created from two camera position and the point of intersection \(o\) between pixel ray of point \(p\) and pixel ray of point \(s\) (potential points for pixel \(p\) from truncated epipolar line from neighbouring frames). In the Figure 8b \(c_{1}\)\(c_{2}\) are two camera position in 3D space and the point \(o\) is the intersection between two pixel ray. So from given two line equation and two given camera position, we calculate \(\beta=\frac{\sin(B)}{\sin(180-(A+B))}\times\gamma\). Finally the Equation (5) becomes  
\[NDD(s)=abs(\gamma\times(\frac{\delta(s)\times\sin(180-(A+B))}{\gamma\times\sin (B)}-1)). \tag{6}\]  
Here, \(A\) and \(B\) is the angle made by the triangle shown in Figure 8b at the camera position \(c_{1}\) and \(c_{2}\) respectively.