# Multi-View Visual Relationship Detection with Estimated Depth Map
## 3 Methodology
### Multi-View Features Generator  
As we already obtain the candidate object bounding boxes in an RGB image and corresponding depth values in an estimated depth map, the locations in 3D visual space can be represented. As illustrated in Figure 3, subject A and object B are the detected object candidates; the front view image represents the view of the original monocular RGB image. To simplify the visual feature extraction, \(\triangle w\) and \(\triangle h\) are calculated by centers of the bounding boxes of A and B. Moreover, \(\triangle\)Depth is the difference in depth values between A and B. The depth value of a detected object can be represented as:  
\[\text{Depth}_{n}=\frac{\sum_{i=1}^{w_{n}}\sum_{j=1}^{h_{n}}\text{ GrayValue}(i,j)}{w_{n}h_{n}} \tag{1}\]  
where \(w_{n}\) and \(h_{n}\) are the width and height of the \(n\)th detected object bounding box, and GrayValue\((i,j)\) is the gray value of the point at the \(i\)th column and \(j\)th row in the corresponding depth map. In front view, the multi-view spatial relationship between A and B can be represented as (\(\triangle w,\triangle h,\triangle\)Depth). Thus, in left view, the multi-view spatial relationship is (\(\triangle\)Depth, \(\triangle h,\triangle w\)). Note that, intuitively, \(\triangle w\) represents the \"depth difference\" in left view. Similarly, in top view, the multi-view spatial relationship is (\(\triangle w,\triangle\)Depth, \(\triangle h\)), where \(\triangle h\) represents the \"depth difference\" in top view. Because multi-view images are generated based on only one 3D coordinate system, \(\triangle w\), \(\triangle h\) and \(\triangle\)Depth have the same values, respectively, in different views. Based on the multi-view spatial relationships in three kinds of view images, intuitively we can utilize a fuzzy strategy to map the depth difference between detected objects in multi-view images into threefold feature vectors (and each vector is 3-dimensional). For every multi-view image, the third term of the corresponding multi-view spatial relationship can be regarded as the depth difference between objects. Thus, we can process the last term of the multi-view spatial relationship by fuzzy membership function as follows:  
\[S_{DepthX}(X\in\{F,L,T\})= \begin{cases}(1,0,0)&\triangle D_{X}<-b\\ \left(-\frac{\triangle D_{X}}{b},\frac{\triangle D_{X}}{b}+1,0\right)&-b \leqslant\triangle D_{X}<0\\ (0,-\frac{\triangle D_{X}}{b}+1,\frac{\triangle D_{X}}{b})&0\leqslant\triangle D _{X}<b\\ (0,0,1)&\triangle D_{X}\geqslant b\end{cases}  \tag{2}\]  
where \(S_{DepthX}\) is the depth spatial feature of the X view image (\(X\in\{F,L,T\}\)), \(\triangle D_{X}\) is the third term of the multi-view spatial relationship in the X view image, and \(b\) is the boundary point which can distinguish three kinds of depth relationship (near, middle, far). Clearly, each dimension of \(S_{DepthX}\) represents the possibility that the depth difference in the X view image belongs to each kind of depth relationship (near, middle, far). Because \(\triangle w\) and \(\triangle h\) are generated based on the original RGB image but \(\triangle\)depth is calculated in an estimated depth map, \(S_{DepthF}\) is not completely accurate; however, \(S_{DepthL}\) and \(S_{DepthT}\) are accurate. However, if there is no normalization, the value ranges of the third term of the multi-view spatial relationship in different view images vary. For example, the range of \(\triangle\)Depth  
Figure 3: Multi-view images of two objects in 3D space. In the front view image, \(\triangle w\) and \(\triangle h\) can be calculated according to object center locations extracted by object detector. \(\triangle\)Depth is the difference of mean depth values between two detected bounding boxes. Note that \(\triangle w\), \(\triangle h\) and \(\triangle\)depth in these three multi-view images have the same values. is from 0 to 255, but the ranges of \(\triangle w\) and \(\triangle h\) depend on the size of the original images.