# Multi-View Visual Relationship Detection with Estimated Depth Map
## 3 Methodology
### Visual Relationship Classifier  
Based on the final combined feature vectors output by the multi-view features generator, the visual relationship classifier can extract the predicates between object pairs corresponding to the input feature vectors. Because estimated depth maps are utilized, there is inaccurate information in the final features. The RGB information (\(S_{RGB}\) and \(A_{RGB}\)) is accurate as it is based on the original input images. However, the depth information (\(S_{Depth}\) and \(A_{Depth}\)) is not completely accurate. Note that \(S_{Depth}\) is relatively accurate because of the fuzzy processing. Compared with RDBN [7], these final combined visual features contain different information in multi-view images; thus, we need an improved balanced relationship classifier with novel mask matrices to separately train the different visual components in the final feature vectors. For the visual relationship classifier \(W=[w_{1},\dots,w_{P}]\), every sub-classifier corresponds to one kind of predicate. \(X\) is the matrix of final combined feature vectors; thus, \(XW\) is the predicate classification results matrix. To process multi-view features, we utilize the loss function in [7] with novel mask matrices. The loss function can be represented as:  
\[L=\frac{1-\alpha}{N}\|Y-XAW\|_{F}^{2}+\frac{\alpha}{N}\|Y-XBW\|_{F}^{2}+\lambda \|W\|_{F}^{2} \tag{7}\]  
where \(N\) is the quantity of training samples, \(Y\in\{0,1\}\) is a matrix of ground truth labels, \(X=[x_{1},\dots,x_{N}]^{\text{T}}\) is the final feature vectors matrix where \(x_{n}\) corresponds n_th_ detected object pair, and \(\lambda\) is the coefficient of the regularization term. The remaining parameters are related with mask matrices \(A\) and \(B\). To train the visual relationship classifier with separated multi-view visual components, we redesigned the mask matrices \(A\) and \(B\), which are illustrated in Figure 5. Mask \(A\) and \(B\) are both composed of four diagonal matrices. They can be represented as:  
\[A=Diag\Big{[}I^{S_{RGB}},I^{S_{DepthX(X<L,T,F)}},I^{A_{Depth}},O^{A_{RGB}} \Big{]} \tag{8}\]  
\[B=Diag\Big{[}I^{S_{RGB}},\big{[}I^{S_{DepthL}},I^{S_{DepthT}},O^{S_{DepthT}} \big{]},O^{A_{Depth}},I^{A_{RGB}}\Big{]} \tag{9}\]  
Thus, \(XA\) and \(XB\) in Equation (7) can be represented as:  
Figure 4: An overview of the final combined feature of a pair of detected objects. The final visual feature vector is composed of four parts: \(S_{RGB}\), \(S_{Depth}\), \(A_{Depth}\), \(A_{RGB}\). The spatial feature is extracted by the second module of the proposed multi-view VRD framework, and the appearance feature is generated by the object detector. Note that the depth spatial feature \(S_{Depth}\) is composed of \(S_{DepthL}\), \(S_{DepthT}\) and \(S_{DepthF}\). \[XA= \begin{bmatrix}S_{RGB},\left[S_{DepthL},S_{DepthT},S_{DepthF}\right],A_{ Depth},A_{RGB}\end{bmatrix}  \tag{10}\] \[\cdot Diag\left[I^{S_{RGB}},I^{S_{DepthX(X\in L,T,F)}},I^{A_{Depth} },O^{A_{RGB}}\right]\]  
\[XB= \begin{bmatrix}S_{RGB},\left[S_{DepthL},S_{DepthT},S_{DepthF} \right],A_{Depth},A_{RGB}\end{bmatrix}  \tag{11}\] \[\cdot Diag\left[I^{S_{RGB}},\left[I^{S_{DepthL}},I^{S_{DepthT}},O ^{S_{DepthF}}\right],O^{A_{Depth}},I^{A_{RGB}}\right]\]  
The visual components in the final combined feature, which correspond to the zero terms in the mask, will be eliminated. Particularly, in mask \(B\), for the sub diagonal matrix corresponding to \(S_{Depth}\), the last three terms are set to zero. Intuitively, this is because in \(S_{Depth}=\left(S_{DepthL},S_{DepthT},S_{DepthF}\right)\), \(S_{DepthL}\) and \(S_{DepthT}\) are more accurate than \(S_{DepthF}\) which is based on an estimated depth map. Thus, in mask \(B\), \(S_{DepthF}\) and \(A_{Depth}\) are both inaccurate and are eliminated together. In Equation (7), with mask matrices, \(\alpha\) represents the training weight of the depth information and the RGB information. The visual triplet score which evaluates the confidence of the detected visual relationship can be constructed as:  
\[