# Multi-View Visual Relationship Detection with Estimated Depth Map
## 5 Conclusions  
In this paper, a novel multi-view visual relationship detection framework is proposed for utilizing the multi-view visual features hidden in monocular RGB information. The multi-view VRD framework achieved state-of-the-art performance of zero-shot learning on a VRD dataset [1] and UnRel dataset [4] in specific depth conditions.  
Firstly, we constructed the novel multi-view VRD framework composed of three main modules, based on estimated multi-view features, to improve the detection performance and generalization ability of the VRD model. Secondly, a novel multi-view images generation method, using only the original monocular RGB images, was proposed to transfer flat visual space to 3D multi-view space. Thirdly, we improved the visual relationship balanced classifier and mask matrices to enable the relationship predictor to be able to adapt to multi-view features. Finally, detailed comparison experiments were conducted to demonstrate the effectiveness of the multi-view VRD framework. Furthermore, we also analyzed the performance based on different feature vector normalization methods.  
However, although the multi-view VRD framework reached SOTA performance in zero-shot learning, it did not achieve the SOTA performance in normal VRD conditions. How to further reduce the negative effects from inaccurate visual information and apply multi-view information into video visual relationship detection require investigation in future research.  
To the best of our knowledge, this is the first time that multi-view information has been utilized in a visual relationship detection task. Our research has demonstrated the possibility that, even when input is only flat monocular RGB information, the multi-view features can be extracted and embedded in a VRD model.  
Conceptualization, X.L. and M.-G.G.; methodology, X.L.; software, X.L. and Y.H.; validation, X.L. and Y.H.; formal analysis, X.L.; investigation, X.L. and M.-G.G.; resources, X.L.; data curation, X.L.; writing--original draft preparation, X.L.; writing--review and editing, M.-G.G.; visualization, Y.H.; supervision, M.-G.G.; project administration, M.-G.G.; funding acquisition, M.-G.G. All authors have read and agreed to the published version of the manuscript.  
This work was supported by the National Key R&D Program of China under Grants 2020 YFB1708500.  
The authors declare no conflict of interest.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **With GT** & \multicolumn{3}{c}{**With Candidates**} \\ \cline{2-5}  & **-** & **Union** & **Sub** & **Sub-Obj** \\ \hline \multicolumn{5}{l}{_IoU \(\geqslant\) 0.3:_} \\ \hline Weakly [4] & 56.8 & 17.0 & 15.9 & 13.4 \\ RDBN [7] & 60.1 & 17.3 & 16.3 & 13.5 \\
**Multi-view** (with ImageNet) & **60.7** & 17.3 & 16.3 & 13.5 \\
**Multi-view** (with RegionNorm) & 60.2 & 17.3 & 16.3 & 13.5 \\ \hline \multicolumn{5}{l}{_IoU \(\geqslant\) 0.5:_} \\ \hline Weakly [4] & 56.8 & 15.2 & 11.0 & 7.7 \\ RDBN [7] & 60.1 & 15.6 & 11.6 & 7.8 \\
**Multi-view** (with ImageNet) & **60.7** & 15.6 & 11.6 & 7.8 \\
**Multi-view** (with RegionNorm) & 60.2 & 15.5 & 11.6 & 7.8 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Image retrieval performance (% mAP) of different VRD frameworks on UnRel dataset [4]. The IoU threshold is higher, the retrieval task is harder.