# Multi-View Visual Relationship Detection with Estimated Depth Map
## 1 Introduction  
Visual relationship detection (VRD) focuses not only on detecting separate objects but also extracting the interactions between objects, which plays a vital role in the understanding of visual scenes. As illustrated in Figure 1, there are different visual relationships between separate detected objects. With the development of object detection, numerous studies on visual relationship detection [1; 2; 3; 4; 5; 6; 7] have been conducted in recent years. [PERSON] et al. [4] combined appearance RGB features with flat spatial features between detected bounding boxes, to improve the model performance of visual relationship detection and zero-shot learning. However, in practical scenes, numerous objects were distributed in 3D space instead of 2D flat space. Thus, it is important to research the RGB-D visual relationship detection framework. [PERSON] et al. [7] proposed a novel VRD framework which extended [PERSON] et al. 's work [4] from RGB features to RGB-D features to dramatically enhance the generalization ability of a VRD model in complex visual scenes. Consequently, depth maps corresponding to RGB images are valuable in visual relationship detection because the depth information can reveal additional semantic relationships which are not contained in RGB information. Several papers have introduced depth maps in visual relationship detection tasks to extract additional provided visual information on object interactions, such as RDBN [7] and the VG-DepthModel [8]. As we obtain the 3D visual information, multi-view methods can be utilized to further improve VRD performance. Existing multi-view methods have been widely used in visual scene understanding tasks, such as classification [9; 10], object detection [11; 12; 13],knowledge-base relation detection [14, 15] and image text detection [16, 17]. However, to the best of our knowledge, the multi-view method based on RGB-D information has not been researched in visual relationship detection, as there are some difficulties associated with this novel task. The first challenge is how to generate corresponding multi-view images only based on monocular RGB images. As described in [7], existing datasets for visual relationship detection only contain RGB information. Furthermore, in complex visual scenes, obtaining accurate depth data with an RGB-D camera is also difficult. Thus, first, we use a monocular depth estimation model [18] to generate inaccurate depth maps corresponding to original RGB images. Then, we propose a novel VRD framework, including a visual strategy, to transform the obtained estimated RGB-D information into multi-view images which can reveal additional visual semantic relationships. The second challenge is how to reduce the negative effects of inaccurate information in estimated depth maps and generated multi-view images. In [7], depth information was mapped into feature vectors before they were input to a visual relationship classifier where RGB and depth information were processed to reduce negative effects from inaccurate information. However, for multi-view images generated from estimated depth maps, there is even more inaccurate information than in depth maps, because every depth map corresponds to several multi-view images. In this paper, we improve the balanced visual relationship classifier to combine flat RGB images with corresponding multi-view images. In summary, there are three main contributions of this paper:  
1. To introduce the multi-view information, based only on limited monocular RGB images, into a visual relationship detection task, we construct a novel multi-view VRD framework composed of three modules to generate and utilize multi-view images. This framework can take advantage of multi-view information to extract potential visual features hidden in flat RGB images. 2. We improve the visual relationship classifier and mask matrices in a novel balanced classifier to process multi-view features, and then predict relationships with information in different views. 3.