# Multi-View Visual Relationship Detection with Estimated Depth Map
## 3 Methodology
### Multi-View VRD Framework  
To utilize the multi-view information in visual relationship detection, based only on monocular RGB input images, we constructed a multi-view VRD framework composed of three modules. An overview of the proposed framework is illustrated in Figure 2.  
The first module is the RGB-D object detector. Following the settings of [7], this module includes a depth estimation network [18] and two Faster R-CNN models [22] with  
Figure 2: An overview of the proposed multi-view visual relationship detection framework. This framework is composed of three main modules: **(1) RGB-D object detector.** For a monocular RGB image, first it is input into a depth estimation network [18] to obtain the corresponding depth map. Then, the original RGB and generated depth map are input into two different Faster-RCNNs [22] to locate and classify objects. **(2) Multi-view features generator.** After obtaining the locations of objects in the RGB image and estimated depth map, multi-view images (front view, left view and top view) are generated by specific strategies. Then the spatial feature vector is extracted by an extractor. Moreover, the RGB and depth appearance feature vector are both concatenated with a spatial feature vector. **(3) Visual relationship classifier.** The concatenated final feature vector is input into a visual relationship classifier to predict the visual relationship triplets.  
different parameters. These deep neural networks are all trained on a VRD dataset [1]. Firstly, the original monocular RGB image is input into the depth estimation network to generate the depth map. Then, a monocular RGB image and the corresponding depth map are input into the RGB and depth Faster R-CNN, respectively, to detect candidate object pairs in the visual scene. Furthermore, we can obtain the RGB and depth appearance feature vectors in fully connected layers of the Faster R-CNN.  
The second module is a multi-view features generator. A function of this module is to generate multi-view images according to a monocular RGB image, corresponding depth map and object bounding boxes. Moreover, after obtaining multi-view images based on specific proposed strategies, flat and depth spatial feature vectors in different views can be generated by a spatial feature extractor in this module. The authors of [4] introduced a flat spatial feature into monocular RGB information to extract the potential semantic relationships in visual features. After that, RDBN [7] combined a flat visual feature with estimated depth information to further extend the dimensions of feature vectors. For our proposed multi-view VRD framework, visual features in different views, which include flat visual features and depth visual features, are fused together to utilize more comprehensive visual information.  
The third module is a visual relationship classifier. The final visual feature vectors, composed of appearance features and spatial features generated by the second module, are input into a visual relationship classifier to extract the visual relationship triplets. Based on the balanced classifier in RDBN [7], we construct an improved classifier which can process multi-view visual features simultaneously.  
The settings of the first module are the same as RDBN [7]; thus, we elaborate the remaining two modules in Sections 3.2 and 3.3, respectively.