# Multi-View Visual Relationship Detection with Estimated Depth Map
## 2 Related Work
### Relationship Triplet Learning  
A relationship triplet can be represented as subject, predicate, and object, where subject and object are located by an object detector and the predicate is extracted by a visual relationship classifier. Relationship triplet learning is a core module in visual scenes understanding, such as visual relationship detection [2; 3; 5; 23], visual question answering [24; 25] and image retrieval [26; 27; 28].  
Generally, in a visual relationship detection framework, the first module is an object detector. [PERSON] et al. [1] located objects with R-CNN [21], [PERSON] et al. [4] regarded Fast R-CNN [29] as the object detector, and [PERSON] et al. [7] utilized Faster R-CNN [22] to generate candidate object pairs.  
After obtaining visual features generated by an object detector, the features need to be processed and then input into a visual relationship classifier to extract the categories of predicates between the detected object pairs. [PERSON] et al. [1] proposed a novel VRD framework which combined a visual module with a language module to share knowledge among visual relationship classifiers (each classifier corresponded to one kind of predicate). Thus, the relevance of object pairs which had already appeared in a training dataset was enhanced, and the VRD accuracy was improved. However, in practical visual scenes, there are numerous combinations of relationship triplets. Although subject, object and predicate may appear in a training dataset, the combination of them may not appear. We call this situation zero-shot learning. To improve zero-shot learning performance, [PERSON] et al. [4] extracted spatial features by encoding detected bounding box coordinates into spatial vectors, and then combined appearance features with spatial features. Furthermore, ref. [4] proposed an UnRel dataset including unusual visual relationships to evaluate the generalization ability and zero-shot learning performance of a VRD model. To extract more visual information in a VRD task, [PERSON] et al. [7] introduced estimated depth maps into original flat RGB images. In [7], an RDBN framework was proposed to combine RGB information with inaccurate depth features and to balance them in a visual relationship classifier training process. Consequently, depth information was able to improve the accuracy and zero-shot learning performance of a VRD framework, especially in specific depth conditions.  
In this paper, we extend the visual feature dimensions in the same limited VRD dataset. First, we estimate depth maps corresponding to monocular RGB images by depth estimation deep neural networks. Second, using a novel multi-view strategy, multi-view images are generated based on original RGB images and inaccurate estimated depth maps. Finally, using a proposed novel visual module, multi-view visual feature vectors are extracted and input into a novel relationship classifier which can process multi-view RGB and depth information simultaneously. The results of experiments undertaken showed that the coordinated multi-view information was able to improve the detection accuracy and zero-shot learning performance of a VRD framework.