Improving Visual Relation Detection using Depth Maps

Relational learning is an established research area in machine learning . State-of-the-art approaches ( Nickel et al. , 2016 ) describe relations as triples of the form ( subject , predicate , object ) , such as ( Man , rides , Bike ) . The triples , which all together form a knowledge graph , are typically extracted from structured data such as the infoboxes in Wikipedia and other sources . In the last years , the detection of relations from images , i.e. , visual relation detection , has also gained a lot of interest . One reason is that understanding relations between entities can play an important role in decision making . For example , detecting whether a man is on a bike or next to a bike is a crucial challenge in autonomous driving . State-of-the-art works in this area utilize information of objects in the scene such as class labels , bounding boxes and RGB features , to capture pairwise relations using different models . In this paper , we argue that relation detection can additionally benefit from objects ’ 3D information . This information can help to distinguish between many relations such as behind , in front of and even improve detection in situations where the objects are nearby such as covered in . Unfortunately , most available datasets , specifically the ones with relational annotations such as Visual Relation Detection ( VRD ) ( Lu et al. , 2016 ) and Visual Genome ( VG ) ( Krishna et al. , 2017 ) , lack this 3D information since acquiring them is a cumbersome task requiring specialized hardware . We solve this issue by synthetically generating the corresponding depth maps of images in these datasets . Depth maps provide the objects ’ distance from the camera and the availability of large corpora of RGB-D pairs , e.g . NYU-Depth-v2 ( Nathan Silberman & Fergus , 2012 ) dataset enables us to learn the mapping between any RGB image to its corresponding depth map using a fully convolutional neural network . We can then apply the trained network on images from VG , converting them into depth maps . We call this dataset extension VG-Depth1 . The extracted features from depth maps , together with other object information extracted from the RGB images , are the basis for relation detection in our framework . We note that the typically employed Recall @ K metric , which we call as Micro Recall @ K , can not properly reveal the improvements of under-represented relations in highly imbalanced datasets such as Visual Genome . This might be an issue in applications such as autonomous driving where it is important to ensure that the model is capable of predicting also more important but less represented predicates such as walking on 1This dataset will be made publicly available upon publication . ( 648 in VG test set ) and not just wearing ( 20,148 in VG test set ) . We tackle this issue by proposing to employ Macro Recall @ K , where the Recall @ K is computed per predicate class and averaged over , thereby eliminating the effect that over-represented classes have in Micro Recall @ K setting . Reporting on both Macro @ K and Micro @ K , we study the effect of each object information on visual relation detection . In summary , our contributions are as follows : 1 . We perform an extensive study on the effect of different object information in visual relation detection , including 3D information . We show in our empirical evaluations using the VG dataset , that our model can outperform competing methods by a margin of up to 8 % points , even those with contextualization or using information extracted from external language sources . 2 . We propose using Macro Recall @ K to better reveal the improvements in visual relation detection when dealing with highly imbalanced datasets such as Visual Genome . 3 . To compensate for the lack of 3D data in Visual Genome , we use an RGB-to-Depth model trained on separate corpora of available pairs in both modalities and apply it to VG images and releasing a new dataset , VG-Depth . In general , in this work we aim to answer the following questions : 1 . If we are given only depth maps of unknown objects in a scene , how accurately can we infer the distribution of possible pairwise relations ? How do other sources of object information compare to it ? 2 . Current visual relation detection frameworks commonly rely on extensive object information such as class labels , bounding boxes , RGB features , contextual information , etc . Does it bring any additional information , if we also include depth representations in these frameworks or would that only bring redundant knowledge about the scene ? 3 . Do the current evaluation metrics properly reflect the improvements of under-represented relations within highly imbalanced datasets ? 1 RELATED WORKS . Knowledge Graph Learning In Knowledge Graph learning , the aim is typically to find embeddings or latent representations for entities and predicates , which then can serve to predict the probability of unseen triples . These methods mostly differ in how they model relations . In RESCAL ( Nickel et al. , 2011 ) each relation is defined as a linear transformation in the embedding space of entities , producing a triple probability . TransE ( Bordes et al. , 2013 ) employs a similar idea but limits each relation to a translation . In comparison to RESCAL , it has fewer parameters ; as a disadvantage , it can not model symmetric relations . DistMult ( Yang et al. , 2014 ) considers each relation as a vector , similar to TransE , but minimizes the trilinear dot product of subject , predicate and object vector . DistMult can also be understood as a form of RESCAL , where the transformation matrix is diagonal . ComplEx ( Trouillon et al. , 2016 ) extends DistMult to complex-valued vectors of embeddings . A multilayer perceptron ( MLP ) architecture ( Dong et al. , 2014 ) extends these methods to non-linear transformations and has shown to be competitive to the other discussed approaches on most benchmarks ( Nickel et al. , 2016 ; Socher et al. , 2013 ) . Visual Relation Detection Visual relation detection received a huge boost by the availability of large corpora of annotated images such as the Visual Relation Detection ( VRD ) ( Lu et al. , 2016 ) and the VG ( Krishna et al. , 2017 ) , containing the visual form of entities , and relations . In VRD , Word2Vec representations of the subject , object , and the predicate were used to train a model jointly with the corresponding image section describing the predicate . In particular , they consider the joint bounding box of subject and object as the image representation for the predicate . Follow-up work achieved improved performance by incorporating a knowledge graph , constructed from the annotated triples in the training set ( Baier et al. , 2017 ) . In general , the distribution of the predicate bounding box in these works is much more long-tailed than of entities alone . Therefore , separating the models for objects and predicates , as employed in VTransE ( Zhang et al. , 2017 ) reduces the complexity of training such a model . VTransE is a generalization of TransE to visual relation detection in which the last convolutional layer of the image detector , together with the location of entities and their class labels , is taken as the input vector to the TransE algorithm . More recently , Yu et al . ( Yu et al. , 2017 ) proposed a teacher-student model to distill external language knowledge to improve visual relation detection . Iterative Message Passing ( Xu et al. , 2017 ) , Neural Motifs ( Zellers et al. , 2018 ) and Graph R-CNN ( Yang et al. , 2018b ) incorporate context within each prediction using RNNs and graph convolutions respectively . Depth Maps While several works have leveraged depth maps to improve object detection ( Bo et al. , 2013 ; Eitel et al. , 2015 ; Gupta et al. , 2014 ) , the idea of using depth maps in the relation detection task has only been explored in a recent parallel work ( Yang et al. , 2018a ) . While the focus of that work is on human-centered relations , we carry out our studies on a broader set of relations ( VG ) , releasing a dataset extension , and also providing a more extensive study on the role of different object information in visual relation detection , with a competent metric . 2 FRAMEWORK . In this section we introduce the general framework employed for this study . Let E = { e1 , e2 , ... , en } be the set of all entities , including subjects ( s ) and objects ( o ) , and P = { p1 , p2 , ... , pm } the set of all predicates . Each entity ei can appear in images within a bounding box bbi = ( xi , yi , wi , hi ) , where ( xi , yi ) are the coordinates of the bounding box and ( wi , hi ) are its width and height . In this work we apply the pre-trained ( on ImageNet ( Russakovsky et al. , 2015 ) ) and fine-tuned ( on Visual Genome ) Faster R-CNN ( Ren et al. , 2015 ) on each image I to extract a feature map fmapI , together with object proposals as a set of bounding boxes bb and class probability distributions c. For each RGB image , we generate a depth map D where the same bounding box areas encompass the entities ’ distance from the camera . In the next section , we first describe the employed network for synthetic generation of Ds and then discuss the extraction of depth features . In the end , we describe the relation detection head where the late fusion of pairwise features and their relational modelling takes place . 2.1 DEPTH MAPS FOR RELATION DETECTION . 2.1.1 GENERATION . We incorporate an RGB-to-Depth model within our visual relation detection framework . As shown in Figure 2 , this is a fully convolutional neural network ( CNN ) that takes an RGB image as input and generates its predicted depth map . This model can be pre-trained on any datasets containing pairs of RGB and depth maps regardless of having the object or predicate annotations . This enables us to work with currently available visual relation detection datasets without requiring to collect additional data , and also mitigates the need for specialized hardware in real-world applications . The architectural details are explained in Section 3 . 2.1.2 FEATURE EXTRACTION . Depth maps have been employed in tasks such as object detection and segmentation ( Eitel et al. , 2015 ; Hazirbas et al. , 2016 ) . In those works , it is common to simply render a depth map as an RGB image and extract depth features using a CNN pre-trained for RGB images . There , it has been argued that the edges in depth maps might yield better object contours than the edges in cluttered RGB images and that one may combine edges from both RGB and depth to obtain more information ( Hazirbas et al. , 2016 ) . Therefore , they aimed to get similar , complementary features from both modalities . However , the practice of employing a model pre-trained on a particular source modality , e.g . RGB , and applying it on a different target modality , e.g . depth map , is sub-optimal in many applications2 . Therefore , similar to other works on depth-based image segmentation or classification we employ a CNN , to generate a feature map fmapD from an input depth map , but unlike those works , we train this network from scratch , using depth maps and specifically for relation detection .

Summary: This paper proposes to leverage the depth information for relation prediction, arguing that the depth information benefit the prediction of some predicates. To solve the lack of 3D data, an RGB-to-Depth model is trained on external available dataset and then applied to images from visual relation dataset. In the experiments, they investigate different strategies to extract features from depth maps and the explore effectiveness of depth information by comparing the model that only used depth map as input with those which use RGB information. The comparisons with other methods and ablation studies under both Zero-shot setting and normal setting demonstrate the effectiveness of depth information.