Improving Visual Relation Detection using Depth Maps
Authors: Sahand Sharifzadeh
Sina Moayed Baharlou
Max Berrendorf
Rajat Koner
Volker Tresp
Venue:  (2020)
Keywords: Visual Relation Detection
Scene Graph Generation

Abstract: State of the art visual relation detection methods mostly rely on object information extracted from RGB images such as predicted class probabilities, 2D bounding boxes and feature maps. In this paper, we argue that the 3D positions of objects in space can provide additional valuable information about object relations. This information helps not only to detect spatial relations, such as \textit{standing behind}, but also non-spatial relations, such as \textit{holding}. Since 3D information of a scene is not easily accessible, we propose incorporating a pre-trained RGB-to-Depth model within visual relation detection frameworks. We discuss different feature extraction strategies from depth maps and show their critical role in relation detection.
Our experiments confirm that the performance of state-of-the-art visual relation detection approaches can significantly be improved by utilizing depth map information.

Under review as a conference paper at ICLR 2020
IMPROVING VISUAL RELATION DETECTION USING DEPTH MAPS
Anonymous authors Paper under double-bling review
ABSTRACT
We study the effect of using different object information for visual relation detec- tion with a focus on depth maps. State of the art visual relation detection methods mostly rely on object information extracted from RGB images such as predicted class probabilities, 2D bounding boxes and feature maps. In this paper, we argue that the 3D positions of objects space can provide additional valuable information about object relations. This information helps not only to detect spatial relations, such as standing behind, but also non-spatial relations, such as holding. Since 3D information of scene is not easily accessible, incorporate pre-trained RGG-to-Depth model within a visual relation detection framework and release new dataset VG-Depth the highly imbalanced distribution of relations in the available datasets, typical evaluation metries for visual relation detection cannot reveal improvements of under-represented relations. To address this problem, we propose using Macro Recall @ for visual relation detection. Our experiments confirm that the perfor- mance of state-of of-the-art visual relation detection approaches can significantly be improved by utilizing depth map information.
extension Visual Genome. We note that given
Relational learning is an established research area in machine learning. State-of-the-art ap- proaches (Nickel et al., (016)) describe relations triples of the form (subject, predicate, object), such as (Man, rides, Bike). The triples, which all together form a knowledge graph, are typically extracted from structured data such as the infoboxes in Wikipedia and other sources.
In the last years, the detection relations from images, i.e., visual relation detection, has also gained a lot of interest. One reason is that understanding relations between entities can play important role in decision making. For example, detecting whether a man is on a bike or next to a bike is a crucial challenge in autonomous driving. State-of-the-arr works in this area utilize information of objects in the scene such as class labels, bounding boxes and RGB features, to capture pairwise relations using different models. In this paper, we argue that relation detection can additionally benefit from objects' 3D information. This information can help to distinguish between many relations such as behind, in front of and even improve detection in situations where the objects are nearby such as covered in.
Unfortunately, most available datasets, specifically the ones with relational annotations such as Visual Relation Detection (VRD) (Lu et al. 2016) and Visual Genome (VG) (Krishna et al., 2017). lack this 3D information since acquiring them is a cumbersome task requiring specialized hardware. We solve this issue by synthetically generating the corresponding depth maps of images in these datasets. Depth maps provide the objects' distance from the camera and the availability of large corpora of RGB-D pairs, c.g. NYU-Depth-v2 (Nathan Silberman & Fergus 2012) dataset enables us to learn the mapping between any RGB image to its corresponding depth map using a fully convolutional neural network. We can then apply the trained network on images from VG, converting them into depth maps. We call this dataset extension VG-DeptA
The extracted features from depth maps, together with other object information extracted from the RGB images, are the basis for relation detection in our framework. We note that the typically employed Recall K metric, which we call as Micro Recall @ K, cannot properly reveal the improve- ments of under-represented relations in highly imbalanced datasets such as Visual Genome. This might be an issue in applications such as autonomous driving where it is important to ensure that the model is capable of predicting also more important but less represented predicates such as walking on
This dataset will he made publicly available upon publication.

---

Under review as a conference paper at ICLR 2020
Figure I: An image from the Visual Genome dataset showing several men sitting next to each other and the corresponding generated depth map. Bright colors indicate a larger distance to the camera. Knowing the depth of, e-g., the individuals, the house and the car, plays important role in detecting relations such as next to and behind.
(648 in VG test set) and not just wearing (20,143 148 in VG test set). We tackle this issue by proposing to employ Macro Recall @K, where the Recall@K K is computed per predicate class and averaged over, thereby eliminating the effect that over-represented classes have in Micro Recall setting. Reporting both Macro @ K and Micro @ K, we study the effect of each object information visual relation detection.
In summary, contributions are follows:
1. We perform an extensive study on the effect different object information in visual relation detection, including 3D information. We show in our empirical evaluations using the VG dataset, that our model can outperform competing methods by a margin of up to 8% points, even those with contextualization or using information extracted from external language sources.
2. We propose using Macro Recall@ K to better reveal the improvements in visual relation detection when dealing with highly imbalanced datasets such as Visual Genome.
3. To compensate for the lack of 3D data in Visual Genome, trained on separate corpora of available pairs in both modalities and apply it to VG images and releasing new dataset, VG-Depth.
an RGB-to-Depth model
In general, in this work we aim to answer the following questions:
1. If we are given only depth maps of unknown objects in a scene, how accurately can we infer the distribution of possible pairwise relations? How do other sources of object information compare to it?
2. Current visual relation detection frameworks commonly rely on extensive object infor- mation such class labels, bounding boxes, RGB features, contextual information, etc. Does bring any additional information, if we also include depth representations these frameworks would that only bring redundant knowledge about the scene?
3. Do the current evaluation metrics properly reflect the improvements of under-representee relations within highly imbalanced datasets?
RELATED WORKS
Knowledge Graph Learning In Knowledge Graph learning, the aim is typically to find embed- dings latent representations for entities and predicates, which then can serve to predict the probabil- ity of unseen triples. These methods mostly differ in how they model relations. RESCAL (Nickel et al. 2011) each relation is defined as linear transformation in the embedding space of entities, producing a triple probability. TransE (Bordes et al. 2013) employs a similar idea but limits each relation to translation. In comparison to RESCAL, it has fewer parameters; cannot model symmetric relations. DistMult (Yang et al. 2014) considers each relation as a vector, similar TransE, but minimizes the trilinear dot product of subject, predicate and object
vintaaa
2

---

Under review as a conference paper at ICLR 2020
vector. DistMult also be understood as form of RESCAL, where the transformation matrix is diagonal. ComplEx (Trouillon et al. 2016) extends DistMult to complex-valued vectors of em- beddings. A multilayer perceptron (MLP) architecture (Dong et al.] 2014) extends these methods to non-linear transformations and has shown to be competitive to the other discussed approaches on most benchmarks (Nickel al. 2016: Socher et al. 2013).
Visual Relation Detection Visual relation detection received huge boost by the availability of large corpora of annotated images such as the Visual Relation Detection (VRD) (Lu et al. (016)) and the VG (Krishna et al.) 2017). containing the visual form of entities, and relations. In VRD, Word2Vec representations of the subject, object, and the predicate were used to train a model jointl with the corresponding image section describing the predicate. particular, they consider the joint bounding box of subject and object as the image representation for the predicate. Follow-up work achieved improved performance by incorporating a knowledge graph, constructed from the annotated triples in the training set (Baier et al. 2017). In general, the distribution of the predicate bounding box in these works is much more long-tate toiget than of entities alone. Therefore, separating the models for objects and predicates, training such a model. V'ransEE is a generalization of TransE to visual relation detection in which the last convolutional layer of the image detector, together with the location of entities and their class labels, is taken as the imput vector to the TransE algorithm. More recently. Yu et al. (Yu et al.) 2017) proposed a teacher-student model distill external language knowledge improve visual relation detection. Iterative Message Passing (Xu et al. 2017), Neural Motifs (Zellers et al., (018)) and Graph R-CNN (Yang et al., 2018b) incorporate context within each prediction using RNNs and graph convolutions respectively.
employed in VTransE (Zhang al.. (017)) reduces the complexity of
Depth Maps While several works have leveraged depth maps to improve object detection (Bo et al., Eitel et al. 2015 Gupta et al. 2014), the idea of using depth maps in the relation detection task has only been explored in recent parallel work (Yang al., 2018a). While the focus of that work is on human-centered relations, we carry out our studies on a broader set of relations (VG), releasing dataset extension, and also providing more extensive study on the role of different object information in visual relation detection, with competent metric.
FRAMEWORK
In this section we introduce the general framework employed for this study. Let E = {el, .z,... ..., en be the set of all entities, including subjects(s) and objects(o), and P = {P1, P2. [P1.P2,
pm the set of all predicates. Each entity 6 can appear in images within bounding box bb; (Ix, Ui, W4, h2). where (I,, y:) are the coordinates of the bounding box and (w;, h;) are its width and height. In this work we apply the pre-trained (on ImageNet (Russakovsky et al. 2015)) and fine-tuned (on Visual Genome) Faster R-CNN (Ren et al. 2015) on each image / to extract a feature map fmapr together with object proposals
set of bounding boxes bb and class probability distributions c. For each RGB image, we generate a depth map D where the same bounding box areas encompass the entities' distance from the camera. In the next section, we first describe the employed network for synthetic generation of Ds and then discuss the extraction of depth features. In the end, we describe the relation detection head where the late fusion of pairwise features and their relational modelling takes place.
2.1
DEPTH MAPS FOR RELATION DETECTION
2.1.1 GENERATION
We incorporate an RGB-to-Depth model within our visual relation detection framework. As shown in Figure [ this is a fully convolutional neural network (CNN) that takes an RGB image as input and generates its predicted depth map. This model can be pre-trained on any datasets containing pairs of RGB and depth maps regardless of having the object or predicate annotations. This enables us to work with currently available visual relation detection datasets without requiring to collect additional data, and also mitigates the need for specialized hardware in worl-world applications. The architectural details explained in Section 3
3

---

Under review as a conference paper at ICLR 2020
Geoereed Depth Mapto)
Deyth Fottar
Nitwork
PGB Fasan
YouR DGB
HGl Poatan
Figure We propose utilizing 3D information in visual relation detection by synthetically generating depth maps using an RGB-to-Deppe model incorporated within relation detection frameworks. On the left side, we see the RGB image and its depth map. We use separate CNNs to extract feature maps from both modalities and create pairwise feature vectors dga (pooled from depth feature maps), Iso (from bounding boxes), cgo (from class labels) and Vso (from the pooled visual features). These vectors then concatenated and fed into relation detection layer to infer the predicate.
2.1.2 FEATURE EXTRACTION
Depth maps have been employed in tasks such object detection and segmentation (Eitel et 2015 Hazirbas et al. 2016). In those works, it is common to simply render a depth map as an RGB image and extract depth features using a CNN pre-trained for RGB images. There, it has been argued that the edges in depth maps might yield better object contours than the edges in cluttered RGB images and that one may combine edges from both RGB and depth to obtain more information (Hazirbas) al. 2016). Therefore, they aimed to get similar, complementary features from both modalities. However, the practice of employing model pre-trained on particular source modality, e.g. RGB, and applying it on different target modality, e.g. depth map, is sub-optimal in many applications Therefore, similar to other works on depth-based image segmentation or classification we employ a CNN, to generate feature map fmapp from an input depth map. but unlike those works, train this network from scratch, using depth maps and specifically for relation detection.
2.2 RELATION MODEL
In the previous section, we described methods for the extraction of individual object features. Here, outline the model that infers relations using pairwise combination of these features. For each pair of detected objects within an image, we create a scale-invariant location feature 1, = (lz, ly lwo 4,) with: to (I, zo) Wo. ty (y, yo)/ fu, tu logí /./ w..) ft. = (hg(ha/ hh,) and similarly lo- We then pool the corresponding features and vo from fiapr and create a visual feature vector [vs; v.). Similarly, we create a depth feature vector [dg; d.). by extracting bb, and bb, from fmapD and class vector [c,; col and similarly lo). Each of these vectors are fed into separate layers, yielding sso. Iso. Cso and dso before being concatenated altogether and fed to the relation head which projects them the relation space such that:
Cp (WWW/so Iuoi Cyo dyoo)..
Vsoi
(1)
Here, W describes a linear transformation and ff is non- -onerrnear function. We realize this as fully connected layers neural network with ReLU activations and dropout. ep is the embedding vector for predicate which will be learned jointly with other parameters. This simple relation prediction model a generalization the model used by (Dong et al. 2014)) to construct knowledge graphs.
One should also keep in mind that even fine-tuning some layers of network does not change the very early convolutional filters.
4

---

Under review as a conference paper at ICLR 2020
In that paper, latent features for the predicates are also part of the network input, whereas here we use a separate output for each predicate which is easier to train regarding negative sampling, and also has fewer parameters. As shown in earlier works, using more sophisticated models for context propagation between objects with RNNs or graph convolutions, can further improve the prediction accuracy. However, the aim here is study the effect of including depth maps additional object representations in visual relation detection and as will be shown later, even with this simple model, they can be more effective supplement than e.g. propagated contex Clearly, those other models can also further enrich their understanding of object relations, by employing these additional representations.
To learn the parameters, we consider each relation (s, p. o) with an associated Bernoulli variable that takes if the triple is observed and otherwise, following locally closed world assumption (Nickel et al. 2016). Given the set of observed triples T, the loss function is the categorical cross entropy between the one-hot targets and the distribution obtained by softmax over the network's output defined as:
exp / p [rr[[[[[oo [WVos lao; Csoi dsol))
f Σ log
EpEP exp (w") T /WW (WW[voo lso; Caoi dso)))
(s.2.0)ET
where w', p is the weight vector corresponding to p in the last layer (linear classification layer).
3 EVALUATION
3.1 DATASETS
We test our approach on the Visual Genome (Krishna al. 2017) dataset. We use the more commonly used subset of VG dataset proposed by (Xu et al. [2077) which contains 150 object classes and 50 relations with 75,651 images used for training and (5000 for validation) and 32,422 images for testing.
3.2 ARCHITECTURES
RGB-to-Dee CG--to-Depth Network: We use the RGB-to-Deple model architecture introduced in (Laina et al. 2016) which is a fully convolutional neural network built on ResNet-50 (He et al., 2015), and trained in end-to-end fashion data from NYU Depth Dataset v2 (Nathan Silberman & Fergus 2012). Training on the outdoor images from Make3D dataset (Saxena et al. (007) instead, did not show promising results in framework. This observation is not surprising since Make3D images contain mostly outdoor scenes with too few objects.
RGB Feature Extraction: To extract embeddings and class probabilities of RGB images, we use the VGG-16 architecture (Simonyan & Zisserman, 2014) pre-trained on ImageNet (Russakovsky 2015) and fine-tuned to our data for relation detection.
al.
Depth Map Feature Extraction: For depth map extraction we use ResNet18 proposed in (He et al. 2015). We trained this model from scratch following the earlier discussions in Subsection 2.1.2 We trained this network on a pure depth-based, relation detection task using Adam (Kingma & Ba 2014 with learning rate of 10-4 and batch size of 32 for eight epochs.
Finally, given the features extracted from previous models with the location features described in Subsection 2.1.2] we trained our relation detection model. We connected each feature pair described in the previous section, separately to fully connected hidden layer of 64, 200, 4096 and 20 neurons each, with a dropout rate of 0.1, 0.8, 0.8 and 0. with scaling layer initialized as 1.0, 0.3, 0.5 and 1.0. The penultimate layer contains 4096 neurons with 0.2 dropout. We trained this network by Adam (Kingma & Ba 2014], with a learning rate of 10 We used a batch size of 16 and eight epochs of training. All of the layers were initialized with Xavier weights (Glorot & Bengio 2010). During the training of this network, RGB Feature Exraction and RGB-to-Depth network weights were frozen.
Relation Detection Network:
'e carried on Visual Genome experiments in an isolated framework, where all pre-trained weights and hyper-parameters, except for the depth map channel, were kept identical the ones employed by Neural Motifs (Zellers al. 2018). The code will he released upon publication.
5

---

Under review as a conference paper at ICLR 2020
Table 1: Predicate prediction recall values on VG test set. We have omitted for better readability. When the depth maps are utilized together with all other features (Ours-l, 1, v. d), we gain large improvement compared to the state-of-the-an In Macro setting, this improvement almost as large as Ours-c to Ours-c, v, demonstrating the importance of depth features in relation detection. One can also see that even replacing depth maps with visual features (Ours-l, c, compared to Ours-l, v) can yield better results. Additionally, comparing Ours-c, v, to VTransE and Neural Motifs reveals the advantage of our simple model regardless of depth maps.
Strategy
Macro Predicate Pred.
Micro Predicate Pred.
Task
Metric
R@ 100
R@50
R@20
R@ 100
R@ R@20
VTransE (hhang et 2017
62.87
62.63
Yu's-S et
49.88
models
Yu's-S Tu cl ml 2017
55.89
[Xu ct 20
53,00
Gruph
R-UNN Yang 2018h
59,10
54.20
NM (Zellers et al 2018
14.39
13,20
10.25
67.10
65,20
$8.50
Ouurs 7
9.3
8.46
6.35
54,72
51.90
"43866"
Ours-e
15.65
13.09
8.56
64.82
60.54
49.89
Ours w
13.88
12.24
8.99
61.72
58.50
50.41
ablations
Ours
5.19
4.66
3.57
49.07
46.13
37.48
Ours v,
15.47
14.04
10.83
62.88
60.52
53.07
Ours 1,
15.76
14.40
11.07
63.06
61183
$3.55
Ours -l,,
21.67
19.56
15.12
67.97
66.09
59.13
Ours -,
19.16
17.72
13.93
67.94
66.06
59.14
Ours - 1,
22.72
20.74
16.40
68.00
66.18
59.44
3.3 METRICS
Some relations might not be annotated in the test set. Still, due to the model's generalization, they might get higher prediction values than the annotated ones. Therefore, similar to previous works (Nickel et al. 2016 Krishna al. 2017), we report Recall K (R@K) K). Recall per image is defined as the ratio of ground truth labels that appear in the model's top K predictions. However, as the distribution of relations in VG is highly imbalanced, this score is dominated by frequently labeled relations and might not reflect the improvements in some important but under-represented classes. Thus, we propose to compute Macro R@ K and name the former computation metric as Micro R@ K. For discussion upon Micro versus Macro for F1 measure refer to (Schütze et al. 2008). Macro R@K, K, we compute Micro R@K separately for each predicate class and average over it. In this setting, the prediction accuracy of under-represented classes has the chance to have stronger effect on general accuracy.
3.4 COMPARING METHODS
We compare our results with VTransE (Zhang et al. 2017) that takes visual embeddings and projects them to relation space using TransE. We also compare to the student network of Yu al. (2017) (Yu's s-S), and their full model (Yu's s-S+7) that employs external language data from Wikipedia. In the context propagating methods we report Neural Motif's (Zellers al. 2018). Graph R-CNN (Yang et al., (0188) and IMP Xu et al.) (2017). In an ablation study, we report our relation prediction results under several settings in which different combinations of object information are employed for prediction.
3.5 EXPERIMENTS
Our main goal is to investigate the role of depth maps and other object information in relation detection. Therefore, we do not focus on improving the object detection accuracy and report predicate prediction results. In this setting, the relation detection performance is isolated from the object detector's ' error by using ground truth bounding boxes of the entities and their class labels to predict the most likely predicates. We carried different random seeds. The maximum variance was no more than 0.01. In the following we provide discussion over the quantitative and qualitative results.
experiments by training each model 8 times with

---

Under review as a conference paper at ICLR 2020
Figure 3: Using depth maps improves the understanding of perspective and increases the general detection rate of relations such as behind (e.g. right image). However, in some cases such as in the left image, after including depth maps, (bushes, behind, bench) has been mislabelled. One should note that here the bushes (left corner of the image) have similar depth values as the benches. Equal depth values can be strong indicator of the predicate next to rather than behind. In this case, the human point of view seems to have caused a labelling error recovered by the depth- based model.
The complete predicate prediction results are shown in Table] 1 strates the results directly reported from other works while the lower two parts present the results from ablation study on our model. For NM, we have computed the Macro @@K results from their available model and code (reproducing the same Micro R @ K results reported in their paper). We can that our full model achieves the highest accuracy in comparison to the others in all settings specially under Macro R@K. It is also interesting to note that when using only depth maps we can already achieve significant accuracy in predicate prediction, emphasizing the value of relational information stored within depth maps. We can observe the improvements that externaly generated depth maps can bring to Ours-v by comparing it to Ours-v, d. Also comparing Ours-c, d, to Ours-c, v. / is specially informative from two aspects: (1) shows that while some results are almost equal in Micro settings, one can observe significant difference under Macro settings, thus observing the changes in under-represented classes. (2) Considering that alone has better R@ than alone when adding them separately to we can see that d has more to offer. In other words, has more redundant information for c, u compared to d. One should note that the improvement is not a linear function of extra information and the combination of different information sources might have non-linear effects on the outcome.
The upper part of the table demon-
To get better intuition of the improvements that gain after including depth maps (Ours-v, d) to Ours-u, we plotted the changes in prediction accuracy for each predicate in Figure 4 We used darker shades for over-represented classes and lighter shades for under-represented ones. This helps to also gain a better intuition of improvement versus representativeness. We can see that in general the accuracy of relations including the predicates such as under, in front of and behind has been improved. However, we also observed in our data, that in some cases the predicate behind was mislabeled after adding the depth maps. To further explore the potential reason behind some of such successes or failures, we further examined the test images. Figure 3 gives an example of a failure and a success case in predicting behind. While in the right image the annotations indicate that (bushes, behind, bench) based on the corresponding depth map, they are in the same distance from the camera (bushes can be seen in the left corner the image). One can argue that in fact, bushes are next to the bench rather than behind, indicating the noisy ground truth. However, it is also important to note that humans describe a predicate from their point of view which is not necessarily associated with the camera's point of view. This effect might appear in many other instances within the dataset, leading to worse performances when dealing with such predicates. A simple way to overcome such problems by having richer set of data.
We provide some samples of generated depth maps from VG images, with high and low quality in Figure 6 [] Note that the potential imprecisions within generated low-quality depth maps, e.g. sky and reflective objects such as glasses, are inevitable.
4 CONCLUSION
We identified 3D information as an important attribute for visual relation detection. Since this information is typically not provided in visual data sets, employed RGB-to-Depte network, which was pre-traines trained on a large corpus of data, to generate depth information. We provided metric, Macro R@ K for better evaluation. In extensive empirical evaluations, we demonstrated the effect of
7

---

Under review as a conference paper at ICLR 2020
0.00
9.15
0.25
0.08
1392
Figure 4: This plot shows the prediction changes per predicate, going from Ours-u Ours-v, d. The classes with zero changes are omitted from the plot. The darker shades indicate larger number of that class within the test set whereas the lighter shades are under-represented classes. An improvement in predicates with more frequency has a larger effect on the Micro R@K whereas this effect is eliminated within Macro @@k K. We can see that indeed the improvements by using depth maps are mostly happening within the less- classes.
Figure 5: Examples of high quality depth maps generated from visual genome images.
Figure 6: Examples of low quality depth maps generated from visual genome images.
8

---

Under review as a conference paper at ICLR 2020
different object information in visual relation detection and showed that by using depth information, one achieves significantly better performance compared to other state-of-the of-ee-of-he-arr art methods.
REFERENCES
Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using semantic modeling of scene descriptions. In International Semantic Web Conference, pp. 53-68. Springer, 2017.
Liefeng Bo, Xiaofeng Ren, and Dieter Fox. Unsupervised feature learning for rgb-d based object recognition. In Experimental Robotics, pp. 387-402. Springer, 2013.
Antoine Bordes, Nicolas Usunier, Alberto Gareia-Duran, Jason Weston, and Ok- Yakhnenko. Translating embeddings for modeling multi-relational data. In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q. Weinberger (eds.), Advances in Neural Information Processing Systems 26, pp. 2787-2795. Curran Associates, Inc., 2013. URL
ceaaapers.iisee wwt.:iissss..ccpppooooeoeoo 5071 httl==-aassaaaae---ellaaii------ http:paw...............ooo...oa.
pdf
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy. Thomas Strohmann, Shaohua Sun, and Wei Zhang. Knowledge vault: A web-scale approach to proba- bilistic knowledge fusion. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 601-610. ACM, 2014.
Andreas Fitel, Jost Tobias Springenberg. Luciano Spinello, Martin Riedmiller, and Wolfram Burgard. Multimodal deep learning for robust rgb-d object recognition. In Intelligent Robots and Systems (IROS), 2015 IEEE/RSJ International Conference on, pp. 681-687. IEEE, 2015.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-2566 2010.
Saurabh Gupta, Ross Girshick, Pablo Arbeláez, and Jitendra Malik. Learning rich features from rgb-d images for object detection and segmentation. In European Conference on Computer Vision, pp- 345-360. Springer, 2014.
Caner Hazirbas, Lingni Ma, Csaba Domokos, and Daniel Cremers. Fusenet: Incorporating depth into semantic segmentation via fusion-based enn architecture. In Asian Conference on Computer Vision, pp. 213-228. Springer, 2016.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, hbb//1512.0338. 2015. (RRp:tt/://axxii/ http://www...otr//a///11........ 03385
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14Xi::1412.6980 2014.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
Iro Laina, Christian Rupprecht, Vasileios Belagiannis, Federico Tombari, and Nassir Navab. Deeper depth prediction with fully convolutional residual networks. In 3D Vision (3DV), 2016 Fourth International Conference on, pp. 239-248 IEEE, 2016.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference Computer Vision, pp. 852-869. Springer, 2016.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rghd images. In ECCV, 2012.

---

Under review as a conference paper at ICLR 2020
Maximilian Nickel, Volker Tresp, and Hans-Peter Kriegel. A three-wa way model for collective learning on multi-relational data. In ICML, volume 11, pp. 809-816, 2011.
Maximilian Nickel, Kevin Murphy, Volker Tresp, and Evgeniy Gabrilovich. A review of relational machine learning for knowledge graphs. Proceedings of the IEEE, 104(1):11-33 2016.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-252, 2015. doi: 10.1007/s1 1263-015-0816--
Ashutosh Saxena, Min Sun, and Andrew Y Ng. Learning 3-d scene structure from single still image. In Computer Vision, 2007. ICCV 2007. IEEE 1 11th International Conference on, pp. 1-8, IEEE, 2007.
Hinrich Schüitee, Christopher D Manning, and Prabhakar Raghavan. Introduction to information re- trieval. In Proceedings of the international communication of association for computing machinery conference, pp. 260, 2008.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. In Advances in neural information processing systems, pp. 926-934, 2013.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International Conference on Machine Learning, pp. 2071-2080, 2016.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410-5419, 2017.
Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6577 2014.
Hsuan-Kung Yang, An-Chieh Cheng. Kuan-Wei Ho, Tsu-Jui Fu, and Chun- Yi Lee. Visual relationship prediction via label clustering and incorporation of depth information. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 0-0, 2018a.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruy Batra, and Devi Parikh. Graph r-enn for graph generation. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 670-685, 2018b.
Ruichi Yu, Ang Li, Vlad I Morariu, and Larry S Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In IEEE International Conference on Computer Vision (ICCV), 2017.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5831-5840, 2018.
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embedding network for visual relation detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pp. 3107-3115. IEEE Computer Society, 2017. ISBN 978- 1577---538-- doi: 10. 11109/CVVR.201...00 URL https: /oii
1109/VV.. 201..30171
10


Reviews:
[{"review_id": "SylMkK2CYH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to leverage the depth information for relation prediction, arguing that the depth information benefit the prediction of some predicates. To solve the lack of 3D data, an RGB-to-Depth model is trained on external available dataset and then applied to images from visual relation dataset. In the experiments, they investigate different strategies to extract features from depth maps and the explore effectiveness of depth information by comparing the model that only used depth map as input with those which use RGB information. The comparisons with other methods and ablation studies under both Zero-shot setting and normal setting demonstrate the effectiveness of depth information.\n\n+Strength:\n(1) The motivation is reasonable and what the authors make an attempt to explore is very meaningful. Visual relation especially the spatial relation is not likely to be predicted accurately without 3D information. In other words, it seems that visual relation prediction task will be extended to 3D images rather than staying within 2D images. Thus what the authors do is a good exploration for further extensions.\n(2) Comparisons with previous methods and the results show that the depth information is useful to some extent, but not so obvious.\n(3) The writing of this article is good and it’s very easy to understand.\n\n-Weakness:\n(1) The RGB-to-Depth Network is pretrained on other dataset. Is there any gap when it is used for VG or VRD dataset? \n(2) Although the depth map feature extraction seems to work well, it seems to be a little trivial. Why a CNN, e.g. AlexNet, or VGG, can be used to extract depth features? And why the AlexNet trained from scratch performs better than AlexNet pretrained on RGB images for object detection task and VGG net? If the author can give more explanations, this part will be more insightful.\n(3) From the plot which shows the top 10 percent absolute changes in prediction performance per predicate, the advantage of Depth is not obvious compared with RGB. And Depth does not bring the advantage claimed in Abstract. It’s a little hard to understand why depth information can rectify the prediction of (Tower, taller, trees). To sum up, the qualitative results are not so satisfying.\n(4) In Table 1, what really functions seems to be c_so, v_so, and l_so, while the improvement brought by depth is limited.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SylGDVKEqS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "OVERVIEW:\nThe authors propose to use depth information to better predict the visual relation between objects in an image. They do this by incorporating a pre-trained RGB-to-Depth model within existing frameworks. They claim the following contributions:\n1. First to utilize 3D information in visual relation detection. They synthesize depth images for existing benchmark datasets of VRD and VG using a pre-trained RGB-to-Depth model trained on NYUv2 to generate RGB-D data for visual relation detection.\n2. Discuss and empirically investigate different strategies to extract features from depth maps for relation detection.\n3. Study the quantitative and qualitative benefits of incorporating depth maps. \"We show in our empirical evaluation using the VRD and VG datasets, that models using depth maps can outperform competing methods by a margin of up to 3% points\".\n\nMAJOR COMMENTS:\n1. I liked the idea of using depth information to inform visual relationships but I am not sure if the proposed approach is the way to go. Given a depth image of the scene, we can generate a reconstruction of the scene in 3D, even if it is partial/imperfect. Direct reasoning in 3D should now be possible instead of going via deep networks as proposed in the paper. I believe a direct 3D approach would make a meaningful baseline at the very least and needs to be discussed.\n2. The authors use a pre-trained RGB-to-Depth network trained on NYU-v2 to predict depth for the images of VRD and VG. There is very little discussion about the quality of predicted depth maps. Ideally, this needs to be quantified to convince the reader that the generated depth maps are \"good\" but at the very least the authors need to show qualitative examples (both good, typical and bad) to prove that the pre-trained network generates meaningful depth maps.\n3. To use a siamese (shared weights) feature extractor between RGB and Depth images or not, is not a significant contribution by itself. In principle, separate feature extractors lead to larger model complexity/learning capability and make sense given domain separation between RGB and Depth. \n\nMINOR COMMENTS:\n1. Figure 2 seems to indicate that a Faster-RCNN is used on both RGB and Depth steams which is backed up by text in Section 2 (first paragraph). However, in Section 3.2, under RGB Feature Extraction and Depth Map Feature Extraction, the discussion is about VGG-16 and AlexNet-BN networks. The VGG-16 network is pre-trained in ImageNet and finetuned to relevant data but it is not clear for what task? If the task is object detection, it needs to be trained for it (not fine-tuned, unless it is being initialized from COCO pre-training). The AlexNet-BN depth model is trained for relation detection using only depth. But it is not clear if it is using proposals/boxes generated by RGB detection model or using ground-truth boxes. Basically, the object-detection component of the pipeline is not clear at all.\n\nNOTE:\nI would like to mention that I have published in monocular object pose estimation and work in the object recognition. I am not as familiar with the visual relation detection field but I understand all the components proposed by the authors in this work. I believe I understood the paper and reviewed it fairly (to the best of my ability).", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SkenUk7MFB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\n********* Post Rebuttal *********\n\nI appreciate the authors' effort in providing thorough responses and revised manuscript. \n\nI agree with the authors that \"the finding not being surprising\" is not a ground for rejection. I tried to word my final decision carefully but it seems it has still caused confusion for the authors. As I have mentioned in my original review, the rating was a result of the 4 points considered *together* . \n\nThat is, if one exploits privileged information that needs extra sensory data and/or annotation (point 2), *and*, this privileged information is clearly related and thus should be normally useful for the final task (point 1), *and* achieve marginal improvements (point 3), it can be a ground for rejection. Especially, given that prior works with similar arguments exists (point 4). \n\nThe rebuttal has alleviated the issue of marginal improvements (point 3) by introducing meanR@K (or as the revised paper refer to it, Macro R@K). Here, the improvements are more significant both compared to the state of the art and ablated baselines.\n\nThe authors also argue that the related [Yang et al. 2018] paper (point 4) should be considered a concurrent submission since the authors original submission was to AAAI18. \n\nThe rebuttal also addresses other clarity or experimental issues which improves the quality of the revised work.\n\nFinally, I understand that the privileged information is only required during training time which is a good point.\n\nAll in all, *assuming that [Yang et al. 2018] is considered a concurrent work* according to ICLR, I think the revised paper becomes slightly above borderline and thus I change my rating to \"weak accept\". If [Yang et al. 2018] is not considered concurrent work, then, a conclusive comparison is required for the acceptance of the current work.\n\n\n********* Summary *********\n \nThe paper poses the question of whether depth information is informative for visual relationship prediction using still images. It is intuitive that 3D arrangement of objects in an image can be a useful cue for predicting their relationship. As such it is important to see whether and to what extent depth information complements RGB information for visual relation detection. That is the focus of this paper.\nThe paper proposes to use an off-the-shelf monocular depth estimation networks to augment the available RGB information towards better visual relation detection. For that, it proposes a specific network two-stream structure working on RGB image and (predicted) depth image. The proposed model demonstrates improved results upon state of the art for visual relation prediction.\n \n\n********* Strengths and Weaknesses *********\n \n+ A comprehensive set of tests has been conducted. \n+ Zero-shot prediction results are particularly interesting.\n+ The experiment on ranking the predicate classes based on the change in prediction accuracy before and after using depth information (Figure 4) is interesting and intuitive.\n* The final results improve upon the state-of-the-art, especially on the zero-shot learning regime. However, it seems that the improvement is mainly coming from the new architecture as opposed to the inclusion of the depth information. That is, ours_{c,v,l} brings most of the improvement already the last step to ours_{c,v,l,d} is negligible for non-zero-shot case.\n- Along the same line, it’s possible that this small difference between ours_{c,v,l} and ours_{c,v,l,d} for the standard predicate prediction, can be due to a hyper-parameter optimization that is (only or more thoroughly) done for ours_{c,v,l,d}. The hyper-parameter optimization scheme for different baselines is not described. \n- Given the small difference of ablation levels, the comparison will be stronger if done multiple times and reporting mean and standard deviation of the results.\n- For a fair comparison the visual feature vector v_{so} should be tried as the feature of the union bound box of both subject and object same way as it is done for depth feature vector d_{so}. \n- The paper refers to “Ours-d’_{so}” as a baseline that *only* uses depth information with no image/label information. However, it seems that the region proposals for this feature are coming from the image-based network that uses image information. \n \n- Important related but uncited works:\n(1) [“Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information” ECCV workshops 2018] studies the same question as part of their work.\n \n\n********* Final Decision *********\n \nI do not find the paper passing the acceptance bar mainly due to the following reasons together:\n1) The finding is not surprising since most of the visual relations are either explicitly depth-related (e.g., behind) or are semantically constrained by depth (e.g. riding cannot happen at different depths when the image is taken orthogonal to the rider).\n2) an additional depth dataset is used which provides the model with privileged information. Should it have been the case that depth information were inferred without an additional offline dataset, the results would have been more interesting.\n3) the improvements due to the additional depth network are not significant or conclusive.\n4) there is a prior uncited work with the same research question for effectiveness of depth information in visual relation detection which uses a similar approach.\n \n\n********* Minor points *********\n\n- the code is not available. This is especially important since the paper is outperforming prior works which could be a contribution if reproducible.\n- Section 2.2: is l_{so} concatenation of l_s and l_o?\n- Section 2.2: y_{spo} is defined but never used.\n- Equation 2: why do we have both e_p and f in the exponents? Aren’t they the same?\n- Equation 2:  P is never defined.\n- Page 5: “a fully connected hidden layer of 64, 200, 4096 and 20 neurons”: this amounts to 3 hidden layers.\n- Why VGG network for visual feature and AlexNet for depth features?\n- zero-shot learning results on visual genome is missing\n- training procedure is a bit unclear: the text suggest that the fine tuning and/or learning of the three components might happen separately. It is important to clearly state if they are done in an end-to-end fashion and simultaneously or separately; and why.\n- It’s good to name the method in table 2 in the same fashion as table 1. With the current naming (based on architecture) it is a bit confusing to understand the content without additional cross referencing. For instance AlexNet-BN - Raw seems to correspond to Ours_{c,v,l,d}\n- Figure 4: the frequency represented as different shades of red or blue is really hard to notice especially on a printed paper. The red vs blue color coding is not necessary since the bars going up or down indicate the same quality. So, it might be better to use red/blue for frequency instead (e.g. dark red high frequency to dark blue low frequency)\n- Section 3.2: the AlexNet reference seems wrong, it should be \"ImageNet Classification with Deep Convolutional Neural Networks\" NIPS , 2012\n- The structure of section 3.5 is currently flat while the content seems to be nested (two experiments and two sets of corresponding discussions). It will read better if they are organized into subsections.\n \n \n********* Points of extensions (improvement) *********\n\n- I believe *unsupervised* discovery of depth information for visual relation detection can be an interesting direction since it is not limited to the availability of relevant depth dataset. \n- It is not clearly motivated why one should use two separate networks for depth and RGB inputs in light of the additional complexity. For instance, it is good to discuss what is the advantage of the proposed (computationally more expensive) method over the following two simpler baselines:\n- Faster RCNN is used on RGBD input to produce a single feature vector\n- above case with RGB input but have the Faster RCNN predict the depth map as an auxiliary loss.\n\n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes to improve visual relation prediction by using depth maps.  Since existing RGB images do not contain depth informations, the authors use a monocular depth estimation method to predict depth maps.  The authors show that using depths maps, they are able to improve prediction of relations between ground truth object bounding boxes and labels.  

The paper got relatively low scores (with 3 initial weak rejects).  After the revision and suggested improvements, one of the reviewers updated their score so the paper now has 2 weak rejects and 1 weak accept. 

The paper had the following weaknesses:
1. The paper has limited technical novelty as it combines off the shelf components.  The components also used different backbones (ResNet at some places, VGGNet at others) that were directly from prior work.  Was there any attempt to have an unified architecture? As the main novelty of the work is not in the model aspect, the paper needs to have stronger experiments and analysis.
2. More analysis on the quality of the depth estimation is needed.  Ideally, the work should provide some insight into whether some of the errors is due to having bad depth estimation?  The depth estimation method used is from 2016, there are newer depth estimation methods now.  Would having better depth estimation give improved results?  Experiments that illustrates that method works well with predicted bounding boxes instead of ground truth bounding boxes will also strengthen the paper.  
3. There was the question of whether the related Yang et al. 2018 workshop paper should be included as basis for comparison.  In the AC's opinion, Yang et al. 2018 is not concurrent work and should be treated as prior work.  However, it is not clear whether it is feasible to compare against that work.  The authors should attempt to do so and if infeasible, clearly articulate why that is the case.
4. As pointed out by R3, once there is a depth map available, it is also possible to compare against 3D methods (such as those that operate on point clouds)

Overall the paper had a nice insight by proposing the simple but effective idea of using depth information to help with visual relation prediction.  Still the work is somewhat borderline in quality.  In the AC's opinion, the main contribution and insight of the paper is of limited interest to the ICLR community, and it would be more appreciated in a computer vision conference.  The authors are encouraged to improve the paper with stronger experiments and analysis, incorporate various suggestions from the reviewers, and resubmit to a vision conference.

Decision: Reject