Title: IMPROVING VISUAL RELATION DETECTION USING DEPTH MAPS

Abstract: We study the effect of using different object information for visual relation detection with a focus on depth maps. State of the art visual relation detection methods mostly rely on object information extracted from RGB images such as predicted class probabilities, 2D bounding boxes and feature maps. In this paper, we argue that the 3D positions of objects in space can provide additional valuable information about object relations. This information helps not only to detect spatial relations, such as standing behind, but also non-spatial relations, such as holding. Since 3D information of a scene is not easily accessible, we incorporate a pre-trained RGB-to-Depth model within a visual relation detection framework and release a new dataset VG-Depth as an extension to Visual Genome. We note that given the highly imbalanced distribution of relations in the available datasets, typical evaluation metrics for visual relation detection cannot reveal improvements of under-represented relations. To address this problem, we propose using Macro Recall@K for visual relation detection. Our experiments confirm that the performance of state-of-the-art visual relation detection approaches can significantly be improved by utilizing depth map information.

:

Relational learning is an established research area in machine learning. State-of-the-art approaches (Nickel et al., 2016) describe relations as triples of the form (subject, predicate, object), such as (Man, rides, Bike). The triples, which all together form a knowledge graph, are typically extracted from structured data such as the infoboxes in Wikipedia and other sources.

In the last years, the detection of relations from images, i.e., visual relation detection, has also gained a lot of interest. One reason is that understanding relations between entities can play an important role in decision making. For example, detecting whether a man is on a bike or next to a bike is a crucial challenge in autonomous driving. State-of-the-art works in this area utilize information of objects in the scene such as class labels, bounding boxes and RGB features, to capture pairwise relations using different models. In this paper, we argue that relation detection can additionally benefit from objects' 3D information. This information can help to distinguish between many relations such as behind, in front of and even improve detection in situations where the objects are nearby such as covered in.

Unfortunately, most available datasets, specifically the ones with relational annotations such as Visual Relation Detection (VRD) (Lu et al., 2016) and Visual Genome (VG) (Krishna et al., 2017) , lack this 3D information since acquiring them is a cumbersome task requiring specialized hardware. We solve this issue by synthetically generating the corresponding depth maps of images in these datasets. Depth maps provide the objects' distance from the camera and the availability of large corpora of RGB-D pairs, e.g. NYU-Depth-v2 (Nathan Silberman & Fergus, 2012) dataset enables us to learn the mapping between any RGB image to its corresponding depth map using a fully convolutional neural network. We can then apply the trained network on images from VG, converting them into depth maps. We call this dataset extension VG-Depth1 .

The extracted features from depth maps, together with other object information extracted from the RGB images, are the basis for relation detection in our framework. We note that the typically employed Recall@K metric, which we call as Micro Recall@K, cannot properly reveal the improvements of under-represented relations in highly imbalanced datasets such as Visual Genome. This might be an issue in applications such as autonomous driving where it is important to ensure that the model is capable of predicting also more important but less represented predicates such as walking on (648 in VG test set) and not just wearing (20,148 in VG test set). We tackle this issue by proposing to employ Macro Recall@K, where the Recall@K is computed per predicate class and averaged over, thereby eliminating the effect that over-represented classes have in Micro Recall@K setting. Reporting on both Macro@K and Micro@K, we study the effect of each object information on visual relation detection.

In summary, our contributions are as follows:

1. We perform an extensive study on the effect of different object information in visual relation detection, including 3D information. We show in our empirical evaluations using the VG dataset, that our model can outperform competing methods by a margin of up to 8% points, even those with contextualization or using information extracted from external language sources. 2. We propose using Macro Recall@K to better reveal the improvements in visual relation detection when dealing with highly imbalanced datasets such as Visual Genome. 3. To compensate for the lack of 3D data in Visual Genome, we use an RGB-to-Depth model trained on separate corpora of available pairs in both modalities and apply it to VG images and releasing a new dataset, VG-Depth.

In general, in this work we aim to answer the following questions:

1. If we are given only depth maps of unknown objects in a scene, how accurately can we infer the distribution of possible pairwise relations? How do other sources of object information compare to it? 2. Current visual relation detection frameworks commonly rely on extensive object information such as class labels, bounding boxes, RGB features, contextual information, etc. Does it bring any additional information, if we also include depth representations in these frameworks or would that only bring redundant knowledge about the scene? 3. Do the current evaluation metrics properly reflect the improvements of under-represented relations within highly imbalanced datasets?

1. RELATED WORKS:

Knowledge Graph Learning In Knowledge Graph learning, the aim is typically to find embeddings or latent representations for entities and predicates, which then can serve to predict the probability of unseen triples. These methods mostly differ in how they model relations. In RESCAL (Nickel et al., 2011) each relation is defined as a linear transformation in the embedding space of entities, producing a triple probability. TransE (Bordes et al., 2013) employs a similar idea but limits each relation to a translation. In comparison to RESCAL, it has fewer parameters; as a disadvantage, it cannot model symmetric relations. DistMult (Yang et al., 2014) considers each relation as a vector, similar to TransE, but minimizes the trilinear dot product of subject, predicate and object vector. DistMult can also be understood as a form of RESCAL, where the transformation matrix is diagonal. ComplEx (Trouillon et al., 2016) extends DistMult to complex-valued vectors of embeddings. A multilayer perceptron (MLP) architecture (Dong et al., 2014) extends these methods to non-linear transformations and has shown to be competitive to the other discussed approaches on most benchmarks (Nickel et al., 2016; Socher et al., 2013) .

Visual Relation Detection Visual relation detection received a huge boost by the availability of large corpora of annotated images such as the Visual Relation Detection (VRD) (Lu et al., 2016) and the VG (Krishna et al., 2017) , containing the visual form of entities, and relations. In VRD, Word2Vec representations of the subject, object, and the predicate were used to train a model jointly with the corresponding image section describing the predicate. In particular, they consider the joint bounding box of subject and object as the image representation for the predicate. Follow-up work achieved improved performance by incorporating a knowledge graph, constructed from the annotated triples in the training set (Baier et al., 2017) . In general, the distribution of the predicate bounding box in these works is much more long-tailed than of entities alone. Therefore, separating the models for objects and predicates, as employed in VTransE (Zhang et al., 2017) reduces the complexity of training such a model. VTransE is a generalization of TransE to visual relation detection in which the last convolutional layer of the image detector, together with the location of entities and their class labels, is taken as the input vector to the TransE algorithm. More recently, Yu et al. (Yu et al., 2017) proposed a teacher-student model to distill external language knowledge to improve visual relation detection. Iterative Message Passing (Xu et al., 2017) , Neural Motifs (Zellers et al., 2018) and Graph R-CNN (Yang et al., 2018b) incorporate context within each prediction using RNNs and graph convolutions respectively.

Depth Maps While several works have leveraged depth maps to improve object detection (Bo et al., 2013; Eitel et al., 2015; Gupta et al., 2014) , the idea of using depth maps in the relation detection task has only been explored in a recent parallel work (Yang et al., 2018a) . While the focus of that work is on human-centered relations, we carry out our studies on a broader set of relations (VG), releasing a dataset extension, and also providing a more extensive study on the role of different object information in visual relation detection, with a competent metric.

2. FRAMEWORK:

In this section we introduce the general framework employed for this study. Let E = {e 1 , e 2 , ..., e n } be the set of all entities, including subjects(s) and objects(o), and P = {p 1 , p 2 , ..., p m } the set of all predicates. Each entity e i can appear in images within a bounding box bb i = (x i , y i , w i , h i ), where (x i , y i ) are the coordinates of the bounding box and (w i , h i ) are its width and height. In this work we apply the pre-trained (on ImageNet (Russakovsky et al., 2015) ) and fine-tuned (on Visual Genome) Faster R-CNN (Ren et al., 2015) on each image I to extract a feature map fmap I , together with object proposals as a set of bounding boxes bb and class probability distributions c. For each RGB image, we generate a depth map D where the same bounding box areas encompass the entities' distance from the camera. In the next section, we first describe the employed network for synthetic generation of Ds and then discuss the extraction of depth features. In the end, we describe the relation detection head where the late fusion of pairwise features and their relational modelling takes place.

2.1. DEPTH MAPS FOR RELATION DETECTION:



2.1.1. GENERATION:

We incorporate an RGB-to-Depth model within our visual relation detection framework. As shown in Figure 2 , this is a fully convolutional neural network (CNN) that takes an RGB image as input and generates its predicted depth map. This model can be pre-trained on any datasets containing pairs of RGB and depth maps regardless of having the object or predicate annotations. This enables us to work with currently available visual relation detection datasets without requiring to collect additional data, and also mitigates the need for specialized hardware in real-world applications. The architectural details are explained in Section 3. On the left side, we see the RGB image and its depth map. We use separate CNNs to extract feature maps from both modalities and create pairwise feature vectors d so (pooled from depth feature maps), l so (from bounding boxes), c so (from class labels) and v so (from the pooled visual features). These vectors are then concatenated and fed into a relation detection layer to infer the predicate.

2.1.2. FEATURE EXTRACTION:

Depth maps have been employed in tasks such as object detection and segmentation (Eitel et al., 2015; Hazirbas et al., 2016) . In those works, it is common to simply render a depth map as an RGB image and extract depth features using a CNN pre-trained for RGB images. There, it has been argued that the edges in depth maps might yield better object contours than the edges in cluttered RGB images and that one may combine edges from both RGB and depth to obtain more information (Hazirbas et al., 2016) . Therefore, they aimed to get similar, complementary features from both modalities. However, the practice of employing a model pre-trained on a particular source modality, e.g. RGB, and applying it on a different target modality, e.g. depth map, is sub-optimal in many applications2 . Therefore, similar to other works on depth-based image segmentation or classification we employ a CNN, to generate a feature map fmap D from an input depth map, but unlike those works, we train this network from scratch, using depth maps and specifically for relation detection.

2.2. RELATION MODEL:

In the previous section, we described methods for the extraction of individual object features. Here, we outline the model that infers relations using pairwise combination of these features. For each pair of detected objects within an image, we create a scale-invariant location feature l s = (t x , t y , t w , t h ) with: 

t x = (x s -x o )/w o , t y = (y s -y o )/h o , t w = log(w s /w o ), t h = log(h s /h o )

; l o ].

Each of these vectors are fed into separate layers, yielding v so , l so , c so and d so before being concatenated altogether and fed to the relation head which projects them to the relation space such that:

EQUATION

Here, W describes a linear transformation and f is a non-linear function. We realize this as fully connected layers in a neural network with ReLU activations and dropout. e p is the embedding vector for predicate which will be learned jointly with other parameters. This simple relation prediction model is a generalization of the model used by (Dong et al., 2014) to construct knowledge graphs.

In that paper, latent features for the predicates are also part of the network input, whereas here we use a separate output for each predicate which is easier to train regarding negative sampling, and also has fewer parameters. As shown in earlier works, using more sophisticated models for context propagation between objects with RNNs or graph convolutions, can further improve the prediction accuracy. However, the aim here is to study the effect of including depth maps as additional object representations in visual relation detection and as will be shown later, even with this simple model, they can be a more effective supplement than e.g. propagated context3 . Clearly, those other models can also further enrich their understanding of object relations, by employing these additional representations.

To learn the parameters, we consider each relation (s, p, o) with an associated Bernoulli variable that takes 1 if the triple is observed and 0 otherwise, following a locally closed world assumption (Nickel et al., 2016) . Given the set of observed triples T , the loss function is the categorical cross entropy between the one-hot targets and the distribution obtained by softmax over the network's output defined as:

EQUATION

where w p is the weight vector corresponding to p in the last layer (linear classification layer).

3. EVALUATION:



3.1. DATASETS:

We test our approach on the Visual Genome (Krishna et al., 2017) dataset. We use the more commonly used subset of VG dataset proposed by (Xu et al., 2017) which contains 150 object classes and 50 relations with 75,651 images used for training and (5000 for validation) and 32,422 images for testing.

3.2. ARCHITECTURES:

RGB-to-Depth Network: We use the RGB-to-Depth model architecture introduced in (Laina et al., 2016) which is a fully convolutional neural network built on ResNet-50 (He et al., 2015) , and trained in an end-to-end fashion on data from NYU Depth Dataset v2 (Nathan Silberman & Fergus, 2012) .

Training on the outdoor images from Make3D dataset (Saxena et al., 2007) instead, did not show promising results in our framework. This observation is not surprising since Make3D images contain mostly outdoor scenes with too few objects.

RGB Feature Extraction: To extract embeddings and class probabilities of RGB images, we use the VGG-16 architecture (Simonyan & Zisserman, 2014) pre-trained on ImageNet (Russakovsky et al., 2015) and fine-tuned to our data for relation detection.

Depth Map Feature Extraction: For depth map extraction we use ResNet18 proposed in (He et al., 2015) . We trained this model from scratch following the earlier discussions in Subsection 2.1.2. We trained this network on a pure depth-based, relation detection task using Adam (Kingma & Ba, 2014), with a learning rate of 10 -4 and batch size of 32 for eight epochs.

Relation Detection Network: Finally, given the features extracted from previous models with the location features described in 2.1.2, we trained our relation detection model. We connected each feature pair described in the previous section, separately to a fully connected hidden layer of 64, 200, 4096 and 20 neurons each, with a dropout rate of 0.1, 0.8, 0.8 and 0.1, with a scaling layer initialized as 1.0, 0.3, 0.5 and 1.0. The penultimate layer contains 4096 neurons with 0.2 dropout. We trained this network by Adam (Kingma & Ba, 2014) , with a learning rate of 10 -4 . We used a batch size of 16 and eight epochs of training. All of the layers were initialized with Xavier weights (Glorot & Bengio, 2010) . During the training of this network, RGB Feature Exraction and RGB-to-Depth network weights were frozen. 3.3 METRICS Some relations might not be annotated in the test set. Still, due to the model's generalization, they might get higher prediction values than the annotated ones. Therefore, similar to previous works (Nickel et al., 2016; Krishna et al., 2017) , we report Recall@K (R@K). Recall per image is defined as the ratio of ground truth labels that appear in the model's top K predictions. However, as the distribution of relations in VG is highly imbalanced, this score is dominated by frequently labeled relations and might not reflect the improvements in some important but under-represented classes. Thus, we propose to compute Macro R@K and name the former computation metric as Micro R@K. For a discussion upon Micro versus Macro for F1 measure refer to (Schütze et al., 2008) . In Macro R@K, we compute Micro R@K separately for each predicate class and average over it. In this setting, the prediction accuracy of under-represented classes has the chance to have a stronger effect on general accuracy.

3.4. COMPARING METHODS:

We compare our results with VTransE (Zhang et al., 2017) that takes visual embeddings and projects them to relation space using TransE. We also compare to the student network of Yu et al. ( 2017) (Yu's-S) , and their full model (Yu's-S+T) that employs external language data from Wikipedia. In the context propagating methods we report Neural Motifs (Zellers et al., 2018) , Graph R-CNN (Yang et al., 2018b) and IMP Xu et al. (2017) . In an ablation study, we report our relation prediction results under several settings in which different combinations of object information are employed for prediction.

3.5. EXPERIMENTS:

Our main goal is to investigate the role of depth maps and other object information in relation detection. Therefore, we do not focus on improving the object detection accuracy and report predicate prediction results. In this setting, the relation detection performance is isolated from the object detector's error by using ground truth bounding boxes of the entities and their class labels to predict the most likely predicates. We carried on our experiments by training each model 8 times with different random seeds. The maximum variance was no more than 0.01. In the following we provide a discussion over the quantitative and qualitative results. The complete predicate prediction results are shown in Table 1 . The upper part of the table demonstrates the results directly reported from other works while the lower two parts present the results from ablation study on our model. For NM, we have computed the Macro R@K results from their available model and code (reproducing the same Micro R@K results as reported in their paper). We can see that our full model achieves the highest accuracy in comparison to the others in all settings specially under Macro R@K. It is also interesting to note that when using only depth maps we can already achieve a significant accuracy in predicate prediction, emphasizing the value of relational information stored within depth maps. We can observe the improvements that externally generated depth maps can bring to Ours-v by comparing it to Ours-v, d. Also comparing Ours-c, d, l to Ours-c, v, l is specially informative from two aspects: (1) It shows that while some results are almost equal in Micro settings, one can observe a significant difference under Macro settings, thus observing the changes in under-represented classes.

(2) Considering that v alone has a better R@K than d alone when adding them separately to c, v we can see that d has more to offer. In other words, v has more redundant information for c, v compared to d. One should note that the improvement is not a linear function of extra information and the combination of different information sources might have non-linear effects on the outcome.

To get a better intuition of the improvements that we gain after including depth maps (Ours-v, d) to Ours-v, we plotted the changes in prediction accuracy for each predicate in Figure 4 . We used darker shades for over-represented classes and lighter shades for under-represented ones. This helps to also gain a better intuition of improvement versus representativeness. We can see that in general the accuracy of relations including the predicates such as under, in front of and behind has been improved. However, we also observed in our data, that in some cases the predicate behind was mislabeled after adding the depth maps. To further explore the potential reason behind some of such successes or failures, we further examined the test images. Figure 3 gives an example of a failure and a success case in predicting behind. While in the right image the annotations indicate that (bushes, behind, bench) based on the corresponding depth map, they are in the same distance from the camera (bushes can be seen in the left corner of the image). One can argue that in fact, bushes are next to the bench rather than behind, indicating the noisy ground truth. However, it is also important to note that humans describe a predicate from their point of view which is not necessarily associated with the camera's point of view. This effect might appear in many other instances within the dataset, leading to worse performances when dealing with such predicates. A simple way to overcome such problems is by having a richer set of data.

We provide some samples of generated depth maps from VG images, with high and low quality in Figure 6 and 5. Note that the potential imprecisions within generated low-quality depth maps, e.g. sky and reflective objects such as glasses, are inevitable.

4. CONCLUSION:

We identified 3D information as an important attribute for visual relation detection. Since this information is typically not provided in visual data sets, we employed an RGB-to-Depth network, which was pre-trained on a large corpus of data, to generate depth information. We provided a metric, Macro R@K for a better evaluation. In extensive empirical evaluations, we demonstrated the effect of An improvement in predicates with more frequency has a larger effect on the Micro R@K whereas this effect is eliminated within Macro R@K. We can see that indeed the improvements by using depth maps are mostly happening within the less-represented classes. different object information in visual relation detection and showed that by using depth information, one achieves significantly better performance compared to other state-of-the-art methods.

This dataset will be made publicly available upon publication.

One should also keep in mind that even fine-tuning some layers of a network does not change the very early convolutional filters.

We carried on Visual Genome experiments in an isolated framework, where all pre-trained weights and hyper-parameters, except for the depth map channel, were kept identical as the ones employed by Neural Motifs(Zellers et al., 2018). The code will be released upon publication.

Figures and Tables:

Figure 1: An image from the Visual Genome dataset showing several men sitting next to each other and the corresponding generated depth map. Bright colors indicate a larger distance to the camera. Knowing the depth of, e.g., the individuals, the house and the car, plays an important role in detecting relations such as next to and behind.

Figure2: We propose utilizing 3D information in visual relation detection by synthetically generating depth maps using an RGB-to-Depth model incorporated within relation detection frameworks. On the left side, we see the RGB image and its depth map. We use separate CNNs to extract feature maps from both modalities and create pairwise feature vectors d so (pooled from depth feature maps), l so (from bounding boxes), c so (from class labels) and v so (from the pooled visual features). These vectors are then concatenated and fed into a relation detection layer to infer the predicate.

Figure3: Using depth maps improves the understanding of perspective and increases the general detection rate of relations such as behind (e.g. right image). However, in some cases such as in the left image, after including depth maps, (bushes, behind, bench) has been mislabelled. One should note that here the bushes (left corner of the image) have similar depth values as the benches. Equal depth values can be a strong indicator of the predicate next to rather than behind. In this case, the human point of view seems to have caused a labelling error recovered by the depth-based model.

Figure 4: This plot shows the prediction changes per predicate, going from Ours-v to Ours-v, d. The classes with zero changes are omitted from the plot. The darker shades indicate larger number of that class within the test set whereas the lighter shades are under-represented classes.An improvement in predicates with more frequency has a larger effect on the Micro R@K whereas this effect is eliminated within Macro R@K. We can see that indeed the improvements by using depth maps are mostly happening within the less-represented classes.

Figure 5: Examples of high quality depth maps generated from visual genome images.

Figure 6: Examples of low quality depth maps generated from visual genome images.

and similarly l o . We then pool the corresponding features v s and v o from fmap I and create a visual feature vector [v s ; v o ]. Similarly, we create a depth feature vector [d s ; d o ], by extracting bb s and bb o from fmap D and a class vector [c s ; c o ] and similarly [l s

Table content: <table/>

Predicate prediction recall values on VG test set. We have omitted so for better readability. When the depth maps are utilized together with all other features (Ours-l, c, v, d), we gain a large improvement compared to the state-of-the-art. In Macro setting, this improvement is almost as large as Ours-c to Ours-c, v, l, demonstrating the importance of depth features in relation detection. One can also see that even replacing depth maps with visual features (Ours-l, c, d compared to Ours-l, c, v) can yield better results. Additionally, comparing Ours-c, v, l to VTransE and Neural Motifs reveals the advantage of our simple model regardless of depth maps.

Table content: <table><tr><td/><td>Strategy</td><td/><td>Macro</td><td/><td/><td>Micro</td><td/></tr><tr><td/><td>Task</td><td colspan="3">Predicate Pred.</td><td colspan="3">Predicate Pred.</td></tr><tr><td/><td>Metric</td><td colspan="6">R@100 R@50 R@20 R@100 R@50 R@20</td></tr><tr><td/><td>VTransE (Zhang et al., 2017)</td><td>-</td><td>-</td><td>-</td><td>62.87</td><td>62.63</td><td>-</td></tr><tr><td/><td>Yu's-S (Yu et al., 2017)</td><td>-</td><td>-</td><td>-</td><td>49.88</td><td>-</td><td>-</td></tr><tr><td>models</td><td>Yu's-S+T (Yu et al., 2017) IMP (Xu et al., 2017) Graph R-CNN (Yang et al., 2018b)</td><td>---</td><td>---</td><td>---</td><td>55.89 53.00 59.10</td><td>-44.80 54.20</td><td>---</td></tr><tr><td/><td>NM (Zellers et al., 2018)</td><td>14.39</td><td>13.20</td><td>10.25</td><td>67.10</td><td>65.20</td><td>58.50</td></tr><tr><td/><td>Ours -d</td><td>9.51</td><td>8.46</td><td>6.35</td><td>54.72</td><td>51.90</td><td>43.86</td></tr><tr><td/><td>Ours -c</td><td>15.65</td><td>13.09</td><td>8.56</td><td>64.82</td><td>60.54</td><td>49.89</td></tr><tr><td>ablations</td><td>Ours -v Ours -l Ours -v, d Ours -l, v, d</td><td>13.88 5.19 15.47 15.76</td><td>12.24 4.66 14.04 14.40</td><td>8.99 3.57 10.83 11.07</td><td>61.72 49.07 62.88 63.06</td><td>58.50 46.13 60.52 60.83</td><td>50.41 37.48 53.07 53.55</td></tr><tr><td/><td>Ours -l, c, d</td><td>21.67</td><td>19.56</td><td>15.12</td><td>67.97</td><td>66.09</td><td>59.13</td></tr><tr><td/><td>Ours -l, c, v</td><td>19.16</td><td>17.72</td><td>13.93</td><td>67.94</td><td>66.06</td><td>59.14</td></tr><tr><td/><td>Ours -l, c, v, d</td><td>22.72</td><td>20.74</td><td>16.40</td><td>68.00</td><td>66.18</td><td>59.44</td></tr></table>

Peer review (iclr2020):
********* Post Rebuttal ********* I appreciate the authors* effort in providing thorough responses and revised manuscript. I agree with the authors that *the finding not being surprising* is not a ground for rejection. I tried to word my final decision carefully but it seems it has still caused confusion for the authors. As I have mentioned in my original review, the rating was a result of the 4 points considered *together* . That is, if one exploits privileged information that needs extra sensory data and/or annotation (point 2), *and*, this privileged information is clearly related and thus should be normally useful for the final task (point 1), *and* achieve marginal improvements (point 3), it can be a ground for rejection. Especially, given that prior works with similar arguments exists (point 4). The rebuttal has alleviated the issue of marginal improvements (point 3) by introducing meanR@K (or as the revised paper refer to it, Macro R@K). Here, the improvements are more significant both compared to the state of the art and ablated baselines. The authors also argue that the related [Yang et al. 2018] paper (point 4) should be considered a concurrent submission since the authors original submission was to AAAI18. The rebuttal also addresses other clarity or experimental issues which improves the quality of the revised work. Finally, I understand that the privileged information is only required during training time which is a good point. All in all, *assuming that [Yang et al. 2018] is considered a concurrent work* according to ICLR, I think the revised paper becomes slightly above borderline and thus I change my rating to *weak accept*. If [Yang et al. 2018] is not considered concurrent work, then, a conclusive comparison is required for the acceptance of the current work. ********* Summary ********* The paper poses the question of whether depth information is informative for visual relationship prediction using still images. It is intuitive that 3D arrangement of objects in an image can be a useful cue for predicting their relationship. As such it is important to see whether and to what extent depth information complements RGB information for visual relation detection. That is the focus of this paper. The paper proposes to use an off-the-shelf monocular depth estimation networks to augment the available RGB information towards better visual relation detection. For that, it proposes a specific network two-stream structure working on RGB image and (predicted) depth image. The proposed model demonstrates improved results upon state of the art for visual relation prediction. ********* Strengths and Weaknesses ********* + A comprehensive set of tests has been conducted. + Zero-shot prediction results are particularly interesting. + The experiment on ranking the predicate classes based on the change in prediction accuracy before and after using depth information (Figure 4) is interesting and intuitive. * The final results improve upon the state-of-the-art, especially on the zero-shot learning regime. However, it seems that the improvement is mainly coming from the new architecture as opposed to the inclusion of the depth information. That is, ours_{c,v,l} brings most of the improvement already the last step to ours_{c,v,l,d} is negligible for non-zero-shot case. - Along the same line, it’s possible that this small difference between ours_{c,v,l} and ours_{c,v,l,d} for the standard predicate prediction, can be due to a hyper-parameter optimization that is (only or more thoroughly) done for ours_{c,v,l,d}. The hyper-parameter optimization scheme for different baselines is not described. - Given the small difference of ablation levels, the comparison will be stronger if done multiple times and reporting mean and standard deviation of the results. - For a fair comparison the visual feature vector v_{so} should be tried as the feature of the union bound box of both subject and object same way as it is done for depth feature vector d_{so}. - The paper refers to “Ours-d’_{so}” as a baseline that *only* uses depth information with no image/label information. However, it seems that the region proposals for this feature are coming from the image-based network that uses image information. - Important related but uncited works: (1) [“Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information” ECCV workshops 2018] studies the same question as part of their work. ********* Final Decision ********* I do not find the paper passing the acceptance bar mainly due to the following reasons together: 1) The finding is not surprising since most of the visual relations are either explicitly depth-related (e.g., behind) or are semantically constrained by depth (e.g. riding cannot happen at different depths when the image is taken orthogonal to the rider). 2) an additional depth dataset is used which provides the model with privileged information. Should it have been the case that depth information were inferred without an additional offline dataset, the results would have been more interesting. 3) the improvements due to the additional depth network are not significant or conclusive. 4) there is a prior uncited work with the same research question for effectiveness of depth information in visual relation detection which uses a similar approach. ********* Minor points ********* - the code is not available. This is especially important since the paper is outperforming prior works which could be a contribution if reproducible. - Section 2.2: is l_{so} concatenation of l_s and l_o? - Section 2.2: y_{spo} is defined but never used. - Equation 2: why do we have both e_p and f in the exponents? Aren’t they the same? - Equation 2: P is never defined. - Page 5: “a fully connected hidden layer of 64, 200, 4096 and 20 neurons”: this amounts to 3 hidden layers. - Why VGG network for visual feature and AlexNet for depth features? - zero-shot learning results on visual genome is missing - training procedure is a bit unclear: the text suggest that the fine tuning and/or learning of the three components might happen separately. It is important to clearly state if they are done in an end-to-end fashion and simultaneously or separately; and why. - It’s good to name the method in table 2 in the same fashion as table 1. With the current naming (based on architecture) it is a bit confusing to understand the content without additional cross referencing. For instance AlexNet-BN - Raw seems to correspond to Ours_{c,v,l,d} - Figure 4: the frequency represented as different shades of red or blue is really hard to notice especially on a printed paper. The red vs blue color coding is not necessary since the bars going up or down indicate the same quality. So, it might be better to use red/blue for frequency instead (e.g. dark red high frequency to dark blue low frequency) - Section 3.2: the AlexNet reference seems wrong, it should be *ImageNet Classification with Deep Convolutional Neural Networks* NIPS , 2012 - The structure of section 3.5 is currently flat while the content seems to be nested (two experiments and two sets of corresponding discussions). It will read better if they are organized into subsections. ********* Points of extensions (improvement) ********* - I believe *unsupervised* discovery of depth information for visual relation detection can be an interesting direction since it is not limited to the availability of relevant depth dataset. - It is not clearly motivated why one should use two separate networks for depth and RGB inputs in light of the additional complexity. For instance, it is good to discuss what is the advantage of the proposed (computationally more expensive) method over the following two simpler baselines: - Faster RCNN is used on RGBD input to produce a single feature vector - above case with RGB input but have the Faster RCNN predict the depth map as an auxiliary loss.