Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: OVERVIEW:
The authors propose to use depth information to better predict the visual relation between objects in an image. They do this by incorporating a pre-trained RGB-to-Depth model within existing frameworks. They claim the following contributions:
1. First to utilize 3D information in visual relation detection. They synthesize depth images for existing benchmark datasets of VRD and VG using a pre-trained RGB-to-Depth model trained on NYUv2 to generate RGB-D data for visual relation detection.
2. Discuss and empirically investigate different strategies to extract features from depth maps for relation detection.
3. Study the quantitative and qualitative benefits of incorporating depth maps. "We show in our empirical evaluation using the VRD and VG datasets, that models using depth maps can outperform competing methods by a margin of up to 3% points".

MAJOR COMMENTS:
1. I liked the idea of using depth information to inform visual relationships but I am not sure if the proposed approach is the way to go. Given a depth image of the scene, we can generate a reconstruction of the scene in 3D, even if it is partial/imperfect. Direct reasoning in 3D should now be possible instead of going via deep networks as proposed in the paper. I believe a direct 3D approach would make a meaningful baseline at the very least and needs to be discussed.
2. The authors use a pre-trained RGB-to-Depth network trained on NYU-v2 to predict depth for the images of VRD and VG. There is very little discussion about the quality of predicted depth maps. Ideally, this needs to be quantified to convince the reader that the generated depth maps are "good" but at the very least the authors need to show qualitative examples (both good, typical and bad) to prove that the pre-trained network generates meaningful depth maps.
3. To use a siamese (shared weights) feature extractor between RGB and Depth images or not, is not a significant contribution by itself. In principle, separate feature extractors lead to larger model complexity/learning capability and make sense given domain separation between RGB and Depth. 

MINOR COMMENTS:
1. Figure 2 seems to indicate that a Faster-RCNN is used on both RGB and Depth steams which is backed up by text in Section 2 (first paragraph). However, in Section 3.2, under RGB Feature Extraction and Depth Map Feature Extraction, the discussion is about VGG-16 and AlexNet-BN networks. The VGG-16 network is pre-trained in ImageNet and finetuned to relevant data but it is not clear for what task? If the task is object detection, it needs to be trained for it (not fine-tuned, unless it is being initialized from COCO pre-training). The AlexNet-BN depth model is trained for relation detection using only depth. But it is not clear if it is using proposals/boxes generated by RGB detection model or using ground-truth boxes. Basically, the object-detection component of the pipeline is not clear at all.

NOTE:
I would like to mention that I have published in monocular object pose estimation and work in the object recognition. I am not as familiar with the visual relation detection field but I understand all the components proposed by the authors in this work. I believe I understood the paper and reviewed it fairly (to the best of my ability).
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes to leverage the depth information for relation prediction, arguing that the depth information benefit the prediction of some predicates. To solve the lack of 3D data, an RGB-to-Depth model is trained on external available dataset and then applied to images from visual relation dataset. In the experiments, they investigate different strategies to extract features from depth maps and the explore effectiveness of depth information by comparing the model that only used depth map as input with those which use RGB information. The comparisons with other methods and ablation studies under both Zero-shot setting and normal setting demonstrate the effectiveness of depth information.

+Strength:
(1) The motivation is reasonable and what the authors make an attempt to explore is very meaningful. Visual relation especially the spatial relation is not likely to be predicted accurately without 3D information. In other words, it seems that visual relation prediction task will be extended to 3D images rather than staying within 2D images. Thus what the authors do is a good exploration for further extensions.
(2) Comparisons with previous methods and the results show that the depth information is useful to some extent, but not so obvious.
(3) The writing of this article is good and it’s very easy to understand.

-Weakness:
(1) The RGB-to-Depth Network is pretrained on other dataset. Is there any gap when it is used for VG or VRD dataset? 
(2) Although the depth map feature extraction seems to work well, it seems to be a little trivial. Why a CNN, e.g. AlexNet, or VGG, can be used to extract depth features? And why the AlexNet trained from scratch performs better than AlexNet pretrained on RGB images for object detection task and VGG net? If the author can give more explanations, this part will be more insightful.
(3) From the plot which shows the top 10 percent absolute changes in prediction performance per predicate, the advantage of Depth is not obvious compared with RGB. And Depth does not bring the advantage claimed in Abstract. It’s a little hard to understand why depth information can rectify the prediction of (Tower, taller, trees). To sum up, the qualitative results are not so satisfying.
(4) In Table 1, what really functions seems to be c_so, v_so, and l_so, while the improvement brought by depth is limited.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: 

********* Post Rebuttal *********

I appreciate the authors' effort in providing thorough responses and revised manuscript. 

I agree with the authors that "the finding not being surprising" is not a ground for rejection. I tried to word my final decision carefully but it seems it has still caused confusion for the authors. As I have mentioned in my original review, the rating was a result of the 4 points considered *together* . 

That is, if one exploits privileged information that needs extra sensory data and/or annotation (point 2), *and*, this privileged information is clearly related and thus should be normally useful for the final task (point 1), *and* achieve marginal improvements (point 3), it can be a ground for rejection. Especially, given that prior works with similar arguments exists (point 4). 

The rebuttal has alleviated the issue of marginal improvements (point 3) by introducing meanR@K (or as the revised paper refer to it, Macro R@K). Here, the improvements are more significant both compared to the state of the art and ablated baselines.

The authors also argue that the related [Yang et al. 2018] paper (point 4) should be considered a concurrent submission since the authors original submission was to AAAI18. 

The rebuttal also addresses other clarity or experimental issues which improves the quality of the revised work.

Finally, I understand that the privileged information is only required during training time which is a good point.

All in all, *assuming that [Yang et al. 2018] is considered a concurrent work* according to ICLR, I think the revised paper becomes slightly above borderline and thus I change my rating to "weak accept". If [Yang et al. 2018] is not considered concurrent work, then, a conclusive comparison is required for the acceptance of the current work.


********* Summary *********
 
The paper poses the question of whether depth information is informative for visual relationship prediction using still images. It is intuitive that 3D arrangement of objects in an image can be a useful cue for predicting their relationship. As such it is important to see whether and to what extent depth information complements RGB information for visual relation detection. That is the focus of this paper.
The paper proposes to use an off-the-shelf monocular depth estimation networks to augment the available RGB information towards better visual relation detection. For that, it proposes a specific network two-stream structure working on RGB image and (predicted) depth image. The proposed model demonstrates improved results upon state of the art for visual relation prediction.
 

********* Strengths and Weaknesses *********
 
+ A comprehensive set of tests has been conducted. 
+ Zero-shot prediction results are particularly interesting.
+ The experiment on ranking the predicate classes based on the change in prediction accuracy before and after using depth information (Figure 4) is interesting and intuitive.
* The final results improve upon the state-of-the-art, especially on the zero-shot learning regime. However, it seems that the improvement is mainly coming from the new architecture as opposed to the inclusion of the depth information. That is, ours_{c,v,l} brings most of the improvement already the last step to ours_{c,v,l,d} is negligible for non-zero-shot case.
- Along the same line, it’s possible that this small difference between ours_{c,v,l} and ours_{c,v,l,d} for the standard predicate prediction, can be due to a hyper-parameter optimization that is (only or more thoroughly) done for ours_{c,v,l,d}. The hyper-parameter optimization scheme for different baselines is not described. 
- Given the small difference of ablation levels, the comparison will be stronger if done multiple times and reporting mean and standard deviation of the results.
- For a fair comparison the visual feature vector v_{so} should be tried as the feature of the union bound box of both subject and object same way as it is done for depth feature vector d_{so}. 
- The paper refers to “Ours-d’_{so}” as a baseline that *only* uses depth information with no image/label information. However, it seems that the region proposals for this feature are coming from the image-based network that uses image information. 
 
- Important related but uncited works:
(1) [“Visual Relationship Prediction via Label Clustering and Incorporation of Depth Information” ECCV workshops 2018] studies the same question as part of their work.
 

********* Final Decision *********
 
I do not find the paper passing the acceptance bar mainly due to the following reasons together:
1) The finding is not surprising since most of the visual relations are either explicitly depth-related (e.g., behind) or are semantically constrained by depth (e.g. riding cannot happen at different depths when the image is taken orthogonal to the rider).
2) an additional depth dataset is used which provides the model with privileged information. Should it have been the case that depth information were inferred without an additional offline dataset, the results would have been more interesting.
3) the improvements due to the additional depth network are not significant or conclusive.
4) there is a prior uncited work with the same research question for effectiveness of depth information in visual relation detection which uses a similar approach.
 

********* Minor points *********

- the code is not available. This is especially important since the paper is outperforming prior works which could be a contribution if reproducible.
- Section 2.2: is l_{so} concatenation of l_s and l_o?
- Section 2.2: y_{spo} is defined but never used.
- Equation 2: why do we have both e_p and f in the exponents? Aren’t they the same?
- Equation 2:  P is never defined.
- Page 5: “a fully connected hidden layer of 64, 200, 4096 and 20 neurons”: this amounts to 3 hidden layers.
- Why VGG network for visual feature and AlexNet for depth features?
- zero-shot learning results on visual genome is missing
- training procedure is a bit unclear: the text suggest that the fine tuning and/or learning of the three components might happen separately. It is important to clearly state if they are done in an end-to-end fashion and simultaneously or separately; and why.
- It’s good to name the method in table 2 in the same fashion as table 1. With the current naming (based on architecture) it is a bit confusing to understand the content without additional cross referencing. For instance AlexNet-BN - Raw seems to correspond to Ours_{c,v,l,d}
- Figure 4: the frequency represented as different shades of red or blue is really hard to notice especially on a printed paper. The red vs blue color coding is not necessary since the bars going up or down indicate the same quality. So, it might be better to use red/blue for frequency instead (e.g. dark red high frequency to dark blue low frequency)
- Section 3.2: the AlexNet reference seems wrong, it should be "ImageNet Classification with Deep Convolutional Neural Networks" NIPS , 2012
- The structure of section 3.5 is currently flat while the content seems to be nested (two experiments and two sets of corresponding discussions). It will read better if they are organized into subsections.
 
 
********* Points of extensions (improvement) *********

- I believe *unsupervised* discovery of depth information for visual relation detection can be an interesting direction since it is not limited to the availability of relevant depth dataset. 
- It is not clearly motivated why one should use two separate networks for depth and RGB inputs in light of the additional complexity. For instance, it is good to discuss what is the advantage of the proposed (computationally more expensive) method over the following two simpler baselines:
- Faster RCNN is used on RGBD input to produce a single feature vector
- above case with RGB input but have the Faster RCNN predict the depth map as an auxiliary loss.



review_title: Official Blind Review #1

Meta-review: The paper proposes to improve visual relation prediction by using depth maps.  Since existing RGB images do not contain depth informations, the authors use a monocular depth estimation method to predict depth maps.  The authors show that using depths maps, they are able to improve prediction of relations between ground truth object bounding boxes and labels.  

The paper got relatively low scores (with 3 initial weak rejects).  After the revision and suggested improvements, one of the reviewers updated their score so the paper now has 2 weak rejects and 1 weak accept. 

The paper had the following weaknesses:
1. The paper has limited technical novelty as it combines off the shelf components.  The components also used different backbones (ResNet at some places, VGGNet at others) that were directly from prior work.  Was there any attempt to have an unified architecture? As the main novelty of the work is not in the model aspect, the paper needs to have stronger experiments and analysis.
2. More analysis on the quality of the depth estimation is needed.  Ideally, the work should provide some insight into whether some of the errors is due to having bad depth estimation?  The depth estimation method used is from 2016, there are newer depth estimation methods now.  Would having better depth estimation give improved results?  Experiments that illustrates that method works well with predicted bounding boxes instead of ground truth bounding boxes will also strengthen the paper.  
3. There was the question of whether the related Yang et al. 2018 workshop paper should be included as basis for comparison.  In the AC's opinion, Yang et al. 2018 is not concurrent work and should be treated as prior work.  However, it is not clear whether it is feasible to compare against that work.  The authors should attempt to do so and if infeasible, clearly articulate why that is the case.
4. As pointed out by R3, once there is a depth map available, it is also possible to compare against 3D methods (such as those that operate on point clouds)

Overall the paper had a nice insight by proposing the simple but effective idea of using depth information to help with visual relation prediction.  Still the work is somewhat borderline in quality.  In the AC's opinion, the main contribution and insight of the paper is of limited interest to the ICLR community, and it would be more appreciated in a computer vision conference.  The authors are encouraged to improve the paper with stronger experiments and analysis, incorporate various suggestions from the reviewers, and resubmit to a vision conference.

Decision: Reject