Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents an extension of FCOS3D that predicts depth probabilistically and integrates it in a graph neural network to estimate relative depths across an entire scene. The resulting system achieves state of the art mono-depth-based 3D detection results on nuScenes and KITTI.
strengths_and_weaknesses: + Easy to understand (well written)
+ simple and interesting idea with good performance
- Please add FCOS3D to T1 and T2

I like this paper. It's well written, easy to follow and has a refreshingly new take on monocular 3D detection. The paper is well executed, including good ablations and comparisons. There only minor improvements required to make this work publishable at CoRL.

Please add FCOS3D to table 1 and table 2, since PGD is already built on that baseline.

Minor:
* PointPillars [2] in T2 can work much better with the right backbone, consider citing the numbers in [50]
* "Monocular 3D detection, as an economical solution compared to conventional settings relying on binocular vision or LiDAR, has drawn increasing attention recently but still yields unsatisfactory results.": It's unclear to me why a ($50 or less if mass produced) stereo camera is more expensive than a mono-camera with a beefy GPU attached to it (LiDAR I agree is currently still more expensive)? Please comment, or rephrase.
summary_of_recommendation: I like the paper, and hope the authors incorporate the above feedback to lift the submission well above bar.
issues: See above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a monocular depth estimation approach which predicts probabilistic depth (via discrete buckets) for objects in a scene and then performs graph-based reasoning (along with some assumptions about ground location) to refine the single-object estimates. In this way, low uncertainty objects can improve the depth estimates for proximate higher uncertainty ones. The authors validate their approach against multiple competing methods and baselines on two autonomous driving datasets.

Update (Post Author Response):

I appreciate the authors' response to questions. Fundamentally my impression of the paper is unchanged and my initial recommendation of acceptance is unchanged.
strengths_and_weaknesses: The paper proposes an interesting mechanism for injecting additional structure into the monocular depth-estimation setting. They conduct a reasonable assortment of experiments and do a nice job experimentally validating their approach.

My biggest question about this paper is how the method transfers to less structured settings. Would this work in something like a household environment where object ground locations are less consistent? Similarly, what if some objects are not on the ground (e.g. tree branches, hanging signs, etc)? The authors only report results for vehicle detection (which are very well behaved w.r.t. this manually injected inductive bias). I would have also liked to see the results of an ablation looking at only the probabilistic depth formulation (without any of the local geometric consistency or constraints).



Minor comment: Some of the prose is unnecessarily floury and the meaning can be a bit hard to parse in places. I suggest an editing pass focused on shortening sentences and simplifying language as much as possible.
summary_of_recommendation: The paper presents an interesting extension of current monocular depth estimation approaches. While some of the assumptions the authors inject seem overly suited to vehicles, the paper still provides valuable insight to others in the field and their proposed method looks promising.

I dislike the lack of granularity in some of the scores (I don't know why the two positive options for impact are <little> and <a ton>. This paper has the potential to have moderate impact if the results end up being replicated in future works by different groups. This is generally the case for most "good" papers: their impact lies somewhere in between a "3" and a "4".
issues: - Please elaborate on when you expect the assumptions you introduce to be violated and how much of a problem that will be.

- (Minor) Do a general editing pass for flow and clarity.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper describes an approach to improving the performance of monocular 3D object detection systems by improving their performance in *depth estimation* specifically.  The main contributions consist of two parts:

* A quantitative study of the performance-limiting elements of a monocular 3D object detection pipeline.  The paper investigates the effect that several estimation subtasks (depth estimation, offset, size, orientation, etc.) have on overall 3D object detection performance using an *oracle analysis*: in this approach, various subsets of these *estimated* quantities are replaced with their *ground truth* values, in order to determine the sensitivity of end-to-end system performance on each of these estimation subtasks.  The results provide convincing evidence that depth estimation is by far the most influential subtask in determining overall system performance.

* Based on this insight, the paper proposes an approach for improving the performance of instance depth estimation by incorporating contextual depth cues obtained from other nearby detected objects, using a simple geometric argument based upon (i) perspective camera geometry and (ii) the assumption (reasonable in autonomous driving applications) that detected objects are supported by a common ground plane.

Both the oracle investigation and experimental evaluation of the proposed method are evaluated on two standard large-scale datasets (KITTI and nuScenes).

strengths_and_weaknesses: In my estimation the paper makes a solid contribution both scientifically and technologically.  From a scientific perspective, it sets out to investigate a fundamental and clearly practically-important question (namely: what is the “performance limiting step” in an existing monocular 3D object detector?), and devises a very simple but effective means of answering it.  [I want to specifically call out the study reported in Section 3 as particularly praiseworthy: this is one of the most well-thought-out and scientifically rigorous empirical investigations that I have seen amongst the learning-based papers that I have recently read.]  Similarly, from a technological perspective the proposed approach to improve depth estimation by leveraging the geometry of perspective projection is both elegant and (as the experiments show) practically effective.

As a result, I have only a few (relatively minor) comments / suggestions for improving the paper:

* One aspect of the proposed approach that seemed a little odd / mildly unsatisfying to me is that it appears to have at least 3 separate (i.e. independently-estimated) values relating to the estimated depth of an object instance: (i) a direct depth regression D_R, (ii) a “probabilistic depth” D_P (which appears to be modeling a belief over a set of discretized depth intervals), and (iii) a “depth confidence score” S_D (which is used as a kind of weight in the geometric fusion step).  Since the proposed method already incorporates the notion of estimating a distribution P over depth, it seems like all three of the above roles could be filled simply by P itself (i.e. without the need to introduce any additional “fudge factors”).  I would be curious to know: have the authors considered simply replacing the regression of the depth D_R with e.g. a regression of the parameters for a (continuous) *distribution* over depth? For example, a basic implementation of this might involve regressing a mean and covariance for a Gaussian distribution over depth; the mean itself could then take the place of D_R above (as a point estimate), and the covariance would presumably be used in the geometric fusion.  Potential benefits of this approach might be (i) a more streamlined / interpretable regression formulation, (ii) perhaps this framing might lead to e.g. Bayesian fusion formulas for the geometric fusion [in eqs. (7) and (8)], (iii) there would no longer be a need for discretization in the depth estimation, including the choice of binning and spacing parameters.

* As I mentioned earlier, I really liked the design of the oracle study reported in Sec. 3.  However, the figures of merit used in this study are rather complex composite *classification* scores that are designed to assess the performance of an end-to-end monocular 3D object detector, and which appear to combine (in some fashion) multiple distinct measures of performance (sometimes including arbitrary weighting parameters, thresholds, etc.).  While these are certainly informative with respect to the end-to-end downstream task (monocular 3D object detection), since ground truth is available, I would also really have liked to see a more direct assessment of the proposed approach’s performance on the depth estimation task itself.  That is: what is the direct effect of the proposed geometric fusion approach on *depth estimation*, rather than *classification score*? 

* In lines 213 - 215 the paper mentions that gradients are not backpropagated through the geometric depth computation in order to “avoid influencing the learning of other components”.  However, since this computation *is* meant to be used as part of the overall pipeline, this choice seems to be at odds with the idea of joint end-to-end training.  Can the authors perhaps say a bit more about why this decision was made?

* How is lambda chosen in (2)?

* I think “D_L” in line 155 should be “D_R”

summary_of_recommendation: My recommendation is based upon the following considerations:

* The oracle study in Section 3 addresses a fundamental scientific question about the performance of existing monocular 3D object detectors (what is the performance-limiting step?) in a rigorous, thorough, and convincing manner, and arrives at a sharp and practically-useful conclusion (depth estimation).
* On the basis of this study, the paper proposes an elegant and practically-effective method for improving the performance of depth estimation by leveraging perspective geometry.

Both of these contributions are likely to be of interest to a sizable fraction of the robotics community.
issues: I have no major issues (beyond the minor suggestions I mentioned in "Strengths and Weaknesses") that I would specifically like the authors to address during revision.
review_title: null

Meta-review: This paper analyzes FCOS3D by replacing upstream depth predictions with the ground-truth oracle, showing that mAP scores of 3D vehicle detections are substantially improved when depth predictions are no longer a bottleneck. This identifies depth prediction as a large bottleneck in 3D detection and its downstream semantic tasks, so the authors focus on improving depth estimation by using an uncertainty-aware refinement step of the depth. 

Reviewers agree that the work is original ("Good; some advance"), has Good or Excellent clarity, and contains incremental or better contributions to the field of robotics. 

I think reviewer 2epL brings up good comments & suggestions, and I'd like the authors to respond to all of them. In particular I'm also curious about why there are separate predictions D_R, D_P, S_P when they could be computed from D_P itself. 

---- update 20210904
All reviewers have voted to weakly accept and the authors have addressed most reviewer comments, so I am recommending accepting this paper. 

Decision: Accept (Poster)