Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose a method for 3d detection and reconstruction from a single image. The main novelty of the paper in my opinion is the PCA_SDF shape representation which improves inference time and helps capture details in the reconstruction. 
main_review: Strong Points
- The paper improves the SOTA in 3d reconstruction from a single image
- The PCA_SDF improves details while being computationally effiicient.

Weak Points
- The paper does 3d detection using a 3d centernet. It is unclear what is the objective of performing this detection. There is a large body of work on monocular 3d object detection that this paper completely ignores in related work and in experiments. Is the 3d detection a way to regularize the network? Does the network benefit from multi-task effects? 
- The overall novelty of the approach is marginal. Each module is isolation has been well studied. In that case, the main novelty is piecing these individual elements together which is more engineering that research. The fact that the authors do both detection and reconstruction dilutes the contributions of the paper. The authors should focus on reconstruction and dive into details. 
- The PCA_SDF borrows ideas from EIgenSDF [63] and Jiang et. al. [60]. The authors need to do more ablation studies by swapping out their PCA-SDF with eigen-sdf/ local implicit grids and be more thorough about the benefit of the pca-sdf. A lot of work on sdf has emerged since deepsdf and it is important to validate this work against the latest and greatest. 

Post Rebuttal: Based on the response of the authors, and having evaluated the other reviews, I am updating my rating for this paper. Although some of the other reviewers mirrored my concerns of the paper, I agree that the improvement in results with simple innovations is of broad appeal to the research community working on 3D reconstruction. 
limitations_and_societal_impact: The authors do not discuss this. Perhaps, a sentence or two can be added beyond stating that there are no obvious negative impacts. The non-obvious ones are the ones to think and discuss.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a novel method for 3D detection and reconstruction of multiple objects from single images, that combines 2D feature extraction and 2D-3D lifting with a modified (3D) version of the CentreNet detector and  novel coarse-to-fine reconstruction stage.
main_review: Positive:
* The paper is well organised, very well written and easy to understand.
* The approach has several novel components and (i) appears to produce state of the art results while (ii) improving on inference speed over other competing methods.
* The experimental section is quite extensive and does appear to cover most essential benchmarks and ablations.

Negative / Questions:
* Proposed 1-4 is somewhat confusing — maybe put more text in the caption. I’d be good to expand the text on lines 275 - 282.
* What if you had bigger rotation perturbations? At what angled would we expect PCA-SDF to fail?
* Are there any advantages of PCA-SDF over using PCA over a standard occupancy? 
* Is the importance of the positional encoding benchmarked anywhere?
* Did you do any ablation on the size of the PCA-SDF local volume? Would smaller volumes require more fewer basis functions and generalize better, or does the size of the volume not matter much?
* Some typos (eg line 283).

Overall, I am very much in favour of acceptance. The proposed approach is principled and produces very good results, and the paper is very well written.
limitations_and_societal_impact: -
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a joint framework for detecting multiple 3D objects from input images while also reconstructing the 3D shapes. The proposed method improves upon previous work [20,21] for joint 3D object detection and reconstruction mostly at the system/architectural level. The authors also propose a new way of compactly representing 3D shapes with signed distance functions (SDF) using PCA that can be naturally incorporated into the framework. Experiments are benchmarked with synthetic multi-object ShapeNet renderings, Pix3D, and a new dataset with annotations for multi-object detection & reconstruction.
main_review: Strengths:
+ A nice overview of related literature is provided. 
+ The method description is clear, and it is easy to follow and understand. The overview diagram is also helpful in understanding the entire framework.
+ The proposed PCA-SDF representation is particularly interesting, as high-quality SDF shapes can be broken down into a collection of SDF parts that can be compactly represented using a pretrained dictionary, which is generalizable to out-of-category shapes and greatly speeds up inference time.
+ Experimental results show that the proposed system outperforms the baseline methods on different benchmarks.

Weaknesses:
- The introduction doesn't give much context on where this work currently stands in the problem scope of 3D object detection and reconstruction. Why is joint 3D detection/reconstruction important? It is also unclear what the current major problems are for this specific task, and the intro is written rather much like another related work section.
- In the related work section, it is unclear how two-stage pipelines [17,19,51] would be inferior and why one would not prefer these methods over the proposed (L87-88). Personally, I do not see how decoupling detection and object reconstruction would be undesirable. It is also unjustified why depth estimation in [21] is undesirable and why the proposed CenterNet-3D can improve detection accuracy (L92-93,150-151). The authors should also discuss the difference between this work and [A], a very related prior work on voxel-based 3D detection/reconstruction.
- "CenterNet" seems to be an important part of the pipeline (detection) and the term has been repeated throughout the paper, but it was never explained what CenterNet is. I think the authors should devote at least a paragraph explaining what CenterNet is and how it is useful for the design of the proposed method. The authors should not assume it is a commonly known concept.
- It is unclear which component in the system is the most crucial to the performance improvement. Is it the use of voxel features, the design of CenterNet-3D, or the use of the new PCA-SDF representation? It is also unclear from the experiments section, as the authors reported results without justifying why there was the improvement. In the current form, I cannot see what the key message or takeaway is from this paper.
- It is difficult to compare the presented visual results. For one, the ground-truth 3D CAD models are not shown (Fig. 4, 5, 6) so it's hard to tell which method is actually better, despite the results from the proposed method looking slightly more plausible. Ground-truth shapes were shown in the supplementary videos but lacks baseline comparisons, and I think having all of them together is important for evaluation. On a side note, the visualization between CoReNet and the proposed in Fig. 4 are different, which slightly prevents a direct comparison.
- I appreciate the authors' effort to create a new real-world dataset for more supportive evaluation, but I think there should be at least some visualization of what the new dataset looks like by including some example images and ground-truth CAD models processed from Scan2CAD. The quality of the dataset, and thus the evaluation quality, is currently unclear. Also, why is [21] not compared against on this new dataset? Evaluating the detection mAP but only comparing against [20] (which does not perform detection) is not meaningful. Finally, will the new dataset be released to the public?

I think in general, the authors seem to aim to get better benchmarking results as the ultimate goal, but it is not clear what components really contributes to the system. As many of the components seem orthogonal (e.g. 3D feature lifting, coarse-to-fine voxelization, and PCA-SDF), it would be good to analyze and ablate each individual components in depth, and show that an integrated system with all the improvements would be most desirable.

Other minor problems:
- Why are $\delta_h$, $\delta_w$, $\delta_l$ residual sizes? This term is misleading as they are actually the logarithm of the sizes.
- Concatenating (x,y,z) coordinates in convolutional operations is not positional encoding, but rather CoordConv [B]. Positional encoding refers to a deterministic mapping for coordinate-based representations, which is different. On a side note, the original concept for positional encoding should be from Transformers [C] instead of NeRF [66].
- What does "might receive the same 2D feature" mean in L136?
- What do "200 surfaces" mean (L231,264)? What counts as "a surface"?
- The citation for focal loss is wrong -- it should be [D].
- It is unclear why predicting a higher-dimensional code is more challenging for neural networks (L290).

[A] Tung et al. "Learning Spatial Common Sense with Geometry-Aware Recurrent Networks." CVPR 2019  
[B] Liu et al. "An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution." NeurIPS 2018  
[C] Vaswani et al. "Attention Is All You Need." NIPS 2017  
[D] Lin et al. "Focal Loss for Dense Object Detection." ICCV 2017
limitations_and_societal_impact: Adequately addressed
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper proposes a technique for joint detection and reconstruction of objects from images. The core idea is to express the signed distance function (used for reconstruction) within a set of pre-trained linear bases, and regress the bases coefficients by a deep network. The effectiveness of the solution is evaluated on a number of datasets against recently published work, and shows good performance.
main_review: Overall, I think the technique is solid and easy to understand, the results very competitive against SOTA (with a simpler technique), and the writing simply **excellent** (congrats! the introduction was the *only* section I did not enjoy reading). I am willing to raise the score by one point if my questions are answered properly, and I do not score the paper even higher, simply because I have not personally worked on all of the datasets this work is evaluated on (I would like to see the other reviewer's comments before taking a championing stance).

## Main questions (Q):
1. my main concern is the overall case of DeepLS vs. PCA-SDF. While without doubt the latter should be more computationally performant, I am skeptical that the accuracy (or latent space) of the former to not be more expressive. Aside from experiments, the paper lacks a theoretical justification of why this should be the case, and I would like to hear the authors' opinion in regards. For example, the recent NGLOD and ACORN papers use *tiny* decoders, and they represent geometry to very high accuracy?
1. the use of positional encoding in the UNet structure is one of the main differences w.r.t. past works (I am aware of), and I am simply amazed it was not quantitatively ablated? I only saw qualitative comparisons in Suppl/Fig.2. I would expect the method to collapse without this enabled in the multi-object setting?
1. Why euler angles and not quaternions, expmaps, or other vector representations? euler is especially a challenge as the representation is non-periodic (0==360). How can this not be an issue?
1. is reconstruction performed within each object's bounding box domain, or at the scene level (I assume the latter)? regardless of the answer, picking one begs you should compare against the other in terms of reconstruction accuracy? Why was this not done?
1. if an image is captured in the wild, knowing where to place the grid becomes a chicken and egg problem (you could estimate the grid position from detections, but detections are derived from the grid). Shouldn't this limitation be clearly acknowledged?
1. how finnicky is thresholding the detection maps? And if this becomes noisy, wouldn't this require some form of maximal suppression?
1. Sec 4.2: You are using the Pix3D splits from MeshRCNN, yet provide no quantitative comparisons to this paper... can you justify?
1. you mention "truncated objects" and that other methods have "no special design for feature smearing"... does yours have it? and can you spell out the issue?

## Technical details (T):
1. I am pretty sure in (1) you want to write everything in homogeneous coordinates.. otherwise the math is actually wrong (e.g. $c_3d$ becomes a scale rather than a translation)
1. You mention interpolation in L199, but I do not recall seeing any results/ablations in regards? Also note that NSVF and NGLOD both allow for this?
1. in (3) you employ the L1 norm for projecting the signal to the PCA basis in the loss, but PCA uses the L2 norm in its optimization, have you tried L2? Doesn't L1 result in suboptimal performance due to the mismatch?
1. L213: the L1 norm of a matrix is the maximum absolute column sum... pretty sure that this is not what you meant to say, are you referring to the Frobenius norm?

## Further comments (C)

1. Title: not sure that this paper is voxel based, as when I think of voxels I do not think of implicit functions... consider changing your title to reflect this? 
1. L32-37 should be moved to the related works. It communicated very little to me as a reader
1. L112-113: not sure this sentence is true, I think it is more proper to say that this is not used within a deep network for inference task (it was used as a prior for compression)?
1. L123-126: it would be good to acknowledge past work from which you inherited this design, I recall learning it from  "learnable triangulation of human pose" (LTUP), but I am sure there was an earlier paper...
1. L127-144: same as above, this is almost identical to LTUP, with the exception of positional encoding?
1. L161: residual with respect to what? it was only mentioned later on in the paper, and you should mention it here
1. L155: it would be good to note that, compared to past works such as KeypointNet or LTUP this is performed because you need multi-modal instance detection. 
1. L170: voXELization
1. L188: q is defined and never used (this is also typically called a `meshgrid`)
1. L189: what does "toward the surface" mean?
1. L209: please add a line to mention why this focal loss is needed?
1. L215: typo "without as"
1. L303: how? by the classical inner product of the lexicographically-unrolled SDF with the PCA projection matrix? (hint: cite or math, your choice)
1. L34: please avoid undeclared acronyms at all costs.. BEV? Bird Eye View? maybe...
limitations_and_societal_impact: Not sure, and not sure I should care for a paper of this type.
review_title: null

Meta-review: This paper initially had mixed reviews (8,8,5,5). After reading the rebuttal and checking the other reviewers' comments, the two negative reviewers were convinced and raised their score to 6. Reviewer 7tQr appreciated the simple innovation this paper offers, which is of broad appeal to the 3D reconstruction community. Reviewer NjXM stated that the rebuttal addressed their concerns adequately. Hence, this paper now has 4 positive scores and it should be accepted.


Decision: Accept (Poster)