Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method for single-view category-specific 3D reconstruction. The pipeline starts with deriving a canonical pose with a partial point cloud. Then a neural deformation is used to reconstruct the object's 3D surface. Finally a joint optimization of pose and shape is to further improve the results.

strength_and_weaknesses: ### Strength
- Qualitative and quantitative results seem impressive, there are notable improvements over previous work.
- The appendix is useful to understand more details of the work.

 ### Weakness
- Overall the work feels more like a pipeline to combine different components together. It is not fully clear to me what the major technical contribution of the paper is, compared to the previous work. It would be useful if authors can highlight and contrast over the previous work.
- The writing and presentation of the paper can be improved. Currently it's not very easy to follow the paper.
- Ablation study and analysis is a bit lacking, thus it's also hard to understand the individual impact of the components.

### Additional Qs:
- In table 2, why the improvement of car is larger compared to other categories?

clarity,_quality,_novelty_and_reproducibility: The presentation of the paper is not very clear. Quality, novelty and reproducibility are also not very convincing.
summary_of_the_review: To me the paper is clearly below the bar from multiple aspects: clarity of presentation, technical novelty and insufficient experimental validation. Thus I would vote for reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper present a deep learning based pipeline to reconstruct the 3D surface of an object from a RGBD image. A network is trained to encode the category shape priors and then deform the shape priors to fit the object point cloud. A separate network is needed to transorm the object point cloud from the camera coordinate to the canonical coordinate frame. The proposed method is evaluated on both synthetic and real datasets and shows superior performance to the compared peer works. 
strength_and_weaknesses: Strength:
- The method achieves better results compared to some recently proposed learning based methods. It also addresses some of their limitations which moves it closer to practical application (still a long way to go though)
- The method shows good results on the picked occlusion cases. It would be interesting to see a more thorough evaluation or demonstration. 

Weakness:
- It is not clear how fast the method runs in practice, especially considering that it requires segmenting the objects from the image or point cloud and lifting the point cloud to the canonical coordinate frame.
- The proposed method seems to be a combination of ideas from existing works. The novelties over these existing works are not clearly discussed. 
- Several parts of the paper need to be revised. 
  - Abstract "A limitation of current color image-based 3D reconstruction models is that they do not generalize across datasets": this should only apply to learning based methods.
  - 2.1 "Neural rendering and neural fields provide an alternative representation to overcome these limitations. Surface fields, such as Signed Distance Functions (SDFs) (Xu et al., 2019; Lin et al., 2020; Duggal & Pathak, 2022) or volumetric representations such as occupancy (Ye et al., 2021)": both SDF and occupancy grid are classic represetnations, they are not proposed or provided by neural rendering and neural fields.
  - 3.1 "Instead of directly mapping a low dimensional latent code zi ∈ Rn to the 3D shape through concatenation": concatenation is quite confusing.
  - Eq 2: what is \hat{s}?
  - 3.1 What does "instance space to the template space" mean?
clarity,_quality,_novelty_and_reproducibility: The idea has been clearly presented. However, since the method consists of several steps and some tech details are not provided, the work might be hard to reproduce. 
summary_of_the_review: The work seems to be a cobination of existing ideas and the novelty is marginal. My current rating is borderline reject.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: In this paper is proposed a method to recover (complete) the 3D reconstruction of an object category from a single depth picture. The method exploits synthetic training data and a point cloud pose canonicalization to learn a category-specific geometry. As a consequence, the learning of these 3D priors helps to retrieve the 3D reconstruction even for occluded bodies. The method is evaluated on different scenarios by using depth images or LIDAR-like point clouds; a comparison with competing techniques, ablation studies and failure cases are also reported. 
strength_and_weaknesses: The authors claim their method does not require ground truth camera pose or 3D reconstruction for supervision, being the goal to learn a category-specific 3D prior model. I have no problem with that, in fact, I like the idea. However, similar ideas were exploited in the non-rigid structure-from-motion community, a community that was not included in the discussion of this paper. I know the differences, but I feel a discussion of these approaches could help the reader. We can find some works on the literature, such as: “Unsupervised 3D Reconstruction and Grouping of Rigid and Non-Rigid Categories” and “Structure from category: A generic and prior-less approach.”

The proposed method is a combination of well-known ideas and architectures in the literature. In this context, the technical contribution is a bit poor. 

Be consistent with notation. For example, see x in Fig. 2 and Eqs. (1), (2), etc. 

Reproducibility: some weight coefficient values, for example, see Eq. (7) are never provided in the main paper. 

A calibrated camera is required. This should be commented on in the introduction. 

Why not use additional categories? A priori, I think more everyday categories are included in the datasets the authors are using, so I cannot understand why just three are considered. 

The authors report quantitative evaluation in terms of chamfer distance and F1 score. To be honest, for a 3D reconstruction algorithm I would hope for a different metric where we evaluate the 3D model, instead of one based on chamfer deviation. 

The method provides a good performance in comparison with competing techniques. Moreover, it can handle occlusion properly. However, the occlusion seems not to be very realistic. Instead of that, I would like to see some cases where the occlusion can explain a natural and real case. The method can naturally also handle missing observations.

I agree with the authors; the proposed method produces more smooth solutions to the point of avoiding capturing details. Can the authors imagine an explanation of that?
clarity,_quality,_novelty_and_reproducibility: In general, the paper is well written and it is clear enough. While the contribution seems to be a bit incremental, or a combination of previous models and ideas, the experimental analysis and the final results are competitive. In terms of reproducibility, I feel the authors also provide most of the details we need to reproduce the results. 
summary_of_the_review: As I commented before, the results obtained are promising, and even though some priors want to be simplified with respect to other techniques in the literature, the novelty of the proposal is somewhat incremental.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The work tackles the task of 3D reconstruction given a single RGB-D image. The reviewers raise concerns regarding the novelty of the method as, although the overall pipeline is interesting, the core components (shape completion, canonicalization) are built on existing approaches. Further, the AC is also concerned that the empirical results compare this approach to baselines which do not require depth and this is not a fair comparison. Moreover, the text claims that these RGB methods require camera pose an inference but this maybe incorrect e.g. Sec 4. of TARS states “Each training example consists of cropped RGB image (centered around the object), corresponding segmentation map and camera pose. At inference time, we only need the object image as input.” Overall, given the concerns in the empirical results and contributions, the AC is leaning towards rejection.
justification_for_why_not_higher_score: There are significant concerns regarding the empirical results/setup (i.e. comparing an approach with RGB-D input to prior RBG-based methods) as well contributions (while the pipeline is novel, it primarily combines existing components).
Decision: Reject