Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors propose to separate out camera view point and template deformation using a differentiable optimization layer, and only delegate correspondence estimation to a CNN. The authors demonstrate improved performance on standard benchmarks with such a modular approach.   
main_review: Strong points
- Formulating the camera viewpoint and template optimization as a differentiable layer and enabling end-to-end training
- Geometry driven losses requiring no annotation

Weak Points
- The experiments are limited. Especially, there is no experiment discussing the performance with/without the silhouette as is done in the related work, 
- The improvement due to separating the optimization tasks should be better motivated and substantiated with additional experiments. For e.g., what happens if we were to use a cnn to regress the view point/deformation parameters with correspondence as input. How does it compare with the differentiable optimizer? What is the inflection point in performance wrt to number of iterations? As neural networks are general function approximaters, the burden of multi task optimization can be alleviated with more network capacity or additional data. All of these are design choices that are not discussed. 
- The main idea of separating out using a CNN for correspondence and classical optimization for pose and shape regression is not new. Real-time Pose and Shape Reconstruction of Two Interacting Hands With a Single Depth Camera by Mueller et. al. were the first to do so to the best of my knowledge and there is related work in body/hand shape estimation which is not discussed in this paper. Learning to Reconstruct 3D Human Pose and Shape via Model-fitting in the Loop by Kolotouros et. al. is also relevant.  
- No mention of run time or compute analysis. 

Pose Rebuttal Review:
I apologize for missing the fact that mask is necessary for the approach. It was a detail overlooked by me while skimming related work as this is not my primary domain of research. Feeding correspondence to a network can be trivially accomplished by passing tuple of positions or their embeddings thereof. As demonstrated in transformers, this information can help the network learn better. I am still curios of such an analysis. I would also be more enthusiastic about an acceptance if the code is made available, as replication of the approach is non-trivial and marginal improvement over SOTA might be difficult to achieve in practice without the authors divulging the 'secret sauce' as common among the best of contributions in this domain. I am slightly upgrading the rating based on other reviews and their assessment along with the additional experiments on latency. 
limitations_and_societal_impact: Augmented reality is here and this work directly impacts it. The authors should include a broader discussion of negative impacts from an augmented reality perspective. Research in nascent stage is the best place to identify potential negative impact on society.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper tackles the problem of learning single-image 3D object reconstruction from single-view collections. The proposed method couples a correspondence prediction network with an iterative optimization procedure that solves shape and pose based on the predicted correspondences, starting with a category template shape.
The claim is that, with this iterative optimization procedure in the loop, it alleviates additional supervision for camera viewpoint such as keypoints, or multiple manual hypotheses.

The results show that this method is able to achieve comparable reconstruction accuracy without keypoints compared to previous methods that require keypoints. Overall, this paper presents an interesting approach of incorporating iterative optimization in a shape learning framework, but the method also has its limitations and the evaluation is rather limited.

main_review: ## Strengths
### S1 - Method
The proposed approach is based on the same idea as in UMR[16] and ACSM[15]: separating the correspondence prediction and reconstruction, which is the key to classical structure-from-motion techniques. Unlike [15] and [16], the proposed method adopts an iterative optimization procedure in the reconstruction branch, whereas [15] and [16] use networks to predict shape and pose directly. Specifically, the camera is solved by L-BFGS optimization and the shape is solved in closed form with a optimized shape basis.

This has the advantage of being more robust against local minima given the shape pose ambiguity, which is a key challenge in unsupervised 3D reconstruction.
In contrast to existing methods that rely on additional keypoint supervision or multi-camera hypotheses, the proposed method is more data- and (perhaps) computation-efficient, although it still involves an iterative optimization process.

### S2 - Writing
The paper is well-written. The motivation is clearly introduced, and methodology clearly explained.


## Weaknesses
### W1 - Method
- W1.1: One major drawback of this optimization-based solution is that the same optimization procedure is also required at inference time, even though the authors show that even one iteration can already produce accurate results. How fast is each optimization iteration?
- W1.2: Does this optimization always yield a good solution? It seems to me the camera viewpoint can fall into local minima with BFGS optimization as well. Moreover, it relies heavily on the predicted correspondences, and cannot be robust against erroneous correspondences as a learning approach could with learned priors.
- W1.3: Another drawback is the way the correspondences are predicted using points. It does not guarantee continuity, as is also shown in the visualizations where the points can get messy in some parts. CSM [15] and part segmentation [16] have the advantage of producing smoother correspondences with an image. This also makes more concerned with the robustness of the optimization against errors in correspondence prediction.
- W1.4: One more major limitation is the need for a category template shape similar to [15] and [16]. This has largely simplified the problem of category-specific shape reconstruction and at the same time restricted the reconstructed shape from deviating too much from the template. The reconstructed airplanes for example are almost always the same despite the variation of the inputs (see Fig 9 in the sup. mat.).

### W2 - Evaluation
The evaluation has a number of limitations.
- W2.1: Why not compare with UMR[16]? It obtains part correspondences through a self-supervised part segmentation model, and does not require a template shape.
- W2.2: The numerical results are not very convincing. Mask mIoU only measures the shape from one single viewpoint, and does not evaluate the 3D shape. Keypoint reprojection is also very sparse and noisy, and may not be indicative of the 3D shape accuracy. It is unclear whether the comparison on these noisy metrics is actually meaningful.
- W2.3: The evaluation on Pascal3D+ with 3D mIoU is more meaningful (although the 3D shape is also "pseudo GT"), but the gap is relatively small. But more importantly, I wonder how well the template shape alone achieves in terms of metric?
- W2.4: Most of the visual results are visualized in the input viewpoint, which makes it hard to tell the how good the reconstructed 3D shapes are. From the only one novel view provided in Fig. 2 and 3 and in the sup. mat., the reconstructed 3D shapes do not look very plausible. The birds for example appear quite flat. The authors should provide videos or visualizations with multiple viewpoints to showcase the reconstructed 3D shapes. How do they compare to UMR and ACMR visually?
- It probably also makes sense to plot distribution of the predicted viewpoints. Is the model confused with some viewpoint, eg front vs back, left vs right?
- It could also help with understanding the pose and shape prediction to color the predicted mesh with the same color pattern in the template mesh.


## Clarification needed
- How exactly is texture symmetry enforced? The method samples texture from the input image using the predicted 2D points. In line 182, it briefly describes the texture symmetry enforced by averaging textures from two symmetric viewpoints. Does it mean taking the average of the sampled colors of two symmetric points (defined by the template)? If so, does this also rely on the visibility prediction? More details should be provided.
- This is interesting because texture symmetry alone can provide pretty strong cue for shape and pose. Does the model utilize this cue for reconstruction?
- There are 9 loss terms in total (if not more). How are they balanced?


## Additional comments
- Eq (3): $\mathbf{V}$ should be $\mathbf{T}$, input $\mathbf{C}$ should be lower case $\mathbf{c}$.
- Line 207: "vertical flipping"? ie, about the horizontal axis and the objects would be upside down?


---
## Post-Rebuttal
I appreciate the authors' efforts in the rebuttal, which has addressed many of my questions, including the computational cost, comparison with UMR, and template shape baseline performance on Pascal3D+. Regarding the robustness, it is still not clear how robust the correspondence prediction and subsequently the pose estimation are. This is hard to quantify, but more discussion on failure cases in the final version would be very helpful.

Overall, this paper has sufficient interesting material for acceptance. I will keep my original review as borderline accept.
limitations_and_societal_impact: A few failure examples are provided in the supplementary, but they all seem very challenging cases, eg flying birds, heavy occlusion and small crops.
Are there failure cases on more regular images? Does the viewpoint prediction get stuck in local minima? Does the correspondence prediction fail dramatically, and what happens in the shape and pose optimization then?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a method for category-level 3D mesh reconstruction from single image. The key innovation is a correspondence-driven optimization layer that produces camera pose and object deformation given a single image, as opposed to using conventional pose CNNs. Overall, the idea is sensible, and the results are better than prior methods under the same setup, either given keypoint annotation or without keypoint annotation. My concerns is mostly on significance and writing.
main_review: Originality and significance:
- The problem is well-motivated and the discussion of bad local minima in NR-SFM is helpful for understanding the significance of the problem. 
- The proposed solution of casting the pose estimation problem as correspondence regression + optimization is also sensible. I can see how the extra optimization step could produce more accurate shape at test time. However, it's not obvious why it will address the bad local minima issue at training time. It would help to provide further insights or experiment evidence.  

Clarity:
- The presentation is clear and easy to follow in general.

Method:
- The method section is well written in general. However, some details are missing from Sec 3.2 and makes it difficult to evaluate the technical significance. For instance, l136 the authors mentioned the optimization is lightweight, but without mentioning the complexity of each iteration and number of iterations needed for convergence.  In table2, the number of iterations is reported but it's not clear whether this is at test time or training time. It will help to compare to prior CNN-based methods in terms of accuracy and speed.
- From an input image, TTP first predicts 2D-to-3D-template correspondence, and then solves a nonrigid perspective-n-points problem that produces camera pose and shape deformation. However, I'm able to find a concrete description of how is correspondence trained. Is it fair to say the correspondence is trained by cycle consistency loss L_cycle?
- It is not clear how much deformation can the proposed approach handle. In Eq (7) the authors models shape deformation with a linear shape basis, that has a closed-form solution. However, a linear basis is likely to be insufficient for modeling large motion, such as articulation of human bodies.


Minor
- l106: I'm not following why "3D to 2D". From the description, it's the other way around.
- l142: weigh->weight
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper tackles the problem of 3D shape texture and pose estimation from a single image without 3D, multi-view nor camera calibrations. This is an extremely challenging problem. Concurrent works are CMR, U-CMR, CSM and A-CSM. **The key novelty** is to avoid learning pose, texture and shape with several networks heads. Instead, the author propose to learn the 2D positions corresponding to the 3D template vertices, then solve a per-sample optimisation to estimate the pose and template deformation, enabling texture recovery through differentiable rendering. They use implicit differentiation to differentiate through the L-BFGS solver of their inner PnP problem. They achieve successful reconstruction of cars, birds and motorcycles. They outperform their competitors on the Cub dataset and Pascal3D+ dataset, showing exciting successes even without keypoint supervision.

main_review: 
# Writing

The method is complicated, and the high quality of the writing helps communicate hard concepts.
How the method relates to the state-of-the-art is not really discussed  in the related work section. I am not sure "Non-rigid structure from motion (NR-SfM)" deserves a section.
  



# Strengths

The method is simpler than U-CMR and sets the new state of the art in shape, texture, and pose estimation from a single image. Avoiding to learn things that can be optimised per-sample seems like a more robust and simple approach.
I found the approach very original and am excited by the adoption of implicit differentiation in 3D deep learning. I greatly enjoyed the paper.

 # Post-rebuttal comment

I have read the other reviews and thank the author for their response. I found this approach very original and maintain my review.
limitations_and_societal_impact: # Weaknesses

-   A few things are still unclear to me. How are the unseen parts of the object textured? Using the predicted **u** ? Does this make sense considering they are not not supervised by the texture loss due the the visibility term? You may have discussed this in this sentence that I did not understand: “We also apply the loss on the symmetric texture predictions by using a bilateral symmetric viewpoint and average the two viewpoints. The soft symmetry constraint ensures that the texture of the non-visible side is still inline with the visible side.”
-   The current limitations of the method are not very clear. Is it easy/hard/impossible to train on a new category like humans? What is the current bottleneck limiting the quality of the reconstructions? Data, deformation model, convergence… ?
review_title: null

Meta-review: The paper presented a new, iterative optimization procedure in the reconstruction branch of the conventional 3D reconstruction framework, instead of training network s to predict shape and pose directly as in UMR and ACSM. In general, the reviewers were a bit mixed in the initial review. After the rebuttal and the discussion phase, R1 increased the rating. Please include many of the rebuttal points in the final version, especially run time analysis and comparison with UMR.
Decision: Accept (Poster)