Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a method for 3D pose estimation of PACAL3D objects (6 vehicle categories). The method bootstraps from 7-50 annotations per category. The idea is: given the initial annotations, finetune a Resnet50 and a featurized cuboid mesh with contrastive losses that bring the 2D and 3D features close to each other, and make them discriminative for position/viewpoint. Then, compute 2D features for the entire dataset, and rotate the 3D featuremap a few degrees in each direction, and compute match scores for all pairs. The confident matches are then used for subsequent re-optimization of the 2D and 3D features, and this process repeats ~100 times, gradually increasing the number of rotation bins covered by the 3D featuremap. Results are quite good against non-bootstrapping baselines, trained just with the 7-50 annotations. 

main_review: I like this paper. Pose estimation is a hard problem, and this boot-strapping/EM solution combines insights from pre-trained ImageNet feature-learning and structured 3D representations in a clever way, to gradually cover the full set of viewpoints/angles of the objects. The approach is a little constrained, in terms of assuming perfect detections and known scale/distance and centroid, but maybe this is OK. 

I was surprised that the approach does not also involve some optimization of the 3D mesh of the category, and instead keeps it as a cuboid. Have the authors tried optimizing this, or perhaps optimizing occupancy inside the grid, and using occupancy as part of the rendering?

What about the drift? EM/bootstrapping methods like this inevitably drift (and also sometimes recover). I think it is important to show pose estimation errors over EM rounds, and maybe to visualize some "hard" examples for which the the wrong pose has a better match score than the right pose.

How does the pose analysis-by-synthesis work exactly? Is it SGD on the three angles? Does this require multiple initializations, or does it work even if it starts 180deg off?

I understand the idea to compare against baselines under the same semi-supervised setting, but I think it is important to also report those models' fully-supervised results, to put things into perspective. 

It would be interesting (but maybe not essential) to compare against "Image GANs meet differentiable rendering for inverse graphics and interpretable 3d neural rendering" (ICLR 2020), which I think can also be interpreted as an EM approach: an image generator is used to optimize a 3D inverse graphics net, and the inverse graphics net is used to optimize the latent space of the image generator. I do not think they evaluated on pose estimation, but they could have. 

It would be nice to visualize the 2D and 3D features, with PCA.

Some typos: 
- Groundturth -> Groundtruth
- wholes -> holes
limitations_and_societal_impact: No concerns.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the problem of estimating the pose of an object, given an image of it. The proposed method aims to use as few labeled samples as possible, and generate pseudo-labels for a large collection of unlabelled data during training. The method works in an EM-fashion in which two stages (neural view synthesis and matching) are optimized by alternating.
 
In the first stage, the method extracts a feature map from the input annotated image and shapes it as a mesh cuboid where each vertex is assigned a feature vector depending on its position. The mesh is then rotated at regular angle intervals and rasterized into the image plane. In the matching part, features from unlabeled images are extracted and matched with rotated view rasterizations using a cosine similarity metric.
 
The experimental evaluations show that the proposed method is able to learn the pose prediction task from as few as 7 annotated images during training. The method mostly outperforms the baselines in comparisons against them. 

main_review: The proposed method is novel in the way that it generates 3D pose pseudo-labels for large amounts of unlabeled data which is difficult to annotate in practice. I believe that the method can be useful in future research in this area.

Overall, I think the paper is organized well. There are a few typos, some of which are included at the end of this review. My questions and comments are listed below.

- The term neural view synthesis sounds somewhat misleading in my opinion, since the proposed method does not synthesize and render RGB images in the way the literature on view synthesis does. I would suggest clarifying what is meant by view synthesis and adding how it is different from the literature. 

- How does the method ensure that the extracted features from the image have meaningful 3D information? What does it mean to rotate the features extracted by a network pre-trained on ImageNet?

- How does the method avoid trivial solutions of assigning the nearest neighbor’s label as the pseudo-label? What is the intuition behind preventing such mode collapses? For instance, if there are images with labels of 30 and 45 degrees azimuth in the annotated dataset, how does the method infer an angle of 35 degrees? 

- Similarly, how is the model able to differentiate between the pose angles from the front and back views, or from left and right side views? Are there any failure modes involving such examples?

- Do the results in the tables average over all three output angles (azimuth, elevation and in-plane rotations)? How does the method perform in these individual terms?

- Is ∆θ only along the azimuth axis or does it also include the elevation and in-plane rotations?

- Can you detail the optimization at test time more? Does the method use additional labeled data at this stage? What is the computation time of the pose optimization at inference time? 

- What is the procedure to select the few images with labels? Does it need to cover the viewing hemisphere? Any discussion would be helpful to include in my opinion.

- Does the approach of averaging features from multiple annotated images have any drawbacks? Can having a large number of labeled data (> 100) degrade the performance due to features averaging out each other? 

- What is the curriculum strategy for starting with small ∆θ and increasing it while training?

- Why are the angle errors increased for the case of ∆θ = 0, after the semi-supervised training in Figure 2?
 

Minor comments:

- Why is the feature cuboid named a mesh rather than a voxel grid? Since the mesh is a grid, and the edges and faces are not used, I believe calling it a voxel grid is more accurate.
- What do the features projected to the image plane look like? I would suggest adding those generated feature images to the paper.
- Please include the standard deviation of the results in the tables as well, for a more complete evaluation. 
- Typos: line 53: the the; line 63: for for; line 65: annotated -> annotate, line 78: between between, line 161: wholes -> holes

Update: after my concerns were clarified in the response, I raised my score to accept.

limitations_and_societal_impact: The limitations of this work, along with potential societal impacts are discussed in the conclusion. The inability of trivial generalization to articulated objects such as humans is listed as a limitation. I believe that there are others as well though, such as the existence of a large error margin (40%) for pose estimation even when the model is evaluated at 30 degree thresholds. I encourage the authors to detail the limitations section of the paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper presents a learning based method for 3D pose estimation from single image that works with very few annotated data. Given some exemplar images with 3D pose annotation, the 3D pose of a test image could be estimated by two steps: 1) neural view synthesis and 2) feature matching. In the neural view synthesis step, a 3d feature is generated by unprojecting a pretrained 2d feature map of the image. The 3d feature is then probably "projected" to a set of novel views for feature matching and 3d pose estimation.
main_review: Weakness/questions:

The paper proposed a few shot learning method for 3D pose estimation from 2d images that utilizes the underlying 3D structure. A 2d feature map is unprojected to a spatial grid and this kind of 3D representation is desired to be pose-invariant. However, it is not very clear to me how occlusion reasoning is performed here and how a holistic shape representation is inferred from a single image here. For example, how are the features on the grid are sampled from a 2D view? Is any self-occlusion taken into account when the 2d feature are unprojected to the 3d grid? How about the features at the occluded vertex? Do they take the feature from the object that is in front of them? Also in the neural view synthesis step, how is the rasterization performed? Are all vertex considered to be non-transparent? it is a bit unclear to me for this part and I hope the author could provide more details about this part.

This paper seems to be similar to another relevent work NeMo which is also cited as [44] in this paper. I am wondering if the author could elaborate more on the difference between those two works and the novelty of this work among NeMo?
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper investigates a new problem of semi-supervied 3D viewpoint estimation using few-shot labeled examples. It further proposes a new method for the task based on learning a 3D cuboid feature for a category with features at its vertices which are viewpoint invariant. The authors propose a method of rotating the cuboid to generate feature representations for novel views. Another key insight of the paper is how to expand the ranges of novel views for which the cuboid can be rotated by introducing a contrastive loss to update the cuboid feature representation to be more rotation invariant by combining with the pseudo--labeled images. The authors compare their method to two baseline SoTA methods and outperform the existing methods by a large margin in the few-shot setting.

main_review: Originality: The paper introduces both a new challenging task of few-shot semi-supervised learning as well as a novel and effective method for it with a strong baseline performance. The work is quite original and opens up a new research area.

Quality: The proposed method is technically sound. The experiments are adequate and well formulated. I note a few shortcoming to improve the paper.

1. The authors describe that they manually chose the 7 few-shot labeled examples per category to train with such that they equally span the entire range of pose angles. It seems to me that this choice is critical to the success of their proposed method. In the worst case, if all the 7 few-shot poses were clustered at a single point, I doubt that the authors would be able to expand the pseudo labels to beyond a small range of angles around the labeled ones. The authors should discuss this limitation of their approach more clearly and provide experiments to clearly show the affect of the choice of few-shot samples on the accuracy of their algorithm. Does their algorithm support a random choice of angles?

2. From Table 1, the proposed method does not seem to perform as well as NeMo for 50 training samples. This behavior for larger number of trained samples is not observed for the KITTI datasets, however. So, perhaps it is an artifact of the Pascal3D+ dataset. I would like to hear a further discussion of this from the authors and whether they have analyzed  the cause of this observation. I am also curious as to what happens with greater than 50 labeled samples and for 100% labeled training data? Does the proposed approach perform as well as NeMo on both datasets in fully labeled scenarios as well or is it limited in applicability only to less labeled scenarios? 

3. Few-shot Viewpoint estimation has been explored previously in the work: Few-Shot Viewpoint Estimation, Tseng et al, BMVC 2019. However it was in a different context of exploring few-shot generalization to previously unseen categories of objects, which is different from this current work. Nevertheless it would be good to cite this closely related prior work.

Clarity: The paper is mostly clear. Code is provided. There are several small typos throughout the paper which would be helped with a thorough editorial review. 

Significance: Labeling viewpoint or 3D pose is a hard problem. The proposed work opens up a new line of research; proposes a novel method with several interesting new insights, including the use of the 3D feature projection and of using the contrastive loss in an EM-style framework to expand to more viewpoint; performs significantly better than the SoTA approaches and generalized to many different object categories. This is a significant and broadly impactful piece of research.
limitations_and_societal_impact: It would be good to discuss further the points 1 and 2 stated in the previous answer as potential limitations of the method in the paper, if they are indeed limitations of the method.
review_title: null

Meta-review: This paper addresses semi-supervised 3D viewpoint estimation by lifting 2D CNN features into object-centric feature cuboids, which are rotated and rendered to generate features maps of alternative views, while also updates in a contrastive manner exploiting labelled and pseudo-labelled examples in an iterative manner. All reviewers agree on the novelty of the approach pursued in the paper. The rebuttal submitted by the authors clarified many concerns of the reviewers regarding handling of  occlusions and self-occlusions, scalability of the method with respect to the number of annotated examples, evaluation metric and thresholds used. 

Decision: Accept (Poster)