Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The paper proposes a method that utilizing unlabeled data for training to enable the proposed category-level 6Dpose estimation algorithm generalised to new scenarios. A new dataset is collected for implementing the idea. Comparisonal experiments were conducted on the REAL275 dataset and the proposed Wild6D dataset. The proposed algorithm has certain advantages over the listed algorithms. 

strengths_and_weaknesses: Strengths:

1. The new method is able to utlizing the unlabeled real-scene data.

2. The authors give a large RGB-D real dataset and annotate the testing dataset, which may be useful for certain tasks.

3. The authors promise to make the code publicly available

Weaknesses and Questions:

1. The value of IOU-0.5 of CPS++ reported in Table 5 is way much lower than the value reported in the paper [1], please explain this inconsistency.

2. The author simply compared the performance of RePoNet with CPS++ when presenting the effectiveness of RePoNet on processing unlabeled dataset. While many SOTA unsupervised algorithms, for example, UDA-COPE [1], behaves better than RePoNet on the REAL275 dataset without any real annotations. It is necessary to compare more relevant SOTA methods for a fair comparison.

3. In Table 6, the CASS, Shape-Prior, DualPoseNet are all trained on the CAMERA75+REAL275, while RePoNet-syn and RePoNet-semi are trained either on CAMERA75 or CAMERA75+Wild6D. The author validated all these algorithms on the Wild6D and then emphasize the superiority of RePoNet. The lack of consistency on training dataset makes this comparison, again, unfair and invalid.

4. For line 327, the authors stated that 'FS-Net does not release the model, we cannot experiment on it.' I am not fully convinced by the explanation. Although the FS-Net doesn't release the pre-trained model, the code is available on GitHub (https://github.com/DC1991/FS_Net). Given the prominent performance of FS-Net on IOU-0.5, it is worth to try, which I believe it is also possible, to experiment with FS-Net on Wild6D dataset based on the published code. 

[1] 'UDA-COPE: Unsupervised Domain Adaptation for Category-level Object Pose Estimation', CVPR2022
questions: please check the strengths and weaknesses.
limitations: I didn't see any negative societal impact bring by this paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This dataset-and-method paper proposed a new unlabeled large-scale dataset for category 6D pose estimation in the wild, along with a new network design that can utilize that dataset in a weakly supervised manner. Most notably, the weakly supervision branch is on top of the desired shape and pose branches, thus allowing training both even without or with only a sporadic strong supervision signal. The proposed method is evaluated on existing datasets and existing methods are evaluated on the new dataset, showing that the proposed method exceeds state of the art.
strengths_and_weaknesses: + The paper is well written and good to follow. All steps are well motivated and described, the network can likely be reproduced based on the detailed description and well-done figures.

+ The dataset acquisition is well thought off. Using "Turkers" to take videos with their own phones - compared to scientists - is very likely to have a reduced bias regarding scene selection, acquisiton etc.

+ The dataset is a significant improvement over prior datasets regarding size and variety. However, it is not fully annotated, thus potentially limiting its long-term impact.

+ The semi-supervised loss is well thought off, especially that it is based on the 6DoF pose which is the actual target (compared to a parallel branch / head in the network).

+ The proposed network and training protocol is well thought off. It is based on well established building blocks and combines them in a way that allows weakly supervised training. This allows it to efficiently leverage a large amount of unlabeled or weakly annotated real data.

+ The experiments are convincing and exhaustive. The proposed method is validated against prior art based on established datasets. Ablation studies show the impact of different building blocks. Some prior art methods are evaluated on the proposed dataset, forming a baseline evaluation.

+ The results are very good compared to prior art.
questions: (1) Since iPhones were used, presumably with an own app to also capture depth, the rather accurate IMU could have been used to track the phone's movement (compared to the data-dependend models TEASER++/ICP, l.145).

(2) What is the estimated accuracy of the annotated poses, based on the errors introduced in labelling and tracking?

(3) Is the instance segmentation network pre-trained (and if yes, how / on which data), or trained along with the rest of the network in an end-to-end manner? For the latter case, I'd expect it to be rather unstable in the beginning of the training.
limitations: Yes, limitations and impact were appropriately addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes a new dataset for 6D pose estimation and a method named RePoNet, for estimating 6D object pose and size. The method falls in the category-level approach that predicts an object pose only based on its category. 
In particular, the presented method is defined by two main branches a Pose Network and a Shape Network. The Pose Network uses the NOCS map as an intermediate representation to get the object pose, and the Shape Network uses the specific category 3D shape prior to leveraging the intra-class variation with the object mask.

The new dataset, named Wild6D, collects 5166 videos for five object categories under multiple views. 
The method utilises the NOCS synthetic dataset, with all 6Dpose annotations, and the real dataset together with the proposed Wild6D to obtain masks and corresponding depth maps for training. The authors perform a detailed ablation analysis of the method on the datasets used. Results show that the proposed method outperforms the considered competing methods. Comparisons with state-of-the-art follow the metrics of [44].

strengths_and_weaknesses: The paper poses the relevant question of mitigating the annotation effort in category-level 6D pose estimation. The authors propose to exploit a common strategy in many fields, based on training on synthetic images and resolving the domain gap with real images.
The authors refer to the NOCS dataset [44] (Camera25, with 300K  synthetic images, and Real275, with 8K real images). At the same time, the authors' contribution is to use only synthetic images from CAMERA25, with full annotations including  CAD models, segmentation and 6D pose parameters,  and taking from Real275 only the RGBD information (depth map and masks). Due to the domain gap between synthetic and real images, the authors claim that the method is semi-supervised.

The authors also contribute with a new annotated dataset, Wild6D, built from videos recorded with iPhone and a depth sensor. The Wild6D is annotated with 6D object poses every 50 frames, and the intermediate ones follow a tracking strategy.
The strength of the paper is in the competitive results on the 6D pose obtained by the semi-supervised method.   

Major weaknesses of the paper are:
1. presentation.
2. The motivation of using only the synthetic parts with all the annotations.
3. A lack of comparison with most recent works and a discussion on them.

The description of the architecture is quite hard to follow as there are a huge amount of component networks, introduced via references, not always precise,  as in lines 207-209 ([43] does not even mention the NOCS, and it refers to a dense correspondence map, see also "Cdpn: Coordinates-based disentangled pose network for real-time RGB-based 6-dof object pose estimation").

The main difficulty in category-level 6D pose estimation is the intra-class object variation. As in previous work, the paper uses a 'shape prior' to face the problem. Adapting the shape-prior to the current object refers to a deformation network (supposedly as in [37]). No details on the accuracy of the deformation network and the loss used are provided, nor qualitative results.

The whole architecture, collecting several modules and networks, lacks substantial motivation.
Indeed,  the two losses,  the one for fully supervised training and the one for semi-supervised training, differ only in the data they collect. The balance parameters are the same, too, implying that there is no specificity in the semi-supervised learning for pose-sensitive features and geometric relationships.

The choice of training with the fully annotated synthetic images seems to be necessary for dealing with the prior shape. The motivation that synthetic data are at no cost is not convincing.

Recent works use only real data, following a different semi-supervised pattern to minimise the amount of training data. 
For example, "GPV-Pose: Category-level Object Pose Estimation via Geometry-guided Point-wise Voting "(CVPR2022) uses just real data for training and uses for evaluation not only CAMERA25 and REAL275 but also LineMod (not considered by the paper). FS-Net [6] also uses only 'real data' for training, despite Table 5  treating it as fully supervised.  

On the other hand, there are recent works relying only on synthetic datasets such as "CPPF: Towards Robust Category-Level 9D Pose Estimation in the Wild" (CVPR2022), "Self-Supervised Category-Level 6D Object Pose Estimation with Deep Implicit Shape Representation" (AAAI2022), "Category level object pose estimation via neural analysis-by-synthesis" (ECCV202), which the authors do not consider.

questions: Please explain the implicit functions \Psi_{nocs}, and what is used to train the MLP.  
Please explain \Psi_{deform}, how the accuracy of \Psi_{deform} is evaluated and the loss used. 

Even if no qualitative results on the shape are provided,  line 320 reads ‘our approach can perform shape reconstruction’.  
How relevant is the shape to generate the object bounding box? Why it is not used in inference (lines 224-225).

If the shape network is not used in inference, is the mask required? How is it used?
Please provide the parameters of the whole network and possibly runtime.

Please report correctly the comparisons with the semi-supervised methods using only real data, such as FS-Net.

limitations: Limitations are discussed, in particular, the difficulty to generalize.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper presents a dataset and semi-supervised method for 6DoF object pose estimation with the aim of generalizing better. The first contribution is a dataset called Wild6D which is a dataset with 5166 RGBD videos of 1722 object instances from 5 categories. The second contribution is a neural network RePoNet that is trained to estimate 6DoF pose by training on both synthetic data (with full supervision) and real data (partial supervision). Experimental results indicate that the proposed method outperforms other methods on the 6DoF pose estimation task.
strengths_and_weaknesses: Overall, this paper has a nice contribution and I would recommend acceptance. In the following, I will list strengths, weaknesses, and other questions.

## Strengths

- The paper addresses an important problem of great practical importance. Generalizing 6DoF pose estimation is one of the grand challenges in 3D vision.
- The dataset contribution is important and can help make rapid progress in the community.
- The results are better than other state-of-the-art approaches.
- The idea of using full supervision from synthetic data and partial supervision from real data is obvious, but surprisingly no other paper has tried it as far as I am aware.

## Weaknesses

- Although the proposed dataset has lots of instances, it is limited to 5 categories. I presume this is to keep it consistent with the NOCS synthetic dataset. However, this limitation should be mentioned, in my view.
- While the architecture proposed in Figure 3 makes sense, there is not much justification for *why* this particular architecture was chosen over alternatives. For instance, why PSPNet was chosen? What is the rationale for combining RGB and geometric features in that way?
- The description of the Shape Network is missing details. It's not clear how the categorical prior is implemented. For instance, does it take a random vector as input?
- The results from Mask R-CNN can often be noisy, especially at the boundaries. It's not clear how this influences the result. It would have been nice to have a discussion on this.


questions: It would be nice to see a discussion of the following points (in addition to the above limitations) in the rebuttal:

- How does Mask R-CNN quality affect the performance of the differentiable renderer?
- Could more categories be trained using the proposed approach?
limitations: The paper includes a short limitations section. It would have been nice to include a discussion of societal impacts -- this is missing currently.
review_title: null

Meta-review: metareview: This paper received 4 reviews with the following scores: SR - BR - WA - A. The reviewers acknowledged importance of the addressed problem, the dataset contribution, clear presentation, and a meaningful approach with solid empirical performance. Main disagreements were around comparisons with existing methods (some published at CVPR'22), and fairness of training setups (supervised-only vs augmentation with real data).
The AC confirms that per NeurIPS policies the lack of comparisons with CVPR'22 publications indeed cannot be a basis for rejection. 
However, looking at the additional results in Table 5 (including the CVPR 2022 paper) it looks like methods have actually been compared on both datasets using the various training regimes. Given that and that the remaining concerns of the reviewers were largely addressed in the rebuttal, both the AC and the SAC recommend acceptance. 
confidence: Certain
Decision: Accept