Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: ## Summary

Paper proposes a self-adaptation framework to align the unpaired samples of 3D pose and video frames to solve 3D human pose estimation from in-the-wild images. This cross-modal alignment is realized through a relation distillation scheme. The proposed approach achieves compelling results on 3D human pose datasets among self-supervised and unsupervised methods.
main_review: ## Strengths

- Paper tackles a real problem of scarcity of 3D pose labels for in-the-wild unconstrained images. They propose a self-adaptation method that helps to utilize web videos and large-scale motion capture datasets both of which are available in abundance.  

- Proposed domain adaptation approach sounds novel. Motivation and implementation make sense to be able to perform unsupervised adaptation.

- Experiments are detailed. Three major human pose estimation datasets are used for the model's performance. Ablation experiments show that propose the contribution of each relation distillation term into the final performance.

## Weaknesses

- Paper writing is quite hard to follow. Especially in the intro, there are lots of technical terms which is not familiar to the human pose estimation community. These are mostly domain adaptation or knowledge distillation terms and they should be introduced in a logical flow to the reader.

- CMU mocap dataset is chosen as the unpaired 3D pose dataset. AMASS dataset is much larger in terms of quantity, the number of subjects, pose variation, and actions performed. It also includes CMU as a subset. So, AMASS could be a better choice for the proposed method.

- A motivation of the proposed approach is overcoming the dataset bias. However, the Dp is kept frozen when used with G. This means that the manifold learned/encoded by Z is biased towards the unpaired 3D pose dataset which is CMU mocap in this case. I think this is a major limitation of the current method and this should be tackled to overcome the dataset bias.

### Post Rebuttal

Authors addressed my comments in a satisfactory way. Considering the other reviews and author's responses to them I would like to keep my initial score of "6: Marginally above the acceptance threshold".
limitations_and_societal_impact: Yes, the limitations and societal impact are discussed in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposed to learn to predict the 3D human pose for real-world images via adapting the model from the source data (label constraints) to unlabeled data. The data used for training contains: (1) the labeled source domain data (e.g. human3.6m or surreal); (2) real human video sequence without pose annotations; and (3) motion capture data (3D pose without images). (2) and (3) are unpaired. The authors proposed to devise: (a) image-to-latent; (b) pose-to-latent; and (c) latent-to-pose, to learn and predict 3D poses. In particular, (a) and (c) forms the test time predictions for the real-world target domain, while (b) and (c) constitutes an auto-encoder. The core problems in here is that the latent space in (ac) and (bc) might be mismatched. The authors proposed a smart way to make sure they are aligned (see Strength below) thanks to the contrastive loss (for setting up latent space non-local grouping rules) as well as the loss from the source domain data for regularization to make sure the two latent domains align with each other. 
main_review: Strength:

I really appreciate the main novelty of the paper. To put the spark of the paper in one sentence: the paper proposed an innovated self supervised method to align the human pose latent space from unpaired image-to-latent and the pose-to-latent, via enforcing several latent space rules that is available for unpaired image-to-latent and pose-to-latent domains (e.g. symmetric poses should be close, named as the "positive non-local coupling" by the author). The loss from the source data (D^s) would serve as the extra constraint to make sure the two domains align. As long as this is the first time to propose this type of the human-pose-specific rules of latent space (I have not checked existing literature thoroughly), I would be quite positive on this paper. The experimental part provides fairly good results (while a notch below very strong visual results, see below weakness part).

Weaknesses:

1. I have moderate concerns to the experimental part, as the visual results and comparisons are not particularly exciting. Comparisons to several SoTA seems missing (e.g. Frank [a] and SMPL-X [b]) although these are not self-supervised or domain adaptation approaches. I wish the authors can also provide prediction overlay and their rotated 3D visual results. The current given visualization seems quite difficult to judge the accuracy. Besides, since one of the major domain is the video and pose sequence, I hope the authors can provide results for several human video clips.

2. The writing of the paper seems not smooth to me, and I need to take efforts to reach the spark of the paper. It seems the introduction part did not well express the core novelty of the proposed framework. I would suggest the authors add one small illustrative figures in the intro part, particularly illustrating the non-local positive coupling is available in both the image-to-latent and the pose-to-latent mappings. I hope that can help the readers to get the message very quickly. 

3. Small issue: It seems the term "non-local" used in this paper is irrelevant to that in [c]. I wish the authors can either provide a clarification in the related works section, or use a slightly different name. 

[a] Joo, Hanbyul, Tomas Simon, and Yaser Sheikh. "Total capture: A 3d deformation model for tracking faces, hands, and bodies." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8320-8329. 2018.

[b] Pavlakos, Georgios, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J. Black. "Expressive body capture: 3d hands, face, and body from a single image." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10975-10985. 2019.

[c] Wang, Xiaolong, Ross Girshick, Abhinav Gupta, and Kaiming He. "Non-local neural networks." In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803. 2018.

Considering the good novelty and the fair experimental results of the paper, I rate this work as "borderline accept".

** Post Rebuttal ** After reading all the reviews and the authors' responses, I believe this paper provided innovated insights, while all the reviewers mentioned the clarity concern of the paper. I would keep my original rating considering all these factors.

limitations_and_societal_impact: The authors have addressed them well in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This work regards monocular 3D human pose estimation as a domain adaptation problem to solve the problem of insufficient outdoor training data. The framework is composed of an image-to-latent encoder, which is pre-trained on the source domain and adapted to the target domain using unlabeled videos, and a latent-to-pose decoder, which is trained on unpaired 3D poses. During the encoder adaptation, non-local relations are utilized to improve the performance, which is the main contribution.

main_review: Pros:
Non-local relations are involved in the domain adaptation, which is novel and proved to be effective by experiments.
Achieve SOTA results on several datasets under the semi-supervised and unsupervised settings.
Well organized and easy to follow.

Cons:
There is an important hypothesis in this paper: Large embedding distance of non-local motion relations is beneficial to the domain adaptation. However, this hypothesis is not carefully verified. An ablation study with different embedding distances is needed, which can be achieved with in-plane rotation with different angles.

When compared with other methods, the training data is different, which makes the comparison unfair.

In the experiment, larger rotation angle results in a larger embedding distance, so is it possible to achieve a larger embedding distance than flipping by increasing rotation angle or combining flipping and rotation to a new motion? Will this further improve the performance?

Some Typos:
In table5, the metric should be PA-MPJPE rather than MPJPE. And we recommend showing both MPJPE and PA-MPJPE.

limitations_and_societal_impact: Yes. Both limitation and societal impact is addressed in this paper.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper focuses on 3D human pose estimation in a scenario of domain adaptation. To avoid dataset-bias, the authors address the problem in a more restrict setting: any form of paired supervision or auxiliary cues should not be used. 

The authors first use AAE modules to learn pose and motion latent spaces. Secondly, the authors train a pose encoder which tries to map the input image to the learned pose latent space. Finally, given the target-domain videos, the authors train the pose encoder together with the decoder of the pose AAE. The model is optimized by minimizing relational energies. 
main_review: Strength
- This paper present a good solution for domain adaptive 3D human pose estimation. Based on the proposed setup, it is technically valid, and it should work well.
- Good figures to better illustrate the proposed idea and the learned embedding.
- Good ablation study of the use of different relational energies.
- The results are strong. In some cases, the proposed method even outperforms the multi-view approaches.


Weakness
- The presentation is very difficult to follow. This is probably due to the writing style of the authors. Most of the things are described in a very abstract manner. It is difficult for me to understand the proposed method in details. I spent hours to read the paper, but still I am not confident about all details in the proposed solution.  Also, I can't find strong connections with the cognitive science. The authors tend to introduce new terms, but I find them a bit distracted.  

- The proposed method involves multi-stage training, but not end-to-end training. Each training stage includes a new model, and each would affect the final performance. Given many uncertainties, it is difficult to understand the performance. I would suggest to ablate each training stage, and show the effectiveness of each module.

- What is performance of the AAEs? What quality of the pose embedding is required? Same question for the motion embedding. 

- Instead of training AAEs to generate pre-learned latent feature, the authors may want to use SMPL pose parameters as an alternative choice of the latent feature. Since SMPL parameters are also pre-trained and commonly used in the literature, this may serve as a good baseline to better promote the proposed method.  

- Why should we use a pre-learned output pose embedding, but not fully end-to-end learned embedding? The pre-learned embedding may lead to sub-optimal performance. The design choice of the proposed method is not well motivated. 

- Lack of baselines: One important baseline is to use pseudo pairs, and perform distillation, as the authors described in line181. Another baseline is domain adaptation with adversarial training. It is better to provide concrete performance comparisons to justify the proposed method.

- The authors argue that they do not use paired supervision or auxiliary cues to avoid dataset bias. However, the proposed method replies on Faster RCNN, which may imply it still require supervised pre-training on large number of paired training data. The authors may want to add more discussions about how they avoid the dataset bias.


Overall:
- The paper introduces an interesting method for human pose estimation in the setting of domain adaptation. However, due to bad writing, it is very difficult to follow (at least to me). Many things remain unclear in the paper. Since the target scenario is relatively different from prior works, I would suggest the authors to build their own baselines, in order to better justify the proposed solution. After careful consideration, I think this paper is not ready for a publication.


*******Post rebuttal*********
Thank you for the detailed response. My overall comments remain the same. I think the presentation is difficult to follow, and requires several revisions. 

limitations_and_societal_impact: n/a
review_title: null

Meta-review: This paper studies the problem of 3D human pose estimation by combining information from labeled sources and unlabeled data in the wild. The paper received mixed reviews, in the beginning, tending near borderline. The reviewers' major concern was regarding the presentation clarity. Multiple reviewers found it hard to follow. The authors provided a rebuttal that addressed some of the reviewers' concerns and promise to improve the presentation. The paper was discussed and most reviewers responded to the rebuttal. One of the reviewers raised their rating but one reviewer did not. The main complaint is still around the writing and presentation. AC agrees with the reviewers that the paper has a good contribution to be accepted and urges the authors to look at the reviewers' feedback, incorporate their comments and clarify the writing as promised in the camera-ready.
Decision: Accept (Poster)