Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: A model is proposed to perform 2d human pose estimation from a monocular image. To train, the model requires 2 images from the same video of the same person: one supervised with a 2d pose and one without supervision. The system simultaneously learns self-supervised keypoints and supervised keypoints, as well as a transformation between them using a transformer network. During inference, this design allows fine-tuning of some parts of the network with additional non-annotated images from the video. The results show that this self-supervised personalization scheme brings improvements over the only-supervised baseline.
main_review: Strengths:
1) Personalizing human pose estimation is an important topic. 
2) The methodology is well explained, self-contained, which makes it easy to follow.
3) The way the splitting of the self-supervised and supervised keypoint estimation is performed (using a transformer to explicitly model their relation using an affinity matrix) is a nice idea.

Weaknesses:

My main concern is the practicality of this approach:

1) First, it has no training regime for learning from datasets of only supervised static images (for example, COCO). This means that it cannot benefit from the large diversity offered by these datasets. 
2) Second, the method is also limited to datasets where the source and target images have small background variation. This is the case during both training and inference personalization. The following argument is being made: “our method has no requirement of consecutive or continuous frames but only unlabeled images belonging to the same person”. Yet, there is no experiment where the same person comes from different videos with different backgrounds (or even just camera motion). 
3) Third, there is no discussion on speed. How does video personalization compare to video-based methods which are far more accurate? Would TTP be practical?
4) Fourth, the results are far from the state of the art. While I understand that the setup is not the same, this is caused mostly by the limitations of the method (e.g.  necessity to use small input resolution).

Clarity: 
1) It should be clear from the abstract that 2d (not 3d) pose estimation is of interest here.
2) L242:  $\Psi^{sup}$ is not defined at this point. 
3) Batching Strategy (from Supp Material) should go into the main paper. It is an important detail without which it is difficult to understand how TTP works.
4) How is $H_t^{gt}$ built?
5) L227-228: [1] actually uses half the distance of the head as a threshold in PCKh@0.5. Is this what you are calculating and reporting? 
6) Complementary to Temporal Methods : The increase in accuracy when performing smoothing is too small so say that “the performance gain of our method does not come from temporal information”. It is also contradicting this statement “After smoothing, our method presents a similar improvement to baseline”. Can you please clarify?


Post Rebuttal:

I thank the other reviewers for their comments and the authors for their answers! I have decided to maintain my rating: 5: Marginally below the acceptance threshold. 
This is mainly motivated by the current method not being able to use annotated static images during training and not convincing enough experiments for background variation. 

limitations_and_societal_impact: Please see Weaknesses in Main Review.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Paper proposes to personalize a human pose estimator during test time without any supervision. This is achieved through training model with self-supervised and supervised keypoint detection objectives jointly. During test time, the model is finetuned using the self-supervised objective with test instances. Doing so ensures personalized keypoint estimations. Results on video 2D pose datasets supports that personalization is helpful. 
main_review: ## Strengths

- Overall, I find the idea neat and refreshing. It is a good application of test time optimization for human pose with a novel technical contribution i.e. usage of transformer for affinity betweeen self-supervised keypoints and semantic keypoints.

- Paper is clear, easy to follow, and nicely written. SupMat provides and explanatory video and the code.

## Weaknesses

### Experiments:

- I appreciate the current set of experiments and analysis conducted on 3 different datasets. However, I think that the selection of datasets is a bit poor. L.21 motivates that such a test time personalization (TTP) is can be helpful for occlusion and unusual poses. But none of these datasets are challenging enough to prove this point. H36M and PennAction have shots where almost full body of the subjects are visible in all frames. BBC only has upper body annotations. PoseTrack dataset as an example is much more rich in terms of challenging scenarios. Results on PoseTrack can improve the presentation of the idea.

- I am a bit confused about the results presented in the supmat pdf and video. Do they show the results before or after TTP? Can you provide before and after results to be able to see the improvement with TTP. Can you also provide results on videos with occlusion and more challenging poses to evaluate the motivation in L.21?

- Authors should provide failure examples and discuss the limitations of the proposed method.

### Other comments:

- L.125: Is this operation differentiable? could you elobarate what this replacement operation look like?

- As a suggestion, the real potential of this method can be shown by training on a large scale unlabeled video dataset with only L_self. Then, learned self-supervised keypoints can be finetuned to a specific dataset through an additional training with L_self + L_sup. This can potentially yield more robustness and improved performance.

- L.320. Did you perform the same experiment with offline TTP?

- EFT[1] is a method that proposes test time optimization for 3D human body regression. Even though the idea is fundamentally different, authors can still discuss it in the related work section.

[1] Joo et al, Exemplar Fine-Tuning for 3D Human Pose Fitting Towards In-the-Wild 3D Human Pose Estimation, Arxiv 2020.

### Post Rebuttal
I appreciate the authors' effort to reply reviewer comments. Several concerns i.e. weak baseline, background variation, limited novelty are raised by reviewers. Authors responded them in a satisfactory way by providing experimental evidence when needed. Even though the experimental setting can be improved, I see this paper as a proof of concept in the direction of personalizing pose estimation methods. Hence, I decided to increase my score to "7: good paper, accept".
limitations_and_societal_impact: Limitations are not discussed properly. It is advised in the main review to provide elaborate discussion of limitations and failures.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a 2D human pose estimation learning framework that uses self-supervision for test-time personalization. The model is first trained with labeled data with both a self-supervised and a supervised task, and then is fine-tuned only with the self-supervised task at test time with unlabeled personalized data. The paper shows consistent improvement over the fully supervised baseline on three datasets under three scenarios (without, with online, and with offline test-time personalization).
main_review: My main concern about the paper is the novelty (as a NeurIPS submission). The test-time personalization framework, and each component used in this framework (image reconstruction with pose and appearance features, Transformer-based attention masking, etc.) have been proposed in previous literature, the paper focuses on their application to the human pose estimation problem.

The paper claims that test-time personalization allows models to generalize better to out-of-distribution data, which is the key point of doing test-time personalization. However, I think the experiments do not sufficiently back up this claim. One missing experiment to show this can be to conduct full training (self-supervised and supervised) on one dataset (e.g., Human3.6M), and apply test-time personalization on a different dataset (e.g., Penn Action). In this case, it would also be interesting to compare with a baseline that trains with labeled data on the main domain (e.g., Human3.6M) with/without a small amount of labeled data on the target domain (e.g., Penn Action).

Line 126, the paper mentions by limiting the heatmap channel number to avoid encoding appearance features. However, this does not guarantee removing appearance features in F_t, which is fed to the Transformer. This potential appearance feature leakage can potentially cause test-time personalization to fail when the supervised domain and the unsupervised test domain have large discrepancy. In this sense, the experiment mentioned in my second point is necessary.

Line 142, any citation for the claim “similar-looking tasks do not necessarily help each other even when sharing features”? Also, it is not clear to me how this leads to the proposal for using a Transformer decoder to “further strengthen their coupling”. Please clarify.

The writing of the test-time personalization online scenario (line 199-201) is not very clear to me. When we have a new testing image at time T, do we run multiple training steps on all the data before time T, or do we only need to train one or a few steps only with this testing image at T? I assume it is the latter (thus the “Update Iteration” experiment in Table 2), because otherwise I wonder how real-time prediction can be possibly achieved. Then it is confusing to me that the paper also mentions using data from s <=T and t <= T (line 201). Please clarify this in Section 3.2.

The statement that “the gap keeps increasing” on line 311 but “starts to drop” on line 315 is confusing. Looking at Figure 3, the gaps always increase and drop for the three datasets.

What architecture is used for the appearance encoder \phi^{app}?

What happens when there is only one image for a target person at test time?

The current method focuses on single-person pose estimation, which I assume relies on person detection. Any thoughts on how to apply this to multi-person problems (detection, pose estimation, etc.)?

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes to train personalized human pose estimators for different people by combining ideas from test-time training [57] with self-supervised keypoint estimation [26]. There is a self-supervised component during training using an image reconstruction task based on [26], and additionally, a transformer is trained in a supervised fashion to map the self-supervised keypoints to supervised keypoints (illustrated in Figure 2). There is a decent set of experiments performed on common human pose datasets - Human3.6M, Penn Action, and BBC Pose. Results demonstrate that the proposed method that is tuned for each subject is able to outperform baselines on pose estimation metrics.
main_review: Strengths - 
1. While elements of the proposed method has been studied in existing works (ex: [57], [26]), applying it to test-time personalization of human pose estimation is a sufficiently new application, and the authors also studied new elements, such as the transformer to map unsupervised keypoint estimates to supervised keypoints.
2. The experiments do demonstrate the authors’ claims that test-time personalization improves pose estimation performance. While it is not surprising that personalizing pose estimators can lead to better results, there is a number of additional experiments in Section 4.4 & 4.5 that provide a few more insights on the model (but I have some additional questions - see weaknesses).

Weaknesses - 

1. I am wondering how camera viewpoint changes, occlusion, and actions taken by the subject during the online or offline test-time personalization phase affect the results. In particular, there are some poses in Human3.6M with heavy occlusion, where I anticipate the self-supervised keypoints would not work well. In cases such as this, is the performance of test-time personalization also lower? I feel the paper would be stronger with some comments about this aspect.
2. The model performance seems sensitive to some parameters. For example, Table 2 shows a big drop in performance on Penn Action with more iterations (would be good to show the effect of iterations on Human3.6M). Overfitting also seems to be an issue based on the experiments on Section 4.4.
3. No error bars are reported, and only a single evaluation run seems to be done. 

limitations_and_societal_impact: I appreciate the paragraph the authors added at the end of the paper on societal impacts, but I’m not clear as to what the authors meant in line 359: “we are committed to limit our methods from any potential malicious usage”. How do you plan on doing this? Seems the method could also be used to train personalized pose detectors without a person’s knowledge or consent (although supervised pose estimation methods have this limitation too, but the proposed method could make this more efficient).
review_title: null

Meta-review: This work addresses the problem of test time personalization for human keypoint estimation from images. It builds on existing self-supervised keypoint discovery methods (e.g. [26]) by adding a Transformer-based adaptor for learning the transformation between self-supervised and supervised keypoints. The novelty lies in the use of a learned affinity matrix that maps between the two representations. During personalization, the model is adapted to the test set in either an online or offline setting. Quantitative results are presented on three different human pose datasets.     

There were concerns in the reviews regarding the choice of datasets used. For instance, in Human3.6M, while there can be changes induced by the different cameras, the overall variation is very small and is highly correlated with the individual cameras. One reviewer was also concerned that the method, as presented, could not be trained on the large numbers of existing static datasets (e.g. COCO) due to the need for source and target frames featuring the same individual. The choice of datasets directly influence the difficulty of the task e.g. the range of poses, the appearance variation in the backgrounds, etc. 

During the discussion, the authors have provided additional experiments investigating occlusion, more training iterations, and cross dataset transfer. These experiments were valuable for better understanding the strengths and weaknesses of the proposed method. In the end, some of the reviewers increased their scores and the majority leaned towards acceptance. The authors are strongly encouraged to include these additional results and explanations in the revised paper along with adding additional discussion of the limitations mentioned by the reviewers of the experiments (e.g. the challenges associated with large changes in background appearance). 

Decision: Accept (Poster)