Test-Time Personalization with a Transformer for Human Pose Estimation

1 Introduction . Recent years have witnessed a large advancement in human pose estimation . A lot of efforts have been spent on learning a generic deep network on large-scale human pose datasets to handle diverse appearance changes [ 59 , 64 , 7 , 15 , 43 ] . Instead of learning a generic model , another line of research is to personalize and customize human pose estimation for a single subject [ 10 ] . For a specific person , we can usually have a long video ( e.g. , instructional videos , news videos ) or multiple photos from personal devices . With these data , we can adapt the model to capture the person-specific features for improving pose estimation and handling occlusion and unusual poses . However , the cost of labeling large-scale data for just one person is high and unrealistic . In this paper , we propose to personalize human pose estimation with unlabeled video data during test time , namely , Test-Time Personalization . Our setting falls in the general paradigm of Test-Time Adaptation [ 58 , 35 , 61 , 69 ] , where a generic model is first trained with diverse data , and then it is fine-tuned to adapt to a specific instance during test time without using human supervision . This allows the model to generalize to out-of-distribution data and preserves privacy when training is distributed . Specifically , Sun et al . [ 58 ] propose to generalize image classification by performing joint training with a semantic classification task and a self-supervised image rotation prediction task [ 18 ] . During inference , the shared network representation is fine-tuned on the test instance with the self-supervisory signal for adaptation . While the empirical result is encouraging , it is unclear how ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the rotation prediction task can help image classification , and what is the relation between two tasks besides sharing the same feature backbone . Going beyond feature sharing with two distinct tasks , we introduce to perform joint supervised and self-supervised human keypoint estimation [ 26 ] tasks where the supervised keypoint outputs are directly transformed from the self-supervised keypoints using a Transformer [ 60 ] . In this way , when fine-tuning with the self-supervised task in test time , the supervised pose estimation can be improved by transforming from the improved self-supervised keypoints . We adapt the self-supervised keypoint estimation task proposed by Jakab et al . [ 26 ] . The task is built on the assumption that the human usually maintains the appearance but changes poses across time in a video . Given a video frame , it trains a network to extract a tight bottleneck in the form of sparse spatial heatmaps , which only contain pose information without appearance . The training objective is to reconstruct the same frame by combining the bottleneck heatmaps and the appearance feature extracted from another frame . Note while this framework can extract keypoints to represent the human structure , they are not aligned with the semantic keypoints defined in human pose estimation . Building on this model , we add an extra keypoint estimation objective which is trained with human supervision . Instead of simply sharing features between two objectives as [ 58 ] , we train a Transformer model on top of the feature backbone to extract the relation and affinity matrix between the self-supervised keypoint heatmap and the supervised keypoint heatmap . We then use the affinity matrix to transform the self-supervised keypoints as the supervised keypoint outputs . With our Transformer design , it not only increases the correlation between two tasks when training but also improves Test-Time Personalization as changing one output will directly contribute to the the output of another task . We perform our experiments with multiple human pose estimation datasets including Human 3.6M [ 24 ] , Penn Action [ 71 ] , and BBC Pose [ 8 ] datasets . As shown in Figure 1 , our Test-Time Personalization can perform on frames that continuously exist in a video and also with frames that are non-continuous as long as they are for the same person . We show that by using our approach for personalizing human pose estimation in test time , we achieve significant improvements over baselines in all datasets . More interestingly , the performance of our method improves with more video frames appearing online for the same person during test time . 2 Related Work . Human Pose Estimation . Human pose estimation has been extensively studied and achieved great advancements in the past few years [ 59 , 64 , 7 , 15 , 43 , 67 , 45 , 21 , 65 , 13 , 57 , 75 , 44 , 5 , 14 ] . For example , Toshev et al . [ 59 ] propose to regress the keypoint locations from the input images . Instead of direct location regression , Wei et al . [ 64 ] propose to apply a cascade framework for coarse to fine heatmap prediction and achieve significant improvement . Building on this line of research , Xiao et al . [ 65 ] provides a simple and good practice on heatmap-based pose estimation , which is utilized as our baseline model . While in our experiments we utilize video data for training , our model is a singleimage pose estimator and it is fundamentally different from video pose estimation models [ 1 , 19 , 62 ] which take multiple continuous frames as inputs . This gives our model the flexibility to perform pose estimation on static images and thus it is not directly comparable to approaches with video inputs . Our work is also related to personalization on human pose estimation from Charles et al . [ 10 ] , which uses multiple temporal and continuity constraints to propagate the keypoints to generate more training data . Instead of tracking keypoints , we use a self-supervised objective to perform personalization in test time . Our method is not restricted to the continuity between close frames , and the self-supervision can be applied on any two frames far away in a video as long as they belong to the same person . Test-Time Adaptation . Our personalization setting falls into the paradigm of Test-Time Adaptation which is recently proposed in [ 51 , 50 , 3 , 58 , 35 , 61 , 69 , 28 , 42 , 20 ] for generalization to out-ofdistribution test data . For example , Shocher et al . [ 51 ] propose a super-resolution framework which is only trained during test time with a single image via down-scaling the image to create training pairs . Wang et al . [ 61 ] introduce to use entropy of the classification probability distribution to provide fine-tuning signals when given a test image . Instead of optimizing the main task itself during test time , Sun et al . [ 58 ] propose to utilize a self-supervised rotation prediction task to help improve the visual representation during inference , which indirectly improves semantic classification . Going beyond image classification , Joo et al . [ 30 ] propose a method that proposes test time optimization for 3D human body regression . In our work for pose personalization , we try to bridge the self-supervised and supervised objectives close . We leverage a self-supervised keypoint estimation task and transform the self-supervised keypoints to supervised keypoints via a Transformer model . In this way , training with self-supervision will directly improve the supervised keypoint outputs . Self-supervised Keypoint Estimation . There are a lot of recent developments on learning keypoint representations with self-supervision [ 55 , 72 , 26 , 38 , 32 , 27 , 68 , 36 , 40 ] . For example , Jakab et al . [ 26 ] propose a video frame reconstruction task which disentangles the appearance feature and keypoint structure in the bottleneck . This work is then extended for control and Reinforcement Learning [ 32 , 36 , 40 ] , and the keypoints can be mapped to manual defined human pose via adding adversarial learning loss [ 27 ] . While the results are encouraging , most of the results are reported in relatively simple scenes and environments . In our paper , by leveraging the self-supervised task together with the supervised task , we can perform human pose personalization on images in the wild . Transformers . Transformer has been widely applied in both language processing [ 60 , 16 ] and computer vision tasks [ 63 , 46 , 23 , 49 , 56 , 17 , 11 , 4 , 73 , 6 , 37 ] , specifically for pose estimation recently [ 66 , 54 , 41 , 33 ] . For example , Li et al . [ 33 ] propose to utilize the encoder-decoder model in Transformers to perform keypoint regression , which allows for more general-purpose applications and requires less priors in architecture design . Inspired by these works , we apply Transformer to reason the relation and mapping between the supervised and self-supervised keypoints . 3 Method . Our method aims at generalizing better for pose estimation on a single image by personalizing with unlabeled data . The model is firstly trained with diverse data on both a supervised pose estimation task and a self-supervised keypoint estimation task , using a proposed Transformer design to model the relation between two tasks . During inference , the model conducts Test-Time Personalization which only requires the self-supervised keypoint estimation task , boosting performance without costly labeling or sacrificing privacy . The whole pipeline is shown in Figure 2 . 3.1 Joint Training for Pose Estimation with a Transformer . Given a set of N labeled images of a single person I = { I1 , I2 . . . , IN } , a shared encoder φ maps them into feature space F = { F1 , F2 . . . , FN } , which is shared by both a supervised and a selfsupervised keypoint estimation tasks . We introduce both tasks and the joint framework as follows . 3.1.1 Self-supervised Keypoint Estimation . For the self-supervised task , we build upon the work from Jakab et al . [ 26 ] which uses an image reconstruction task to perform disentanglement of human structure and appearance , which leads to self-supervised keypoints as intermediate results . Given two images of a single person Is and It , the task aims at reconstructing It using structural keypoint information from target It and appearance information from source Is . The appearance information F app s of source image Is is extracted with a simple extractor φapp ( see the bottom stream in Figure 2 ) . The extraction of keypoints information from the target image follows three steps as below ( also the see the middle stream in Figure 2 ) . Firstly , the target image It is forwarded to the encoder φ to obtain shared feature Ft . The selfsupervised head ψself further encodes the shared feature Ft into heatmaps Hselft . Note the number of channels in the heatmap Hselft is equal to the number of self-supervised keypoints . Secondly , H self t is normalized using a Softmax function and thus becomes condensed keypoints . In the third step , the heatmaps are replaced with fixed Gaussian distribution centered at condensed points , which serves as keypoint information F kpt . These three steps ensure a bottleneck of keypoint information , ensuring there is not enough capacity to encode appearance features to avoid trivial solutions . The objective of the self-supervised task is to reconstruct the target image with a decoder using both appearance and keypoint features : Ît = φrender ( F apps , F kp t ) . Since the bottleneck structure from the target stream limits the information to be passed in the form of keypoints , the image reconstruction enforces the disentanglement and the network has to borrow appearance information from source stream . The Perceptual loss [ 29 ] and L2 distance are utilized as the reconstruction objective , Lself = PerceptualLoss ( It , Ît ) + ∥∥∥It − Ît∥∥∥2 ( 1 ) Instead of self-supervised tasks like image rotation prediction [ 18 ] or colorization [ 70 ] , choosing an explicitly related self-supervised key-point task in joint training naturally preserves or even improves performance , and it is more beneficial to test-time personalization . Attention should be paid that our method requires only label of one single image and unlabeled samples belonging to the same person . Compared to multiple labeled samples of the same person or even more costly consecutively labeled video , acquiring such data is much more easier and efficient .

Summary: This paper proposes a 2D human pose estimation learning framework that uses self-supervision for test-time personalization. The model is first trained with labeled data with both a self-supervised and a supervised task, and then is fine-tuned only with the self-supervised task at test time with unlabeled personalized data. The paper shows consistent improvement over the fully supervised baseline on three datasets under three scenarios (without, with online, and with offline test-time personalization).