Test-Time Personalization with a Transformer for Human Pose Estimation

1 Introduction . Recent years have witnessed a large advancement in human pose estimation . A lot of efforts have been spent on learning a generic deep network on large-scale human pose datasets to handle diverse appearance changes [ 59 , 64 , 7 , 15 , 43 ] . Instead of learning a generic model , another line of research is to personalize and customize human pose estimation for a single subject [ 10 ] . For a specific person , we can usually have a long video ( e.g. , instructional videos , news videos ) or multiple photos from personal devices . With these data , we can adapt the model to capture the person-specific features for improving pose estimation and handling occlusion and unusual poses . However , the cost of labeling large-scale data for just one person is high and unrealistic . In this paper , we propose to personalize human pose estimation with unlabeled video data during test time , namely , Test-Time Personalization . Our setting falls in the general paradigm of Test-Time Adaptation [ 58 , 35 , 61 , 69 ] , where a generic model is first trained with diverse data , and then it is fine-tuned to adapt to a specific instance during test time without using human supervision . This allows the model to generalize to out-of-distribution data and preserves privacy when training is distributed . Specifically , Sun et al . [ 58 ] propose to generalize image classification by performing joint training with a semantic classification task and a self-supervised image rotation prediction task [ 18 ] . During inference , the shared network representation is fine-tuned on the test instance with the self-supervisory signal for adaptation . While the empirical result is encouraging , it is unclear how ∗Equal contribution 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the rotation prediction task can help image classification , and what is the relation between two tasks besides sharing the same feature backbone . Going beyond feature sharing with two distinct tasks , we introduce to perform joint supervised and self-supervised human keypoint estimation [ 26 ] tasks where the supervised keypoint outputs are directly transformed from the self-supervised keypoints using a Transformer [ 60 ] . In this way , when fine-tuning with the self-supervised task in test time , the supervised pose estimation can be improved by transforming from the improved self-supervised keypoints . We adapt the self-supervised keypoint estimation task proposed by Jakab et al . [ 26 ] . The task is built on the assumption that the human usually maintains the appearance but changes poses across time in a video . Given a video frame , it trains a network to extract a tight bottleneck in the form of sparse spatial heatmaps , which only contain pose information without appearance . The training objective is to reconstruct the same frame by combining the bottleneck heatmaps and the appearance feature extracted from another frame . Note while this framework can extract keypoints to represent the human structure , they are not aligned with the semantic keypoints defined in human pose estimation . Building on this model , we add an extra keypoint estimation objective which is trained with human supervision . Instead of simply sharing features between two objectives as [ 58 ] , we train a Transformer model on top of the feature backbone to extract the relation and affinity matrix between the self-supervised keypoint heatmap and the supervised keypoint heatmap . We then use the affinity matrix to transform the self-supervised keypoints as the supervised keypoint outputs . With our Transformer design , it not only increases the correlation between two tasks when training but also improves Test-Time Personalization as changing one output will directly contribute to the the output of another task . We perform our experiments with multiple human pose estimation datasets including Human 3.6M [ 24 ] , Penn Action [ 71 ] , and BBC Pose [ 8 ] datasets . As shown in Figure 1 , our Test-Time Personalization can perform on frames that continuously exist in a video and also with frames that are non-continuous as long as they are for the same person . We show that by using our approach for personalizing human pose estimation in test time , we achieve significant improvements over baselines in all datasets . More interestingly , the performance of our method improves with more video frames appearing online for the same person during test time . 2 Related Work . Human Pose Estimation . Human pose estimation has been extensively studied and achieved great advancements in the past few years [ 59 , 64 , 7 , 15 , 43 , 67 , 45 , 21 , 65 , 13 , 57 , 75 , 44 , 5 , 14 ] . For example , Toshev et al . [ 59 ] propose to regress the keypoint locations from the input images . Instead of direct location regression , Wei et al . [ 64 ] propose to apply a cascade framework for coarse to fine heatmap prediction and achieve significant improvement . Building on this line of research , Xiao et al . [ 65 ] provides a simple and good practice on heatmap-based pose estimation , which is utilized as our baseline model . While in our experiments we utilize video data for training , our model is a singleimage pose estimator and it is fundamentally different from video pose estimation models [ 1 , 19 , 62 ] which take multiple continuous frames as inputs . This gives our model the flexibility to perform pose estimation on static images and thus it is not directly comparable to approaches with video inputs . Our work is also related to personalization on human pose estimation from Charles et al . [ 10 ] , which uses multiple temporal and continuity constraints to propagate the keypoints to generate more training data . Instead of tracking keypoints , we use a self-supervised objective to perform personalization in test time . Our method is not restricted to the continuity between close frames , and the self-supervision can be applied on any two frames far away in a video as long as they belong to the same person . Test-Time Adaptation . Our personalization setting falls into the paradigm of Test-Time Adaptation which is recently proposed in [ 51 , 50 , 3 , 58 , 35 , 61 , 69 , 28 , 42 , 20 ] for generalization to out-ofdistribution test data . For example , Shocher et al . [ 51 ] propose a super-resolution framework which is only trained during test time with a single image via down-scaling the image to create training pairs . Wang et al . [ 61 ] introduce to use entropy of the classification probability distribution to provide fine-tuning signals when given a test image . Instead of optimizing the main task itself during test time , Sun et al . [ 58 ] propose to utilize a self-supervised rotation prediction task to help improve the visual representation during inference , which indirectly improves semantic classification . Going beyond image classification , Joo et al . [ 30 ] propose a method that proposes test time optimization for 3D human body regression . In our work for pose personalization , we try to bridge the self-supervised and supervised objectives close . We leverage a self-supervised keypoint estimation task and transform the self-supervised keypoints to supervised keypoints via a Transformer model . In this way , training with self-supervision will directly improve the supervised keypoint outputs . Self-supervised Keypoint Estimation . There are a lot of recent developments on learning keypoint representations with self-supervision [ 55 , 72 , 26 , 38 , 32 , 27 , 68 , 36 , 40 ] . For example , Jakab et al . [ 26 ] propose a video frame reconstruction task which disentangles the appearance feature and keypoint structure in the bottleneck . This work is then extended for control and Reinforcement Learning [ 32 , 36 , 40 ] , and the keypoints can be mapped to manual defined human pose via adding adversarial learning loss [ 27 ] . While the results are encouraging , most of the results are reported in relatively simple scenes and environments . In our paper , by leveraging the self-supervised task together with the supervised task , we can perform human pose personalization on images in the wild . Transformers . Transformer has been widely applied in both language processing [ 60 , 16 ] and computer vision tasks [ 63 , 46 , 23 , 49 , 56 , 17 , 11 , 4 , 73 , 6 , 37 ] , specifically for pose estimation recently [ 66 , 54 , 41 , 33 ] . For example , Li et al . [ 33 ] propose to utilize the encoder-decoder model in Transformers to perform keypoint regression , which allows for more general-purpose applications and requires less priors in architecture design . Inspired by these works , we apply Transformer to reason the relation and mapping between the supervised and self-supervised keypoints . 3 Method . Our method aims at generalizing better for pose estimation on a single image by personalizing with unlabeled data . The model is firstly trained with diverse data on both a supervised pose estimation task and a self-supervised keypoint estimation task , using a proposed Transformer design to model the relation between two tasks . During inference , the model conducts Test-Time Personalization which only requires the self-supervised keypoint estimation task , boosting performance without costly labeling or sacrificing privacy . The whole pipeline is shown in Figure 2 . 3.1 Joint Training for Pose Estimation with a Transformer . Given a set of N labeled images of a single person I = { I1 , I2 . . . , IN } , a shared encoder φ maps them into feature space F = { F1 , F2 . . . , FN } , which is shared by both a supervised and a selfsupervised keypoint estimation tasks . We introduce both tasks and the joint framework as follows . 3.1.1 Self-supervised Keypoint Estimation . For the self-supervised task , we build upon the work from Jakab et al . [ 26 ] which uses an image reconstruction task to perform disentanglement of human structure and appearance , which leads to self-supervised keypoints as intermediate results . Given two images of a single person Is and It , the task aims at reconstructing It using structural keypoint information from target It and appearance information from source Is . The appearance information F app s of source image Is is extracted with a simple extractor φapp ( see the bottom stream in Figure 2 ) . The extraction of keypoints information from the target image follows three steps as below ( also the see the middle stream in Figure 2 ) . Firstly , the target image It is forwarded to the encoder φ to obtain shared feature Ft . The selfsupervised head ψself further encodes the shared feature Ft into heatmaps Hselft . Note the number of channels in the heatmap Hselft is equal to the number of self-supervised keypoints . Secondly , H self t is normalized using a Softmax function and thus becomes condensed keypoints . In the third step , the heatmaps are replaced with fixed Gaussian distribution centered at condensed points , which serves as keypoint information F kpt . These three steps ensure a bottleneck of keypoint information , ensuring there is not enough capacity to encode appearance features to avoid trivial solutions . The objective of the self-supervised task is to reconstruct the target image with a decoder using both appearance and keypoint features : Ît = φrender ( F apps , F kp t ) . Since the bottleneck structure from the target stream limits the information to be passed in the form of keypoints , the image reconstruction enforces the disentanglement and the network has to borrow appearance information from source stream . The Perceptual loss [ 29 ] and L2 distance are utilized as the reconstruction objective , Lself = PerceptualLoss ( It , Ît ) + ∥∥∥It − Ît∥∥∥2 ( 1 ) Instead of self-supervised tasks like image rotation prediction [ 18 ] or colorization [ 70 ] , choosing an explicitly related self-supervised key-point task in joint training naturally preserves or even improves performance , and it is more beneficial to test-time personalization . Attention should be paid that our method requires only label of one single image and unlabeled samples belonging to the same person . Compared to multiple labeled samples of the same person or even more costly consecutively labeled video , acquiring such data is much more easier and efficient .

Summary: A model is proposed to perform 2d human pose estimation from a monocular image. To train, the model requires 2 images from the same video of the same person: one supervised with a 2d pose and one without supervision. The system simultaneously learns self-supervised keypoints and supervised keypoints, as well as a transformation between them using a transformer network. During inference, this design allows fine-tuning of some parts of the network with additional non-annotated images from the video. The results show that this self-supervised personalization scheme brings improvements over the only-supervised baseline.