Non-local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation

1 Introduction . Human pose estimation systems have garnered immense attention due to their innumerable applications [ 55 , 19 , 85 ] . The successes of such systems are mostly driven by large-scale datasets containing images with paired 3D pose annotations [ 25 ] . Unlike 2D joint landmarks , annotating a 3D pose requires body-worn sensors or multi-camera structure-from-motion setup [ 71 ] which is challenging to install outdoors . Hence , the available datasets are either captured in constrained laboratory settings or are limited in size and diversity . Unsurprisingly , models trained on such datasets exhibit poor cross-dataset generalization . Addressing this , several approaches [ 11 , 77 , 56 , 34 ] resort to weaklysupervised methods that rely on images with paired 2D landmark annotations . Certain methods require additional supervision such as depth [ 81 , 11 ] or multi-view image pairs [ 66 , 33 ] . However , they still suffer from dataset-bias due to their strong reliance on some form of paired supervision . In this work , we digress from any form of paired supervision or auxiliary cues ( multi-view or depth ) thereby avoiding the curse of dataset-bias . We thus introduce a self-supervised domain adaptation framework for 3D human pose estimation ( Fig . 1 ) . In the proposed setting , we consider access to three different datasets . First , a labeled source dataset obtained from graphics-based synthetic environments such as SURREAL [ 79 ] or in-studio datasets such as Human3.6M [ 25 ] . Second , unlabeled video sequences from the target domain . Third , a set of unpaired 3D pose sequences . Following this , image-to-pose inference is carried out via two explicit mappings i.e. , image-to-latent CNN followed by a latent-to-pose network . Here , the latent-to-pose network is pre-learned as a decoder of a prior-enforcing generative adversarial auto-encoder [ 47 ] ( AAE ) in order to restrict the 1Webpage : https : //sites.google.com/view/sa3dhp 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . pose predictions within the plausibility limits . We follow the same to pre-learn a generative motion embedding via a recurrent AAE setup which operates on the latent pose space instead of the raw pose sequences . Both pose and motion embeddings are trained solely on the unpaired 3D pose dataset . Next , we prepare image-to-latent CNN by supervising on the labeled source dataset . After finishing the above pre-learning steps , our prime objective is to train or adapt only the image-to-latent CNN so that it can perform well on the unlabeled target domain samples . In order to align the output manifold of the image-to-latent with the pre-learned latent pose manifold ( in the absence of cross-modal pairs ) , one must formalize innovative ways to represent the dark-knowledge based on which the student ( i.e . image-to-latent ) output can align to that of the teacher ( i.e . pose-to-latent ) . Several studies [ 18 , 20 ] on human cognitive development advocate that , in a self-supervised paradigm , new knowledge is acquired by relating entities based on some semantic rule . Motivated by this , we explore different ways of formalizing inter-entity relations . A relation can be characterized as lower order or higher order based on the number of entities that participate in defining a relationship tuple . For instance , in contrastive learning [ 57 , 12 ] , a pose space triplet relation expresses a coupling of just 3 pose entities , and is thus considered a lower order relation . However , a similar contrastive triplet defined in the hierarchical motion space ( i.e . temporal pose sequences of fixed sequence length T ) would couple 3T pose entities , thus is considered a higher order relation . Next , we adapt the target-specific image-tolatent by minimizing the relational energies derived from the contrastive relations . Unlike in prior-arts [ 12 , 51 ] where the output embedding is learned from scratch , we are restricted to operate on a pre-learned output pose embedding . Consequently , the model often converges to a degenerate solution exhibiting instance-level misalignment [ 46 ] . We realize that the positive coupling in contrastive relations is limited to local neighborhood structures resulting in suboptimal alignment . This motivates us to develop a new set of relations that would express positive coupling of diverse non-local relations ( beyond the structural neighborhood ) , thereby characterizing long-range latent pose interactions in a much effective manner . We define tangible non-local relations separately in the pose and motion space that are categorized under a ) lower order non-local relations and b ) higher order non-local relations , respectively . For instance , “ pose-flip ” is a lower order non-local relation which associates anchor poses with their left-right flipped versions . Similarly , “ flip-backward ” is a higher order non-local relation which couples anchor pose-sequences with their flip-backward counterpart which is obtained via temporal reversal ( backwards ) of the individually flipped frames . Here , the corresponding relational energy is devised via latent space relation networks . These are essentially frozen neural networks that are trained to regress the latent embedding of the relational counterpart given latent embedding of the anchor as input . In a nutshell , the relational energies aim to preserve the equivariance of higher order spatio-temporal relations between the two modalities as a means to perform the cross-modal alignment . It turns out that , among various relations , relations coupling the most diverse non-local samples result in a better cross-modal alignment . We perform extensive experiments to validate the efficacy of our approach and demonstrate superior generalizability on samples from in-the-wild environments . We summarize our contributions as follows : • The proposed solution for self-adapting 3D human pose model involves cross-modal alignment between the unpaired samples from the input and output modalities , via relation distillation . Highly non-local instances are associated using novel relation networks to specifically cater to the instance-level misalignment . • We provide insights to select the most effective non-local relations . This involves quantifying non-localness of a relation as the average latent-distance between the coupled entities . • We evaluate different self-adaptation settings and demonstrate state-of-the-art 3D human pose estimation performance against the available semi-supervised and weakly-supervised prior arts on Human3.6M [ 25 ] , MPI-INF-3DHP [ 49 ] , and 3DPW [ 80 ] . 2 Related Works . Table 1 shows a comparison of the proposed setting against the prior-arts for the task of 3D human pose estimation . Our framework is equipped with self-adaptation capability while only utilizing unpaired 3D poses and synthetic 3D pose supervision . Knowledge distillation ( KD ) . The process involves transferring knowledge from a larger model to a smaller model to learn a concise knowledge representation with negligible information loss . Faster evaluation and reduced memory footprint renders distilled models more deployment friendly . More specifically , in KD [ 24 ] , the smaller model ( student ) focuses on mimicking the output behaviour of a pre-trained larger model ( teacher ) . Traditional techniques aim to match instance level behaviours such as matching the output activation [ 68 ] , gram matrix [ 87 ] , gradient [ 72 ] , attention maps [ 88 ] , etc . Recently , several techniques proposed to align pairwise distance [ 76 , 63 ] or similarity graphs [ 58 , 43 ] to look beyond the instance-level matching . Further , to capture higher order output dependencies , researchers adopt techniques [ 75 ] inspired from the advances in self-supervised contrastive learning literature [ 57 , 4 ] . Essentially , a balanced distillation of both structural and instance-level knowledge leads to superior generalization [ 83 ] . However , in the proposed setting , the teacher , pose-to-latent and the student , image-to-latent do not share the same input modality . Further , the absence of paired correspondence restricts us from directly employing such distillation techniques . Monocular 3D human pose estimation . There are two broad categories of models that infer 3D poses from a single image , i.e . a ) those using model-based parametric representation [ 5 , 8 , 62 , 7 ] and b ) others that infer the 3D pose representation directly [ 69 , 2 , 6 ] . Models belonging to the first category map the input image to the latent parameters of a pre-defined parametric human model . This setting establishes the base to impose various pose priors through techniques such as adversarial training . Recent works [ 28 , 61 ] have also aimed to build on the parametric body models and extend it to express finer movements such as hand gestures and facial expressions . Those belonging to the second directly map the input image to the corresponding 3D pose . These models are further categorized into a ) one-stage methods [ 93 , 73 , 59 , 54 , 60 , 86 ] that directly map input image to the 3D pose , and b ) two-stage methods [ 92 , 48 , 53 , 73 ] that adopt a mapping from the image to an intermediate 2D pose , followed by lifting of 2D-to-3D . In our work , the shared latent pose can be seen as a parametric form to represent plausible 3D poses . Domain Adaptation . A model trained on some source data may not be a suitable choice for deployment in its nascent form to an unknown target environment . In such a scenario , domain adaptation aims to tackle this domain shift induced due to the difference in source and target data distributions by adapting the model trained on some related labeled source domain to the target domain . A number of works [ 37 ] aim to tackle the problem of domain shift induced due to the vastly different training and deployment scenarios . For the animal pose estimation task , Cao et al . [ 9 ] apply discriminator based discrepancy minimization technique . Zhang et al . [ 91 ] leverage multi-modal input such as depth and body segmentation masks . Similarly , Doersch et al . [ 16 ] utilize optical-flow and 2D key-points as input representations . Further , Zhang et al . [ 90 ] propose to use 2D landmarks , obtained from a powerful off-the-shelf 2D pose detector , as the input modality . They perform instance-level geometry-aware self supervised learning on each target for inference time 2D-to-3D lifting . These approaches rely on alternative input modalities that are least affected by domain shift unlike the raw RGB images . The proposed approach does not utilize any such auxiliary inputs thus addressing one of the most challenging adaptation settings . 3 Approach . In the following , we first define the basic notations and network components required to discuss the proposed training setup . Then we provide more details about the proposed self-supervised adaptation . 3.1 Notations and pre-learning steps . We start with listing the available datasets based on which we introduce our learning setup . Pre-learning Pre-learning pose embedding 3.1.1 Datasets . We consider access to three different datasets as follows ( refer Fig . 2 ) . a ) A labeled source dataset ( xs , ys ) ∈Ds , where ( xs , ys ) denotes a tuple of a source image paired with its 3D pose annotation , b ) A set of unlabeled video sequences from the target domain , X = [ x1 , x2 , .. , xT ] ∈ X̃ where T denotes the sequence length . Here , xt∈X denotes a single RGB image frame sampled from the target domain sequence X . And , c ) a set of unpaired 3D pose sequences , Y = [ y1 , y2 , .. , yT ] ∈Ỹ . Here , a single 3D pose frame is represented as yt∈Y . 3.1.2 Network components . The image-topose inference is to be carried out via the imageto-latent CNN , G : X →Z followed by a latentto-pose network , Dp : Z → Y ( see Fig . 2C ) . We introduce two latent embeddings , i.e. , pose embedding and motion embedding , as follows . a ) The pose embedding is denoted as z∈Z is constrained to follow a uniform prior distribution , z∈ [ −1 , 1 ] 32 . This is realized by training an AAE ( adversarial auto-encoder [ 47 , 35 , 36 ] ) whose encoder and decoder mappings are represented as , Ep : Y→Z and Dp : Z→Y ( Fig . 2A ) . b ) The motion embedding is denoted as v ∈ V . In line with the preparation of pose embedding , the encoder and decoder mappings of the motion-AAE ( see Fig . 2B ) are denoted as Em : Z̃ → V and Dm : V → Z̃ . Here , the motion embedding is learned as a hierarchical temporal embedding of sequence of pose embeddings , Z = [ z1 , z2 , .. , zT ] ∈ Z̃ . Moreover , the recurrent auto-encoder operates on the sequence of pose embeddings ( Z∈Z̃ ) instead of the raw pose sequences ( Y ∈Ỹ ) . 3.1.3 Pre-learning steps . We introduce separate domain-specific image-to-latent mappings for the source and target domains i.e. , Gs : X s→Z and G : X →Z respectively . The pose and motion autoencoders are trained using the unpaired 3D pose data , Y ∈Ỹ and kept frozen in later training stages ( see Fig . 2 ) . Here , Gs is prepared by training on Ds using the following loss term : ||Dp◦Gs ( xs ) − ys|| , where ◦ denotes functional composition . Given a target image-sequence ( or video ) X , its motion embedding is obtained as v̂ = Em◦G ( X ) . Here , G ( X ) separately processes each temporal frame of X to obtain the latent pose sequence Ẑ which is passed through Em to obtain v̂ . Sec . 3.3 details the procedure to adapt G after initializing it from Gs .

Summary: This paper proposed to learn to predict the 3D human pose for real-world images via adapting the model from the source data (label constraints) to unlabeled data. The data used for training contains: (1) the labeled source domain data (e.g. human3.6m or surreal); (2) real human video sequence without pose annotations; and (3) motion capture data (3D pose without images). (2) and (3) are unpaired. The authors proposed to devise: (a) image-to-latent; (b) pose-to-latent; and (c) latent-to-pose, to learn and predict 3D poses. In particular, (a) and (c) forms the test time predictions for the real-world target domain, while (b) and (c) constitutes an auto-encoder. The core problems in here is that the latent space in (ac) and (bc) might be mismatched. The authors proposed a smart way to make sure they are aligned (see Strength below) thanks to the contrastive loss (for setting up latent space non-local grouping rules) as well as the loss from the source domain data for regularization to make sure the two latent domains align with each other.