Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation
Year: 2021

Abstract:
Available 3D human pose estimation approaches leverage different forms of strong (2D/3D pose) or weak (multi-view or depth) paired supervision. Barring synthetic or in-studio domains, acquiring such supervision for each new target environment is highly inconvenient. To this end, we cast 3D pose learning as a self-supervised adaptation problem that aims to transfer the task knowledge from a labeled source domain to a completely unpaired target. We propose to infer image-to-pose via two explicit mappings viz. image-to-latent and latent-to-pose where the latter is a pre-learned decoder obtained from a prior-enforcing generative adversarial auto-encoder. Next, we introduce relation distillation as a means to align the unpaired cross-modal samples i.e. the unpaired target videos and unpaired 3D pose sequences. To this end, we propose a new set of non-local relations in order to characterize long-range latent pose interactions unlike general contrastive relations where positive couplings are limited to a local neighborhood structure. Further, we provide an objective way to quantify non-localness in order to select the most effective relation set. We evaluate different self-adaptation settings and demonstrate state-of-the-art 3D human pose estimation performance on standard benchmarks.1

heading: 1 Introduction
text: Human pose estimation systems have garnered immense attention due to their innumerable applications [55, 19, 85]. The successes of such systems are mostly driven by large-scale datasets containing images with paired 3D pose annotations [25]. Unlike 2D joint landmarks, annotating a 3D pose requires body-worn sensors or multi-camera structure-from-motion setup [71] which is challenging to install outdoors. Hence, the available datasets are either captured in constrained laboratory settings or are limited in size and diversity. Unsurprisingly, models trained on such datasets exhibit poor cross-dataset generalization. Addressing this, several approaches [11, 77, 56, 34] resort to weaklysupervised methods that rely on images with paired 2D landmark annotations. Certain methods require additional supervision such as depth [81, 11] or multi-view image pairs [66, 33]. However, they still suffer from dataset-bias due to their strong reliance on some form of paired supervision.
In this work, we digress from any form of paired supervision or auxiliary cues (multi-view or depth) thereby avoiding the curse of dataset-bias. We thus introduce a self-supervised domain adaptation framework for 3D human pose estimation (Fig. 1). In the proposed setting, we consider access to three different datasets. First, a labeled source dataset obtained from graphics-based synthetic environments such as SURREAL [79] or in-studio datasets such as Human3.6M [25]. Second, unlabeled video sequences from the target domain. Third, a set of unpaired 3D pose sequences. Following this, image-to-pose inference is carried out via two explicit mappings i.e., image-to-latent CNN followed by a latent-to-pose network. Here, the latent-to-pose network is pre-learned as a decoder of a prior-enforcing generative adversarial auto-encoder [47] (AAE) in order to restrict the
1Webpage: https://sites.google.com/view/sa3dhp
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
pose predictions within the plausibility limits. We follow the same to pre-learn a generative motion embedding via a recurrent AAE setup which operates on the latent pose space instead of the raw pose sequences. Both pose and motion embeddings are trained solely on the unpaired 3D pose dataset. Next, we prepare image-to-latent CNN by supervising on the labeled source dataset. After finishing the above pre-learning steps, our prime objective is to train or adapt only the image-to-latent CNN so that it can perform well on the unlabeled target domain samples. In order to align the output manifold of the image-to-latent with the pre-learned latent pose manifold (in the absence of cross-modal pairs), one must formalize innovative ways to represent the dark-knowledge based on which the student (i.e. image-to-latent) output can align to that of the teacher (i.e. pose-to-latent).
Several studies [18, 20] on human cognitive development advocate that, in a self-supervised paradigm, new knowledge is acquired by relating entities based on some semantic rule. Motivated by this, we explore different ways of formalizing inter-entity relations. A relation can be characterized as lower order or higher order based on the number of entities that participate in defining a relationship tuple. For instance, in contrastive learning [57, 12], a pose space triplet relation expresses a coupling of just 3 pose entities, and is thus considered a lower order relation. However, a similar contrastive triplet defined in the hierarchical motion space (i.e. temporal pose sequences of fixed sequence length T ) would couple 3T pose entities, thus is considered a higher order relation.
Next, we adapt the target-specific image-tolatent by minimizing the relational energies derived from the contrastive relations. Unlike in prior-arts [12, 51] where the output embedding is learned from scratch, we are restricted to operate on a pre-learned output pose embedding. Consequently, the model often converges to a degenerate solution exhibiting instance-level misalignment [46]. We realize that the positive coupling in contrastive relations is limited to local neighborhood structures resulting in suboptimal alignment. This motivates us to develop a new set of relations that would express positive coupling of diverse non-local relations (beyond the structural neighborhood), thereby characterizing long-range latent pose interactions in a much effective manner.
We define tangible non-local relations separately in the pose and motion space that are categorized under a) lower order non-local relations and b) higher order non-local relations, respectively. For instance, “pose-flip” is a lower order non-local relation which associates anchor poses with their left-right flipped versions. Similarly, “flip-backward” is a higher order non-local relation which couples anchor pose-sequences with their flip-backward counterpart which is obtained via temporal reversal (backwards) of the individually flipped frames. Here, the corresponding relational energy is devised via latent space relation networks. These are essentially frozen neural networks that are trained to regress the latent embedding of the relational counterpart given latent embedding of the anchor as input. In a nutshell, the relational energies aim to preserve the equivariance of higher order spatio-temporal relations between the two modalities as a means to perform the cross-modal alignment. It turns out that, among various relations, relations coupling the most diverse non-local samples result in a better cross-modal alignment. We perform extensive experiments to validate the efficacy of our approach and demonstrate superior generalizability on samples from in-the-wild environments. We summarize our contributions as follows:
• The proposed solution for self-adapting 3D human pose model involves cross-modal alignment between the unpaired samples from the input and output modalities, via relation distillation. Highly non-local instances are associated using novel relation networks to specifically cater to the instance-level misalignment.
• We provide insights to select the most effective non-local relations. This involves quantifying non-localness of a relation as the average latent-distance between the coupled entities.
• We evaluate different self-adaptation settings and demonstrate state-of-the-art 3D human pose estimation performance against the available semi-supervised and weakly-supervised prior arts on Human3.6M [25], MPI-INF-3DHP [49], and 3DPW [80].

heading: 2 Related Works
text: Table 1 shows a comparison of the proposed setting against the prior-arts for the task of 3D human pose estimation. Our framework is equipped with self-adaptation capability while only utilizing unpaired 3D poses and synthetic 3D pose supervision.
Knowledge distillation (KD). The process involves transferring knowledge from a larger model to a smaller model to learn a concise knowledge representation with negligible information loss. Faster evaluation and reduced memory footprint renders distilled models more deployment friendly. More specifically, in KD [24], the smaller model (student) focuses on mimicking the output behaviour of a pre-trained larger model (teacher). Traditional techniques aim to match instance level behaviours such as matching the output activation [68], gram matrix [87], gradient [72], attention maps [88], etc. Recently, several techniques proposed to align pairwise distance [76, 63] or similarity graphs [58, 43] to look beyond the instance-level matching. Further, to capture higher order output dependencies, researchers adopt techniques [75] inspired from the advances in self-supervised contrastive learning literature [57, 4]. Essentially, a balanced distillation of both structural and instance-level knowledge leads to superior generalization [83]. However, in the proposed setting, the teacher, pose-to-latent and the student, image-to-latent do not share the same input modality. Further, the absence of paired correspondence restricts us from directly employing such distillation techniques.
Monocular 3D human pose estimation. There are two broad categories of models that infer 3D poses from a single image, i.e. a) those using model-based parametric representation [5, 8, 62, 7] and b) others that infer the 3D pose representation directly [69, 2, 6]. Models belonging to the first category map the input image to the latent parameters of a pre-defined parametric human model. This setting establishes the base to impose various pose priors through techniques such as adversarial training. Recent works [28, 61] have also aimed to build on the
parametric body models and extend it to express finer movements such as hand gestures and facial expressions. Those belonging to the second directly map the input image to the corresponding 3D pose. These models are further categorized into a) one-stage methods [93, 73, 59, 54, 60, 86] that directly map input image to the 3D pose, and b) two-stage methods [92, 48, 53, 73] that adopt a mapping from the image to an intermediate 2D pose, followed by lifting of 2D-to-3D. In our work, the shared latent pose can be seen as a parametric form to represent plausible 3D poses.
Domain Adaptation. A model trained on some source data may not be a suitable choice for deployment in its nascent form to an unknown target environment. In such a scenario, domain adaptation aims to tackle this domain shift induced due to the difference in source and target data distributions by adapting the model trained on some related labeled source domain to the target domain. A number of works [37] aim to tackle the problem of domain shift induced due to the vastly different training and deployment scenarios. For the animal pose estimation task, Cao et al. [9] apply discriminator based discrepancy minimization technique. Zhang et al. [91] leverage multi-modal input such as depth and body segmentation masks. Similarly, Doersch et al. [16] utilize optical-flow and 2D key-points as input representations. Further, Zhang et al. [90] propose to use 2D landmarks, obtained from a powerful off-the-shelf 2D pose detector, as the input modality. They perform instance-level geometry-aware self supervised learning on each target for inference time 2D-to-3D lifting. These approaches rely on alternative input modalities that are least affected by domain shift unlike the raw RGB images. The proposed approach does not utilize any such auxiliary inputs thus addressing one of the most challenging adaptation settings.

heading: 3 Approach
text: In the following, we first define the basic notations and network components required to discuss the proposed training setup. Then we provide more details about the proposed self-supervised adaptation.

heading: 3.1 Notations and pre-learning steps
text: We start with listing the available datasets based on which we introduce our learning setup.
Pre-learning
Pre-learning pose embedding
3.1.1 Datasets. We consider access to three different datasets as follows (refer Fig. 2). a) A labeled source dataset (xs, ys) ∈Ds, where (xs, ys) denotes a tuple of a source image paired with its 3D pose annotation, b) A set of unlabeled video sequences from the target domain, X = [x1, x2, .., xT ] ∈ X̃ where T denotes the sequence length. Here, xt∈X denotes a single RGB image frame sampled from the target domain sequence X . And, c) a set of unpaired 3D pose sequences, Y =[y1, y2, .., yT ]∈Ỹ . Here, a single 3D pose frame is represented as yt∈Y . 3.1.2 Network components. The image-topose inference is to be carried out via the imageto-latent CNN, G : X →Z followed by a latentto-pose network, Dp : Z → Y (see Fig. 2C). We introduce two latent embeddings, i.e., pose embedding and motion embedding, as follows.
a) The pose embedding is denoted as z∈Z is constrained to follow a uniform prior distribution, z∈ [−1, 1]32. This is realized by training
an AAE (adversarial auto-encoder [47, 35, 36]) whose encoder and decoder mappings are represented as, Ep : Y→Z and Dp : Z→Y (Fig. 2A). b) The motion embedding is denoted as v ∈ V . In line with the preparation of pose embedding, the encoder and decoder mappings of the motion-AAE (see Fig. 2B) are denoted as Em : Z̃ → V and Dm : V → Z̃ . Here, the motion embedding is learned as a hierarchical temporal embedding of sequence of pose embeddings, Z = [z1, z2, .., zT ] ∈ Z̃ . Moreover, the recurrent auto-encoder operates on the sequence of pose embeddings (Z∈Z̃) instead of the raw pose sequences (Y ∈Ỹ). 3.1.3 Pre-learning steps. We introduce separate domain-specific image-to-latent mappings for the source and target domains i.e., Gs : X s→Z and G : X →Z respectively. The pose and motion autoencoders are trained using the unpaired 3D pose data, Y ∈Ỹ and kept frozen in later training stages (see Fig. 2). Here, Gs is prepared by training on Ds using the following loss term: ||Dp◦Gs(xs)− ys||, where ◦ denotes functional composition. Given a target image-sequence (or video) X , its motion embedding is obtained as v̂ = Em◦G(X). Here, G(X) separately processes each temporal frame of X to obtain the latent pose sequence Ẑ which is passed through Em to obtain v̂. Sec. 3.3 details the procedure to adapt G after initializing it from Gs.

heading: 3.2 Problem formulation
text: The prime objective is to train or adapt the image-to-latent mapping, G to better generalize it to the unlabeled target domain samples (i.e. x or X).
a) Domain adaptation perspective. One can see it as an unsupervised domain adaptation (DA) problem [17, 70] which aims to adapt a source trained Gs in order to obtain a target specific mapping G. However, unlike segmentation or classification, 3D pose is a structured regression output. Consequently, the usual statistical or adversarial discrepancy minimization based [78, 44] DA solutions usually fail to improve the adaptation performance against the corresponding pre-adaptation baselines [16].
b) Knowledge distillation perspective. The problem in hand can also be perceived as a manifold alignment problem [82] where the output manifold of the image-to-latent G, needs to be aligned with the pre-learned pose manifold, Z . In knowledge distillation (KD), the same is achieved by formalizing the dark-knowledge based on which the output of a student network can align to that of a teacher network. However, in the proposed setting, the teacher, pose-to-latent Ep : Y→Z and the student, image-to-latent G : X →Z do not share the same input modality (i.e. image space X vs. pose space Y). The most common form of distillation aims to align the instance-wise embeddings [24, 68]. However, such approaches are infeasible in the absence of cross-modal pairing i.e., (x, y) ∈ X×Y in
the unlabeled target domain. Though one can obtain pseudo (x, y) pairs by forwarding the unlabeled target images through the source specific Gs (via Dp◦Gs : X →Y), the obtained pose predictions are highly unreliable due to the input domain-shift. This motivates us to look for innovative ways to represent the dark-knowledge without relying on cross-modal pairs.

heading: 3.3 Proposed solution
text: We propose to represent the dark-knowledge by forming relational associations that aim to relate a group of intra-modal entities. For example, distilling triplet associations as used in self-supervised contrastive learning [12, 51] literature can be seen as a trivial form of representing the dark-knowledge.

heading: 3.3.1 Distilling local neighborhood relations via contrastive learning
text: General contrastive learning can be applied at both latent pose space, Z and motion space, V . a) Lower-order contrastive. Pose space contrastive learning relies on relational associations of the form Cx = {(xt, x+t , {x−t })where (xt, x+t ) positive and (xt, {x−t }) negative} (Fig. 3C, bottom panel). Here, {x−t } represents a batch of unique instances that are in no way related to xt. positive indicates a positive coupling of xt and x+t . Here, x + t is obtained via a pose-invariant image space augmentation of xt i.e., x+t = A(xt), and G(xt) = G(x + t ). However, in the negative coupling, one just knows that the corresponding poses are unrelated. Thus, Cx mostly focuses on characterizing the lower-order latent local neighborhood structure. Drawing inspiration from InfoNCE [57], the corresponding relational energy loss (with τ as the temperature hyperparameter) is represented as,
LLCR = − log ∑
(xt,x + t ,{x − t })∈Cx
e(G(xt).G(x + t )/τ)/(e(G(xt).G(x + t )/τ) + ∑ {x−t } e(G(xt).G(x − t )/τ)) (1)
b) Higher-order contrastive. Similarly, motion space contrastive characterizes higher-order local neighborhood relation i.e., CX = {(X,X+, {X−})where (X,X+) positive and (X, {X−}) negative}. The corresponding relational energy is denoted as LHCR. LHCR takes the same form as of Eq 1 by replacing G(xt) with Em◦G(X), and similarly for X+ and X− (Fig. 3C, top panel). Higher-order vs lower-order relations. A natural question that arises is: why to use motion embedding when the goal task is to realize an image-to-pose mapping? We hypothesize that formalization of higher-order pose association would enhance the expressibility of the dark-knowledge which is of prime interest towards realizing superior cross-modal alignment. Here, higher-order refers to relational association of a large number of pose entities. In CX , each individual triplet relation expresses a positive coupling of 2T pose entities against just 2 in Cx. Thus, the hierarchical motion embedding facilitates a suitable platform to formalize temporally structured (temporal order must be retained) higher-order relational associations of the pose space entities (see Fig. 3C).

heading: 3.3.2 Distilling non-local relations via equivariance consistency
text: We observe that adapting the image-to-latent mapper G just by minimizing LLCR and LHCR results in poor pose estimation performance. In general self-supervised contrastive learning prior-arts, the latent embedding is learned from scratch alongside the training of image-to-latent mapping. However, in the proposed setting, we aim to adapt the image-to-latent network G in order to align its output with the pre-learned pose and motion embeddings (i.e. the latent embeddings of frozen {Ep, Dp} and
{Em, Dm}). Consequently, we need to address a special degenerate scenario where the distillation losses completely converge even when the model exhibits severe instance-level misalignment [46].
Local vs non-local relations. We hypothesize that relations having non-local positive coupling, equivalent to relations in a knowledge-graph (KG), are the best way to express the instance-grounded dark-knowledge. Note that, the positive coupling in contrastive-based relations are limited to local neighborhood, as here, the positive counterpart of an anchor is obtained via simple pose-invariant augmentations (i.e. x+t = A(xt) in both Cx and CX ). To this end, our goal is to come up with a new set of relational associations with non-local positive couplings. This aims to characterize long-range latent pose interactions, i.e. a positive coupling of two entities that are grounded far away in the latent pose or motion space (see Fig. 4C). To this end, we introduce relation networks.
Relation networks. Relation networks operating on the pose and motion embeddings are neural network mappings of the form, Tzrz : Z→Z and T v rv : V→V respectively. Here, rz and rv denote independent embedding specific rule-ids for the pose and motion embeddings respectively. The equivalent rule-specific image space relation transformation operating on the image and imagesequence are represented as Txrz : X →X , and T X rv : X̃ → X̃ . And the same on the raw pose and motion space is represented as Tyrz : Y →Y , and T Y rv : Ỹ → Ỹ . Note that, except T z rz and T v rv , all other transformations are simple affine-like operations without involving neural network mappings.
a) Distilling lower-order non-local relations. For each target image xt ∈X , we first construct a dataset of positive coupling represented as N xrz = {(xt, x + t ) where x + t = T x rz (xt)}. Here, one is already aware of the corresponding pose relation, i.e. y+t = T y rz (yt). And, the same relation in the latent pose space is expressed as z+t = T z rz (zt).
Let us consider a non-local relation termed as "pose-flip" with rule-id rz=1. Here,N x1 is obtained by pairing each target image with its flipped version obtained via a simple image-flip transformation operation, (i.e. Tx1 ). Note that, a similar pose space transformation would involve swapping the left side joints with that of the right (i.e. Ty1). And, the corresponding relation network T z 1 is a multi-layer neural network which is trained to regress z+t =Ep ◦ T y 1(yt) while feeding zt=Ep(yt) as the input. Here, yt is sampled from the unpaired 3D pose data Y . Next, we formalize the corresponding relational energy loss that uses the frozen relation network Tz1 and is represented as,
Lz1 = ∑
(xt,x + t )∈Nx1
‖Tz1 ◦ G(xt)− G(x+t )‖ (2)
b) Distilling higher-order non-local relations. One can define higher order non-local relations on the hierarchical motion space to devise a similar relational energy loss.
Let us consider a non-local relation termed as "flip-backward" with rule-id rv = 1. Here, NX1 = {(X,X+) where X+ = TX1 (X)} is obtained by pairing each target sequence X with its flip-backward counterpart which is obtained via temporal reversal (backwards) of the imageflipped frames. Accordingly, the corresponding relation network Tv1 is a multi-layer neural network trained to regress v+=Em◦Ep◦TY1(Y ) while feeding v=Em◦Ep(Y ) as the input, where Y is sampled from the unpaired 3D pose sequences Ỹ . And, the corresponding relational energy loss is represented as,
Lv1 = ∑
(X,X+)∈NX1 ‖Tv1 ◦ G(X)− G(X+)‖ (3)
Finally, cross-modal alignment is achieved by simultaneously minimizing all the relational energies, i.e. L = LCR + LNLR; where LCR = LLCR + LHCR and LNLR = ∑ rz Lzrz + ∑ rv Lvrv (4)
Next, we discuss certain insights which can help us to formalize effective non-local relations.

heading: 3.3.3 What makes non-local relations more effective?
text: In the latent pose space, we conceptualize two different relation networks viz. a) pose-flip, Tz1 and b) in-plane-rotation, Tz2. The corresponding relational energies are denoted as Lz1 and Lz2 respectively. Though minimizing Lz1 yields a substantial improvement over Ours-LL (just minimizing LCR without LNLR), jointly minimizing both Lz1 and Lz2 does not yield much improvement on the final pose estimation performance. However, the combination of flip and rotation in a single relation yields a considerable improvement in performance. We call this rule "flip+inplane-θ", with rule-id rz=3. We also notice improvement in performance when using in-plane rotation used in conjunction with the higher order relation “flip-backward“ (rotation of individual flipped frames). This rule
is termed "flip+inplane-θ-backward", with rule-id rv = 2. We also introduce a higher-order non-local relation "slow-backward", with rule-id rv=3. The relational pairs for slow-backward are constructed by sampling the original sequence at 15 FPS, to enable a temporally slow 30 FPS slow backward video taken from the middle region (see Fig. 4C).
A. Pre-learning steps (training on source data) 1) Train Ep and Dp on unpaired 3D poses, y ∈ Y 2) Train Em and Dm on unpaired motion, Y ∈ Ỹ 3) Train Gs on labeled source dataset, (xs, ys) ∈ Ds 4) Train Tz3 for "flip+inplane-θ" on Z using y ∈ Y 5) Train Tv2 for "flip+inplane-θ-backward" on V
using Y ∈ Ỹ 6) Train Tv3 for "slow-backward" on V using Y ∈ Ỹ
B. Unsupervised target adaptation 7) After initializing G from Gs, train G by minimizing the
local relational energy, LCR alongside minimizing the non-local relations Lz3, Lv2 and Lv3 .
Algorithm 1: Overview of the optimization steps. Note that, Lz3, Lv2 and Lv3 rely on frozen Tz3, Tv2 and Tv3 to distill the embedded relational darkknowledge.
Quantifying non-localness via latentdistance. We hypothesize that, “relations coupling diverse samples (long-range interactions) lead to better cross-modal alignment”. To evaluate this hypothesis, we define a distance metric to apprehend the extent of non-localness of individual relations which is termed as latentdistance. Here, the latent-distance dz1 = mean(zt,z+t )∈N z1 ‖zt−z + t ‖measures the average distance between relation pairs (zt = Ep(yt), z + t = Ep ◦T y 1(yt)), for relation-id rz = 1, where y(yt) is sampled from the unpaired 3D pose data Y . In Fig. 4A, the bar-plot comparing the latent-distance for various relations clearly highlight the gap in the degree of non-localness thus justifying the aforementioned observation regarding the performance gap. We perform a similar metric analysis to select the best set of relations defined on the motion space V (see Fig. 4B and 4C). Finalizing the non-local relations. At the end, we take up a single pose relation, i.e., flip+inplane-θ and two diverse motion relations, i.e., flip+inplane-θ-backward and slow-backward to realize our final cross-modal alignment framework (Fig. 4). With Lz3 being the relational energy specific to the pose space relation flip+inplane-θ, we denote Lv2 and Lv3 as the relational energies specific to the motion relations flip+inplane-θ-backward and slow-backward respectively. Please refer to Supplementary for an ablation on the effect of changing the angle θ.
Overview of the optimization steps. Algorithm 1 summarizes the training as a step-by-step learning process. Here, each step uses frozen models obtained from the previous step. After initializing G from Gs only a minimal set of network parameters (Res-3 block of ResNet-50) are updated to learn target-specific mapping. This can be see as a regularization for the unsupervised adaptation training. We freeze Dp while training G as it is crucial to regularize the unsupervised adaptation to avoid degenerate solutions (or mode-collapse). Updating Dp while training G requires us to impose an additional adversarial discriminator loss (as used in HMR [29]) in order to uphold its ability to decode plausible pose predictions. Note that, updating Dp would update the manifold structure of
the embedding space. Further, as the relation networks (Tz3, T v 2, T v 3) operate on the frozen latent embeddings, the pre-learned relations networks (used to define the relation distillation objective) would no longer be useful and are required to be updated alongside. Such unconstrained optimization greatly destabilizes the self-adaptive process (in the absence of cross-modal pairs) and degrades the performance significantly. Note that the proposed self-adaptation step does not involve any complex adversarial loss component, which greatly simplifies the training procedure.

heading: 4 Experiments
text: We demonstrate effectiveness of the proposed framework by evaluating it on a variety of cross-dataset adaptation settings.
Implementation details. The backbone of Gs constitutes of an ImageNet initialized ResNet-50 [23] (till Res-4F) followed by a series of fully-connected (FC) layers to obtain the latent pose representation, z ∈ R32. The pose auto-encoder, {Ep, Dp} are FC networks operating on the 3D pose y (17 joints). The motion auto-encoder, {Em, Dm} is composed of bidirectional LSTMs [22] with 128 hidden units operating on a fixed sequence length of 30 (30 FPS) where the intermediate motion-embedding, v ∈ R128. The relation networks constitute of simple FC layers. We associate separate Adam optimizer [32] to each relational energy term which are optimized in alternate training iterations. The adaptation is performed on an Nvidia V100 GPU with each batch containing 8 videos each of frame-length 30 (see Suppl. for more details).
Datasets. We use the CMU-MoCap [1] dataset as the sample set for unpaired 3D poses Y and unpaired pose sequences Ỹ . We use the synthetic SURREAL (S) dataset [79] as one of the source datasets. The sample set for the unpaired videos X̃ constitutes of single-person action videos (dance forms, sports, etc.) collected from the Sports-1M dataset [31]. The raw video frames are forwarded through a person-detector [64] to obtain the person-focused image sequences. We name it as webdataset (W). For a fair evaluation, we use the standard, in-studio Human3.6M (H) dataset [25] as both source or target domain, in different problem settings. MPI-INF-3DHP (M) [49] is used as an unlabeled target domain to evaluate cross-studio adaptation. Further, 3DPW [80], and LSP [27] datasets are used to evaluate our cross-dataset generalizability, without involving these during training.

heading: 4.1 Adaptation settings
text: We introduce the following adaptation settings with various source and target dataset selections.
a) Ours(S→H) We use labeled synthetic SURREAL (S) as the source while unlabeled Human3.6M (H) acts as the target without involving the web-dataset (W). The resultant model is tuned to work well only for the specific in-studio environment, thus may fail to generalize for in-the-wild data.
b) Ours(H→M) Here, labeled Human3.6M is used as the only source domain while MPI-INF-3DHP (M) samples are used as the unlabeled target. This evaluates our cross-studio adaptation performance.
Table 4: Quantitative results comparing our approach against prior-arts for 3D pose estimation on 3DPW. Full (3D) supervision denotes using ground-truth 3D supervision on 3DPW for training. Direct Transfer denotes training on a variety of labeled source dataset and directly evaluating the resultant model on unseen 3DPW test set. We achieve state-of-the-art result upon using both Human3.6M (H) and SURREAL (S) as the source and the web-data as the target. + denotes number taken from [30]. * denotes using parametric mesh models.
Training Methods PA-MPJPE ↓ Full (3D)
Supervision Arnab et al. [3]* 77.2 Sun et al. [74]* 69.5
Direct Transfer
Martinez et al. [48]+ 157.0 Dabral et al. [15]+ 92.3 Kanazawa et al. [30]* 80.1 Doersch et al. [16]* 82.4 Kanazawa et al. [29]*+ 76.7 Ours(S→W) 79.3 Ours(SH→W) 72.1
c) Ours(S→W) Here, labeled SURREAL (S) is used as the only source domain while the unlabeled target videos are extracted from the web-dataset (W). The resultant model is evaluated for cross-dataset generalization on 3DHP and 3DPW (see Table 3 and Table 4). Note that, web-dataset is the most challenging in-thewild dataset with substantial diversity in pose, apparel, and backgrounds.
d) Ours(SH→W) Here, the source domain is a combination of labeled SURREAL (S) and Human3.6M (H) datasets while the unlabeled web-dataset (W) acts as the target. This setting enjoys the advantage of strong source supervision on natural but in-studio Human3.6M alongside SURREAL.

heading: 4.2 Evaluation on benchmark datasets
text: In this section, we evaluate the proposed approach on the standard benchmark datasets. MPJPE and PA-MPJPE [25] denote standard mean per joint position error metric computed before and after Procrustes Alignment [21]. Following prior arts [49], we report PCK and AUC, i.e. percentage of correct keypoints and area under the curve respectively for the 3DHP dataset.
a) Evaluation on Human3.6M. Table 2 lists a comparison of Ours(S→H) against the prior arts. Under unsupervised training setup, Ours(S→H) outperforms the prior state-of-the-art (SOTA) by a significant margin. We attribute this improvement to our attempt to utilize domain randomization via SURREAL and the proposed cross-dataset alignment procedure. Under semi-supervised training, the target adaptation is supervised on a small subset of labeled samples (i.e. subject S1 in Human3.6M) alongside unsupervised alignment on the rest. The resultant model, Ours(S→H, Semi) also outperforms the prior semi-supervised works even in the absence of additional multi-view supervision as used in some of the prior-arts.
b) Evaluation on 3DHP. Table 3 shows a detailed quantitative comparison of three of our adaptation variants; Ours(H→M), Ours(S→W), and Ours(SH→W), under two training setups. Ours(SH→W) achieves state-of-the-art unseen transfer performance validating superior generalizability of the proposed framework. Ours(SH→W) uses in-studio but real Human3.6M as an additional labeled source data alongside SURREAL, thus reducing the domain gap when compared against Ours(S→W). c) Evaluation on 3DPW. Table 4 reports a detailed comparison of our variants against the prior-arts on the in-the-wild 3DPW dataset. All the methods under direct transfer do not use 3DPW samples even for any supervised or unsupervised training. A lower PA-
MPJPE for Ours(SH→W) clearly highlight our superior cross-dataset generalizability against the prior approaches which also utilize additional information such as the SMPL mesh model [45].
d) Ablation study on Human3.6M. Table 5 shows an ablative analysis highlighting the effectiveness of individual relational energies towards realizing a better unsupervised alignment. As compared
Click here for Additional plots
to the source-only baseline (first row of Table 5), distillation of contrastive relations in LCR yields the first stage of improvement. Minimizing the non-local pose space relational energy Lz3 results in further improvement by effectively distilling an instance-grounded dark-knowledge. Additionally, distilling higher order non-local motion relations (i.e. Lv2 and Lv3) yields a substantial improvement by effectively attenuating the instance-level misalignment.
Qualitative evaluation and limitations. Fig. 5 illustrates the generaliability of Ours(SH→W) proposed approach under diverse pose, apparel, and background variations. Results with red base show some of the failure cases. The model is restricted from predicting implausible 3D pose outcomes as these are inferred through the prior-enforcing generative pose decoder. However, the model fails under certain drastic scenarios such as high background clutter, multi-level body-part occlusion, and rare athletic poses. Other limitations include body-truncation scenarios i.e., scenarios where certain body-parts are either occluded by external objects or are outside the image frame. In such cases there are multiple plausible 3D pose outcomes, thus asking for a future exploration of probabilistic pose estimation modeling. Taking a different direction, one can explore innovative ways to utilize inputs from auxiliary modalities (such as depth, foreground segmentation, etc.) as and when available from the deployed target environment (see Suppl. for more details).
Negative societal impacts. While we do not foresee our framework causing any direct negative societal impact, it may be leveraged to create malicious applications utilizing human tracking and action recognition. Estimating a human pose does not require any identity information. However, methods such as gait recognition can be indirectly used to identify personal attributes thereby raising privacy concerns. We urge the readers to make use of the work detailed responsibly, limiting the usage to legal use-cases.

heading: 5 Conclusion
text: We presented a cross-modal alignment technique to align the learned representations from two diverse modalities. Our unsupervised technique allows adaptation to the wildest unlabeled samples gathered from web while initializing the base model on substantially diverse but unrealistic SURREAL. We analyzed the importance of higher order, non-local relations in expressing the rich instance-grounded dark knowledge as required to attenuate the instance-level misalignment. In future, we plan to extend our framework for multi-modal alignment in presence of additional input modalities.
Acknowledgments and Disclosure of Funding
This work was supported by a Google Ph.D. Fellowship (Jogendra) and a project grant from MeitY (No.4(16)/2019-ITEA), Govt. of India.

Review summaries:
This work regards monocular 3D human pose estimation as a domain adaptation problem to solve the problem of insufficient outdoor training data. The framework is composed of an image-to-latent encoder, which is pre-trained on the source domain and adapted to the target domain using unlabeled videos, and a latent-to-pose decoder, which is trained on unpaired 3D poses. During the encoder adaptation, non-local relations are utilized to improve the performance, which is the main contribution. 

## Summary  Paper proposes a self-adaptation framework to align the unpaired samples of 3D pose and video frames to solve 3D human pose estimation from in-the-wild images. This cross-modal alignment is realized through a relation distillation scheme. The proposed approach achieves compelling results on 3D human pose datasets among self-supervised and unsupervised methods.

This paper proposed to learn to predict the 3D human pose for real-world images via adapting the model from the source data (label constraints) to unlabeled data. The data used for training contains: (1) the labeled source domain data (e.g. human3.6m or surreal); (2) real human video sequence without pose annotations; and (3) motion capture data (3D pose without images). (2) and (3) are unpaired. The authors proposed to devise: (a) image-to-latent; (b) pose-to-latent; and (c) latent-to-pose, to learn and predict 3D poses. In particular, (a) and (c) forms the test time predictions for the real-world target domain, while (b) and (c) constitutes an auto-encoder. The core problems in here is that the latent space in (ac) and (bc) might be mismatched. The authors proposed a smart way to make sure they are aligned (see Strength below) thanks to the contrastive loss (for setting up latent space non-local grouping rules) as well as the loss from the source domain data for regularization to make sure the two latent domains align with each other. 

The paper focuses on 3D human pose estimation in a scenario of domain adaptation. To avoid dataset-bias, the authors address the problem in a more restrict setting: any form of paired supervision or auxiliary cues should not be used.   The authors first use AAE modules to learn pose and motion latent spaces. Secondly, the authors train a pose encoder which tries to map the input image to the learned pose latent space. Finally, given the target-domain videos, the authors train the pose encoder together with the decoder of the pose AAE. The model is optimized by minimizing relational energies.