Non-local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation
Authors: Jogendra Nath Kundu
Siddharth Seth
Anirudh Gururaj Jamkhandi
Pradyumna YM
Varun Jampani
Anirban Chakraborty
Venkatesh Babu Radhakrishnan
Venue: NeurIPS 2021 Poster (2021)
Keywords: 3D human pose estimation
knowledge distillation
domain adaptation

Abstract: Available 3D human pose estimation approaches leverage different forms of strong (2D/3D pose) or weak (multi-view or depth) paired supervision. Barring synthetic or in-studio domains, acquiring such supervision for each new target environment is highly inconvenient. To this end, we cast 3D pose learning as a self-supervised adaptation problem that aims to transfer the task knowledge from a labeled source domain to a completely unpaired target. We propose to infer image-to-pose via two explicit mappings viz. image-to-latent and latent-to-pose where the latter is a pre-learned decoder obtained from a prior-enforcing generative adversarial auto-encoder. Next, we introduce relation distillation as a means to align the unpaired cross-modal samples i.e., the unpaired target videos and unpaired 3D pose sequences. To this end, we propose a new set of non-local relations in order to characterize long-range latent pose interactions, unlike general contrastive relations where positive couplings are limited to a local neighborhood structure. Further, we provide an objective way to quantify non-localness in order to select the most effective relation set. We evaluate different self-adaptation settings and demonstrate state-of-the-art 3D human pose estimation performance on standard benchmarks.

Non-Local Latent Relation Distillation for Self-Adaptive 3D Human Pose Estimation
Jogendra Nath Kundu'
Siddharth Seth' Anirban Chakraborty
Anirudh Jamkhandi'
Pradyumna YM
Varun Jampani
R. Venkatesh Babu
Indian Institute of Science, Bangalore
Google Research
Abstract
Available 3D human pose estimation approaches leverage different forms of strong (2D/3D pose) or weak (multi-view or depth) paired supervision. Barring synthetic in-studio domains, acquiring such supervision for each new target environment is highly inconvenient. To this end, we cast 3D pose learning as a self-supervised adaptation problem that aims transfer the task knowledge from a labeled source domain to a completely unpaired target. We propose to infer image-to-pose via two explicit mappings viz. image-to-latens and latent-to-pP toese- where the latter is a pre-learned decoder obtained from prior-enforcing generative adversarial auto-encoder. Next, we introduce relation distillation a means align the unpaired cross-modal samples i.e. the unpaired target videos and unpaired 3D pose sequences. To this end, we propose a new set of non-local relations in order characterize long-range latent pose interactions unlike general contrastive relations where positive couplings are limited to a local neighborhood structure. Further, we provide objective way to quantify non-localness in order to select the most effective relation set. We evaluate different self-adaptation settings and demonstrate state-of-the-ard 3D human pose estimation performance standard benchmarks.
Introduction
Human pose estimation systems have garnered immense attention due to their innumerable applica- tions [55, 19, 85]. The successes of such systems are mostly driven by large-scale datasets containing images with paired 3D pose annotations [25]. Unlike 2D joint landmarks, annotating 31) pose requires body-worn sensors or multi-camera structure-frome motmmootioon srrctmrr-frommmmotiin setup [71] which is challenging install outdoors. Hence, the available datasets are either captured in constrained laboratory settings or are limited in size and diversity. Unsurprisingly, models trained on such datasets exhibit poor cross-dataset generalization. Addressing this, several approaches [11, 77, 56, 34] resort weakly- supervised methods that rely on images with paired 2D landmark annotations. Certain methods require additional supervision such as depth [81, 11] or multi-view image pairs [66, 33]. However, they still suffer from dataset-bias due to their strong reliance some form of paired supervision.
In this work, digress from any form of paired supervision auxiliary thereby avoiding the curse of dataset-bias. We thus introduce a self-supervised domain adaptation framework for 3D human pose estimation (Fig. 1). the proposed setting, we consider access to three different datasets. First, a labeled source dataset obtained from graphics-based synthetic environments such as SURREAL [79] in-studio datasets such as Human3. 3.6 [25]. Second, unlabeled video sequences from the target domain. Third, a set of unpaired 3D pose sequences.
(multi-view depth)
Following this, image-to-pose inference is carried out two explicit mappings i.e., image-to-latee CNN followed by a latent-to-pon po-p network. Here, the latent-to to-poe network is pre-learned as decoder of prior-enforcing generative adversarial auto-encoder [47] (AAE) in order restrict the
Webpage: https://siteet www.::ittte.gggge.eeeeeeoo google. com/vien/sa3dhp
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

pose predictions within the plausibility limits. We follow the same to pre-learn generative motion embedding via recurrent AAE setup which operates on the latent pose space instead of the raw pose sequences. Both pose and motion embeddings are trained solely on the unpaired 3D pose dataset. Next, we prepare image-to-latent CNN by supervising on the labeled source dataset. After finishing the above pre-learning steps, our prime objective is to train or adapt only the image-to-latent CNN so that it can perform well on the unlabeled target domain samples. In order to align the output manifold of the image-to to-ge-t--latent with the pre-learned latent pose manifold (in the absence of cross-modal pairs), one must formalize innovative ways to represent the dark-knowledge based on which the student (i.e. image-to-latee -o-tent)) output can align to that of the teacher (i.e. pose-to to-latent..
Several studies [18, 20] on human cognitive development advocate that, in a self supervised paradigm. new knowledge is acquired by relating entities based on some semantic rule. Motivated by this, we explore different ways of formalizing inter-entity relations. A relation can characterized as lower order or higher order based on the number of entities that participate in defining relationship tuple. For instance, in contrastive learning [57, 12], a pose space triplet relation expresses a coupling of just 3 pose entities, and is thus considered a lower order relation. However, similar contrastive triplet defined in the hierarchical motion space (i.e. temporal pose sequences of fixed sequence length T)) would couple 37 pose entities, thus is considered higher order relation.
Next, we adapt the target-specific image-to- latent by minimizing the relational energies de- rived from the contrastive relations. Unlike in prior-arts [12, 51] where the output embedding learned from scratch, we are restricted to op- erate on pre-learned output pose embedding. Consequently, the model often converges to a de- generate solution exhibiting instance-level mis- alignment [46]. We realize that the positive coupling in contrastive relations is limited to local neighborhood structures resulting in sub- optimal alignment. This motivates us to develop a new set of relations that would express positive coupling of diverse non-local relations (beyond the structural neighborhood), thereby character- izing long-range latent pose interactions in much effective manner.
A. Unpaired motion
Ap-bachward
GUF
RAAARASA
5 PAAT PAATRRG
Higheronne Retason
post 1o latem
Disiilation higher order rebilions
Imaoc lsten
is absence of cross modal pait
Iswer 50000
Paant:
Unpaired videos
We align samples from unpaired pose (or motion) and unpaired images (or videos) at a shared latent pose space by distilling higher order (associ- ating multiple instance via motion) non-local (e.g. (ap-backward)) relations. Relations are equivalent to form of data-interlinking as done in anowledge-graphs.
Figure 1:
We define tangible non-local relations separately in the pose and motion space that are categorized under a) lower order non-local relations and b) higher order non-l relations, respectively. For instance, "pose-flip" is lower order non-local relation which associates anchor poses with their left-right flipped versions. Similarly, "flip-backward" baakwarrdd"" is a higher order non-local relation which couples anchor pose-sequences with their flip- backwadd counterpart which is obtained via temporal reversal (backwards) of the individually flipped frames. Here, the corresponding relational energy is devised via latent space relation networks. These essentially frozen neural networks that trained to regress the latent embedding of the relational counterpart given latent embedding of the anchor as input. In a nutshell, the relational energies aim preserve the equivariance of higher order spatio-temporal relations between the two modalities as a means to perform the cross-modal alignment. It turns out that, among various relations, relations coupling the most diverse non-local samples result in a better cross-modal alignment. We perform extensive experiments to validate the efficacy of our approach and demonstrate superior generalizability on samples from in-the-wild environments. We summarize our contributions as follows:
The proposed solution for self-adapting 3D human pose model involves cross-modal align- ment between the unpaired samples from the input and output modalities, via relation distillation. Highly non-local instances are associated using novel relation networks to specifically cater to the instance-level misalignment.
We provide insights to select the most effective non-local relations. This involves quantifying non-localness of a relation as the average latent-distance between the coupled entities.
We evaluate different self-adaptation settings and demonstrate state-of-the-ard 3D human pose estimation performance against the available semi-supervised and weakly-supervised prior arts on Human3.6, [25], MPI-INF-3DHP [49], and 3DPW [80].

---

2
Related Works
Table shows a comparison of the proposed setting against the prior-arts for the task of 3D human pose estimation. Our framework is equipped with self-adaptation capability while only utilizing unpaired 3D poses and synthetic 3D pose supervision.
Knowledge distillation (KD). The process involves transferring knowledge from larger model to smaller model to learn a concise knowledge representation with negligible information loss. Faster evaluation and reduced memory footprint renders distilled models more deployment friendly. More specifically, in KD [24], the smaller model (student) focuses on mimicking the output behaviour of a pre-trained larger model (teacher). Traditional techniques aim match instance level behaviours such as matching the output activation [68], gram matrix [87], gradient [72], attention maps [88], etc. Recently, several techniques proposed to align pairwise distance [76, 63] or similarity graphs [58, 43] to look beyond the instance-level matching. Further, to capture higher order output dependencies, researchers adopt techniques [75] inspired from the advances in self-supervised contrastive learning literature [57, 4J. Essentially, a balanced distillation of both structural and instance-level knowledge leads to superior generalization [83]. However, in the proposed setting, the teacher, pose-to-latent and the student, soage-to-latery do not share the same input modality. Further, the absence of paired correspondence restricts us from directly employing such distillation techniques.
Monocular 3D human pose estimation. There two broad categories of models that infer 3D poses from a single image, iee. a) those using model-based parametric representation [5, 8, 62, 7] and b) others that infer the 3D pose representation directly [69, 2, 6]. Models be- longing to the first category map the input image to the latent parameters of pre-defined para- metric human model. This setting establishes the base to impose various pose priors through techniques such as adversarial training. Recent works [28, 61] have also aimed build on the parametric body models and extend it to express finer movements such as hand gestures and facial expressions. Those belonging to the second directly map the input image to the corresponding 3D pose. These models are further categorized into a) one-stage methods [93, 73, 59, 54, 60, 86] that directly map input image to the 3D pose, and b) two-stage methods [92, 48, 53, 73] that adopt a mapping from the image to an intermediate 2D pose, followed by lifting of 2D-to-3D. In our work, the shared latent pose can be seen as a parametric form to represent plausible 3D poses.
Table I:
Characteristic table comparing positive and negative attributes of Ours against prior works.
Real Paired Sup [ Lnpaired
Sypparee/Syllhetiic
Self-
Methods
3D3 2D Muln
pose
3D pose Sup.
aduptation capability
pose pose
view
Sup.
Rhodin et al. [6S]
X
V
X
X
X
Chen et al. [13]
/
x
X
Wandt et al. [81]
/
x
/
X
Chen er al. [1]]
X
/
X
Doersch af. [16]
X
X
lybal et at. [26]
X
V
J
X
X
X
Kundu et [38]
X
/
Zhang ef al. [90]
/
X
X
Ours
X
X
Domain Adaptation. A model trained on some source data may not be suitable choice for deployment in its nascent form to an unknown target enviromment. adaptation aims to tackle this domain shift induced due to the difference in source and target data distributions by adapting the model trained on some related labeled source domain to the target domain. A number of works [37] aim to tackle the problem of domain shift induced due to the vastly different training and deployment scenarios. For the animal pose estimation task, Cao et al. [9] apply discriminator based discrepancy minimization technique. Zhang et al. [91] leverage multi-modal input such as depth and body segmentation masks. Similarly, Doersch et al. [16] utilize optical-flow and 2D key-points input representations. Further, Zhang et al. [90] propose to use 2D landmarks, obtained from a powerful off-the-shelf 2D pose detector, as the input modality. They perform instance-level gcometry-awand self supervised learning on each target for inference time 2D-to-3D lifting. These approaches rely on alternative input modalities that are least affected by domain shift unlike the raw RGB images. The proposed approach does not utilize any such auxiliary inputs thus addressing one of the most challenging adaptation settings.
such a scenario, domain
3 Approach
In the following, we first define the basic notations and network components required to discuss the proposed training setup. Then we provide more details about the proposed self-supervised adaptation.
3.1 Notations and pre-learning steps
We start with listing the available datasets based on which we introduce our learning setup.
3

---

3.1.1 Datasets. We consider access to three different datasets follows (refer Fig. 2). a) A labeled source dataset (s*, yf) ) E D', where (x*, """""") denotes a tuple of source image paired with its 3D pose annotation, b) A set of unla- beled video sequences from the target domain,
yEy
Yey
Unpaired 3D
Unpaired
("',)"))
3D pose
Motion
A
B
Labeled Source
Training AAE Pose embedding
Training AAE motion embedding
data
X = [Ti. T2, 2T] E X where T denotes the sequence length. Here, 2y X denotes single RGB image frame sampled from the target do- main sequence X. And, c) a set of unpaired 3D pose sequences, Y [y/1, y2 ... yrl y. Here, single 3D pose frame is represented as yr E V
C
Ep
Dp
Em
Dm
Supervised training
G
Dp
2EZ
v V
image-to -latent
latent-to- pose
D
frozern ".
Inttiaizo
Minimize
G
Dp
relational energy
3.1.2 Network components. The image-to- pose inference is to be carried out via the image- to-latent CNN, G: X' + followed by a latent- to-pose network, Dp Z + y (see Fig. 2C). We introduce two latent embeddings, i.c., pose embedding and motion embedding, as follows.
X Unpairred
target videos
Unsup. adaptation
Proposed training setup. A and B. Train- ing pose and motion AAEs on the unpaired pose and motion data. C. Initializing G by supervising on labeled source data. D. Adapting G on unlabeled target videos.
Figure 2:
a) The pose embedding is denoted as E 2 is constrained to follow a uniform prior distribu- tion, z 1, 1]32, This is realized by training an AAE (adversarial auto-encoder [47, 35, 36)) whose encoder and decoder mappings are represented as, E.p - Z and D, 2 y (Fig. 2A).
b) The motion embedding is denoted as VEV. In line with the preparation of pose embedding. the encoder and decoder mappings of the motion-AAE (see Fig. 2B) are denoted as E, 2 -
and D. V + Z. Here, the motion embedding is learned as a hierarchical temporal embedding of sequence of pose embeddings. Z [21, #2, ... 27] E Z. Moreover, the recurrent auto-encoder operates on the sequence of pose embeddings (Z E Z) instead of the raw pose sequences (Y e))
3.1.3 Pre-learning steps. We introduce separate domain-specific image-to-latent mappings for the source and target domains ie., G : + Z and G : X - 2 respectively. The pose and motion auto- encoders trained using the unpaired 3D pose data, Y and kept frozen in later training stages (see Fig. 2). Here, G is prepared by training on D using the following loss term: ||D,o0*(z* where o denotes functional composition. Given target image-sequence (or video) X, its motion embedding is obtained as v = E, G Here, G(X) separately processes each temporal frame of X to obtain the latent pose sequence Ž which is passed through E, to obtain é. Sec. 3.3 details the procedure adapt G after initializing it from G
y"
3.2 Problem formulation
The prime objective is to train or adapt the image-to-latens mapping, G to better generalize it to the unlabeled target domain samples (i.e. X).
a) Domain adaptation perspective. One can see it as an unsupervised domain adaptation (DA) problem [17, 70] which aims to adapt source trained G in order to obtain target specific map- ping G. However, unlike segmentation or classification, 3D pose is a structured regression output. Consequently, the usual statistical or adversarial discrepancy minimization based [78, 44] DA solu- tions usually fail to improve the adaptation performance against the corresponding pre-adaptation baselines [16].
b) Knowledge distillation perspective. The problem in hand can also be perceived as a manifold aligmment problem [82] where the output manifold of the image-to-latent G, needs to be aligned with the pre-ledrned pose manifold, Z. In knowledge distillation (KD), the same achieved by formalizing the dark-knowledge based on which the output of a student network can align to that of a teacher network. However, in the proposed setting, the teacher, pose-to-latent Ep ) - and student, image-to-latent G X 2 do not share the same input modality (i.e. image space X' vs. pose space )²). The most common form of distillation aims to align the instance wise embeddings [24, 68). However, such approaches are infeasible in the absence of cross modal pairing ie., (z. y) E X X'x in

---

A. Unpaired motion
rive coupling coupling
K
Latent
Pose-to -latent
B
motion
Non-local Reiational energy
Non-local energy
Image-to latent
Latent Pose space
:2-22
C. General contrastive learning: *ve coupling of local entities
D. Distilling non-local association:
B. Unpaired videos
ve coupling of non-local entities
Figure 3: We align samples from unpaired pose (or motion) and unpaired images (or videos) at a shared latent pose space by distilling higher order (associating multiple instance via motion) non- local (e.g. baagg-fll--cckiie flip-backward) relations. Relations are equivalent to a form nata-interlinking as done in knowledge-graphs.
the unlabeled target domain. Though one can obtain pseudo (vc, v) pairs by forwarding the unlabeled target images through the source specific G (via D : +)),,,, the obtained pose predictions are highly unreliable due to the input domain-shift. This motivates to look for innovative ways to represent the karkllnowledge without relying on cross-modal pairs.
3.3 Proposed solution
We propose to represent the dark-knowledge by forming relational associations that aim to relate a group of intra-modal entities. For example, distilling triplet associations as used in self-supervised contrastive learning [12, 51] literature can be seen as trivial form of representing the dark-knowledge.
3.3.1 Distilling local neighborhood relations via contrastive learning
General contrastive learning can be applied at both latent pose space, Z and motion space, V.
Pose space contrastive learning relies on relational associations of the form C* {(x,, att {., where (x+, 2x positive and (Ire {., negative) (Fig. 3C, bottom panel). Here, {., represents a batch of unique instances that are in no way related Tt- positive indicates positive coupling of x; and _ Here, _ is obtained via pose-invariant image space augmentation of 24 ie., _ A(xz), and G(x²) G(x:). However, in the negative coupling, one just knows that the corresponding poses are unrelated. Thus, C* mostly focuses on characterizing the lower-order latent local neighborhood structure. Drawing inspiration from InfoNCE [57], the corresponding relational energy loss (with 7 as the temperature hyperparameter) is represented as,
a) Lower-order contrastive.
CLCR = log E(zeat Aa, Dec= c(((a)).a(x)) )/+) ((((((
E(x;) (((())))(())) )/t) (1)
b) Higher-order contrastive. Similarly, motion space contrastive characterizes higher-order local neighborhood relation i.e., C {(X, ([X,X* X ( (X-]) where (X, x+) + positive and (X, (X {X- }} negative). The corresponding relational energy is denoted as LHER- LHCR takes the same form as of Eq by replacing G(r:)) with E, oG(X), and similarly for X and X (Fig. 3C, top panel).
Higher-order vs lower-order relations. A natural question that arises is: why to use motion embedding when the goal task is to realize image-to- to-e mapping? We hypothesize that formalization of higher-order pose association would enhance the expressibility of the dark-knowledge which is of prime interest towards realizing superior eross-mout mossl alignment. Here, higher-order refers to relational association of a large number of pose entities. In C²² C each individual triplet relation expresses positive coupling of 27 pose entities against just 2 in C". Thus, the hierarchical motion embedding facilitates a suitable platform to formalize temporally structured (temporal order must be retained) higher-order relational associations of the pose space entities (see Fig. 3C).
3.3.2 Distilling non-local relations via equivariance consistency
We observe that adapting the image-to-latent mapper G just by minimizing LLCK and LACR results poor pose estimation performance. In general self-supervised contrastive learning prior-arts, latent embedding is learned from scratch alongside the training of image-to-laten mapping. However, in the proposed setting, we aim to adapt the image-to-laten network in order align output with the pre-learned pose and motion embeddings (i.e. the latent embeddings of frozen {Ep, Dp and
5

---

{Ea, D.)). Consequently, we need to address special degenerate scenario where the distillation losses completely converge even when the model exhibits severe instance-level misalignment [46].
Local vs non-local relations. We hypothesize that relations having non-local positive coupling, equivalent relations in knowledge-gras (KG), are the best way express the instance-grounded dark-knowledd Note that, the positive coupling in contrastive- based relations are limited to local neighborhood, as here, the positive counterpart of an anchor is obtained via simple pose-invariant augmentations (i.e. _ = A(xz) in both C" and )x). To this end, our goal is to come up with new set of relational associations with non-local positive couplings. This aims to characterize long-range latent pose interactions, i.e. positive coupling of two entities that are grounded far away in the latent pose or motion space (see Fig. 4C). To this end, we introduce relation networks.
Relation networks. Relation networks operating on the pose and motion embeddings are neural network mappings of the form, T3 2 Z and Ty. > د V respectively. Here, r= and ry denote independent embedding specific rule-ids for the pose and motion embeddings respectively. The equivalent rule-specific image space relation transformation operating on the image and image- sequence are represented as TT. : :X' - XX'>> X. and T X xX. And the same on the raw pose and motion space is represented Tę -. and T.. : V. Note that, except T; and T; all other transformations are simple affine-like operations without involving neural network mappings.
a) Distilling lower-order non-local relations. For each target image I E X, we first construct a dataset of positive coupling represented as N {<1), x7" where x Tr. (2t)3. Here, one is already aware of the corresponding pose relation, i.e. y." = TV, (w+). And, the same relation in the latent pose space is expressed as 2,- = =T: (2x).
Let us consider a non-local relation termed as "pose-flip" with rule-id r. 1. Nf by pairing each target image with its flipped version obtained via a simple image-flip transformation operation, (i.e. Tς). Note that, similar pose space transformation would involve swapping the left side joints with that of the right (i.e. TT). And, the corresponding relation network Tf is a multi-layer neural network which is trained to regress 2;- =Ep o T (y)) while feeding 2 = Ep(1)) as the input. Here, Vr is sampled from the unpaired 3D pose data V Next, we formalize the corresponding relational energy loss that uses the frozen relation network Ti and is represented as,
Wf is obtained
C = E(xeat)
||T] G(xz) G(z)) II
b) Distilling higher-order non-local relations. One can define higher order non-local relations on the hierarchical motion space to devise a similar relational energy loss.
Let us consider non-local relation termed as "flip-backward" with rule-id r. = 1. Here, N/ = {(X, x+) where X+ = T (X)} is obtained by pairing each target sequence X with its flip-backward counterpart which obtained via temporal reversal (backwards) of the image- flipped frames. Accordingly, the corresponding relation network Tÿ is multi-layer neural network trained to regress ut = Eg oTT))((()) while feeding v ==E, as the input, where Y is sampled m
from the unpaired 3D pose sequences y. And, the corresponding relational energy loss is represented
as,
CT = EXxx+)NNN |[TV o G(X) - -G(
(3)
Finally, cross-modal alignment is achieved by simultancously minimizing all the relational energies, i.e.
L= LcR + LNIR; where LCR LLCR Lucr and LNIR =.. Ci Er. fy.
Next, we discuss certain insights which can help us to formalize effective non-local relations.
3.3.3 What makes non-local relations more effective?
In the latent pose space, we conceptualize two different relation networks viz. a) pose-flip. Ti and b) dd-plane-rotatie
Tž. The corresponding relational energies are denoted as L and L respectively. Though minimizing Li yields substantial improvement over Ours-LL LL (just minimizing Lex without LNLR). jointly minimizing both Li and L5 does not yield much improvement on the final pose estimation performance. However, the combination of flip and rotation in a single relation yields considerable improvement in performance. We call this rule "fliptinplane- r= =3. We also notice improvement in performance when using in-plane rotation used conjunction with the higher order relation "flip-backwar (rotation of individual flipped frames). This rule
with rule-id

---

A. Pose-embedding
B. motion-embedding
Crounded relations on the mation space
E9500
"onlysFape
PaTE SEUUUS Oi Socco Exty
Non local
region
Tlipeipplane. backward
8
onny bacaward
Local
Slow--sadoward
in-plane-8 in-piane-10 10P
Alp+5d (all frames)
Latert dist.
anky-backward
pose-Ai
Aip+ backwardddd
RPp+50P
alow-bsckward
Input senuence
Figure 4: Embedding visualizations. A. Grid-interpolation of pose embedding (pose-flipp or- ange boxes; in-plane-5": blue boxes) with bar-plot showing the avg. latent-distance. B. Grid- interpolation of speed of the generated motions with latent-distance bar-plot comparing the non-local motion relations. C. 2D PCA projected motion embedding points with glances of the anchor video (middle-point) and the corresponding transformed videos obtained by operating over various motion- related non-local relation candidates. Note that, only the relations in blue are selected because of their high non- non-localness indicated by the latent-distances d' and d".
is termed warppp-inpnaee---bacwaaarr with rule-id r, 2. We also introduce higher-order non-local relation baw-bbackward,, with rule-id r. 3. The relational pairs for slow-backward constructed by sampling the original sequence 15 FPS, to enable temporally slow 30 FPS slow backward video taken from the middle region (see Fig. 4C).
Quantifying non-localness via latent- distance. We hypothesize that, "rela-
A. Pre-learning steps (training on source data)
1) Train Ep and D. on unpaired 3D poses, E )
tions coupling diverse samples (long-range interactions) lead to better cross- mods- alignment"" To evaluate this hypothe- sis, we define a distance metric to appre- hend the extent of non-localness of indi- vidual relations which termed as latent- distance. Here, the latent-distance di mean, JEN; || 2,, measures the erage distance between relation pairs (z, Ep(y)); 25 = Ep TT (y:)). for relation-id r= 1, where y(y)) is sampled from the unpaired 3D pose data y. In Fig. 4A, the bar-plot comparing the latent- ditett-distanc for various relations clearly highlight the gap in the degree of localocllness thus justify- ing the aforementioned observation regarding the performance gap. We perform a similar metric analysis to select the best set of relations defined the motion space V (see Fig. 4B and 4C).
2) Train Eg and Dg on unpaired motion, Y )
3) Train G on labeled source dataset, (x" E D
4) Train тž for "fliptinplane-O on using y
5) Train тA for "ELiptinplaae-O-bacakee --aaaaee----aaaardd- on V
using Y E
6) Train Tğ for "slos-backuard" on V using Y e y
B. Unsupervised target adaptation
7) After initializing G from G", train by minimizing the local relational energy, LoR alongside minimizing the non-local relations L3, L and Lz
Overview the optimization steps. Note that, C3 L and Lÿ rely on frozen Tž T and Тÿ to distill the embedded relational dark- knowledge.
Algorithm
Finalizing the non-local relations. At the end, we take up a single pose relation, i.e.. fliptinplane-0 and two diverse motion relations, i.e., Sapptinplane---bacaaee blow-backward realize our final cross-modal alignment framework (Fig. 4). With L3 being relational energy specific to the pose space relation fliptinplame-)) we denote L3 and L3 as the re- lational energies specific to the motion relations fliptinplane coane-0-bacee b-backward and slow- backward respectively. Please refer to Supplementary for an ablation on the effect of changing the angle 0.
and
Overview of the optimization steps. Algorithm 1 summarizes the training as step-by-step learning process. Here, each step uses frozen models obtained from the previous step. After initializing from G only a minimal set of network parameters (Res-3 block of ResNet-50) are updated to learn target-specific mapping. This can be see as regularization for the unsupervised adaptation training. We freeze Dp while training as crucial to regularize avoid degenerate solutions (or mode-collapse). Updating D, while training G requires to impose an additional adversarial discriminator loss (as used HMR [29]) order to uphold its ability to decode plausible pose predictions. Note that, updating Dp would update the manifold structure of
unsupervised adaptation to
7

---

Table 2: Comparison of 3D pose estimation re- sults on Human3.6.) dataset. Full (3D) supervi- sion denotes using GT 3D supervision for training. Semi-Sup. denotes supervised training only on subject $1, Unsup. refers to training on unpaired Human3. 6.6M samples (unlabeled). We achieve SOTA results on both semi-supervised and un- sup. training methods. Our unsupervised training outperforms the SOTA even in presence of huge domain gap between the SURREAL and H3.6M datasets. (MV) denotes usage of multi-view pervision.
Table 3: Quantitative results for 3D pose estima- tion on 3DHP under three different supervision settings. Full (3D) supervision denotes using GT 3D supervision on 3DHP during training. Unsu- pervised Adaptation refers to training on labeled Human3.6M and adapting to unlabeled 3DHP. Di- rect Transfer denotes training on labeled source dataset, adapting to unlabeled web-dataset and evaluating on unseen 3DHP dataset. ddenotes the implementation taken from [89]. (MV) denotes
using multiview data.
Training Methods
PCK
AUCT PA-MPJPE]
Training
Methods
PA-MPIPE J MPJPE
Full
Rogez et al. [67]
59.6
27.6
158.4
Chen et af. [10]
82.7
(3D)
Mchta
[50]
76.6
40.4
124.7
Martinez et [48]
47.7
Sup.
Kocabas et al. [33]
77.5
Full (3D)
Iqbal et al. [26]
83.0
Li et al. [41]
38.0
Sup.
Xu al. [84]
36,2
45.6
Iabal af. [26
76.5
122.4
Chen et al. [14]
32.7
47.3
Kundu ef al. [40]
79.2
43.4
99.2
Unsup. Adupt.
Wandt et [81]*
81.6
47
95.4
Mitra et al. [52]
90.8
120.9
Kundu al. [39]
83.2
58.7
97.6
Semi-sup. (sup. on S1)
Li et al. [42]
Rhodin et al. [65]
66.5
65.1
88.8
Zhan ct al. [92]*
Ours(H -M)
86.0
89.8
46.7
59.3
79.6
96.8
Kocabus et al. [33]
60.2
Chen of ol. [TT]
64.3
31.6
Lybal et al. [26]
$1.4
62.8
Kundu et al. [40]
76.5
39.8
115.3
Ourx(S -H, Semi)
48.2
$7.6
Direct Transfer
Li al. [41]
81.2
46.1
Kundu al. [39]
99.2
Kundu et al. [39]
82.1
56.3
103.8
Unsup.
Kundu al. [40]
89.4
OurstS - W)
79.1
43.4
114.9
Ourx(S-H)
86.2
97.8
Ours(SH
85.5
60.7
94.1
the embedding space. Further, as the relation networks (Tž, Tý, Tÿ) operate on the frozen latent embeddings, the pre-learned relations networks (used to define the relation distillation objective) would no longer be useful and are required be updated alongside. Such unconstrained optimization greatly destabilizes the self-adaptive process (in the absence of cross-modal pairs) and degrades the performance significantly. Note that the proposed self-adaptation step does not involve any complex adversarial loss component, which greatly simplifies the training procedure.
4 Experiments
We demonstrate effectiveness of the proposed framework by evaluating it on variety of cross-dataset adaptation settings.
Implementation details. The backbone of G constitutes of an ImageNet initialized ResNet-50 [23] (till Res-4F) followed by series of fully-connected (FC) layers to obtain the latent pose representation, E 3R The pose auto-encoder, (Ep. pp} are FC networks operating on the 3D pose y (17 joints). The motion auto-encoder, (Ea, D is composed of bidirectional L.STMs [22] with 128 hidden units operating on a fixed sequence length of 30 (30 FPS) where the intermediate notion-embedddig, E 128 The relation networks constitute of simple FC layers. We associate separate Adam optimizer [32] to each relational energy term which are optimized in alternate training iterations. The adaptation is performed on an Nvidia V100 GPU with each batch containing 8 videos each of frame-length 30 (see Suppl. for more details).
Datasets. We use the CMU-MoCap [1] dataset as the sample set for unpaired 3D poses y and unpaired pose sequences )r. We use the synthetic SURREAL (S) dataset [79] as one of the source datasets. The sample set for the unpaired videos X constitutes of single-person action videos (dance forms, sports, etc.) collected from the Sp dataset [31]. The video frames
forwarded
through a person-detector [64] to obtain the person-focused image sequences. We name it as web- dataset (W). For a fair evaluation, we use the standard, in-studio Human3.6N, 3.6M (H) dataset [25] as
both source or target domain, in different problem settings. MPI-INF-3DHP (M) [49] is used as
an unlabeled target domain to evaluate cross-studio adaptation. Further, 3DPW [80], and LSP [27]
datasets are used to evaluate our cross-dataset generalizability, without involving these during training.
4.1 Adaptation settings
We introduce the following adaptation settings with various source and target datasct selections.
8

---

a) Ours(S-AH)) We use labeled synthetic SURREAL (S) as the source while unlabeled Human3 6M (H) acts as the target without involving the web-dataset (W). The resultant model is tuned work well only for the specific in-studio environment, thus may fail to generalize for in-the-wild data.
b) Ours(H->M) Here, labeled Human3.6M is used the only source domain while MPI-INF-3DHP (M) samples are used as the unlabeled target. This evaluates our cross-studio adaptation performance.
с) Ours(S-WW) Here, labeled SURREAL (S) used as the only source domain while the unlabeled target videos are extracted from the web-dataset (W). The resultant model is evalu- 3DHP
Table 4: Quantitative results comparing our ap- proach against prior-arts for 3D pose estimation on 3DPW. Full (3D) supervision denotes using ground-truth 3D supervision on 3DPW for train- ing. Direct Transfer denotes training on a variety of labeled source dataset and directly evaluating the resultant model on unseen 3DPW test set. We achieve state-of- ofte-of-thhe-rt result upon using both Hu- man3.6M (H) and SURREAL (S) as the source and the web-data as the target. denotes number taken from [30]. denotes using parametric mesh models.
ated for cross-dataset generalization and 3DPW (see Table and Table 4). Note that, web-dataset is the most challenging in-the- wild dataset with substantial diversity in pose, apparel, and backgrounds.
d) Ours(SHI>W) Here, the source domain is combination of labeled SURREAL (S) and Human3. ..M (H) datasets while the unlabeled web-dataset (W) acts as the target. This setting enjoys the advantage of strong source supervi- sion on natural but in-studio Human3.6M, along-
Training
Methods
PA-MPIPE
Full (3D)
Amab et al. [3]
77.2
Supervision
Sun
[74]*
69.5
Martinez et al. [48]
157.0
side SURREAL.
Dubral et [15]*
92.3
Kanazawa et al. [30]*
80.1
4.2 Evaluation on benchmark datasets In this section, we evaluate the proposed ap- proach on the standard benchmark datasets. MPJPE and PA-MPJPE [25] denote standard mean per joint position error metric computed
Direct Transfer
Doersch et al. [16]*
82.4
Kanazawa et al. [29]**
76.7
Ours(S-3)
79.3
Ours(SH-
72.1
before and after Procrustes Alignment [21]. Following prior arts [49], we report PCK and AUC, i.e. percentage of correct keypoints and area under the curve respectively for the 3DHP dataset.
a) Evaluation Human3.6M. Table lists comparison of Ours(S->H) against the prior arts Under unsupervised training setup, Ours(S outperforms the prior state-of-the-art (SOTA) by a significant margin. We attribute this improvement to our attempt utilize domain randomization via SURREAL and the proposed cross-dataset alignment procedure. Under semi semi-supervised training, the target adaptation is supervised on a small subset of labeled samples (i.e. subject S1 in Human3.61 6 alongside unsupervised alignment on the rest. The resultant model, Ours(S-*H Semi) also outper- forms the prior semi-supervised works even in the absence of additional multi-view supervision as used in some of the prior-arts.
b) Evaluation 3DHP. Table 3 shows detailed quantitative comparison of three of our adaptation vari- ants; Ours(H--M), Ours(S-W),, and Ours(SH-WW) W), under two training setups. Ours(SH-WW achieves state-of-the-art unseen transfer performance validating superior generalizability of the proposed framework. Ours(SH-W)) uses in-studio but real Human3.6. as an additional labeled source data alongside SURREAL, thus reducing the domain gap when compared against Ours(S->))
Table 5: Ablation study on Human3.6n .6M dataset (i.e. Ours(S-+H)) Starting from the baseline (inference through the SURREAL trained model), usage of different relational energies results in a substantial improve- ment in the adaptation performance.
Modules Involved
Ablation
MPJPE
Source-only
G, Dp
209.6
+LLCR
G, Dp
193.4
c) Evaluation 3DPW. Table 4 reports a detailed comparison of our variants against the prior-arts on the in-the-wild 3DPW dataset. All the methods un- der direct transfer do not use 3DPW samples even for any supervised or unsupervised training. A lower PA- MPJPE for Ours(SH-a) W) clearly highlight our superior cross cross-dttaset generalizability against the prior approaches which also utilize additional information such as the SMPL. mesh model [45].
+LHCR
+E.
172.1
+
+Tī
139.7
+C5
+Ts
91.8
+C3
+T3
86.2
d) Ablation study on Human3.6M. Table 5 shows an ablative analysis highlighting the effectiveness of individual relational energies towards realizing better unsupervised alignment. As compared

---

A. Web-dataset (in-the-wild)
Human3.6M dataset (in-sludio)
3DHP datasel (in-thewild)
3DPW datasel (in-the-wild)
LSP (unseen, in-the-wild)
Figure 5: Predictions obtained via Ours(SH-W generalize well to both in-studio and in m--thewiil datasets. The failure cases (rare poses or poses with inter-limb occlusion) are highlighted by results on red bases.
to the source-only bascline (first row of Table 5), distillation of contrastive relations in LeR yields the first stage of improvement. Minimizing the non-local pose space relational energy L5 results in further improvement by effectively distilling an instance -arance-grounded distilling higher order non-local motion relations (i.e. L and L3) yields a substantial improvement by effectively attenuating the instance-level misalignment.
knkwleddee Additionally,
Qualitative evaluation and limitations. Fig. 5 illustrates the generaliability of Ours(SII~>W) proposed approach under diverse pose, apparel, and background variations. Results with red base show some of the failure cases, The model is restricted from predicting implausible 3D pose outcomes as these are inferred through the prior-enforeing generative pose decoder. However, the model fails under certain drastic scenarios such as high background clutter, multi-level body-part occlusion, and rare athletic poses, Other limitations include tody-truunation scenarios i.e. scenarios where certain mody-parts are either occluded by external objects are outside the image frame. In such cases there are multiple plausible pose outcomes, thus asking for a future exploration of probabilistic pose estimation modeling. Taking a different direction, one can explore innovative ways to utilize inputs from auxiliary modalities (such as depth, foreground segmentation, etc.) as and when available from the deployed target environment (see Suppl. for more details).
Negative societal impacts. While do not foresee our framework causing any direct negative societal impact. it may be leveraged to create malicious applications utilizing human tracking and action recognition. Estimating a human pose does not require any identity information. However, methods such as gait recognition can be indirectly used to identify personal attributes thereby raising privacy concerns. We urge the readers to make use of the work detailed responsibly, limiting the usage to legal use-cases.
5 Conclusion
We presented cross-modal alignment technique to align the learned representations from two diverse modalities. Our unsupervised technique allows adaptation to the wildest unlabeled samples gathered from web while initializing the base model on substantially diverse but unrealistic SURREAL. We analyzed the importance of higher order, non-local relations in expressing the rich instance-groundd dark knowledge as required to attenuate the instance-level misalignment. future, we plan to extend our framework for multi-modal alignment in presence of additional input modalities.
Acknowledgments and Disclosure of Funding
This work was supported by Google Ph.D. Fellowship (Jogendra) and project grant from MeitY (No.44(6Y2219-IITEA), Govt. of India.
10

---

References
[1] CMU graphics lab motion capture database. available: http://mocappes.cmu..uuu..
[2] A. Agarwal and B. Triggs. Recovering 3d human pose from monocular images. Transactions on Pattern Analysis and Machine Intelligence, 2006.
[3] Anurag Amab, Carl Doersch, and Andrew Zisserman. Exploiting temporal context for 3d human pose estimation in the wild. In CVPR, 2019. 9
[4] Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saunshi. theoretical analysis of contrastive unsupervised representation learning. arXiv preprint arXiv:1902.009) 2019. 3
[5] V. Belagiannis, S. Amin, M. Andriluka. B. Schiele, N. Navab, and S. Ilce. 3d pictorial structures for multiple human pose estimation, In CVPR, 2014.
[6] L. Bo and Sminchisescu. Structured output-associative regression. CVPR, 2009.
[7] Federica Bogo, A. Kanazawa, Christoph Lassner, Gehler, J. Romero, and Michael J. Black. Keep smpl: Automatic estimation of 3d human pose and shape from single image. ECCV, 2016.
[8] M. Burenius, J. Sullivan, and S. Carlsson. 3d pictorial structures for multiple view articulated pose estimation. In CVPR, 2013.
[9] Jinkun Cao, Hongyang Tang, Hao-Shu Fang, Xiaoyong Shen, Cewu Lu, and Yu-Wing Tai. adaptation for animal pose estimation. In ICCV, 2019. 3
[10] Ching-Hang Hang Chen and Deva Ramanan, 3D human pose cstimation= 2d pose estimation+ matching. In CVPR, 2017. 8
[11] Shing-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Rohith MV, Stefan Stojanov, and James M Rehg. Unsupervised 3D pose estimation with geometric self-supervision. In CVPR, 2019. 1, 8
[12] Ting Chen, Simon Korbliihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020. 2,5
[13] Xipeng Chen, Kwan- Yee Lin, Wentao Liu, Chen Qian, and Liang Lin. Weakly-supervised discovery of geometry-aver aware representation for 3D human pose estimation. CVPR, 2019.
[14] Zerui Chen, Yan Huang, Hongyuan Yu, Bin Xue, Ke Han, Yiru Guo, and Liang Wang. Towards part-aware monocular human pose estimation: An architecture search approach. ECCV, 2020.
[15] Rishabh Dabral, Anurag Mundhada, Uday Kusupati, Safeer Afaque, Abhishek Sharma, and Arjun Jain. Learning 3D human pose from structure and motion. ECCV, 2018.
[16] Carl Doersch and Andrew Zisserman. Sim2real transfer learning for 3d human pose estimation: motion the rescue. NewrIPS, 2019. 3,4,9
[17] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015. 4
[18] Dedre Gentner and Kenneth J. Kurtz. Relational categories. In Categorization inside and outside the laboratory: Essays in honor of Douglas I. Medin. 2005. 2
[19] Michael Gleicher. Retargetting motion to new characters. In Proceedings of the 25th annual conference on Computer graphics and interactive techniques, 1998.
[20] Micah Goldwater, Hilary category learning. Journal of Experimental Psychology: General, 2018. 2
Don, Moritz Krusche, and Evan Livesey. Relational discovery in
[21] John C Gower. Generalized procrustes analysis. Psychometrika, 1975.
[22] Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidircctional Istm and other neural network architcctures. Neural networks, 2005.
[23] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 8
[24] Geoffrey Hinton. Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint ar3.v: 2015. 3.4

---

[25] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m 6:: Large scale datasets and predictive methods for 3D human sensing in natural environments. IEEE transactions pattern analysis and machine intelligence, 2013. 1,2, 8,
[26] Umar lqbal, Pavlo Molchanov, and Jan Kautz. Weakly-supervise 3d human pose learning via multi-view images in the wild. In CVPR, 2020. 3,8
[27] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance models for human pose estimation. In BMVC, 2010.
[28] Hanbyul Joo, Tomas Simon, and Yaser Sheikh. Total capture: A 3d deformation model for tracking faces, hands, and bodies. In CVPR, 2018.
[29] Angjoo Kanazawa, Michael J Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In CVPR, 2018. 7.9
[30] Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik. Learning 3d human dynamics from video. In CVPR, 2019. 9
[31] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014. 8
[32] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6441 2014. 8
[33] Muhammed Kocabas, Salih Karagoz, and Emre Akbas. Self-supervised learning of 3D human pose using multi-view geometry. In CVPR, 2019. 1,8 8
[34] Chen Kong and Simon Lucey. Deep non-rigid structure from motion. ICCV, 2019.
[35] Jogendra Nath Kundu, Maharshi Gor, and R Venkatesh Babu. Bihmp-gan: Bidirectional 3D human motion prediction gan. AAAI, 2019.
[36] Jogendra Nath Kundu, Maharshi Gor, Phani Krishna Uppala, and R Venkatesh Babu. Unsupervised feature learning of human actions as trajectories pose embedding manifold. WACV, 2019. 4
[37] Jogendra Nath Kundu, Nishank Lakkakula, and R Venkatesh Babu. Um-adapt: Unsupervised multi-task adaptation using adversarial cross-task distillation. In ICCV, 2019. 3
[38] Jogendra Nath Kundu, Ambareesh Revanur, Govind V Waghmare, Rahul M Venkatesh, and R Venkatesh Babu. Unsupervised cross-modal alignment for multi-person 3d pose estimation. In ECCV, 2020. 3
[39] Jogendra Nath Kundu, Siddharth Seth, Varun Jampani, Mugalodi Rakesh, R Venkatesh Babu, and Anirban Chakraborty. Self-supervised 3d human pose estimation via part guided novel image synthesis. In CVPR, 2020. 8
[40] Jogendra Nath Kundu, Siddharth Seth, Rabul M V. Rakesh Mugalodi. R Venkatesh Babu, and Anirban Chakraborty. Kinematic-strucuun--reeeee representation for unsupervised 3D human pose estimation. In AAAI, 2020. 8
[41] Shichao Li, Lei Ke, Kevin Pratama, Wing Tai, Chi-Keung Tang, and Swang-Ting Cheng. Cascaded deep monocular 3d human pose estimation with evolutionary training data. In CVPR, 2020.
[42] Z. Li, X. Wang, Wang, and P. Jiang. On boosting single-frame 3d human pose estimation via monocular videos. ICCV, 2019.
[43] Yufan Liu, Jiajiong Cao. Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yunqiang Duan. Knowledge distillation via instance relationship graph. CVPR, 2019.
[44] Mingsheng Long, Yue Cao, Jianmin Wang, and Michael Jordan. Learning transferable features with deep adaptation networks. In ICMI., 2015. 4
[45] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM transactions on graphics, 2015.
[46] Yawei Luo, Liang Zheng, Tao Guan. Junqing Yu. and Yi Yang. Taking closer look at domain shift: Category-level adversaries for semantics consistent domain adaptation. In CVPR, 2019. 2, 6
[47] Alireza Makhzani, Jonathon Shlens. Navdeep Jaitly, lan Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint 1515v45511.05644, 2015. 1.4
12

---

[48] Julieta Martinez. Rayat Hossain, Javier Romero, and James J Little. A simple yet effective baseline for 3D human pose estimation. In ICCV, 2017. 8,
[49] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3D human pose estimation in the wild using improved enn supervision. In 3DV, 2017. 2, 8,
[50] Dushyant Mchta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafici, Hans-Pete Scidel, Weipeng Xu, Dan Casas, and Christian Thcobalt. Vnect: Real-time 3D human pose estimation with single rgb camera. ACM Transactions Graphics, 2017.
[51] Ishan Misra and Laurens van der Maaten, Self-supervised learning of pretext-invariant representations. In CVPR, 2020. 5,5
[52] Rahul Mitra, Nitesh B. Gundavarapu, Abhishek Sharma, and Arjun Jain. Multiview-consistent semi- supervised learning for 3d human pose estimation. CVPR, 2020.
[53] Francese Moreno-Noguer. 3D human pose estimation from a single image distance matrix regression. In CVPR, 2017. 3
[54] B. X. Nie, P. Wei, and S. Zhu. Monocular 3d human pose estimation by predicting depth on joints. In ICCV, 2017. 3
[55] Kent I. Norman, Cyherpsychology: An introduction to human-compiter interaction. interaaciiono Cambridge university press, 2017.
[56] David Novotny, Nikhila Ravi, Benjamin Graham, Natalia Neverova, and Andrea Vodaldi. C3DPO: Canonical 3D pose networks for non-rigid structure from motion. In ICCV, 2019.
[57] Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03744 2018. 3.5
[58] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019.
[59] G. Pavlakos. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to- fine volumetric prediction for single-image 3d human pose. In CVPR, 2017. 3
[60] Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis. Ordinal depth supervision for 3D human posc estimation. In CVPR, 2018. 3
[61] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019.
[62] Georgios Pavlakos. Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, and Michael J. Black, Expressive body capture: 3d hands, face, and body from single image. In CVPR, 2019. 3
[63] Baoyun Peng. Xiao Jin, Jiaheng Liu, Dongsheng Li, Yichao Wu, Yu Liu, Shunfeng Zhou, and Zhaoning Zhang. Correlation congruence for knowledge distillation. In CVPR, 2019. 3
[64] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object dctection with region proposal networks. In NeurIPS, 2015. 2015. 8
[65] Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware representation for 3D human pose estimation. ECCV, 2018. 8
[66] Helge Rhodin, Jörg Spöri, Isinsu Katircioglu, Victor Constantin, Frédéric Meyer, Erich Müller, Mathieu Salzmann, and Pascal Fua. Learning monocular 3D human pose estimation from multi-view images. In CVPR, 2018.
[67] Gregory Rogez, Philippe Weinzaepfel, and Cordelia Schmid. Ler-net: Lecalizziion-classfiiaaoooreeegs for human pose. In CVPR, 2017. 8
[68] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets, arXiv preprint
2014, 3,4
[69] Rómer Rosales and S. Sclaroff. Learning body pose via specialized maps. In NeurIPS, 2001.
13

---

[70] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. In ECCV, 2010. 4
[71] Leonid Sigal, Alexandru O Balan, and Michael J Black, Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International journal of computer vision, 2010.
[72] Suraj Srinivas and Francois Fleuret. Knowledge transfer with jacobian matching. In ICML, 2018.
[73] Xiao Sun, Bin Xiao, Fangyin Wei, Shuang Liang, and Yichen Wei. Integral human pose regression. In ECCV, 2018. 3
[74] Yu Sun, Yun Ye, Wu Liu, Wenpeng Gao, YiL.i Fu, and Tao Mei. Human mesh recovery from monocular images a skeleton- diseetaned representation. ICCV, 2019.
[75] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2019. 3
[76] Frederick Tung and Greg Mori. Similarity-preserving knowledge distillation. In CVPR, 2019.
[77] Hsiao-Ya Yu Fish Tung, Adam W Harley, William Seto, and Katerina Fragkiadaki. Adversarial inverse graphies networks: Learning 2D-to-3D lifting and image-to-image translation from unpaired supervision. In ICCV, 2017.
[78] Erie Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, 2017. 4
[79] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael Black, Ivan Laptev, and Cordelia Schmid. Learning from synthetic humans. In CVPR, 2017. 1,8
[80] Timo von Marcard, Roberto Henschel, Michael Black, Bodo Rosenhahn, and Gerard Pons-Moll. Re- covering accurate 3d human pose in the wild using imus and moving camera. In ECCV, 2018. 2,
[81] Bastian Wandt and Bodo Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection network for 3D human pose estimation. In CVPR, 2019. 1,
[82] Chang Wang and Sridhar Mahadevan. Heterogeneous domain adaptation using manifold alignment. In IJCAI, 2011. 4
[83] Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self supl-suuerv In ECCV, 2020. 3
[84] Jingwei Xu, Zhenbo Yu. Bingbing Ni, Jiancheng Yang, Xiaokang Yang, and Wenjun Zhang. Deep kinematics analysis for monocular 3d human pose estimation. In CVPR, 2020.
[85] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In AAAI, 2018.
[86] W. Yang. W. Ouyang. X. Wang. Ren, H. Li, and X. Wang. 3d human pose estimation in the wild by adversarial learning. CVPR, 2018.
[87] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim. A gift from knowledge distillation: Fast optimization, network minimization and transfer learning. In CVPR, 2017. 3
[88] Sergey Zagoruyko and Nikos Komodakis. Paying more attention to attention: Improving the performance of convolutional neural networks via attention transfer. arXiv preprint 1612: 03928, 2016.
[89] Jianfeng Zhang, Xuecheng Nie, and Jiashi Feng. Inference stage optimization for cross-scenario human pose estimation. In NeurlPS, 2020. 8
[90] Jianfeng Zhang. Xuecheng Nie, and Jiashi Feng. Inference stage optimization for cross noss-scenarii human pose estimation. In NeurlPS, 2020.
[91] Xiheng Zhang, Yongkang Wong, Mohan S. Kankanhalli, and Weidong Geng. Unsupervised domain adaptation for 3d human pose estimation. In ACMMM, 2019.
[92] Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In CVPR, 2019. 3,
[93] Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue. and Yichen Wei. Towards 3d human pose estimation in the wild: A supervi- approach. In ICCV, 2017.
14


Reviews:
[{"review_id": "rh3Aw_opCGG", "reviewer": "Reviewer_jckK", "summary": "The paper focuses on 3D human pose estimation in a scenario of domain adaptation. To avoid dataset-bias, the authors address the problem in a more restrict setting: any form of paired supervision or auxiliary cues should not be used. \n\nThe authors first use AAE modules to learn pose and motion latent spaces. Secondly, the authors train a pose encoder which tries to map the input image to the learned pose latent space. Finally, given the target-domain videos, the authors train the pose encoder together with the decoder of the pose AAE. The model is optimized by minimizing relational energies. ", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength\n- This paper present a good solution for domain adaptive 3D human pose estimation. Based on the proposed setup, it is technically valid, and it should work well.\n- Good figures to better illustrate the proposed idea and the learned embedding.\n- Good ablation study of the use of different relational energies.\n- The results are strong. In some cases, the proposed method even outperforms the multi-view approaches.\n\n\nWeakness\n- The presentation is very difficult to follow. This is probably due to the writing style of the authors. Most of the things are described in a very abstract manner. It is difficult for me to understand the proposed method in details. I spent hours to read the paper, but still I am not confident about all details in the proposed solution.  Also, I can't find strong connections with the cognitive science. The authors tend to introduce new terms, but I find them a bit distracted.  \n\n- The proposed method involves multi-stage training, but not end-to-end training. Each training stage includes a new model, and each would affect the final performance. Given many uncertainties, it is difficult to understand the performance. I would suggest to ablate each training stage, and show the effectiveness of each module.\n\n- What is performance of the AAEs? What quality of the pose embedding is required? Same question for the motion embedding. \n\n- Instead of training AAEs to generate pre-learned latent feature, the authors may want to use SMPL pose parameters as an alternative choice of the latent feature. Since SMPL parameters are also pre-trained and commonly used in the literature, this may serve as a good baseline to better promote the proposed method.  \n\n- Why should we use a pre-learned output pose embedding, but not fully end-to-end learned embedding? The pre-learned embedding may lead to sub-optimal performance. The design choice of the proposed method is not well motivated. \n\n- Lack of baselines: One important baseline is to use pseudo pairs, and perform distillation, as the authors described in line181. Another baseline is domain adaptation with adversarial training. It is better to provide concrete performance comparisons to justify the proposed method.\n\n- The authors argue that they do not use paired supervision or auxiliary cues to avoid dataset bias. However, the proposed method replies on Faster RCNN, which may imply it still require supervised pre-training on large number of paired training data. The authors may want to add more discussions about how they avoid the dataset bias.\n\n\nOverall:\n- The paper introduces an interesting method for human pose estimation in the setting of domain adaptation. However, due to bad writing, it is very difficult to follow (at least to me). Many things remain unclear in the paper. Since the target scenario is relatively different from prior works, I would suggest the authors to build their own baselines, in order to better justify the proposed solution. After careful consideration, I think this paper is not ready for a publication.\n\n\n*******Post rebuttal*********\nThank you for the detailed response. My overall comments remain the same. I think the presentation is difficult to follow, and requires several revisions. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "n/a", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "72"}, "extra_text": {}}, {"review_id": "jgIhJ6lSehe", "reviewer": "Reviewer_fswa", "summary": "## Summary\n\nPaper proposes a self-adaptation framework to align the unpaired samples of 3D pose and video frames to solve 3D human pose estimation from in-the-wild images. This cross-modal alignment is realized through a relation distillation scheme. The proposed approach achieves compelling results on 3D human pose datasets among self-supervised and unsupervised methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- Paper tackles a real problem of scarcity of 3D pose labels for in-the-wild unconstrained images. They propose a self-adaptation method that helps to utilize web videos and large-scale motion capture datasets both of which are available in abundance.  \n\n- Proposed domain adaptation approach sounds novel. Motivation and implementation make sense to be able to perform unsupervised adaptation.\n\n- Experiments are detailed. Three major human pose estimation datasets are used for the model's performance. Ablation experiments show that propose the contribution of each relation distillation term into the final performance.\n\n## Weaknesses\n\n- Paper writing is quite hard to follow. Especially in the intro, there are lots of technical terms which is not familiar to the human pose estimation community. These are mostly domain adaptation or knowledge distillation terms and they should be introduced in a logical flow to the reader.\n\n- CMU mocap dataset is chosen as the unpaired 3D pose dataset. AMASS dataset is much larger in terms of quantity, the number of subjects, pose variation, and actions performed. It also includes CMU as a subset. So, AMASS could be a better choice for the proposed method.\n\n- A motivation of the proposed approach is overcoming the dataset bias. However, the Dp is kept frozen when used with G. This means that the manifold learned/encoded by Z is biased towards the unpaired 3D pose dataset which is CMU mocap in this case. I think this is a major limitation of the current method and this should be tackled to overcome the dataset bias.\n\n### Post Rebuttal\n\nAuthors addressed my comments in a satisfactory way. Considering the other reviews and author's responses to them I would like to keep my initial score of \"6: Marginally above the acceptance threshold\".", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the limitations and societal impact are discussed in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours"}, "extra_text": {}}, {"review_id": "1TnfeIfs5qf", "reviewer": "Reviewer_UoE5", "summary": "This paper proposed to learn to predict the 3D human pose for real-world images via adapting the model from the source data (label constraints) to unlabeled data. The data used for training contains: (1) the labeled source domain data (e.g. human3.6m or surreal); (2) real human video sequence without pose annotations; and (3) motion capture data (3D pose without images). (2) and (3) are unpaired. The authors proposed to devise: (a) image-to-latent; (b) pose-to-latent; and (c) latent-to-pose, to learn and predict 3D poses. In particular, (a) and (c) forms the test time predictions for the real-world target domain, while (b) and (c) constitutes an auto-encoder. The core problems in here is that the latent space in (ac) and (bc) might be mismatched. The authors proposed a smart way to make sure they are aligned (see Strength below) thanks to the contrastive loss (for setting up latent space non-local grouping rules) as well as the loss from the source domain data for regularization to make sure the two latent domains align with each other. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\nI really appreciate the main novelty of the paper. To put the spark of the paper in one sentence: the paper proposed an innovated self supervised method to align the human pose latent space from unpaired image-to-latent and the pose-to-latent, via enforcing several latent space rules that is available for unpaired image-to-latent and pose-to-latent domains (e.g. symmetric poses should be close, named as the \"positive non-local coupling\" by the author). The loss from the source data (D^s) would serve as the extra constraint to make sure the two domains align. As long as this is the first time to propose this type of the human-pose-specific rules of latent space (I have not checked existing literature thoroughly), I would be quite positive on this paper. The experimental part provides fairly good results (while a notch below very strong visual results, see below weakness part).\n\nWeaknesses:\n\n1. I have moderate concerns to the experimental part, as the visual results and comparisons are not particularly exciting. Comparisons to several SoTA seems missing (e.g. Frank [a] and SMPL-X [b]) although these are not self-supervised or domain adaptation approaches. I wish the authors can also provide prediction overlay and their rotated 3D visual results. The current given visualization seems quite difficult to judge the accuracy. Besides, since one of the major domain is the video and pose sequence, I hope the authors can provide results for several human video clips.\n\n2. The writing of the paper seems not smooth to me, and I need to take efforts to reach the spark of the paper. It seems the introduction part did not well express the core novelty of the proposed framework. I would suggest the authors add one small illustrative figures in the intro part, particularly illustrating the non-local positive coupling is available in both the image-to-latent and the pose-to-latent mappings. I hope that can help the readers to get the message very quickly. \n\n3. Small issue: It seems the term \"non-local\" used in this paper is irrelevant to that in [c]. I wish the authors can either provide a clarification in the related works section, or use a slightly different name. \n\n[a] Joo, Hanbyul, Tomas Simon, and Yaser Sheikh. \"Total capture: A 3d deformation model for tracking faces, hands, and bodies.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8320-8329. 2018.\n\n[b] Pavlakos, Georgios, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AA Osman, Dimitrios Tzionas, and Michael J. Black. \"Expressive body capture: 3d hands, face, and body from a single image.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10975-10985. 2019.\n\n[c] Wang, Xiaolong, Ross Girshick, Abhinav Gupta, and Kaiming He. \"Non-local neural networks.\" In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803. 2018.\n\nConsidering the good novelty and the fair experimental results of the paper, I rate this work as \"borderline accept\".\n\n** Post Rebuttal ** After reading all the reviews and the authors' responses, I believe this paper provided innovated insights, while all the reviewers mentioned the clarity concern of the paper. I would keep my original rating considering all these factors.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed them well in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "0SO6rYzQbx", "reviewer": "Reviewer_vt6r", "summary": "This work regards monocular 3D human pose estimation as a domain adaptation problem to solve the problem of insufficient outdoor training data. The framework is composed of an image-to-latent encoder, which is pre-trained on the source domain and adapted to the target domain using unlabeled videos, and a latent-to-pose decoder, which is trained on unpaired 3D poses. During the encoder adaptation, non-local relations are utilized to improve the performance, which is the main contribution.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nNon-local relations are involved in the domain adaptation, which is novel and proved to be effective by experiments.\nAchieve SOTA results on several datasets under the semi-supervised and unsupervised settings.\nWell organized and easy to follow.\n\nCons:\nThere is an important hypothesis in this paper: Large embedding distance of non-local motion relations is beneficial to the domain adaptation. However, this hypothesis is not carefully verified. An ablation study with different embedding distances is needed, which can be achieved with in-plane rotation with different angles.\n\nWhen compared with other methods, the training data is different, which makes the comparison unfair.\n\nIn the experiment, larger rotation angle results in a larger embedding distance, so is it possible to achieve a larger embedding distance than flipping by increasing rotation angle or combining flipping and rotation to a new motion? Will this further improve the performance?\n\nSome Typos:\nIn table5, the metric should be PA-MPJPE rather than MPJPE. And we recommend showing both MPJPE and PA-MPJPE.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. Both limitation and societal impact is addressed in this paper.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical issues found.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of 3D human pose estimation by combining information from labeled sources and unlabeled data in the wild. The paper received mixed reviews, in the beginning, tending near borderline. The reviewers' major concern was regarding the presentation clarity. Multiple reviewers found it hard to follow. The authors provided a rebuttal that addressed some of the reviewers' concerns and promise to improve the presentation. The paper was discussed and most reviewers responded to the rebuttal. One of the reviewers raised their rating but one reviewer did not. The main complaint is still around the writing and presentation. AC agrees with the reviewers that the paper has a good contribution to be accepted and urges the authors to look at the reviewers' feedback, incorporate their comments and clarify the writing as promised in the camera-ready.
Decision: Accept (Poster)