Optimising 2D Pose Representation: Improving Accuracy, Stability and Generalisability inUnsupervised 2D-3D Human Pose Estimation
Authors: Peter Timothy David Hardy
Srinandan Dasmahapatra
Hansung Kim
Venue: Submitted to ICLR 2023 (2023)
Keywords: Unsupervised Learning
3D Human Pose Estimation
Data Representation
Adversarial Learning

Abstract: This paper addresses the problem of 2D pose representation during unsupervised 2D to 3D pose lifting to improve the accuracy, stability and generalisability of 3D human pose estimation (HPE) models. All unsupervised 2D-3D HPE approaches provide the entire 2D kinematic skeleton to a model during training. We argue that this is sub-optimal and disruptive as long-range correlations are induced between independent 2D key points and predicted 3D ordinates during training. To this end, we conduct the following study. With a maximum architecture capacity of 6 residual blocks, we evaluate the performance of 5 models which each represent a 2D pose differently during the adversarial unsupervised 2D-3D HPE process. Additionally, we show the correlations between 2D key points which are learned during the training process, highlighting the unintuitive correlations induced when an entire 2D pose is provided to a lifting model. Our results show that the most optimal representation of a 2D pose is that of two independent segments, the torso and legs, with no shared features between each lifting network. This approach decreased the average error by 20% on the Human3.6M dataset when compared to a model with a near identical parameter count trained on the entire 2D kinematic skeleton. Furthermore, due to the complex nature of adversarial learning, we show how this representation can also improve convergence during training allowing for an optimum result to be obtained more often.

Under review as a conference paper at ICLR 2023
OPTIMISING 2D POSE REPRESENTATION: IMPROVING ACCURACY, STABILITY AND GENERALISABILITY IN UNSUPERVISED 2D-3D HUMAN POSE ESTIMATION
Anonymous authors Paper under double-bling review
ABSTRACT
This paper addresses the problem of 2D pose representation during unsupervised 2D to 3D pose lifting to improve the accuracy, stability and generalisability of 3D human pose estimation (HPE) models. All unsupervised 2D-3D HPE approaches provide the entire 2D kinematic skeleton to a model during training. We argue that this is sub-optimal and disruptive long-ramn rongerange correlations are induced between independent 2D key points and predicted 3D) ordinates during training. To this end, conduct the following study. With maximum architecture capacity of 6 residual blocks, we evaluate the performance of 5 models which each represent 2D pose differently during the adversarial unsupervised 2D-3D HPE process. Additionally, we show the correlations between 2D key points which are learned during the training process, highlighting the unintuitive correlations induced when an entire 2D pose is provided to a lifting model. Our results show that the most optimal representation of 2D pose that of two independent segments, the torso and legs, with shared features between each lifting network. This approach decreased the average error by 20% on the Human3.66 66M dataset when compared to model with a near identical parameter count trained on the entire 2D kinematic skeleton. Furthermore, due to the complex nature of adversarial learning, we show how this representation can also improve convergence during training allowing for an optimum result to be obtained more often. Code and weights will be made available
INTRODUCTION
Monocular 3D human pose estimation (HPE) aims to reconstruct the 3D skeleton of the human body from 2D images or video. This is known to be an ill-posed inverse problem as multiple different 2D poses can correspond to the same 3D pose. Even with this hurdle, deep learning has allowed for accurate 2D-3D pose regression mappings to be learned allowing for remarkable results when trained and tested on 3D pose datasets (Wandt & Rosenhahn 2019: Martinez et al., 2017 Pavlakos) al. 2018 Yang al. 2018 Cheng al. Pavilo al., 2019p. Unfortunately, the difficulty of obtaining 3D datasets leads to poor performance when evaluating in domains where rigid environ- mental constraints (lighting, action, camera location, etc) are unable to controlled. Recent work (Chen et al. 2019 Wandt et al., 2022: Drover et al.) 2019 Yu et al. 202I) has investigated if an un- supervised solution for 3D HPE is possible. These approaches utilize geometric self-supervision cycle through random rotations to create a consistent lifting network, with some form of pose or probability discriminator to see if the rotated pose, once reprojected back to 2D, is realistic. As 2D data is cheaper to obtain, more efficient for computations and readily available in many circum- stances, improving the performance of unsupervised 2D-3D HPE networks would therefore allow for accurate 3D poses to be obtained in many unconstrained scenarios.
An overlooked aspect in prior work however, is the representation of the 2D pose being given to the lifting model. We posit that when full 2D pose is provided to a lifting model during training, long-range correlations are induced between a key points 3D prediction and all of the poses' other 2D key point coordinates (i.e. the 3D prediction of the left elbow will be influenced by the 2D coordinate of the right ankle). Although supervised approaches have touched upon this topic by learning joint dependency through graph convolutional networks (GCN) (Lee et al. 2018
Zhao

---

Under review as a conference paper at ICLR 2023
al. 2019), relationships between joints via relational networks Park & Kwak 2018 splitting and recombining the limbs of pose ZZng et al. 2020). To our best knowledge, this has never been done in unsupervised setting. and it has never been assumed that the pose could be two or more independent structures with no inter-relational correspondence needed. Additionally, the large variations in network architecture and optimisation within prior work mean we unable fairly compare the results between approaches to find an optimum representation.
We address this problem by training 5 models with near identical amounts of parameters and iden- tical training approaches on different 2D pose representations. By evaluating the results obtained from each model we will be able to determine an optimum representation that future work can use to obtain the best performance. We also show the correlations induced between 2D key points during training when a full pose is provided to a model as well as our best performing 2D representation model, providing some intuition behind the improved performance. To summarise our paper makes the following contributions:
We show the effect when using different 2D pose representations for the unsupervised adversarial 2D-3D HPE process, where by changing the 2D pose representation we can reduce the average error by 20%.
Our findings can be easily implemented within current the state the art as our approach
utilizes the popular residual block introduced by Martinez (Martinez et al., 2017) and used
within (Chen et al. 2019; Wandt et al. 2022 Wandt & Rosenhahn 2019 Drover et al,
2019 Yu et al. 20211.
We show the correlations induced between key points for full 2D pose representation model and our hest 2D pose representation model highlighting the sub-optimal learning when full 2D pose is provided to a network.
We show the adversarial stability of our best pose representation model against full 2D pose representation model, highlighting that the improvement is consistent across multiple random initializations.
2 RELATED WORK
There currently exists two main avenues of deep-learning research for 3D HPE. The first learns the mapping of 3D joints directly from a 2D image (Pavlakos al.) 2017; Lie al., 2019; Mehta al. 2017a Li et al. 2015 Tome et al., 2017). The second builds upon an accurate intermediate 2D pose estimate, with the 2D pose obtained from an image through techniques such as Stacked Hourglll Architectures (Newell et al., (016) or Part Affinity Fields (Cao et al.) (021). and lifts this 2D pose to 3D. This work focuses on the latter 2D to 3D lifting avenue which can be organized into the following categories:
2.1 FULLY SUPERVISED
Fully supervised approaches seek to learn mappings from paired 2D-3D data which contain ground truth 2D locations of key points and their corresponding 3D coordinates. Martinez et al. (2017)
introduced baseline fully connected regression model which learned 3D coordinates from their relative 2D locations. Exemplar approaches such as Chen & Ramanan (2017) and Yang et al. (2019) use large dictionaries/databasee of 3D poses with a -eighbuur search determine an optimal 3D pose. [Pavllo et al.) (209)) used temporal convolutions over 2D key points to predict the pose of the central or end frame in a time series, whereas Mehta et al. (20176) utilized multi-task learning to combine convolutional pose regressor with kinematic skeleton fitting for real-time 3D HPE.
2.2 WEAKLY-SUPERVISH
Seakly-Supervieed approaches do not use explicit 2D-3D correspondences and instead use either augmented 3D data during training or unpaired 2D-3D data to learn human body priors (shape or articulation). Pavlakos et al. (2018) and Ronchi al. 2018)) proposed the learning of 3D poses from 2D with ordinal depth relationships between key points (e.g. the right wrist is behind the right elbow). Wandt & Rosenhahn (2019) introduced weakly-supervised adversarial approach where
2

---

Under review as a conference paper at ICLR 2023
they transformed their predicted and ground truth 3D poses into kinematic chain (Wandt et al. 2018) before being seen by a Wasserstein critic network (Gulrajani et al., 2017). Yang et al. (2018) lifted 2D poses from in the wild scenes where no ground truth data was available with critic network that compared these predictions against ground-truth 3D skeletons. By contrast, /hou et al. (2017) did not use a critic network but transfer learning that used mixed 21) and 31) labels in unified network to predict 3D poses in scenarios where no 3D data is available. Drover et al. (2019) investigated if 3D poses can be learned through 2D self-consistency alone, where they would rotate predicted 3D pose and reproject it back into 2D before passing it back to the model for comparison. This led to the discovery that 2D critic network was also needed and self-consistency alone is not sufficient.
2.3 UNSUPERVISED
Unsupervised approaches do not utilize any 3D data during training, unpaired or otherwise. Kudo et al. (2018) introduced one of the first unsupervised adversarial networks which utilized random reprojections and a 2D critic network, under the assumption that any predicted 3D pose once rotated and reprojected should still produce a believable 2D pose. [Cen et al. (2019) expanded this work and that of Drover et al. (2019) by introducing an unsupervised adversarial approach with a self- consistency cycle. They also provided ablation studies highlighting a 7% improvement when using temporal cues during training. Yu et al. (202I) built upon Chen et al. (2019) highlighting that these temporal constraints may hinder models performance due to balancing multiple training objectives simultaneously and proposed splitting the problem into both a lifting and scale estimation module.
2.4 PRIOR WORK ON 2D POSE REPRESENTATIONS
Park & Kwak (0188) introduced one of the first studies that investigated if the human pose could be constructed of multiple interrelated structures. By utilizing relational networks, which were passed limb pairs during training (lleft arm and right arm], [left arm and right legl. etc.), they aimed to discover shared feature information between limbs that would aid in the 2D-3D lifting process. [eee] let al. (2018) utilized a propagating LSTM architecture where each LSTM would sequentially add another key point to its input. This intrinsic approach allowed joint relationships to be learntó over time each key point was added. Diang (2010) worked
exemplar approach where their 3D) dictionary was split into torso and legs key points during lookup. Though their idea behind this was to speed up the nearest-neighhour search process, they may have improved the generalisability of their model as it could find similar legs in one training example and similar torsos within another, increasing the number of possible library combinations. Zeng al. (2020) investigated splitting a pose into different parts to learn localized correlations between key points first then global corre- lations between these localized segments afterward. Unlike our study however, Zeng et al. (2020) is supervised network that still assumes that entire 2D pose is one independent structure via feature sharing between localized groups throughout their network. Our study utilizes completely unsupervised networks, where two of the pose representations investigate are independent with the idea that localized key-point groups could be unrelated to one another.
3 METHODOLOGY
In this section we describe the different 2D representations used and the intuition behind each rep- resentation. We then explain the adversarial learning approach of our various models.
3.1 2D POSE REPRESENTATIONS
I...N, with the root key point, the midpoint between the left and right hip joint, being located at the origin (0, Because of the fundamental scale ambiguity associated with monocular images, it is impossible to infer absolute depth from a single view alone Nishimura et al. (220). Therefore, we used max- nax-oarmaiisation 2D poses to scale their 2D key point coordinates between -1 and I. In our study we investigate the effects of using the following five 2D pose representations for the unsupervised 2D)3D HPE process:
yi). Our 2D poses consist of N key points (I,,
each of our
3

---

Under review as a conference paper at ICLR 2023
Full 2D Pose Representation: The standard approach where model is taught to predict the 3D ordinates from the entire 2D pose. The theory being that a good model should learn to correlate co-dependent key points and disassociate independent ones. In this scenario, we will have one network take an entire 2D pose as input and predict the 3D ordinates for the full pose.
Split and Recombine Leg and Torso key points: A concept introduced by Zeng et al. (2020), the idea behind this representation is to first learn localized key point correlations, followed by the global context of a pose. For this representation, two separate networks will first learn the localized correlations between the leg and torso key points and output a vector of features learned. A further network will learn to combine these features and predict the 3D ordinates for the full pose.
Independent Leg and Torso key points: In many actions (e.g. waving, eating, sitting etc) the legs and torso of a person can be more independent than co-dependent. Therefore another 2D pose representation will be that of completely independent torso and leg key points. For this representation we will have two independent lifting networks, one that accepts the 2D key points and predicts the 3D ordinates for the torso, and the other the 2D) key points and 3D ordinates of the legs.
Split and Recombine 5 Limbs key points: As humans, we are able to move each limb independently of one another. For this representation, separate networks will learn the localized correlations of each limbs (left arm, right arm, torso, right leg and left leg) key points and output a vector of features learned. A further network will learn to combine these features and predict the 3D ordinates for the full pose.
Independent 5 Limbs key points: Similar to the Independent leg and torso representation, this approach will look at the possibility of treating the entire pose as 5 independent limbs. For this representation, 5 separate networks will each take input and predict the 3D) ordinates for each limb independently (i.e. one network will take 3 key points as input belonging to the right arm and predict the 3D ordinates for the right arm).
3.2 LIFTING NETWORK
The lifting networks (G) used are fully connected neural networks whose architecture was based on Martinez et al.] (017)) and can be seen in Figure 3 in Appendix A These predict one 3D ordinate
relative to the root for each 2D key point provided:
G(x, y; y;
where (x, y) are the 2D coordinates provided to the network, à are the predicted 3D ordinate for each key point and w are the weights of our model learned during training. Once our lifting networks had made their predictions they are concatenated with the original 2D key points to create our final predicted 3D pose (x. y, 2). A more detailed architecture explanation of each pose representation model can be seen in Appendix
3.3 REPROJECTION CONSISTENCY
Similar to prior work (Chen et al. 2019 Drover et al. 2019 Yu et al. 2021). we utilize a self- consistency cycle through random 3D rotations to reproject our predicted 3D poses to new synthetic 2D viewpoints. Let Y E N be our full 2D pose. Once a prediction G(Y) is made and a full 3D pose (x y. 2) obtained, random rotation matrix R will be created by uniformly sampling an azimuth angle between | 8T 8x and an elevation angle between I [= _1 The predicted 3D pose will be rotated by this matrix and reprojected back to 2D via projection P. obtaining new synthetic viewpoint of the pose and the matrix Y E N where Y = PR[(x, y, 2)). To enable our model to learn consistency, we now pass Y the same lifting network, perform the inverse rotation R-1 on the newly predicted 3D pose (x, y, ?) and reproject it back into 2D, we should obtain our original matrix of 2D key points Y. This cycle allows our lifting networks to learn self-consistency during training where they seek to minimize the following components in the loss function:
9
C2D = -PR--- -R-PRR (R-'(G(())]I
(2)
4

---

Under review as a conference paper at ICLR 2023
In the case of our independent leg and torso and independent 5 limb 2D pose approaches, each net- work will receive its own L2D loss based on the error between the key points that they predicted. As an example, part of the Lan loss for the right arm lifter within our independent 5 limbs represen- tation would include the difference between the original 2D key point coordinate of the right wrist, and its 2D coordinate once Y was inversely rotated and reprojected. This error is not included in the CDD loss for the left arm lifter it does not predict the 3D ordinate for this key point.
3.4 90 DEGREE CONSISTENCY
During our study, we found that increasing self-consistens was key to reducing the evaluation error (see Appendix [B. Therefore, we introduce new self-consistency constraints during training based on rotations around the y/ axis at 90" increments. Let (x, y, 2) be the predicted 3D pose from our model. If we assume a fixed camera position and rotate our pose 90", then the depth component of pose (2) prior to rotation will now lie on the z axis from are normalised between -1 and 1, a 90 clockwise rotation of the 3D pose (x. 2) will produce the pose y, x). Therefore, providing y2. input to our generators should result in as its predictions. This fact allows for the inclusion of three additional consistency constraints the loss function of our generators which teach consistency at 90" clockwise rotation, 90% anticlockwise rotation and 180" rotation, which are as follows:
cameras viewpoint. As poses
||G(2.y: y: w) + x|| =0
(3)
|IG(-2,; x1|2
(4)
y|G(x,;;w) G(-x.y;
(5)
The left parts of these constraints are summed in the final loss function to produce Looe and included in the optimisation function of our models. Similar to Lnp in the case of our independent 2D pose representation networks, each lifting network will have its own Log depending points that it predicted for. Although we could include three similar constraints for 90" rotational increments around the axis, we found that these hinder the performance of the model. This is due to 90 x axis rotations producing a birds eye and ground-up up view of a 2D pose, which contains little variation between 2D key points.
the 2D key
3.5 DISCRIMINATOR Loss
Although self-consistency is important, alone it is insufficient for generating realistic 3D skeletons (Chen et al.) 2019). Therefore, we utilize a 2D discriminator D, that takes as input a 2D pose and outputs probability value of that pose being plausible or unrealistic. The architecture of our discriminator a fully connected neural network made of 3 residual blocks (Martinez et al. 2017) and ending with linear prediction function. It learns to discriminate between the real 2D poses within our data Y. and our reprojected 2D pose Y. This provides feedback to the generators during training, enabling the learning of geometric priors such as joint angles and limb length ratios. Our discriminator D utilized the least squares loss (Mao et al. 2017) as found it performed better than the standard adversarial loss during our experiments:
min E[D(( - -1]3 E(D
min E(D(Y)- - 1]²
Unlike our consistency constraints, we do not provide a unique version of Ladv to each lifting network in our independent 2D pose representation networks and instead provide the same loss (with a different weight) to each network. This is due to two reasons; Firstly, although our independent lifting networks are being trained separately, we still want them to produce believable reprojected 2D pose together and having one discriminator see the entire pose will provide this feedback during training. Secondly, poor feedback during training. An example being that 2D legs and arms are normally represented as straight or bent lines, making it hard to discriminate between plausible and implausible.
found that trying to discriminate between segments of 2D pose provided
5

---

Under review as a conference paper at ICLR 2023
3.6 TRAINING
As discussed, our lifting networks were trained adversarially with a random reprojection and 90 consistency constraints. The network parameters are then updated to optimise the total loss for each lifting model given by:
== w
w2C2D Wy Wy
(7)
where, W1, W2 and wa are the relative weights for the adversarial loss component, reprojection con- sisteney loss component and 90º consistency loss component respectively and defined numerically in Appendix A We trained
models unsupervised following (Chen et al.) 2019) using batch size of 8192 and the Adam optimiser (Kingma & Ba 2015) with a learning rate 0.0002. Our experiments use N 16 key points. During discriminator training employ label flipping with 10% chance.
4 EVALUATION AND RESULTS
Here we compare and evaluate the performance of our different 2D pose representation models against each other and various state-of-the-art models. Though our aim is not to beat the current state of the art, we believe that comparing against current models is important to examine how just a simple change in 2D pose representation adjustments. Our results show that not only can an optimum 2D pose representation improve upon the default full pose representation by 20%, but it can also improve generalisability unseen poses, even when compared against supervised and weakly-supervised approaches.
compare against the improvement from more complex
4.1 QUANTITATIVE RESULTS ON HUMAN3.66
Human3.66 6M (H36M) (Ionescu al. 2014) one of the largest and most widely used 3D human pose datasets, containing 3.6 million 3D human poses. It consists of both video and motion capture (MoCap) data from viewpoints of female and male subjects performing specific actions (e.g- talking on the phone, taking photo, cating, etc.). There are two main evaluation protocols for the H36M dataset, which use subjects 1, 5, 6. and for training and subject 9 and 11 for evaluation. Both protocols report the Mean Per Joint Position Error (MPJPE), which is the Euclidean distance in millimeters between the predicted and ground truth 3D coordinates. We report the protocol-II per- formance of model on all frames of the validation which employs rigid alignment between the ground truth and predicted pose before evaluation. The results of our different 2D pose repre- sentation models can be seen in Table From our results, we can see that representing a 2D pose as an independent torso and legs improves performance the most, as this representation achieved 20% lower average error than our full 2D pose network. Additionally, our split-combine leg and torso and split-combine 5 limbs representation also showed improved performance, decreasing the average error by 6% and 9% respectively. Our independent limbs model did not improve results which may be due to this representations models using 3 or 4 key points to predict 3D ordinates which may not contain enough information to learn the task at hand. By looking at our leg and it additionally performed well against several fully supervised models. In
torso results we fact. by using this representation we achieved the second highest performance in the Discussing, Posing, Smoking and Walking action. Although improving the Discussing, Posing and Smoking action seems intuitive, as we would assume the arms and legs of a person to be independent during these actions, it is surprising that achieved such high results in walking we would expect to see larger levels of co-dependence between the torso and legs during this movement.
4.2 QUANTITATIVE RESULTS ON MPI-INF-3DHP
MPI-INF-3DHP (Mehta et al. 7a) is markerless MoCap dataset containing the 3D human poses of 8 actors performing 8 different activities. We use this dataset to highlight the effect that 2D pose representations have on a model's generalisability to unseen poses when trained on the H36M dataset. The evaluation metrics used are the percentage of correctly positioned key points (PCK3D) and area under the curve (AUC) as defined in (Mehta et al. 2017a). As our poses are normalised, upscale our 3D poses by their original 2D normalizing factor before evaluation. Our results can be seen in Table Similar to our H36M results, all of our 2D representation approaches, aside from our independent 5 limbs representation, performed better unseen poses when compared full

---

Under review as a conference paper at ICLR 2023
Table I: The reconstruction error (MPJPE) the H36M validation set. Legend: (+) denotes extra data during training. (GT) denotes providing 2D ground truth key points to network for prediction. (T) denotes the use temporal information, all compairson results are taking from their respective papers, lower is better, best in bold, second best underlined.
Method
Appronch
Direct.
Discuss
Eat
Grect
Phons
Photo
Posing
Purchase
Mariner al 2017 (GT)
Supervised
39,5
43.2
46.4
47.0
51.0
56.0
41.4
40.6
ea
Supervised
38.7
38.0
45.9
Careel 2019 (GT)
Supervised
36.8
38.7
38.2
41.7
40.7
46.8
37.9
35.6
12l & di 20r9 (+)
Weakly- Supervised
26,9
30.9
36.3
39.9
43.9
47.4
28.8
29.4
(GT)
Weakly- Supervviid
33.6
38.8
32.6
37.5
36.0
Paviakss et au 2013 (+)
Weakly- Supell--p
34.7
39.8
41.8
38.6
42.5
47.5
38.0
36.6
Chen dr 201 (TT)
Unsupervised
(())())
Unsupervised
Full Posc Network (QTr))(T)
Unsupervised
51.5
47.1
48.1
53.7
47.6
56.2
45.9
46.1
Split Combine Leg und Terso Newwrr (GT)
Unsupervised
45.3
44.8
43.4
47.2
46.2
48.2
43.2
41.5
Independent
Torso Network (Ou)))GT)
Lasupervised
38.1
38.7
36.8
41.9
38.8
Split-Combing 5 Limks Network (OursKGT)
Unsupervised
43.6
42.3
43.4
45.9
45.1
49.9
11.5
42.0
Independent 5 Limbs Network (Ours)[CT)
Unsupervised
90.0
98.1
109.4
94.5
100.3
109.3
74.0
105.5
Method
Approach
Sit
SitD.
Smoke
Wait
Walk
WalkD.
WalkT.
Avg.
Martinez al. 2017 (GT)
Supervised
56.5
47.7
Pavili di 2019 (GT)
Supervised
46.8
53.4
41.4
36.9
43.1
30.3
34.8
40.0
(GT)
Supervised
47.6
51.7
41.3
36.8
42.7
31.0
34.7
40.2
Yang et al 2018 (+)
Weakly Supervised
36.9
58.4
41.5
30.5
29.5
42.5
32.2
37.7
Whimi & Rosentshn 2019 (GT)
Seuukly-Supervieel
19.2
52.0
37.5
39.8
34,1
40.3
34,9
38.2
Paviakos
Weakly Supervised
50.7
41.8
Chea ela.. 2019 (GT))
Unsupervised
51.0
2021 (TXXT)
Unsupervised
42.0
Full 2D Pose Network (Qur))GT)
Unsupervised
55.4
63.9
47.0
47.5
46.5
50.7
46.6
50.0
Split- Coii--Cmmbine Leg and Torso Network (GT))
Unsupervised
55.3
66.9
45.2
43.5
39.8
47.4
43.1
46.8
Independent Leg and Torso Network (OursXGT)
Unsupervised
47.4
60.6
40.6
38.0
34.0
46.0
37.1
41.7
Split-Combine Limbs Network (GT)
Unsupervised
50.1
63.4
43.
42.6
42.6
46.9
42.4
45.5
Independent Limbs Network (GTr))GT)
Unsupervised
123.1
102.2
Table 2: Results the MPI-INE-3DHP dataset. Legend: (3DHP) denotes model being trained on the MPI-INF-3DHP dataset. (H36M) denotes a model being trained on the Human3.6M dataset. (+) denotes additional training data. (*) uses transfer learning during from 2Dposenet. (T) denotes the use of temporal information during training, all comparison results are taking from their respective papers, higher better, best in bold, second best underlined.
Method
Approach
PCK3D
AUC
Mehta et ml
2017n (3DHPXH3GM)()
Supervised
76.5
40,8
Zeng
(H36M)
Supervised
77.6
43.8
et
2018 (H26M1(+)
Weakly-Supervised
69.0
32.0
Koseershn 2019 (H36M)
Weakly-Superviee
81.8
54.8
Kanazawa el al. 2018 (3T)
Weakly Supervised
77.1
40.7
Clen 2019 (3DHPX]]
Unsupervised
71.1
36.3
Kunda el al. 2022 (1136M)
Unsupervised
76.5
39.8
2021 (H36MXT)
Unsupervised
82.2
46.6
Full 2D Pose Network (Hun))(Ours)
Unsupervised
75.4
45.8
Splii- Combine and Torso Network (Oun)))(Ors)
Unsupervised
76.1
47.5
Independent Leg and Tarso Network (H33MM)Ours)
Unsupervised
78.5
48.5
Spli- Combine 5 Limbs Network (Oun)))Ouns)
Unsupervised
77.7
48.2
Independent 5 Limbs Network (Hur))
Unsupervised
47.2
25.6
2D pose representation approach. Our independent leg and torso representation improved results the most with a 4% increase in PCK3D and 6% increase in AUC when compared to the full 2D pose network. Additionally, this representation achieved the second highest overall AUC metric, even compared to other supervised and weakly approaches.
REDUCING UNINTUITIVE KEY POINT CORRELATIONS
Our intuition behind the improved results when using an independent leg and torso representation when compared to the full 2D pose representation, is due model learning more intuitive cor- relations between 2D key points and 3D ordinates. To validate our assumption we performed the following additional study. By changing the 2D coordinate of particular key point, we observed how the 3D predictions of our models were affected. From this, we could then see any unintuitive correlations have been made between key points. For example, if changed the key point location
7

---

Under review as a conference paper at ICLR 2023
for the left knee and the largest change in 3D prediction was for the right shoulder, then say that the model has learned to correlate the 2D key point of the left knee with its 3D prediction of the right shoulder. To conduct this study each 2D key point within the H36M validation set was scaled in 1% increments between -5% and 5% of its original size. Fach key point was changed individually and we observed the change in prediction for each 3D ordinate caused by changing that particular key point. Figure | provides an example of the results from this study.
Solo Representation eeft Fibow Change
Salo Reprosentation Right Change
0.15
0.10
0.03
Change 2 gredicton
-0.00
-4.15
Leg and Torso Left Elbow Change
Leg and Torso Right Ankle Change
0.15
6.13
0.00
0.00
-0.15
Change in the elbow coordinats Location
Figure 1: Showing the effect that changing the left elbows 2D key point coordinates has the 3D ordinate prediction of the left elbow and right ankle for the full 2D pose representation model (top- left and top-right respectively), and the left elbow and right ankle for the independent leg and torso model (bottom-left and bottom-right respectively).
Examining Figure we can see that the full 2D pose representation model has learned to correlate any change in the 2D coordinate for the left elbow with the 3D ordinate prediction of both the right ankle and left elbow. This highlights the unintuitive correlations learned when using a full 2D pose representation to train the model, humans are able to move their left elbow and right ankle in- dependently. By contrast, we examine our independent leg and torso correlations, the correlation between the left elbow and right ankle has been removed while maintaining the correlation between the left elbows 2D coordinate and 3D ordinate prediction. We believe that by removing these unintu- itive correlations we able to obtain are less sensitive to a change within one specific key point. The full results this study can be viewed within the supplementary material.
improved results
independent models predictions
IMPROVING ADVERSARIAL TRAINING STABILITY
One fundamental question when utilising unsupervised networks is when stop training. How- ever, we find a lack of information within prior work detailing how long authors have trained their models for. Therefore, we assume that prior work trained for set amount of epochs and picked the weights across these epochs that performed best on an validation set. Though fine from an evaluation viewpoint, in practice this would not work if we had no ground truth data. In truly un- supervised scenario there are three approaches we could use to decide when to stop training. Firstly, we could monitor the discriminators loss and stop training when it too weak or strong. Though there intuition for this approach, in practice strong discriminator vanishing gradients (Arjovsky & Bottou/ (2017) and a weak discriminator provides poor feedback to generator reducing its performance. Secondly, could visualise the predictions per epoch and
cause generator to fail due to
8

---

Under review as a conference paper at ICLR 2023
Leg and Torso Error
110
Uncertainty
Sala Generator Eraor
Solo Generator Uncertainty
90
Error (MPIPE)
80
50-
100 200
300
400
300
800
800
Epochs
Figure 2:
The average evaluation error (MPJPE) and uncertainty of leg and torso generators com- pared against 5 solo generators on the H36M validation dataset for each training epoch.
decide by eye which pose is the best. Though having potentially hundreds of epochs and thousands of poses, this is not an efficient solution. Lastly, and most realistically, we could pick the final weight during the training of our model or average the weights between certain range of epochs to use. For this scenario we show the stability of solo and 5 independent leg and torso generators during adversarial training which can be seen in Figure As can see, by having leg and torso generator training independently not only is the MPJPE lower, but all our models converge to similar minima during adversarial training. This is especially apparent around epoch 450 where our solo generators' errors begin to diverge away from one another. However, our leg and torso generators' error does not exhibit this same pattern and the errors are tighter to the mean during training following stable downwards curve. As these models were trained for 800 epochs, if we chose the last epochs weights evaluate on then the average error of the leg and torso generators' would have been 47.6 0 4.0mm and our solo generators' average error would have been 56.6 + 5.8mm. From epoch 500 to 800 the average error and standard deviation of our leg and torso generators' was 45.6 + 2.0mm, the average eтror and standard deviation of our solo generators' by comparison was 56.3 + 5.5mm. Taking the epoch where we observed the minimum eтror for each model would give us a result of 42.61 + 0.57mm for the leg and torso generators' and 51 51.55 + 2. 2.13mm for the solo generators'.
7 CONCLUSION
To conclude, we present a rigorous study investigating how different pose representations affects the unsupervised adversarial 2D-3D lifting process. Our results show that the most optimum pose representation for this process is that of an independent leg and torso which reduced the average error by 20%. In fact, by having two lifting networks learn independently, model. Furthermore, we have shown that when trained on
can improve both
the accuracy and generalisability full pose, unintuitive correlations are induced between 2D coordinates and 3D ordinate predictions which we believe leads to poorer results. Additionally, we have shown that the best pose repre- sentation model can be recreated with greater consistency as different instances converge to similar minima during training whereas full 2D pose representation approach diverges work within this area could look using attention mechanisms, which would allow the model to learn its own best representation for the 2D pose. We believe however, that for single frame scenar- ios it may fall victim to the same problems as our full 2D pose representation model, as our own experiments with selective attention have observed model simply learning to get by with the initial 2D representation provided rather than change its own input. However, the main statement from this paper that nearly
time. Future
our different pose representations improved upon the standard full pose representation, highlighting how a simple change to the input to your model can significantly improve your results.
REFERENCES
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. ArXiv, abs/1701.045s 778622 2017.

---

Under review as a conference paper at ICLR 2023
Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016. URL httpps:/
arxiv. wrxiv.org// .rrg abs/ 107. 06450)
Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, and Nadia Magnenat Thalmann. Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2272-2281, 2019. doi: 10.1 109/CCV. 2019.00236.
Cao, G. Hidalgo, T. Simon, Wei, and Y. Sheikh. Openpose: Realtime multi-person 2d pose estimation using part affinity fields. IEEE Trunsuctions on Pattern Analysis and Machine Intelligence, 43(01):172-186, jan 2021. ISSN 1939-3539. doi: 10. 101110999M2222...999999 2929277....
Ching-Hang Chen and Deva Ramanan. 3d human pose estimation 2d pose estimation + matching. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5759 17779-777.22 2017.
Ching- Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Rohith MV, Stefan Stojanov, and James M. Rehg, Unsupervised 3d pose estimation with geometrie self-superiision. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 5707-5717, 2019. doi: 10.1 1O09//VR..009.00S..
Yu Cheng, Bo Yang. Bo Wang, and Robby T. Tan. 3d human pose estimation using spatio-temporal networks with explicit occlusion training. Proceedings of the AAAI Conference on Artificial Intelligence, 34(07): 10631-10638, Apr. 2020. doi: 16409/aai..34447.6666
URL https://ojs. www.i....aaaaa.................. org/ index.
6689
Dylan Drover, Rohith M. V. Ching-Hang Chen, Amit Agrawal, Ambrish Tyagi. and Cong Phuoc Huynh. Can 3d pose be learned from projections alone? Laura Leal-Taixé and Stefan Roth (eds.), Computer Vision ECCV 2018 Workshops. pp. 78-94, Cham, 2019. Springer International Publishing. ISBN 978-3-030- 11018-5.
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky. Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vish- wanathan, and R. Garnett (eds.), Advances Neural Information Processing Systems, volume 30, Cur- ran Associates, Inc., 2017. URL https: http:::://prcooo pttp///tttpcccccccccc/pp/ www. http://..........cc//oo//oo./1..
19923blc6dccdd293366 2cbdd wodd22/dddd0ffccc Ff633d6 Paper.
Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3. Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325-1333 11399, jul 2014.
Hao Jiang. 3d human pose reconstruction using millions of exemplars. In 2010 20th International Conference on Pattern Recognition, pp. 1674-1677, 2010. doi: 10.1000/22.2000.4444
Angjoo Kanazawa, Michael Black, David W. Jacobs, and Jitendra Malik. End-to-end recovery human shape and pose. In 2018 JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7122- 7131, 2018. doi: 10. 201.9//CV..211..00777..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.699000 2015.
Yasunori Kudo, Keisuke Ogaki, Yusuke Matsui, and Yuri Odagiri. Unsupervised adversarial learning of 3d human pose from 2d joint locations, 2018.
Jogendra Nath Kundu, Siddharth Seth, Mayur Rahul, M. Rakesh, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. Einemmati-structur--preeer representation for unsupervised 3d human pose estimation. In AAAI, 2020.
Kyoungoh Lee, Inwoong Lee, and Sanghoon Lee. Propagating Istm:
pose estimation based on joint interde- pendency. In Vittorio Ferrari. Martial Hebert, Cristian Sminchisescu. and Yair Weiss (eds.), Computer Vision -ECCV 2018, pp. 141, Cham, 2018. Springer International Publishing. ISBN 978 978-3-030-01234-2.
Sijin Li, Weichen Zhang, and Antoni B. Chan. Maximun-margin structured learning with deep networks for 3d human pose estimation. International Journal of Computer Vision, 122: 12:1149-11 168, 2015.
Wen-Nung Lie, Guan-Har Lin, Lung- Sheng Shih, Yuling Hsu, Thang Huu Nguyen, and Quynh Nguyen Quang Nhu. Fully convolutional network for 3d human skeleton estimation from a single view for ac- tion analysis. In 20/9 IEFF International Conference on Multimedia Expo Workshops (ICMEW), pp. 1 1-6, 2019. doi: 10. W0W.C/9W.200999000 120..
10

---

Under review as a conference paper at ICLR 2023
X. Mao, Q. Li, H. Xie, R. K. Lau, Z. Wang, and S. Smolley. Least squares generative adversarial net- works, 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2813 2813-2821, Los Alami- tos, CA, USA, Oct 2017. IEEE Computer Society, doi: 10.1 109//CCV.2211.304 URL https://doo:
pccccomputorsoclety.ooo org 100 http//1...11//911 ICCV, 204
Julieta Martinez, Rayat Hossain, Javier Romero, and James J. Little. A simple yet effective baseline for 3d human pose estimation. In 2017 IEFF. International Conference on Computer Vision (ICCV), pp. 2659- 2668, 2017. doi: 10.1 109/1CCV2001..88.
Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko, Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in the wild using improved enn supervision. In Vision (3DV), 2017 Fifth International Conference on. IEEE, 2017a. doi: 10.1 10.110993d..2017.00066 URL kttp: 7/gvv mpi - inf mpg. de 3dap dataset
Dushyant Mehta. Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, Mohammad Shafiei. Hans-Peter Seidel, Weipeng Xu, Dan Casas, and Christian Theobalt. Vnect: Real-time 3d human pose estimation with single rgb camera. volume 36, 2017b. doi: 1114///3779999..7735666 URL/ttpp:
mpg de
Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In Bastian Leibe. Jiri Matas, Nicu Sebe, and Max Welling (eds.), Computer Vision - ECCV 2016, pp. 483-499, Cham, 2016. Springer International Publishing. ISBN 378-3-319-466484
Mark Nishimura, David B. Lindell, Christopher Metzler, and Gordon Wetzstein. Disanbiguating monocular depth estimation with single transient. Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 139-155, Cham, 2020. Springer International Publishing. ISBN 978-3-033--8558--1
Sungheon Park and Nojun Kwak. 3d human pose estimation with relational networks. In British Machine Vision Conference 2018, BMVC 2018, Neweastle. UK, September 3-6, 2018, pp. 129. BMVA Press, 2018.
/tRp::tt:///mm/c2000 org/ contents pttptt/s/pppee//50000
htts://otteett/pappe///333.. pdf
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G. Derpanis, and Kostas Danillidis. Coarse-to-line volumet- ric prediction for single-image human pose. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2017.
2017.
Georgios Pavlakos, Xiaowei Zhou, and Kostas Danillidis. Ordinal depth supervision for 3d human pose esti- mation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7307-7316, 2018. doi: 10. 109/CVPR. 2012201180000 8.00763.
Dario Pavilo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d human posc estimation in vidco with temporal convolutions and supervise training. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7745-7754, 2019. doi: 10.1109/CVP.2001..007
Mattco Ruggero Ronchi, Oisin Mac Aodha, Robert Eng, and Pietro Perona. It's all relative: Monoc- ular 3d human pose estimation from wcakly supervised data. In British Machine Vision Conference 2018, BMVC 2018, Northumbria University, Newcastle, UK, September 3-6, 2018, pp. 300, 2018. URL http:/ /tw/c//wcc....ccc.../..//./ org/ wwt.:/tsttttcccoooo/c wttpeeestttppppese/eeeeem wttpppppppppp....eeee.pppp pdE
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? In S. Bengio, H. Wallach, H. Larochelle, Grauman, Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Cur- ran Associates. Inc., 2018. URL
(0505eclac1dad1411550
D Tome, Christopher Russell, and L Agapito. Lifting from the deep: Convolutional 3d pose estimation from single image, 2017.
Bastian Wandt Bodo Rosenhahn. Repnet: Weakly supervised training of an adversarial reprojection net-
work for 3d human pose estimation. In Computer Vision and Pottern Recognition (CVPR), June 2019.
Bastian Wandt, Hanno Ackermann, and Bodo Rosenhahn. A kinematic chain space for monocular motion capture. In ECCV Workshops, 2018
Bastian Wandt, James J. Little, and Helge Rhodin. Elepose: Unsupervised 3d human pose estimation by predicting camera elevation and learning normalizing flows 2d poses. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), pp. 6635-6645, June 2022.

---

Under review as a conference paper at ICLR 2023
Jingjing Yang. Lili Wan, Wanru Xu, and Shenghui Wang. 3d human pose estimation from single im- age exemplar augmentation. Journal Visual Communication and Image Representation, 59:371- 379, 2019. ISSN 1047-3203. doi: https://dboor///ll..0011/vy..n.d scicncedirect. com/ htt.://cc/.ccccceetttttccc////// http:///ow..........///////////0
URL htLtps://ww.
Wei Yang, Wanli Ouyang, Wang, Jimmy Ren, Hongsheng Li, and Xiaogang Wang. 3d human pose estimation the wild by adversarial learning. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5255-5264, 2018.
Zhenbo Yu, Bingbing Ni, Jingwei Xu, Junjie Wang, Chenglong Zhao, and Wenjun Zhang. Towards alleviat- ing the modeling ambiguity of unsupervised monocular 3d human pose estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pp. 8651 8660, October 2021.
Ailing Zeng, Xiao Sun, Fuyang Huang, Minhao Liu, Qiang Xu, and Stephen Lin. Srnet: Improving generaliza- tion in 3d human pose estimation with a split-and-reoombine approach. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision ECCV 2020, pp. 507-523, Cham, 2020.
Springer International Publishing. ISBN 978-3-030-55566866
Long Zhao, Xi Peng, Yu Tian, Mubbasir Kapadia, and Dimitris N. Metaxas. Semantic graph convolutional networks for 3d human pose regression. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 3420-3430, 2019. doi: 10.1109/CVR.2201.00023
Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang and Yichen Wei. Towards human pose estimation in the wild: A scakly-supervised approach. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
A TRAINING APPROACH AND MODEL ARCHITECTURE FOR EACH 2D POSE REPRESENTATION
Here will highlight the overall architectures and the optimisation functions with their numeric weights and reasoning for each our 2D pose representation approaches. In Figure 3 you can the residual block architecture which was included within all of our 2D pose representation networks.
Safch Norta
Batch Nora
Rell
Re
Dropout
Leupost
Figure 3: The residual block used within all our representation networks inspired by Martinez al. (017)
A.1 FULL 2D POSE NETWORK
In Figure you can see the full architecture used for our full 2D pose representation network. As we have only one network, we used one optimisation function to update the network during training. This can be seen in equation where w1, Wl and U/ are 1, 10 and respectively.
W1 Lade WaLaD wa WgLo
(8)
12

---

Under review as a conference paper at ICLR 2023
A.2 SPLIT- LEG AND TORSO NETWORK
In Figure you can see the full architecture used for our split-combine leg and torso network. In this scenario the full 2D pose was initially split into the leg and torso key points. The leg key points consisted of the ankles, knees and hips. The torso key points consisted of the wrists, elbows, shoulders, spine, neck, head, and crown key point. These were passed through two separate networks containing two residual blocks and ending with linear layer which contained the local feature feature relationships between the key points within the leg and torso key points respectively. The number of neurons in the output layers relate to how many key points each network before it was provided to predict for. In other words the torso network provided features relating to 10 out of the 16 3D ordinates of the entire pose (where containing features we carried forward during concatenation was higher from this network to reflect this. Once the outputs were concatenated they were passed through an additional two residual blocks ending with a TanH function which predicted the all 3D ordinates of the pose. Similar to the full 2D pose representation, we have one network which predicts all 3D ordinates. Therefore we used one optimisation function and weights as Equation g
1024 640). Therefore the amount of neurons
A.3 INDEPENDENT LEG AND TORSO NETWORK
In FigureBlyon can see the full architecture used for our independent leg and torso network. this representation, similar to our split-combine network, we split up 2D pose into its respective leg and torso key points. These are then passed to two completely independent networks which predict the 3D ordinates solely for the key points they provided. Unlike the split-combine network each network within this process is trained with its own optimisation function which contain the loss representative of the keypoints it predicts for (see section (33) Therefore the optimisation functions and their respective weights for this approach was as follows:
Lugs = W1 Lode + W22 Cregs 2D + WyL 90 Ciors W. Ladv + W2 Crors 2D + W3 Ciorso 90>
where Wi, W2, Wy and W4 are 16. 10, 3 and is respectively. The discrepancy between wi and wa stems from the fact that the adversarial loss provide to each model was the same (see section 3.5) however as the torso network predicted T6 is higher to reflect that any incorrect pose more likely to be because of this network.
10 of the 3D ordinates for the full pose, its adversarial loss
A.4 SPLIT-COMBINE 5 LIMBS NETWORK
In Figure 7 you can see the architecture of our split-combine 5 limbs network. In this scenario the full 2D pose was initially split into 5 limbs comprising of the left arm, right arm, left leg, right leg and torso. Each limb contained 3 key point coordinates aside from the torso which contained 4. We first passed through each limb through its residual block which learnt local group features. The outputs of these network were passed through a linear layer. The number of neurons in each output linear layer corresponds how many key points that residual block had as input, so for each limb with 3 key points it has 192 neurons ( 2 x 1024) and for the torso as this has 4 key points it 256. These features are then concatenated and passed through a final residual block which learns how to combine these features, and finally all 3D ordinates are predicted via a TanH function. As we have one network which predicts the entire 3D ordinates of a pose, therefore we use the same optimisation function and weights as in Equation
A.5 INDEPENDENT LIMBS NETWORK
In Figure g you can see the architecture of our independent limbs network. In this scenario the full 2D pose was initially split into 5 limbs stated previously. Each limb was passed through its own residual block network which is followed by a TanH function which outputted the 3D ordinates belonging that specific limb. These outputs pose to obtain our full 3D pose. As we are training networks independently of one another they each receive their own specific optimisation function which contains the losses relative the key
concatenated and combined with the input 2D
13

---

Under review as a conference paper at ICLR 2023
points that they predict for (see section 3.3 For this network the 5 optimisation functions and weights were as follows:
liettleg W1 Ladv W2 log 2D + Wy Clett leg 90 Fright leg = w1 Lade + uuy Lrghht leg + Wyg fight leg
Liell urm Ladv + W2 Stea am 2D + W3 Cin amn 90 Light arm = wi Ladu + W2 Light 2D + Wy right
(10)
Luso W4Lade + Wyy Ciooos 2D Wy tooo yue
where Wi, W2, W3 and Wy are 16. 10, 3 and 4 and wa is due to the fact that the all limb networks predict 3 ordinates but the torso network predicts 4.
T6 respectively. The discrepancy between weights w1
B IMPROVING THE SELF-CCONSISENNC CYCLE
We found during our study that are able to achieve a better quantitative error by improving self- consistency even if this leads to a pose that is easy to discriminate against. This can be seen clearly in Table 3 where
noticeable decrease in MPJPE between the results in Drover et al. (2019) and our re-creation with the additional consistency constraints mentioned within our work.
Table 3: The results of Drover et al. (2019) and our recreation with our additional consistency constraints and changing the weight values the optimisation function increase the importance of self-consistency.
Model
Avg.
Drover et Drover et al. (2019)
38.2
Ours (Improved Consistency)
34.9
Because of this, we sought to replace the random rotation self-consistency cycle with something more efficient. This was due random rotation lending itself long training times, where the longer model is trained the more random rotations it will see and therefore the more consistent it will become. However, this could be problem as highlighted in Figure 2 where longer training period may lead to GAN instability. By contrast 90% consistency constraints allows for 3 speci- fied angles consistency to learned per training iteration, while also being more computational efficient then randomly rotation 3D object and reprojecting it. These by themselves however aren't 't sufficient lear self-consistency as the model only learns specific angles during training and in the wild many more viewpoints exist. We therefore sought an optimisation formula similar to 90% consistency that would satisfy all possible viewpoints around the axis. First let us determine the end position of the pose (x, y, 2;) after random 3D rotation R& along the y axis:
Tcos(B) (8)
0
sin
R
1
ssin (0)
0
cos(O)
(11)
Txcos(0
2sin(0)
R.
y
2
(s (0) + 2c)
we substitute these new positions within our network function (Equation)
x0) + 2c = G(xcos(0) (0)n y; w)
(12)
for 0 <<< 1:
x0 à G(x 20, y:
(13)
14

---

Under review as a conference paper at ICLR 2023
perform Taylor series expansion while ignoring terms of power 2 and above for the small angle 8:
0
x0 G(x,y; w)
20 G(x,y;w)
(14)
0x
cancel z with ((x, y; w) and remove 0:
x -2 G(x.y:
(15)
0x
this leaves us with our final consistency constraint for all angles:
8 y: -0 x + 2 0x
(16)
Although theoretically simple, in practice the above is difficult to implement and leads to lackluster factors; firstly the derivative component Jacobian matrix,
performance. This is due which to calculate numerically within current deep learning languages is computationally inefficient, requiring over 100 minutes to train one epoch on an RTX-8000 GPU. Secondly as we are finding the derivative with respect to the inputs, to maintain gradient independence all batch-norm layers have be removed from our model as these normalises across the batch dimension. This has the effect of lowering the rate at which our model learns and decreasing its stability while training training[Santurkar (Santurkar et al.) (2018. We are currently investigating if layer norm Ba et al. (2016) would be suitable replacement for batch norm this would retain gradient independence and allow the implementation of the above within future work.
Tot 10 Tow Ropstrs
Twcsh Ntmm
Text:
Tell Tree Trecketen
Figure 4: The architecture of our full pose lifting network
Butns Notes
Teetirl
Deriaal
Treer
Reil
Hosihal
Rodball
Block
Blics
304
Rell
Figure 5: The architecture of split-combine Leg and Torso Network
15

---

Under review as a conference paper at ICLR 2023
1804
Expaits
Narm
Reciiall
Recisal
Pradittoon
Fel New 20 Kesports
Lieae
20 lag Kepportt
lassh Kum
brill
Falli
Figure 6: The architecture of independent Leg and Torso Network
Lese
Seepetta
Jitro
Tor dov
Lseer
Lasa
Latt Aegueiro
Teir
Fmi
Liner
1004
Tileee
113
Bacc
Lef Am 20 Seppoias
Miro Nore
Tendeol
Liber
Theth Am 20 Seppetts
Tatn Norm
Liver
Ixto
Figure 7: The architecture of our split-combine 5 limbs Network
16

---

Under review as a conference paper at ICLR 2023
Deporers
Taart Nime
Bock
Tartl
Chert30 Freeetio
Lawe
Dkg Repputtta
Nore
Reskial
Led Dog Predktan
Laac
Seyposh
Right Kopennta
Nere
Reciital
Right Lay
B064
Lawe:
Regar km Exparats
Kew
Testbal
Rijit Prodiriess
Figure 8: The architecture of our 5 independent limbs Network
17


Reviews:
[{"review_id": "tJcgKV03_3", "reviewer": "Reviewer_p3BW", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper conducted investigation on how different 2D pose representations affects the unsupervised adversarial 2D-3D lifting process. Experiments are conducted on both Human3.6M dataset and MPI-INF-3DHP dataset. The results show that reducing the unintuitive key point correlations can reduce the average error. In addition, the paper explores the factors that affects the convergence and adversarial training stability. ", "summary_of_the_review": "The experiments are conducted on two datasets and are valuable as a case study. However, the improvement over the baselines looks marginal. Also, the effectiveness of each of the loss components is not very clear. I believe that addressing my points of weakness to make them more clear would largely benefit the paper.", "strength_and_weaknesses": "Strength:\n1. The author has validated the effectiveness of reducing unintuitive key point correlations. The model using an independent leg and torso representation performs better than using the full points. The proposed idea is very simple but effective.\n2. A new matrix, 90-degree consistency, is proposed. As far as I know, this matrix is novel.\n3. Detailed explanation of the model and experiment parameters, good for reproducing.\n\nWeakness\n1. My main concern is the performance compared to the state-of-the-art. In the results on the MPI-INF-3DHP in Table  2, the best one of the proposed models (leg and torso) has a PCK3D of 78.5, still less than the 82.2 of the model by Yu et al.\n2. Can the author provide an ablation study to show the effectiveness of each of the loss components, saying adversarial, reprojection consistency, and 90-degree consistency losses? It would be interesting to see the effectiveness of each one and the interaction between them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "qDc0hIWhU_", "reviewer": "Reviewer_rPD4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This proposes the unsupervised 2D-to-3D lifting approach by exprimenting with 5 network structures: 1) Popularly used 2D-to-3D lifting model, 2) extracting features from body and legs and estimate 3D full pose by concatenating 2 features, 3) Separately estimating poses for body and legs, 4) Separately estimate 5 part features (2 arms, 2 legs and body) and estimate 3D full pose using them, 5) Estimate 3D pose for 5 different parts. Further the consistency loss is proposed. Experimental results demonstrate the superiority of the proposed method.\n", "summary_of_the_review": " I think this paper is proposing some novel method; while the qualitative results and ablative studies are lacking. At this moment, I am currently at marginally below threshold for this paper; while I could change mine to the higher score if authors could rebut my points.\n", "strength_and_weaknesses": "Reprojection loss is novel and well motivated to refine the poses in the aspect of different camera viewpoints.\n\nMinor errata: “This intrinsic approach allowed joint relationships to be learnt0 over\ntime as each key point was added.” in page 3.\n\nTable3 in page 14 does not show how much it is improved in the aspect of the depth. I recommend authors to add this.\n\nI’m curious how much the variations on leg poses affect the depth values of the arms. I recommend authors to report any additional results that can explain this.\n\nThere is lack of qualitative results and ablative studies. I recommend adding these.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q6BfGmbWL6", "reviewer": "Reviewer_2qG2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "N/A", "summary_of_the_review": "N/A", "strength_and_weaknesses": "N/A", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "mS-TOFvefaa", "reviewer": "Reviewer_QAQe", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for the problem of 2D to 3D pose lifting. The authors propose to divide the all joints of a pose into several groups, and then use separate networks to lift each group of joints independently. A recombining network then takes as input the features of lifting networks and estimates the full 3D pose. The above grouping strategy is applied to the unsupervised 2D-to-3D pose lifting task. Experiments are implemented on two datasets, and the results show that the split-and-recombine method significantly outperforms the full pose lifting method using the same backbone model, and also performs better than unsupervised SOTAs. When trained on HUMAN3.6M and tested on MPI-INF-3DHP, the proposed method outperforms most of supervised and weakly-supervised competitors. ", "summary_of_the_review": "The proposed divide-and-conquer strategy achieves promising performance, but I expect the authors obtain more research achievement based on this observation. ", "strength_and_weaknesses": "Strength: \n- Experimental results are promising, which verify the effectiveness of the proposed split-and-recombine strategy. \n\nWeaknesses:\n- Novelty is relatively limited. Divide-and-conquer is a common approach in computer vision. It is interesting to observe that joint grouping can improve the accuracy significantly. But I expect more in this work. For example, the authors could investigate novel methods to decently encourage the interaction between joins belonging to one group and suppress the interaction between joins belonging to different groups, in order to improve the efficiency.\n- In my understanding, the proposed method should be applicable for all 2D-to-3D pose lifting methods. Why do the authors only focus on the unsupervised one? What about the experimental results on supervised and weakly-supervised methods?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6t8yujNaLuW", "reviewer": "Reviewer_cAex", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a study investigating the influence of different representations of 2D key points to regress 3D depth in an unsupervised fashion. The hypothesis studied is that when the whole 2D skeleton is input into a lifting network, the network extracts undesirable correlations between key points, which hurt performance. The most successful proposed model splits key points into two groups (torso and legs) and regresses depth individually for each group. ", "summary_of_the_review": "Overall, the paper empirically tests a hypothesis in a pretty rigorous fashion. Some of the statements have to be modified/weakened and the authors should consider testing their method with different state-of-the-art models to support their claims. I also recommend rethinking the paper layout a bit and putting the main focus on the main contributions.", "strength_and_weaknesses": "Strengths:\n* The paper designs a limited, small-scale study to evaluate a single design aspect of human pose regression networks in a controlled way. As such, the paper can vary factors individually and study the outcomes on reprojection error. \n* The paper is overall well-written and easy to understand. I have some points regarding improving readability/clarity below.\n* The paper studies the problem on two different datasets. The Human3.6M dataset is used to study the accuracy of the model on the training/testing dataset. The second dataset, MPI-INF-3DHP, is used to study generalization.\n\nWeaknesses:\n* The paper mentioned that their proposed input modality (split torso/leg groups) can be used as drop-in replacements to improve existing approaches that use full-body key points as input. However, there is no experiment in this paper that shows this to be true. In my opinion, this is an over-generalization and should be validated. As the paper tries to encourage the use of a different input form through a controlled study,  I think the paper could be made much stronger by showing evidence that the proposed method improves performance for one of the multiple state-of-the-art models. In my opinion, this should be rather easy to validate and will add a lot to the paper. \n* The paper mentions multiple times that the proposed 3D pose representation is \"optimal\", implying that there can be no other representation that is better. As there is no evidence to support this claim, please remove it. Instead, the paper could refer to it as: \"improved 2D representation\".\n* Figure 1: It is unclear to me what this figure shows. What are the colors representing? And how can the paper claim to see a clear correlation between the two selected joints? Explain more, please. \n* Section 6. The paper mentions that their paper is truly unsupervised and therefore it is not entirely clear when to stop training. However, the method does NEED ground truth 2D poses to train the GAN and therefore is not fully unsupervised and this argumentation is not well supported. If 2D ground truth poses are not available, how would you train your discriminator? If this is not possible, why not just use a portion of the data for validation? In my opinion, this space could be better used by showing that the proposed approach benefits existing methods too. \n\nMinor concerns:\n* Section 3.1: the paper mentions ordinates instead of coordinates. Maybe a search/replace error?\n* Table 2 caption: uses transfer learning during from...", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject