Using Motion Cues to Supervise Single-frame Body Pose & Shape Estimation in Low Data Regimes
Authors: Andrey Davydov
Alexey Sidnev
Artsiom Sanakoyeu
Yuhua Chen
Mathieu Salzmann
Pascal Fua
Venue: Accepted by TMLR ()

Abstract: When enough annotated training data is available, supervised deep-learning algorithms excel at estimating human body pose and shape using a single camera. The effects of too little such data being available can be mitigated by using other information sources, such as databases of body shapes, to learn priors. Unfortunately, such sources are not always available either. 
We show that, in such cases, easy to-obtain unannotated videos can be used instead to provide the required supervisory signals. Given a trained model using too little annotated data, we compute poses in consecutive frames along with the optical flow between them. We then enforce consistency between the image optical flow and the one that can be inferred from the change in pose from one frame to the next. This provides enough additional supervision to effectively refine the network weights and to perform on par with methods trained using far more annotated data.

Published in Transactions on Machine Learning Research (02/2024)
Using Motion Cues to Supervise Single-Frame Body Pose and Shape Estimation in Low Data Regimes
Andrey Davydov* CVLab, EPFL
Alexey Sidnev Meta AI
asidnev@@meta.cm
Artsiom Sanakoyeu
osanakoy@mmeta.com
Meta AI
Yuhua Chen
yubuac@metta.com
Meta AI
Mathieu Salzmann CVLab, EPFL
aachien.salmmann@eep...dd
Pascal Fua CVLab, EPFL
poscal fua @uapff...e @epfl.ch
Reviewed on OpenReview: https: //
// openreview. net/ forum? id- fUh0b14sQu
Abstract
When enough annotated training data is available, supervised deep-learning algorithms excel at estimating human body pose and shape using a single camera. The effects of too little such data being available can be mitigated by using other information sources, such as databases of body shapes, to learn priors. Unfortunately, such sources are not always available either. We show that, in such cases, easy-to-obtain unannotated videos can be used instead to provide the required supervisory signals. Given trained model using too little annotated data, compute poses in consecutive frames along with the optical flow between them. We then enforce consistency between the image optical flow and the one that can be inferred from the change in pose from one frame to the next. This provides enough additional supervision to effectively refine the network weights and to perform on par with methods trained using far more annotated data.
1 Introduction
Given enough annotated training data, supervised deep-learning algorithms for estimating human body shape and pose using single camera have bocome remarkably effective (Moon Lee
2020 Choi al. 2021 Wei et al. 2022). The very recent. transformer-basee) architecture of (Goel al.]
2023 embodies the current
state-of- tfate-of- the-art. It is pre-trained on 300 million images and fine-tuned most SMPL data sets in existence using a GAN discriminator. This makes it excellent foundation model but, unfortunately, such a large amount of data is not always forthcoming and requires massive amounts of computing power. Furthermore, even though the datasets are publicly available for research purposes, networks trained on them may not usable commercially for licensing or privacy reasons. Specific licenses often restrict the use of data for commerciil purpose; which drives companies lo either spend resources for their own data acquisition or
"This work supported in part by the Swiss National Science Foundation and done part during internship Meta
AL.

---

Published in Transactions on Machine Learning Research (02/2024)
B2
I[
I.
Training
unlabelled videos
pose&shape estimator
F F Vx2
V x pranie)
moterr)
C B, F Fo)
Optical Flow estimator
register to veriex
F OF xx2
MxWa x3
H W
single frames with GT
B
test frames
B
Inference
pose&shupe estimator
L(B, gtt)
pose& estimator
Figure 1: Method Overview. Self-supervised training with optical flow guidance (Top). Given pre-trained network that takes single frames as input, we compute SMPL body mesh estimates in consecutive frames along with the optical flow For between them. This lets us write loss term based on the consistency of For and of the flow Fa that can inferred from the changes in B across the frames. We then minimize a weighted sum of this loss and the one used to pre-train the network (Bottom left). At inference time (Bottom right), the refined body estimator can run on single images and does not require the optical flow anymore.
develop other approaches that require less supervision. Meanwhile, privacy issues might entail legal risks for further use of data, especially human poses that can be viewed as personal information.
To overcome these issues, there have been many attempts at obtaining good results with less supervision, such as imposing body pose and motion priors (Kanazawa et al.) 2018 Davydov al.) 2022; Kocabas al. 2020 Luo et al. 2020) creating synthetic training data (Sengupta et al. 2021 Hewitt et al. 2023) Most of these approaches to making the training more data-efficient require external sources of information, such as 2D/3D keypoints or silhouettes, in addition to the images themselves. In addition to the privacy and licensing issues ioned above, such targeted data acquisition can be labor intensive. This motivates our search for more data-cfficient techniques. In this paper, we advocate exploiting a widely available source of information. which has largely been overlooked for training purposes. Since most cameras can acquire video sequences, we propose relying on motion consistency within these sequences to provide additional supervision-withunee further annotations- and to refine the network weights so that it ultimately delivers better result in single images at inference time. In other words,
the sequences at training time but the refined run-lime network does not require them, which is not true for current approaches to exploiting motion information (Wan et al. 2021 Li et al. 2022 Wei et al. 2022). This is design limitation that restricts their gencrality.
In practice, we starl from potentially poorly trained network for example because not enough annolated images were available train better that outputs predictions in terms of the SMPL body model (Loper et al. 2015). We then use off-the-shelf algorithm (Teed & Deng 2020) to estimate the flow in unanootated video sequences and rely the SMPL fixed topology to refine the network weights that its predictions in individual images are consistent with it. Fig. 1 depicts this approach. This does not require changing the architecture of the network we want to refine. Furthermore, the optical flow estimator is trained on synthetic data, such as with body motion estimation and does not require human annotation. In other words, our approach is flexible, generic, and easy to deploy.
(Mayer et al. 2016 or Sintel (Butler et al. 2012), which has nothing to do
We show that this outperforms the state-of-the-art sumi-super TexturePose (Pavlakos et al. 2019)
pose and shape estimation technique that exploits color consistency instead of motion. This is significant because TexturePose is the only semi-supervised technique we know of that extracts useful body-shape information from unlabeled videos. Furthermore, color and motion-consiitency are complementary sources of information. Hence, we show that using them jointly works even better. Even though our main focus is in using only unlabelled vidcos for additional supervision, we compare against an off-the-shelf 2D keypoint estimator that uses large amount of posing data during its training. Although the keypoints provide a useful signal, our use of optical flow delivers additional enhancement. Additionally, we demonstrate that
2

---

Published in Transactions on Machine Learning Research (02/2024)
using more unannotated videos during training significantly improves the performance of the model. Finally, even though our approach is designed to operate on single images, with only minor changes, it can exploit sequences at run time and bridge the gap between monocular and video-based pose estimators.
Our method delivers these benefits for various network architectures and our experiments show that our approach further strengthens already strong baseline models relying on different backbones. Thus, our contribution is an approach to using videos refine a network designed to work on single images, which has never been done before. This lets start with insufficiently trained network and refine it without having to provide additional annotated data or to modify it in any way. This makes our method useful in practical settings where not enough annotated data is available to properly train the network in fully supervised manner.
2 Related Work
Our focus is on providing self-supervision for methods that regress SMPL models solely from images, while requiring as little additional information as possible beyond the images themselves. Thus we first examine existing semi-supervised approaches and argue that they require far more additional information than the off-the-shell optical flow estimator we rely on. We then turn the very few methods that take similar tack we do. Finally, discuss rideo-based methods, which, while powerful, are not well suited for the single-image setting we address here.
Semi-supervised Pose and Shape Estimation. A common approach to making the training of 3D body pose estimation networks less data intensive is to rely on strategies that lift 2D poses to 3D (Deng ctt al.
2021 Chen et al. 2019). In the specific case of networks that regress SMPL paramcters (Loper ct al.
2015).
Ne3D (Triiptti al. 2020) uses sequential leacher-stuudnt training to do this, while Hybrik (Li 2021 relies on an elaborate well-trained 3D pose estimator coupled with complex inverse kinematics computations. Unfortunately, this still requires considerable amounts of 3D and SMPL ground-truth.
et al.
In theory, could also use techniques that are model-agnostic. They rely on camera priors (Wandt et al, 2022), kinematics preservation (Kundu et al. 2020a Chen et al. 2022), or multi-camera consistency (Rhodin et al. 2018 Gholami et al.] 2022). However, they are designed to work under very specific circunstances and lack the generality of what we propose here.
Optical Flow for Pose and Shape Estimation. The central idea of this paper is to rely solely on consecutive images and the optical flow (OF) between them to refine a pre-trained network. OF has clearly been used before for human pose estimation, but not in this particular way.
In [Romero et al. 2015), 2D body poses directly estimated from the video using only optical flow. (Arko et al., 2022 introduces an optimization bootstrapping algorithm to enhance off-the-shelf 3D skeleton body and optical flow estimators at inference time. In (Tung et al. 2017), optical flow is used along with keypoints and silhonettes to refine the output of a pretrained SMPL-based body shape estimator. In DDersch & Zissorman 2019), a CNN-based body regressor is split in two independently trained segments. The first is trained to estimate OF and 2D heatmaps directly from images using real-world video datasets. The second regresses body parameters explicitly from OF maps and heatmaps, and the supervision is taken from synthetic datascts. The unconditioning of body estimates from the noisy irrelevant image cues helps close the domain gap between syuthelic and real video sequences. However, this approach requires the optical flow at inference time and, thus, cannot handle single images. In (Ranjan et al.] 2018 2020), the authors follow up by introducing novel synthetic dataset containing very clean OF maps paired with image-body data.
Color Consistency for Pose and Shape Estimation. Instead of using OF, one can use color con- sistency. In (Pavlakos et al., 2019). textures of the rendered body avatar are extracted from images, and the networks are trained so that they conform with what is seen in different views. This idea is extended in (Kundu et al. 2020b) by selecting pairs of images with various background colors and taking into account
3

---

Published in Transactions on Machine Learning Research (02/2024)
the visibility of particular body points. Hence, this approach is complementary ours. We will show that jointly using color consistency and motion information yields better results than using either alone.
Video-Based Pose and Shape Estimation. Soon after SMPL body estimators from single im- ages (Kanazawa et al. 2018 appeared, they were extended to handle video inputs and mostly differ in how they encode temporal relations.
A common approach (Kanazawa et al., 2019 Kocabas al. 2020; Luo al. 2020; Choi al. 2021 is to extract per-frame features and then mix them through temporal encoding. HMMR. (Kanazawa et al, 2019) exploits 2D poses extracted from videos to predict past and future poses from the current frame. VIBE Kocabas et al., 2020) uses recurrent layers to mix temporal features and penalizes unrealistic body sequences by adversarial training. In MEVA [Luo et al. 2020), the discriminator is replaced by a VAF-based motion prior, while TCMR (Choi et al. 2021) disentangles static and temporal features to increase temporal consistency. Recent advances in self-attention mechanisms have given risc a new gencration of methods that rely on spatial and temporal transformers, including MAED (Wan et al. 2021 DTS-VIBE (Li et. al. 2022 and the state-of-the-art MPS-Net (Wei et al. 2022).
All these methods require large sets of annotated video data, including 2D, 3D and ground-truth SMPL parameters, when available. Furthermore, they are typically designed to take as input a complete video. Hence, they cannot be used for single-frame pose and shape estimation. We demonstrate that, in this sctting, our algorithm can exploit temporal context and bridge the gap between monocular and rideo-based methods.
3 Method
In this section, we present our approach to fine-tuning a network by enforcing consistency between the motion flow that can be inferred from detections in consecutive images and the actual optical flow that be estimated from these images. In Sec. 3.1. start by describing the standard approach to estimating body pose and shape using& baseline model. In Sec. 3.2 we introduce our proposed Of-based technique to refining the baseline given unlabelled videos. Finally in Sec. 3.3 we argue that a minor extension of our single-image based approach allows temporal inference to bridge the gap with video-based methods.
3.1 Problem Statement
Let us assume we are given a pre-trained body shape and pose estimation network, which we will refer to as BL ("bascline"), from the so-called human mesh recovery family of models (Kanazawa et al. 2018 Kolotouros et al., 2019 Joo et al., 2020). It takes as input a single image I and outputs a human pose representation terms of the SMPL. (Loper et al., 2015: Bogo al., 2016)) pose and shape paramoters, 0 and respectively, along with camera projection estimate म. As we are mostly concerned with body shape and pose estimation, camera parameter estimation is of secondary importance. Hence, in many other approaches, we rely on the weak perspective projection camera model. This follows common practice and provides us with volumetric and semantic body representations we need. We will refer to the estimated body mesh vertices as B of size V x 3.
Our goal is to fine-tune BL using the information provided by the optical flow between consecutive video frames, without having supply any additional information or annotations. Once this done, BL does not need the motion information anymore and can operate on single images.
3.2 Optical Flow as Weak Supervision
Given two consocutive images I and I2, we define a loss that quantifies the discrepancy between
the OF map F1-2 from I, 12 estimated by an off-the-shelf optical flow algorithm Teed & Deng 2020),
the flow map F that can be computed from the vertex displacements between B, and By the SMPL body meshes estimated from both booges--rrjectetd into I and Iz.
4

---

Published in Transactions on Machine Learning Research (02/2024)
gtz
g12
X
OMI!
Clpa. gla)
Cips. Ps)
P2 Pl
P2
P1
Figure 2: Point-wise nature the OF motion-alignument loss. Points pi p2 are the predictions for the corresponding frames. If groud-truth labels are given (left), then supervision is straightforward. In the absence of ground truth (right), we use Pı displaced by the OF estimate as a weak label for p2-
Ideally, F For and Fn should be equal, and we define
oF = |F 2 [TI (I, Ml12 F F 2 (Bi),
(1)
where M is the intersection of the binary masks of the vertices visible in each frame. These visibilily masks are V-dimensional vectors computed by differentiable rendering of the 3D body mesh. II and Il denote the projection of the 3D vertices into the images and O (B,)) stands for the OF values at the vertex projections. Fig. I illustrates this computation.
In the computation of LOF?; the two images play asymmetric roles. To remedy this, we also compute L'F by reversing the roles of the images and take our full loss to be
LoF 1 - + LOF)) 2
2
In other words, we compute the flow in both directions and average the corresponding losses. This contributes stabilizing the training for two reasons. First, by getting OF information both directions, we effectively average the estimates and reduce the noise. Similarly, by using two distinct body estimates B1 and BN, we also reduce the influence of slight errors in these. Later in Sec. 5 we discuss the technical design of the LoF term of Eq. 2 and the pitfalls to be avoided when implementing it.
OF Alignment as Weak Labelling. A key feature of our approach that we use vidco sequences at training time but we don't need them inference time. This raises the question of why this additional optical flow supervision helps. Our interpretation is that minimizing Lor aligns the prodictions of the network BL with the motion, illustrated by Fig. 2 The subtraction in Eq.
I can be rewritten follows. For any vertex with projections Pı and p2 in images I and Ig, have
FB - Forfu] = (p2 - p1) OF[p
= p2 - (pl + OF[pi)
p2 p2
(3)
with P22 P1 + OFPii. Thus, P2 can be viewed as a weak label for the p2 estimate. In other words, motion alignment essentially serves as weak labelling process and Eq. 2 can be interpreted as a point-to-poing loss when the inference performed for image pairs.
Anchoring Strategy. Given our pre-trained network BL, simply minimizing LoF of Eq 2 lends to en- courage the network to shrink the area of its projected predictions. In theory, the loss function can be brought to zero by covering small region with constant flow, even zero flow, with the body in image I and performing translation, or no motion at all, to L,
Hence, during fine-tuning, we do not want to deviate too much from the initial state of the pre-trained model. Assuming that the data used for pre-training is still available, we can take the full loss to
L Loup + LOF
(4)
5

---

Published in Transactions on Machine Learning Research (02/2024)
where Leu is the loss originally used to train the network. This is simple but effective. In the supplementary material we discuss alternative approaches that can be used when the original training data is unavailable.
3.3 Providing a Temporal Context
Statc-of-tho-art vidco-basod methods use special-purpose layers- GRUs, LSTMs transformers to account for temporal consistency. Hence, they require image sequences both for training and testing. By contrast, distill the knowledge from unlabeled videos during training, regardless of whether such information is available for testing. What is more, if full video sequence is available at test time, we can exploit it follows.
In practice, the BL of Sec. 3.1 relies on a ResNet (He et al. 2016) feature encoder connected to a SMPL decoder. We exploit this by introducing a feature-wise affine transformation conditioned on the predictions for the previous frames, inspired by (Perez al. 2018). Let assume we are given N consccutive frames. We use our BL to produce SMPL estimates for - of them, [[0r, क mi).... 10, n We feed these N - estimates to an auxiliary lightweight 2-MLP context network that outputs the parameters 5} of an affine transformation that is applicd to the activation f of the last lincar layer of the RosNet feature encoder. We write
f=yof+ō
(5)
and food the Nth frame to the network thus modifiod.
In other words, we still use the same single-frame network as before but inject temporal context via feature normalization. In effect, the MLP provides temporal context for pose estimation in frame N. The training as described in the previous section is mostly unchanged. The only difference is that the auxiliary "context" network now used to estimate By during training and learns cues from the previous estimates. Once trained, we use it at inference time as well. We provide more details in the supplementary material.
4 Experiments
In this section, first discuss our experimental setup (Sec. 4.1 and demonstrate the superiority of our approach over state-of-the-art competing method (oexturePose) alone, along with the complementarity of the two approaches given varying amounts of annotations (Sec. 4.2 Next, we showcase our ability to exploit sets unannotatod video sequences at training time (Sec. 4.3 and to exploit temporal consistency not only at training time but also al lest, time (Sec. 4.5 We then show that our approach boosts performance regardless of backbone architecture (Sec. 4.6 Finally, we analyze the quality of OF a supervisory signal
(Sec. 4.7
4.1
Experimental Setup
We use a well established ResNet-based architecture (anazawa et al. 2018 Kolotouros et al. 2019 Joo
et al. 2020 Kocabas et al. 2020 as the BL baseline of Sec. 3.1 for most. of the experiments unless stated
otherwise. More sophisticatod architectures, such as the volumetric heatmap-based models (Li al., 2021 the bransedrmer--bseed HMR 2.0 architecture (Goel et al. 2023) deliver state-of-the-art results in human pose and shape estimation. However, they very data-demanding and do not work properly under only limited supervision (Sun et al. 2018 Iskakov et al. 2019 Dosovitskiy et al. 2020h. Hence, they are less suitable for our main experiments in the low-data regime, yet we demonstrate that our method can be applied to more sophisticated architectures, such as HRNet--W3 (Sun et al. 2019 Kocabas et al.) (021). To estimate optical flow we use the off-the-shelf pre-trained RAFT TTed & Deng 2020), without fine-tuning it.
Pro- Training and Datascts. To pre-train our bascline network BL, we use subsets obtained by randomly picking 10%, 20%, 50% and 100% of the samples from the COCO [Lin et al. (2014) dataset (only training subset, 75K) of in-the-wild images with high-quality pseudo-ground truth supervision acquired by EFT {Joo

---

Published in Transactions on Machine Learning Research (02/2024)
et al. 2020 We will refer to them as COCO*% where p is the percentage of the COCO images and corresponding ground-truth labels used to pre-train the baseline model.
The COCO-EFT dataset exhibits many good features. First, it is a compact yet high-quality dataset that is sufficient to train a strong body estimator model approaching state-of-the-arr performance. Even when only a small fraction of the data used (COCO10% COCO20%) the quality of the trained baseline is satisfactory and serves as a good starting state for further refinement.
For datasets containing videos, also use 3DPW (von Marcard et al. 2018 (both training and testing subsets), Human3.6M (Tonescu et al. 2014), and PennAction (Zhang et al. 2013 (we will refer to it as PA).
These datasets differ in size, image resolution, and scene complexity (samples from each of the datasets can be found in the supplementary material). During the fine-tuning experiments, we utilize only individual images without reference to the additional data provided these datasets. For evaluation, we use common pose benchmarks, such as 3DPW (von Marcard et al.) 2018), Human3. 3.6M (Ionescu et al.) (014) and MPI- INF-3DHP (Mehta al. 2017]. We rely the Adam (Kingma & Ba 2015) optimizer, and the loss weights were chosen via grid search. We provide the details in the supplementary material.
Metrics. We adopt the widely used 3D Mean-Per-Joint Position Error with Procrustes Alignment (P- MPJPE, mm). To compare against video-based methods (Luo et al. 2020 Choi et al. 2021 Kocabas et al.) 2020), we estimate the pose in each image individually and, even though our method does not feature a temporal module, we compute the acceleration error (Accel.Err, mm//²) between the predicted and ground- truth 3D joints. This metric has been used in previous works (Kocabas et al.. 2020 Kanazawa et al. 2019:
Choi et al. 2021 to quantify the trade-off between 3D pose accuracy and motion consistency.
4.2 Fine-tuning with Optical Flow Loss
66
87.
cocoie
TF
OF
61
mix
01
P-MPPPEE 600
61
20%
50%
100%
87.
H3.6M
3DPW--rain
COCD
Figure 3: Our method vs ToxturePose. We plot P-MPJPE as function of the percentage of the an- notations we used before refinement for the baseline model and after semi-supervised refinement using ei- ther TexturePose, our approach with OF only, or both OF and TexturePose. Our approach outper- forms TexturePose, and, as expected, the less anno- tated data use, the greater the improvement. For video dataset, Human3. 6M was used. All metrics are J-MPJPE 3DPW-test set.
Figure 4: More unlabelled videos. We plot P- MPJPE as function of the amount of unlabelled data used for fine-tuning BL using our approach. As expected, the more unannolated data use, the greater the improvement. All metrics are P-MPJPE 3DPW-test set.

---

Published in Transactions on Machine Learning Research (02/2024)
Table 1: Fine-tuning the baseline model
More unannotated data. Starting from the BL pre- trained using COCO we refine it using images from either H36M alone, or H36M and 3DPW-tr, H36M, 3DPW-tr, and PA. We evaluate on common pase benchnarks. use, the better the performance. The last row of 3DPWttest evaluations corresponds to using the test-set itself for supervision, which is meaningful and practical in some cases because the loss is self-supervised. This yields the best performance but using H36M + 3DPW-tr + PA comes close. All metrics are P-MPJPE, except Eccel.Err for 3DPW-test evaluation (in brackets).
Table 2:
BL using our Of Of-guided method (OF), Tex-
turePose (Pavlakos et al. 2019) (TP), 2D key-
points (2D) {Wu et al. 2019) or mixing OF
expected, the more images
with each of these. For video dataset, Hu-
man3.6M was used. All metrics are P-MPJPE 3DWWttest set.
10%
20%
50%
100%
BL
65.3
62.2
59.4
58.2
TP
62.7
60.0
58.8
58.1
OF
61.3
59.3
58.0
57.1
3DeWttest
H36M (P2)
MPI-IN--DDHP
TP+OF
60.4
38.7
57.6
56.9
BL (COCO 100%)
58.2 (29.0)
64.4
67.3
2D
58.4
57.1
56.4
56.0
H36M
57.1 (24.4)
59.7
65.2
2D+OF
57.6
56.5
56.0
55.8
H36M 3DPW-tr
56.4 (24.0)
59.4
65.0
H36M 3DPW-tr PA
56.2 (23.9)
59.1
64.7
3est
55.8 (20.2)
We compare our approach against TosturePose PPavlakos et al. 2019). the only other self-supervised pose
and shape estimation method we know of that relies mostly on image content as we do. Along with color- consistency, TexturePose exploits silhouettes and 2D keypoints, which we do not.
We first use the ground-truth body poses of the COCO0% datasets to pre-train BL. We then use the video sequences of Human3.6M (Tonescu et al.] 2014] and only them--we ignore the ground-truth pose data- to refine the model weights by minimizing the loss of Eq. | that includes the optical flow loss of LoF of Eq. 2 or, instead, one that includes the texture-aware loss term of TexturePose. To this end, we re-implemented the texture extraction algorithm of TexturePose because the original code is not publicly available. This has enabled us to compute the loss
L Loup + LTP
(6)
where LTp is the texture-aware loss of TexturePose, that plays the same role as LoF in our formulation. Our training batches are combinations of independent frames with single-frame ground-truth supervision and pairs of consecutive frames without annotations. For TexturePose, we sample 5-frame sequences as in the original paper (Pavlakos et al., 2019).
We evaluate both approaches on the test set of the 3DPW dataset, which was not used in any part of the training. In Fig. we plot the P-MPJPE metric function of what percentage of the annotated data we used to pre-train the pose estimation baseline BT. Using either method, the less initial supervision is used initially the more noticeable the improvement brought about by the unannotated sequences. However, our approach consistently outperforms TexturePose and delivers an improvement even when the baseline has been trained using all the available annotations, whereas TexturePose does not. We attribute this to the fact that our approach directly leverages all the 3D points of the body mesh, whereas TexturcPose works by interpolation in the pixel space. Note that the model that uses only 10% of the available supervision coupled with the proposed OF loss beats the baseline model that uses twice many samples. The same can be seen in Fig, 3 when comparing models of 05(00% and BL(100%).
To demonstrate that the two approaches are in fact complementary, we repeat the experiment by minimizing a loss that combines ours and that of TexturePose. We take it to be
L L + LOF + LTP,
where Leup and LoF are the losses of Eq. while LTp is the texture-aware loss of TexturePose. As can be seen in Fig. B this yiclds a further improvement over optical flow alone, especially when using small fractions of the available annotations to pre-train the network. Our interpretation is that minimizing the Ltp term helps correct the body estimates so that more of its projections overlap the true body, which ensures that the flow is used more effectively.
To put these numbers in context, the best current method (Goel et al. 2023) delivers a P-MPJPE of 44.4 but relies on vast amounts of training data and transformer architecture that is unlikely to do well with
8

---

Published in Transactions on Machine Learning Research (02/2024)
reduced amount of training of data. Similarly, the method of (Li et al. 2021 delivers 48.8 but heavily relies on a strong 3D heatmap estimator that also requires abundant supervision.
4.3 Using more Unannotated Data
The results shown above were obtained using Human3.6i M. as the sole source of unlabeled videos. Since we have additional datasets, we can use them as additional source of unannotated images. We ran this experiment and report the results in Table 2 (across various evaluation benchmarks) and in Fig. (across baseline models that use different amount of supervision). We started from the baseline models BL pre- trained with subsets of COCO and refined using only Human3.6M (Tonescu et al. 2014) as before, or adding images from 3DPW-train (von Marcard et al. 2018), or from both 3DPW-train and PennAction (Zhang ct al. 2013). As expected, the more unannotated images we use, the better the result across all benchmarks.
Furthermore, because our method semi-supervised, we can use the test data to further refine the network. In the last row of Table 2
we report the result of using the test vidcos to refine the BL(COC)108) network. This yields the best, results of all and there scenarios in which this would be practical. However, when using all the available data but not the test data, obtain similar results.
As before, the less supervision the model has received, the more valuable unsupervised OF motion alignment becomes (-6mm in P-MPJPF for COCO10%). The baseline model trained with COCO50% is considerably outperformed by the fune-tuned one that has seen 5 times less supervision (COC010%) but used the proposed weakly-supervised OF -F-induced motion alignment.
To put these results in context in terms of P-MPJPF (mm), our model that utilized 20% of available body supervision (F(CC0020% model (57.8) outperforms such models, as SPIN (Kolotouros et al. 2019) (59.2), РyMAF (Zhang et al. 2021 (58.9) and performs par with 12L-MeshNet. (Moon & Lee 2020 (57.7). All these methods rely on full body supervision using millions of samples, whereas our approach uses only about 15K.
Although additional data brings significant boost to the models' performance, it comes at cost of longer training time. We provide these statistics in the supplementary material.
4.4 Beyond Unannotated Videos
In keeping with the central focus of this paper, so far, we have only shown results using unlabeled videos to provide additional supervision. This restricts the range of methods against which we can compare. To broaden it, we now use the Detectron2 2D keypoint estimator (Wu et al. 2019) as an alternative baseline. Although training Det2ctron2 requires a large amount of body-related data, which runs against our desire to use less supervision, we can treat it as a black-box model that provides additional supervision and run it on our video frames during training. In addition, one can also combine our optical flow approach with this additional 2D keypoint supervision to show their complementarity.
More specifically, we take the loss to be the combination of the fully supervised Lsup and the additional Lan aligument between 2D keypoints predicted by off-the-shelf 2D pose estimator and the joints extracted from the predicted body shape. As we did with TexturePose in Section 4.2 we repeat the experiment by minimizing a loss that combines ours and 2D keypoint alignment, as in Eq. 7 In the last two rows of Tab. we compare 2D keypoint supervision (2D) and the combination of our method with 2D keypoints (2D OF). Even though 2D keypoints encode more body-related information, optical flow provides useful additional information and enhances the model's performance.
4.5 Using the Temporal Context
As discussed in Sec. 3.3 OF OF-guidance can be used to exploit temporal information in video sequences. In other words, even though one often distinguishes between single-frame and vidco-basod methods, our approach bridges the gap between the two. It enables single-frame model to exploit motion information training time to provide more realistic motions at inference time.
9

---

Published in Transactions on Machine Learning Research (02/2024)
Table 3: Our approach with and without temporal context. Starting from a pre-trained baseline model using COCO we report results using from operating strictly on one single frame, as in Sec. 1.2 We also compare against state-of-the-arl video-based methods. Our method performs on par with them in both words, it provides less shaky estimates, even though it was not tailored for video-based estimation. All metrics are P-MPJPE 3DPW-test set.
previous frames. Here 0 stands for not using the temporal context and
in pose and, most importantly
in acceleration. In other
P-MPJPE
Arcel.Err
0
57.1
24.4
ctx length
57.0
24.0
2
56.9
22.6
56.7
56.7
18.8
video-bussed
HMMR
Kanazawa
72,6
15.2
VIBE
Kornhhs
57.6
25.4
MEVA
Lao 0s al.
2020
54.7
11.6
TCMR
Choi at
2021
52.7
6.8
MPS-Ne
Wei
2022
62.1
7.4
OF (ReeNirN))
(HRNNer
d(Pp.For alFop: d(Fop.Forr
10
PAMPIPE
ibon disunce
My Wal Mom Whlnor la Wom
M.y Wn mamnee Humaen
10%6
20%
COCO data
50%
100%
50
100
185 Prame index
2000
250
300
Figure 5: Different backbones. We apply our fine-tuning strategy using two different backhones, ResNet50 (solid) and HRNet-W32 (dashed). Our approach delivers a significant improvement in both cases. Note that the complex volumetric features of HRNet- W32 perform better than those ResNet when lot training dala and worse when do not.
Figure OF quality. We compute the distances dor and dø described in Sec. 1.7 for particular video sequence. We that the motion flow induced by the body estimator lies always further from the ground-truth, than the optical flow estimate. proves the effectiveness of the off-the-shelf optical flow model.
To demonstrate this, we use the same experimental setup as in Sec. 4.2 We compare the performance of the network pre-trained using COCO100% and refined using the Human3.6M 6M sequences, with the same the predictions obtained in the previous N images, with N ranging from to 8. We report the results in Table 3 Even very short 2-frame time window gives small P-MPJPE boost. It increases slightly for N 8 but not for greater values of N. More importantly, exploiting temporal consistency provides a more significant boost in the Acceleration Error introduced in Sec. 4.1 mcaning that the sequence of recovered poses will be far less jerky, even though it ls produced by a single-frame inference model without recurrent layers. We would like to emphasize that prediction--from-videe is not our primary focus. Instead, we demonstrate how exploit temporal information that naturally encoded in the optical flow, and how it makes the angle-frame model compatible with the video inputs. More experiments involving temporal context are provided in the supplementary material.
network trained
10

---

Published in Transactions on Machine Learning Research (02/2024)
Table 4: OF quality. The mean and median of the ratios between d(FB. B, Far) and d(F Fer) across 1000 random samples from each datasel, as described in Sec. 4.7 At is time distance between frames in sequence. Oracle
refers to the case where OF is computed at ground-truth vertices instead of estimates of the body regressor.
3DPW
H36M
PennAction
LR
HR
LR
HR
LR
HR
233 / 1.6
2.6 /
24 / 1.7
27
3.2 /
3.3
/
1.9
1.6 / 1.3
1.
/ 1.3
1.6 / 1.3
1.8
1.5
2.8
f
1.4
2.3
1.4
1.4 / 1.1
1.5
/ 12
1.5
1.2
1.6
0.6/13
1.3
2.7
1.4
2.1
/.1/
1.3 / 1.1
1.4
/ 1.2
14 / 1.2
1.5
/
1.3
2.2
1.3
2.5
1.3
(oracle)
6.4 / 4.0
6.7
/
7.9 /
8.8
/
4.9
6.6
3.2
7.0
4.6 Changing the Backbone Architecture
To illustrate that our method improves performance regardless of backbone architecture, we replace ResNet- 50, which we have used so far, with the high-resolution backbone H3Ne Tt was first proposed in (Sun cct al.j 2019) and later adapted for SMPL estimation in (Kocabas ct al. 2021). We pre-train (BL) and fine-tune (OF) this new architecture as in Sec. 4.2 We report our results in Fig. 5 HRNet-W33 learns high-resolution volumetric features. This is very effective when there is enough training data and HRNet- W32 provides state-of-the-art result when using all the training data. However, in the data-scarce regime, the volumetric features are much harder to learn (Sun et al. 2018 Iskakov et al. 2019 Hence, under these conditions, ResNet delivers better results.
In any event, our proposed fine-tuning strategy improves performance considcrably, as did the case of the ResNet backbone presented in Sec. 4.2 In both cases, the less data is used for supervision, the more noticeable the improvement is. In other words, our method is model-agnostic and is beneficial independently of the specific network architecture being used.
4.7 Quality of OF as a Supervisory Signal
In this section, we demonstrate that the OF-predicted motion is far better aligned with the real motion of the body than the motion that can be inferred from the pre-trained baseline, which goes a long way towards explaining why it provides a useful supervisory signal.
To this end, exploit the fact that datasets feature 2D ground-truth keypoints. For any given pair of frames I and L we compute three motion flows at the positions of visible keypoints:
from the ground-truth 2D keypoints, FGT:
from body estimates, B,
3. and the optical flow one, For.
Then, we compute dor = d(F oF, FGT) and dp = d(FB, FCT), where d is the usual Euclidean distance. In Fig. 6 we plot these two distances dor blue and dg in red-for a specific sequence. Note that the blue curve systematically below the red one, indicating the optical flow estimate closer than the body regressor estimate the ground-truth.
To show that this is true not only for single sequence but in general, generated Table 4 The two numbers in each cell are the mean and the median of the ratios of the two distances for randomly chosen set of 1000 images from each datasct. The higher these numbers the more additional information OF is likely to provide. Since image resolution strongly affects the quality of the OF, for each image, we compute OF map at two resolutions, low (224 pixels, BL imput format) and high (maximum available). Each line in the table corresponds to different time interval between frames in which we compute the OF. Since the predicted bodies do not project perfectly on the image, the OF can be corrupted by background pixels. To gauge this effect, recomputed dor using the ground-truth body models to ensure that all pixels involved
11

---

Published in Transactions on Machine Learning Research (02/2024)
Lop. without normalization
LFF. with normalization
3DPW
3DPW
113.6M
113.6M
PennAction
PennAction
10
0
2
6
LOF
LOF
Figure 8: Scaling of OF loss. We found that the differences in motion behavior may drastically differ across different datasets or even same video sequences. On the left, we show the values of Lop (Eq. 2 for 2 samples of each of the video datasets. We propose to normalize Los for every sample
Figure 7: Randomization. The first two columns in every show pairs of consec- utive frames from different video datasets, 3DPW, H36M and PeunAction, with overlaid body estimates. These estimates are very sim- ilar each other but wrong. The third col- umn feature the body estimate for the second frame after color randomization. Even if this does not give the correct motion flow or OF, it disentangles body estimates from each other.
the mean of the OF norms across all visible vertices. The result is on the right. Loss normalization significantly stabilizes the training.
are foreground pixels. This can be viewed as an oracle and is given on the final line the table. As the training progresses and the predictions improve, the ratios can be expected to approach these values. In any event, these results confirm our hypothesis that OF brings valuable additional information and that A ==1. 1, ie. using consecutive frames, is the best way to exploit it.
5 Implementation details
In this section, we discuss the technical design of the Lor from Eq. and what pitfalls should be avoided when implementing it. In particular, we added color noise to differentiate the predictions of two neighboring frames, threshold and scale the loss values for computational stability.
Rigidity of body estimators. For the experiment carried out in Sec. 4.2 found that the LoF from Eq. [ must be implemented carcfully. Body estimates for neighboring frames, especially when acquired by fixed camera as in Human3.6M (Ionescu et al. 2014). look very much alike, because the images and the features extracted from them are also similar. Consequently, similar frames can yield similarly bad body estimates, see the first two columns in Fig. 7 for an illustration. Hence, the seemingly natural assumption that body estimates for different inputs are independent does not hold, which can lead to correlated errors in OF estimates of Eq. I We can eliminate this effect by adding random noise to the colors of input frames. This makes the body estimation model more robust to perturbations in pixel colors and allows it provide different oulputs. See the third column of Fig. 7
Thresholding of OF loss. Additionally, for computational stability, very large values of Lop should be discarded. Our tests show that such high spikes in OF estimates connected to either wrongly chosen
12

---

Published in Transactions on Machine Learning Research (02/2024)
pairs of frames- in some sequences, some frames were cut out-ot or to partial occlusions. We threshold LoF by the maximum of OF norms || orll across all visible vertices.
Scaling of OF loss. When using different video datasets for refinement, the motion characteristics, such as default FPS, can be very different and very different behaviors can be seen even among the pairs of frames inside one video sequence. For example, in Human3.6W, (Ionescu et al.) 2014) in most of the vidcos, the subject stands still the first several frames before starting to move to perform actions. In such cases, the estimated motion from both optical flow or body pose&shape predictor may differ in orders of magnitude, as shown in Fig. S (left). We found it crucial to compensate for this effect when minimizing the loss of Eq, 1 because it has supervised part that is independent from the motion changes. To this end, for given body estimate, we compute the mean value of the OF norms ||F across all visible vertices, and divide the value of LOF over it. In Fig. & we plot histograms of LOF values before fine-tuning the baseline model. Every histogram contains 2000 datapoints randomly sampled from cach of the datasets. The plot on the left shows how diverse and inhomogeneous the motion is. The plot on the right shows the same values after normalization. Now all datasamples are equally distributed.
6 Conclusion
We have shown that optical flow between image pairs could be used to refine a pre-trained network so that it works better on single images. This can be ascribed to the fact that the motion estimates delivered by good but standard OF estimator correspond better the real motion than what can be inferred from the pre-trained pose and shape estimator's predictions. This is useful observation because all takes to exploit are video sequences of people in motion, which are easy to acquire. Thus, even though we acknowledge the prevailing "data is king" paradigm, such scenarios do occur due to privacy and licensing issues, which makes this useful. Furthermore, our method also aligns with ethical considerations, while respecting data minimization principles.
In principle, our optical flow- technique could be applied for other moving objects. We plan to extend our approach a very challenging domain of animal pose and shape estimation. There an abundant amount of unannotated videos, yet few methods and models can exploit them effectively because there is very little annotated data for this task. We will also exploit motion priors that can be obtained from existing databases to further constrain the refinement.
References
A.R. Arko, K.M. Yi, and J.J. Little. Bootstrapping Human Optical Flow and Pose. ar Preprint, 2022.
F. Bogo, A. Kanazawa, C. Lassner, P. Gehler, J. Romero, and M. J. Black. Keep It SMPL: Automatic Estimation of 3D Human Pose and Shape from Single Image. In European Conference on Computer Vision, 2016.
D. Butler, J. Wullf, G. B. Stanley, and M. J. Black. A Naturalistic Open Source Movie for Optical Flow Evaluation. In European Conference on Computer Vision, 2012.
C. Chen, A. Tyagi, A. Agrawal, D. Drover, R. MV, S. Stojanov, and J. M. Rehg. Unsupervised 3D Pose Estimation with Geometric Sell-Supervision. In Conference on Computer Vision and Pallern Recognition, 2019.
S. Chen, Y. Xu, Z. Pu, J. Ouyang, and B. Zou. Skeletonpose: Exploiting human skeleton constraint for 3d human pose estimation. Knowledge-Basse Bascd Systems, 2022.
Н. Choi, G. Moon, J.Y. Chang, and K.M. Lee. Beyond Static Features for Temporally Consistent 3D Human Pose and Shape from a Video. In Conference on Computer Vision and Pottern Recognition, 2021.
A. Davydov, A. Remizova, V. Constantin, S. Honari, M. Salzmann, and P. Fua. Adversarial Parametric Pose Prior. In Conference on Computer Vision and Patlern Recognition, 2022.
13

---

Published in Transactions on Machine Learning Research (02/2024)
Y. Deng, C. Sun, J. Zhu, and Y. Sun. SVMAC: Unsupervised 3D Human Pose Estimation from Single Image with vinell-iew-mulli-annle Consistency. Tn International Conference on 3D Vision, 2021.
C. Doersch and A. Zisserman. Sim2real transfer learning for 3d human pose estimation: motion to the rescue. In Advances in Neural Information Processing Systems, 2019.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghami, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In arviv Preprint, 2020.
M. Gholami, Rezaci, H. Rhodin, R. Ward, and Z.J. Wang. Self-Supervised 3D Human Pose Estimation from Video. In Neurocompuling, volume 488, pp. 97 106. Elsevier, 2022.
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and Tracking Humans with Transformers. In International Conference on Computer Vision, 2023.
K. He, X. Zhang, S. Ren, and J. Sun. Deep Residual Learning for Image Recognition. In Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Charlie Hewitt, Tadas Baltrušaitis,, Erroll Wood, Lohit Petikam, Louis Florentin, and Hanz Cuevas Ve- lasquez. Procedural Humans for Computer Vision. arXiv Preprint, 2023.
C. Tonescu, I. Papava, V. Olaru, and C. Sminchisescu. Human3. 66: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014.
K. Iskakov, F. Burkov, V. Lempitsky, and Y. Malkov. Learnable Triangulation of Human Pose. In Interna- tional Conference on Computer Vision, pp. 7718-7727, 2019.
H. Joo, N. Neverova, and A. Vodaldi. Exemplar Fino-Tuning for 3D Human Pose Fitting Towards In-tho-Wii Wild 3D Human Pose Estimation. In International Conference on 3D Vision, 2020.
A. Kanazawa, M. J. Black, D. W. Jacobs, and J. Malik. End-To-Enn End Recovery of Human Shape and Pose. In Conference Computer Vision and Pattern Recognition, 2018.
A. Kanazawa, Zhang, P. Felsen, and Malik. Learning 3D Human Dynamics from Video. In Conference on Computer Vision and Pattern Recognition, 2019.
D. P. Kingma and Ba. Adam: A Mothod for Stochastic Optimisation. In International Conference on Learning Represendations, 2015.
M. Kocabas, N. Athanasiou, and M. J. Black. VIBE: Video Inference for Human Body Pose and Shape Estimation. In Conference on Computer Vision and Pattern Recognition, pp. 5252 5252-5262, June 2020.
M. Kocabas, C.-H.P. Huang, O. Hilliges, and M.J. Black. PARE: Part attention regressor for 3D human body estimation. In International Conference on Computer Vision, 2021.
N. Kolotouros, G. Pavlakos, M. J. Black, and K. Danillidis. Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop. In Conference on Computer Vision and Pattern Recognition, 2019.
J. N. Kundu, S. Seth, M.V. Rahul, M. Rakesh, R. V. Babu, and A. Chakraborty. Kineenatic-Strueturee Preserved Representation for Unsupervised 3D Human Pose Estimation. In AAAT Conference on Artificial Intelligence, volume 34, pp. 11312 11319, 2020a.
J.N. Kundu, M. Rakesh, V. Jampani, R.M. Venkatesh, and R.V. Babu. Appearance Consensus Driven Self-Supervised Human Mesh Recovery. In Europeon Conference on Computer Vision, 2020b.
J. Li, C. Xu, Z. Chen, S. Bian, L. Yang, and C. Lu. Hybrik: A hybrid analytical-neural inverse kinemat- ics solution for 3d human pose and shape estimation. In Conference on Computer Vision and Pattern Recognition, 2021.
14

---

Published in Transactions on Machine Learning Research (02/2024)
Ziwen Li, Bo Xu, Han Huang, Cheng Lu, and Yandong Guo. Deep Two-Stream Video Inference for Human Body Pose and Shape Estimation. In IEEE Winler Conference Appliculions of Compuler Vision, pp. 430 409, 2022.
T- Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C.L. Zitnick. Microsoft COCO: Common Objects in Context. In European Conference Computer Vision, pp. 740-755, 2014.
M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M.J. Black. SMPL: A Skinned Multi-Person Lincar Model. ACM SIGGRAPH Asia, 34(6), 2015.
Z. Luo, S.A. Golestaneh, and K.M Kitani. 3D Human Motion Estimation via Motion Compression and Refinement. In Asian Conference on Compuler Vision, 2020.
N. Mayer, Tlg, P. Häusser, P. Fischer, Cremers, A. Dosovitskiy, and T. Brox. Large Datasel Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation. In Conference on Computer Vision and Pattern Recognition, 2016.
D. Mchta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt. Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision. In International Conference on 3D Vision 2017.
G. Moon and K.M. Lee. 12I-MeshNet: Image-to-Tixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from Single RGB Image. In European Conference on Computer Vision, 2020.
G. Pavlakos, N. Kolotouros, and K. Danillidis. Texturepose: Supervising Human Mesh Estimation with Texture Consistency. In International Conference on Computer Vision, 2019.
E. Perez, F. Strub, H. de Vries, V. Dumoulin, and A.C. Courville. FILM: Visual Reasoning with a General Conditioning Layer. AAAI Conference on Artificial Intelligence, 2018.
A. Ranjan, J. Romero, and M.J. Black. Learning Human Optical Flow. In British Machine Vision Confer- ence, 2018.
Ranjan, D.T. Hoffmann, D. Tzionas, S. Tang, J. Romero, and M.J. Black. Learning Multi-Human Optical Flow. International Journal of Computer Vision, 2020.
H. Rhodin, M. Salzmann, and P. Fua. Unsupervised Geometry- Aware Representation for 3D Human Pose Estimation. In European Conference on Computer Vision, 2018.
J. Romero, M. Toper, and M.J. Black. FlowCap: 2D human pose from optical flow. In Pattern Recognition, 2015.
Akash Sengupta, Ignas Budvytis, and Roberto Cipolla. Hierarchical Kinematic Probability Distributions for 3D Human Shape and Pose Estimation from Images in the Wild. In International Conference on Computer Vision, October 2021.
K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representation learning for human pose estima- tion. In Conference on Computer Vision and Pattern Recognition, 2019.
X. Sun, F. Wei B. Xiao, S. Liang, and Y. Wei. Integral Human Pose Regression. In Europeon Conference on Computer Vision, 2018.
Z. Teed and J. Deng. RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. In European Conference on Computer Vision, 2020.
S. Tripathi, S. Ranade, A. Tyagi, and A. Agrawal. Poscnet3d: Learning temporally consistent 3d human pose via knowledge distillation. In International Conference on 3D Vision, 2020.
H.-Y. Tung, H.-W. Tung, E. Yumer, and K. Fragkiadaki. Self-Supervised Learning of Motion Capture. In Advances in Neural Information Processing Systems, pp. 2017.
15

---

Published in Transactions on Machine Learning Research (02/2024)
T. Marcard, R. Henschel, M. Black, B. Rosenhahn, and G. Pons-Moll. Recovering Accurate 3D Human Pose in the Wild Using IMUs and Moving Camera. In European Conference on Compuler Vision, 2018.
Z. Wan, Z. Li, M. Tian, J. Liu, S. Yi, and H. Li. Encoder- with Multi-level Attention for 3D Human Shape and Pose Estimation. In International Conference Computer Vision, 2021.
B. Wandt, J.J. Little, and H. Rhodin. Flepose: Unsupervised 3d Human Pose Estimation by Predicting Camera Elevation and Learning Normalizing Flows on 2D Poses. In Conference on Computer Vision and Pattern Recognition, 2022.
W.-L. Wei, J.-C. Lin, T-L. Liu, and H.-YMM. Liao. Capturing Humans in Motion: Temporal-Attentive 3D Human Pose and Shape Estimation from Monocular Video. In Conference on Compuler Vision and Pattern Recognition, 2022.
Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick. 2019.
Detectron2.
https://github.com
facebookresearchdetteee
H. Zhang, Y. Tian, X. Zhou, W. Ouyang, Liu, L. Wang, and Z. Sun. Pymaf: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feodback Loop. In International Conference Computer Vision, 2021.
W. Zhang, M. Zhu, and K. Derpanis. From actemes to action: strongly-supervieed representation for detailed action understanding. In International Conference on Computer Vision, 2013.
16

---

Published in Transactions on Machine Learning Research (02/2024)
Table 5: Overview of the video datasets used in this work with the number of images and image resolution.
Datasets
#samples
Image resolution
3DPW-train
23k
1080x1920 px
3DPW-test
34k
1080x1920 px
Human3.61 M
293k
1002x1000 px
PennAction
162k
360x480 px
Table 6: Using more unannotated data from 3DPW-train and PA. Despite PA being much larger (x7) than 3DPW-train, the improvement coming from it is smaller. We explain it with the low image resolution of PA (and hence, the lower quality of OF) and the larger domain gap with the 3DPW-test set used for evaluation.
P-MPJPE (4)
Accel Err (L)
BL (C0CO100%)
58.2
29.0
PA
57.8
27.8
3DPW-train
57.4
25.8
3DPW-tr PA
57.2
25.2
In this supplementary material, provide the quality of the estimated motions with the corresponding videos attached (Sec. A), the choice of lyper-parameters (Sec. B the samples from each of the datasets with the pre-processing details (See. Cj, additional experiments on using more unlabelled data with combinations of the datasets not used in the main paper (Sec. D) the architectural design of the temporal context injection
(Sec. E). additional experiments on using temporal context for baselines of less supervision (Sec. F} and
alternative strategics of exploiting optical flow loss, when the supervision is not available (Soc. The code
and supplementary files are publicly available http::///iilabbcccmmm////lll/l/p
A Motion quality analysis
Here evaluate the estimated motions particular sequences qualitatively and quantitatively. In Fig. g plot acceleration errors for four video sequences and compare the baseline trained on COCO with its OF-refined versions without and with the temporal context (lines "0" and "8" in Table 3 in the main paper). It is clear that the proposed OF strategy improves the estimated motion considerably.
Furthermore, we provide video files with the corresponding body renderings for qualitative comparison.
B Optimization details
In all our experiments in the main paper, we used grid search to find the optimal balance between different terms in loss functions. The values for the balance weights are: A[Laup] 1, A[LoF] 0.01, X[LTP] 10.
C Datasets
We provide samples from each of the datasets used in the present paper in Fig. 10
Pre-processing of video datasets. We follow the common pre-processing of video datasets, first pro- posed in (Kocabas et al. 2020). Their pre processee dictionaries contain lists of frames grouped by videos. We provide the number of samples and image resolution of the video datascts in Table ( During our ex- periments, we did not use any image augmentation on video frames, except color noise, when inferring body estimates (the details are in Sec. 5]
17

---

Published in Transactions on Machine Learning Research (02/2024)
100
(COCOLOON)
18.1
Sequence 1
Accel.Err Err, mmis?
OF
80
15.1
OF + tmp cbx (8)
13.3/ 11.6
60
40
Way
150
250
300
BL (COCO)00%)
80
31.4/ 29.0
Sequence 2
Accel.Err, mm/s²
/ 25.3
60-
tmp (8)
19.6 18.4
40
50
100
150
200
250
300
100
29.6
(COCOL00%)
Sequence 3
Accel.Err, mm/s?
OF
27.6 24.4
OF + tmp etx (8)
23.3 22..0
60
40
150
250
BL (COCD(UN))
60
20.6 / 19.6
Sequence
Accel.Err, mmis?
17.5 / 17.1
tmp
40
20
50
100
150 Frame index
200
250
300
Figure 9: We plot Accel.Err for four particuluar video sequences. Colored lines show the performance of the baseline model BL (C0CO100%; its OF-refined version (Sec. 4.2 of the main paper) and OF-refieed with temporal extension (Sec. 4.5 of the main paper). In the legends, report the evident that the proposed OF-based semi-supervised technique and its temporal context version significantly improve human motion, despite the fact that the architecture is single-frame and does not contain any recurrent layers. We provide corresponding video files with side-by-side motion comparison.
and the median across all frames in sequence. It is
D Additional results on "Using more Unannotated Data"
In Table 2 of the main paper, we presented how adding more unannotated data brings more improve- ment to the performance of the body estimator pretrained using 10CO100% In the main paper, took
18

---

Published in Transactions on Machine Learning Research (02/2024)
Samples from the datasets exploited in this work. From top to bottom: COCO (Lin et al. 2014). 3DPW fvon Marcard al. 2018). H36M [Tonescu et al. 2014) and PennAction (Zhang et al. 2013). COCO and its extended version from (Joo al. 2020) provides high-quality psendo ground-truth bodies for individual frames. All other datasets comprise video sequences that use without labels.
Figure 10:
Table 7: The cost of using more data. The use of more data comes with larger cost of training time. The "#samples" column denotes the number of additional training samples, and the "#iterations" shows the number of the network weights updates needed for model convergence during fine-tuming.
P-MPJPE (4)
Arcel.Err (L)
#samples
Training time, #iterations
H36M
57.1
24.4
293k
150k
H36M + 3DPW-tr
56.4
24.0
316k
160k
H36M + 3DPW-tr + PA
56.2
23.9
478k
270k
3DPW-test
55.8
20.2
34k
30k
Human3.61 6. (Tonescu et al. 2014 as the starting set and then extended it. Here we present additional experiments with combinations of 3DPW-train (von Marcard et al. 2018 and PennAction (PA) (Zhang et al. 2013). see Table
We note in the main paper that OF quality depends on the image resolution. The relatively low resolution of PennAction iZhang el al. 2013) images (see T) partially explains the low performance of fine-tuning using these frames, despite the fact that the amount of samples several times larger than 3DPW. Another reason is the large domain gap between the 3DPW and PennAction images (as can be seen in Fig, 10). By contrast, the gap between the training and testing sets of 3DPW is minimal, which explains why 3DPW-tramt outperforms PA al refinement.
In Sec. 4.3 of the main paper, we speculate that using more data requires larger resources during training. We provide the statistics on the mumber of samples and the training time Table 7 for the 3DPW-tes benchmark evaluation experiments, discussed in Table the main paper. The "#samples" column denotes the number of additional training samples, and the "#iterations" shows the number of the network weights updates needed for model convergence during fine-tuning.
E Temporal context network
In Sec. 3.3 of the main paper, we present the Temporal Context network. takes body regressor's predictions
of sequence of N frames as input and outputs the paramcters an affine transformation y and 8 for the Nth frame. Every prediction (0. 3, π is of size d 24 3 + 10 + 3 85. The sequence of predictions is concatenated, that the size of the input tensor becomes N 1 x d (for each sample in the batch).
19

---

Published in Transactions on Machine Learning Research (02/2024)
Table 8: Temporal context with less single-frame supervision. We report P-MPJPE and Arr for the experiments with the lemporal extension of our method, when less single-frame supervision used. As expected, when less supervision is used, the greater the improvement..
length
1OCO.0%
COCOa0%
COCO
COCO100%
61.3 / 25.8
59.3 / 25.4
58.0 / 25.0
57.1
/ 24.4
61.1
24.6
59.1
24.6
37.9
24.3
57.0
24.0
23.1
22.9
37.7
22.7
56.9
22.6
59.6
21.9
58.4
21.8
37.6
/ 19.7
56.7
19.4
59.6
21.5
58.2
21.5
37.4
19.2
56.7
18.8
Table 9: Unsupervised NN fine-tuning. Initial body predictions are used for consistency according to the optimization of Eq. Evaluation is performed the 3DPW-test set.
P-MPJPE (4)
Accel.Err (1)
BL (COCO100%)
58.2
29.0
3DPW-test
57.4
23.9
3ttain
58.1
27.3
First, the imput tensor is filtered with a ix1 convolution layer along the context axis, with an output of size x This tensor reshaped to a vector and fed to a linear layer 16 d 128. This 128-size feature is mapped to y and 8 by two independent linear layers 128 + 2048. We add to the obtained y, so that the network learns its deviation from O. Finally, we transform the inner activation (of size 2048) of the ResNet's backbone using and 8, according to the Eq. 5f the main paper. For non-linearities, Leaky Relu(O.2). We initialize the weights of the network according to normal distribution (N00 0.01). We found it crucial to keep the parameters perlurbation incurred lo the network's ' output to be loo strong.
close to 1 and 0, respectively; otherwise, we found the
F Temporal context under less supervision
In Sec. 1.2 of the main paper, have shown that the less annotated data use to pre-train the baseline model, the greater the improvement from our method. Following this strategy. we refine the basclines of various single-frame supervision using the temporal context network from Sec. 3.3 in the main paper and Sec. above.
The results can found in Table (it an extended version of Table in the main paper). We see that both the position error P-MPJPE, and the motion alignment Accel.Err decrease with larger temporal context, and the effect is stronger when less data is used for supervision.
G Alternative ways of using OF
G.1 Unsupervised fine-tuning
In Sec. 3.2 of the main paper, we assumed that the annotations used to pre-train our baseline were still available when refining it using the unannotated video sequences. This may not always be the case. In such casc, propose alternative approach to preventing the network weights from drifting too far away from their original values. Instead of minimizing the loss of Eq. 4 of the main paper, we mininize
T= 0o 0112 3 B11 AorLor
where 0 and 9 are the SMPL shape parameters introduced Sec. 3.1 of the main paper and the subscript denotes their initial values. We report the results of this alternative approach in Table g where we use the 3DPW-test set for evaluation.
20

---

Published in Transactions on Machine Learning Research (02/2024)
Table 10: Unsupervised optimization of 3DeW-test body sequences with Eq. 8
All sequences are optimized sepa-
rately. Evaluation 3DPW-test set.
P-MPJPE (4)
Accel.Erros (J)
BL (COCO100%)
58.2
29.0
OF
58.0
18.2
OF + smoothing
57.9
13.3
Since our refining method is self-supervised, we can use the test videos for training (first row of Table 9 This regime is the so-called "test-time optimization". We see an improvement in position error, yet the acceleration error relatively high (compared with the semi-supervised fine-tuning of Sec. 4.2). This shows that the pre-trained learning model capable of local improvements, yet the "hard' anchoring of Eq. g does not allow the model weights go too far from their initial state, hence they get trapped in local oplimum.
If the unlabeled videos are taken from the training set (last row of Table 9 we marginal improvement in both metries. The over-reliance on the initial weights in Eq. 8 prevents generalization to the test set. Note, however, 1hat we already saw the significant effect from the 3DPW-train set. in Table 0 when single-frame supervision was used as anchoring.
G.2 Body sequence optimization
As the network we use to regress body pose and shape in its basic configuration takes a single frame at a time and does not contain any recurrent modules, the output body sequence is very shaky.
Instead of fine-tuning the network, one can refine the body sequence predictions directly to make the sequence realistically smoother using OF-guidance (Eq. 2 of the main paper). Specifically, the SMPL parameters of the estimated sequences optimized explicitly, using the loss from Eq. g anchored to the initial estimates. A similar experiment was carried out in (Tung et al. 2017), yet they used multi-camera setups, and silhouette and keypoint estimators. We show that without bells and whistles, using just well pre-trained single-frame baseline and an off-the-shelf OF estimator, the results can be satisfactory. We optimized all video sequences separately, and report the average results on 3DPW-test in Table I0 It is also useful to add smoothing as a post-processing (second row Table 10). To do that, we added an additional term to Eq. S pulling the parameters and B their moving average (we found the best window size to be 30 frames).
We found that the human motion consistency improves considerably, while keeping the position error close to the initial.
21


Reviews:
[{"review_id": "sJnPPUDpMW", "reviewer": "Reviewer_LJLC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- the proposed approach is novel. It is clearly and sensibly motivated, and the components are described\n- the specific idea of using optical flow to directly regularise the vertex displacements of a SMPL model is straightforward and elegant\n- the method works mainly in the setting of image-to-pose, but there is also an extension presented for the video-to-pose setting, requiring only small modifications (and still using the original per-image network without retraining)\n- the evaluation is detailed and largely comprehensive, covering the proposed method and one self-supervised baseline (as well as their combination), across different supervision levels\n- the evaluation uses different video datasets for the self-supervised part, and shows that increasing amounts of video data from diverse sources improves the model performance\n- the writing is generally very good – clear, literate, and pleasant to read; there are very few typos\n\nWeaknesses\n- saying (relatively old) TexturePose is only relevant competitor because others rely on keypoints is a little hopeful – in particular one could use such a method with 2D keypoints predicted from a pretrained model (in much the same way that the proposed work uses optical flow predicted from a pretrained model)\n- 4.1 says use a resnet because newer, fancier models require lots of training data – but the method is explicitly sold in the introduction as improving on the best (\"our approach can even improve the results of state-of-the-art networks\")\n- also the statement in the introduction that \"Our method delivers these benefits no matter what network architecture is used\" seems to be contradicted in 4.1. Either should actually show it working with various recent architectures, or should not make this claim\n- the part on prediction-from-video is somewhat 'detached' from the rest of the work. There is plenty of literature on this task, yet the proposed approach simply presents a straightforward addition to the per-image model that allows it to work on video. Instead, if prediction from video is a goal, I would have expected authors to use an alternative, spatiotemporal network (e.g. a 3D CNN) for this setting, and to show that when trained with the proposed self-supervised loss, performance is better than without\n- all the experiments use datasets that actually have keypoint annotations available; hence the argument that the method is important in the 'practical' setting of limited data is weak. The paper would be much stronger if it showed (or at very least described) a use-case where there is much more limited ground-truth available, and this ground-truth is harder to obtain (in contrast with humans where pretrained models can create respectably good pseudo-labels for in-the-wild datasets).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work considers the task of pose and body-shape estimation (i.e. estimating SMPL parameters) from images. It proposes a new training method incorporating self supervision from unannotated video data. Here a standard, pre-trained optical flow model (RAFT) is run on the video frames, and this estimated flow is used to regularise the predicted poses from adjacent frames to be consistent. The proposed method is demonstrated using a CNN backbone with a mixture of labelled and unlabelled data from several sources; the self-supervised training is shown to significantly improve on supervised training on the labelled data only, and also to improve on another self-supervised method.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- see the points under weaknesses – in particular, some of the claims need adjusting (or, preferably, evidencing properly)\n- give some more convincing motivation (e.g. find some genuinely-low-data scenario where the method is useful and performs well)\n- typo – p4 – \"visibilty\"", "broader_impact_concerns": "None -- this work does not raise any signficant concerns (nor is there any broader impact statement present)", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "1JNSLA2a6q", "reviewer": "Reviewer_C5dk", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n- The presentation and writing are exemplary. The method is very well motivated, explained and justified. Code is provided in the supplement so reproducibility is not a concern.\n- Using optical flow methods as an additional source of supervision is well motivated, and this work implements this idea effectively. The novel technique can be applied as an additional loss term for any 3D pose estimation method that outputs SMPL parameters.\n-  The proposed approach works for single images or videos at test time. This is possible because of a small MLP that is trained to output an affine transform for the current output of the image encoder, given N previous SMPL parameter estimates. Table 3 empirically validates the effectiveness.\n- The proposed approach decreases pose estimation error, especially in the low data regime. Adding additional unannotated data further improves performance.\n\n*Weaknesses*\n- Despite the motivation to improve 3D human pose estimators using unlabelled video data, the proposed approach is demonstrated to be effective but lags behind the state of the art. It would be a great improvement of the paper if the proposed technique for supervision is shown to positively benefit state of the art methods.\n- When 100% of labeled 2D data is available, the relative improvements appear to be marginal. The authors explain that the proposed method is still useful because of licensing and privacy restrictions. It would be beneficial to be explicit here.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a novel technique for training 3D human pose estimation methods by supplementing labelled 2D data with unlabelled videos of humans. The approach works by first taking consecutive frames of videos, computing two sets of 2D pixel motion for the human (1) with an off-the-shelf optical flow method and (2) by projected mesh vertices of a 3D pose estimator outputting a SMPL model. Given these two sets of estimates, a loss is derived to encourage the visible projected vertices to follow the optical flow estimation.\n\nThe quality of using optical flow as a source of supervision is validated through analysis (Sec. 4.6) and empirically by showing improvements in pose estimation error when added to a baseline pose estimation method. The positive impact of the proposed technique is greater when there is less labelled 2D data available.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To improve the justification of the proposed technique, I suggest the authors be precise about the sources of 3D pose estimation training data that are available and are unrestricted because of licensing problems, and potentially demonstrate how their technique practically allows for these restrictions to be overcome.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9IvDDnMXsv", "reviewer": "Reviewer_GTP2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Paper strengths \n\n+ The paper is well written and easy to follow. The related work is complete and it is clear where the paper advances over the previous work. In addition, the method is well written and clearly explained. \n\n+ The paper shows promising results on a standard benchmark. Without relying on labels, it can show better performance than the previous work that uses additional amount of information for training. \n\n+ The experiment of including additional datasets shows good generalisation capabilities. \n\n\nPaper Weaknesses\n\n+ The proposed method lacks novelty. It does not present a new methodology. However, it combines existing approaches in a novel way and this at least adds a solid contribution to the paper. \n\n+ The paper is based solely on the SMPL model. It would add value to the paper to discuss or present an ablation study where the approach works with another body pose and shape model. \n\n+ The paper focuses mainly on comparison with TexturePose and does not include other baselines and previous work for comparison. As the direct related work for comparison is limited, more baselines could be formed to conclude on the performance of the paper. \n\n+ Table 2 would be more helpful if it included the additional number of training images. It could also include the additional training time to have an overview of the cost of the additional performance. \n\n+ It is important to include more evaluations than the 3DPW test set.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an approach to 3D human pose and shape estimation in the absence of annotated data. It is assumed that video information is available to exploit the optical flow information. A deep neural network based on the SMPL model is pre-trained on the available data to be used later to extract pseudo-labels of shape and pose from the unlabelled videos. The optical flow is then used to propagate the SMLP model prediction from a previous frame and apply a consistency loss. In addition, the available pre-training data is used as another source of supervision. The approach is mainly compared with previous work on TexturePose on the 3DPW dataset.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The experimental part should be further worked based on the above comments.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall all reviewers found the paper to be of sufficient interest to the TMLR community and a useful addition to the literature so I am happy to recommend acceptance.
Decision: Accept (poster)