Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery
Authors: Yongwei Nie
Mingxian Fan
Chengjiang Long
Qing Zhang
Jian Zhu
Xuemiao Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: Human Mesh Recovery; Meta-Learning; Test-Time Optimization; Training Optimization
Primary area: deep_learning_architectures

Abstract: Human Mesh Recovery (HMR) is the task of estimating a parameterized 3D human mesh from an image. There is a kind of methods first training a regression model for this problem, then further optimizing the pretrained regression model for any specific sample individually at test time. However, the pretrained model may not provide an ideal optimization starting point for the test-time optimization. Inspired by meta-learning, we incorporate the test-time optimization into training, performing a step of test-time optimization for each sample in the training batch before really conducting the training optimization over all the training samples. In this way, we obtain a meta-model, the meta-parameter of which is friendly to the test-time optimization. At test time, after several test-time optimization steps starting from the meta-parameter, we obtain much higher HMR accuracy than the test-time optimization starting from the simply pretrained regression model. Furthermore, we find test-time HMR objectives are different from training-time objectives, which reduces the effectiveness of the learning of the meta-model. To solve this problem, we propose a dual-network architecture that unifies the training-time and test-time objectives. Our method, armed with meta-learning and the dual networks, outperforms state-of-the-art regression-based and optimization-based HMR approaches, as validated by the extensive experiments. The codes are available at https://github.com/fmx789/Meta-HMR.

Incorporating Test-Time Optimization into Training with Dual Networks for Human Mesh Recovery
Yongwei Nie', Mingxian Fan', Chengjiang Long", Qing Zhang Jian Zhu", Xuemiao Xu!
South China University of Technology, China
Meta Reality Labs, USA
Sun Yat-sen University, China
*Guangdong University of Technology, China
(nieyongwei, @scut, edu. cn, fanmingxiant wttpaigglaa1120@aal..mmmm com
cjfyXxx@mmil..co, zhangq93@mail sysu. edu. en, rockeyzhue163.co
Abstract
Human Mesh Recovery (HMR) the task of estimating parameterized 3D human mesh from an image. There is kind of methods first training a regression model for this problem, then further optimizing the pretrained regression model for any specific sample individually at test time. However, the pretrained model may not provide an ideal optimization starting point for the test-time optimization. Inspired by meta-learning, we incorporate the test-time optimization into training, performing step of test-time optimization for each sample in the training batch before really conducting the training optimization over all the training samples. In this way, we obtain meta-model, the test-time optimization. At test time, after several test-time optimization steps starting from the meta-parametes we obtain much higher HMR accuracy than the test-time optimization starting from the simply pretrained regression model. Furthermore, we find test-time HMR objectives are different from training-ting time objectives, which reduces the effectiveness of the learning of the meta-model. To solve this problem, we propose a dual -newwork architecture that unifies the training- time and test-time objectives. Our method, armed with meta-learning and the dual networks, outperforms -fate-of-the-art regression- baseed and optimization- based HMR approaches, validated by the extensive experiments. The codes available http:///thhbb.ooo////////////e
meta meraa-er of which is friendly to
Introduction
Human mesh recovery (HMR) from a single image is of great importance to human-related applica- tions, such action capture without MoCap device, action transfer with vision-based system, and VR/AR entertainments, etc. This topic has received extensive research during past years, for which most of previous approaches represent 3D human mesh by the parametric human model SMPL (39] with parameters e (0, B), where encodes the pose of the mesh and / describes the body shape. The aim is thus estimate Ô of human given image.
Originally, the problem solved by optimizing a standard human mesh so that its 2D projection
matches the 2D joints of the target human (c.g., SMPLify (5)). Later, works of [23 33 58 37 g
67 73 (66] propose end-to-end networks trained on large datasets to directly output a 3D SMPL
mesh given an input image. In SPIN (28), the regression- based approach [23] and optimization-
approach SMPLify [5] are combined together by interleaved training, where the regression method
"Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

provides an initial solution for optimization and then the optimization method provides supervision for regression. Different from SMPLify [5] that directly optimizes a 3D mesh, works of EFT [22] and BOA 17 finetune pretrained regression model on each test sample, which indirectly optimize the target human mesh (i.e., the the outcome of the regression models).
In this paper, we are particularly interested in the test-time optimization approaches of EFT [22] and BOA [17]. Since BOA is developed for video leveraging consistency properties between frames, we mainly discuss and compare with EFT that performs HMR for images, like ours. Our observation is that the test-time optimization is analogous to one-shot learning. That is, it finetunes a pretrained model on specific sample before really applying the model to solve the human mesh recovery task for that sample. However, the pretrained model, which is not specially tailored for the one-shot learning problem, may not be so effective for the test-time adaptation as desired.
Based the above analysis and inspired by Kim et al. [2 incorporate the test-time optimization into the training process, re- re-oouuula the test-time optimization from the perspective of learning to learn, i.e., meta learning [14]. Specifically, given batch of training samples (or saying set of tasks), our method first performs test-time optimization on each sample to update the regression network parameters temporally for that sample. Then, based on all pieces of regression parameters after test-time optimizations, we further optimize the training-time objectives over the whole batch of training samples. By performing training- maining-time optimization after test-time optimization, we imagine that the training optimization works as faithful supervision to correct the wrong optimization directions of the test-time optimizations. After the training, the obtained parameters of the regression network can be viewed as meta- meraa-aaaaeteess which will be instantiated to parameters actually used for human mesh recovery through several test-time optimization steps.
We find that the test-time objectives for the human mesh recovery task are different from training-time objectives, because we sometimes have ground-truth human meshes training time but forever not at test time. This may produce an obstacle in the meta-learning process, since the optimization directions of the test-time and training optimizations are not identical. To alleviate this problem, we design dual-network. structure to implement our method, which owns main regression network and an auxiliary network. The auxiliary network provides the main network with pseudo ground -tuuth SMPL meshes, by which we unify the training and test-time objectives elegantly.
We demonstrate through extensive experiments that our method equipped with meta-learning and the dual networks greatly outperforms state-of-the art approaches. To summarize, our main contributions three-fold: (1) We propose novel dual-network HMR framework with test-time optimization involved into the training procedure, which improves the effectiveness of the test-time optimizations. (2) We ensure the test-time objectives identical the training objectives, further facilitating the joint-training of the test-time and training-time optimizations. (3) Extensive experiments validate that our results outperform those of previous approaches both quantitatively and qualitatively.
2 Related Work
Regression-based HMR methods typically employ neural networks to regress the human body mesh
representation from images. Methods of (26 48 45 72
23
42
63
68
[6] 13 36 65|
35
34
choose to regress parametric human body model, i.e., SMPL 39 HMR 23 was the first employing
CNN [9 to extract features and MLP layers to output 3D mesh parameters. Later, sophisticated
networks were proposed for improving the reasoning accuracy. For example, PyMAF (72 71
extracted features in a pyramid structure and iteratively aligned 3D vertices with human body in the image. Xue et al. [66] used a learnable mask to automatically identify the most discriminative features related to 3D mesh recovery. Works of [27 33 58 44] observed that prior works overlooked
the importance of camera parameters. Among them, CLIFF (33 innovatively considered using the
cropping bounding boxes input to reduce the ambiguity of reprojection loss. Zolly (58) considered the camera distortion produced by perspective projection. Recently, Nie et al. 44 proposed a
Rol-aware feature extraction and fusion network, guided by camera consistency and contrastive loss
functions tailored to the multi-Rot Molti-RRlooa setting.
There are also non-parameterized methods directly regressing mesh vertices. For example, METRO 37 utilized Transformer to model the global relationship between human keypoints and mesh vertices. FastMETRO [8] separated backbone features from the features corresponding to keypoints

---

and vertices. Recently, 667 combined pyramid structure in PyMAF 71 with the Transformer- based HMR regression method, further improving the regression accuracy.
The regression model is a key component in our method. In this paper, We test HMR 23 and CLIFF as the regression network in our method.
33
Optimization-based HMR methods [7 30 16 18
22 52 usually attempt to estimate
3D body mesh consistent with 2D image cues. Bogo al. [5 proposed an approach called
SMPLify, which iteratively adjusts SMPL parameters to fit detected 2D keypoints. SPIN [28] combined regression- based methods with SMPLify in interleaved training strategy. CycleAdapt [3] alternately trained HMR network and motion denoising network to enhance each other. Unlike SMPLify, EFT [22] and BOA [17 fine-tuned pretrained regression network via 2D reprojection loss or temporal consistency loss at test phase, updating the SMPL parameters indirectly. Some approaches proposed learning stronger 3D) priors [29 47
12 4
utilizing trainable neural networks 53 Inverse kinematics (IK) has also been
to update parameters in lieu of gradient updates [69
explored. These methods address IK problems by decomposing relative rotations [32], designing networks that integrate forward and inverse kinematics [3I] or incorporating UV position maps [51]. Different from all the above, our method integrates test-time optimization into the training process, obtaining a meta-model and meta-parameters.
Meta Learning Our method is most related to the model-agmostic meta- weaanleg (MAML, more precisely FOMAML) [14]. MAML first samples a number of tasks, then performs local optimization on each task, and finally conducts global optimization to update the parameters of the original network. Similarly, our method first executes test-time optimization on each training sample and then performs training optimization on a batch of training samples. Although our method is inspired by MAML and its extensions [2] 49, 55) our goal is fundamentally different from theirs. The goal of MAML is usually for few-shot learning or domain adaptation, which assumes there is ground-truth labeled data in the target domain. In contrast, our goal is to adapt the network to a single test sample which is free of ground-truth human mesh. We have noted MAML has been proven effective in various domains, such as talking head generation (70), SVBRDF recovery [75] [5]. image super resolution [4)). etc. As far as we know, the work of Kim al.
(24] is the first that applies meta learning to HMR. The key difference is that that Kim et al. 244 used the 2D reprojection loss only (please see Eq. in their paper) in both inner and outer loops of meta learning, while we incorporate the ground-truth 3D SMPLs into the outer loop of meta learning and additionally generate pseudo SMPLs and incorporate them into the inner loop of the metal learning. The utilization of the GT and pseudo SMPLs greatly improves the results of our method upon the method of [24].
3 Our Method
Our goal is estimate 3D SMPL human mesh parameterized by θ (0,3) together with a camera т from a given image I of a person, where 0 E IR 24x3 and 3 E 18 10 are pose and shape parameters of the SMPL human model [39], respectively. Let {I.j. θ.,, Jr,,
B.M == be training dataset, where
I, is a training image, ,,,, is the ground-truth (GT) human mesh, J.., is the 2D GT joints (or joints detected by such as OpenPose (7]) of the human in the input image, B is the number of batches, and M is the batchsize. End-to-end HMR regression approaches usually train a neural network fw: Lig - (Oig; (i by minimizing the following training-time loss function Lirain:
M
W = arg min
Σ
Cirasn (fw (Is,s), Gu,,
(1)
1=
where
Efrain(w(In Jom) LaD(fw (I. ((d) LSD (L.)),
J..)
(2)
with
J.,1 LSD (I.,) -xx X( (3) As seen, firain is composed of 2D reprojection loss L2D and a 3D loss LaD. The 2D loss first projects mesh Oi,j to the 2D plane by the camera π and then computes difference between the projected 2D joints and the given 2D joints Jı.- The 3D loss computes difference between 3D meshes, where X can be an identity transformation or transformations computing 3D human joints or mesh vertices from the mesh parameters.
L2D (f
3

---

The tt betch Images: J
Towards Winata
Towards Meett
fu
fw
fu
fw
fu
Texs- Timee Airi Frddion
Pseudo
SMPL Tanaer
SMPL
Johnts
Gradient Descent
Gradient Descent.
Grzdient Desccun
Wildd
E
Serain
Estimated
GT
GT 2D
SMPL
SMPL
Joints
Outer
E
Ceeani
Creatt
Patuda
GT
GT 2D
Juints
Training Optimization
Figure 1: Overview of the dual-network meta-learning HMR method, composed of main HMR regression network fw and an auxiliary network fu- Both networks have the same architecture but different parameters. Given 7th batch of images, test-time optimization is first executed for each training image L in the batch individually, updating fw to fw;,, by performing gradient descent step w.r.t. the test-time loss function Liest-u- Then based {fw:,, E M}} (M is the batch size), the training optimization is executed to update the parameters of both main and auxiliary networks by Ltrain with different arguments respectively. Wmete and Umeta are the finally generated meta-parameters. fu generates "Pseudo SMPLs" that are used in the test-time loss to supervise the learning of the "Estimated SMPL Inner". GT SMPLs are used in the training loss to supervise the learning of "Estimated SMPL Outer" and the Pseudo SMPLs.
3.1 Test-time Optimization
Exemplar-Fine- Tuming (EFT) 22 was the first work proposing test-time optimization for HMR. In particular, the pretrained network Wre is further finetuned by performing the following test-time optimization loss function on a specific test sample I,
w = arg min Liest W.,
Crnst(Sws, (Sws.s (Ir.)) Jes), initially Wi.j = W pres
with
C2n Crest (fw (f..g) Can (I.
(5)
The test-time optimization starts from the initial solution Wpre provided by the pretrained model. It resembles one-shot learning but actually does not, because the pretrained model is obtained using normal supervised learning techniques while not introducing any strategy for guaranteeing the properties of one-shot learning. The parameters of the pretrained model may be not ideal the starting point for the test-time optimization.
3.2 Incorporating Test-time Optimization into Training
To solve the above problem, we propose to integrate the test-time optimization into the training procedure as shown in Figure I inspired by optimization- based meta-learning [14]. Specifically, for each sample in a batch, we first perform test-time optimization on that sample to update the parameters of the regression network temporally corresponding the sample. After that, we perform training optimization over all M training samples, based on the M temporally updated regression

---

Algorithm Meta learning of Dual networks for 3D Human Recovery
The stage of training
Require: Training dataset [I.g. J.
J. ا=
B.M B.M
Require: fu: main and auxiliary networks, randomly initialized
Require: a,8: step size hyperparameters
1: while not done do
2: Sample batch of images (I. E
U(I
U uniform distribution
3: for all It do
4:
Compute SMPL meshes by fw (I. and fu(I.,), respectively
5: Compute Lust-u in Eq. 9. evaluate V. Lrest-u. and update w'. + a VVw List--u
6: end for
7:
Compute Liain in Eq.
8
evaluate V Ii and update w <- w BE V "w.s Litain
wh, is used in I
8:
Compute Laain in Eq. 8
evaluate Vuliam and update u f u BE, V Vuluan
9:
end while
10:
meta
W,
Umeta
The stage of testing
Require: Input image I
Require: Main and auxiliary networks with meta parameters Wmetn and Umeta
1: w Wmeta
2: for = 1 to do
> Iterate test-time optimization times Using frozen auxiliary network Given input I, the pseudo GT mesh, and 2D joints
3: Pseudo GT mesh + fun. (I)
4: Compute Lisst- using Eq.
5: Evaluate W
6: Update
V Trese.u
7: end for
8: Wfinal
9: Compute SMPL mesh by fwam (I)
networks. This process is formulated as:
B
M
Wmeta arg min w
Crrain (fwi, (Lij), O.y. J.y),
(6)
1= 1=1
where,
w aV Crest (fw (Ii)), Jzy).
The difference between Eq, and Eq. iis in the parameters of f to optimized. Instead of directly optimizing the current parameters of using the training objective Ltrain, we first perform step of test-time optimization using Eq. 7 on each sample I, to obtain network parameters w', specific to that sample. Then, ( (w's E [1, MT over all training samples in a batch are in turn used in Eq. is the learning rate of the test-time optimization.
evaluate the training objective. In Eq.
By performing test-time optimization before training optimization in Eq. and 7 we take test-time optimization into consideration in the training procedure. That means, the "test-time optimization" is trained on the training dataset, thus having better generalization ability to test samples. The proposed method resembles the pptimization-base meta-learning [14 and we call the obtained parameters Wmeta meta-parameters.
3.3 Unifying Training and Test-time Optimization Objectives with Dual Networks
There are ground-trui truth human meshes at training time while not at test time, causing the difference between Lirain (see Eq, 2 and Ltest (see Eq. (5.. Since both the test-time and training optimizations update parameters of the same network, the difference between the two optimization objectives yields different gradient descent directions, causing potential conflicts that reduce the effectiveness of the training (see ablation studies in Section 4.4

---

To make the test- test-time optimization more compatible with the training objective, we propose method that unifies the training and test-time optimization objectives by introducing an auxiliary regression network fu parameterized by u which is trained together with the main network:
M ΣE
(Cirasn (Jw,,, Or.s), + Ciram (I.s) 3...) (8)
W Wmefa, Umeea
arg mriin
The above equation is combination of Eq. and Eq. 6 (superscript and are used to denote the first and second term respectively), with Eq. applied to the auxiliary network fu. and Eq. 6 applied to fw- We use Ju to generate pseudo GT mesh θμ, for training image Iij. i.e., 6%, (Tgg) and use the pseudo label to supervise the gradient descent in the test-time optimization:
w a ww (I.,); 6% Jt.))-
(9)
Please compare between Eq. and Eq. The difference is that there is an additional input @ to Liest-u u in Eq. [ and note that this new Torm of Ctest-u u is identical to the form of Lirain.
3.4 Inference with Dual Networks
Both the training and testing pseudo codes of our method are given Algorithm I The training process is fully elaborated in the above sections. Now we introduce how to perform inference at test time. For each test sample I, our hand are two networks fw and fu with w Weeta and Umeta. respectively. We freeze the parameters of the auxiliary network, and use it to compute the pseudo GT human mesh for the test image Then, under the supervision of the pseudo mesh, we compute ftest-u u and use Eq. to iteratively update the parameters w of f from mest to final- We run at most m = 14 iterations, and automatically stop the iteration losses of two consecutive iterations are close enough. We finally use fwijinal to estimate the human mesh for image I.
3.5 Implementation Details
We implement the main network fw and auxiliary network fu with the same network architecture
but different parameters. Specifically, we use HMR [23 or CLIFF (33 as f due to their simplicity. The two methods and many other approaches (37 8 66 148 [54] to extract features from the input image, and estimate human mesh based on the features. We provide results of both kinds of backbones.
72 71 26 67 adopt ResNet-50 [9] or
We implement our method in PyTorch using the Adam optimizer (2 with 31 = 0.9 and B2 = 0.999.
The batchsize for ResNet backbone is 40, and for HRNet backbone is 30. The number of training epochs for ResNet backbone is 65, and for HRNet backbone is 25. The learning rate a used in the test-time optimization is le-5, and the learning rate 3 (see Algorithm TT for the training optimization is le-4. Our method takes about days training on a single NVIDIA RTX3090 GPU.
Experiments
4.1 Datasets
Following previous work 33 26 73
we employ the following datasets in our experiments: (1) Human3.6M (20). indoor dataset with precise GT human mesh and 2D joints captured through
MoCap devices. (2) MPI-INF-3DHP 4T], another widely used indoor dataset whose GT human
meshes are obtained through multi-view reconstruction. (3) COCO (38 and (4) MPII (1, two in-the-wild outdoor datasets with human annotated 2D joints for which we use the pseudo GT mesh provided by (33 (5) 3DPW [57]. challenging in-the-wild dataset providing accurate human mesh fitted from IMU sensor data.
4.2 Training, Testing and Metrics
Following prior arts 37 66 & 4, we first train our method on mixture of four datasets, including
Human3.66 20), MPI-INF- 3DHP 41, COCO 38 and MPII (I) and then test our method on
The authors Yongwei Nie and Mingxian Fan signed the license and produced all the experimental results in this paper. Meta did not have access to the datasets.

---

the test dataset of Human3.6M, [0]]. After that, we further fine-tune our model for 5 epochs by introducing the training dataset 3DPW [57), and then evaluate our method on the test dataset of 3DPW (57 We use MPJPE (Mean Per Joint Position Error), PA-MPJPE, (Proerustes-aligned MPJPE), PVE (Mean Per-vertex Error) as the metrics to evaluate our method.
Table I: Quantitative comparison with state-of-the-art methods on 3DPW 157 and Human3.o6 66M 20
T using 2D joints detected by OpenPose 7 "*" using 2D joints detected by RSN 6
3DPW
Humam3.60
Method
Backbone
MPJPET
PA-MPIPEJ
PVEL
MPIPET
PA PA-MPIPEL
HMR
23
18
Res-50
130.0
81.3
$8.0
56.8
PARE
26
W3R-
74.5
46.5
88.6
ROMP
55
21
HR W32
76.7
47.3
93.4
Rogression-base
PyMAF
72
21
HR-W48
74.2
45.3
87.0
$4.2
37.2
METRO
W64
FasMETRO
8 22
W6-
73.5
44.6
84.1
52.2
33.7
CLIFF
HR-W48
43.0
47.1
32.7
LearSSample
66
HR-W33
70.5
43.3
82.7
45.9
33,5
ProPuse
13
W48
68.3
40.6
79.4
45.7
29.1
POTTER
13 23
VIT
75.0
44.8
87.4
56.5
35.1
DeFormer
17
23
72.9
44.3
82.6
44.8
31.6
LeamedGD
53
20
55.9
56.4
Opeimization based
HUND 669
Res-50
81.4
$7.5
69.5
$2.6
SPIN 28
21
Res-50
96.9
59.2
116.4
62.5
41.1
EFT [2] 21
Res-50
85.1
52.2
98.7
63.2
43.8
Res-44
NIKI [3] 23
HR-W48 W48
71.3
40.6
86.6
ReFit [5]
HR-W48
41.0
32.2
PLIKS '23 '3
HR W48
66.9
42.8
82.6
49.3
34.7
OusScLiFpt OuscIIFF*
HR-W48
62.9
39.7
30.3
HR-W488
62.4
39.5
78.1
42.0
29.1
4.3 Comparison with Previous Approaches
Quantitative results. We present accuracy comparison with SOTA methods Table [ including prreession-baeeee approaches 23
72 g 66) and optimization- based For all the compared approaches, we report the best results their papers provide. For method, we adopt CLIFF B3 as and HRNet-VV W48 as the backbone network. Since our method needs 2D joints for test-time optimization, we report results using joints estimated by OpenPose 7 (denoted by 1) and RSN 6 (denoted by *).
26
55
37
33
13
73
67
approaches [28] 22 32 59
31
51
53
69
Please compare "OurSCLIFFF (ER)"W8)" Table with SOTA approaches that also use HRNet- W48 as backbone. Our method outperforms most of previous approaches. Compared with 333 improve it from 69.0 to 62.9 taking MPJPE 3DPW as an example, which is large margin. using RSN joints for the test-time optimization, our method can further improve the metrics.
Qualitative results. We show qualitative comparison with CLIFF [33 and Refit [59 in Figure 2 Our method estimates faithful human poses and meshes which are better than those of the compared approaches. Comparisons with HybrIK [32 NIKI [1] ProPose the supplementary material.
[3 and EFT EFTCLIFF can found in
4.4 Ablation study
Influence of Regression Model. The adopted regression model influences the effectiveness of our method. In Table 2 show the results of using HMR (23) or CLIFF 33 as the regression model. Since CLIFF is stronger baseline than HMR, our method based on CLIFF performs better than that based on HMR.
Influence of Accuracy 2D Joints. Since our method needs 2D joints as the supervision at test time, it is interesting to see how the quality of the 2D joints affects the effectiveness of the method. We have already repported results on 2D joints detected by OpenPose 7 and RSN (6) In Table 2 we further test GT 2D joints. As seen, our method with GT joints outperforms our method using detected joints by OpenPose and RSN. This indicates our method will become more effective as 2D pose detectors continue to develop.

---

Ours+ Novel Figure 2: Qualitative comparison with SOTA methods. We show results produced by CLIFF [33] ReFit (59] and our method (f: OpenPose, RSN). All the three methods use HRNet-Wal as the backbone. In the novel views, green represents the ground truth, orange represents CLIFF, purple represents ReFit, pink and blue represent the two variants of our method, respectively.
Input
CLIFF
Ourst
Ours+ Novel view
Input Refit
Oursf
Table 2: Ablation study regression model and 2D joints on 3DPW 57
and fluman3.6M 20
"f" using 2D joints detected by OpenPose 17]. "o" using GT 2D joints.
3DPW
Human3.6M
Method
Backbone
MPJPF|
PA-MPIPEL
PVE
MPJPFL
PA-MPIPE
Res-50
73.3
44.3
90.3
55.8
36.4
Oussmmn
Res-50
68.9
39.6
85.5
53.7
33.8
OuscLieri
W48
O
HR.WWE W48
57.8
35.3
74.4
39.4
27.5
Oursgaat
EFTCuPF
OurccuIE
100
62
1.8e-01
97
1.4e-01
94
PA-MPJPE (mm)
59
g
11.0e-01
76
56-
time
6.0c-02
MPJPE
73
2.0c-02
70
04
67
5.0c-04
64
3.0e-04
61
38 0
I 1.0c-04
12
Optimization steps
Optimization steps
Optimization steps
(a)
(b)
(c)
Figure 3: Influence optimization steps during inference. Our method outperforms EFT when
using the same regression model. As optimization proceeds, our results continuously become better, while those of EFT become better at first and then become worse (see (a) and (b)). (c) shows that our
method achieves faster convergence compared to EFT.
Influence of Optimization Steps at Inference Time. At inference time, we perform at most m test-time optimization steps. In Figure (a) and (b), we show how the evaluation metrics become as the number of optimization steps increases. As seen, our results consistently become better in terms of both MPJPE and PA-MPJPE. We also show the results of HMRT and EFTCLIFF. With the same regression model, our method is better than EFT 22 The results of EFT become better the first few optimization steps, but become worse more optimization steps execute. This is probably because EFT is only finetuned with 2D reprojection loss and is more sensitive to the errors the joints. At first several optimization steps, the estimated 3D SMPL approaches the joints from relatively distant initialization, therefore the result gets better gradually. With more optimization steps, the SMPL may overfit the 2D joints whose annotation-errort then distort the SMPLs, thus yielding worse evaluation metrics. In contrast, our method is guided by both 3D and 2D

---

Input
Step O
Step
Step
Step
Step
Final
Figure 4: Stepwise visualization. From left to right, we showcase results after different steps of test-time optimization during testing.
Table Ablation study on meta-learning and auxiliary network. Models are trained on COCO 35
and tested on 3DPW [57]. "Test. Opt." is "test-time optimization".
Model
Integrating Test. Opt. (meta-loarming)
Auxiliary Net
MP/PE
PA-MPIPE
HFTeLiF
84.6
54.2
OUSCcLIER
78.5
49.9
OuSSCLIFP
76.7
49,5
Table 4: Quantitative comparison with EFTcLIFF on the LSP-Extended dataset (21).
Table 5:
Quantitative comparison with EFTcLIFF on the Human3.6k dataset (20
Method
2D Loss
Mehhod
MPSPE4
PA-MPJPEJ
EFTcLiPF
8.3c-3
EFTCLiFF
85.5
$1.0
Ourecrave
6.1e:3
Ourecrape
83.8
48.6
supervisions. The 3D pseudo SMPLs plays the role of regularization that mitigates the influences of errors in 2D joints (see more explanations in the supplemental material), In Figure (c), present the loss curve of the test-time optimization of EFT and our method. Our method converges in about 6 steps, demonstrating faster convergence compared with EFT.
Figure 4 shows our meshes after different optimization steps. Initially, the mesh does not fit with the target human. After more steps, the mesh progressively deforms itself achieve perfect fitting.
Influence of Meta-Learning and Dual Networks. We propose meta-learning to improve the perfor- mance of test-time optimization. We also introduce auxiliary network unify the formulation of test-time and training optimizations, and hope this can reduce the conflict in training and improve the estimation accuracy. We conduct ablation study to validate the two components as shown in Table 3 We train all the models in the ablation study on COCO 38 and test them on 3DPW (57.. The first row in Table shows results of training with test-time (Test.) optimization (Opt.) and no auxiliary network, i.e., CLIFF 333. The second row shows our method without auxiliary network. The third row shows our full method. As shown, the introduction of the meta-learning and auxiliary network strategies both improve the evaluation results.
Out-of-Domale Adaptation. Test-time optimization performs post-processing on each test image and has the ability of adaptation to out-of-domain data. To comprehensively validate our method's effectiveness in out-offdistrrbutien with EFT CLIFF on the -SP-Exxended dataset, as shown in Table 3 (Training dataset: COCO, MPII, MPI-INF-3DHH Human3.6M M. 3DPW, Backbone: HR-W48). Since LSP provides ground truth 2D joints but not GT SMPLs, the comparison is based on 2D loss relative to the GT joints. Our method achieves lower 2D loss than EFT with results of 6.1e-3 (our) versus 8.3e-3 (EFTcLIFF), indicating that our method is more accurate in approaching the GT joints.
(OOD) scenarios, first conduct quantitative comparison
To further evaluate the OOD performance, dataset (an outdoor dataset) and test them on the Human3.6M dataset (an indoor dataset). With ground truth SMPLs available, report results using MPJPE and PA-MPJPE metrics. As shown
train both our method and EFTCLIFF on the COCO

---

Ours Figure 5: Out-of-domain adaptation. Please the depth order of the arms where EFTCLIFF fails to infer. Our method correctly identify the correct configuration.
Input
GT Joints
EFTcLIFF
in Table our method demonstrates superior performance over EFTCLIFE, further validating its effectiveness in OOD scenarios.
Figure 5 shows two examples from the LSP-Extended dataset. The persons in the two images take complex actions. The shadows in the images and the similar color of the black pants and shoes make it difficult even for humans to identify the configuration of the 3D meshes. Our method successfully estimates correct meshes, while the arms in the results of EFTcLIFF exhibit wrong depth orders.
Conclusion
To conclude, this paper presents a new training paradigm towards better test-time optimization performance test time. We mainly propose two strategies. First, integrate the test-time optimization into the training procedure, which performs test-time optimization before running the typical training in each training iteration. Second, propose dual-network architecture to implement the proposed novel training paradigm, aiming at unifying the space the test-time and training optimization problems. Experiments and comparisons prove that the proposed training scheme improves the effectiveness of the test-time optimization during testing, demonstrating that successfully learns meta-parameters that benefit the test-time optimization for specific samples. Our method can perform even better with stronger regressor baseline or better 2D joints, and can adapt to -ut-of-domain challenging test cases.
Acknowledgments
The work was supported by Guangdong Provincial Natural Science Foundation for Outstanding Youth Team Project (No. 150400000, the Fundamental Research Funds for the Central Universities (No. 2024ZYGXZR021, China National Key R&D Program (Grant No. 2023YFE0202000, Key- Area Research and Development Program of Guangzhou City NNo.2023B0LJ0022 National Natural Science Foundation of China for Key Program (No. 62237001), Natural Science Foundation of China (No. 62072191).
References
[1] Andriluka, M., Pishchulin, L., Gehler, P., Schiele, B.: 2d human pose estimation: New bench- mark and state of the art analysis. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition. pp. 3686-3693 (2014)
10

---

[2] Antoniou, A., Edwards. H., Storkey, A.: How to train your maml. In: International Conference on Learning Representations (2019)
[3] Bälan, A.O., Black, M.J.: The naked truth: Estimating body shape under clothing. In: Com- puter Vision-ECCV 2008: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part II 10. pp. 15-29. Springer (2008)
[4] Black, M.J., Patel, P., Tesch, J., Yang. J.: Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7726-8737 (2023)
[5] Bogo, F. Kanazawa, A., Lassner, C.,, Gehler, P., Romero, J., Black, MJJ.: Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. In: Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14. pp. 561-578, Springer (2016)
[6] Cai, Y., Wang. Z., Luo, Z., Yin, B., Du, A., Wang, H., Zhang, X., Zhou, X., Zhou, E., Sun, J.: Learning delicate local representations for multi-person pose estimation. In: Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part Springer (2020)
III 16. pp.
[7] Cao, Z., Simon, T., Wei, S.E., Sheikh, Y:: Realtime multi-person 2d pose estimation using part affinity fields. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 7291-7299 (2017)
[8] Cho, J., Youwang, K., Oh, T.H.: Cross-attention of disentangled modalities for 3d human mesh recovery with transformers. In: European Conference on Computer Vision. pp. 342-359. Springer (2022)
[9] Choutas, V., Bogo, F., Shen, J., Valentin, J.: Learning to fit morphable models. In: European Conference Computer Vision. pp. 160-179. Springer (2022)
[10] Dang, L., Nie, Y., Long, C., Zhang, Q., Li, G.: Msr-gen: Multi-scale residual graph convolu- tion networks for human motion prediction. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11467-11477 1176 (2021)
[11] Dang. L., Nie, Y., Long, C., Zhang, Q.. Li. G.: Diverse human motion prediction via gumbel- softmax sampling from an auxiliary space. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 5162-5171 (2022)
[12] Davydov, A., Remizova, A., Constantin, V., Honari, S., Salzmann, M., Fua, P.: Adversarial parametric pose prior. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 10997-11005 (2022)
[13] Fang, Q.. Chen, K., Fan, Y., Shuai, Q., Li, Zhang, W.: Learning analytical posterior proba- bility for human mesh recovery. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8781-8791 (2023)
[14] Finn, C.,, Abbeel, P., Levine, S.: Model-agnostic meta-learning for fast adaptation of deep networks. In: International conference on machine learning. pp. 1126-1135. PMLR (2017)
[15] Fischer. M., Ritschel, T.: Metappearance: Meta-learning for visual appearance reproduction. ACM Transactions on Graphics (TOG) 41(6), 1-13 (2022)
[16] Guan, P., Weiss, A., Balan, A.O., Black, MJJ.: Estimating human shape and pose from single image. In: 2009 IEBE 12th International Conference on Computer Vision. pp. 1381-1388. IEEE (2009)
[17] Guan, Xu, J., Wang, Y., Ni, B., Yang, X.: Bilevel online adaptation for out-of-domain human mesh reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10472-100 10481 (2021)
[18] Hasler, N., Ackermann, H., Rosenhahn, B., Thormählen, T., Seidel, H.P.: Multilinear pose and body shape estimation of dressed subjects from image sets. In: 2010 IEEE Computer Society Conference Computer Vision and Pattern Recognition. pp. 1823-1830. IEEE (2010)
11

---

[19] He, K., Zhang, X., Ren, S., Sun, J.: Identity mappings in deep residual networks. In: Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14. pp. 630-645. Springer (2016)
[20] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3, 6m: Large scale datasets and predictive methods for 3d human sensing natural environments, IEEE transactions on pattern analysis and machine intelligence 36(7), 1325-1339 (2013)
[21] Johnson, S., Everingham, M.: Learning effective human pose estimation from inaccurate annotation. In: CVPR 2011. pp. 1465-1472. IEEE (2011)
[22] Joo, H., Neverova, N., Vedaldi, A.: Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In: 2021 International Conference on 3D Vision (3DV).
pp. 42-52. IEEE (2021)
[23] Kanazawa, A.. Black, M.J,, Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7122-7131 (2018)
[24] Kim, M., Min, Y., Kim, J., Kim, S.: Meta-learned initialization for 3d human recovery. In: 2022 IEEE International Conference on Image Processing (ICIP), pp. 4238 4242. IEEE (2022)
[25] Kinga, D., Adam, J.B., et al.: A method for stochastic optimization. In: International conference on learning representations (ICLR). vol. 5, p. 6. San Diego, California; (2015)
[26] Kocabas, M., Huang, C.H.P., Hilliges, O., Black, M.J.; Pare: Part attention regressor for 3d human body estimation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11 11127-11137 (2021)
[27] Kocabas, M., Huang, C.H.P., Tesch, J., Müller, L., Hilliges, O., Black, M.J.: Spec: Seeing people in the wild with an estimated camera. In: Proceedings the IEEE/CVF International Conference Computer Vision. pp. 1003--10453 (2021)
[28] Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.: Learning reconstruct 3d human pose and shape via model-fitting in the loop. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 2252-2261 (2019)
[29] Kolotouros, N., Pavlakos, G., Jayaraman, D., Daniilidis, K.: Probabilistic modeling for human mesh recovery. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11605-11614 (2021)
[30] Lassner, C., Romero, J., Kiefel, M., Bogo, F., Black, M.J., Gehler, P.V.: Unite the people: Closing the loop between 3d and 2d human representations. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6050-6059 (2017)
[31] Li, J., Bian, S., Liu, Q.. Tang, J., Wang, F., Lu, C.: Niki: Neural inverse kinematics with invertible neural networks for 3d human pose and shape estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2933-112422 (2023)
[32] Li, J. Xu, C., Chen, Z., Bian, S., Yang, L., Lu, C.: Hybrik: A hybrid analytical- inverse kinematics solution for 3d human pose and shape estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 3383-3393 (2021)
[33] Li, Z., Liu, J., Zhang. Z., Xu, S., Yan, Y.: Cliff: Carrying location information in full frames into human pose and shape estimation. In: European Conference on Computer Vision. pp. 590-606. Springer (2022)
[34] Liao, H.R., Lin, J.C., Lec, C.Y.: Progressive hypothesis transformer for 3d human mesh recovery. In: Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision. pp- 6323 6332 (2024)
[35] Liao, X., Zhang, C., Xu, J., Su, W. Chen, Z., Tao, W.: Instahmr: Instance-aware one-stage multi-person human mesh recovery. IEEE Transactions on Visualization and Computer Graphics (2024)
12

---

[36] Lin, K., Lin, C.C., Liang, L., Liu, Z., Wang, L.: Mpt: mesh pre-training with transformers for human pose and mesh reconstruction. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3415-3425 (2024)
[37] Lin, K., Wang, L., Liu, Z.: End-to-end human pose and mesh reconstruction with transformers. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 1954-1963 1963 (2021)
[38] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D.. Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context. In: Computer ViCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer (2014)
[39] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: Smpl: A skinned multi- person linear model. ACM Transactions on Graphics 34(6) (2015)
[40] Ma, T., Nie, Y., Long, C., Zhang, Q. Li, G.: Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp.
(2022)
[41] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.: Monocular 3d human pose estimation in the wild using improved cnn supervision. In: 2017 international conference on 3D vision (3DV). pp. 506-516. IEEE (2017)
[42] Moon, G., Choi, H., Lee, K.M.: Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2308-2311 (2022)
[43] Nam, H., Jung, D.S., Oh, Y., Lee, K.M.: Cyclic test-time adaptation on monocular video for 3d human mesh reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 14829-14839 (2023)
[44] Nie, Y., Liu, C., Long, C., Zhang, Q., Li, G., Cai, H.: Multiple-crop human mesh recovery with contrastive learning and camera consistency in a single image. In: Proceedings the European Conference Computer Vision (ECCV) (2024)
[45] Omran, M., Lassner, C., Pons-Moll, G., Gehler, P., Schiele, B.: Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In: 2018 international conference on 3D vision (3DV). pp. 484-494, IEEE (2018)
[46] Park, S., Yoo, J., Cho, D., Kim, J., Kim, T.H.: Fast adaptation super-resolution networks via meta-learning. In: Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVII 16. pp. 754-769. Springer (2020)
[47] Pavlakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A.A., Tzionas, D., Black, M.J.: Expressive body capture: 3d hands, face, and body from single image. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10975-10985 (2019)
[48] Pavlakos, G., Zhu, L., Zhou, X., Daniilidis, K.: Learning to estimate 3d human pose and shape from single color image. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 459-468 (2018)
[49] Raghu, A., Raghu, M., Bengio, S., Vinyals, O.: Rapid learning or feature reuse? towards under- standing the effectiveness of maml. In: International Conference on Learning Representations (2020)
[50] Rajeswaran, A., Finn, C., Kakade, S.M., Levine, S.: Meta-learning with implicit gradients. Advances in neural information processing systems 32 (2019)
[51] Shetty, K., Birkhold, A., Jaganathan, S., Strobel, N., Kowarschik, M., Maier, A., Egger, B.: Pliks: A pseudo-linear inverse kinematic solver for 3d human body estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 574-584 (2023)
13

---

[52] Sigal, L.. Balan, A., Black, M.: Combined discriminative and generative articulated pose and non-rigid shape estimation, Advances in neural information processing systems 20 (2007)
[53] Song, J., Chen, X.. Hilliges, O.: Human body model fitting by learned gradient descent. In: European Conference on Computer Vision. pp. 744-760. Springer (2020)
[54] Sun, K., Xiao, B., Liu, D., Wang. J.: Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 5693-5703 (2019)
[55] Sun, Y., Bao, Q. Liu, W., Fu, Y., Black, M.J., Mci, T.: Monocular, one-stage, regression of multiple 3d people. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 11179- 11199-1188888111111111111111-111 (2021)
[56] Tan, V., Budvytis, I., Cipolla, R.: Indirect deep structured learning for 3d human body shape and pose prediction. In: BMVC. vol. 3, p. 6 (2017)
[57] Von Marcard, T., Henschel, R., Black, M.J., Rosenhahn, B., Pons-Moll, G.: Recovering accurate 3d human pose in the wild using imus and moving camera. In: Proceedings of the European conference on computer vision (ECCV). pp. 601-617 (2018)
[58] Wang, W., Ge, Y., Mei, H., Cai, Z., Sun, Q., Wang, Y., Shen, C., Yang. L., Komura, T.: Zolly: Zoom focal length correctly for perspective-distorted human mesh reconstruction. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 3925- 3935 (October 2023)
[59] Wang, Y., Danillidis, K. Refit: Recurrent fitting network for 3d human recovery. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 14644-14654 (October 2023)
[60] Xiao, P., Xie, Y., Xu, X., Chen, W., Zhang, H.: Multi-person pose forecasting with individual interaction perceptron and prior learning. In: European Conference on Computer Vision, pp. 402-419. Springer (2025)
[61] Xiao, W., Xu, C., Zhang, H., Xu, X.: Spatial-aware gan for instance-guided cross-spectral face hallucination. In: CAAI International Conference on Artificial Intelligence, pp. 93-105. Springer (2022)
[62] Xu, C., Chen, Z., Mai, J., Xu, X., He, S.: Pose-and attribute-consiistnt person image synthesis. ACM Transactions on Multimedia Computing, Communications and Applications 19(2s), 1-21 (2023)
[63] Xu, X., Chen, H., Moreno-Noguer, F., Jeni, L.A., De la Torre, F.: 3d human shape and pose from a single low-resolution image with sulf-super learning. In: Computer Fision-ECCy 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16. pp. 284-300, Springer (2020)
[64] Xu, X., Li, K., Xu, C., He, S.: Gdface: Gated deformation for multi-view face image synthesis. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 12532-12540 (2020)
[65] Xuan, H., Zhang, J., Lai, Y.K., Li, K.: Mh-hmr: Human mesh recovery from monocular images via multi-hypothesis learning. CAAI Transactions on Intelligence Technology (2024)
[66] Xue, Y., Chen, J., Zhang, Y., Yu, C., Ma, H., Ma. H.: 3d human mesh reconstruction by learning to sample joint adaptive tokens for transformers. In: Proceedings of the 30th ACM International Conference Multimedia. pp. 6765-6773 (2022)
[67] Yoshiyasu, Y.: Deformable mesh transformer for 3d human mesh recovery. In: Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 17006-17015 (2023)
14

---

[68] Zanfir, A., Bazavan, E.G., Xu, H.. Freeman, W.T., Sukthankar, R., Sminchisescu, C.: Weakly supervised 3d human pose and shape reconstruction with normalizing flows. In: Computer VisoonFECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceed- ings, Part VI 16. pp. 465-481. Springer (2020)
[69] Zanfir, A., Bazavan, E.G., Zanfir, M., Freeman, W.T., Sukthankar, R., Sminchisescu, C.: Neural descent for visual 3d human pose and shape. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 14484-14493 (2021)
[70] Zhang, B., Qi, C., Zhang, P.. Zhang, B., Wu, H., Chen, D., Chen, Q. Wang, Y., Wen, F: Metaportrait: Identity-preservie talking head generation with fast personalized adaptation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 22096-22105 (2023)
[71] Zhang, H., Tian, Y., Zhang, Y., Li, M., An, L., Sun, Z., Liu, Pymaf-x: Towards well-aligg well-aligned full-body model regression from monocular images. IEEE Transactions on Pattern Analysis and Machine Intelligence (2023)
[72] Zhang, H., Tian, Y., Zhou, X., Ouyang, W., Liu, Y., Wang, L., Sun, Z. Pymaf: human pose and shape regression with pyramidal mesh alignment feedback loop. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1 11446-11556 (2021)
[73] Zheng, C., Liu, X., Qi, G.., Chen, C.: Potter: Pooling attention transformer for efficient human mesh recovery. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 161 1-1620 (2023)
[74] Zhou, S., Fu, H., Liu, L., Cohen-Or, D., Han, X.: Parametric reshaping of human bodies in images. ACM transactions on graphics (TOG) 29(4), 1-10 (2010)
[75] Zhou, X., Kalantari, N.K.: Look-ahead training with learned reflectance loss for single-image svbrdf estimation. ACM Transactions on Graphics (TOG) 41(6), 1-12 (2022)
15

---

A Appendix
A.1 Inference Speed Comparison with Previous Methods
In Table we provide a comparison with ReFit [59]. NIKI [I], and PLIKS [51] in terms of inference speed. Except for NIKI using ResNet-34, all others employ HRNet-W48, and all tested on a single NVIDIA RTX3090 GPU. OUrSCLIFF takes 0.072s for an iteration, and about 1.1s for 14 iterations in default. ReFit, NIKI, and PLIKS take 0.043s, 0.068s, 0.041s, respectively, which are faster. When our method performs single-step optimization, the time required is comparable to the above three methods. When we conduct additional iterations of optimization, our method consumes more time. As reward, the additional optimizations improve human mesh recovery accuracy upon regression approaches.
Table 6: Inference speed comparison with ReFit [59]. NIKI B1, PLIKS [51]
Method
Inference Speed (per sample)
ReFit
0.043s
NIKI
0.0688
OUSSCLIFF
0.072s (1 iteration)
A.2 Ablation on Learning Rate
In Table we report the experimental results when using different learning rates in our method. We adjusted the learning rates for both the test-time optimization and the ordinary training optimization. Among the feasible learning rates, we observe that utilizing and x 10-1 for the training optimization is the most suitable configuration. We find that our training process is unstable under some combinations of the two learning rates, e-g., when the two learning rates are very different from each other (1e-6 for the test-time optimization and lc-4 for the training optimization), or using too large learning rates (c.g., 1e-3).
10-5 for the test-time optimization
Table 7: Ablation study of different learning rate settings, with COCO 38 as the training dataset and 3DPW (57 as the testing dataset. "_" means the training not stable, and no result is obtained. Gray row is the default setting.
Test time
Training Ir
MPJPE
PA PAIPPE
PVE
1e-6
16-5
lc-4
76.8
49,5
90.1
1e-4
le-4
77.1
49,6
89,1
1e:3
le.4
le-4
le-3
1o-4
lc-5
78.8
50.8
91.0
1c-4
le-6
121.6
74.0
134.5
A.3
Ablation on the Number of Test-time Optimization Steps at Training
In the main paper, we perform just one step of test-time optimization in each training iteration. Here, we explore using more test-time optimization steps during training, and show its impact on the model performance. The results are shown in Table 8 It can be observed that with the increment in the number of steps, there a slight improvement in model performance. However, this comes at the cost of increased memory usage and longer training time. To save training time, opted for single test-time optimization step at the training stage.
A.4 Per-Joint Error Analysis
As shown in Figure 6 we explore the performance gains of OurSCLIFF over EFTCLIFF at each joint. Specifically, we compute the per-joint error of MPJPE and PA-MPJPE, then we subtract the result of EFTCLIFF from our result. Darker red means our method is more better. It can be observed that the joints on feet achieve larger performance gains, primarily due to the higher motion frequency in foot
joints. This phenomenon was also observed in the study 40 10 60 62 61 64). The advantages
16

---

Table Ablation study on using different test-time optimization (Opt.) steps at training stage. with COCO [38] as the training dataset and 3DPW [57] as the testing dataset. Gray row is the default
setting. Bold values are the best.
Test-time Opt. Steps at Training
MPJPE
PA-MPIPE 1
PVE 4
Training Time (mins/cpoch)
76.6
49.5
90.1
5.8mins
76.7
49.0
90.0
6.8mins
76.3
48.5
89,5
7.4mins
2.74
33.36
-12.0
-12.0
0,22
0,53
2.76
-2.44
2.3
1.16
-0.92
-0.79
-3.37
2.6
-9.0
-9.0
-6.0
-6.0
00.74
0.99
4.45
-6.5
-3.0
-3.0
-1.13
-085
0,0
0.0
3.0
3.0
10.87
14.63
9.14
10.77
MPJРE
PA-MPIPE
Figure 6: Per-joint error analysis between OUrcLiFr and EFTCLIFF. The testing dataset is 3DPW
57
Table 9: Ablation study 3D Pseudo SMPLs. Models are trained full training datasets and tested on 3DPW [57].
Method
MPJPE
MPIPEJ
PVE.
Ours cLIer (Res-50) w/o pacudo OUSScIPP (Res 50) w/ pseuda
44.7
66.0
42.1
83.6
of our method more obvious on PA-MPJPE, i.e., our method outperforms EFT in terms of PA-MPJPE for all joints, demonstrating that our method can better capture the pose and shape of the 2D human than EFTCLFFF.
A.5 More Explanations about Unifying the Training and Testing Objectives with Dual Networks
There is concern that with the introduction of the auxiliary network, still can not fully achieve the goal of matching training and testing objectives, i.e., there still remains a discrepancy between the training loss (using ground truth labels) and the testing-ting time loss (using pseudo labels). We argue that, since the auxiliary network is trained simultaneously with the main network, the auxiliary network learns the pseudo 3D) meshes that most suitable for optimizing the network during test-time optimization. In other words, we use the pseudo 3D labels generated by the auxiliary network help mitigate the gap between training and testing losses, compared with using the 2D joints only as supervision at the test stage.
To verify the impact of 3D pseudo SMPLs, have already conducted an ablation study, as shown in Rows and of Table 3 In Row 2, we discard the auxiliary network, meaning the pseudo SMPLs not used in the test-time optimization function. The comparison
generated by the auxiliary network reveals that the inclusion of pseudo SMPLs improves the model's 's performance, demonstrating their contribution to the optimization process.
Since the above ablation was conducted using a smaller dataset (COCO), we further validate the results by repeating the experiment on the full training datasets. As shown in Table 9 these additional
17

---

Table 10: Ablation Study about Ensembling Effects. Models are trained on COCO on 3DPW [57].
38
and tested
Method
MPJРE
PA. PP-MPP
EFTcLIFF
EFT2CLIPF
82.7
53.6
OurScr.wP
76.7
49.5
Table 11: Quantitative Comparison with Kim et al. [24).
Method
Backhone
Training Dataset
Testing Dataset
PA. MPJPEL
Kim et
24
SPIN 2X HMR
COCO, MPII, Human3.6M, MPLINN--DHHP. 3DPW, L.P
3DPW 3DPW
57,88
Ours
23
COCO, MPII, Human3.6M, MPI-INF-3DHP. 3DPW
44,3
findings reinforce the effectiveness of the generated pseudo 3D SMPLs, confirming their positive influence on model performance.
One may concern that using dual networks introduces an ensembling effect, with the second term of Eq. | serving
adjustment towards an intermediate estimation between the two networks.
To evaluate ensembling effect introduced by our dual-network setup, we conducted an experiment using two CLIFF models to generate SMPLs and computed their average SMPL,, on which EFT optimization was performed. The final results, including the averaged output, are shown in the Table 10 EFT2CLIFFs demonstrating the impact of using two CLIFFs within EFT.
The results indicate that incorporating two CLIFFs does indeed improve EFT performance, high- lighting an ensembling effect. However, even with these improvements, 2FT2CLIFSs underperforms compared to our method. This suggests that the ensembling effect achieved through meta-learning in our approach is more effective than simply using two CLIFF networks.
A.6 Comparison with Kim et al. 24 and Analysis
To provide a quantitative comparison, we collect results from Kim et al. [24]. Kim et al. 24 use the SPIN backbone-a stronger model than our HMR backbone, and include the LSP dataset in their training data (which we exclude). Our method achieves a better PA-MPIPE score (44.3 vs. 57.88), as shown in Table TT. This result indicates that approach surpasses Kim et al.'s performance despite using a relatively simpler backbone and fewer training resources.
We elaborate the key difference between Kim et al. (24] and our method in detail.
The key difference that that Kim al. use the reprojection loss only (please see Eq. in their paper) in both inner and outer loops of meta learning, while our method uses 2D and 3D losses. This is small difference in formulation, but large difference in contextualization.
EFT 22 is performed under the guidance of 2D reprojection loss. Kim et al. [24] extended EFT 22
both the inner and outer loops of meta learning. In this sense, the method of Kim et al. 24 is
a direct extension of EFT 22 to meta learning.
In contrast, our method considers meta learning from the perspective of the complete HMR model trained with 2D and 3D losses. In other words, we use the complete HMR model both inner and outer loops of meta learning. This is more reasonable, our aim is to personalize the HMR model on each test sample, rather than a model trained with only 2D reprojection loss.
The above differences enable us to design dual-networf architecture that is very different from the network used in Kim et al. (24). Overall, we find that incorporating 3D SMPLS into meta learning is very helpful. It is the utilization of the GT SMPLs that greatly improves the results of our method. Besides utilizing GT 3D SMPLs in the outer loop of meta learning, we additionally generate pseudo SMPLs and incorporate them into the inner loop of the metal learning.
18

---

A.7 More Intermediate Results
we show more stepwise optimization results during testing. can be seen from these examples that the mesh deviates more or less from the target human after step 0, and the deviation is progressively repaired after several optimization steps. For example, please check the results in row 1. The left arm does not match with the evidence in the image at the very beginning, and this error is corrected after about optimization steps. We show more examples from row 5 where the arms are wrong at first and corrected afterwards. In row 6 and 7, please pay attention to the legs. In the last three rows, please pay attention to the whole bodies.
In Figure 7
A
Input
Step 0
Step 1
Step 2
Step 3
Step
Step 5
Final
Figure 7: Stepwise visualization. From left to right, we showcase results after different steps of test-time optimization during testing.
19

---

A.8 More Qualitative Results
In Figure and Figure 9 we show more qualitative comparisons with the latest approaches on
test/validation datasets of COCO [38, 3DPW [57] and Human3.61. 6M [20]. Besides CLIFF [33 and
ReFit (59] that have been compared with in the main paper, we additionally bring HybrIK [2] NIKI
31. ProPose [3] and EFT into the comparison. In the shown examples, HybrIK produces
misaligned right arm in Figure@columm
NIKI produces wrong head orientations (Figure column
2 and column 4). For ProPose, ReFit and CLIFF, please pay attention to the feet in Figure 8 column
EFFCLLFF produces misaligned feet in FigureBccolume column and Tigure9colune 3. Besides, CLIFF
produces wrong left leg for the example in Figure 8 column 1. For these examples, our method produces visually better results.
A.9 Failure Cases
One kind of failure cases of our method are shown in Figure g and Figure 9 where there is still slight misalignment between projected mesh and the target 2D evidence, though the misalignment smaller than that of previous approaches. For example in column of Figure 8 the feet of Ourf and Our do not exactly match with the target 2D feet in the image. We observe that the ground-truth meshes in training datasets after projection also show such artifacts. To tackle this problem, one may need to provide more accurate human annotations.
Figure shows another kind of failure cases. In the example of row 1, our reconstructed mesh and the target 2D person are well-aligned. However, from a novel perspective, there a misalignment between our 3D mesh and the ground truth. This discrepancy arises from the inherent ill- ill-posedness of inferring 3D mesh from 2D information in monocular image. the second row of Figure IO] we showcase person with partial occlusion. Please notice the left foot of the person, where the such a scenario, the accuracy of the corresponding 2D joints not
person exhibits self-occlusion. high, posing a challenge to our method.
Figure II shows that the 3D mesh projected the 2D image performs poorly in the foot region, probably because the SMPL model itself is not flexible enough to capture the large distortion of the two legs.
Due to the severe occlusion or since the distance from the person to the camera is too far in Figure the quality of the estimated human mesh is poor, not well fitted with the target person.
20

---

Input
HybrIK 32
NIKI 31
13
ProPose ose
ReFit 39
CLIFF 35
EFTCLIFF
Oursf
Ours*
Figure 8: More qualitative comparisons with SOTA methods. We show results produced by
HybrlK [32). NIKI [31), ProPose [13 ReFit [59] CLIFF 33), EFTCLIFF, and our method (f:
OpenPose, *: RSN
21

---

Input
FESCOM
FENC
3
Hybrik
FES OM
ERCOM
kra leete
FE OO
NIK
LERCOM
13
ProPose
FE OOM
LERCOM
ReFit 59
FE OO
LEBCOm
33
CLIFF
FEM
EFTCLIFF
FES OM
Ourst
FES OM
FENCE
Ours
FE OOM
ZERCO
Figure 9: More qualitative comparisons with SOTA methods. We show results produced by
HybrlK (32). NIKI (31], ProPose [1]. ReFit (59. CLIFF 333, EFTCLIFF, and our method (f:
OpenPose, *: RSN
22

---

Figure 10: Failure eases. The projected mesh matches with the target 2D human body exactly, but there is misalignment in the 3D space. Green meshes represent ground-truth.
AFG
BAG
BROY
Figure 11: Failure case. The SMPL model's limited flexibility likely causes poor 3D mesh projection in the foot region.
Figure 12: Failure case. Severe occlusion far camera distance results in a low-quality human mesh estimation.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract presents the main idea, highlighting its innovative aspects that distinguish from other works. The introduction clearly states the main contributions, which are supported by both theoretical analyses (Section 3 and experimental results (Section The claims align with the results presented in the paper, and the scope is well-defined.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: While the main body of the paper does not include dedicated section on limitations, we have discussed several failure (A.9 and potential limitations in the appendix. These cases highlight scenarios where there is still slight misalignment between projected mesh and the target 2D evidence. We chose to include these discussions in the appendix due to space constraints in the main text, prioritizing detailed descriptions of our methodology and results.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The theory of this paper, which includes test-time optimization, incorporation of testing-time time into training, unifying training and test-time optimizations, and inference
with dual networks, is discussed in Section (3). This section introduces the loss function, the architecture of the network, and the algorithm for updating gradients according to meta- meta-learning.
Guidelines:
The answer NA means that the paper does include theoretical results. the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
The paper provides detailed descriptions of the experimental setup, including dataset descriptions that were used, hyperparameters, and evaluation metrics. All nec- essary steps to reproduce the experiments are included either in the main text or in the supplementary material. Additionally, it offers both qualitative and quantitative results for comparison.
Justification:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
25

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The paper has released the code and provided the download path for the data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https soneddeeumisssiooPeee
/ //nips httblss.ccppubli/guuueeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
The paper relies on public datasets which have provided the information of the data splits along with the datasets. The paper provides detailed descriptions of the training and test experiments, including datasets, chosen hyperparameters, the architecture network, and the criteria for their selection. These details are included in the section and Experiments in main text.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
26

---

7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: The paper does not include error bars, confidence intervals, or statistical significance tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
The paper specifies the types of compute resources used, including GPU models and memory specifications. This information is detailed in the section Experiments(4)
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Fthics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: //tt://nnsipsse cc.ppubllc/tthhccssueeelll...eee
Answer: [Yes]
Justification:
We ensured that all data and code used was obtained and processed ethically.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
27

---

If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid-
eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: Human pose estimation has several broader impacts. Positively, it enhances healthcare, optimizes athlete performance, improves human-come coman-computer interactions, and reduces workplace accidents. Negatively, it raises privacy concerns, risks misuse, and may lead to unfair outcomes if training data lacks diversity. Due to space limitations, this paper does not discuss these societal impacts in detail.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
28

---

12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All external assets used in the paper are properly credited. The licenses and terms use for these assets are explicitly mentioned in the references section and supplementary materials. Human3.6M4 CC BY-NC-ND 4.0; MPL-INF-3DHP: CC BY-NC 4.0; 3DPW: CC BY-NC-ND 4.0; COCO: CC BY 4.0; MPII: CC BY 4.0; LSP-Extended: CC BY-NC4.00.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
29

---

According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve research with human subjects requiring IRB Approvals.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
30


Reviews:
[{"review_id": "YwgbPu9ZZv", "reviewer": "Reviewer_PPKd", "summary": "This paper introduces a novel method to enhance Human Mesh Recovery (HMR) from images. The approach integrates test-time optimization into the training process, inspired by meta-learning. The dual-network structure is designed to align training and test-time objectives, improving the starting point for test-time optimization. Extensive experiments demonstrate that this method outperforms state-of-the-art HMR approaches, providing higher accuracy and better generalization to test samples. The work emphasizes the advantage of combining test-time and training-time optimization for robust human mesh recovery.", "questions": "In line 7 of Algorithm 1, why use the gradient w.r.t. $w'$ to update $w$ (similar to Reptile), instead of the gradient w.r.t $w$ (as in MAML)?", "limitations": "No negative societal impact", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The authors introduce an innovative approach by incorporating test-time optimization into the training phase, which addresses a gap in existing HMR methods. This blend of meta-learning principles ensures the optimization process at test time is more effective, enhancing performance and accuracy.\n\n2. The proposed method integrates seamlessly with existing HMR models using a dual-network structure, ensuring minimal additional computational load during training. This efficiency is achieved without compromising result quality, making the approach practical for real-world applications.\n\n3. The authors present their work in a well-structured and accessible manner. The logical flow, clear explanations of technical terms, and use of diagrams ensure that even complex concepts are easily understood, enhancing the overall impact of the paper.", "weaknesses": "1. The inclusion of dual networks does not fully achieve the goal of matching training and testing objectives. There remains a discrepancy between the training loss (using ground truth labels) and the testing-time loss (using pseudo labels). There is no guarantee or analysis provided that the gradients from ground truth labels and pseudo labels align.\n\n2. Using dual networks will introduce the ensembling effect. The second term of $\\bigtriangledown_{\\omega} L_{test}$ ($\\bigtriangledown_{\\omega} L_{3D}$) acts as an adjustment towards an intermediate estimation between the two networks. To fully eliminate the ensembling effects, another row in Table 3 should be included showing the results of EFT_CLIFF with two networks (averaged results) for a more accurate comparison.\n\n3. While the last subsection in Section 4 is a good start, it is insufficient. Test-time optimization (TTO) methods typically perform significantly better than their counterparts on out-of-domain datasets, which are critical for demonstrating the full potential of TTO solutions. A quantitative comparison with EFT on the LSP-Extended dataset would provide a more comprehensive validation of the method's effectiveness in OOD scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** There remains a discrepancy between the training loss (using ground truth labels) and the testing-time loss (using pseudo labels).\n\n**Our Response:** Thanks a lot for the insightful question which deserves further interpretations.\n\nThe design of dual networks is one of the core contributions of our method. We use the dual networks to unify the ***formulations*** of training and testing objectives, but the ***content*** of them cannot be completely unified, as one uses GT labels and the other uses pseudo labels. \n\n*We interpret this mismatch and provide analysis as follows.*\n\n1. There are 3D losses in the training objectives, while there are not at test time. We find this yields a gap between the training and testing optimizations, which motivates us to design the auxiliary network to generate pseudo 3D labels to mitigate the gap. Although there still exists discrepancy between GT and pseudo labels, ***the gap is greatly reduced compared to using only 2D loss without introducing the pseudo labels***. The experiments in Table 3 validates this key design of our method.\n\n2. In our system, the auxiliary network in combination with the main network is trained simultaneously using our proposed training strategy. Through this process, the auxiliary network learns to estimate pseudo 3D meshes that are effective for optimizing the network during test-time optimization, which ***guarantees to a certain extent*** the alignment between the gradients from ground truth labels and pseudo labels.\n\n**Question 2:** Eliminate ensembling effects: include results of EFT_CLIFF with two networks (averaged results).\n\n**Our Response:** Thanks a lot for the valuable comment. Adding this experiment can further validate the meta-architecture of our method. \n\nFollowing your suggestion, we use two CLIFFs to generate SMPLs and compute the averaged SMPL of them, based on which we conduct EFT optimization which in turn optimizes the two CLIFFs. This process is iterated for 20 rounds. The finally obtained CLIFFs and their averaged result is used for comparison.\n\nThe comparison results are shown in the following table.\n\n|             | MPJPE | PA-MPJPE |\n|:-----------:|:-----:|:--------:|\n| EFT_CLIFF   | 84.6  | 54.2     |\n| EFT_2CLIFFs | 82.72 | 53.59    |\n| Ours_CLIFF  | 76.7  | 49.5     |\n|             |       |          |\n\nThe row of EFT_2CLIFFs shows results of using two CLIFFs in EFT. It shows that this really works, indicating there indeed exists ensembling effects. However, although the two CLIFFs improve EFT, the improved results are still much worse than ours, proving that the ensembling effect by meta-learning outperforms that by simply using two CLIFFs.\n\nWe will add this new comparison into Table 3. Thanks again for the great suggestion.\n\n**Question 3**: A quantitative comparison with EFT on the LSP-Extended dataset would provide a more comprehensive validation of the method's effectiveness in OOD scenarios.\n\n**Our Response**:  Thanks a lot for the suggestion. We perform the following quantitative experiments to validate the OOD effectiveness of our method compared with EFT.\n\nFirst, we perform a quantitative comparison with EFT on the LSP-Extended dataset. Since LSP provides GT 2D joints but not GT SMPLs, we compare the 2D loss (measured w.r.t. the GT joints) with EFT.\n\n|            | LSP-Extended |\n|:----------:|:------------:|\n|            | 2D Loss      |\n| EFT_CLIFF  | 8.3e-3       |\n| Ours_CLIFF | 6.1e-3       |\n|            |              |\n\n>**Training dataset:** COCO, MPII, MPI-INF-3DHP, Human3.6M, 3DPW  \n>**Backbone:** HR-W48\n\nOur method approaches the GT joints more nearby than EFT, i.e., 6.1e-3 (ours) vs. 8.3e-3 (EFT). \n>The calculation method for the 2D loss is as follows: Both the predicted values and GT joints are transformed into the range [-1,1] using the formula 2*y/crop_img_height-1 (similar for the width). Then, the mean squared error (MSE) loss is calculated.\n\nSecond, to further evaluate the OOD effectiveness of our method, we train our method and EFT on COCO (an outdoor dataset), and test them on Human3.6M (an indoor dataset). \n\n|            |Human3.6M |          |\n|:----------:|:---------:|:--------:|\n|            | MPJPE     | PA-MPJPE |\n| EFT_CLIFF  | 85.5      | 51.0     |\n| Ours_CLIFF | 83.8      | 48.6     |\n|            |           |          |\n\n>**Training dataset:** COCO  \n>**Backbone:** Res-50\n\nSince there are GT SMPLs, we report MPJPE and PA-MPJPE. As can be seen, our results are also better than EFT. \n\nFinally, The above quantitative experiments  will be edited into the main text of the revised paper, to validate the OOD effectiveness of our method with quantitative evidences.\n\n**Question 4:** In line 7 of Algorithm 1, why use the gradient w.r.t. w′ instead w?\n\n**Our Response:** Thanks for the careful checking. We are sorry that this is a typo. We do use the MAML algorithm. Specifically, we use FOMAML that discards the second-order derivatives. So, w’ should be corrected to w. We will correct this typo in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GmFPxiPVIS", "reviewer": "Reviewer_Cdc4", "summary": "Human mesh recovery (HMR) is improved by incorporating test-time optimization into the mesh prediction model’s training loop, allowing the learned parameters to potentially be more readily adapted to examples at test time. This is referred to as a meta learning approach, and it departs from prior work that also introduced this meta learning strategy by incorporating a secondary network to make predictions for pseudo ground truth meshes during the test time computations (during training and actual testing). This novelty as well as the design choices of the approach as a whole are validated through ablation studies, and the final method is shown to perform strongly relative to SOTA competing methods.", "questions": "Line 153 and Algorithm 1: the referenced meta learning approach MAML involves computing gradients through other gradient computations (i.e., second order derivatives). Could you please confirm my understanding that Algorithm 1 does not do this, and (if it does not) explain why it's described as “simulating” this meta learning approach? \n\nPlease add a quantitative comparison to the prior work that introduced meta learning to this problem, Kim et al. (2022), using the same training/testing data for each method. Also, please highlight the differences between these methods when discussing their different results, and connect this discussion to Table 3 because I believe its row 2 corresponds to Kim et al. (2022). \n\nLine 251: I would add quantitative results to this section or avoid claiming strong OOD performance (e.g. relative to EFT). The provided qualitative results with 2 images do not provide sufficient support for the OOD claims.\n\nConsider replacing the backwards sum notation in Figure 1 by placing the usual notation on the left side of the summands.", "limitations": "The main text does not discuss any limitations. The appendix has some discussion of them that should be moved to the main text.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "Originality: Adding to the meta learning approach an auxiliary network that provides a pseudo mesh for the test time adaptation is a novel contribution. Ablation studies illustrate the effects of key method hyperparameters.\n\nQuality: Incorporation of the test-time adaptation into the training procedure is well motivated. The method is tested with multiple backbone networks on multiple datasets, against multiple strong baseline methods of varying types (optimization and regression). Quantitatively, it significantly outperforms the state of the art baselines. The ablation studies support the contention that the proposed learning approach will be effective as joint estimation and mesh prediction models advance. The results (especially Figure 3 and Table 3) support the idea that meta learning is a critical framework for solving the HMR problem. Table 3 also supports the use of the dual network approach.\n\nClarity: The paper is reasonably well written and the figures/tables helpfully convey key concepts and results. \n\nSignificance: The problem of predicting human mesh parameters from image data is actively studied, and progress on it impacts various applications. The submission makes notable contributions that improve the ability to solve this problem.", "weaknesses": "The main weakness is presentation, which I think can be improved in a few ways. I outline these below and in the Questions section. Importantly, I think the results in this paper are strong, and I would be happy to raise my score if these weaknesses are addressed.\n\nMost importantly, the contextualization relative to the prior work “META-LEARNED INITIALIZATION FOR 3D HUMAN RECOVERY” (Kim et al., 2022) needs to be improved. \n\n - The related work section references Kim et al. (2022), which also uses meta learning for HMR, but other than that reference the phrasing in the submission suggests that the meta learning idea is a novel contribution of this submission. I think this phrasing should be adjusted to better reflect that the meta learning component here is not novel. Some examples of phrasing I would change follow: the submission says that the relevance of meta learning to HMR is its “key observation”, that it is “rethinking” the HMR problem by introducing meta learning, that it “proposes” to use meta learning, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** Rephasing “key observation”, “rethinking”, “proposes”, etc., as Kim et al. 2022 have used meta learning already.\n\n**Our Response:** Thanks very much for the suggestion. Yes, Kim et al. 2022 also adopted meta learning, as referenced in our paper. The difference is that their method is a direct extension to EFT, while we take a different perspective from the Human Mesh Regression model. Please see the difference between Kim et al. 2022 and our method in the response to ***Question 3***.\n\nFollowing your suggestion, we will rephase the above presentation to credit the contribution of Kim et al. 2022 more appropriately. \n1. *“The key observation in this paper is that the pretrained model may not provide an ideal optimization starting point for the test-time optimization”* ***will be changed to:***   \n>“However, the pretrained model may not provide an ideal optimization starting point for the test-time optimization”.\n\n2. *“The above analysis motivates us to re-think the test-time optimization problem in the framework of learning to learn, i.e., meta learning. Inspired by optimization-based meta-learning [12], we incorporate test-time optimization into the training process”* ***will be changed to:***   \n>“Based on the above analysis and **inspired by Kim et al. [22]**, we incorporate the test-time optimization into the training process, formulating the test-time optimization in the framework of learning to learn, i.e., meta learning [12].”  \n\nBesides the above modifications, we will add analysis about the difference between Kim et al. 2022 and our method, and revise relevant places of the paper accordingly. \n\n**Question 2:** Does Algorithm 1 compute second-order derivatives like MAML? Explain why “simulating” this meta learning approach in Line 153.\n\n**Our Response:** We do use the MAML algorithm. However, the original MAML needs to compute second-order derivatives which is too much time-consuming and memory-cost. We thus adopt FOMAML instead, a simplified version of MAML suggested by the Finn et al. [12] which discards second-order derivatives.\n\nTo clarify, there is a typo in Line 7 of Algorithm 1, where the gradient should be computed with respect to w instead of w’, as indicated by Reviewer PPKd. \n\nTherefore, we confirm that the statement of our algorithm “simulates” MAML in Line 153 is correct.\n\n**Question 3:** Quantitative comparison with Kim et al. (2022). Highlight the difference.\n\n**Our Response:** \n\n| Method          | Backbone                      | Training Dataset                               | Testing Dataset | PA-MPJPE |\n|:---------------:|:-----------------------------:|:----------------------------------------------:|:---------------:|:--------:|\n| Kim et al. 2022 | SPIN (Kolotouros et al. 2019) | COCO, MPII, Human3.6M, MPI-INF-3DHP, 3DPW, LSP | 3DPW            | 57.88    |\n| Ours            | HMR (Kanazawa et al. 2018)    | COCO, MPII, Human3.6M, MPI-INF-3DHP, 3DPW      | 3DPW            | 44.3     |\n|                 |                               |                                                |                 |          |\n\nPlease see the above quantitative comparisons. Due to time limit, we collect the result from Kim et al. 2022 for the comparison. \n1. Backbone: the backbone of Kim et al. 2022 is SPIN, which is a stronger backbone than ours HMR. \n2. Training dataset: the training datasets are nearly the same except that Kim et al. 2022 additionally uses LSP while we do not. \n\nAs can be seen, the PA-MPJPE of Kim et al. is 57.88 which is worse than ours 44.3, indicating that our method performs better than Kim et al. 2022. \n\n*Now, we elaborate the key difference between Kim et al. 2022 and our method in detail.*\n\nThe key difference is that that Kim et al. 2022 use the 2D reprojection loss only (please see Eq. 1 in their paper) in both inner and outer loops of meta learning, while our method uses 2D and 3D losses. This is a ***small difference in formulation***, but a ***large difference in contextualization***. \n\nEFT is performed under the guidance of 2D reprojection loss, and Kim et al. (2022) extend EFT to both the inner and outer loops of meta learning. In this sense, the method of Kim et al. (2022) is a direct extension of EFT to meta learning.\n\nIn contrast, our method considers meta learning from the perspective of the complete HMR model trained with 2D and 3D losses. In other words, we use the complete HMR model in both inner and outer loops of meta learning. ***This is more reasonable, as our aim is to personalize the HMR model on each test sample, rather than a model trained with only 2D reprojection loss.*** \n\nThe above difference enables us to design a dual-network architecture that is very different from the network used in Kim et al. 2022. Besides, the introducing of 3D losses improves the HMR quality significantly, as shown by the above quantitative comparisons and those in the paper.\n\n**Question 4:** Line 251, add quantitative results to support the OOD claims.\n\n**Our Response:** Thanks very much for the suggestion. We have added quantitative comparisons. Due to space limit, please refer to our answer to ***Question 3 of Reviewer PPKd*** (i.e., Reviewer 1) for more information.\n\n**Question 5:** Place the backwards sum notation in Figure 1 on the left side of the summands.\n\n**Our Response:** Thanks for the suggestion. We have revised Figure 1. Please see the attached PDF document for the modified figure.\n\n**Question 6:** Move discussion of limitations in Appendix to the main text.\n\n**Our Response:** Thanks a lot for the suggestion. We will move them to the main text as suggested.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yJNygRYfQm", "reviewer": "Reviewer_Qn92", "summary": "This paper targets on the Human Mesh Recovery task. The authors adopt a test-time per-instance optimization method similar to EFT, but proposes a meta-learning-like framework to pre-optimize the pretrained human pose estimation network, aiming to provide a better starting point for test-time optimization. In addition, the authors also propose a dual-network mechanism to solve the test-time and training-time objective misalignment. The idea is novel and interesting. Experimental results also fully demonstrate the effectiveness of the propsoed method.", "questions": "Please see above.", "limitations": "The authors didn't include Broader Impacts in the paper. Although two failure cases are provided, more are encouraged to give a comprehensive overview.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* Incorporating meta learning idea in test-time optimization is interesting and demonstrates leading performance.\n* The authors conduct extensive ablations to validate the proposed method.\n* Paper is well organized and written.", "weaknesses": "* Some experimental results were not clearly explained. E.g. In Fig. 3, EFT gets worse after several optimization steps, while the proposed method keeps getting better. Could you please give some intuition on this? Why EFT gets worse and which design of the proposed approach alleviate this issue?\n* Fig. 1 is a bit dense and difficult to follow. Perhaps the authors could try to reorganize and simplify it.\n* The paper consists a large number of experiments. However, the authors don't provide statistical significance for any of the experiment as suggested by the NeurIPS Paper Checklist 7. It is suggested to at least include the standard deviation or the standard error for the main results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** In Fig. 3, Why EFT gets worse and which design of the proposed approach alleviates this issue?\n\n**Our Response:** This is probably because EFT is only finetuned with 2D reprojection loss and is more ***sensitive to the errors in the 2D joints***. At the first several optimization steps, the estimated 3D SMPL approaches the 2D joints from a relatively distant initialization, therefore the result gets better gradually. However, with more optimization steps, the SMPL may overfit the 2D joints whose annotation-errors then distort the SMPLs, thus yielding worse evaluation metrics. \n\nIn contrast, our method is guided by both 3D and 2D supervisions. The 3D pseudo SMPLs ***plays the role of regularization*** that mitigates the influences of errors in 2D joints.\n\n**Question 2:** Fig. 1 is a bit difficult to follow. Reorganize and simplify Fig. 1.\n\n**Our Response:** Thanks for the suggestion. We have revised Fig. 1. Please ***see the new figure in the attached PDF document***. \n\nOverall, we think the difficulty in reading Fig. 1 comes from the fact that the illustrations of SMPL meshes and 2D joints in the testing and training losses are free of text descriptions. In the new figure, we increase space between losses, and add descriptions for the inputs of the loss functions.\n\n**Question 3:** Provide statistical significance for experiments as suggested by the NeurIPS Paper Checklist 7. \n\n**Our Response:** \n\n| 3DPW       |          |                  |                  |                  |\n|:----------:|:--------:|:----------------:|:----------------:|:----------------:|\n| Method     | Joints   | MPJPE            | PA-MPJPE         | PVE              |\n| Ours_CLIFF | OpenPose | 62.93$\\pm$0.003  | 39.72$\\pm$0.002  | 80.06$\\pm$0.004  |\n| Ours_CLIFF | RSN      | 62.42$\\pm$0.0002 | 39.47$\\pm$0.0001 | 78.14$\\pm$0.0003 |\n|            |          |                  |                  |                  |\n\n| Human3.6M  |          |                 |                  |\n|:----------:|:--------:|:---------------:|:----------------:|\n| Method     | Joints   | MPJPE           | PA-MPJPE         |\n| Ours_CLIFF | OpenPose | 43.93$\\pm$0.006 | 30.31$\\pm$0.004  |\n| Ours_CLIFF | RSN      | 41.96$\\pm$0.002 | 29.18$\\pm$0.0006 |\n|            |          |                 |                  |\n\nThanks very much for the reminding. During this rebuttal period, we have conducted experiments to include the standard errors for the main results. Please see the above tables. The above results are obtained by running the test-time optimization 5 times using different initializations. More statistical experimental results of both training and testing for the main and ablation experiments will be added in the revised paper finally.\n\n**Question 4:** Although two failure cases are provided, more are encouraged to give a comprehensive overview.\n\n**Our Response:** Thanks very much for the suggestion. We have added more failure cases. Please see the attached PDF document for more information.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ODjpGshL8J", "reviewer": "Reviewer_Nccj", "summary": "The paper presents a new training paradigm for better test-time optimization performance at test time. Specifically, two strategies are proposed. (1). integrate the test-time optimization into the training procedure, which performs test-time optimization before running the typical training in each training iteration. (2). propose a dual-network architecture to implement the proposed novel training paradigm, aiming at unifying the space of the test-time and training optimization problems.", "questions": "see weakness", "limitations": "see weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a new training paradigm that can integrate test-time optimization into the training procedure.\n2. The overall procedure is clear and easy to understand.", "weaknesses": "My biggest concern is that the performance improvement is not significant when compared with the existing method.  The Table 1 compares the results of an existing method PLIKS and the method proposed in this paper. However, the comparison of 3DPW is not fair, since the results of the last two lines use 3DPW for training, while the reported PLIKS does not use 3DPW for training. PLIKS trained on 3DPW can achieve 40.1 PA-MPJPE and 73.3 PVE, which greatly outperforms the results in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Question 1:** The biggest concern is that the performance improvement is not significant when compared with the existing method. The comparison of 3DPW is not fair, since the results of the last two lines use 3DPW for training, while the reported PLIKS does not use 3DPW for training. PLIKS trained on 3DPW can achieve 40.1 PA-MPJPE and 73.3 PVE, which greatly outperforms the results in this paper.\n\n**Our Response:** Thanks very much for the comments. We have checked the results of PLIKS again. After the checking, we confirm that our comparison is fair, and our results outperform those of PLIKS.\n\nFor convenience, we copy the following content from Table 1 of the paper of PLIKS (see https://openaccess.thecvf.com/content/CVPR2023/papers/Shetty_PLIKS_A_Pseudo-Linear_Inverse_Kinematic_Solver_for_3D_Human_Body_CVPR_2023_paper.pdf).  \n\n| 3DPW          |          |       |      |\n|:-------------:|:--------:|:-----:|:----:|\n| Method        | PA-MPJPE | MPJPE | PVE  |\n| PLIKS+(HR32)  | 42.8     | 66.9  | 82.6 |\n| PLIKS++(HR32) | 40.1     | 63.5  | 76.7 |\n| PLIKS++(HR48) | 38.5     | 60.5  | 73.3 |\n|               |          |       |      |\n\n>***From caption of Table 1 of PLIKS:*** PLIKS+ means that the network was additionally trained with ***3DPW***. PLKIS++ means that the network was additionally trained with ***3DPW and AGORA***.\n\nIn our paper, we compare with PLIKS+ which is trained with 3DPW too. So, from the aspect of training dataset, our comparison is fair. \n\nAs for the results of 40.1 (PA-MPJPE) and 73.3 (PVE) of PLIKS++, they are obtained by further finetuning PLIKS on 3DPW and Agora. Since our method is not finetuned with Agora, we are thus more appropriate to compared with PLIKS+ instead of PLIKS++. \n\nHowever, we find there is a typo in our paper. That is, we write that PLIKS adopts the backbone of HRNet-W48, but actually it uses HRNet-W32. This makes it somewhat unfair between PLIKS and our method. Therefore, we make the following fair comparison, which also shows our method outperforms PLIKS.\n\n| 3DPW                       |          |       |      |\n|:--------------------------:|:--------:|:-----:|:----:|\n| Method                     | PA-MPJPE | MPJPE | PVE  |\n| PLIKS+ (HR32)               | 42.8     | 66.9  | 82.6 |\n| Ours_CLIFF$\\dagger $ (HR32) | 40.1     | 62.9  | 80.9 |\n|                            |          |       |      |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear ACs and Reviewers:

We sincerely thank you and all reviewers for your time and review. 

All reviewers endorse the novel idea of this paper. For example, Reviewer PPKd says "The authors introduce ***an innovative approach***", Reviewer Cdc4 says "Adding to the meta learning approach an auxiliary network ... is a ***novel contribution***, is ***well motivated***". Reviewer Qn92 provides the comment that "Incorporating meta learning idea in test-time optimization is ***interesting*** and demonstrates ***leading performance***". Reviewer Nccj says "The paper proposes ***a new training paradigm***". All the reviewers think our paper is "well structured", and "easy to understand". 

Reviewer Cdc4 mainly raise the concern that Kim et al. (2022) have already adopted meta learning to improve EFT. We have interpreted the difference between Kim et al. (2022) and our method in detail. 

Both Reviewer PPKd and Cdc4 suggest to add quantitative results about the OOD performance of our method. We have conducted two kinds of experiments that well validate the OOD effectiveness. 

Reviewer PPKd provides a very insightful comment that our method has ensembling effect and suggests us to provide a comparison with the baseline that also incorporates ensembling. After adding this experiment, our paper is more solid. 

According to the comments of Reviewer Qn92, we re-plot Figure 1 to make the pipeline clearer. We also provide two more failure cases in the attached PDF document. 

We would like to indicate that there are ***some factual errors*** in the comments of Reviewer Nccj who may misread the results of PLIKS. We confirm again that our comparison with PLIKS is fair and our method outperforms PLIKS. Please see our response to the question of Reviewer Nccj.

In summary, we have proposed a novel Human Mesh Recovery (HMR) method. Our method incorporates HMR into the framework of meta learning. We propose a novel dual-network architecture to mitigate the discrepancy between the optimizations in the inner and outer loops of the meta learning. Besides, we have provided thorough experiments to validate the effectiveness of the proposed method. 

We sincerely hope that this SOTA paper on an important topic will be considered suitable for publishing in NIPS 2024. Thank you for your service and for considering our paper for the conference.

Sincerely

Authors.

Meta-review: This work addresses the problem of human mesh recovery (HMR) using a meta-learning framework, where a dual network is designed to align training and test-time objectives.  Overall, the presentation is clear, the problem and method are well-motivated, and the performance is good. It receives positive scores after the rebuttal discussions. AC agrees this is a nice application of meta-learning on HMR and thus recommends acceptance. The authors should properly reference the prior work “META-LEARNED INITIALIZATION FOR 3D HUMAN RECOVERY”(Kim et al., 2022) and highlight the key difference which already applied meta-learning on HMR.
Decision: Accept (poster)