ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization
Authors: Wenhao Shen
Wanqi Yin
Xiaofeng Yang
Cheng Chen
Chaoyue Song
Zhongang Cai
Lei Yang
Hao Wang
Guosheng Lin
Venue: ICML 2025 poster (2025)
Keywords: human mesh recovery
direct preference optimization
Primary area: applications->computer_vision
TL;DR: A framework to align diffusion-based human mesh recovery methods via direct preference optimization.

Abstract: Human mesh recovery (HMR) from a single image is inherently ill-posed due to depth ambiguity and occlusions. Probabilistic methods have tried to solve this by generating numerous plausible 3D human mesh predictions, but they often exhibit misalignment with 2D image observations and weak robustness to in-the-wild images. To address these issues, we propose ADHMR, a framework that **A**ligns a **D**iffusion-based **HMR** model in a preference optimization manner. First, we train a human mesh prediction assessment model, HMR-Scorer, capable of evaluating predictions even for in-the-wild images without 3D annotations. We then use HMR-Scorer to create a preference dataset, where each input image has a pair of winner and loser mesh predictions. This dataset is used to finetune the base model using direct preference optimization. Moreover, HMR-Scorer also helps improve existing HMR models by data cleaning, even with fewer training samples. Extensive experiments show that ADHMR outperforms current state-of-the-art methods. Code is available at: [*https://github.com/shenwenhao01/ADHMR*](https://github.com/shenwenhao01/ADHMR).

ADHMR:
Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization
Wenhao Shen |
Wanqi Yin?
Xiaofeng Yang'
Cheng Chen Chaoyue Song' Zhongang Cai Lei Yang²
Hao Wang²
Guosheng Lin
Abstract
Prevailing approaches usually adopt a deterministic style, generating a single prediction for each image (Cai et al., 2024a; Goel et al., 2023; Li et al., 2023; Moon et al., 2022). However, this task faces inherent uncertainty when lifting 2D observations to 3D models, due to depth ambiguity and occlusion. Accordingly, the community is shifting to probabilistic methods. Probabilistic methods tackle the un- certainty by generating multiple plausible human mesh pre- dictions for each image (Kolotouros et al., 2021; Sengupta et al., 2023). For instance, recent approaches (Foo et al., 2023; Cho & Kim, 2023) frame this task as a denoising diffu- sion process. However, these probabilistic approaches suffer from limited emphasis on obtaining accurate estimates.
Human mesh recovery (HMR) from single im- age is inherently ill- ill-posed due to depth ambiguity and occlusions. Probabilistic methods have tried to solve this by generating numerous plausible 3D human mesh predictions, but they often ex- hibit misalignment with 2D image observations and weak robustness to in-the-wild images. To ad- dress these issues, propose ADHMR, frame- work that Aligns a Diffusion- based HMR model in a preference optimization manner. First, we train a human mesh prediction assessment model, HIMR-Scorer, capable of evaluating predictions even for in-the-wild images without 3D annota- tions. We then use HMR-Scorer create prefer- ence dataset, where each input image has a pair of winner and loser mesh predictions. This dataset is used to finetune the base model using direct preference optimization. Moreover, HMR-Scorer also helps improve existing HMR models by data cleaning, even with fewer training samples. Exten- sive experiments show that ADHMR outperforms current thee-----t-ht--rttee art methods. Code is available at: http:/iwwww...ooo////eeeeeeenoee /ADMM.
Specifically, the current state-of-the-an probabilistic method ScoreHypo (Xu et al., 2024b) tackles this by designing an additional network for test-time prediction selection after the diffusion- based prediction model. However, we observe that ScoreHypo still exhibits the following shortcomings: (1) misalignment between 3D mesh predictions and 2D image cues, and (2) poor performance on in-the-wild images. This is primarily because end-to-end diffusion models predicting from pure noise typically avoid 3D reprojection loss, as early denoising steps yield unrealistic poses, making such loss ineffective (Huang et al., 2024). Instead, the diffusion loss focuses on generating the target human mesh distribution rather than precisely aligning joints. While this produces plausible poses, may neglect the alignment between the 3D mesh and the image. Moreover, existing datasets often use aptimization-basee HMR methods to generate pscudo 3D annotations for in in-the-wild images, which inevitably contain some inaccurate or noisy data.
1. Introduction
Human mesh recovery (HMR) is a fundamental challenge in computer vision, focused on estimating the 3D human shape and pose from single RGB image. HMR enables various downstream applications, including clothed human reconstruction (Shuai al., 2022; Hong & Shen, 2024; Yao et al., 2025), virtual try-on, AR/VR content creation (Xu et al., 2024c; Yang et al., 2024) and etc.
To address the above challenges, we introduce Aligned Diffusion- Human Mesh Recovery (ADHMR). The key insight is to distill the knowledge of powerful scorer into the 3D human mesh predictor in a preference optimiza- tion manner. Technically, we begin by training the HMR- Scorer, essentially a reward model that assigns quality score to quantify the human mesh prediction quality. HMR- Scorer gives higher scores to predictions better aligned with the input image ("winners"") and lower scores to those poorly aligned ("losers""). In order to increase the sensitivity of
'Nanyang Technological University 'SenseTime Research 'The Hong Kong University of Science and Technology (Guangzhou). Correspondence to: Guosheng Lin <gslin@ntuuedu.sge Hao Wang <haowang @ hhuustgee edu..z..>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

ADHMR: Aligning Diffusion- based Human Mesh Recovery via Direct Preference Optimization
HMR-Scorer to nuances in the image cues, we extract multi- scale image features as conditions, which provide global and local pixel-aligned features sampled by reprojected human keypoints, enabling HMR-Scorer to identify misalignment between the predicted mesh and 2D image cues.
We show that using HMR-Scorer for data cleaning boosts the performance of state-of-the-art HMR mod- els, even when trained on fewer data.
2. Related work
Then we draw on the concept of direct preference optimiza- tion (DPO) for diffusion models (Wallace et al., 2024) to optimize a diffusion- beafusiion-based HMR base model. Traditional joint-wise or pixel-wise losses could overfit noisy labels in real-world data. Besides, a trade-off between 2D repro- jection fidelity and 3D accuracy exists due to imprecise camera estimation (Dwivedi et al., 2024). In contrast, DPO focuses on the relative prediction quality, being more robust to imperfect data. However, DPO requires an annotated preference dataset, which is costly to obtain (Rafailov al., 2024). To this end, employ HMR- -eoorr to evaluate and rank the predictions generated by the base model, re- sulting in a preference dataset composed of (winner, loser) prediction pairs. Guided by this synthetic dataset, ADHMR refines the HMR base model towards producing human pose predictions that are both more plausible and more closely aligned with 2D image cues. Moreover, ADHMR improves its robustness by finetuning on in-the-wild images without the need for pseudo labels.
2.1. Human Mesh Recovery from a Single Image
Current HMR approaches can be broadly categorized into two paradigms: deterministic and probabilistic. Determin- istic approaches(Goel et al., 2023; Cai et al., 2024a; Moon et al., 2022; Shen et al., 2024; Yin et al., 2024) produce a single estimate for each input. However, due to intrinsic re- construction ambiguities, probabilistic approaches focus on generating multiple plausible hypotheses or capturing prob- abilistic distributions. ProHMR (Kolotouros et al., 2021) leverages a conditional normalizing flow to model con- ditional probability distribution. Fang et al. (Fang et al., 2023) propose learning probability distributions over human joint rotations by utilizing a learned analytical posterior probability. EgoHMR (Zhang et al., 2023) proposes a 3D scene-conditioned diffusion approach for reconstructing hu- man meshes from egocentric views. ScoreHypo (Xu et al., 2024b) uses a diffusion- verse set of plausible estimates, and separate network is employed to choose from these estimates. Despite their effectiveness, they often require generating numerous candi- date poses for selection or averaging. In contrast, we employ direct preference optimization to improve the performance of the prediction model directly.
generator to produce di-
Notably, HMR-Scorer can also be leveraged to improve the performance of state-of-the-art HMR models through data cleaning. Many models (Yin et al., 2025; Pang et al., 2024; Sun et al., 2024) incorporate in-the-wild datasets for training to enhance their generalizability. However, as men- tioned earlier, the 3D pseudo-labels in these datasets are often unreliable. Prior work relies on expensive manual curation (Lassner et al., 2017) or rigid reprojection-ent -erro filtering (Kolotouros et al., 2019) to combat these issues. Instead, we propose to conduct a fully automated data clean- ing process to build higher-quality training datasets. We sort the labuled images in a dataset based on their scores given by HMR-Scorer and only retain samples with scores above certain threshold. By filtering out poorly annotated data, we reduce the influence of noisy annotations and boost model performance.
2.2. Human Preference Optimization
The initial efforts to learn from human preferences orig- inated in training agents (Christiano et al., 2017; Ibarz et al., 2018), later expanding to incorporate human feedback (RLHF) for enhancing tasks like translation (Kreutzer et al., 2018) and summarization (Stiennon et al., 2020; Ziegler et al., 2019). These methods first train reward model to align with human preferences and then finetune language model to maximize this reward using reinforcement learn- ing techniques such as PPO (Schulman et al., 2017). Sev- cral solutions have been proposed to simplify this complex pipeline: HIVE (Zhang et al., 2024) uses offline reinforce- ment learning align instruction editing. Direct Preference Optimization (DPO) (Rafailov et al., 2024) directly opti- mizes the model using supervised classification objective on preference data. This approach is now being increas- ingly adopted across other domains. For instance, ImageRe- ward (Xu et al., 2024a) and Lee et al. (Lee et al., 2023) apply RLHF to eext-to-image synthesis models; DreamRe- ward (Ye et al., 2024) and CADCrafter (Chen et al., 2025) use RLHF for text-to-3D generation. Diffusion-DPO (Wal- lace et al., 2024) adapts the DPO objective for Diffusion Models, improving the performance of models like Stable
Comprehensive experimental results demonstrate the ef- fectiveness of our approach. The main contributions are summarized below:
We propose ADHMR, a novel framework for improv- ing existing diffusion- based HMR models by adapting human preference optimization methods to an unla- beled setting, thus outperforming existing state-of-the- art probabilistic HMR methods.
We introduce HMRRScoores a robust reward model that effectively quantifies the aligument between human mesh predictions and corresponding input images.
2

---

ADHMR: Aligning Diffusion- Human Mesh Recovery via Direct Preference Optimization
Rocsermacionn Quality OP0e graiie graeN
HMR
HMR New Model
Wimars
Base
HMR-Scorer
Model
LosTS
Figure 1. Overview of ADHMR. We aim to finetune probabilistic HMR base model that generates multiple human mesh predictions conditioned the input image. We first train the Scorer that assesses the reconstruction quality given image and corresponding human mesh predictions. The reconstruction quality annotations Q" are computed using standard HMR metrics, including PVE Q MPJPE QTPPIPe, PA-MPIPE Qrajse, and PA-PVE Quapue, Next, construct synthetic human preference dataset, where each sample a (winner, loser) prediction pair rated by the HMR-Scorer. Finally, ADHMR uses this synthetic human preference dataset to finetune the base model preferentially generate predictions that are more plausible and better aligned with the image cues.
Diffusion for enhanced visual appeal and textual coherence. Our method is inspired by Diffusion-DDR but differs in its implementation. Rather than depending on curated manu- ally labeled human feedback datasets, we devise method to automatically generate a human preference dataset using HMR-Scorer, offering greater flexibility for our scenario.
eters into components: swing, derived from 3D joint positions, and twist, representing rotational details for each body part. These two elements are combined into single data sample, which is then processed through forward diffusion step to gradually add noise. The noisy samples are mapped to a high-dimensional feature space using a multilayer perceptron. guide the denoising process, the model incorporates image features extracted through con- volutional neural network backbone. These preprocessed image features are concatenated and fed into transformer- encoder (Vaswani, 2017) based network. The transformer integrates global image context through cross-attention mechanism, aligning the denoising process with the input image. Finally, the network reconstructs the human pose by removing the added noise. The human shape parameters are estimated by the convolutional backbone.
3. Preliminary
3.1. HMR Evaluation Metrics
We use four standard metrics for HMR: Mean Per Vertex Position Error (PVE) and Mean Per Joint Position Error (MPJPE), along with their Procrustes-aligned variants (PA- PVE and PA-MPJPE). All metrics compute the average distance (in mm) between predicted and ground-truth po- sitions, with the pelvis joint aligned as the reference point. We apply the joint regressor of SMPL(-X) to the predicted mesh to obtain 3D joint coordinates.
4. Method
4.1. Overview
3.2. Diffusion-based HMR Base Model
An overview of ADHMR is presented in Figure 1. Given an input image I, we aim to reconstruct the 3D human mesh in parameterized way, which to predict the pose parameters R24x3 and shape parameters
Modeling HMR as reverse diffuusion process by noisy sam- ples (x, TT the base model HypoNet (Xu et al., 2024b) is denoiser that progressively denoises random pose noise based on the input image I to reconstruct the human mesh. I' is the total number of timesteps. This process is formulated as:
R the predefined SMPL model (Pavlakos et al., 2019). We formulate this problem as generation process conditioned the input image to tackle the inherent reconstruction ambiguity.
Po (x,-1 x.) .'' (x,-1; Grer (,,
We begin by training a diffusion-basel HMR base model (Sec. 3.2). Next, we construct synthetic human prefer- ence dataset (Sec. 4.3), where candidate human mesh pre- dictions generated by the base model and then paired
where is the posterior mean of the forward process.
Specifically, the base model follows (Li et al., 2021) 1) by breaking down the SMPI. (Loper al., 2015) pose param-
3

---

ADHMR: Aligning Diffusion-barse based Human Mesh Recovery via Direct Preference Optimization
based scores provided by the assessment model HMR- Scorer (Sec. 4.2). To distill knowledge from this synthetic preference dataset, we propose a preference optimization framework ADHMR that finetunes the base model to prefer- entially generate winner predictions over losers (Sec. 4.4). Furthermore, thanks to the strong capacity of HMR-Scorer to assess mesh predictions, we filter training data to enhance several popular HMR models (Sec. 4.5).
output of the preceding MHSA unit. Through the cross- attention mechanism, the geometric information from the human mesh predictions is effectively aligned with image features, ensuring a coherent integration of structural and visual cues. Finally, a decoder network, implemented as an MLP, is employed to estimate the score s.
Training. We construct a training dataset comprising human mesh predictions for corresponding images and their quality labels to train the generated by adding joint-wise Gaussian noise to the ground truth SMPL pose to simulate rotational errors, with magni- tudes empirically determined. The reconstruction quality labels are annotated using standard HMR metrics, includ- ing PVE Quee, MPJPF QmPixe, PA-MPIPE Quaipe, and PA-PVE Quapre, Details of these metries are provided in Sec. 3.1. To accurately capture subtle quality differences, the training process is designed to learn relative quality pref- erences among predictions. Inspired by RankNet (Burges et al., 2005), we utilize a probabilistic ranking cost function:
Specifically, predictions are
4.2. HMR-Scorer
Given a set of predictions {Pm - (0, 3, m M for an input image I, HMR-Scorer aims to assign an estimated quality score (sm E R R)) to each prediction. M repre- sents the number of predictions, ll is the predicted camera parameters. Higher scores should be assigned to predictions with higher quality and better aligned with the image.
Model architecture. We first introduce the input features into HMMR-Screr. Instead of directly encoding pose param- eters, which are prone ambiguities in representing joint positions and deficient in spatial context, we leverage UVD coordinates as the input to HMR-Score This provides unified and consistent representation of the 3D human skele- ton and preserves the geometric relationships of the skeleton. Specifically, using the camera model, we first project human body keypoints to the input image space to get their UVD coordinates J wed E IR Vx3, where N 29 the number of keypoints. We use multilayer perceptron (MLP) to map Jand to high-dimensional feature vector F E C
Lyn (8mns Gmnn == Vmn log Smn
- (1 - ymnn) log (1 - (1-$m))
where Smn Sigmoid (sm 8n) is the relative quality difference probability between predictions Pm and Px. and Ymu is the ground truth quality label representing the quality difference between predictions based on each of the above- mentioned four HMR metrics. For instance, for the PVE benchmark, the label is represented your (yee if Qeee < Quee and 0 otherwise). The overall training loss for HMR-Scorer is defined as follows:
We utilize multi-scale image features as the image condi- tion, denoted as initially divided into fixed-size patches through patch em- bedding mechanism, producing a sequence of image tokens. These tokens are subsequently processed using a ViT-Base model (Dosovitskiy et al., 2021) to generate a scries of global image feature tokens, denoted as F E xCC**H* xW The global image feature tokens are then passed through a convolutional network to derive the low-channel global fea- tures, represented as F E RC' x x"", A de-convolution head is deployed on the global feature F to obtain the high- resolution local feature map F' E IR XH'' xw" C* and H* x W* denote the feature channel and size, respectively. We sample the local feature F' according to the re-projected 2D joint positions J Juv and obtain pixel-aligned local image features FL E RC' xN for each joint.
The input image I is
ClIMR- Scoree
M
(3)
(Crue +
pappee + Cpaipe + mnn
Cmpipe). L
mn
112, mim1000mm
4.3. HMR Preference Dataset Construction
We leverage preference- optimization rather than tradi- tional supervised training to finetune the base HMR model. However, traditional preference optimization methods re- quire human preference datasets labeled by human annota- tors, which are currently unavailable for this field.
To this end, we propose to use HMR- Scorer to synthesize an HMR preference dataset D = {1, x² : xb) }. where each sample contains the input image I and a pair of human mesh predictions generated from the HMR base model Eyet- Specifically, given a set of predictions (P m J HMR- Scorer assigns scores {sm E { for these predictions. According to the scores, these predictions undergo ordinal arrangement, which emulates human preference in ranking human mesh reconstructions, from highest to lowest fidelity.
The concatenated features of F and FL are subsequently fed into a transformer-encoder based network comprising B fundamental blocks. Each block integrates multi-head self-attention (MHSA) mechanism, a cross-attention (CA) unit, and a feed-forward network (FFN). Within the CA unit, the global image feature F serves as the key and
value features, while the query feature derived from the
4

---

ADHMR: Aligning Diffusion-barse Human Mesh Recovery via Direct Preference Optimization
GTA-Human
DNA-Rendering
PVE
MPIPE
PA-MPIPE
PVL
MPJPE
PA-MPIPE
PLOC
SRCC T
PI.CC 1
SRCC 1
PI.C1
SRCC
PLOCC
SRCC 1
PLOCC 1
SRCC1
PL.C 1
SRCC1
ScoreNet (Xu et al., 2024b)
0.52
0.49
0.52
0.50
0.47
0.43
0.55
0.51
0.55
0.50
0.50
0.46
HMR- Scorer-P P
0.30
0.28
0.28
0.26
0.29
0.25
0.34
0.31
0.32
0.29
0.34
0.27
RMR--Sorer-2D
0.59
0.58
0.59
0.58
0.50
0.49
0.62
0.60
0.63
0.61
0.56
0.53
HMR--corer (Ours)
0.63
0.62
0.63
0.62
0.57
0.54
0.66
0.64
0.66
0.64
0.62
0.58
Table 1.
Score prediction results on the GTA ITA-- (Cai ct al., 2024b) and DNA Rendernng (Cheng ct al., 2023) datasct. We report the PLCC and SRCC between the predicted scores and the PVE, MPJPE, and PA-PVE ground truth errors, respectively.
This hierarchical organization facilitates the extraction of paired samples (P", ,P), denoting superior (winner) and inferior (loser) predictions respectively, where their associ- ated scores satisfy the preference relation (P" > P'] I). The pairing process involves stochastic selection of winners from the top K highest-scoring predictions, coupled with losers from the K lowest-scoring predictions, generating K distinct pairs per image. For studio-captured datasets with precise human mesh annotations, the prediction quality ordering is directly determined by computing the reconstruc- tion error against ground truth labels.
where x = axx + ore" is drawn from q (x} xx) with E" I), Here, X = of/o? denotes the signal-to- noise ratio, w (Az) is weighting function, and the constant T is factored into B.
During training, the model improves by comparing points along the diffusion trajectory with examples from the syn- thetic preference dataset. This helps the model better de- noise winner mesh predictions compared to losers, as evalu- ated by HMR-Scorer. Hence, this methodology guides the model to generate human mesh predictions that not only align closely with the input image but also adhere to real- istic distribution of human poses. By exclusively finetuning the denoiser component within the latent space, the ap- proach achieves more generalized and well-aligned results without overfitting to noisy labels, especially for in-the-wild datasets.
4.4. ADHMR
ADHMR aligns the base model Ege with the constructed preference dataset D (I, x xi)} } to produce superior predictions. The aligned model Eg is initialized using the parameters of the base model Get, while keeping the base model's parameters frozen throughout the training process. The proposed optimization framework extends the direct preference optimization (DPO) method. The principle of DPO lies in its direct optimization of conditional distri- bution eg (Xo c). contrasting with RLHF's approach of optimizing a reward model r (c, Xo), while simultaneously constraining the KL-divergence from a reference distribu- tion Eje
4.5. Data Cleaning
To further evaluate the efficacy of our trained HLMR-Scoree we propose apply it to the training data cleaning process, aiming to determine whether the proposed scorer can effec- tively identify noisy data. While many indoor datasets have ground truth labels, in-the-wild datasets often rely on noisy pseudo labels, which hinders model training and generaliza- tion. Therefore, we use HMR-Scorer to remove low-quality samples, ensuring reliable training dataset.
max Bew (e 60 [r xu)] 3KL [ea (No |
(4)
The data cleaning process begins with score computation, where a quality score s; [0. L] is assigned to each sample (Li, 0., 8r, I1.) in the dataset X' using HMR-Scorer. The score evaluates the alignment of predictions with the input image and the plausibility of the model parameters. Next, a threshold is applied to filter out low-quality samples, resulting in the cleaned dataset Xetemn = {(T,, 0,, 3r, I1.) I s; 2 T). Finally, only high-confidence pscudo-labels are retained for model training.
c) ||err (xo cc].
Following (Wallace et al., 2024), a significant challenge in applying DPO to diffusion models is the intractability of the parameterized distribution co (xo c), which stems from the necessity marginalize over all possible diffusion trajectories (xI...., .T) that culminate in Xg- Through some mathematical techniques, this challenge is addressed by formulating an objective function that operates on the complete denoising trajectory Xo.r:
5. Experiments
(D) = - [xxx x))= -xxxx"xxxxxXaa xx
1-go(-3Tw (A:)
5.1. Setup
_|e" - co (x:",)]1² 2)]1 |e" Eref (x." ; t))12
Training. HMR-Scorer is trained on five datasets, including
HI4D (Yin et al., 2023), BEDLAM (Black et al., 2023), ONA-Rendering (Cheng et al., 2023), GTA-Human (Cai
- -(1le' es (x), t) 1 l|e - -ref (x/: t) ))
(5)
5

---

ADHMR: Aligning Diffusion- based Human Mesh Recovery via Direct Preference Optimization
3DPW
Human3.cm 6M
Method
M
PVE
MPJPE
PA-MPJPE
PVE
MPJPE
PA-MPJPE |
HMR (Kanazawu et al., 2018)
152.7
130.0
81.3
96.1
56.8
HybrIK (Li et al., 2021)
86.5
74.1
45.0
65.7
34.5
PyMaf (Zhang ct al., 2021)
110.1
92.8
58.9
40.5
POTTER (Zheng et al., 2023)
87.4
75.0
44.8
56.5
35.1
ImpHMR (Cho et al., 2023)
87.1
74.3
45.4
Zolly (Wang et al., 2023)
76.3
39,8
49.4
32.3
HMR (Goel al., 2023)
44.5
33.6
HMR (Goel et al., 2023)
54.3
50.0
32.4
ScorcHMR (Stathopoulos al., 2024)
76.8
51.1
10
79.4
56.6
59.2
42.2
Biggs al. (Biggs 2020a)
25
75.8
55.6
58.2
42.2
Sengupta et al. (Sengupta et al., 2021) 1)
25
75.1
47.0
ProHMR (Kolotouros et al., 2021)
25
52.4
36.8
HuManiFlow (Sengupta et al., 2023)
100
65.1
39.9
HMDIf (Foo et al., 2023)
25
82.4
72.7
44,5
49.3
32.4
10
79.8
68.5
41.0
52.5
42.4
29.0
HypoNet (Base Model)
100
73.4
63.0
37,6
47.5
38.4
26.0
200
71.9
61.8
36.1
46.4
37.4
25.3
73.8
38.3
52.1
41.8
28.4
ADHMR
65.4
33.5
45.9
36.9
24.8
63.5
55.7
32.5
44.6
35.8
24.1
71.3
61.3
37.1
52.2
41.9
28.3
ADIMMR (TTW)
100
62.6
54,2
32.0
45.9
37.0
24.8
200
60.5
52.6
30.8
44.6
35.9
24.0
Table 2. Comparison with sale--oftthe-ate on the 3DPW (Von Marcard et al., 2018) and fluman3.6M (Ionescu et al., 2013) dataset. M is the number of predictions in probabilistic methods. ADHMR is finetuned on the target benchmark dataset, while ADHMR (ITW) further finetuned on the preference dataset constructed from an in-the-wild datasct.
et al., 2024b), and SPEC (Kocabas et al., 2021). These datasets contain accurate 3D annotations for human pose, which plays an important role in training an effective scorer. The HMR base model is the current state-of-the-- art proba- bilistic HMR method: HypoNet from (Xu et al., 2024b).
3DP
Method
PVE
MPJPE
PA-MPIPE
HypoNet (Base Model)
73.4
63.0
37.6
(a) Finetune on target benchmark dataset
Supervised finetuning ADHMR
68.0
59.4
35.9
65.4
57.2
33,5
Evaluation metrics. We use four standard human mesh reconstruction metrics: PVE, MPJPE, PA-PVE, and PA- MPJPE, as detailed in Sec. 3.1. To evaluate the scorer, we follow score prediction assessment (Zhai & Min, 2020) to employ two standard metrics: the Pearson linear correlation coefficient (PLCC) and Spearman rank correlation coeffi- cient (SRCC). These correlation coefficients quantify the alignment between the predicted scores and ground truth reconstruction error.
(b) Finetune on in-the-wild dataset
Supervised finetuning ADHMR
70.2
61.3
36.5
62.6
54.2
32.0
Table 3. Ablation of preference finetuning 3DPW (Von Marcard et al., 2018) dataset. M 100 for all models.
predictions of HMR models. We save the corresponding reconstruction errors for the noised predictions. We then measure the PLCC and SRCC between the ground truth reconstruction metrics and the predicted scores.
5.2. HMR-Scorer Evaluation
Test benchmark. Since there are no existing datasets for score prediction of HMR models, construct test set from synthetic dataset GTA-Human (Cai et al., 2024b), which is produced with rendering engines (e.g., Unreal En- gine) and contains accurate 3D annotations. We also use ONA-Rendering (Cheng et al., 2023), a large-scale multi- view studio-based dataset with an ultra-high resolution, to construct the other test set to show the capacity for com- studio-based scenes. We adopt the original test set of the selected datasets. Then perturb the ground truth pose labels with random gaussian noise to simulate
Results. Table presents the comparison results for score prediction. We observe that our scorer outperforms all base- line methods in both PLCC and SRCC metrics, showcasing its efficacy. For comparison, we modify the reward model in ScoreHypo baseline. We also study two ablations of HMR-Scoree HMR- HMR-SSoore-P. accepts SMPL(-X) pose rotation vectors input, and joint positions (without joint depth) as input. Results show that HMR- -corrr outperforms the baselines in aligning the scores with the real reconstruction errors, which underscores effectiveness indicating human mesh prediction quality.
accepts 2D
6

---

ADHMR: Aligning diffusion-based Human Mesh Recovery via Direct Preference Optimization
Input
Ours
ScoreHypo
Input
Ours
ScoreHypo
(a)
(b)
cc)
(c)
(e)
(
(g
(h)
Shart
Short
Figure 2. Qualitative comparison of the state-of-ther tort--------ttttt probabilistic model ScoreHypo (Xu et al., 2024b) and our ADHMR, Our framework significantly improves image alignment and in-the-wild rohustness. (a) ~ (f) are from the 3DPW (Von Marcard al., 2018) dataset, and (g) (h) challenging in mn-the-wild images.
3DPW
further finetuned on Insta Variety dataset (Kanazawa et al., 2019), which contains various in-the-wild images annotated with noisy pseudo-labels collected from Instagram. Please note that we do not use the original 3D labels but use the HMR-Scores to construct a preference dataset. Notably, we observe that our finetuned model achieves better per- formance using M = 10 predictions than the base model using M 200 predictions on the in-the-will benchmark 3DPW, showcasing that our finetuning pipeline greatly en- hances the generalizability in-the-wild datasets and its efficiency. Moreover, ADHMR consistently outperforms existing state-of-the-art methods by a substantial margin.
Method
PVE
MPJPE
PA-MPIPE
HypoNet (Base Model)
71.9
61.8
36.1
Supervised finetuning
69.9
59.7
35.2
ADHMR (ITW)
60.5
52.6
30.8
Table 4. Ablation of extra training data 3DPW (Von Marcard al., 2018) dataset. We use multiple training datasets the scorer to perform supervised finetuning. M = 200 for all models.
5.3. ADHMR Evaluation
Comparisons with state-of-the-art methods. In Table 2, compare the accuracy of the ADHMR with state-of- the-art methods on two widely used benchmark datasets Human3.66 66 (Ionescu et al., 2013) and 3DPW (Von Mar- card et al., 2018). 3DPW an in-the-wild dataset. We show results of both deterministic and probabilistic meth- ods. Following the conventions of standard probabilistic approaches (Xu et al., 2024b; Biggs et al., 2020b), we gen- erate multiple estimates and report the min-MPJPE and min-PVE of the M predictions. When finetuned directly on the target benchmark dataset, ADHMR achieves further enhancements, showcasing its strong ability to adapt to and balance -pmaii- distribution. To evaluate the effec- tiveness under the un-the-wild setting, ADHMR (TTW) is
Qualitative results. Fig. 2 shows qualitative comparisons between ADHMR and the previous state-of-the-a proba- bilistic method ScoreHypo. We show randomly selected results of ScoreHypo and ADHMR on 3DPW and internet images with A 10 candidate predictions. We can see that the finetuned model can produce more accurate results for body pose under challenging cases, such as dense human- environment interactions. The base model, however, cannot achieve good image-mesh alignment. For example, in the (a) instance, our method provides more reasonable poses for the occluded right arm. In the (c) instance, ScoreHypo gives erroneous prediction for the person's 's arms, while ours gives more accurate body pose prediction, proving the

---

ADHMR: Aligning Diffusion-barse Human Mesh Recovery via Direct Preference Optimization
3DPW
Human3.6M
EHF
Method
MPJPE 4
PA-MPJPE 1
MPJPE +
PA-MPIPE 4
PA-PVE 4
PVE J
PA-MPJPE 4
Hand4Whole (Moon et al., 2022)
115.2
75.4
78.8
57.7
57.8
89.2
70.2
+ data cleaning
112.3
73.7
77.9
57.0
56.2
88.8
69.8
OSX (Base) (Lin al., 2023)
100.4
66.4
69.5
48.9
54.7
86.6
63.7
data cleaning
99.4
65.2
65.7
46.4
53.1
84.0
62.4
SMPLer-X-BB -a-e (Cai et al., 2024a)
99.5
64.2
59.8
45.8
51.0
82.4
59.7
data cleaning
97.9
62.9
57.5
43.8
51.4
78.6
58.6
Table Quantitative comparisons between several state-of-the methods with and without data cleaning on the 3DPW (Von Marcard et al.. 2018), Human3.6M (lonescu et al., 2013) and EHF (Pavlakos et al., 2019) dataset. All methods are trained on four commonly used datasets. After cleaning the training data, these models achieve higher accuracy even when trained on smaller subset.
efficacy of the proposed finetuning pipeline. In the (g) in- stance, ScoreHypo produces inaccurate elbow poses, while our prediction fits the input image better. Results show that ADHMR is more robust for challenging internet images than ScoreHypo. Please zoom in to observe our improvement over the base model.
towards generating high-quality predictions.
5.4. Data Cleaning Results
Current HMR models are trained using quite differ- ent datasets. For fair and comprehensive compari- son, we selected several state-of-the-art HMR methods and retrained them using four commonly used datasets: MSCOCO (Lin et al., 2014), MPII (Andriluka et al., 2014), Human3.66 (Ionescu et al., 2013), and MPI-INF- 3DHP (Mehta et al., 2017). We compare training the models on the full training sets of these datasets and training on the filtered datasets obtained after applying data cleaning with the HMR-Scorer. We set the filter threshold 0.6.
Ablation of preference finetuning. As shown in Table 3, we conduct an ablation study on different finetuning meth- ods to demonstrate the advantages of ADHMR over tradi- tional supervised finetuning. We construct two baselines where the base model is finetuned using the ground truth labels in the datasets. We first finetune the base model on the training sets of the two target benchmarks (3DPW and Human3. 6M). We also finetune on the pseudo labels of Insta Variety simulate training on noisy pseudo-labeled in- the-wild data. Results show that ADHMR consistently out- performs traditional supervised finetuning in both settings. When finetuned directly on the target benchmark dataset, our method achieves further enhancements than supervised fine- tuning, showcasing its strong ability to adapt to and balance domain-specific distribution. the meantime, supervised finetuning may overfit one training dataset (3DPW) and the performance on the other test benchmark (Human3. ) could be corrupted. On the in-the- dataset, ADHMR demonstrates robustness under noisy pseudo-label condi- tions, while supervised finetuning on the noisy labels could overfit the training dataset and harm its performance on 3DPW benchmark.
Quantitative results are in Table 5. The results demonstrate that models trained on the cleaned datasets achieve better performance despite using less training data. This improve- ment highlights the utility of our method in filtering out low-quality training samples, thereby enabling the models to focus on higher-quality data for learning. This finding pro- vides new perspective for improving large HMR models by strategically cleaning training data. By leveraging the HMR- Scorer to curate datasets, we can achieve higher-quality model training with even less data, making it a valuable tool for effortless performance gain.
6. Conclusion
In this work, we propose ADHMR, the first framework for aligning diffusion-based HMR models with direct prefer- ence optimization. We leverage trained HMR-Scorer to synthesize a preference dataset automatically without the need for manual annotation. This dataset is then used to align the diffusion-based HMR model through direct pref- erence optimization. Additionally, HMR- -corer improves the performance of several state-of-the-arr HMR models by filtering out low-quality training data. Extensive exper- iments validate the effectiveness of ADHMR. We believe that our work will pave the way for future advancements in alignment techniques for human mesh recovery.
Ablation of extra training datasets. In Table 4, we aim to confirm that the improvements achieved by ADHMR contributed to its inherent optimization strategy, rather than including extra information from other datasets used during scorer training. So we additionally finetune the base model on the scorer's 's training sets. Supervised finetuning on scorer training datasets yields insignificant performance gains, confirming that the improvements are not primarily due to external dataset information. This suggests that sim- ply exposing the base model to more data does not guarantee better performance. In contrast, ADHMR benefits from the ability of LMMR-Scorer to implicitly guide the base model
8

---

ADHMR: Aligning Diffusion-bars based Human Mesh Recovery via Direct Preference Optimization
Acknowledgements
Chen, C., Wei, J., Chen, T., Zhang, C., Yang, X., Zhang, S., Yang, B., Foo, C.-S., Lin, G., Huang, Q. et al. Cad- crafter: Generating computer-aided design models from unconstrained images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
This study is supported under the RIE2020 Industry Align- ment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from the industry partner(s). This research is also supported by S under its MTC Programmatic Funds (Grant No. M23L760021) and the MoE AcRF Tier 1 grant (RG14/22).
Cheng, W., Chen, R., Fan, S., Yin, W., Chen, K., Cai, Z., Wang, J., Gao, Y., Yu, Z., Lin, Z., et al. Dna-rendering: A diverse neural actor repository for high-fidelity human- centric rendering. In Proceedings of the IEEE/CVF In- ternational Conference Computer Vision, pp. 19982- 19993, 2023.
Impact Statement
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Cho, H. and Kim, J. Generative approach for probabilistic human mesh recovery using diffusion models. In Pro- ceedings of the IEEF/CVF International Conference on Computer Vision, pp. 4183-4188, 2023.
References
Andriluka, M., Pishchulin, L., Gehler, P., and Schiele, B. 2d human pose estimation: New benchmark and state of the art analysis. In Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp. 3686-3693, 2014.
Cho, H., Cho, Y., Ahn, J., and Kim, J. Implicit 3d human mesh recovery using consistency with pose and shape from unseen-view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21148-2111 1158, 2023.
Biggs, B., Novotny, D., Ehrhardt, S., Joo, H., Graham, B., and Vedaldi, A. 3d multi- bodiiss Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems, 33:20496-20507,, 2020a.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances neural information pro- cessing systems, 30, 2017.
Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x words: Transformers for image recognition at scale. ICLR, 2021.
Biggs, B., Novotny, D., Ehrhardt, S., Joo, H., Graham, B., and Vedaldi, A. 3d multi-bodics Fitting sets of plausible 3d human models to ambiguous image data. Advances in neural information processing systems, 0-22049-225017 2020b.
Dwivedi, S. K., Sun, Y., Patel, P., Feng, Y., and Black, M. J. Tokenhmr: Advancing human mesh recovery with a tokenized pose representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024.
Black, M. J., Patel, P., Tesch, J., and Yang, J. Bedlam: A synthetic dataset of bodies exhibiting detailed lifelike animated motion. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 8726-8737, 2023.
Recognition, pp.
Fang. Q. Chen, K., Fan, Y., Shuai, Q., Li, and Zhang, W. Learning analytical posterior probability for human mesh recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8781- 8791, 2023.
Burges, C., Shaked, T., Renshaw, E., Lazier, A., Deeds, M., Hamilton, N., and Hullender, G. Learning to rank using gradient descent. In Proceedings of the 22nd international conference on Machine learning, pp. 89-96,920 2005.
Cai, Z., Yin, W., Zeng, A., Wei, C., Sun, Q., Yanjun, W., Pang, H. E., Mei, H., Zhang, M., Zhang, L., et al. Smpler- x: Scaling up expressive human pose and shape estima- tion. Advances in Neural Information Processing Systems,
Foo, L. G., Gong, J., Rahmani, H., and Liu, J. Distribution- aligned diffusion for human mesh recovery. In Proceed- ings of the IEEE/CVF International Conference Com- puter Vision, pp. 9221-9232, 2023.
36, 2024a.
Goel, S., Pavlakos, G., Rajasegaran, J., Kanazawa, A., and Malik, J. Humans in 4d: Reconstructing and tracking hu- mans with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 14783- 14794, 2023.
Cai, Z., Zhang, M., Ren, J., Wei, C., Ren, D., Lin, Z., Zhao, H. Yang, L., Loy, C. C., and Liu, Z. Playing for 3d human recovery. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
9

---

ADHMR: Aligning Diffusion- based Human Mesh Recovery via Direct Preference Optimization
Hong, Z. and Shen, W. Free-viewpoint video in the wild using flying camera. In ECCV 2024 Workshop on Wild 3D: 3D Modeling, Reconstruction, and Generation in the Wild, 2024.
Lassner, C., Romero, J., Kiefel, M., Bogo, F,, Black, M. J., and Gehler, P. V. Unite the people: Closing the loop be- tween 3d and 2d human representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6050-6059, 2017.
Huang, B., Li, C., Xu, C., Pan, L., Wang. Y., and Lee, G. H. Closely interactive human reconstruction with proxemies and physics-guides adaption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Lee, K., Liu, H.,, Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXiv:2302. 12192, 2023.
Li, J., Xu, C., Chen, Z., Bian, S., Yang. L., and Lu, C. Hybrik: A hybrid analytical inverse kinematics solution for 3d human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3383-33923 2021.
Ibarz, B., Leike, J., Pohlen, T., Irving, G., Legg, S., and Amodei, D. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
Ionescu, C., Papava, D., Olaru, V., and Sminchisescu, C. Human3. 6m: Large scale datasets and predictive meth- ods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-1339, 2013.
Li, J., Bian, S., Xu, C., Chen, Z. Yang, L., and Lu, C. Hybrik-x: Hybrid analytical- inverse kine- matics for whole-body mesh recovery. arXiv preprint arXiv:2304.0566 2023.
Lin, J., Zeng, A., Wang, H., Zhang, L., and Li, Y. One-stage 3d whole-body mesh recovery with component aware transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 21159 21168 2023.
Kanazawa, A., Black, M. J., Jacobs, D. W., and Malik, J. End-to-end recovery of human shape and pose. In Proceedings of the IEEE conference on computer vision and patlem recognition, pp. 7122-7131, 2018
Lin, T.- Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Visi 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Kanazawa, A., Zhang, J. Y., Felsen, P., and Malik, J. Learn- ing 3d human dynamics from video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5614-5623, 2019.
Kocabas, M., Huang, C.-H. P., Tesch, J., Müller, L., Hilliges, O., and Black, M. J. Spec: Sceing people in the wild with an estimated camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11035- 11045, 2021.
Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., and Black, M. Smpl: skinned multi-person linear model. ACM Transactions on Graphics (TOG), 34(6):1-16, 2015,
Mchta, D.,, Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., and Theobalt, C. Monocular 3d human pose estimation in the wild using improved enn supervision. In 2017 international conference on 3D vision (3DV), pp- 506-516. IEEE, 2017.
Kolotouros, N., Pavlakos, G., Black, M. J., and Daniilidis, K. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2252-2261,,000
Moon, G., Choi, H., and Lee, K. M. Accurate 3d hand pose estimation for whole-body 3d human mesh estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2317-2317, 2022.
Kolotouros, N., Pavlakos, G., Jayaraman, D., and Daniilidis, K. Probabilistic modeling for human mesh recovery. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1100--16644 2021.
Pang, H. E., Cai, Z., Yang, L., Tao, Q. Wu, Z., Zhang, T., and Liu, Z. Towards robust and expressive whole-body human pose and shape estimation. Advances in Neural Information Processing Systems, 36. 2024.
Kreutzer, J., Uyheng, J., and Riezler, S. Reliability and learnability of human bandit feedback for sequence-to- sequence reinforcement learning. In Proceedings of the 56th Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. 1777
Paylakos, G., Choutas, V., Ghorbani, N., Bolkart, T., Osman, A. A., Tzionas, D., and Black, M. J. Expressive body capture: 3d hands, face, and body from a single image.
1788, 2018
10

---

ADHMR: Aligning Diffusion-basee Human Mesh Recovery via Direct Preference Optimization
In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10975-10985, 2019.
Von Mareard, T., Henschel, R., Black, M. J., Rosenhahn, B., and Pons-Moll, G. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pp. 601-617, 2018.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wallace, B., Dang, M., Rafailov, R., Zhou, L., Lou, A., Purushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct pref- erence optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8228-828888888 8238, 2024.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 17X7:07777.6647,, 2017.
Sengupta, A., Budvytis, I., and Cipolla, R. Hierarchical kinematic probability distributions for 3d human shape and pose estimation from images in the wild. In Pro- ceedings of the IEEE/CVF international conference on computer vision, pp. 1 12119-11229, 2021.
Wang, W., Ge. Y., Mei, H., Cai, Z., Sun, Q.. Wang, Y., Shen, C., Yang, L.., and Komura, TT. Zolly: Zoom focal length correctly for cesseetive-distoree human mesh recon- struction. In Procedingss of the IEEF/CVF International Conference on Computer Vision, pp. 3925-3935, 2023.
Sengupta, A., Budvytis, I., and Cipolla, R. Humaniflow: Ancestor-conditooee normalising flows on so (3) mani- folds for human pose and shape distribution estimation. In Proceedings of the IEEF/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 4779-4789, 2023.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong, Y. Imagereward: Learning and evaluating hu- man preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024a.
Xu, Y., Ma, X., Su, J., Zhu, W., Qiao, Y., and Wang, Y. Scorehypo: Probabilistic human mesh estimation with hypothesis scoring. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 979-989, 2024b.
Shen, W. Yin. W., Wang, H., Wei, C., Cai, Z., Yang. L., and Lin, G. Hmr-adapter: A lightweight adapter with dual-path cross augmentation for expressive human mesh recovery. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 1093-6102, 2024.
Xu, Z., Song, C.,, Song, G., Zhang, J., Liew, J. H., Xu, H., Xie, Y., Luo, L., Lin, G., Feng, J., et al. High quality human image animation using regional supervision and motion blur condition. arXiv preprint arXiv:2409. 0904409.19880 19580, 2024c.
Shuai, Q., Geng, C.,, Fang, Q., Peng. S., Shen, W., Zhou, X., and Bao, H. Novel view synthesis of human interactions from sparse multi-view videos. InACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
Yang. E., Chen, T., He, X., Cai, Z., Yang. L.,, Wu, S., and Lin, G. Attrihuman-3d: Editable 3d human avatar gen- eration with attribute decomposition and indexing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10596-10605, 2024.
Stathopoulos, A., Han, L., and Metaxas, D. Score-guided diffusion for 3d human recovery. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 906-90 915, 2024.
Stiennon, N., Ouyang, L., Wu, J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Yao, N., Zhang, G., Shen, W., Shu, J., and Wang, H. Unify3d: An augmented holistic end-to-end monocular 3d human reconstruction via anatomy shaping and twins negotiating. arXiv preprint arXiv:2504. 18215, 2025.
Ye, J., Liu, F., Li, Q., Wang, Z., Wang, Y., Wang, X., Duan, Y., and Zhu, J. Dreamreward: Text-to-3d generation with human preference. arXiv preprint arXiv:2403.1466 2024.
Sun, Q. Wang, Y., Zeng, A., Yin, W., Wei, C., Wang. W., Mei, H., Leung, C.-S., Liu, Z., Yang, L., et al. Aios: All- in-one-stage expressive human pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1834-1843, 2024.
Yin, W., Cai, Z., Wang, R., Wang. F., Wei, C., Mei, H., Xiao, W., Yang, Z., Sun, Q. Yamashita, A., et al. Whac: World- grounded humans and cameras. In European Conference on Computer Vision, pp. 20-37. Springer, 2024.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.

---

ADHMR: Aligning Diffusion-based Human Mesh Recovery via Direct Preference Optimization
Yin, W., Cai, Z., Wang, R., Zeng, A., Wei, C., Sun, Q., Mei, H., Wang, Y., Pang, H. E., Zhang, M., et al. Smplest-x: Ultimate scaling for expressive human pose and shape estimation. arXiv preprint arXiv:2501.09788 2025.
Yin, Y., Guo, C., Kaufmann, M., Zarate, J., Song, J., and Hilliges, O. Hi4d: 4d instance segmentation of close human interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17016-17027, 2023.
Zhai, G. and Min, X. Perceptual image quality assessment: survey. Science China Information Sciences, 63:1-52, 2020.
Zhang, H., Tian, Y., Zhou, X., Ouyang, W., Liu, Wang, L., and Sun, Z. Pymaf: 3d human pose and shape regres- sion with pyramidal mesh alignment feedback loop. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11446-11453 1456, 2021.
Zhang, S., Ma, Q., Zhang, Y., Aliakbarian, S., Cosker, D., and Tang, S. Probabilistic human mesh recovery in 3d scenes from egocentric views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1989-8000, 2023.
Zhang, S., Yang, X., Feng, Y., Qin, C., Chen, C.-C., Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., et al. Hive: Harnessing human feedback for instructional visual editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9026- 9036, 2024.
Zheng, C., Liu, X., Qi. G.-J,, and Chen, C. Potter: Pooling attention transformer for efficient human mesh recovery. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Puttern Recognition, pp. 1611-1620, 2023.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, P., and Irving, G. Fine-tuning language models from human preferences. arXiv preprint PrXiv:990900893, 2019.
12


Reviews:
[{"review_id": "pldREEuOVX", "reviewer": "Reviewer_WdV4", "summary": "This paper targets improving HMR methods with preference prediction. Therefore, the authors present a prediction assessment model named HMR-Scorer. Further, the authors create a preference dataset using HMR-scorer, which is used to finetune base model and existing HMR methods. The full method, called ADHMR, shows sota results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing the strong performance of our method and the clear presentation of our paper. We deeply appreciate your constructive comments and address them below.\n\n### **Q1. Equation 5 elucidation**\n\nThank you for your helpful comment. Our formulation follows the approach introduced in Diffusion-DPO [A], which provides a detailed derivation (see Section 4 and Supp. S2 of that paper). To improve readability, we will revise our manuscript to explicitly cite this reference at Line 254 and include a pointer to the key steps in the original paper.\n\n[A] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" CVPR. 2024.\n\n### **Q2. Results in Table 5 not in Table 2**\n\nWe separate Table 2 and Table 5 as they serve different goals. Table 2 evaluates the ADHMR framework with SMPL-based methods for fair comparison, while Table 5 focuses on data cleaning with more challenging SMPL-X data. Methods in Table 5 are deterministic and not compatible with ADHMR. We will clarify this in the final version.\n\n\n### **Q3. About ScoreNet training set**\n\nThank you for the insightful comment. While we understand the concern, it’s important to note that ScoreNet and our HMR-Scorer were originally trained on different datasets. Specifically, ScoreNet was trained on H3.6M, 3DPW, MPI-INF-3DHP, MPII, COCO, and UP-3D, while our HMR-Scorer was trained on a set of synthetic and human-interaction datasets with higher annotation accuracy and scene diversity, as detailed in Sec. 5.1.\n\nTo ensure a fair comparison, we re-trained ScoreNet using our HMR-Scorer training sets. Please check the results in the updated Table 1:\n\n| Method              | PVE ↑ PLCC | PVE ↑ SRCC | MPJPE ↑ PLCC | MPJPE ↑ SRCC | PA-MPJPE ↑ PLCC | PA-MPJPE ↑ SRCC | PVE ↑ PLCC | PVE ↑ SRCC | MPJPE ↑ PLCC | MPJPE ↑ SRCC | PA-MPJPE ↑ PLCC | PA-MPJPE ↑ SRCC |\n|---------------------|------------|-------------|---------------|----------------|------------------|-------------------|------------|-------------|----------------|-----------------|------------------|-------------------|\n|                     |       GTA-Human|         |              |         |                 |         |       DNA-Render|\n| ScoreNet            | 0.52       | 0.49        | 0.52          | 0.50           | 0.47             | 0.43              | 0.55       | 0.51        | 0.55           | 0.50            | 0.50             | 0.46              |\n| **ScoreNet$\\dagger$**           | **0.55**   | **0.51**    | **0.54**      | **0.51**       | **0.50**         | **0.46**          | **0.57**   | **0.54**    | **0.56**       | **0.54**        | **0.51**         | **0.48**          |\n| HMR-Scorer (Ours)   | **0.63**   | **0.62**    | **0.63**      | **0.62**       | **0.57**         | **0.54**          | **0.67**   | **0.64**    | **0.66**       | **0.65**        | **0.62**         | **0.59**          |\n\n\n**ScoreNet$\\dagger$** is the ScoreNet that is trained on the same train sets as HMR-Scorer.\nResults show that our HMR-Scorer still outperforms ScoreNet$\\dagger$ using the same set of train sets, validating the design of our scoring strategy rather than dataset bias.\n\nWe will clarify this point and explicitly list the datasets used in Table 1 to improve rigor and transparency.\n\n### **Q4. Seen datasets of methods in Table 2**\n\nThank you for your valuable comment. We will include a clearer description of the datasets used in Table 2 in the revised version. Specifically:\n\n1. ADHMR and HypoNet are trained on the same set of datasets, and ADHMR does not use HMR-Scorer. This comparison demonstrates the effectiveness of our framework even without additional supervision.\n2. ADHMR (ITW) is not directly trained on the datasets used for training HMR-Scorer (e.g., DNA-rendering). However, it may indirectly benefit from these extra datasets. As you rightly pointed out, Table 4 shows that our method is significantly more effective than directly fine-tuning with those datasets, which further highlights the strength of our approach.\n\n### **Q5. About Figure 2**\n\nThank you for your careful analysis and helpful suggestions.\n\nWe have updated Figure 2 by adding ground truth and side-view visualizations for better interpretability. In addition, we have replaced the first example with a clearer case. The revised figure is available here:\n\nhttps://postimg.cc/gx0YG3G8\n\nIn the original first example, the man’s right hand is fully occluded, and our model still predicts a plausible pose based solely on visible cues. In contrast, HypoNet’s result is not consistent with the observed image evidence. It is worth noting that our model processes single-frame images only and does not leverage any temporal context from the sequence.\n\nWe appreciate your detailed observation and have reflected this clarification in the updated figure.", "claims_and_evidence": "I think the claims are clear.", "methods_and_evaluation_criteria": "The proposed method ADHMR has been tested on several standard datasets such as 3DPW and Human3.6M using standard metrics MPJPE, etc.", "theoretical_claims": "At L.254, the authors said \"through some mathematical techniques ...\" and yield Eqn.(5). It would be better to elucidate the procedure in appendix or guide the readers where could find similar provements.", "experimental_designs_or_analyses": "I have some questions about the experiments.\n\n1. Why methods in Table.5 do not appear in Table.2? Any special reasons? \n\n2. In Table 1, does ScoreNet share the same training set with HMR-Scorer? How to ensure this is a fair comparison? \n\n3. Table.2 is a bit unfair because ADHMR is informed of more training data through HMR-Scorer which is trained on DNA-rendering etc. I do not doubt about the effectiveness of ADHMR, Table.4 has proven that. Additional descriptions about seen datasets in Table.2 would make it more rigorous.", "relation_to_broader_scientific_literature": "This paper is based on CVPR 2024 paper HypoNet with significant differences.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "Strength: \n\n1. The paper is well written. The contributions are stated clearly. Incorporating DPO with HMR is an effective idea. \n\n2. Experiments have shown the improvements brought by ADHMR. \n\nWeakness: \n\n1. In Figure 2, it would be more informative to show GT and side view as well. \n\n2. The first visual result in Figure 2 is not very accurate. In this image, the man is playing the baseketball with his left hand. The proposed ADHMR predicts that the right hand is also on the ball. However, the shadow on the ground indicates that the right hand is far away from the ball. In fact, this image is the 78th frame of the seq \"courtyard_basketball_00\". From the whole sequence, we would find that the right hand of the man is placed on the right of his right leg, and is totally occluded. Therefore this sample is not that good.", "other_comments_or_suggestions": "L.087. (Rafailov et al. 2024) duplicated.", "questions_for_authors": "No.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vpqjRYUYJ7", "reviewer": "Reviewer_1fDj", "summary": "This paper proposes the first method to use preference optimziation to improve the Human Mesh Recovery (HMR) models. The paper first introduces a HMR-scorer model to rank the human mesh result produced by (an arbitrary) HMR method. Experiments show the score is strongly correlated to the reconstruction metrics, supporting that this scorer model can be reliably applied to other datasets without 3D ground truths. This scorer is then used to construct preference dataset of {Image, winner result, loser result} triplets. These data are used to fine-tune a base HMR model using the direct preference optimization (DPO) loss. Result: a big leading margin in the HMR accuracy on common benchmarks. Comprehensive experiments validated the different stages in the paper. Overall, I think this is a solid paper.\n\nUpdate after rebuttal: after reading other reviewers' comments and the authors' rebuttal, my rating of acceptance remains unchanged.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for highlighting the novelty of applying preference optimization to HMR, the effectiveness of our framework, and the comprehensive experiments. We truly appreciate your encouraging feedback and respond to your points below.\n\n### **Q1. Equation 5 elucidation**\n\nThank you for your helpful comment. Our formulation follows the approach introduced in Diffusion-DPO [A], which provides a detailed derivation (see Section 4 and Supp. S2 of that paper). To improve readability, we will revise our manuscript to explicitly cite this reference at Line 254 and include a pointer to the key steps in the original paper.\n\n[A] Wallace, Bram, et al. \"Diffusion model alignment using direct preference optimization.\" CVPR. 2024.\n\n### **Q2. Preference dataset application**\n\nThank you for the great question. The preference dataset is not directly applicable to standard non-probabilistic HMR models, as the DPO framework assumes the model produces discrete predictions with associated probabilities. Our current implementation is based on a diffusion-based HMR model, which naturally supports such probabilistic outputs, consistent with the assumptions in Diffusion-DPO.\n\nHowever, we believe adapting deterministic models to fit within this preference framework is both feasible and promising. As demonstrated in TokenHMR [B], one can first train a VQ-VAE to discretize the continuous pose space, and then modify the original regression head (e.g., in HMR2.0) into a classification head over quantized tokens. This adaptation would allow deterministic HMR models (e.g., HMR 2.0) to benefit from our preference datasets and preference optimization framework, unlocking similar gains in robustness and image alignment. We consider this a promising direction for future research.\n\nImportantly, the HMR-Scorer and our data cleaning pipeline are fully model-agnostic, and thus directly applicable to both probabilistic and deterministic models. They provide a scalable way to improve training data quality by filtering out noisy pseudo-labels—particularly useful for in-the-wild datasets where 3D ground truth is often unavailable or unreliable. We view this as a practical step toward broader adoption of preference-guided learning in HMR, and we plan to explore these extensions in future work.\n\n\n\n[B] Dwivedi, Sai Kumar, et al. \"Tokenhmr: Advancing human mesh recovery with a tokenized pose representation.\" CVPR. 2024.\n\n### **Q3. Typos**\n\nThank you for pointing out the duplicated citation. We will fix this in the final version.", "claims_and_evidence": "Claim 1: the proposed method is the first framework that enhances HMR mdoels using preference optimziation. To my knowledge, this is true.\nClaim 2: the HMR-scorer model is effective. This is validated by experiments Sec. 5.2. \nClaim 3: the HMR-scorer improves the SOTA HMR models by filtering out low-quality training data. This is also validated through comprehensive experiments in Sec. 5.3, in particular Table 5.\n\nTo summarize, the major novelty and technical claims are valid and verified.", "methods_and_evaluation_criteria": "The proposed method makes of sense to me, and matches well with the evaluation. \nFirst, for the HMR-scoerer, the motivation is to replace human preference rating in the dataset curation step for the preference optimization methods. If the dataset has precise ground 3D ground truth, the direct computation of the errors are used for the ranking. If the data gt annotations are noisy (e.g. in-the-wild datasets), the scorer function can help effiectively create positive and negative samples. Consequently, the evaluation just need to verify the scores are correctly correlated to the common HMR error metrics (such as per-vertex errors, body joint errors etc.) -- that's how Section 5.1 and 5.2 are designed. Therefore I think this validation is a good design.\n\nSecond, using the constructed preference dataset for DPO of the diffusion-based HMR method -- sounds like a straightforward thinking in hindsight -- is a great idea and this paper is the first to conduct so. The method creates impressive performance improvement on the standard benchmarks (Table 2), which I think sufficiently validates the efficacy of the design.", "theoretical_claims": "To my understanding, this paper is more about applying the DPO method (Wallace et al., 2024) to the specific problem of improving diffusion-based probabilistic human mesh recovery. Therefore the major theoretical claim (while indirectly made), is that DPO is effective for this setting. The experiments have validated this claim.", "experimental_designs_or_analyses": "As discussed above, the experiment are designed to validate the major technical claims. They have achieve this goal.\n1. The first part of the experiment is to validate if the HMR scorer model can align with the common error metrics in HMR. This is validated on the datasets with precise ground truth. Metrics here are the correlation coefficients which I believe are sensibly chosen.\n2. The second part is about whether the preference-rated data can improve common HMR models. This is validated on a detailed benchmarking and ablation study (table 3). \n3. The third part of the experiment is to use the HMR-scorer model to clean up the data. The effectiveness is demonstrated on 3 different base models.\nTherefore, I think the experimental designs are comprehensive and effective.", "relation_to_broader_scientific_literature": "The models and datasets delivered by this work can be applied to more precise human pose/motion estimation from images or video streams. This can facilitate many applications in other areas, such as motion caption in the movie industry, pose estimation in live sports, sport medicine and analysis, etc.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "Strengths:\n- Overall, I think this paper proposes a good idea that is validated through extensive experiments. The scorer network should be a drop-in enhancement for diffusion-based probabilistic HMR models. The method has improved the performance on common HMR benchmarks by a large margin -- such big improvement is not often seen in recent 2 years. The paper is well written too. Therefore I recommend acceptance.\n\nWeakness: L. 254, \"through some mathematical techniques\" -- it would be good to refer the reader to the corresponding location in the original literature.", "other_comments_or_suggestions": "- I particularly appreciate the detailed captions of the tables and figures.\n- Minor: DPO paper is cited twice on L. 87, right column.", "questions_for_authors": "Is the Preference dataset also applicable to non-probabilistic HMR models to boost their performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5SfnOCGZnv", "reviewer": "Reviewer_8ugF", "summary": "The paper adapts the works of Diffusion-based DPO (DDPO) to HMR by proposing ADHMR. Specifically, the paper introduces an HMR-scorer model that generates a reward for image-mesh alignment. This module is given the local, sampled from the UV joint locations, and global image features and outputs a score. The network is then trained similarly to RankNet on synthesized noisy annotations from ground truth SMPL poses. The HMR-scorer is then used for preference dataset synthesis, HMR DDPO, and data cleaning. The paper highlights the positive impact of each of these applications, demonstrating the effectiveness of their scoring strategy. Specifically, the paper shows significant improvements on their base model, HypoNet, through preference fine-tuning and a stronger scoring model. The paper also shows that the data-cleaning strategy can improve the performance of SOTA HMR models.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the effectiveness of our scoring strategy, significant improvements over the base model, and good experiment design. We deeply appreciate your valuable comments and address them below.\n\n### **Q1. Generalizability to unseen data & Bias caused by DPO**\n\nTo further support our claim on in-the-wild robustness, we provide more ITW qualitative results here:\n\nhttps://postimg.cc/jwY2qCgT\n\nThese examples demonstrate improved performance under extreme poses, occlusions, and complex backgrounds—for example, (a–b) involve lying or rolling on the floor, where our method shows better 2D alignment.\nIn the paper, Figure 2 (g–h) shows ITW images without ground truth, allowing only qualitative comparison. Results on 3DPW also partially reflect ITW capability with outdoor, unconstrained scenes.\n\n**Bias brought by DPO.** \nOur framework improves generalization by shifting the objective from fitting noisy in-the-wild pseudo labels to learning from relative preferences, encouraging the model to capture perceptually aligned and semantically meaningful poses (see L55–59). \nBesides, HMR-Scorer implicitly learns 2D image alignment by integrating global and pixel-aligned features. This leads to better robustness and transferability in challenging real-world scenarios.\n\n### **Q2. More relevant references**\n\nThank you for the helpful comment. While our work is the first to apply DPO to HMR, we acknowledge related efforts in motion generation and reward modeling.  We also agree that data cleaning has been explored in prior works. For example, SPIN [A] filters training data based on reprojection error, and Unite the People [B] relies on manual selection. We will include these references in the revision.\n\n[A] Kolotouros et al., Learning to reconstruct 3D human pose and shape via model-fitting in the loop, ICCV 2019.\n\n[B] Lassner et al., Unite the people: Closing the loop between 3D and 2D human representations, CVPR 2017.\n\n### **Q3. About contribution**\n\nThank you for your thoughtful assessment and for acknowledging the strength of our experimental design.\n\nWhile DDPO is well-established in tasks like image generation, its application to HMR is non-trivial. ADHMR introduces key domain-specific innovations:\n- a learned HMR-Scorer capturing perceptual image–mesh alignment, and\n\n- a preference dataset constructed without human labels.\n\nADHMR is the first to successfully apply DPO to HMR, enabling training from relative preferences rather than noisy ITW pseudo labels. In contrast, conventional methods often overfit these labels, degrading 3D accuracy—as also noted in TokenHMR [C].\n\nBeyond performance gains, HMR-Scorer supports automatic data cleaning, making it broadly helpful in enhancing other HMR pipelines (see Sec. 4.5).\n\n[C] Dwivedi, Sai Kumar, et al. \"Tokenhmr: Advancing human mesh recovery with a tokenized pose representation.\" CVPR. 2024.\n\n### **Q4. Generalization to other hypothesis-generation approaches**\n\nThank you for the valuable suggestion. We chose HypoNet as our main baseline as it is the latest SOTA diffusion-based HMR model for general-purpose scenarios. Our framework is compatible with other diffusion-based methods [D, E], and we plan to extend experiments in future work.\n\nDeterministic models (e.g., HMR 2.0) can also adopt our preference framework by discretizing the pose space (e.g., via VQ-VAE in TokenHMR [C]) and replacing the regression network with a classification network for DPO training.\n\nMeanwhile, HMR-Scorer and data cleaning are model-agnostic and can enhance training quality across both probabilistic and deterministic pipelines—offering a practical path for pseudo-label refinement in in-the-wild settings where 3D ground truth is scarce.\n\n[D] Cho, Hanbyel, and Junmo Kim. \"Generative approach for probabilistic human mesh recovery using diffusion models.\" *ICCV*. 2023.\n\n[E] Stathopoulos, Anastasis, Ligong Han, and Dimitris Metaxas. \"Score-guided diffusion for 3d human recovery.\" *CVPR*. 2024.\n\n### **Q5. Duplicate citations**\n\nWe will fix this in the revision.\n\n### **Q6. HMR-Scorer training data preparation**\n\nYes, we add joint-wise Gaussian noise to the ground truth SMPL pose to simulate rotational errors, with magnitudes empirically determined. We will include the exact noise parameters in the final appendix.\n\n\n### **Q7. Training/inference cost**\n\nWe provide the training/inference time and memory cost below:\n\n- **Inference**\n| Model| Time per prediction| Batch Size| Memory|\n|-|-|-|-|\n| HMR-Scorer| 56 ms| 32| 3350 MiB|\n| ADHMR ($M=10$)   | 3 ms| 160| 8400 MiB|\n| ADHMR ($M=100$)  | 1.6 ms| 80| 24 GB|\n| ADHMR ($M=200$)  | 1.5 ms| 40| 24 GB|\nAll inference was conducted on a single NVIDIA V100 GPU. $M$ denotes the number of predictions.\n\n- **Training**\n| Model| Time| Batch Size | Memory per GPU|\n|-|-|-|-|\n| HMR-Scorer   | 24 h| 32| 11500 MiB|\n| ADHMR| 7 h| 40| 18 GB|\nAll training was conducted on 4 NVIDIA V100 GPUs.", "claims_and_evidence": "The paper's claims are supported via experimental evidence. For instance, they show the superiority of ADHMR over previous works, the effectiveness of HMR-scorer in assessing mesh quality, and the impact of data cleaning on 3 state-of-the-art models. The paper makes some minor claims, such as improving in-the-wild robustness, without proper analysis. However, it is acceptable, given the similarity of such claims with the literature.", "methods_and_evaluation_criteria": "The evaluation criteria (PVE, MPJPE, etc.) are the standard and widely used metrics for evaluating HMR methods. The evaluation (3DPW, Human3.6m) and training sets are standard research benchmarks. The paper has also included recent papers up to 2024, showing good awareness of the literature during its comparisons.", "theoretical_claims": "None provided. Only formulations of DDPO are used.", "experimental_designs_or_analyses": "All experiments, including HMR-scorer evaluation, ADHMR evaluation, ablations, data cleaning, and qualitative comparisons, are sound and well-justified.", "relation_to_broader_scientific_literature": "Although DPO has been extensively researched for NLP/LLMs and even image/video diffusion models, I did not find its adaptation to HMR tasks. However, it has been proposed for human motion generation [1], which is an entirely different task with similar input/output parameters and metrics. Reward modeling and scoring multiple hypotheses have also been extensively researched, but the paper introduces several implementation changes over prior works, showing improved performance. Several papers also directly address data cleaning, but the paper does not mention them since they are not the paper's focus.\n\nReferences:\n[1] Pappa, Massimiliano, et al. \"MoDiPO: text-to-motion alignment via AI-feedback-driven Direct Preference Optimization.\" arXiv preprint arXiv:2405.03803 (2024).", "essential_references_not_discussed": "The references are adequate.", "other_strengths_and_weaknesses": "### Strength\n1. While the individual components (diffusion model, DPO, reward modeling) are not entirely new, the paper successfully combines them to improve prior works.\n2. The paper is well-written. It has clear goals and designs adequate experiments to showcase its results.\n3. The paper is well aware of recent literature and is one of the first works adapting DDPO into HMR tasks.\n\n### Weaknesses\n1. The paper makes incremental contributions over the base model by adapting DDPO for HMR. Since the usefulness of DDPO was already well-known, it reduced the impact of the paper's findings.\n2. The choice of baselines is limited to only HypoNet. It would've been much more impactful if HMR-Scorer showed improved performance on other hypothesis-generation approaches.", "other_comments_or_suggestions": "The paper's contributions are incremental since it applies well-established methods to HMR without many changes. However, I found no significant issues in the paper, and the experiments were adequate. My final decision is borderline, leaning on acceptance due to good experiment design.\n\nTypo:\n- Line 87, Right Column: Duplicate citations for (Rafailov et al., 2024) (Rafailov et al., 2024)", "questions_for_authors": "1. How did you prepare the data for training the HMR-Scorer? Did you introduce rotational noise to the ground truth? Could you elaborate on the process?\n2. Could you analyze the biases caused by DPO on the diffusion model? Does it improve rare/uncommon poses like lying or rolling on the floor? Could you provide some qualitative comparisons of generalizability to unseen data?\n3. I understand that it is not common in this line of research to provide runtime information. However, could you give the model's general training/inference time and memory cost?\n4. Please also address my concerns about the weaknesses mentioned above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BVuCS3HUyy", "reviewer": "Reviewer_mE1y", "summary": "The authors propose a framework that integrates a diffusion-based human mesh recovery model with direct preference optimization. The core idea is to train HMR-Scorer, a model that evaluates the quality of human mesh predictions without requiring 3D annotations, and use it to create a preference dataset. This dataset consists of pairs of high- and low-quality predictions, which are then used to fine-tune the base model via preference optimization. Additionally, ADHMR leverages HMR-Scorer for data cleaning, which improves model accuracy. Experimental results show that ADHMR outperforms state-of-the-art probabilistic HMR methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for recognizing our SOTA performance, and the potential for pseudo-label generation. We truly appreciate your constructive comments and address them below.\n### **Q1. Clarification of claims**\nThank you for pointing this out. We would like to clarify that: end-to-end diffusion models predicting from noise typically avoid reprojection loss, as early denoising steps yield unrealistic poses, making such loss ineffective. CloseInt also notes: *“Early iterations are meaningless for human motion.”*\n\nIn contrast to ScoreHMR and CloseInt, which are two-stage (using external pose estimators before diffusion), our method is purely end-to-end, sampling from random noise. This avoids dependency on potentially biased external estimators and improves the robustness of in-the-wild (ITW) data. Visual comparison with ScoreHMR:\n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q1.pdf\n\nWe will revise the phrasing in the final version.\n\n**Consistency with the input image.**\nOur method directly learns 3D human pose distribution conditioned on images without needing an explicit camera model or reprojection loss in training. We follow ScoreHypo to predict UVD joints used for mesh reconstruction and scoring. At inference, we recover 3D joints from UVD using a full-perspective projection with estimated intrinsics.\n\nAs TokenHMR notes, minimizing 2D error can harm 3D accuracy due to imprecise camera parameters, even with full-perspective cameras. Instead, we ensure consistency implicitly via HMR-Scorer, which learns alignment using pixel-aligned image features. On clean data, scoring by 3D error naturally favors well-aligned results; on ITW data, the scorer transfers 2D alignment knowledge. Additional ITW results are provided at: \n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/8ugF_Q1.pdf\n\n### **Q2. Differences  with HypoNet**\nWe introduce a preference optimization framework to improve image consistency and ITW robustness. Rather than proposing a new diffusion model, our method focuses on finetuning existing diffusion-based HMR models like HypoNet. \n### **Q3. How to ensure accurate in-the-wild scoring**\nTo address limited and noisy outdoor data, we train HMR-Scorer on large-scale synthetic datasets (e.g., BEDLAM, GTA-Human) for effective knowledge transfer. As noted in BEDLAM, HMR models trained solely on synthetic data can achieve SOTA accuracy.\nTable 1 in the paper shows strong results on GTA-Human (with ITW scenes).  HMR-Scorer visualization is at:\n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q3.pdf\n### **Q4. About experiment comparison**\nWe updated Table 2 to indicate which methods are finetuned on the target benchmark:\n\n**Updated Table 2**: https://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q4.png\n\nWe added results showing:\n- Our ADHMR outperforms finetuned HMR 2.0a.\n- Prior works (e.g., HMDiff, Zolly) also finetune on 3DPW. To compare fairly, we introduce ADHMR (ITW)†, trained without Human3.6M, which still outperforms others.\nTable 3 and Table 4 in the paper show that finetuning on target or extra datasets offers only marginal gains, highlighting the effectiveness of our framework.\n### **Q5. More details**\nUVD in L176 denotes 2D coordinates (u, v) in the image and their corresponding depth (d) of human keypoints, which aligns 3D joint with image space.\n\nWe will include the formulas for PLCC and SRCC in the revision.\n\nThe datasets used for data cleaning are listed in L401–403 (right column).\n### **Q6. Comparison with ScoreHMR and CloseInt**\n- ScoreHMR is a two-stage method that refines pretrained pose estimates using a diffusion model, applying reprojection loss only during sampling while trained with standard diffusion loss.\n- CloseInt is also a two-stage method and focuses on multi-person interaction. It refines plausible initial poses with a proxemics and physics-guided diffusion model.\n- Our ADHMR is one-stage and learns to synthesize poses from random noise, guided solely by image cues and preference alignment, without handcrafted priors.\n\nThe **Updated Table 2 (see Q4)** includes results of ScoreHMR and CloseInt, where our ADHMR outperforms both.\n### **Q7. Method generalization analysis**\nWe provide more ITW results at:\n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/8ugF_Q1.pdf\n\nADHMR shows generalization under extreme poses (a–c), occlusions (a, b, d, e), and complex backgrounds (f).\n\nHowever, extreme camera angles remain challenging, likely due to limited training coverage, as shown here:\n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q7.pdf\n### **Q8. Qualitative examples and failure case of HMR-Scorer**\nAlthough HMR-Scorer is robust even for ITW scenes **(see Q3)**, it still fails in some occlusion cases:\n\nhttps://anonymous.4open.science/r/ICML_2025_Rebuttal-E36D/mE1y_Q8.pdf\n\nWith limited visual cues, the scorer may prefer predictions aligned with the image but not the 3D GT. We plan to explore temporal scorers to address this.", "claims_and_evidence": "Some claims in the paper are incorrect. For example, \"This is primarily because diffusion-based HMR models cannot use traditional 3D joint reprojection loss.\" However, many existing works [A,B] successfully use reprojection loss to guide the denoising process. Additionally, depending on the camera model, there is a trade-off between 3D joint accuracy and 2D alignment [C]. What camera model does this method use? The proposed approach does not consider reprojection error—how can it ensure that the final results are consistent with the input image?\n\n[A] \"Score-guided diffusion for 3d human recovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[B] \"Closely interactive human reconstruction with proxemics and physics-guided adaption.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[C] \"Tokenhmr: Advancing human mesh recovery with a tokenized pose representation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "methods_and_evaluation_criteria": "- This work builds on top of HypoNet. What are the differences between this method and HypoNet? What is UVD in Line 176?\n- This base model is trained on accurate indoor datasets and synthetic datasets. How can it ensure accurate scoring on in-the-wild images?", "theoretical_claims": "The improvement in training heavily depends on the performance of HMR-Scorer. However, this module is difficult to train on outdoor data, making its effectiveness in outdoor scenarios uncertain.", "experimental_designs_or_analyses": "- The experimental comparison is unfair. The proposed method uses different training data than the baselines. Moreover, it requires fine-tuning on the benchmark, which gives it an unfair advantage over other baselines. For example, HMR2.0 could also be fine-tuned on the benchmark using reprojection error.\n\n- Add more details, such as the specific formulas for PLCC and SRCC.\n\n- Which datasets were used for data cleaning?", "relation_to_broader_scientific_literature": "This work may be insightful for future psedo-label generation in human pose estimation.", "essential_references_not_discussed": "These works use reprojection error to guide the diffusion, which can achieve model-image alignment.\n\n[A] \"Score-guided diffusion for 3d human recovery.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[B] \"Closely interactive human reconstruction with proxemics and physics-guided adaption.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "- A comparison with diffusion-based methods such as ScoreHMR and CloseInt needs to be included.\n- There is no analysis of whether this method generalizes well across different poses, occlusions, or camera angles.\n- The effectiveness of HMR-Scorer is a critical component of ADHMR, yet the experiments evaluating its reliability are somewhat limited. While Table 1 reports correlation metrics (PLCC and SRCC) between HMR-Scorer’s scores and standard reconstruction errors, a more detailed analysis of failure cases is absent. For example, does HMR-Scorer struggle with extreme poses, occlusions, or rare human-object interactions? Qualitative examples of cases where HMR-Scorer fails would strengthen the validation.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper targets improving existing diffusion-based human mesh recovery (HMR) models that generates multiple predictions in a preference optimization manner. Specifically, it trains HMR-Scorer, which quantifying the human mesh prediction, and then uses HMR-Scorer for data cleaning to improve the performance of HMR. Experiments show that the proposed method outperforms state-of-the-art probabilistic HMR methods.

The main strengths of this paper includes:
- it is novel to incorporate direct preference optimization for diffusion-based probabilistic HMR
- it improves the performance significantly

The main weaknesses of this paper are:
- the fairness of training data of HMR-Scorer (Reviewer WdV4, mE1y)
- more baselines and comparisons (Reviewer 8ugF,mE1y)


During the rebuttal phase, all reviewers discussed, acknowledged and provided justification. Overall, this paper received 3 accept and 1 weak reject from the reviewers. Most concerns raised during review were addressed in the later rebuttal. However, one HMR-Scorer issue should be highlighted:
HMR-Scorer can provide accurate evaluations, but with "additional labeled data" like BEDLAM and GTA-Human (See Reviewer mE1y Rebuttal Q3).  This may need more discussion and experiments for clarification like the use of labeled data, a fairer comparison (See Official Comment by Reviewer mE1y).

After reading all the review, the discussion and the rebuttal, the AC agrees its novelty and its superior performance. But the HMR-Scorer labelled data issue is also important and should be carefully handled. The AC acknowledges the merits and tends to accept. The authors are suggested to provide a substantial improvement in final version to clarify the issues raised by reviewers, especially the HMR-Scorer issue (Reviewer WdV4,mE1y).
Decision: Accept (poster)