Unified 2D-3D Discrete Priors for Noise-Robust and Calibration-Free Multiview 3D Human Pose Estimation
Authors: Geng Chen
Pengfei Ren
Xufeng Jian
Haifeng Sun
Menghao Zhang
Qi Qi
Zirui Zhuang
Jing Wang
Jianxin Liao
Jingyu Wang
Venue: NeurIPS 2025 poster ()
Keywords: 3d human pose estimation; pose estimation; VQ-VAE
Primary area: applications
TL;DR: We introduce a calibration-free 3D pose estimator that uses a discrete prior learned via VQ-VAE and integrates it through proposed discrete-continuous attention for robust, accurate prediction.

Abstract: Multi-view 3D human pose estimation (HPE) leverages complementary information across views to improve accuracy and robustness. Traditional methods rely on camera calibration to establish geometric correspondences, which is sensitive to calibration accuracy and lacks flexibility in dynamic settings. Calibration-free approaches address these limitations by learning adaptive view interactions, typically leveraging expressive and flexible continuous representations. However, as the multiview interaction relationship is learned entirely from data without constraint, they are vulnerable to noisy input, which can propagate, amplify and accumulate errors across all views, severely corrupting the final estimated pose.
To mitigate this, we propose a novel framework that integrates a noise-resilient discrete prior into the continuous representation-based model. Specifically, we introduce the \textit{UniCodebook}, a unified, compact, robust, and discrete representation complementary to continuous features, allowing the model to benefit from robustness to noise while preserving regression capability.
Furthermore, we further propose an attribute-preserving and complementarity-enhancing Discrete-Continuous Spatial Attention (DCSA) mechanism to facilitate interaction between discrete priors and continuous pose features.
Extensive experiments on three representative datasets demonstrate that our approach outperforms both calibration-required and calibration-free methods, achieving state-of-the-art performance.

Unified 2D-3D Discrete Priors for Noise-Robust and Calibration-Free Multiview 3D Human Pose Estimation
Geng Chen'
Pengfei Ren' Xufeng Jian Haifeng Sun' Menghao Zhang
Qi Qi
Zirui Zhuang Jing Wang Jianxin Liao Jingyu Wang!
State Key Laboratory of Networking and Switching Technology, Beijing University of Posts and Telecommunications (chengeng, rpf, jianxf, hfsun, zhangmenghao, qiqi8266, zhuangzirui, wangjing, liaojx, wangjingyu] @bupt edu. cn
Abstract
Multi-view 3D human pose estimation (HPE) leverages complementary informa- tion across views to improve accuracy and robustness. Traditional methods rely on camera calibration to establish geometric correspondences, which is sensitive to calibration accuracy and lacks flexibility in dynamic settings. Calibration free ap- proaches address these limitations by learning adaptive view interactions, typically leveraging expressive and flexible continuous representations. However, as the multiview interaction relationship is learned entirely from data without constraint, they are vulnerable to noisy input, which can propagate, amplify and accumulate errors across all views, severely corrupting the final estimated pose. To mitigate this, we propose novel framework that integrates a noise-resilient discrete prior into the continuous representation-bise based model. Specifically, introduce the Uni- Codebook, a unified, compact, robust, and discrete representation complementary to continuous features, allowing the model to benefit from robustness noise while preserving regression capability. Furthermore, we propose an attribute- -prebbrr-rreeriinggg and campllementartt-eennnniii Discrete- Conettnuuu Spatial Attention (DCSA) mechanism to facilitate interaction between discrete priors and continuous pose features. Extensive experiments three representative datasets demonstrate that our approach outperforms both calibration-required and calibration-free methods, achieving state-of-the-an performance.
Introduction
3D human pose estimation aims to estimate the 3D locations of keypoints from images videos. It is important because 3D human skeletons are widely used in scenarios such as action recogni- tion [21, 25], human mesh recovery [8, 12], and robotics manipulation [7, 34], Although monocular pose estimation offers greater convenience in usage, its performance is limited by depth ambiguity
and occlusion issues, which hinder its broader application. Multiview systems [36, 13, 22, 31] offer promising solution by leveraging complementary information from different views, enhancing accu- racy and robustness in pose estimation. Existing multiview methods utilize the geometric constraints provided by camera extrinsics to establish correspondences between views, facilitating the fusion of complementary information enhance 3D pose prediction. There are several drawbacks using camera calibration-basee methods. First, due to the complexity and computational intensity of camera
Eqqual contribution.
Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

pose
transformer
encoder
continuous token
Continuous Transformer-Be Basee Lifting Method
regression head
pose
transformer w/ DCSA
encoder
continuous loken
UniCodebook
server as robust prior
discrete loken
Continuous Transformer- Based lfttngg Method
with Discrete Features Severing Robust Prior
Figure 1: (a) Continuous transformer-basee lifting method, which directly processes 2DD pose inputs to estimate 3D poses. (b) Proposed method, which integrates discrete features as robust prior within a continuous sunfformer-based framework, enhancing robustness to noisy 2D inputs and improving pose estimation accuracy.
calibration, these methods incur higher costs and exhibit limited flexibility. Second, their performance is sensitive to the precision of camera extrinsics, which can be unsatisfactory in complex or dynamic environments. Therefore. uncalibrated multi-view pose estimation has emerged as a significant trend, as it avoids the impact of unreliable camera extrinsics and is more flexible in deployment, leading to better generalization capability and broader applicability in real-world scenarios.
Many uncalibrated methods leverage adaptive multiview interaction patterns, relying on semantic intra-view and inter-view relationships to eliminate reliance on rigid geometric constraints. In recent years, purely transformer-bas based methods [32, 1 have attracted increasing attention in calibration- free pose estimation, they provide unified and scaleable framework for modeling spatial, view, and temporal relationships using attention mechanisms. Notably, FusionFormer [1] and PoselRM [2] have exhibited performance that surpasses even that of calibrated methods. This observation suggests that uncalibrated methods, rather than being constrained by predefined geometric relationships, can instead exploit data-driven adaptive semantic constraints. Such flexibility enables them more effectively capture context-award -awre dependencies across views, thereby learning richer and more robust feature representations. For example, calibrated methods depend heavily on geometric information from external calibration, which occlusions can easily disrupt. In contrast, without calibration, uncalibrated methods are forced to leverage multi-view interactions dynamically correct errors.
The superiority of transformer-based calibration- framework can be attributed to their expressive and flexible continuous representations, which enable fine-grained modeling of pose dependencies when equipped with attention mechanism. Their continuity aligns with the continuous nature of both input and output spaces in 2D-to-3D pose lifting, enabling smooth, consistent, and precise mappings that are free from quantization errors and easy to optimize. However, this flexibility comes at the cost of increased susceptibility overfitting when trained on limited data [5], making them sensitive to noisy 2D poses caused by occlusions and other common real-world challenges. Such errors can propagate, amplify and accumulate across views, affecting the accuracy of predictions from other views. Human pose is fundamentally constrained by inherent anatomical priors. These biological constraints, such as skeletal structure, joint motion limitations, and other biomechanical properties. define a natural boundary for valid poses. By encoding such anatomical knowledge as biomechanical priors in pose estimation models, we can intrinsically restrict the solution space to exclude implausible poses (e.g.. g. hyperextended elbows or inverted knees). Explicitly inject priors for guidance has been proven highly effective other domains, such as image generation [44] and 3D reconstruction [40, 41]. This prior-based regularization complements data-driven approaches by injecting domain-specie knowledge, significantly improving robustness against noisy inputs, and is particularly valuable in uncalibrated scenarios where semantic constraints entirely learned from the input data.
Despite the promise of integrating biomechanical baser- regularization, effectively incorporating them into pose estimation models remains a non-trivial challenge. First, a core difficulty lies in how to represent such priors in a way that is structurally robust in representation. Discrete representations offer a promising solution by clustering large amount of body configurations into a limited set of proto- typical poses, which encourages the learning high-quality dominant pose prototypes. This compres- sion enforces structural regularity by restricting the model to select from predefined pose vocabulary, thereby reducing overfitting to spurious or implausible patterns. Moreover, such many-to-one map- pings, i.e., multiple inputs can be mapped to the same prototype, naturally introduces tolerance to

---

noisy variations, as structurally similar but perturbed poses are treated equivalently. Although it is data- drive, such quantized space can serve as compact, interpretable, and noise-resilient beneficial in ambiguous or weakly constrained estimation scenarios. Second, integrating discrete priors into continuous regression models introduces a representational mismatch. While discrete pose spaces are optimized for compactness and structural regularity, pose regressors demand fine-grained flexibility. Directly constraining outputs to lie the discrete space may hinder expressiveness.
As illustrated in Figure 1, to address the challenges of constructing and leveraging human pose priors, we propose a unified, compact, and noise-resilient discrete prior-enhanced pipeline for 2D-to-3D human pose estimation. Central to our framework is the UniCodebook, which encodes both 2D and 3D poses as compositions of discrete sub-structure tokens within a shared discrete space to server as an informative prior that captures essential pose patterns and promotes resilience to noise. While discrete priors provide strong structural regularization and robustness to noise, their quantized nature inherently mismatches the continuous solution space required for accurate pose regression. Rather than directly regularizing the regressed output, we propose Attention (DCSA) module to enable continuous pose features to selectively attend to relevant discrete prototypes, which preserves the continuity of the original regression pipeline while softly integrating structural guidance from the discrete prior. By avoiding hard projection into the discrete space, DCSA mitigates representational mismatch and allows the model to inherit noise robustness from the discrete prior without sacrificing fine-gra ine-grained precision, thereby effectively reconciling the gap between discrete structure and continuous prediction. We demonstrate the effectiveness by achieving state-of of-tth-arttttt results on three benchmark datasets covering multiple complex scenarios.
Spatial
The main contributions of the article are as follows:
VQ-VAE as a Robust Prior for Pose Estimation: We propose the use of VQ-VAE to serve as a robust prior for pose estimation. By unifying 2D and 3D pose representations within shared codebook, VQ-VAE further enhances its robustness to noisy pose, ensuring more accurate and consistent predictions, especially in challenging scenarios such as occlusion 2D detection errors. Discrete-Continuoui Representation Interaction: We introduce an attribute-presee preservvig and parpll/mennaatyyyeeaaatteee mechanism for the interaction between discrete (codebook-basse and continuous ttransformer-bssed representations. This allows the discrete tokens to guide the continuous features, infusing noise-resilient properties into the backbone while preserving the regression capability of the continuous representation in continuous tasks (i.e., pose estimation). State-of-the-Arr Performance: The proposed approach is capable of calibration human pose estimation even under significant noise, achieving SOTA performance three large datasets.
2
Related Work
2.1 Multiview Calibration-Free 3D HPE
Methods with unknown camera extrinsics typically rely on semantic information rather than geometric information to enable better multiview feature interaction. FLEX [11] exploits the -nvvrranee characteristics of skeleton representations, such as bone lengths and rotation angles, to reconstruct human poses.
[32] and FusionFormer [1] leverage transformers to elegantly and efficiently model multiview, temporal, and spatial information, with the latter notably surpassing the performance of calibrated methods for the first time. PoselRM [2] treats the uncalibrated task as a domain generalization problem and employs the invariant risk minimization paradigm to enhance performance in unseen camera settings. --Net [3] leverages pre-defined alignment proxies, such as meshes and joints, to automatically discover inter-view interaction patterns.
2.2 VQ-VAE in Pose-Related Tasks
Recent advancements in pose-related tasks have leveraged VQ-VAE [37] to enhance representation learning and improve task performance. In PCT [10], they introduced two-step framework for 2D pose estimation, where VQ-VAE is first used to learn compact, discrete pose representation space, followed by a translator that maps image features to this space. This approach effectively prevents unrealistic poses and provides prior knowledge for occluded pose prediction. In TokenHMR [6]. they applied similar pipeline, focusing on learning pose parameters of the SMPL [23] body model using VQ- VAE. In MEGA [9], they utilized VQ-VAE to discretize 3D human meshes, achieving enhanced
3

---

(a) Stage I: Pretrain of the UniCodebook
(c) Spatial MHSA w/ DCSA
2D Encoder
2D Decoder
UnCCedbbook
Softmax
Softmax
2D Pose
2D Pose
heaass
heads
3D Encoder
3D Decodem
K
3D Pose
3D Pose
Linear
Linear
Linear
(b) Stage II: Human Pose L Lifting
Nor
daptive Pusion
Continuous
Discrete
UniCodobook
Tokens
Tokens
Encoder
Continuous- Continuous Discrete- to Continuous
Spatial MHSA w/ DCSA
Temporal MHSA
drs
Spatial Pooling
View MHHA
Temponi MHSS
FC
-FC
Spatial MHSA
Temporal MHSA
Noisy 2D
Pose Pzo w/ DCSA
LSTT
PaD
pose
Figure 2: Two stages of the proposed calibration-free multiview 3D human lifting pipeline (a, b) and the detailed structure of the Spatial Multi- Head Self-Attentic (MHSA) with Discrete- Continuous Spatial Attention (DCSA) (c). In Stage we construct the UniCodebook, a unified discrete representation space, through multi-strategy training scheme (2Dto2D, 2Dto3D, 3Dto2D, and 3Dto3D). this space, both 2D and poses are encoded as sets discrete tokens this shared space to bridge the representation gap between 2D and 3D data. In Stage II, transformer-based continuous model is employed for pose lifting, where codebook tokens generated from the UniCodebook are injected into the hybrid spatial attention block. Here, the proposed DCSA mechanism integrated with conventional MHSA facilitate effective fusion between the noise-resilient discrete priors and expressive continuous pose features, which enhances the robustness to noisy 2D input.
mesh recovery from single RGB images through both deterministic and stochastic generation modes, significantly improving uncertainty estimation and multi-output predictions. DVQVAE [48], they proposed decomposed VA-VAE approach for generating realistic human grasps, where hand components are separately encoded to enhance interaction with objects. DisPose [39] is discrete diffusion model for occluded 3D pose estimation, which combines pose quantization with discrete diffusion process to model 3D poses in latent space, leading to superior performance on occlusion handling and physical plausibility. These methods highlight the versatility of VQ- VAE, demonstrating its potential in improving robustness and generalization across wide range of pose related problems. However, these methods either rely on hard mappings between continuous features and discrete representations [9], constrain the entire prediction generation process to discrete latent spaces [48, 39, 10, 6), where quantization can obscure fine- line-grained details. In contrast, continuous representations excel capturing subtle spatial variations, which are eritical for precise and smooth pose reconstruction. To fully exploit the complementary strengths of discrete structure and continuous precision, method softly integrates discrete structural priors into the continuous regression stream, enabling noise-robust yet precise pose prediction.
3 Methodology
Figure 2 depicts the overall architecture of the proposed methods. In Section 3.1, we describe how to learn an UniCodebook which represents 2D and 3D data simultaneously. Section explains how the pretrained the UniCodebook is used in the human pose lifting task.

---

3.1 Stage I: Pretrain of the UniCodebook
In this section, we first introduce the key components used Stage I. We begin with the Pose Encoder, which processes the original pose into several tokens, followed by the proposed UniCodebook, which retrieves features from this unified compact feature space, Finally, we introduce the Pose Decoder and the loss function used to train the entire pipeline.
Existing codebook-based pose representation methods suffer from modality isolation, ie., they encode either 2D poses [10] or 3D poses [6] in separate models. This isolation creates two critical issues: (I) underutilization of heterogeneous pose data that inherently contains complementary 2D-3D correlations, and (2) failure to align the representation gap between 2D observation and 3D geometry. Since the lifting task involves both 2D and 3D features, this gap forces the backbone model to allocate additional capacity to transform unimodal features into more suitable representation for lifting, rather than focusing entirely on learning robust 2D-to-3D mappings.
To more effectively utilize 2D and 3D data and bridge the gap between them, we propose a unified cross-modal representation learning framework that establishes shared latent embeddings between 2D) and 3D human poses. Specifically, while maintaining dedicated modality-specific encoders and decoders for 2D/3D data processing, we shared codebook, i.e., the UniCodebook, that jointly represents both 2D and 3D pose. We optimize the whole framework through four complementary objectives: 2Dto2D self-reconstrucction, 3Dto3 self-reconstruction, cross-modal 2Dto3D projection, and inverse 3Dto2D back-projection. The first two auto-encoding tasks preserve modality-speciffi reconstruction fidelity and the latter two projection objectives enforce a smaller representation gap through bidirectional translation between modalities.
Pose Encoder. Given human pose P E: R' where = 2 for 2D pose P24 and D = 3 for 3D pose P3d. we adopt the compositional encoding strategy to capture human priors and structural relationships following [10]. Unlike joint-wise encoding that processes each joint independently, decompose the pose into N overlapping sub-structures f; (e.g.. limbs, torso combinations), where each sub-structure corresponds to a group of biomechanically correlated joints. It is noted that the compositional representation contains lots of redundancy due to the fact that different tokens may share overlapping joints. The redundancy, in turn, enhances the robustness of the representation against occlusions of individual parts [10].
We utilize two separate encoders, both identical in structure, to encode 2D and 3D poses, respectively. Following the approach in [10], we adopt the MLP-Mixe Mixer architecture [35] as our encoder fe. which is particularly adept capturing both local joint interactions and global pose semanties. Given an input pose P, we first embed it into Pemb E R J using a linear layer femb- Subsequently, the embedded features are passed through four MLP-Mixer blocks effectively model the relationships between different joints. Finally, we employ another linear layer frans to map the features to N tokens. The overall process described by F.n - [f, f2, [N] frans (Jem (P))) R where each f, E Peat, The number of tokens is set to N = 63, which is larger than the 51 input dimensions (17 joints x 3) of Human3.6M pose, introduce representational redundancy.
Quantization Process With the UniCodebook. We define a shared UniCodebook space Uxx that jointly represents both 2D and 3D poses. Specifically, each
C Ci....,, cy]" [c] codebook entry Ca encodes pose patterns applicable to both modalities. This shared representation of discrete tokens helps to minimize the disparity between heterogeneous pose representations.
For each sub-structure token f,, we perform quantization via nearest neighbor lookup in the codebook:
[ if u arg min, ||f Cyll2
qff |P) = (o otherwise
(1)
We abuse qf,)) to denote the index of the closest codebook entry for token f. The quantized pose representation is given by Fquant = [c,(())) Cg(Eg).....
Cx(fx)) E N. Dis,
Pose Decoder. The pose decoder is designed to recover pose P from quantized feature F quant It mirrors the architecture of the pose encoder but a reverse manner.
Loss. For each strategy, we optimize the entire framework using the following loss function:
M
Ipettrai = smoothe, (P,
3
[ - sg sg(c (
5

---

where sg denotes stopping gradient, and / is hyperparameter which we set to 10.
3.2 Stage II: Human Pose Lifting
Our framework consists of three main components Stage II: (1) 2D pose discretization and initial continuous feature embedding, (2) dual-stream tempaall feature interaction, and (3) cross-view and temporal feature refinement, and 3D pose regression.
Given 2D input skeleton sequence P2D E RVXTXJx Pa, we first feed it into the 2D Pose Encoder of the VQ-VAE trained in Stage I and quantize it into discrete codebook features F quant E I V X xx N Disa, Subsequently, within the transformer backbone, we initially project the 2D pose P2D into a higher- dimensional feature space F I xTxJxDes via fully-connected (FC) layer. We then employ a dual-stream transformer equipped with spatial and temporal Multi-Head Self-Attention (MHSA) mechanisms to derive the feature F pose E IR. xTxxxxxxBaa Das, Within the spatial MHSA, we implement Cocrrte-Conti Spatial Attention (DCSA) to facilitate interaction between continuous features and the discrete features F quant along the spatial dimension.
Next, we apply spatial pooling to obtain F pool AveragcPooling(F pose) E RVXTx This is followed by the application of View MHSA and Temporal MHSA to yield the final continuous pose features F E RVVxxDaa Finally, we regress the predicted 3D pose 3D E RVVxxxxXD through pose
an FC layer. We only apply LI loss between ground truth(GTT) 3D pose P3D and P3D- For simplicity, in the following descriptions, we omit layer notations unless cross-layer feature interaction is involved.
3.2.1 Hybrid Spatial Attention Block
The spatial block contains a Spatial Multihead Self-Attention (MHSA) and a CocctiteCo Spatial Attention (DCSA) module. For simplicity, we describe the computation for single attention head. The multi-head version follows standard practice by computing heads in parallel and projecting the concatenated results.
Spatial MHSA models relationships between joints within the same view and frame. Given a spatial feature Fs I /x from F posce. we first compute its query, key, and value vectors:
Qs Fs so
Ks = F Vs = Fs s
where 1 WK S. w Proxx Dr are learnable projection matrices. The self-attention output is then computed as:
Q
S-MHSA s) = softmax
Vsa
VDK
where DK is the dimension of the key vectors. The temporal MHSA follows the same computation but applies to temporal tokens.
Cearete-Contiuuous Spatial Attention (DCSA) augments the continuous joint interactions with semantic guidance from discrete codebook tokens. We implement this module using cross-attention mechanism. As detailed in our ablation study in Table 4b and Table 7 in the Supplementary Materials, this approach proved to be the most effective and straightforward compared to other alternatives. The mechanism derives queries from the continuous features Fs and keys/values from the discrete codebook tokens F quant R. x Dox:
Qn Fs W D
KD F quant WKK
VD D
F
quant WV
(5)
The discrete-to-coomiinuuu attention is then computed as:
QoKn VD
DCSA(Fs, F quamt) = softmax
D
VD.
(6)
The final spatial representation combines features from both modules through addition:
Foui
S-MHSA s)
DCSA(Fss F quaat))
Continuous Continuous
Discretott--Continnous
The output features undergo residual connections and layer normalization (LayerNorm), followed by an multilayer perceptron (MIP) block with another residual connection and LayerNorm. We denote the entire spatial block by S.

---

3.2.2 Dual-Stream Spatial-Temporal Transformer
Following MotionBert [49], we construct dual-stream spatial-temporal transformer by alternately stacking temporal and hybird spatial blocks in different order, thereby creating two parallel branches.
To enable the model to dynamically adjust the importance of different branches, we compute an adaptive weight for each branch. Adaptive fusion weights a'gr, 07s E RXxVxTxx are predicted by linear projection, as a'yr, d7s = softmax (T (FF )), (1 (FP))))))) where W is a learnable linear transformation, [:, denotes concatenation, i E {I,.... LJ is the current layer number, and L is the number of layers of the dual-stream module.
The features from the two branches are fused additively via predicted weights as Fj = a'sT (F (F)))) + atss (2 (F where o denotes element wise multiplication.
Experiments
4.1 Datasets and Metrics
Human3.6M [14] is a widely used 3D human pose dataset with 3.6 million frames captured by 4 synchronized cameras, covering 15 actions performed by 11 subjects. And the poses are represented by 17 human joints. Following common protocol, we subjects S1, S5, S6, S7, and S8 for training, and S9 and S11 for testing. We evaluate under two settings: using ground- boounddtruth (GT) 2D poses and CPN-detected 2D poses. The CPN detector introduces significant 2D pose noise: the average 2D MPJPE between CPN-detected and ground-truth keypoints is 3.9 pixels on the training set and 6.5 pixels the test set, with maximum errors reaching 55 and 124 pixels, respectively.
MPI-INF-3DHP [27] is a challenging multiview dataset for 3D human pose estimation, consisting of 1.3 million frames performed by 8 subjects in diverse indoor and outdoor scenarios, which is widely used to evaluate the generalization performance in complex real-world scenarios. Following previous studies [20, 16], we train subjects S1-S7 and test subject S8.
FreeMan [38] a large-scale multiview 3D human pose dataset collected under real-worke conditions. containing 11 million frames from 8,000 sequences captured by 8 synchronized smartphones. It covers 40 subjects across 10 diverse indoor and outdoor scenarios with varying lighting. To address the limited distribution gap between the training and testing sets in the original protocol, we re-split the dataset that introduces clearer separation in both scene and camera configurations: scenes 1-8 and cameras (1,3,577) are used for training, while scenes 10-12 and cameras (1,3,5,7) (2,4.6,8) are reserved for testing. The resulting training set consists of 566,484 image pairs, and the test set includes 646,608 pairs. This revised split enables more realistic and challenging evaluation of model generalization.
AMASS [26] is large-scale motion capture dataset with over 40 hours of motion data from 300+ subjects and 11 11,000+ motions. Since it does not provide direct 2D-3D annotations, we render SMPL meshes and apply official Human3.6M and COCO regressors to extract 3D joints, which are then projected onto 2D planes from 8 virtual views to form 2D-3D pairs. We use the official validation and test splits for training and evaluating the VQ-VAE in Stage I only.
Evaluation Metrics. We report Mean Per Joint Position Error (MPJPE), measured in millimeters Percentage of Correct Keypoints (PCK), and Area Under Curve (AUC) with a 150mm threshold.
4.2 Implementation Details
All individual experiments were completed within I day using one server equipped with an Nvidia RTX 3090 GPU, 64GB RAM, and 28-core CPU. We employed the AdamW [24] optimizer with learning rate of 2e-4 for both the Stage I and Stage II. We used cosine annealing learning scheduler with 100 warmup steps. The model was trained for 50 and 100 epochs with a batch size of 256 at Stage and Stage II, respectively. No data augmentation techniques were applied during training and test. For VQ-VAE, we use N
tokens to represent a pose, and the codebook in shape U x Dieat 2048 x 256. The original VQ-VAE suffers from codebook collapse, where significant portion of codebook entries remain underutilized. Following [6], we employ exponential moving average (EMA) and code reset tricks to enhance codebook activation rates.
7

---

Table I: Results Human3.6k 6M are reported using MPJPE as the evaluation metric. CPN, HRNet and 1.5 different 2D pose detectors. GT means using ground truth 2D pose. means this image-to-3d method. f indicates our reimplementation. T represents the number of frames.
Metivl
Veew
Tepat Sentig
Five
Phoce
Pran
Proe
Porc
san
Steclo
WIRD
Wil
WAST
Meeeciiar Metroki
MaSTE (413
CV1583222 2122
(SPN, 1mm 343)
37A
37.3
36.3
42.3
4011
35.5
51.7
45.1
72.8
413
27.9
273
-155
KTT
CVPR 2006
(CPS. T 343)
394
WLS
35.4
35.3
35.6
36.
45.1
45.5
34.7
31.3
50.7
22.1
25.0
35.0
Mad--Vo Molaeaa Wids Cancra
Parakis
Eypede Praion [21]
10EV-3009
11)
3611
212
26.3
26.8
24.1
32.1
36.9
25.0
24,4
36.3
Occentry Raseel Trunsionen [30]
PQ 2026
H8006 TM07)
26.0
Leard Iin Irangiaaiin 115
18M
360
180
16.5
19.4
1K-1
22.1
20.8
267
31.2
31.8
16.7
23.3
3162
313
1933
20.1
1122
MNP [13]
18.6
Mall-Vaew Methech Wiisutt Canara Parmustie
Profroegsbaee [1
P.7-1
31.8
25-1
36.6
21.1
$2.1
35.1
30.5
16.5
24.5
31.4
37.6
2018
36.1
32.01
17.6
1EBIX L11g
BOCV-SEE
(S3R) 27)
31.7
2-20
33.8
AAAE21044
23.7
34.
25.1
27.3
27.1
23.9
26.5
25.9
25.2
2574
23.5
22.8
26.6
Duss
ICHN 220
350
25.2
24.6
24.4
21.5
26.4
35.4
25.6
31.4
22.5
25.7
24.0
37.3
22.9
13.9
36.8
MTTE Trandveee [1]]
TRAMPN339
T-
15.5
117
15.5
16.0
15.8
15.8
16.9
15.3
153
S0uPooree 111)
AAAET004
18.7
154
16.
12.0
16.
13.0
15.1
187
19.8
11.9
11.0
11.0
12.0
11:1
13.0
11.7
Foreer 11]
AAAE 2004
(GT T 375
1.8
8.34
7.36
8.33
7.83
80
5.00
8.18
0.59
1,37
7.83
7.36
7.00
Our
3.37
7.30
7.5
7.80
7.76
7.87
7.89
7.81
7.66
7.20
7.02
7.80
7.88
1.24
7.34
Table Comparison results on the MPI-INF-3DHP. Table Comparison results on the FreeMan dataset. f
Method
For Finetune
Extra Data
MPIPE
PCK]
AUC
indicates our reimplementation.
seen camera MPJPE
unseen camera MPIPE
mean MPJPE
(20)
data
95
PafF [16]
SGraFormer [45]
16.9
98.7
90.2
SGraFormer [45]
22.20
28.69
25.44
MTF-Tannfommr [32]
h36m
14.6
MIF [32]
10.85
13.51
12.18
SGraFarmer [=S]
h36m
10.6
99.9
91.7
Fusionfoemer [1]
9,89
12.72
11.30
PesionFormer
h36m
Ours
9.59
12.71
11.15
4.3 Comparisons With SOTAS
Human 3.6M. Table compares our method with several state-of-the (SOTA) monocular and multi-view approaches the Human 3.6M dataset for 3D human pose estimation (HPE). Our method outperforms all the calibrated methods due to that our method are not limited by predefined geometric relationships between views but instead utilizes data-driven adaptive semantic constraints, allowing for more effective multi-view feature fusion. When compared to calibration-free 3D HPE methods using CPN [4] GT 2D poses, our approach achieves SOTA performance, with MPJPE values of 26.0mm and 7.74mm, respectively, demonstrating its effectiveness. Notably, our method does not rely on any additional tricks but simply adopts L1 Loss.
MPI-INF-3DHD Table 2 presents a detailed comparison with other SOTA calibration methods on the MPI-INF-3DHP dataset. We GT 2D poses as input and keep the same settings [20]. As shown in the table, our method consistently outperforms all other methods across various metrics. Specifically, achieve significant reduction in the MPJPE, outperforming the second-bes method by at least 2.03mm (37.5%). Additionally, our method shows a notable increase in the AUC, improving it by at least 4.24% These indicate that our model not only achieves higher accuracy but also maintains robust performance across different thresholds. It also demonstrates that the UniCodebook not only enhances robustness but also improves accuracy without noise.
FreeMan. Table 3 reports the evaluation results on the FreeMan dataset under seen and unseen camera settings using HRNet-w44 448 [33] 2D poses. Our method achieves the best overall performance with mean MPJPE of 11. 11.15mm, outperforming recent approaches such as Fusionformer [1] (1 (1.30mm) method attains the lowest error under unseen camera settings (9.5 9.59mm), demonstrating strong generalization to novel viewpoints. These results confirm the robustness and adaptability of our approach in real-world scenarios with cross-camera configurations.
and MTF [32] (12. (12.18mm). In particular,
4.4 Ablation Study
Impact of Codebook Training Strategy. Table 4a shows how different codebook training strategies in Stage I affect the lifting performance at Stage II. The best performance is achieved when all strate- gies are used. Without the codebook, performance drops significantly, highlighting its importance as robust prior for the continuous model. The MPJPE around 26.8, 26.7, 26.4 and 26.3 when
8

---

Table 4: Ablation studies key components of the UniCodebook, including training strategies, feature interaction methods, and data sources. MPJPE is calculated at Stage "CPN" and "GT" denote 2D poses from CPN [4] and ground truth, respectively.
(a)
Ablation study on different training strategy combinations for (b) Ablation experiments different inter- the UniCodebook where T=l.
action methods between discrete codebook features and continuous features. T=27.
Strategy
Number of Strategy
2dto2d
2dto3d
3dto2d
3dto3d
MPJPEL
Method
MPJPEL
27.01
Baseline
26.7
Conv + Addition
26.6
26.74
Q-Former [18]
34.1
26.86
Concat MHSA
26.4
V
26.70
V
26.78
DCSA
26.0
V
26. 74
26.73
(c) Ablation training data of VQ-VAB
T=27.
V
26.40
Data Source
MPJPEL
v
26.53
H36M CPN
26.43
26.45
H36M GT AMASS
26.35
26.34
26.02
Add No to
Adc Nobe
Add Notas
Nose Views
Locel noicon
basoite-spn
lesel con sesel codereoooooddoooo Sp*
besel no codsteor-zz
base Sunkhe--rebbt--com
busel gt tanel codetooveg!!
baschts-g: Iase-tre-tteebrrke
sesel na-gt
saxel niw codettinnn
60-
25
50
20
15
tawwline-cpp
Luselinc codebook-450
10
buceline-g
cudebooc-gtt
0
30
6 Interitty
30
hotte incersity
No mo
Noito Intenity
Figure 3: Comparison of MPJPE crrOr across four models (i.e., baseline trained on H36M CPN, baseline with codebook trained on H36M CPN, baseline trained on I136M GT, and baseline with codebook trained on H36M GT) under varying noise intensities without retraining. For each instance (consisting multi-view 2D poses of the same person the same timestamp). we randomly select 1 to views and add Gaussian noise with zero mean and a standard deviation "Noise Intensity" pixels to cach 2D joint. For models trained on H36M CPN, we evaluate them using H36M CPN test data with extra noise. Similarly, models trained on H36M GT are evaluated with H36M GT test data with extra noise. The results show that models with the codebook exhibit robustness across all noise levels, with greater robustness observed at higher noise intensities.
applying one, two, three and four strategies, respectively. This decreasing trend demonstrates that as more strategies are used, the UniCodebook can better capture the relationships between 2D and 3D poses and data utilization rate, making robuster prior for lifting tasks.
Impact of the Interaction Method of Discrete Codebook Features and Continuous Features. In Table 4b, we aim to investigate the impact of different interaction methods between discrete and continuous features on model performance. "Bascline" means no discrete features are used. "Conv + Addition" involves aligning the token count of the codebook features using convolutional layer, followed by adding them to the continuous features. "Q-Former [18]" employs learned queries with the same shape as continuous features and maps discrete feature information onto the queries using a transformer layer. The queries and continuous features then interact via cross-attention. "Concat + MHSA" first concatenates discrete and continuous features along the token dimension and then applies self-attention. All methods are applied in multiple spatial blocks. The results demonstrate that our DCSA method achieves the best performance, improving MPJPE by 0.7mm.
Impact of Noise. To evaluate the robustness our model to noisy inputs, we present in Figure the MPJPE performance of four models under varying noise intensities. The results demonstrate that

---

Table 5: Ablation the number of pose tokens and codebook size in UniCodebook where the frame number The active rate represents the mean activation rate Stage while MPJPE evaluated Stage on Human3.6k 6M [14].
pose tokens
codebook size
Active Rate(%))
MPJPE
21
61.8
26.82
63
2048
83.5
26.34
105
84.3
26.35
1024
84.5
26.70
63
2048
83.5
26.34
4096
75.0
26.47
our method-whett trained on H36M CPN or H36M GT consistently outperforms the baseline in the presence of noise. Moreover, the performance gap widens the noise intensity increases, indicating that the codebook based model exhibits stronger resilience to noisy 2D observations than the continuous wransfommr-bbaee baseline. This robustness stems from the discrete nature of the codehook, which provides a form of structural regularization by anchoring predictions to a set of stable pose prototypes. Such discrete priors may help correct noisy ambiguous joint inputs by guiding the model toward plausible configurations. These findings highlight the effectiveness of our approach in real-world scenarios, where noisy 2D detections are common, and underscore the practical advantages of incorporating discrete structural knowledge into continuous pose estimation models.
Impact of Codebook Training Data Quality. To evaluate the impact of data quality model performance. we conducted experiments in Table 4c using three data sources: H36M CPN, H36M GT, and AMASS. The results show consistent performance increase among them. As data quality improves, the model performance also increases, indicating that higher-quality data leads to better performance. It is worth noting that even when using only H36M CPN data, i.e., without any external data, our model still achieves near-SOTA performance on the Human3.6m 6M benchmark.
Impact of Codebook Configurations. We analyze two key hyperparameters: the codebook size, which defines the capacity of the discrete latent space, and the number of pose tokens, which reflects the granularity of the pose representation. (I) First, we fixed the codebook size to 2048 and varied the number of pose tokens. Increasing the tokens from 21 to 63 significantly boosted the active rate (from 61.8% 83.5%) and improved MPJPE (from 26.82 82 to 26.34). A further increase to 105 tokens yielded no improvement, suggesting that 63 tokens offer a favorable balance between expressiveness and efficiency. (2) Next, fixed the number of pose tokens at 63 and varied the codebook size. A codebook of 2048 achieved the best MPJPE (26.34). A smaller size (1024) suffered from higher error (MPJPE: 26.70 26.70) due limited capacity, while a larger
(4096) slightly degraded performance (MPJPE: 26.47). Overall, our selected configuration of 63 pose tokens and codebook size of 2048 achieves the best trade-off between token utilization and pose estimation accuracy.
Conclusion
In this paper, we propose a novel approach for uncalibrated multi-view 3D human pose estimation by integrating a noise-resilient prior, i.e., the UniCodebook, into a transformer-basee framework. By employing a multi-strategy training scheme, the UniCodebook effectively bridges the gap between 2D and 3D pose representations. Furthermore, we introduce Cearete-Continuous Spatial Attention (DCSA) facilitate interaction between discrete codebook tokens and continuous pose features, ensuring the benefits of noise-resilient priors without disrupting the continuity of the backbone model. Extensive experiments on three benchmark datasets demonstrate that our approach outperforms both calibration-require and calibration-free methods, achieving ofate-of-the-art performance. These results highlight the effectiveness of integrating discrete priors into multi-view pose estimation, demonstrating their potential to enhance the robustness of continuous models.
10

---

Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grants (62406039, 62321001, 62471055, U23B2001, 62171057, 62201072. 6207 1067, 62406039, 62101064), the High-Quality Development Project of the MIIT (2440STCZB2584), the Ministry of Education and China Mobile Joint Fund (MCM20200202. MCM20180101 the Project funded by China Postdoctoral Science Foundation (2023TQ0039, 2024M750257, GZC20230320), the Fundamental Research Funds for the Central Universities (2024PTB-00) -0P4,, and the 2025 Educa- tion and Teaching Reform Project Funding at Beijing University of Posts and Telccommunication
(2025Y7005).
References
[1] Y. Cai, W. Zhang, Y. Wu, and C. Jin. Fusionformer: A concise unified feature fusion transformer for 3d pose estimation. In AAAI Conference on Artificial Intelligence (AAAI), volume 38, pages 900-908, 2024.
[2] Y. Cai, W. Zhang, Y. Wu, and C. Jin. Poseirm: Enhance 3d human pose estimation on unseen camera settings via invariant risk minimization. In IEEE Conference on Computer Vision and
Pattern Recognition (CVPR), pages 2124-2133, 2024.
[3] G. Chen, X. Jian, Y. Chen, P. Ren, J. Wang, H. Sun, Q. Qi, J. Wang, and J. Liao. -een Calibration free multi-view 3d hand reconstruction for enhanced musical instrument learning. In International Joint Conferences on Artificial Intelligence (IJCAI), 2025.
[4] Y. Chen, Z. Wang. Y. Peng, Z. Zhang, G. Yu, and J. Sun. Cascaded pyramid network for multi-person pose estimation. In JEEE Conference on Computer Vision and Pattern Recognition
(CVPR), pages 7 7103-71 12, 2018.
[5] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[6] S. K. Dwivedi, Y. Sun, P. Patel, Y. Feng, and M. J. Black. Tokenhmr: Advancing human mesh recovery with a tokenized pose representation. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pages 1323-1333, 2024.
[7] K. Ehlers and K. Brama. A human-robot interaction interface for mobile and stationary robots based on real-time 3d human body and hand- hand-finger pose estimation. In International Conference on Emerging Technologies and Factory Automation (ETFA), pages 1-6. IEEE, 2016.
[8] H.-S. Fang, Y. Xu, W. Wang, X. Liu, and S.-C. Zhu. Learning pose grammar to encode human body configuration for 3d pose estimation. In AAAI Conference on Artificial Intelligence (AAAI), volume 32, 2018.
[9] G. Fiche, S. Leglaive, X. Alameda- Pineda, and F. Moreno-Noguer. Mega: Masked generative autoencoder for human mesh recovery. arXiv preprint arXiv:2405.188.. 2024.
[10] Z. Geng, C. Wang, Y. Wei, Z. Liu, H. Li, and H. Hu. Human pose compositional tokens. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 660-671, 2023. [11] B. Gordon, S. Raab, G. Azov, R. Giryes, and D. Cohen-Or. Flex: Extrinsic parameters-fre multi-view 3d human motion reconstruction. In European Conference Computer Vision
(ECCV), pages 176-196. Springer, 2022.
[12] R. A. Guler and I. Kokkinos. Holopose: Holistic 3d human reconstruction in wild.. In JEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 10884-10894, 2019.
[13] Y. He, R. Yan, K. Fragkiadaki, and S.-I. Yu. Epipolar transformers. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 7779-7788, 2020.
[14) C. Tonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEFE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI). 36(7): 2013.
[15] K. Iskakov, E. Burkov, V. Lempitsky, and Y. Malkov. Learnable triangulation of human pose. In IEEE International Conference Computer Vision (ICCV), pages 7718-7727, 2019. [16] K. Jia, H. Zhang, I. An, and Y. Liu, Delving deep into pixel alignment feature for accurate multi- view human mesh recovery. In AAAI Conference on Artificial Intelligence (AAAI), volume 37, pages 989-9999997 2023.
11

---

[17] B. Jiang, L. Hu, and S. Xia. Probabilistic triangulation for uncalibrated multi-view 3d human pose estimation. In IEEE International Conference on Computer Vision (ICCV), pages 14850- 14860, 2023.
[18] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ACM International Conference on Machine Learning (ICML), pages 19730-19742, PMLR, 2023.
[19] W. Li, H. Liu, H. Tang, P. Wang, and L. Van Gool. Mhformer: Multi-hypootesis transformer for 3d human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 13147-13156, 2022.
[20] J. Liang and M. C. Lin. Shape-aware human pose and shape reconstruction using multi-view images. In IEEE International Conference on Computer Vision (ICCV), pages 2019.
[21] J. Liu, A. Shahroudy, D. Xu, A. C. Kot, and G. Wang. Skeleton-based action recognition using spatio-temporal Istm network with trust gates. IEEE Transactions on Paltern Analysis and Machine Intelligence (T-PAMII, 40(1 (0:12)3007-3021, 2017.
[22] X. Liu, P. Ren, Y. Chen, C. Liu, J. Wang, H. Sun, Q. Qi. and J. Wang. Sa-fusion: multimodal fusion approach for web-based human- comattc interaction in the wild. In ACM International
World Wide Web Conference (WWW), pages 3883-3891, 2023.
[23] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. Smpl: a skinned multi-
person linear model. ACM Transactions on Graphics (TOG), 34(6):1-16, 2015.
[24] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 2017.
[25] D. C. Luvizon, D. Picard, and H. Tabia. Multi-task deep learning for real-time 3d human pose estimation and action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2366):2752- 2020.
[26] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture as surface shapes. In IEEE International Conference on Computer Vision (ICCV), pages 5442-5451, 2019.
[27] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, and C. Theobalt. Monocular 3d human pose estimation in the wild using improved enn supervision. In International Conference on 3D Vision (3DV), pages 506- IEEE, 2017.
[28] O. Moliner, S. Huang, and K. Aström. Geometry-biased transformer for robust multi-view 3d human pose reconstruction. In IEEE International Conference on Automatic Face and Gesture Recognition (FG), pages 1-8. IEEE, 2024.
[29] J. Peng. Y. Zhou, and P. Mok. Ktpformer: Kinematics and trajectory prior knowledge-enhance transformer for 3d human pose estimation. In IEEE Conference on Computer Vision and Pattern
Recognition (CVPR). pages 1123-1132, 2024.
[30] H. Qiu, C. Wang, J. Wang, N. Wang, and W. Zeng. Cross view fusion for 3d human pose estimation. In IEEE International Conference on Computer Vision (ICCV), pages 4342-4351, 2019.
[31] P. Ren, H. Sun, J. Hao, J. Wang, Q. Qi, and J. Liao. Mining multi-view information: a strong self- supervised framework for depth-based 3d hand pose and mesh estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages
2022.
[32] H. Shuai, L. Wu, and Q. Liu. Adaptive multi-view and temporal fusing transformer for 3d human pose estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 45(4):4122-4435, 2022.
[33] K. Sun, B. Xiao, D. Liu, and Wang. Deep high-resolution representation learning for human pose estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR),
pages 5693-5703, 2019.
[34] T. Tao, X. Yang. J. Xu, W. Wang, S. Zhang, M. Li, and G. Xu. Trajectory planning of upper limb rehabilitation robot based on human pose estimation. In International Conference on Ubiquitous Robots (UR), pages 333-338. IEEE, 2020.
[35] Tolstikhin, N. Houlsby, A. Kolesnikov, L. Beyer, X. Zhai, T. Unterthiner, J. Yung, A. Steiner, D. Keysers, J. Uszkoreit, et al. Mip-mixer: An all-mip architecture for vision. Advances in Neural Information Processing Systems (NeurIPS), 34:24261-2423
2021.
[36] H. Tu, C. Wang, and W. Zeng. Voxelpose: Towards multi-camera 3d human pose estimation in wild environment. In European Conference on Computer Vision (ECCV), pages 197-212.
Springer, 2020.
12

---

[37] A. Van Den Oord, O. Vinyals, et al. Neural discrete representation learning. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[38] Wang, F. Yang, B. Li. W. Gou, D. Yan, A. Zeng. Y. Gao. J. Wang, Y. Jing, and R. Zhang. Freeman: Towards benchmarking 3d human pose estimation under real-worid conditions. In
IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 21978-21988, 2024.
[39] W. Wang. J. Xiao, C. Wang, W. Liu, Z. Wang, and L. Chen. Poseriiccreee diffusion model for occluded 3d human pose estimation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
[40] Z. Yuan, C. Liu, F. Shen, Z. Li, J. Luo, Mao, and Z. Wang. MSP-MVS: Aulti-gramularity segmentation prior guided multi-view stereo. In AAAI Conference Artificial Intelligence (AAAI), volume 39, pages 9753-9762, 2025.
[41] Z. Yuan, J. Luo, Shen, Li, C. Liu, T. Mao, and Z. Wang. DVP-MVS: Synergize depth-edge and visibility prior for multi-view stereo. In AAAI Conference on Artificial Intelligence (AAAI), volume 39, pages 9743-9752, 2025.
[42] Zhang, Y. Cai, S. Yan, Feng, et al. Direct multi-view multi-person 3d pose estimation. Advances in Neural Information Processing Systems (NeurIPS), 34:13153-13164, 2021.
[43] J. Zhang, Z. Tu, J. Yang, Y. Chen, and J. Yuan. Mixste: Seq2seq mixed spatio-temporal encoder for 3d human pose estimation in video. In IEEE Conference on Computer Vision and Putlern Recognition (CVPR), pages 13232-13242, 2022.
[44] L. Zhang, A. Rao, and M. Agrawala, Adding conditional control to text-to-image diffusion models. In IEEE International Conference Computer Vision (ICCV), pages 3836-3847 2023.
[45] L. Zhang, K. Zhou, F. Lu, X.-D. Zhou, and Y. Shi. Deep semantic graph transformer for multi-view 3d human pose estimation. In AAAI Conference Artificial Intelligence (AAAI), volume 38, pages 7205-7214, 2024.
[46] W. Zhang, M. Liu, H. Liu, and W. Li. Svtformer: Spatial-view-temma transformer for multi-view 3d human pose estimation. In AAAI Conference on Artificial Intelligence (AAAI),
volume 39, pages
2025.
[47] Z. Zhang, C. Wang, W. Qiu, W. Qin, and W. Zeng. Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild. International Journal of Computer Vision (IJCV), 129:703-718, 2021.
[48] Z. Zhao, M. Qi, and H. Ma. Decomposed vector-quantized variational autoencoder for human grasp generation. In European Conference on Computer Vision (ECCV), pages 447-463. Springer, 2025.
[49] W. Zhu, X. Ma. Z. Liu, L. Liu, W. Wu, and Y. Wang. Motionbert: A unified perspective learning human motion representations. In IEEE International Conference on Computer Vision (ICCV), pages 15085-15099, 2023.
6 NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly state three contributions: (1) a VQ- VAE-based UniCodebook unifying 2D/3D pose representations to improve robustness, (2) a DCSA mechanism enabling
interaction, and (3) SOTA results on three representative datasets. These claims are directly validated in the methodology and experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
13

---

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the supplemental material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers penalize honesty concerning limitations.
will be specifically instructed
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
14

---

Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The methodological workflow is thoroughly detailed in Section 3, while Section 4.1 specifies the dataset's train-test splitting way and preprocessing procedures.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The paper uses publicly available datasets and provides detailed, reproducible processing methods in Section 4. The code will be released upon acceptance of the paper, and comprehensive model hyperparameters and training details are provided in Section 4.2.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https: //pp:://npssss/ cc/
publi i gu des SCodeSubm as Possoooooliee (cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new soun-s benchmark).
15

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: c/ des /CodeSobdbr ss onPol I ccy) for more details.
/ ps.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper provides comprehensive details on data processing and splits in Section 4. along with hyperparameters, optimizer selection, and model configurations in Section 4.2, ensuring full reproducibility and clear understanding of the experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We do not report error bars or other appropriate information about the statistical
significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma emror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
16

---

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report the compute resources in Section 4.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't tt make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /al:://neeuu /twuriss.ppppp/e
Answer:
[Yes]
Justification:
The research fully complies with NeurlPS ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss potential societal impacts in the supplemental material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
17

---

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks. mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification:
We propose a human pose estimation method and it poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All datasets and models used this work are properly licensed and accompa- nied by accurate citations to their original publications.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assett are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers senersswiie thoode com/ dataset has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
18

---

Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use the LLM to improve writing.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //n/uriiisspp//eee/ee ca/ Cwt.//eeneeeccsccccccccccc 2025/
LLM) for what should should not be described.
19


Reviews:
[{"review_id": "INpLA8zXrt", "reviewer": "Reviewer_imRL", "summary": "This paper propose a model for uncalibrated multi-view human pose estimation. The model has two part, a VQ-VAE and a regressional transformer that leverages the 2D poses and pre-trained priors in VQ-VAE to compute 3D human poses, for which the authors claimed for some architectural novelties. The evaluation is done on several 3D human pose estimation benchmarks, on which the proposed method outperforms some previous works.", "questions": "N/A", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n* The idea of leveraging pre-trained priors in VQ-VAE is technically sound\n* The experiment setting is good and comprehensive enough. Ablation studies are presented to test the influence of each proposed modules. \n\n**Weakness**\n* My biggest concern for this work is that it seems all its proposed modules only have marginal influence on the model performance. From the results in Table 4 (a), (b), (c), I do not think there any significant improvement resulted from the \"new ideas\", e.g., the MPJPE improvements are all less then 3% compared to the baseline. For me, they could just be random noises. \n* When comparing with state-of-the-art, e.g., Table 1, the author should compare against latest methods instead of out-of-date ones. For example, for Human3.6M, I recommend the authors to compare methods listed in this list: https://paperswithcode.com/sota/3d-human-pose-estimation-on-human36m. \n* Some technical details are not clear. For example, in L177, how are the $N$ embeddings computed from the $J$ joints? Why choose $N$ to be 63? It looks like a magic number to me. \n* For the VQ-VAE, why not adding temporal modelling? My gut tells me maybe adding temporal modelling will give some performance boost. \n* For L209-L230, the authors spend a lot of spaces to introduce the basic attention formulation, for which I do not think it's necessary. TBH, although it does make the paper look more \"mathematical\" at the first glance, it does not bring any real values to the readers given the attention and transformers are widely used today. Please do not expect reviewers are fool enough to give accept just because there are some \"nice looking\" formulations for well-known basic stuffs :))", "was_revised": true, "final_justification": "I have read the author rebuttal and other reviewer's comments. I think the proposed method's good performance under heavy occlusion is beneficial to the community, for which I ignored in my first read, thus I am happy to raise my rating.", "rebuttal": "> **Q1: My biggest concern for this work is that it seems all its proposed modules only have marginal influence on the model performance. From the results in Table 4 (a), (b), (c), I do not think there any significant improvement resulted from the \"new ideas\", e.g., the MPJPE improvements are all less then 3% compared to the baseline. For me, they could just be random noises.**\n\n**A1:** **(1) Saturation of Human3.6M and Stronger Results on 3DHP**\n\nWe would like to clarify that Human3.6M is already saturated in terms of performance, and Human3.6m contains some inaccurate annotations[4], making further improvement increasingly difficult. \nUnlike Human3.6M, which is a lab dataset, 3DHP includes both indoor and outdoor scenes, offering higher diversity and better reflecting real-world scenarios. \nOur method shows stronger performance on 3DHP without extra training data, with a **2.03 mm MPJPE reduction and a 4.24 AUC increase** over the second-best method, FusionFormer and SGraFormer.\n\n**(2) Addressing the Overlooked Real-World Noise: A Novel Use of Data-Driven Discrete Priors for Robustness**\n\nOur core motivation is as follows: continuous representations are well-suited for regression tasks but are vulnerable to noise, while discrete representations tend to be noise-resilient but suffer in precision. Our method aims to leverage both to improve robustness without sacrificing accuracy. As shown in **Figure 3 of the supplementary material**, using DCSA and the discrete codebook improves noise robustness significantly, achieving up to **6% (5 mm MPJPE) improvement under large noise**, which is far from marginal.\n\n**(3) Steady Performance Gains Through Progressive Discrete Prior Quality**\n\nTable 4 (a)(c) shows **consistent and stage-wise improvements** when increasing the quality of the VQ-VAE prior (see L304–307, L322–324). These improvements are achieved without adding any model parameters, solely by enhancing the quality of the UniCodebook, which directly correlates the **quality of discrete prior with model robustness** and validating the effectiveness of our method, especially under noisy inputs.\n\n\n> **Q2: When comparing with state-of-the-art, e.g., Table 1, the author should compare against latest methods instead of out-of-date ones. For example, for Human3.6M, I recommend the authors to compare methods listed in paperwithcode.**\n\n**A2:** (1) Thank you for the suggestion. **We have compared our method against all latest open-source available models under the uncalibrated setting** (e.g., SVTFormer (AAAI 2025), SGraFormer (AAAI 2024), FusionFormer(AAAI 2024)). \n\n(2) We followed your suggestion and compared our method with the top-3 approaches listed on Papers With Code. Methods ranked lower show performance significantly inferior to ours.\n\n(2.1) In that leaderboad, the top-ranked method LMT R152[1] operates under a different experimental setup by generating 3D poses directly from images and leveraging SMPL prior, which is **not comparable to our setting** that uses noisy 2D pose inputs. As noted in the main paper (Lines 247–249), the 2D poses predicted by CPN contain significant noise, making it challenging to infer correct 3D poses from erroneous 2D joints. In contrast, image-based methods inherently have access to more accurate 2D information.\n\n(2.2) The second-ranked method, Geometry-Biased Transformer[2], relies on calibrated setups and uses HRNet for 2D pose sequences. Since HRNet’s 2D pose estimation is significantly more accurate than CPN’s, it benefits from better inputs. Despite using noisier inputs and lacking camera calibration, our method achieves comparable results (MPJPE: 26.0 mm VS 26.0 mm), highlighting the robustness and strength of our approach. This also suggests the dataset performance is approaching saturation.\n\n(2.3) Moreover, our method substantially outperforms the third-ranked Epipolar Transformer[3] (26.0 mm vs. 26.9 mm). We will include these methods in the final version and clarify the differences in experiment settings between our method and theirs.\n\n| model                           | calibrate | input                                         | MPJPE |\n| ------------------------------- | :-------: | --------------------------------------------- | :---: |\n| LMT R152 [1]                    |     √     | image -> 3D pose                              |   \\   |\n| Geometry-Biased Transformer [2] |     √     | HRNet 2D pose sequence -> 3D pose             | 26.0  |\n| Epipolar Transformer [3]        |     √     | image -> 3D pose                              | 26.9  |\n| ours                            |     ×     | CPN 2D pose sequence -> 3D pose               | 26.0  |\n\n\n> **Q3: Some technical details are not clear. For example, in L177, how are the $N$ embeddings computed from the $J$ joints? Why choose $N$ to be 63? It looks like a magic number to me.**\n\n**A3:** We apologize for not explaining the origin of this number. Human3.6M contains 17 joints, each with 3D coordinates, totaling 51 dimensions. To increase representation redundancy, we use 63 tokens, roughly corresponding to 21 joints × 3 coordinates. 63 is not a magic number as similar token counts (e.g., 60 or 65) yield similar performance.\n\n| token num | MPJPE |\n| :-------: | ----- |\n| 60        | 26.09 |\n| 63        | 26.04 |\n| 65        | 26.07 |\n\n\n> **Q4: For the VQ-VAE, why not adding temporal modelling? My gut tells me maybe adding temporal modelling will give some performance boost.** \n\n**A4:** (1) Thank you for the insightful suggestion. As have discussed in **Section F in Supplementary Material**, we fully agree that incorporating temporal modeling can potentially enhance performance. However, we intentionally avoid relying on pose sequences in order to maximize the use of single-pose data, which is significantly more abundant than sequential data.\n\n(2) We also conducted experiments incorporating a temporal prior during codebook training by introducing a loss term that encourages the predicted joint velocity across time to match that of the ground truth if two joints are from one sequence. This indeed led to improvements in VQ-VAE reconstruction across all four codebook strategies. When applied to the full backbone, the final MPJPE remained almost unchanged (26.09 mm).\n\n| model                 | 2Dto2D | 2Dto3D | 3Dto2D | 3Dto3D |\n| --------------------- | ------ | ------ | ------ | ------ |\n| ours                  | 0.006  | 28.81  | 0.005  | 7.84   |\n| ours + temporal VQVAE | 0.006  | 28.31  | 0.005  | 7.57   |\n\n| model                 | Stage II MPJPE |\n| --------------------- | -------------- |\n| ours                  | 26.02          |\n| ours + temporal VQVAE | 26.09          |\n\n\n> **Q5: For L209-L230, the authors spend a lot of spaces to introduce the basic attention formulation, for which I do not think it's necessary. TBH, although it does make the paper look more \"mathematical\" at the first glance, it does not bring any real values to the readers given the attention and transformers are widely used today. Please do not expect reviewers are fool enough to give accept just because there are some \"nice looking\" formulations for well-known basic stuffs**\n\n**A5:** Thanks! We will simplify those redundant explanation of the basic attention mechanism and focus directly on the core ideas and contributions of our method.\n\n[1] Learnable Human Mesh Triangulation for 3D Human Pose and Shape Estimation. WACV 2023\n\n[2] Geometry-Biased Transformer for Robust Multi-View 3D Human Pose Reconstruction. FG 2024\n\n[3] Epipolar Transformers. CVPR 2020\n\n[4] Learnable Triangulation of Human Pose. ICCV 2019"}, {"review_id": "1AzSvhMiz3", "reviewer": "Reviewer_iezF", "summary": "This paper introduces a calibration-free multiview 3D human pose estimation method. It combines discrete and continuous representations using a unified VQ-VAE codebook (UniCodebook) and Discrete-Continuous Spatial Attention (DCSA). It uses discrete pose priors to improve the method's robustness against noisy 2D inputs while keeping the accuracy of continuous regression. Experimental results are shown on the Human3.6M dataset, as well as the MPI-INF-3DHP and FreeMan datasets.", "questions": "Is the codebook generalizable to unseen poses?", "limitations": "Assumptions of a multiview setting limit the application of this method. How to apply it on real scenarios, such as moving cameras or video streams?", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nS1: It is novel to unify 2D/3D pose quantization in a shared codebook, mitigating the modality gap. DCSA elegantly resolves the discrete-continuous mismatch via soft attention. The rigorous two-stage training (codebook pretraining + transformer lifting) is reasonable.\n\nS2: The ablations validate design choices (Table 4) are very helpful.\n\nS3: The result shows the proposed method outperforms both recent calibrated (e.g., CrossView Fusion) and calibration-free (e.g., FusionFormer) methods.\n\nWeaknesses:\n\nW1: Missing baselines, especially for the monocular setting, only baselines are listed; it should also evaluate this method's monocular performance on different datasets.\n\nW2: The fixed-size UniCodebook may have difficulty with highly diverse or unusual poses that were not included in the training. This creates a representational bottleneck that might limit generalization.\n\nW3: The paper does not explain how the codebook would adjust to new pose domains. Needing complete retraining could be impractical in real-world situations.", "was_revised": true, "final_justification": "The rebuttal addressed some of my concerns, while still cannot conving me in the following points:\nComparison With Monocular Baselines Using Mixste Backbone: CPN 2D pose is already noise detection result comparing to GT 2D pose, I don't think it is necessary to add extra noise to CPN 2D. While with CPN 2D, the proposed method cannot outperform any baseline with the proposed robust prior.\n\nAdditional Monocular Evaluations With 27-Frame Input: The authors mention the proposed method \"achieves the best monocular performance among all models originally designed for multi-view pose estimation, highlighting its strong generalization and potential for use even in monocular scenarios\", while from its provided results, it cannot outperformance any monocular method under monocular setting. I still believe that the target multi-view setting limits the application of pose estimation methods, and how to apply them to real-world scenarios is still worth more in-depth discussion.\n\nIn the discussion period, the author try to convince reviewers by some non-standard settings such as \"train-on-GT, test-on-CPN setting\", I encourge the authors use the standard comparison in the futher submission version.", "rebuttal": "> **Q1: Missing baselines, especially for the monocular setting, only baselines are listed; it should also evaluate this method's monocular performance on different datasets.**\n\n**A1:** \n(1) Regarding monocular performance, we are currently training our temporal model on sequences of 243 frames, which requires approximately 3 days to converge. Once training is complete, we will report monocular results and analyze the effect of the discrete codebook on noise robustness in the temporal monocular setting within the rebuttal period.\n\n(2) Our method is primarily designed for the multi-view and temporal setting (v=4, f=27), which is a meaningful and practical setup. Multi-view configurations are as important as monocular ones, as they better reflect real-world applications such as motion capture studios and multi-camera systems, where complementary viewpoints enhance robustness and accuracy.\n\n(3) Notably, recent state-of-the-art multi-view, uncalibrated methods like SGraFormer and SVTFormer also do not report monocular performance, as their architectures and training are specialized for multi-view input.\n\n\n> **Q2:The fixed-size UniCodebook may have difficulty with highly diverse or unusual poses that were not included in the training. This creates a representational bottleneck that might limit generalization. Is the codebook generalizable to unseen poses?**\n\n**A2:**\n(1) We agree that a fixed-size UniCodebook may not fully cover all highly diverse or unusual poses. **However, continuous regression models face the same limitation.** \n\n(2) **Noise in real-world scenarios can also hinder generalization, so improving noise robustness directly contributes to better generalization.** As shown in **Figure 3 of the supplementary material**, when noise intensity increases, the poses become more unseen for the UniCodebook (as traing pose is high quality from AMASS). In this case, our model are more robust than the continuous baseline.\n\n(3) **Better data utilization also improves real-world generalization.** The UniCodebook is trained on high-quality poses from AMASS and supports both paired and unpaired inputs, allowing broader data utilization than Stage II .\nThis means that Stage I, based on the UniCodebook, is less likely to become a bottleneck—especially considering that the continuous model in Stage II is both more sensitive to noise and trained on less data.\n\n(4) To address generalization concerns, we propose DCSA to effectively combine the complementary strengths of discrete and continuous representations. \nThis hybrid design ensures that our model does not suffer from more domain generalization issues than fully continuous alternatives.\n\n\n> **Q3: The paper does not explain how the codebook would adjust to new pose domains. Needing complete retraining could be impractical in real-world situations.** \n\n**A3:** \n(1) As shown in Section F of the supplementary material, our model encounters challenges primarily when transferring across different **pose definitions**, rather than across different **pose distribution domains**.\n\nIn our work, a pose is defined by a set of semantically consistent keypoints representing major human joints, their 3D coordinates, and the skeletal connectivity between them. For example, in the Human3.6M dataset, each pose comprises 17 keypoints—including the head, neck, shoulders, elbows, wrists, hips, knees, and ankles—structured as a kinematic chain.\n\nIt is important to distinguish this from domain shifts in pose distributions, which are naturally involved in our main experiments—for example, transferring from AMASS to Human3.6M and other datasets.\n**We highlight this distinction to clarify a potential misunderstanding: the reviewer’s concern appears to conflate variation in pose definitions (i.e., joint types and connections) with shifts in pose distribution (i.e., statistical variation within a fixed definition).**\n\n(2) As shown in **Section F**, we propose a potential solution by training a format-agnostic discrete prior. However, we currently do not have a concrete approach for fine-grained adaptation to new pose definitions, as this is not the primary focus of our work. **This limitation is common to all models that represent poses as 1D sequences, and it is not specific to the use of VQ-VAE.**\n\n(3) Moreover, **many joint formats can be easily converted**, especially when migrating from models with more joints to those with fewer joints. Thus, retraining is often unnecessary unless the joint sets are truly incompatible. In real applications, **models typically rely on mainstream 2D detectors, which use fixed and widely adopted pose definitions.** For example, OpenPose, Detectron2 and MMPose all support COCO format keypoint.\n\n\n\n\n> **Q4: No qualitative evaluation and comparison with baselines are shown in the main paper, which makes it very difficult to show the improvement convincingly.**\n\n**A4:** We include qualitative evaluations and comparisons with baselines in **Figure 7 and the accompanying videos in the Supplementary Materials**. In the final version, we will move part of these qualitative results into the main paper and provide more detailed analysis to better highlight the improvements achieved by our method.\n\n\n> **Q5: Assumptions of a multiview setting limit the application of this method. How to apply it on real scenarios, such as moving cameras or video streams?**\n\n**A5:**\n(1) As noted in our response to Q1, our model can be directly applied to monocular settings by setting views=1, while still maintaining robustness to noisy 2D pose inputs (see **Figure 3 in Supplementary Materials**).\nAlthough our main experiments use a multiview setup, we share the same concern as the reviewer regarding real-world deployment, where noisy 2D poses are common, especially in moving camera scenarios.\n\n(2) Uncalibrated multiview scenarios are susceptible to noisy 2D poses and thus present a realistic and challenging setting. Without camera extrinsics to provide geometric cues, such methods rely more heavily on semantic information for multi-view fusion. This makes continuous-feature-based models more vulnerable to input noise. Our study on robustness under uncalibrated multiview conditions is therefore also helpful to handle temporal jitter caused by moving cameras, offering practical insights for deployment in dynamic real-world environments."}, {"review_id": "Xy4LYI4FKB", "reviewer": "Reviewer_Tfp8", "summary": "- This paper proposes a novel method for 3D human keypoint estimation from (potntially noisy) multi-view 2D keypoint sequences.\n- They introduce *Uni-Codebook*, a noise-resilient discrete codebook that is trained to recover both 2D and 3D keypoints.\n- They then use trained Uni-Codebook as a discrete prior for keypoint estimation, complementing and enhancing conventional estimation with spatio-temporal transformer. They term this Discrete-Continuous Spatial Attention (DCSA).\n- They did experiments to demonstrate that by leveraging Uni-Codebook (trained on the AMASS dataset) as a prior, the proposed method achieves state-of-the-art results on the Human3.6M dataset.", "questions": "- You claim that your model is not affected by \"predefined geometric relationships\" (L283), yet you include view-dependent positional encoding $\\mathbf{P}_{\\mathrm {pos}}^V$ (L199). This appears to entangle positional relationship into the estimation process.\n\t- I would like to see if your method is truly independent from camera geometric relationships, for example, by swapping or reducing input views only during evaluation.\n- You simply add the output of spatial MHSA and DCSA (Fig. 2, Eq. 9). Why wasn't a weighted sum used, as done in L236 for two-path integration?\n\t- The model benefits from adaptively considering continuous and discrete features depeneding on noise levels. Employing a weighted sum to allow the model to prioritize specific feature could lead to improved estimation.\n\t- Currently, since $\\mathrm{head}_i$ are the sum of $V_i$ with the weights derived from $\\mathrm{softmax} (QK)$ (which sums to one) for both of spatial MHSA and DCSA, this may hinder effective feature focusing.", "limitations": "yes, in the supplementary materials", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strength:\n- Uni-codebook offers a clear and concise solution for modality isolation.\n\t- It effectively combines the tokenization approach for pose estimation, previously used in TokenHMR [5] and PCT [9], and extends it to integrate 2D and 3D keypoint representations.\n- They also propose DCSA, an intuitive and effective method to enhance continuous pose estimation with discrete representation.\n- They achieve state-of-the-art results on the Human3.6M dataset. Comparisons and ablation studies are extensive and effectively supports their claims.\n\nWeaknesses: integrated with the questions section.", "was_revised": true, "final_justification": "My questions regarding some uncertainties about their methodology are fully answered by the authors. A minor issue about the necessity of positional encodings in their method is not yet fully experimented in the temporal model (T=27) setting due to the limited timeframe of the rebuttal period. However, as (1) they did an experiment in T=1, and the trend of the results should be the same for the temporal model, and (2) they assured they will include the result and discussion for the temporal experiment in the final paper, I am happy to keep my rating and accept this paper to NeurIPS 2025.", "rebuttal": "> **Q1: You claim that your model is not affected by \"predefined geometric relationships\" (L283), yet you include view-dependent positional encoding  (L199). This appears to entangle positional relationship into the estimation process. I would like to see if your method is truly independent from camera geometric relationships, for example, by swapping or reducing input views only during evaluation.**\n\n**A1:**\n(1) We acknowledge that our model does exhibit some sensitivity to view biases present in the data. However, this issue is not unique to our method and can also be observed in other SOTA approaches such as FusionFormer. Notably, our method demonstrates stronger generalization—even when trained using only 4-view data or a mixture of 3- and 4-view configurations—outperforming FusionFormer under the same conditions.\n\nWe attribute this robustness to the use of discrete codebook features. Continuous representations are typically learned from multi-view data, where the same features must simultaneously encode both pose semantics and inter-view interactions. This entanglement can make them more sensitive to changing of view numbers. In contrast, our discrete features are learned from single-view data, and their integration into the model provides additional information that is less entangled with camera-specific biases.\n\n\n| model        | train views | test views | MPJPE    |\n| ------------ | :---------: | :--------: | -------- |\n| fusionformer |      4      |     4      | 27.0     |\n| fusionformer |      4      |     3      | 66.2     |\n| ours         |      4      |     4      | **26.3** |\n| ours         |      4      |     3      | **50.3** |\n| fusionformer |    4, 3     |     4      | 29.2     |\n| fusionformer |    4, 3     |     3      | 32.0     |\n| ours         |    4, 3     |     4      | **27.8** |\n| ours         |    4, 3     |     3      | **30.2** |\n\n\n(2) As attention mechanism is theoretically permutation-invariant, we suspect that this view sensitivity primarily stems from the use of positional encodings (PE). In ablation experiments where we removed PE from the model, we observed consistent performance improvements when shuffled views only when testing, suggesting that PE may inadvertently introduce view-dependent artifacts. But it still can not generalize to less views which is not observed in training. We believe that using reduce views when training would increate the performance.\n\n| model           | trian shuffle | train views | test shuffle | test views |  MPJPE   |\n| --------------- | :-----------: | :---------: | :----------: | :--------: | :------: |\n| ours            |       ×       |      4      |      ×       |     4      |   26.3   |\n| ours            |       ×       |      4      |      √       |     4      |   50.3   |\n| ours without PE |       ×       |      4      |      ×       |     4      |   26.3   |\n| ours without PE |       ×       |      4      |      √       |     4      | **26.3** |\n| ours without PE |       ×       |      4      |      ×       |     3      |   50.2   |\n\n\n> **Q2: You simply add the output of spatial MHSA and DCSA (Fig. 2, Eq. 9). Why wasn't a weighted sum used, as done in L236 for two-path integration? The model benefits from adaptively considering continuous and discrete features depending on noise levels. Employing a weighted sum to allow the model to prioritize specific feature could lead to improved estimation.**\n\n**A2:** Good suggestions! We apply weighted sum but obtained results similar to the original method when T=1. One possible explanation is that the model implicitly learns to balance the contributions from the two branches. Therefore, a straightforward addition already captures most of the benefits that a more complex weighted sum could provide, without adding extra parameters or complexity.\n\n| model               | MPJPE |\n| ------------------- | ----- |\n| ours                | 26.34 |\n| ours + weighted sum | 26.39 |\n\n\n> **Q3: Currently, since $head_i$ are the sum of $V_i$ with the weights derived from $softmax(QK)$ (which sums to one) for both of spatial MHSA and DCSA, this may hinder effective feature focusing.**\n\n**A3:** \nExcellent observation! In the original design, continuous queries interacted only with continuous keys/values, while a separate set of continuous queries interacted with discrete keys/values. \nHowever, this setup enforces that **a fixed number of continuous and discrete tokens are always involved in the output**, regardless of their actual feature quality or relevance in the current context. This can be limiting, since the relative importance of continuous and discrete features may vary depending on the context.\nTo address this, we redesign the attention mechanism by unifying the two attention branches into a single one: continuous queries now attend to the concatenation of both continuous and discrete keys/values. This not only simplifies the architecture but also allows the model to dynamically and adaptively focus on the most relevant combination of continuous and discrete tokens, without being constrained by fixed ratios.\n\nHowever, preliminary experiments show that this unified attention approach does not improve performance—in fact, it slightly degrades it (MPJPE increases from 26.34 to 26.51).\nThis might because that the two-path design inherently reduces coupling between branches, allowing each to specialize: one branch focuses on modeling continuous spatial pose relationships, while the other learns affinity to discrete tokens. This separation facilitates learning complementary features independently. In contrast, the single-path design requires the model to simultaneously capture both continuous spatial relationships and discrete token affinities within the same attention mechanism. This increased learning burden can cause feature interference and make it harder for the model to effectively disentangle and utilize the distinct modalities, leading to degraded performance.\n\n| model           | MPJPE |\n| --------------- | ----- |\n| ours (two path) | 26.34 |\n| ours (one path) | 26.51 |"}, {"review_id": "KTGktLNifV", "reviewer": "Reviewer_63UD", "summary": "This paper addresses the challenge of multi-view 3D human pose estimation (HPE), where traditional methods rely on precise camera calibration—limiting flexibility in dynamic settings—while calibration-free approaches, though more adaptable, suffer from error propagation due to unconstrained learning of view interactions. To overcome these limitations, the authors propose a novel framework that combines a noise-resilient discrete prior (Uni-Codebook) with continuous feature representations, enhancing robustness without sacrificing regression accuracy. Additionally, they introduce a Discrete-Continuous Spatial Attention (DCSA) mechanism to improve interaction between discrete and continuous features while preserving key attributes. Experiments on three benchmark datasets demonstrate that the proposed method outperforms both calibration-dependent and calibration-free approaches, achieving state-of-the-art performance in multi-view 3D HPE.", "questions": "1. What are the differences in experimental settings between the first row of Table 4(a) and the baseline in Table 4(b) that lead to the performance discrepancy?\n\n2. The title format of Reference 32 is incorrect.", "limitations": "The authors should discuss how the inference efficiency compares with FusionFormer.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**  \n1. The paper is clearly written, with a logical structure and well-organized presentation.\n2. Comprehensive experimental evaluations are conducted across multiple datasets, demonstrating the generalizability of the proposed method.\n\n**Major Weaknesses:**  \n1. Limited methodological novelty: This work proposes an incremental enhancement to existing methods by adding quantized 2D pose features. The DCSA module, while effective for feature fusion, primarily reuses self-attention and cross-attention mechanisms, limiting its conceptual novelty.\n2. Insufficient motivation and justification: The motivation for introducing quantized features remains unclear. First, can the quantized features maintain consistency with the 2D poses from different viewpoints? Second, the input settings differ between the first-stage learning (single action input) and the second-stage learning (multi-view and multi-sequence input)—could this discrepancy amplify feature misalignment and introduce additional noise?\n3. The experimental setup has several questionable aspects. First, the paper fails to provide an evaluation of the first-stage model's learning performance. Moreover, since the performance improvement is marginal, the authors should further validate—through visualization or other means—whether the proposed method genuinely addresses the challenges in camera parameter-free settings.", "was_revised": true, "final_justification": "I sincerely apologize for not replying sooner. I have carefully reviewed the author's response as well as the discussions from the other reviewers. I greatly appreciate the author's detailed reply, which has resolved all of my concerns. Therefore, I have decided to increase my original score by one point.", "rebuttal": "> **Q1: Limited methodological novelty: This work proposes an incremental enhancement to existing methods by adding quantized 2D pose features. The DCSA module, while effective for feature fusion, primarily reuses self-attention and cross-attention mechanisms, limiting its conceptual novelty.**\n\n**A1:** We are the first to introduce a data-drive discrete pose prior into continous 3D pose regression for noise resilience, and to propose to fuse discrete and continuous features with complementary strengths. The use of attention for fusion is not very novel but intuitive and effective. However, **the novelty stems from the insights of combining data-drive robust discrete priors with precise continuous representations to improve noise robustness and regression precsion, not from a single component.** As shown in **Figure 3** of the Supplementary Material, our model outperforms the pure-continuous baseline by up to **5 mm** under increasing noise levels. Across all noise intensities, our method consistently achieves lower MPJPE than the baseline, demonstrating its strong noise resilience.\n\n\n> **Q2: Insufficient motivation and justification: The motivation for introducing quantized features remains unclear. First, can the quantized features maintain consistency with the 2D poses from different viewpoints? Second, the input settings differ between the first-stage learning (single action input) and the second-stage learning (multi-view and multi-sequence input)—could this discrepancy amplify feature misalignment and introduce additional noise?**\n\n**A2:** \n**(1) Viewpoint consistency is neither required nor expected for quantized features.**  \nThe quantized features are not designed to maintain consistency across viewpoints. In fact, enforcing such consistency is neither feasible nor necessary. For example, 2D poses from different views (e.g., front vs. side) can differ significantly in visible joints and geometric patterns, and forcing consistency may lead to loss of view-specific information during Stage I. Our goal is not to align cross-view 2D features, but to learn a **discrete pose prior** that captures **robust and generalizable patterns** within individual views, which is sufficient for supporting downstream regression.\n\n**(2) Stage I and Stage II use different inputs without causing feature misalignment.**  \nStage I uses single-view single-frame inputs to fully exploit large-scale unpaired 2D pose data, while Stage II leverages multi-view multi-sequence data for final 3D regression. This difference is intentional and serves complementary purposes: Stage I learns a **noise-robust discrete prior**, while Stage II captures **cross-view and temporal dependencies** through continuous features.  \nThere is **no misalignment**, as the discrete prior is not required to encode multi-view or temporal cues—it only needs to offer **robust structural guidance**. Multi-view and temporal modeling are deferred to the continuous branch, which is optimized jointly with the discrete tokens during Stage II, ensuring effective integration without conflict.\n\n\n> **Q3: The experimental setup has several questionable aspects. First, the paper fails to provide an evaluation of the first-stage model's learning performance.** \n\n**A3:** We report the performance of Stage I on AMASS in **Table 6 in Supplementary Material**. For single-frame, single-view evaluation on Human3.6M GT, Stage I achieves 40.2 MPJPE. It is important to note that the entire VQ-VAE contains only 5.54M parameters, which is quite small. And as expected, the discrete representation yields lower regression accuracy compared to continuous features. We will include those results in the main paper.\n\n\n> **Q4: Moreover, since the performance improvement is marginal, the authors should further validate—through visualization or other means—whether the proposed method genuinely addresses the challenges in camera parameter-free settings.** \n\n**A4:** \n**(1) Non-Marginal Performance Gains on Saturated Benchmarks.**  \n\n(1.1) Saturation of Human3.6M and Stronger Results on 3DHP\n\nWe would like to clarify that Human3.6M is already saturated in terms of performance, and Human3.6m contains some inaccurate annotations[4], making further improvement increasingly difficult. \nUnlike Human3.6M, which is a lab dataset, 3DHP includes both indoor and outdoor scenes, offering higher diversity and better reflecting real-world scenarios. \nOur method shows stronger performance on 3DHP without extra training data, with a **2.03 mm MPJPE reduction and a 4.24 AUC increase** over the second-best method, FusionFormer and SGraFormer.\n\n(1.2) Addressing the Overlooked Real-World Noise: A Novel Use of Data-Driven Discrete Priors for Robustness.\n\nOur core motivation is as follows: continuous representations are well-suited for regression tasks but are vulnerable to noise, while discrete representations tend to be noise-resilient but suffer in precision. Our method aims to leverage both to improve robustness without sacrificing accuracy. As shown in **Figure 3 of the supplementary material**, using DCSA and the discrete codebook improves noise robustness significantly, achieving up to **6% (5 mm MPJPE) improvement under large noise**, which is far from marginal.\n\n(1.3) Steady Performance Gains Through Progressive Discrete Prior Quality\n\nTable 4 (a)(c) shows **consistent and stage-wise improvements** when increasing the quality of the VQ-VAE prior (see L304–307, L322–324). These improvements are achieved without adding any model parameters, solely by enhancing the quality of the UniCodebook, which directly correlates the **quality of discrete prior with model robustness** and validating the effectiveness of our method, especially under noisy inputs.\n\n**(2) Enhancing Noise Robustness in Camera Parameter-Free Settings Which Is Important for Real-World Deployment**\n\nThe concern you raised is exactly at the core of our work! In real-world camera parameter-free settings, noise is a big challenge. The core problem our method aims to address is noise sensitivity under uncalibrated conditions. Without camera extrinsics to provide geometric cues, such methods rely more heavily on semantic information from the data to perform multi-view fusion. This makes continuous-feature-based models, especially transformers, more vulnerable to noisy input data.\nTo tackle this, we introduce discrete, data-driven human pose priors to support continuous regression. This improves robustness without sacrificing accuracy. As shown in **Figure 3 of the supplementary material**, our model consistently outperforms the continuous-only baseline under varying noise strengths and view counts, with up to a **5 mm (6.25%)** improvement under the highest noise level. This confirms that our method effectively mitigates the noise issue in camera-free settings.\n\n\n> **Q5: What are the differences in experimental settings between the first row of Table 4(a) and the baseline in Table 4(b) that lead to the performance discrepancy?**\n\n**A5:** Table 4 (a) T=1, Table 4 (b) T=27. Unless otherwise specified, all results in the paper default to T = 27. We apologize for not making this clear in the manuscript.\n\n\n> **Q6: The title format of Reference 32 is incorrect.**\n\n**A6:** We have corrected it.\n\n\n> **Q7: The authors should discuss how the inference efficiency compares with FusionFormer.**\n \n**A7:** We report the results in the table below, which shows that the computational cost of our model grows approximately linearly with the number of views and frames, consistent with Supplementary Table 4. Note that the reported parameter count is smaller than that in Figure 4 as we mistakenly included redundant components such as the 2D decoder, 3D encoder, and 3D decoder of VQVAE.\n\n| model        | Params (M) | MACs (G) v=1/2/3/4 and  f=1 | MACs(G) v=4 and f=3/9/27 |\n| ------------ | ---------- | --------------------------- | ------------------------ |\n| Fusionformer | 17.25      | 0.10/0.20/0.30/0.41         | 1.23/3.69/11.08          |\n| Ours         | 23.14      | 0.13/0.27/0.40/0.54         | 1.66/4.98/14.94          |"}]

Author rebuttal:

Meta-review: The paper presents a multi-view 3D human pose estimation method considering a noisy and calibration-free setup by leveraging data-driven discrete priors. The proposed method shows the SOTA performance. 

The paper initially received very mixed ratings. However, after the rebuttal and author–reviewer discussion phase, the authors provided thorough responses with additional verifications. As a result, two reviewers raised their scores, leading to three acceptance ratings and one rejection rating.

The AC acknowledges that the proposed noise-robust discrete priors demonstrate clear efficacy in the target scenarios and that the paper provides sufficient evidence of its strengths. Given this, the AC recommends acceptance.

The rebuttal was particularly important in convincing reviewers, and the authors are strongly encouraged to revisit it and incorporate it into their revision. In addition, the final comments by Reviewer iezF are highly valuable, and the authors should carefully reflect on this feedback.
Decision: Accept (poster)