Title: Towards Generalizable 3D Human Pose Estimation via Ensembles on Flat Loss Landscapes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Lifting-based 3D Human Pose Estimation
Abstract: 3D Human Pose Estimation (HPE) is a fundamental task in the computer vision. Generalization in 3D HPE task is crucial due to the need for robustness across diverse environments and datasets. Existing methods often focus on learning relationships between joints to enhance the generalization capability, but the role of the loss landscape, which is closely tied to generalization, remains underexplored. In this paper, we empirically visualize the loss landscape of the 3D HPE task, revealing its complexity and the challenges it poses for optimization. To address this, we first introduce a simple adaptive scaling mechanism that smooths the loss landscape. We further observe that different solutions on this smoothed loss landscape exhibit varying generalization behaviors. Based on this insight, we propose an efficient ensemble approach that combines diverse solutions on the smooth loss landscape induced by our adaptive scaling mechanism. Extensive experimental results demonstrate that our approach improves the generalization capability of 3D HPE models, and can be easily applied, regardless of model architecture, with consistent performance gains.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOverall, this submission focuses on improving the generalization performance of 3D human pose estimation models. It achieves this with the help of flat loss landscape and ensemble.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\nOverall, this submission is easy to follow while it emphrically also demonstrates good performance.\n\nWeaknesses:\n\n(See questions section for more details).\n\nQUESTIONS:\nOverall, the reviewer currently votes for borderline rejection for this submission due to the following concerns.\n\n1. Firstly, the reviewer is confused about the relationship between the proposed method and investigation and the 3D HPE task. Specifically, while it can be admitted that generalization is important in 3D HPE, it is also important for almost any other computer vision tasks. Thus, why the authors focus on improving the generalization performance of models while specifically focuses on 3D HPE is confusing, especially considering that the reviewer seems to also fail to get the relationshipf between the proposed method and the 3D HPE task. Instead, it seems that both parts of the methods are quite general methods that are not specifically relevant to the 3D HPE task.\n\n2. Moreover, the reviewer also appreciate the authors to better elaborate and explain on the significance behind its key insights. Specifically, the first key insight that the authors seem to aim to elaborate in Sec 2.4 seems to be existing and relatively general. The reviewer thus fails to understand why this insight need to be re-mentioned particularly in the 3D HPE context. Similarly, the second key insight that ensembling can improve generalization capability looks to be also very very general and common sense from the reviewer's perspective.\n\n3. Meanwhile, the authors claim that \"our method requires nearly the same amount of time as training a single model\". Yet, this claim seems to be very inadquately supported. Specifically, the reviewer seems to only see from Table 2 in Appendix that, across different baselines, the proposed method all obviously increase the inference time. Yet, the reviewer seems to see no clear comparison w.r.t. the training time. The reviewer appreciates explanation over the improper support of this claim.\n\n4. Lastly, in the introduction line 23-28, the authors seem to want to show that many 3D HPE methods have focused on enhancing robustness. While this is the case, in the experimental section, the authors seem to compare to none of these robustness enhancing 3D HPE works. Instead, it only uses several methods as baselines and apply the proposed method on these baselines. This leads the experiments to be significantly non-comprehensive.\n\nGiven the reviewer's above concerns, the reviewer currently tends to reject this submission.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-11"}

comments: SUMMARY:
This paper explores the loss landscape in 3D human pose estimation task, and proposes an adaptive scaling mechanism to smooth the loss landscape. The authors visualize the loss landscape and show its impact on performance improvement. Based on this, they introduce a simple, model-agnostic ensemble method that leverages the smoothed landscape to combine diverse solutions. The approach improves performance across various HPE methods on benchmarks like Human3.6M and MPI-INF-3DHP.

STRENGTHS AND WEAKNESSES:
Strengths
1. The paper is well-structured and easy to follow.
2. The adaptive scaling mechanism and ensemble method are simple enough to reproduce.
3. The experiments clearly demonstrate the smoothed loss landscape and the performance improvement.

Weaknesses
1. As mentioned in the limitations, the paper lacks theoretical analysis of the proposed method and its results, making it difficult to understand the underlying principles and justify its effectiveness.
2. Section 3.2 does not explain how the different regression heads are obtained. Are they trained on different datasets, or are they simply initialized with different random seeds?
3. In Figure 6(d), the performance of both methods is similar under small perturbations. At a perturbation level of 0.2, ΔMPJPE exceeds 100, which is significant. Although some robustness is shown under large perturbations, such a large error renders the results practically unusable.
4. It is unclear how the value of K is selected and how it impacts the results. The paper lacks experiments or analysis to support this design choice.

QUESTIONS:
Please see the weaknesses listed above. I would be inclined to increase my rating if these concerns are addressed.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-17

comments: SUMMARY:
This paper poses the critical issue of generalization in 3D Human Pose Estimation (HPE) by shifting focus from network architecture design to the loss landscape geometry. The authors first analyze the complex and fragmented loss landscapes present in HPE, particularly those arising from depth ambiguities. They introduce an adaptive scaling mechanism that encourages flatness in the local minima of the loss surface and propose an efficient ensembling framework based on multiple regression heads. This approach allows the model to explore and average diverse but high-performing solutions within a smoothed loss region. Experiments on Human3.6M and MPI-INF-3DHP demonstrate consistent improvements across multiple architectures (MLP, CNN, GCN, Transformer), both in-domain and cross-dataset.

STRENGTHS AND WEAKNESSES:
1. Novel perspective on generalization in 3D HPE: The paper introduces an underexplored but highly relevant perspective by analyzing the loss landscape of 3D HPE. The findings are well-motivated and supported by visualizations and eigenvalue analysis.
2. Simple yet Effective Adaptive Scaling: The adaptive scaling mechanism is elegant, architecture-agnostic, and incurs minimal additional computational cost. 
3. Thorough Experimental Validation: The paper conducts extensive experiments with multiple architectures and benchmarks, including cross-dataset evaluations, noise robustness, and comparison with SAM. Ablation studies clearly isolate the contribution of each component.

QUESTIONS:
The paper focuses on 3D HPE. Could the proposed method be applied to other tasks with similarly complex loss landscapes, such as 2D pose estimation or object detection?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-19

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of generalization in 3D Human Pose Estimation (HPE). The authors argue that the loss landscape of 3D HPE is highly complex and has multiple disconnected local minima, which can negatively impact model generalization. To address this, they propose a simple adaptive scaling mechanism that smooths the loss landscape, reducing the likelihood of high-loss barriers between local minima. Furthermore, they introduce an efficient ensemble approach that combines diverse solutions on the smoothed loss landscape. The method is designed to be model-agnostic and easy to implement. Experimental results on benchmark datasets such as Human3.6M and MPI-INF-3DHP demonstrate improvements in generalization performance across various network architectures.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\n1. The paper provides a novel analysis of the 3D HPE task from a loss landscape perspective, which is underexplored in previous research. This analysis offers valuable insights into the challenges of optimizing 3D HPE models and their generalization performance.\n\n2. The proposed adaptive scaling mechanism and ensemble approach are simple yet effective. They are designed to induce flatness in the loss landscape, which is associated with better generalization. The method shows consistent performance improvements across different model architectures and datasets.\n\n3. The experiments are comprehensive and well-designed. The authors demonstrate the effectiveness of their method on multiple benchmark datasets and compare it with various state-of-the-art architectures. The results show significant improvements in generalization performance, validating the practical utility of the proposed approach.\n\n4. The method can be easily integrated into existing 3D HPE frameworks regardless of the model architecture. This makes it a versatile enhancement for the community.\n\n\nWeaknesses: \n\n1. While the paper provides empirical evidence of the effectiveness of the proposed method, the theoretical understanding of why and how the adaptive scaling mechanism and ensemble approach work is somewhat limited. More in-depth theoretical analysis could strengthen the paper. \n\n2. The paper primarily focuses on the generalization performance within the context of benchmark datasets. It would be beneficial to discuss the applicability of the method in real-world scenarios where data distributions may vary more significantly.\n\nQUESTIONS:\n1. How do you ensure the diversity of solutions when using different random seeds? Is there a risk of the models converging to similar local minima despite different initialization?\n\n2. Can you provide more insights into how the adaptive scaling mechanism affects the model's learning process? Are there any potential drawbacks or limitations to this mechanism?\n\nLIMITATIONS:\nYes, but it is in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a training and ensemble strategy for 3D human pose estimation (3D HPE) based on loss landscape analysis. The motivation is that the 3D HPE’s loss surface contains multiple disconnected sharp minima that hinder generalization. To address this problem, an adaptive scaling mechanism is adopted to smooth the loss landscape and encourage the discovery of diverse, generalizable solutions. Building on this, they propose a simple and architecture-agnostic ensemble method that effectively leverages solution diversity to enhance robustness, all with minimal training overhead. Experiments are conducted on Human3.6M and MPI-INF-3DHP, to validate the effectiveness of the proposed designs on 4 typical architectures, MLP, CNN, GCN, and Transformer. The proposed designs achieve promising improvement (over 1.6% at most time) in both MPJPE and PA-MPJPE.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths.\n1. New perspective.  \nThis paper introduces a insightful approach by analyzing the 3D HPE at the perspective of loss landscape, which is rarely explored in this field. \n2. Generalizable design.  \nThe proposed adaptive scaling mechanism seems to be easy to added to different model and architecture-agnostic. This great property makes it widely applicable. Experiments on Human3.6M and MPI-INF-3DHP validate it. \n3. Insightful experiments.  \nThis paper provides insightful experiments to show that the proposed method leads to flatter local minima by estimating the top-1 eigenvalue of the Hessian matrix and visualizing both local and global loss landscapes. This analysis shows that adaptive scaling reduces curvature and eliminates high-loss barriers between solutions, which supports the claim of improved generalization.  \nAlso, the comparison with Sharpness-Aware Minimization (SAM), an existing method with global flattening effect, shows that the proposed   method can avoid biasing the landscape around specific local minimal. \n\n\nWeakness. \n1. Lacking sufficient details of experiment settings.   \nTo further convince the readers, we really need to add more details of the experiments on H36M and MPI-INF-3DHP, including the training datasets used in Tab. 3 and 4, how to combine the SAM, etc. The SAM comparison would benefit from stronger experimental support, such as better tuning or combining with ensembling.  \nAlso, the Hessian estimation method (e.g., power iteration, approximation details) is not clearly described and should be explicitly stated for reproducibility.\n2. The models are only trained on Human3.6M. Whether the proposed can help the model trained on more datasets remains unclear.   \nOne potential limitation is that all experiments are conducted with models trained exclusively on the Human3.6M dataset. While the cross-dataset evaluation on MPI-INF-3DHP is valuable, it remains unclear whether the proposed adaptive scaling mechanism can provide similar benefits in more realistic settings where models are trained on large-scale and diverse datasets (e.g., combined H36M + MPI-INF-3DHP + BEDLAM, and validate on 3DPW). In such settings, the optimization landscape may already exhibit flatter or more entangled structures, potentially reducing the marginal gain from the proposed smoothing strategy. The generality of the method in large-data regimes thus remains an open question and could be further explored.\n\nQUESTIONS:\n1. Please add the details of experiment settings. \n2. Experiments that train with more datasets would be very helpful to identify the significance of the proposed method.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-04"}