Title: Background Matters: Robust 3D Human Pose Estimation via Controllable Video Generation
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: 3D Human Pose Estimation, Domain Generalization, Video Generation

Abstract: Deep learning models for 3D human pose estimation (HPE) often fail to generalize across domains with varying environments, camera setups, or data distributions. We address this challenge with a controllable video synthesis framework that systematically varies poses, scenes, and viewpoints to generate diverse human motion sequences. 
Unlike conventional methods that only augment skeletal data, our approach produces RGB video sequences, from which 2D poses can be extracted for both training and inference. This integration of video modality helps bridge the gap between idealized 2D pose inputs and real-world detection noise. We explore different combinations of ideal and real-world 2D inputs during training and evaluation, showing that while ideal conditions yield the best theoretical accuracy, using real-world detections leads to significantly stronger robustness and generalization.
Experiments across multiple benchmarks, including corrupted 2D inputs, demonstrate that models trained with our synthetic videos consistently outperform baselines. These results highlight the importance of scene diversity in 3D HPE and establish controllable video generation as an effective strategy for domain-generalized pose estimation.