# Diverse Humanoid Robot Pose Estimation from Images Using Only Sparse Datasets
## 1 Introduction  
Human-robot interaction (HRI) has become a pivotal area of research in robotics and extended reality (XR), driven by the increasing capabilities and applications of humanoid robots. Recent advancements in humanoid robots showcase remarkable physical abilities that closely mimic human movements, such as fast running, precise object manipulation, and sophisticated AI-assisted actions [1, 2]. As these robots are integrated into diverse environments, accurate pose estimation has become critical to enhance their interaction with humans and their operational effectiveness in XR applications [3, 4]. Pose estimation involves determining the position and orientation of a robot's body parts from visual data, which is crucial for understanding the robot's current status during interactions. This capability not only supports real-time operations, but also enhances the user's immersive experience in XR environments. Despite its significance, accurate pose estimation for a wide variety of robots remains a challenge due to the diversity in robot designs and applications. Traditional pose estimation methods for robots often rely on visual markers or internal feedback mechanisms. Visual markers can be intrusive and limit the robot's operational flexibility, while internal feedback systems are frequently customized for specific robots orenvironments, reducing their adaptability to new or varied scenarios [3; 4]. Moreover, these methods typically require extensive setup and calibration, which can restrict their use in controlled lab environments. Recently, learning-based methods for pose estimation have gained significant attention. Convolutional neural network (CNN)-based approaches have been applied to robot arm pose estimation [5; 6; 7], as summarized in Table 1(a). However, these methods are restricted to arm joints and have been tested on only a limited number of robot types. For humanoid robots, several pose estimation techniques have been proposed, as outlined in Table 1(b). [PERSON] et al. [8] applied their method to an upper-body humanoid robot, estimating dual-arm poses using inverse kinematics (IK), but without addressing full-body estimation. Similarly, [PERSON] et al. [9] employed CNN-based keypoint detection for humanoid robot pose estimation, though their approach was limited to end-effector joints and lacked full-body coverage. A full-body 3D pose estimation method from head-worn cameras was introduced by [PERSON] et al. [10], but the method was validated on only a single type of robot. None of these approaches have been evaluated on a diverse set of robots (e.g., more than ten types) or focused on estimating full-body joints, leaving the general applicability of pose estimation methods an open problem. Collecting pose datasets for a wide variety of robots poses another challenge, as setting up capture systems and recording diverse poses is costly and labor-intensive. Moreover, acquiring a variety of humanoid robot videos from providers is extremely limited, hindering the formation of large-scale datasets. To address these challenges, we present a new pose dataset for diverse humanoid robots, combining small real-world datasets with synthetic data generated using our approach. We collected public videos from the official websites of 11 state-of-the-art humanoid robots [1; 2] and manually annotated them to form sparse real datasets. The combined dataset leverages these sparse pose data for each humanoid robot and augments them with synthetic data generation techniques, including AI-assisted image synthesis, foreground removal, and 3D character simulations. By merging real and synthetic data, we create a robust dataset that provides full-body pose annotations for a variety of humanoid robots, capturing diverse motions such as front, side, back, and partial views in real-world scenarios. To the best of our knowledge, no existing dataset includes full-body pose annotations for diverse humanoid robots. An overview of our approach for diverse humanoid robot pose estimation is presented in Figure 1.