# Depth Segmentation Approach for Egocentric 3D Human Pose Estimation with a Fisheye Camera
## 2 Related Work
### Egocentric 3D Human Pose Estimation  
As HMD-based VR systems become more popular, several researchers [8; 9; 10] introduced methods that are compatible with an HMD. They mostly attached one or two cameras underneath a hat, glasses, or an HMD.  
The earliest study using the egocentric view is [PERSON]'s work [34], which estimated the stereo egocentric pose using a pair of fish-eye cameras. His main approach involved taking 2D features from each fisheye view and optimizing the 3D pose by extracting 3D features. [PERSON] [35] and [PERSON] [36] introduced a similar method involving two image regressions from two cameras. However, because these methods processed two images, they required higher computational resources and more electrical power compared to methods that regress a single image. This is particularly critical for head-mounted displays (HMDs), which rely on battery power.  
[PERSON] et al. [10] proposed the Mo2 Cap2 framework, which uses a single fisheye camera mounted underneath a hat. To overcome the rapid resolution degradation at body joints far from the fisheye camera, such as the ankles and knees, they proposed a two-stream CNN architecture. This architecture regresses a zoom-out image to track the overall body pose and a zoom-in egocentric image to specifically track the lower body joints.  
[PERSON] et al. [8] also used an egocentric view with a single fish-eye camera mounted underneath an HMD. They proposed the xR-EgoPose framework, which supports dual-branch decoders for lifting 2D features to 3D poses. They also published a large synthetic dataset supporting better image quality compared to the previously published Xu dataset [10].  
[PERSON] et al. [9] proposed a 3D pose estimation framework with two branches. The two branches extract the 2D body pose features with a heatmap and scene depth features with a depth map. They extracted the scene depth features after excluding the body region from the images and used the scene depth features to estimate the 3D human pose together with heatmap pose features. Their main idea was that the human pose is generally influenced by the objects around the human (e.g., hands on a keyboard often have a similar depth to the keyboard itself); thus, incorporating scene depth features into the regression can increase accuracy. In embedding these scene depth features, they implemented a scene voxel network [37]. This approach tends to reduce speed performance and consume significant computational resources. The use of scene depth features for 3D pose estimation offers advantages in tracking occluded body parts.  
[PERSON]'s approach [9] is somewhat similar to our framework in regard to using two branches to obtain depth features. Their depth features, however, embedded the scene information while our depth features embedded the body depth information. Additionally, [PERSON]'s approach would not be compatible with VR systems because the essential scene information (real-world objects around a user) in their approach would not be available. This is because people play VR games in empty spaces for safety reasons (to prevent bumping into real-world objects). Our approach, which uses only the body image, would be promising in VR systems regardless of the availability of scene information.  
[PERSON] et al. [38] proposed a self-supervised framework, named EgoFish3D, for 3D pose estimation, using a single fisheye camera and third-person view images. The framework solves key challenges in egocentric fisheye-based estimation, including severe image distortion, frequent occlusions, and a lack of real-world annotated datasets. EgoFish3D integrates three modules: (i) a third-person module that estimates 3D poses using synchronized third-person cameras; (ii) an egocentric module that predicts 3D poses with egocentric fisheye images, and (iii) an interactive module that aligns the third-person and egocentric coordinate systems. The framework shows strong robustness in real-world scenarios, demonstrating the potential of self-supervised methods for egocentric 3D pose estimation.