## 3 Immersive Dance Training System
### Extrinsic Camera Calibration  
Calibrating external camera parameters is vital for 3D multi-view analysis. Traditional methods, which often use 2D features like chessboard corners, struggle with wide camera spacings. Our approach uses human skeletal data from Mediapipe to automatically adjust camera settings as a user moves in a predefined space. This movement allows the camera to collect data without over- or under-collecting. We also introduce room calibration for global positioning, where the user's initial pose and the height of their VR headset are used for scale transformation.  
However, this approach has a limitation: it is highly sensitive to the user's initial pose. Because the frontal plane is determined based on the orientation of the body, it contains a large amount of prediction error, which can introduce instability and affect the accuracy of the calibration. To mitigate this issue, we have developed a method that utilizes the movement trajectory of the VR headset for a more robust estimation that is less dependent on the user's initial pose. Specifically, this new method synchronizes the headset's trajectory data with the 'nose' keypoint coordinates obtained from pose estimation to achieve a more accurate and stable calibration.  
In contrast to prior research [56; 57] that utilized human skeletons as oriented point clouds rather than 2D or 3D corresponding points, our method avoids time-consuming bundle adjustments often required for system optimization. This is particularly beneficial in our case, where the system comprises only two cameras, making the benefits of bundle adjustment marginal at best. Although our method necessitates specific user movements for calibration, it enables rapid and straightforward calibration without the computational overhead of bundle adjustment.  
Table 1 compares our technique with earlier studies with respect to various evaluation metrics. These metrics include the Riemannian metrics \(E_{R}\) for rotation matrices, the root mean square errors \(E_{t}\) for translational vectors, and the root mean square errors \(E_{3D}\) for the accuracy of reconstructed 3D joint positions. \(N_{c}\) is the number of cameras used. Our dual-camera approach not only delivers high real-time performance but also comes close to achieving the highest accuracy when compared to more complex methods that utilize five cameras and bundle adjustment.  
\begin{table}
\begin{tabular}{l c c c c} \hline  & \(E_{R}\) & \(E_{t}\) & \(E_{3D}\) & \(N_{c}\) \\ \hline [PERSON] [57] & 0.368 & 5.355 & - & 2 \\ [PERSON] (L) [56] & 0.043 & 1.414 & 1.817 & 5 \\ [PERSON] (L + B) [56]: & 0.020 & 0.053 & 0.041 & 5 \\ Ours & 0.036 & 0.102 & 0.098 & 2 \\ \hline \end{tabular}
\end{table}
Table 1: Comparison with previous studies. [PERSON] [57]: Metrics obtained using a dual-camera setup. [PERSON] (L) [56]: Metrics obtained with a five-camera setup using linear calibration only. [PERSON] (L + B) [56]: Metrics obtained when bundle adjustment is additionally applied to LEE (L). OURS: Evaluation metrics obtained with our proposed dual-camera method.  
Figure 6: Motion capture process.  
The environment in which camera calibration is performed is as follows. Automatic calibration is performed by walking behind the avatar that appears in the MR space. If the estimated camera position is significantly off, the position can be adjusted manually as shown in Figure 7.