## 2 Materials and Methods
### Development of a Solution to Combine the Results of Two Tracking Methods
Therefore, the transformer network processed the video streams \(V_{f}(t)\) and \(V_{s}(t)\) from the front and side, respectively, over time \(t\) through a dual-stream encoder-decoder structure, where each stream was first processed independently to extract high-dimensional feature embeddings:  
\[F_{f}=\text{TransformerEncoder}(V_{f}(t)) \tag{1}\]  
\[F_{s}=\text{TransformerEncoder}(V_{s}(t)) \tag{2}\]  
These feature embeddings \(F_{f}\) and \(F_{s}\) were then fused using a cross-attention mechanism, which aligned the spatial features from both views. The fused representation was subsequently decoded to predict the 3D joint coordinates for each time step:  
\[S_{f},S_{s}=\text{TransformerDecoder}(F_{f},F_{s}) \tag{3}\]  
Instead of performing a simple rotational alignment, we employed Neural Implicit Fields (NIFs) [28] to model the non-rigid deformations between the side-view skeleton \(S_{s}\) and the front-view skeleton \(S_{f}\). The NIFs were represented as continuous volumetric functions \(\Phi(x,y,z)\), which mapped 3D coordinates to deformation vectors:  
\[\Phi_{s}=\text{NeuralImplicitField}(S_{s}) \tag{4}\]  
These fields allowed us to compute a smooth, differentiable transformation that aligned \(S_{s}\) with \(S_{f}\) without explicitly defining a rotation matrix. The aligned skeleton \(S_{s}^{\prime}\) was obtained by querying the implicit field at each joint's coordinates:  
\[S_{s}^{\prime}=S_{s}+\Phi_{s}(S_{s}) \tag{5}\]  
For depth error correction and depth prediction, we integrated Neural Radiance Fields (NeRF), inspired by [29], which modeled the volumetric scene geometry and appearance from sparse input views. NeRF was used to estimate the depth of joints more accurately by synthesizing novel views and refining the depth coordinates:  
\[D_{f}=\text{NeRF}(V_{f}(t)) \tag{6}\]  
Figure 2: A diagram illustrating the proposed solution algorithm. Figure 2.0  
These depth maps \(D_{f}\) and \(D_{s}\) were used to compute the depth error \(E_{d}\) as the discrepancy between the estimated depths from the front and side views:  
Figure 2.1  
To handle the temporal dynamics of the joint coordinates and mitigate the impact of noise, we implemented a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) cells. The LSTM network processed the sequence of 3D joint coordinates over time, \(S(t)\), applying dynamic smoothing and error correction:  
Figure 2.2  
The output \(\hat{S}(t)\) represents the temporally smoothed and refined 3D skeleton over time, with errors minimized and motion dynamics accurately captured. The final step involved reconstructing the comprehensive 3D human skeleton by integrating information from the processed streams \(\hat{S}_{f}(t)\) and \(\hat{S}_{s}^{\prime}(t)\) and any missing joint points. A Graph Neural Network (GNN) was employed to model the relationships between joints, leveraging the skeletal graph structure:  
Figure 2.3  
The GNN iteratively updated the joint coordinates by aggregating information from neighboring joints, ensuring a consistent and anatomically plausible final 3D skeleton \(S_{final}\).