# End-to-End Online Video Stitching and Stabilization Method Based on Unsupervised Deep Learning
## 2 Methodology
### Trajectory Smoothing  
The trajectory smoothing module aims to eliminate discontinuities in motion trajectories caused by camera shake and complex environmental conditions while maintaining consistency in smoothness across different scene layouts and depth conditions. Traditional methods such as meshflow [10] and RTVSASP [14] typically apply Gaussian filtering or Kalman filtering to smooth global trajectories, using the first video frame as an absolute reference coordinate. However, these approaches are prone to cumulative errors over time, leading to deviations in the final stitched result. To address this issue and achieve real-time video stabilization, this study adopts the concept of DUTnet [15] and proposes a CNN-based online trajectory smoothing method. The core idea is to establish a mapping relationship between the raw camera trajectory of the current frame and the historical smoothed trajectory using a dual-input network structure. The network framework is illustrated in Figure 3.  
To integrate trajectory information over time, a sliding window strategy is adopted. Let \(P_{raw}(t-n:t)\) represent the raw camera trajectory from time \(t-n\) to \(t\), with a dimension of [2,_n,M,N_], where \(M\) and \(N\) denote the number of grid rows and columns in each video frame, respectively. Similarly, let \(P_{smooth}(t-n:t-1)\) denote the historical smoothed trajectory of the preceding \(n-1\) frames, with a dimension of [2,_n_-1,_M,N_]. To fully utilize both sources of information, the method first maps the 2D trajectory features into a 64-dimensional feature space through independent embedding layers:  
\[F_{\text{embed,raw}}=\text{EMbedding}(P_{raw}(t-n:t)) \tag{12}\]  
\[F_{\text{embed,smooth}}=\text{EMbedding}(P_{smooth}(t-n:t-1)) \tag{13}\]  
Subsequently, a concatenation operation is performed to fuse the two embedded features into a unified feature tensor:  
\[F_{combined}=Concat(F_{\text{embed,raw}},F_{\text{embed,smooth}}) \tag{14}\]  
This fused feature tensor contains information from both the current raw trajectory and the historical smoothed trajectory, facilitating the network's ability to capture local temporal and spatial variations.  
A 3D convolutional layer is then applied to extract features from \(F_{combined}\). The 3D convolution has a receptive field in both spatial and temporal dimensions, enabling it to capture continuous trajectory variations over time. After multiple layers of 3D convolution, the network ultimately outputs a set of parameters \(\theta t\), which include smoothing weights \(wk\) and offset values \(bk\), collectively defining the mapping relationship from the raw trajectory to the smoothed trajectory. Given a local smoothing window radius \(l\) (typically set to \(l=3\)), the smoothed trajectory at time \(t\) is updated as follows:  
\[P_{\text{smooth}}(t)=\sum_{k=-l}^{l}\left(w_{k}\cdot P_{raw}(t-k)+b_{k}\right) \tag{15}\]  
By restricting accumulation to a finite time window, this approach effectively reduces error propagation while enhancing the stability and real-time performance of online smoothing.  
Figure 3: Network architecture of trajectory smoothing.  
The proposed trajectory smoothing method integrates raw trajectory and historical smoothed trajectory information using a dual-input network structure and extracts spatiotemporal features via 3D convolutional layers. The final smoothing weights and offsets ensure real-time trajectory smoothing. Additionally, the relative path accumulation strategy significantly minimizes cumulative errors, ensuring a more robust and accurate video stabilization process while maintaining computational efficiency.