# Dual-Modality Cross-Interaction-Based Hybrid Full-Frame Video Stabilization
## 5 Conclusions  
The primary objective of our research was to generate visually useful imagery for Augmented Reality applications by preventing cropping through resolution maintenance and enhancing stabilization performance by minimizing the degradation of stability and distortion. In pursuit of this goal, we maintained a focus on improving performance while adequately preserving execution speed, leading to the proposal of a novel hybrid full-frame video stabilization algorithm, based on dual-modality cross-interaction using neural rendering, not previously explored. Our method was evaluated using stability, distortion, and cropping metrics, demonstrating enhanced stabilization, especially when utilizing an IMU sensor to robustly counter flow inaccuracies. Overall, our method uniquely achieved TOP2 in the S/D/C metric and showed the most significant improvement in Turn envi \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Metric**} & \multicolumn{3}{c}{**Tree**} & \multicolumn{3}{c}{**Brick**} & \multicolumn{3}{c}{**Sand**} \\  & **LPIPS** & **SSIM** & **PSNR** & **LPIPS** & **SSIM** & **PSNR** & **LPIPS** & **SSIM** & **PSNR** \\  & \(\downarrow\) & \(\uparrow\) & \(\uparrow\) & \(\downarrow\) & \(\uparrow\) & \(\uparrow\) & \(\downarrow\) & \(\uparrow\) & \(\uparrow\) \\ \hline L1 path & 0.174 & 0.294 & 20.740 & 0.118 & 0.353 & 15.924 & 0.071 & 0.213 & 22.820 \\ PCA flow & 0.106 & 0.418 & 23.496 & 0.138 & 0.407 & 17.296 & 0.046 & 0.383 & 24.783 \\ NR & 0.068 & 0.558 & 24.271 & 0.021 & 0.866 & 24.853 & 0.016 & 0.818 & 28.711 \\ DIFRINT & 0.086 & 0.579 & 23.774 & 0.069 & 0.591 & 18.962 & 0.029 & 0.629 & 25.535 \\ Ours & 0.040 & 0.856 & 28.689 & 0.031 & 0.846 & 22.771 & 0.024 & 0.744 & 27.481 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Quantitative comparison on a specific region. The first rank is marked in red color, while the second rank is marked in blue color. \begin{table}
\begin{tabular}{c c} \hline \hline
**Method** & **Computational Time Per Frame (Second) \(\downarrow\)** \\ \hline L1 path & 0.674 \\ PCA flow & 1.512 \\ NR & 5.400 \\ DIFRINT & 0.677 \\ Ours & 5.669 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Computational time comparison according to the average result of all four categories. The first rank is marked in red color, while the second rank is marked in blue color. ronments. Further, the visual quality induced by the Distortion value was quantitatively compared using the LPIPS, SSIM, and PSNR metrics, providing a detailed analysis. The use of the combination of an IMU sensor and neural rendering technique showed that while maintaining the Distortion value, it resulted in increased Stability score outcomes, effectively reducing visual artifacts caused by shaking. Our technology has proven valuable in correcting video shake, contributing to accurate target detection and tracking. The application of our technology is contingent upon the availability of IMU sensor values measured concurrently with the original video capture, which presents a limitation in terms of applicability to the vast array of videos available online. Despite this challenge, given that modern AR/VR devices and smartphones are inherently equipped with IMU sensors, leveraging these to acquire videos and applying our technology can result in visually superior, stabilized videos. Additionally, since the warping field depends on optical flow and the IMU sensor values are globally reflected across frames, visual artifacts may still persist at the video edges. Looking forward, we propose three areas for future work. First, we aim to achieve superior motion compensation by fusing the transformation matrices generated by deep learning-based optical flow calculations, assigning greater weights to the transformation matrices as the IMU sensor values increase. Moreover, there is a need to locally reflect this in the frame to enhance compensation at the video edges.