Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations
Authors: Haoming Cai
Jingxi Chen
Brandon Y. Feng
Weiyun Jiang
Mingyang Xie
Kevin Zhang
Cornelia Fermuller
Yiannis Aloimonos
Ashok Veeraraghavan
Christopher Metzler
Venue: NeurIPS 2024 poster (2024)
Keywords: Atmospheric Turbulence Mitigation
Primary area: machine_vision
TL;DR: Improving temporal consistency in video atmospheric turbulence mitigation by separating out and regularizing temporal-wise neural representations

Abstract: Atmospheric turbulence, caused by random fluctuations in the atmosphere's refractive index, introduces complex spatio-temporal distortions in imagery captured at long range. Video Atmospheric Turbulence Mitigation (ATM) aims to restore videos affected by these distortions. However, existing video ATM methods, both supervised and self-supervised, struggle to maintain temporally consistent mitigation across frames, leading to visually incoherent results. This limitation arises from the stochastic nature of atmospheric turbulence, which varies across space and time. Inspired by the observation that atmospheric turbulence induces high-frequency temporal variations, we propose ConVRT, a novel framework for consistent video restoration through turbulence. ConVRT introduces a neural video representation that explicitly decouples spatial and temporal information into a spatial content field and a temporal deformation field, enabling targeted regularization of the network's temporal representation capability. By leveraging the low-pass filtering properties of the regularized temporal representations, ConVRT effectively mitigates turbulence-induced temporal frequency variations and promotes temporal consistency. Furthermore, our training framework seamlessly integrates supervised pre-training on synthetic turbulence data with self-supervised learning on real-world videos, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data. More information can be found on our project page: https://convrt-2024.github.io/

Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations
Haoming Cai"! Jingxi Chen"! Brandon Y. Feng". Weiyun Jiang', Mingyang Xie
Kevin Zhang'. Cornelia Fermuller', Yiannis Aloimonos', Ashok Veeraraghavan', Christopher A. Metzler
University of Maryland, "Massacuusetts Institute of Technology, 'Rice University
Abstract
Atmospheric turbulence, caused by random fluctuations in the atmosphere's re- fractive index, introduces complex spatio-temporal distortions imagery captured at long range. Video Atmospheric Turbulence Mitigation (ATM) aims to restore videos affected by these distortions. However, existing video ATM methods, both supervised and self-supervised, struggle to maintain temporally consistent mitigation across frames, leading to visually incoherent results. This limitation arises from the stochastic nature of atmospheric turbulence, which varies across space and time. Inspired by the observation that atmospheric turbulence induces high- fiehffrey temporal variations, propose ConVRT, novel framework for consistent video restoration through turbulence. ConVRT introduces a neural video representation that explicitly decouples spatial and temporal information into spa- tial content field and temporal deformation field, enabling targeted regularization of the network's temporal representation capability. By leveraging the low-pass filtering properties of the regularized temporal representations, ConVRY effec- tively mitigates turbulence-induced temporal frequency variations and promotes temporal consistency. Furthermore, our training framework seamlessly integrates supervised pre- pre-training on synthetic turbulence data with self-supervised learning on real-world videos, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data. More information can be found on our project page: httpss:/convttt2000 2024.githubb.
1 Introduction
Atmospheric turbulence poses significant challenge in long-range imaging applications, causing unique distortions in captured videos. These tubbulence-distorted videos suffer from spatially-varying and time-varying degradations, including blur and warping effects, due to the random fluctuations of the refractive index in the atmosphere. These distortions significantly hinder the performance of computer vision applications like object detection, recognition, and surveillance systems by obscuring the true shapes, edges, and visual details of objects. Therefore, this work focuses on Video Atmospheric Turbulence Mitigation (ATM). aiming to recover videos degraded by these atmospheric distortions.
Mathematically, the process of capturing video through atmospheric turbulence can be modeled by the following equation
'Euual contributions / Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original
VITT
TMTT
DATUM
DATUM OuTS
10
Figure I: Temporally consistent restoration in video ATM is challenging, State-of-the-art methods like DATUM (2) (CVPR'24) and TMT (TCC2)3) designed for video ATM, fail to maintain temporal consistency real-world atmospheric turbulence. For instance, they produce flickering artifacts on static pole.
Vr distorted frame
Bg o Tij
4 clean frame
ult-then-bbur 7
where B, and T, are blur and tilt process at time stamp. denotes the application of tilt followed by blur (I).
As described by Equation T the key challenge arises from the stochastic nature of atmospheric turbulence, which varies across space and time, making temporally consistent video restoration difficult. Figure [ illustrates the challenge of temporal inconsistency by showing static scene captured with a stationary camera and object. Despite the static setup, the atmospheric turbulence introduces erratic movements of the stationary traffic cone across frames, causing flickering artifacts in the resulting video sequence. Notably, even state-of-the-art methods like DATUM (2), designed for video turbulence mitigation, fail to produce temporally consistent mitigation results, result in flickerng video. This underscores the critical need for novel solutions tailored to address the challenge of temporal consistency in video atmospheric turbulence mitigation.
1.1 Current State-of-the-Art in Atmospheric Turbulence Mitigation
To address this complex and variable degradation in images and videos, var- ious methodologies have been devel- oped. Current state-of of-the-art meth- ods can generally be categorized into supervised and self-supervised learn- ing manner.
Comparison of recent supervised (S), self- supervised (SS), and hybrid (S+SS) learning approaches for image and video ATM.
Table 1:
Supervision
Method
Capability
Critical Performance Factors
TSRWGAN 4
Static Scane Sequences
Advensarial L Learning
TurkkNet [5
Image
Advanced Simulator
8
PiRN-SB 0
Image
Advanced Simulator
Supervised learning techniques in ATM use turbulence simulators gen- erate paired training data (clean and distorted images/video) that can be
Vidao
Physically-
Mnyniclll--Gnonddd Grounded
DATUM
Video
Physically Grounded
Turb Sog: Res
Video
Advanced Simulator
Mao
M
Image
Lacky Imaging & Denoisers
SS
TurbuGAN [
Li al. [00
Stttic Scume Sequenczes
Imagee
Degrashtion Est Adversmiia Sensing Concepi
used for training (14] 15] 16: [ [7] 18). (A) and the supervised learn- ing section of Table 1 depict methods that achieve significant results based
NekI
Stutic Scone Sequences
Degradation Est
Template
[
Static Scene Sequences
Optical Flow
8488
ConVRT (ours)
Vidoo
Repersentation Regularization
on large amounts of paired data. Despite the continual evolution of simulators, the persistent gap between simulated and real-world atmospheric turbulence poses challenges for this design in han- dling unseen real-world data. In vidcos, this drawback is further amplified, leading issues like temporally inconsistent mitigation. To address this temporal inconsistency issue, in addition to better simulators, enlarging the dataset and model capacity are necessary, which substantially increases the computational costs of training.
Self-supervise learning approaches for ATM employ internal learning techniques to leverage data priors such as lucky images, internal data distributions, or blind degradation estimation, as depicted in Fig. B) and self-supervised learning section of Table A key advantage of these methods is their test-time optimization capability, allowing them to adapt to any test data. However, to date these approaches have not been used to enforce temporal consistency in video ATM. Furthermore,

---

Original
Y-t Slice
Slice
Atmospheric turbulence introduces variations in videos due to the chaotic motion of air caused by temperature gradients and other energy sources. These variations manifest as vareing tilt and blur, deviating from the ground truth, as evident in the rapidly fluctuating patterns along the temporal dimension (vertical axis) of the y-t slice of the turbulence-distorted video (left). In contrast, the restored video of SOTA method DATUM (2) (right) exhibits smoother temporal variations, indicating the mitigation of turbulence- induced distortions. This key insight highlights the potential for regularizing temporal information to effectively restore videos affected by atmospheric turbulence.
Figure 2: Inspiration of our method:
temporal
because they don't exploit accurate learned image priors, the performance of self-supervised methods in real-world turbulence mitigation often falls short of supervised learning approaches.
This paper develops a hybrid algorithm that can consistently mitigate real-world atmospheric turbu- Ience across video frames. Our pipeline can leverage the knowledge encoded in pre-trained models while leveraging test-time optimization to adapt to the complexities of real-world turbulence. As shown in Fig. 3JC) and the final section of Table [, our pipeline leverages the strengths of both self-supervised learning and simulation- based pre- training..
1.2 Motivation and Contribution
Our work motivated by the insight, illustrated in Figure 2 that state-of-the-art ATM methods struggle to remove the temporal distortions introduced by turbulence. That is, while existing methods are reasonably effective removing the spatial distortions (e.g.. blur) introduced by turbulence, they do not effectively remove the temporal distortions.
To address this challenge, develop an approach that explicitly decouples spatial and temporal information. This method leverages the low-pass filtering properties of neural networks to reduce turbulenceinduueed degradations. Specifically, we propose self-supervised method called CRn (Consistent Video Restoration through Turbulence). Con forms a neural representation of the reconstucted video that explicitly decouples spatial and temporal information: The video is represented with a spatial content field and a temporal deformation field. This decoupling allows Con VRT to effectively regularize the temporal information while preserving spatial information and fine details.
Through extensive evaluations, demonstrate that ConVRT substantially improves temporally consistency while also marginally improving per-frame restoration quality.
2
Related Work
Implicit neural representations. Our work leverages a coordinate-based implicit neural repre- sentation (INRs), which has been commonly adopted to model 2D images or 3D videos multi- layer perceptions (MLPs). INRs take 2D pixel coordinates (x, y). or 3D pixel coordinates with temporal encoding, (x, y, t) and output the corresponding pixel values. These INRs demonstrate exceptional performance when fitting images (19 20 21 22 23 24 25), videos (26) (22). 3D 30 3I), and optical components (32). Not only they able to represent these 2D or 3D signals, but they also show strong priors for solving inverse problems, such as image
shapes (27 28 26 29
super resolution (33), phase retrieval (34), and reducing optical aberration (35 37 38).
Neural video representation. Our work aligns closely with the evolving field of neural video representation (39 represent video into decomposed layers, these primarily focus on clean videos and are not applicable to videos with severe degradation turbulence. Our work extends the application of neural video
42 While there are existing approaches 45
44 42 45) that seek to
3

---

Supervised learning paired video image
B. selt-supervised learning
Turbulence
Simulator
Prior Y Mitigation Model
Physicully- Grounded
Loss
Mitigation
Model
Degradation
Supervised pre-training self-supervised learning testing video (Ours)
Loss
Our Firal
Network
Reguls-
Video
fization
Temporal
Spatial
Mitigation
Deformation
Content
Model
Fieldi
Viden/Image
Videa/Image
Ground-Trath
Weight
Prozen
Figure 3: Comparison of different learning approaches for image and video ATM. (a) ATM methods in supervised learning face challenges in domain adaptation for real-world data. (b) Self-supervised learning ATM methods mostly explore static video sequences, outputting one frame from multiple frames as input. (c) Our hybrid pipeline is tailored for video ATM, combining supervised pre-training and self-supervised learning to achieve consistent video restoration through turbulence
representation scenarios heavily impacted by atmospheric turbulence. This extension is not trivial, as it involves addressing the unique challenges posed by the dynamic and unpredictable nature of turbulence, which are not considered in conventional video representations.
Atmospheric turbulence mitigation. Attempts to mitigate atmospheric turbulence (46 47) have applied optical flow ( 48). B-spline grid (49), and diffeomorphism (50) to unwarp each distorted image and then fuse and combine these registered distorted images into a clean and sharp image. The fusion is usually modeled patch-wise stitching (9) or blind deconvolution (51) Recent development
of pegf-peeformance GPUs and fast turbulence simulators (16) [8] 17 15 14) leads to new progress (13 11 7 12). However, previous efforts tend to overlook the importance of temporal consistency on the reconstructed video. Our method, ConVRT, is specifically designed to restore temporal consistency with on test-time optimization of a neural video representation.
in turbulence mitigation
Blind Video Restoration via deep video prior. Supervised video restoration methods (52
55
have made significant advancements but are constrained by the need for paired data, which increases the value blind video restoration. One promising direction involves leveraging deep priors. The deep video prior (DVP) and DVP-based blind video consistency methods (56) 57) use convolutional neural networks (CNNs) to learn image operators that exploit the implicit priors in CNNs to remove video artifacts. These approaches have demonstrated impressive results in tasks such as colorization and haite-bblancing.. However, turbulence mitigation presents a more complex challenge compared to these common degradations, involving spatially and temporally varying blur and tilt. This complexity raises unexplored questions for these types of methods
Method
3.1 Overview of the Pipeline
The framework of our method, CRn is presented in Figure In this subsection, we provide a high-level overview covering the design inspiration, the video representation mechanism, and the training process.
Design Inspiration. As discussed in Section 1.2 the core design logic of ConVRT is to apply temporal-wisc regularization in video representation learning. For the representation, our method is inspired by series of works tensor decomposition, commonly used parameterize 3D volumes in implicit neural representations (INR). These approaches enhance the ability to represent 3D signals while reducing the number of required parameters (33 58 59). Building upon this, we developed the
Con VRT method.
Video Representation. CRT represents videos using two main components: the 3D Spatial- Temporal Deformation Field (Thela and the 2D Spatial Content Field (Shet). The process begins with Thea, which receives the pixel location (z. y, as input, where (æ, are spatial coordinates and is the temporal frame index. Tima outputs deformation offsets (Ax, Ag), indicating changes in the pixel's ss spatial position across frames relative to canonical frame. These offsets are then used by

---

Temporal Deformation Field Tyie
Spatial Content Field S Steld
Regularized Representation
[H,W, [.] Spetial (trainable)
Ax
Query Feature
Frume's i height
Fixel s.y.t
Temporal Feature
Spatial-
Deformation MLP
Ax Ay
Warped Spatial Feature
Content MLP
RGB Inreasity
W Prime's width
Tya Time resolution
[H. W. Qz] Canonical Spatial Feature Map (trainable)
[Prss Qx] Temporal Feature N (trainable)
O Hadamard prodact
Figure 4: Illustration of the proposed method. Con represents video with two fields: the Temporal Deformation Field (T'rieta) and the Spatial Content Field (ieta). Regularization is applied by constraining the dimensions of the Temporal Feature Map. Similarly, reducing the size of the deformation MLP serves as additional regularization to promote temporal consistency.
Snew- Sheld queries the Canonical Spatial Feature Map (C) at the modified location (x + Ax, y + Ay) to retrieve the corresponding feature from trainable feature map. This feature is subsequently processed by an MLP to predict the RGB intensity values for the pixel location (x,, y. 1).
Training Overview. During training, the trainable parameters include all feature maps and the Multi Perceptrons (MLP) described below. The loss function measures the difference between the predicted RGB intensity values and the corresponding pixel colors in the restored video, obtained using any ATM method. Since no ground-truth data is available, CRnVRT is designed to overfit each partially restored video; however, its limited capacity for capturing temporal information prevents it from overfitting to turbulence artifacts
3.2 Temporal Deformation Field Triedd with Regularization
We represent the input video's spatial-temporal features using main components: the Spatial Feature Map and the Temporal Feature Map. The Spatial Feature Map (M) acts as a dictionary for spatial features, with dimensions Hxx xQ: where is the frame height, W is the frame width, and Q1 is the number of spatial feature channels. Each pixel coordinate (æ, y) serves as a key to retrieve the corresponding spatial feature vector Ma.y E R The Temporal Feature Map (N) functions as dictionary for temporal features, with dimensions Tf Here, Tres is the regularized temporal resolution and Q1 is the number of temporal feature channels.
To construct the spatial-tempori temporal feature vector v2 at specific pixel location (z, y, t), first query the Spatial Feature Map M using the pixel coordinates (z, y), extracting the spatial feature vector Mz.y Rd: Next, we query the Temporal Feature Map N using the time coordinate extracting the temporal feature vector N, E Q These vectors are then combined using the Hadamard product, which performs element-wiss multiplication, to form the seniiaal feature vector:
Va = Ma. O N₁
This Hadamard product effectively combines the spatial and temporal features, creating a compact and efficient representation of the video's 's spatial-temporal characteristics. This Vr.g,t is then fed into a compact MLP, referred to as the deformation MLP. The details of the deformation MLP are provided in the supplementary material. The deformation MLP outputs the offsets (Az, Ay) necessary for warping the canonical spatial feature map.
To regularize the temporal representation capability of the Temporal Feature Map (N), we constrain its dimensions to IR. xQ, where Tree is much smaller than the total number of video frames (T). Consequently, multiple neighboring frames share the same temporal feature. For example, frames at / 1, I, and + may query the same temporal feature N, due to the reduced temporal resolution. Additionally, we define the deformation MLP with reduced number of parameters. Both regularizations decrease the representation capacity of the temporal features, promoting smoother and more consistent temporal dynamics across frames, as inspired by motivation experiment.
5

---

3.3 Spatial Content Field
The Spatial Content Field focuses on accurately representing the spatial details of each video frame. Unlike the Spatial Feature Map (M) used in the Temporal Deformation Field, we initialize new optimizable feature map, denoted as the Canonical Spatial Feature Map (C), with dimensions # XQ2, where H is the frame height, W is the frame width, and Q2 is the number of spatial feature channels specific to this field.
Each pixel coordinate (I, y) is adjusted by the deformation offsets (Ar, Ay), resulting in new coordinates (x + Az, y Ay). These adjusted coordinates are then used query the Canonical Spatial Feature Map (C), retrieving the spatial feature vector C2+Az.24 Az E g These spatial features are processed by Content MLP, which transforms the spatial feature vector CN+ Am.y+ Ay into the final RGB intensity values for the corresponding pixel. The details of the Content MLP are provided in the supplementary material. This transformation ensures that the spatial details of the video frame are accurately captured and represented.
3.4 Training Objectives
Temporal Consistency Regularization. To ensure temporal stability across video frames, use disparity estimation network (MiDas (60)) to calculate pixel-wise disparities. These disparities serve as weights for the predicted warp (one of Dield's 's outputs), helping maintain spatial consistency over time. The loss is defined as:
Ciemp (1 Disparity(()) |Preddicedd 1arp/
(3)
where Disparity(7) measures the pixel-level disparity, and |Prredictdd Warpllı enforces sparsity in the grid changes. The design of Ltemp minimizes the L1 norm of the predicted warp, conditioned by - Disparity(I)) (/), to prioritize consistency in far regions based on the depth information. This focused approach on temporal consistency significantly reduces the propagation of turbulence-induced distortions, ensuring a smooth transition between frames.
Similarity LOss. The Similarity Loss Term is given by:
Ssim Amsehuse Assim Essim + Mippss
where Amses Aeaims and Atpips are weights for each term. This loss term assesses the fidelity of the predicted output compared to the outputs of arbitrary ATM methods, incorporating Mean Squared Error (MSE), Structural Similarity Index Measure (SSIM) (61), and Learned Perceptual Image Patch Similarity (LPIPS) (62). This multifaceted approach ensures comprehensive evaluation of reconstruction quality.
Overall Loss. The overall loss combines the similarity loss with temporal consistency and semantic enhancement:
Liotal = Leim + Atemp
(5)
4 Experiments
4.1 Datasets and Training Details
We adopt several real-world datasets for evaluation, including the OTIS (33) HeatChamber (5), subset of BVI-CLEAR dataset (64), TSR WGAN dataset 4 and DOST (65). We trained the Con VRT model individually on each video clip with a learning rate of x 10 using the Adam optimizer (66). For each video clip, the batch size equals to the number of frames in that clip. The spatial resolution of both the trainable spatial feature map and the canonical spatial feature map matches the original frame resolution after square cropping. The temporal resolution parameter Trs with parameters Q1 and Q2 configured to 128 and 256, respectively. More details about the network settings are provided in the supplementary material. Training conducted single RTX A6000.
4.2 Evaluation Strategy
We selected VRT(22), TMT3), and DATUM as the base video methods for CRn due their state-of-the-art performance in video restoration and video ATM. TurbNet(5) is selected for base

---

Patch
Y-t
y
Original
VRT
VRT Ours
TMT
TMT + Ours
DATUM
DATUM Ours
Patch
Y-1 Slice
y
Original
VRT
VRT+ Ours
TMT
TMT + Ours
DATUM
DATUM + Ours
Patch
Y-t Siice
Original
VRT Ours
TMT
TMT DATUM DATUM Ours
Figure 5: Visualization of our method's effectiveness in mitigating real-world atmospheric turbulence compared to existing methods. The leftmost image shows the original frame with green box marking the zoom-in crop area for KLT tracking and red line for the Y-t slice, shown in the bottom left. The right side displays two rows: the first shows zoom-in KLT tracking results for baseline methods and their outputs enhanced by our method, and the second shows zoom-in Y-t slices highlighting the temporal consistency achieved. Note the significant reduction in erratic movements our results.
image method. We also directly applied ConVR' the original video without the base methods to assess its standalone performance. To evaluate the consistency of turbulence removal in videos, we employed four metrics for quantitative evaluation and two interframe-relatee methods for qualitative assessment.
Temporal Consistency and Per-frame Quality. We used PSNR and SSIM to measure the per- per-frame reconstruction quality. Following (67), we utilized the average warp error to quantify the temporal consistency of the restored video. The warp error between consecutive frames is defined as:
Ewarp (Ve Ve+1) =
Σ
M
(8)
v.0)
(00 2+1
(6)
ΣN M
where (-( 1++ is the warped frame by optical flow time + and M. E 0, 1 is the occlusion mask estimated by the methods proposed in (68). The average warp across the entire video sequence
calculated as:
T-1
1
Ewarp (V)
Ewamp Vi, Ve+1)
(7)
T

---

Valcobued ATM
Suagehuud
No. Metal
Obc: Roseenam
Typ
Danect
Motic
PANK.
18643
18.33
15.41
19/80
40.9
15.01
25.55
23
24.55
24.18
+0.54
150595
PoNK
18.39
10.06
+021
28.36
+071
40.77
10.0
15.0
48.27
333437
38740
6712.5
$71.36
11-5442
03:00
000.14
134431
581.3
122
31547
130.8)
CLEAM 14]
Flmms 3H166
990.4
01
4003-44
073.77
400
11033 17
99533
8331.00
545
522000
152.0
10.000
13R0WCA0 4
120.23
122.22
21:00
Table 2: Performance improvements achieved by applying our proposed Con VRT across various model architectures and datasets. The No Base Method columns show the results when the methodol- ogy was applied directly to the original frames, labeled Ori. Gains are highlighted for each metric, showing the effectiveness of Con VRT enhancing the temporal consistency in video ATM.
Patch
Sllco
GT
Zoom-In GT
Synth Turb
TMT
TMT Ours
DATUM
DATUM Ours
Figure 6: Comparison of turbulence mitigation techniques on synthetic dataset. Left: Ground truth (GT) frame and corresponding slices. Right: Zoom-in views of KLTT tracking (top row) and Y-t. slices (bottom row) for baseline methods and the enhancements brought by our method.
In addition to the warp loss, inspired by (9), we also calculate the Total Variation loss of the X-t slice, Slicetv, and Total Variation of the optical flow, Flowre. to quantitatively measure whether the temporal variation of time slices in restored videos are small.
KLT Trajectories. We employed the KLT tracker (70) to track feature points and plot their trajec- tories, as shown in Figure 5] KLT tracking is directly based on image gradient information, such that common issues in turbulence restoration, i.e., blurriness, artifacts, and temporal inconsistency, are reflected in the tracked trajectories. Smooth and coherent trajectories indicate temporally con- sistent restoration, while erratic or discontinuous trajectories suggest the presence of artifacts or inconsistencies.
x-f Slice. We plotted æ-t slices to visualize the motion a row of pixels, as illustrated in Figure the video restoration is temporally consistent, the z-- slice plot will exhibit smooth and continuous curves, In contrast, non-smooth or jagged curves in the x-t slice indicate temporal inconsistencies or artifacts in the restored video.
4.3 Qualitative and Quantitative Improvements on Existing Methods.
Qualitative Real-world Cases. Our method, ConVRT, achieves notable temporal consistency in videos distorted by real atmospheric turbu- lence. As shown in Figure 5 the original tur- bulence and bascline methods exhibit "zig-zag" KLT tracking trajectories, indicating erratic mo- tion caused by turbulence. In contrast, incor- porating ConVRT results in smoother trajecto- ries, demonstrating its effectiveness in consis- tently removing turbulence artifacts throughout the video. The x-I slice further illustrates that Con VRT effectively smooths row pixel motion over time, reducing the flickering effects typi-
Table 3: Ablation Study of Ltemp and Tres- Com-
parison of PSNRimg. SSIM, and PSNR,- scores, showing the impact of Laemp and Tres. The exper- iment is conducted on a synthetic dataset created using turbulence simulator The base model TurbNet.
Method TurbNet
T...
Litepp
PSNR
SSIM
PSNR,
22.57
0.673
24.20
ConVVR
15
23.29
0.679
24.86
ConVRT
8
23.91
0.694
25.51
ConVRT
24.16
0.701
26.02
ConVRT
5
24.31
0,709
26.05
8

---

GT Frame
Canonical Image
GT
LQ
TurNett
Ours
Tres
wio L temp
Zoomed Y-1 Slice
Figure 7: Ablation study and canonical image visualization. Our method mitigates residual turbulence using Lump and lower Tres Canonical image is visualizvisualized from Canonical Spatial Feature Map C.
GT Frame
Captured
Captured
DATUM
DATUM-OUrS
Brownian Motion for Camera Shake
Camera Only Zoomed Slice
Shake Turbulence
Zoomed Y-L Slice
Figure 8: Illustration of camera shake simulation using Brownian Motion. In the Y-t slice plots, we observe similarities between camera shake and turbulence. The plots also demonstrate the effectiveness of our approach in handling both camera shake alone and in combination with turbulence.
Original
Li
DATUM Ours
Original
Mao et
ei
DATUM
Figure 9: Experimental results compare CRn with unsupervised and test-time optimization methods by et al. (10) and Mao et al. (9) on moving objects. Both baselines fail to capture motion, replacing moving parts with the average background, while ConVRY effectively handles them
cally observed in crrbulence-distortee videos. This improved performance underscores the capability of CRnVRT to handle real-world turbulence, providing more stable and visually coherent video sequences.
Qualitative Synthetic Cases. Similarly, synthetic video, As shown in Figure our method enhances temporal turbulence removal when applied to baseline methods. The video dynamics gener- ated by CRnVRT closely resemble the ground-truth videos, effectively smoothing out atmospheric turbulence. The improved KLT trajectories further demonstrate this temporal consistency.
Quantitative Results. We evaluated the performance of our proposed VRT method across real- world datasets containing both static and dynamic scenes, shown Table 2 consistent improvements across models and most of datasets, underscoring its broad applicability. On the HcatChamber dataset, which provides real-world paired data through a controlled heating mechanism, we calculated PSNR values to further substantiate Con VRT's effectiveness. on VRT consistently improves PSNR, demonstrating robust enhancement of temporal consistency, especially given PSNR's sensitivity to pixel misalignment.
ConVRT demonstrates
4.4 Ablation Study
Regularized temporal resolution Ties is critical for ensuring temporal consistency. Lowering it results smoother transitions but loses fine details, while higher value preserves details but increases the risk of flickering. We conducted an ablation study on the impact of Tres and Teemp- shown Table with qualitative results Figure 7 These results demonstrate the effectiveness of our representation field design in regularizing irregular turbulence motion.

---

First Frame
Patch
End Frame
Slice
Synthetic Turbulence
Original
Original ours
DATUM
DATUM ours
Figure 10: Mitigation capability of our method without using base restoration methods for pre- processing. The scene is an elephant raising its head.
4.5 Analysis
Why It Works. Our method's effectiveness stems from key factors. First, it leverages the distinct differences in motion patterns, particularly the regularity of optical flow directions within a short time window, between regular object movement and atmospheric turbulence, as discussed Section [.2 This distinction also enables our method to handle camera shake, which shares similar irregular patterns with turbulence. The results of this capability are illustrated in Figure Second, our method includes robust video representation that overcomes limitations in unsupervised methods for static scenes. As illustrated Figure these methods often struggle on moving objects, blending them into static backgrounds. In contrast, our approach preserves object integrity across frames, making it well-suited for video-based turbulence mitigation.
Mitigation Capability without Base Restoration Methods. Even without a base restoration method provide partially restored frames, our approach could improve temporal consistency, shown in Figure I0 However, we recommend combining our method with other restoration techniques. This allows to benefit from the sharpness improvements offered by supervised methods, while also taking advantage of the temporal consistency improvements provided by ConVRT.
Visualizing Trainable Feature Maps. We visualize the canonical image by inputting the canonical spatial feature map into the content MLP without applying Az and Ay. as shown the Figure The canonical image contains most of the video's content, providing base representation from which other frames can be derived. Consequently, the canonical spatial field in our video representation functions similarly key frame video compression, serving reference for other frames in the sequence to query information.
Limitations
While Con VRT offers significant improvements in video atmospheric turbulence mitigation, there are two limitations. First, as neural representation method, Con performance depends on accurate video representation and currently optimized to capture motion with precision short clips. Extending this to longer sequences and more complex motions is potential area for future exploration. Second, Con VRT processes a 25-frame video at 540x540 resolution in approximately 10 minutes, including DATUM as base method. Although much faster than Mao's 's (165 minutes) and Li's (300 minutes) methods, there still room for improving computational efficiency, especially for larger complex sequences.
6 Conclusion
In this paper, we present CRn, a novel approach aimed at enhancing temporal consistency in video ATM tasks. VRT uses dual-field Paproachh-Temorral Deformation Field and Spatial Content Field-to accurately capture spatial information while regularizing temporal information, focusing on regular object motion rather than irregular turbulence. Combined with any ATM method, Con VRT leads to visibly improved temporal consistency.
10

---

Acknowledgment. H.C., M.X., K.Z., and C.A.M. were was supported in part by AFOSR Young Investigator Program Award no. FA9550-22-1 12-00-22--- ONR award no. N000142312752, and NSF CAREER Award no. 2339616. W.J. and A.V. were supported in part by ONR award no. N00014-23- 1-2714.
References
[1] S. H. Chan, "Tilt-then-blur or blur-then-tiit?? clarifying the atmospheric turbulence model," IEEE Signal Processing Letters, vol. 29, pp. 1837,2022 2022.
[2] X. Zhang. N. Chimitt, Y. Chi, Z. Mao, and S. H. Chan, "Spatio -emppral turbulence mitigation: A translational perspective." arXiv preprint arXiv:2401.064
2024.
[] X. Zhang, Mao, Chimitt, and H. Chan, "Imaging through the atmosphere using turbulence mitigation transformer," IEEE Transactions on Computational Imaging, vol. 10, pp. 115-128, 2024.
[4] D. Jin, Y. Chen, Y. Lu, J. Chen, P. Wang, Z. Liu, S. Guo, and X. Bai, "Neutralizing the impact of atmospheric turbulence complex scene imaging via deep learning," Nature Machine Intelligence, vol. 3, no. 10, pp. 876-884, 2021.
Z. Mao, A. Jaiswal, Z. Wang, and S. H. Chan, "Single frame atmospheric turbulence mitigation: benchmark study and a new physics-inspired transformer model," in European Conference on Computer Vision, pp. 430-446, Springer, 2022.
[6] A. Jaiswal, X. Zhang, S. H. Chan, and Z. Wang. "Physics-driven turbulence image restoration with stochastic refinement," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12170-12181, 2023.
[7] X. Zhang. Mao, Chimitt, and H. Chan, "Imaging through the atmosphere using turbulence mitigation transformer," arXiv preprint arXiv:2207.06466 2022.
[8] R. K. Saha, D. Qin, N. J. J. Ye, and S. Jayasuriya, "Turb-seg-res: segseettt pipeline for dynamic videos with atmospheric turbulence." arXiv preprint arXiv:2404. 2024.
[9] 7. Mao, N. Chimitt, and S. H. Chan, "Image reconstruction of static and dynamic scenes through anisopla- natic turbulence," IEFE Transactions on Computational Imaging, vol. pp. 1415-1428, 2020.
[10] N. Li, Thapa, C. Whyte, W. Reed, S. Jayasuriya, and J. Ye, "Unsupervised non- non-rigid image distortion removal via grid deformation,"" in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2522-2532, 2021.
[11] B. Y. Feng, M. Xie, and C. A. Metzler, "Turbugan: An adversarial learning approach to eratially-varyinnn multiframe blind deconvolution with applications to imaging through turbulence," IEEE Journal on Selected Areas Information Theory, vol. no. 3, pp. 543-556, 2022.
[12] Jiang, Boominathan, and A. Veeraraghavan, "Nert: Implicit neural representations for unsupervised atmospheric turbulence mitigation," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4235 4242, 2023.
[13] D. Lao, C. Wang, A. Wong, and S. Soatto, "Diffeomorphic template registration for atmospheric turbulence mitigation." arXiv preprint arXiv:2405.03662. 2024.
[14] N. Chimitt and S. H. Chan, "Simulating anisoplanatic turbulence sampling correlated zernike coefli- cients," 2020 IEEE International Conference on Computational Photography (ICCP), pp. 1-12, IEEE, 2020.
[15] 7. Mao, N. Chimitt, and S. H. Chan, "Accelerating atmospheric turbulence simulation via learned phase- to-space transform," in Proceedings of the IEFF/CVE International Conference Computer Vision, pp. 147,9-14768,, 2021.
[16] N. Chimitt, Zhang, Mao, and S. H. Chan, "Real-time dense field phasc-to-space simulation imaging through atmospheric turbulence," IEEE Transactions on Computational Imaging, vol. 8, pp. 1159-1169, 2022.
[17] N. Chimitt and S. Chan, "Anisoplanatic optical turbulence simulation for continnnts c n 2 profiles without propagation." " Optical Engineering, vol. 62, no. pp. 078103 078103, 2023.
11

---

[18] N. Chimitt, X. Zhang. Y. Chi, and S. H. Chan, "Scattering and gathering for spatially varying blurs," IEEE Transactions on Signal Processing, 2024.
[19] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[20] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan, U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng, "Fourier features let networks Iearn high frequency functions in low dimensional domains," " Advances in Neural Information Processing Systems, vol. 33, pp. 7537-7547, 2020.
[21] Mehta, M. Gharbi, C. Barnes, E. Shechtman, R. Ramamoorthi, and M. Chandraker. "Modulated periodic activations for generalizable local functional representations," in Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 14214-14223, 2021.
[22] B. Y. Feng and A. Varshney, "Signet: Efficient neural representation for light fields," in Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 14224-14233, 2021.
[23] B. Y. Feng. S. Jabbireddy, and A. Varshney, "Viinter: View interpolation with implicit neural representations of images," in SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
[24] B. Feng and A. Varshney. "Neural subspaces for light fields." IEEE Transactions on Visualization and Computer Graphics, 2022.
[25] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphics primitives with a multiresoluution hash encoding," ACM Transactions on Graphics (ToG), vol. 41, no. 4. pp. 1-15, 2022.
[26] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, "Implicit neural representations with periodic activation functions," Advances neural information processing systems, vol. 33, pp. 7462-7473, 2020.
[27] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, "Deepsdf: Learning continuous signed distance functions for shape representation." in Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. (65-174, 2019.
[28] B. Y. Feng. Y. Zhang, D. Tang, R. Du, and A. Varshne,, "Prif: Primary ray-based implicit function," in European Conference on Computer Vision, pp. 138-155, Springer, 2022.
[29] Qadri, K. Zhang, Hinduja, M. Kaess, A. Pediredla, and A. Metzler, "Aoneus: A neural rendering framework for acoustic-optical sensor fusion," in ACM SIGGRAPH 2024 Conference Papers, pp. 1 1-12, 2024.
[30] M. Xie, H. Cai. S. Shah, Y. Xu, B. Y. Feng. J.-B. Huang, and C. A. Metzler, "Flash-splat: 3d reflection removal with flash cues and gaussian splats," in European Conference Computer Vision. pp. 139. Springer, 2025.
[31] H. Alzayer, K. Zhang, Feng, C. A. Metzler, and J.-B. Huang, "Seeing the world through your eyes." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4864-4873, 2024.
[32] S. Shah, M. A. Chan, H. Cai, J. Chen, S. Kulshrestha, C. D. Singh, Y. Aloimonos, and C. A. Metzler, "Codedevents: Optimal boint--pread--uneetee function engineering for 3d-tracking with event cameras," Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[33] Н. Zhou, B. Y. Feng, Guo, M. Liang, C. A. Metzler, C. Yang, el al., "Fpm-inr: Fourier ptycho- graphic microscopy image stack reconstruction using implicit neural representations," arXiv preprint arXiv:2310. 18529, 2023.
[34] H. Wang and I.. Tian, "ooaa conditional neural fields for versatile and generalizable large-scale recon- structions in computational imaging," " arXiv preprint arXiv:2307. 006007, 2023.
[35] B. Y. Feng, H. Guo, M. Xie, V. Boominathan, M. K. Sharma, Veeraraghavan, and C. Metzler, "Neuws: Neural wavefront shaping for guidestar- imaging through static and dynamic scattering media," Science Advances, vol. 9, no. 26, eadg4671, 2023.
[36] E. Y. Lin, Z. Wang, R. Lin, D. Miau, F. Kainz, J. Chen, X. C. Zhang, D. B. Lindell, and K. N. Kutulakos, "Learning lens blur tields." arXiv preprint arXiv:2310.11555. 2023.
12

---

[37] E. Bostan, R. Heckel, M. Chen, M. Kellman, and L. Waller. "Deep phase decoder: self-calibrating phase microscopy with untrained deep neural network." Optica, vol. 7, no. 6, pp. 559-562, 2020.
[38] M. Xie, H. Guo, B. Y. Feng, L. Jin, A. Veeraraghavan, and C. A. Metzler, "Wavemo: Learning wavefront modulations to see through scattering." in Proceedings of the IEEE//VF Conference on Computer Vision and Pattern Recognition, pp. 25276-25285, 2024.
[39] Z. Li, Q. Wang. Cole, R. Tucker, and N. Snavely, "Dynibar: Neural dynamic image-based rendering," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 427344444444 4284, 2023.
[40] Q. Wang, Y.-Y. Chang, R. Cai, Li, B. Hariharan, A. Holynski, and N. Snavely, "Tracking everything everywhere all at once," arXiv preprint arXiv:2306.05422, 2023.
[41] B. Y. Feng, H. Alzayer, M. Rubinstein, W. T. Freeman, and J.-B. Huang, "3d motion magnification: Visual- izing subtle motions from time- varying radiance fields," in Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 9837-9844 2023.
[42] H. Ouyang, Q. Wang, Y. Xiao, Q. Bai, Zhang, K. Zheng, X. Zhou, Q. Chen, and Y. Shen, "Codef: Content deformation fields for temporally consistent video processing." arXiv preprint wxXv:2308.00996. 2023.
[43] Y. Kasten, D. Ofri, O. Wang, and T. Dekel, "Layered neural atlases for consistent video editing," ACM Transactions on Graphics (TOG), vol. 40, no. 6, pp. 1-12, 2021.
[44] Y-C. Lee, J.ZZ. G. Jang, Y.-T. Chen, E. Qiu, and J.-B. Huang, "Shape-aware text-driven layered video editing." " in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14317-14326, 2023.
[45] V. Ye, 7. Li, R. Tucker, A. Kanazawa, and N. Snavely, "Deformable sprites for unsupervised video decomposition." in Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2657-2666, 2022.
[46] D. L. Fried, "Probability of getting lucky short-exposure image through turbulence." " JOSA, vol. 68, no. 12. pp. 1651-1658, 1978.
[47] R. J. Noll, "Zemike polynomials and atmospheric turbulence," JOsA, vol. 66, no. 3, pp. 207-211, 1976.
[48] T. Caliskan and N. Arica, "Atmospheric turbulence miligation using optical flow," 2014 22nd Interna- tional Conference on Pattern Recognition, pp. 883- 888, Iese, 2014.
[49] M. Shimizu, S. Yoshimura, M. Tanaka, and M. Okutomi, "Super-resolution from image sequence under influence of hot-air optical turbulence," " in 2008 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, IEEE, 2008.
[50] J. Gilles, T. Dagobert, and C. De Franchis, "Atmospheric turbulence restoration by diffeomorphic image registration and blind deconvolution," in Advanced Concepts for Intelligent Vision Systems: 10th Interna- tional Conference, ACIVS 2008, Juan-les- -iin-,,s-Pins, France, October 20-24, 2008. Proceedings 10, pp. 400 409, Springer, 2008
[51] N. Anantrasirichai, A. Achim, and D. Bull, "Atmospheric turbulence mitigation for sequences with moving objects using recursive image fusion," in 2018 25th IEEE international conference on image processing (ICIP), pp. 2895-2899, IEEE, 2018.
[52] J. Liang, J. Cao, Y. Fan, K. Zhang, R. Ranjan, Y. Li, R. Timofte, and Van Gool, "Vit: A video restoration transformer," IEEE Transactions on Image Processing, 2024.
[53] K. C. Chan, X. Wang, K. Yu, C. Dong, and C. C. Loy, "Basicvsr: The search for essential components in video super-resolution and beyond," in Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 14474956, 2021.
[54] H. Chen, J. Ren, Gu, H. Wu, X. Lu, H. Cai, and L. Zhu, "Snow removal in video: A new dataset and novel method," in 2023 /EEE/CVF International Conference on Computer Vision (ICCV), pp. 13165-13176, IEEE, 2023.
[55] X. Zhang, H. Dong, J. Pan, C. Zhu, Y. Tai, C. Wang. J. Li, F. Huang, and F. Wang. "Learning to restore bazy video: A new world dataset and a new method," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9248, 2021.
13

---

[56] C. Lei. Y. Xing, and Q. Chen, "Blind video temporal consistency via deep video prior," Advances in Neural Information Processing Systems, vol. 33, pp. 1083-1093, 2020.
[57] C. Lei, Y. Xing, H. Ouyang. and Q. Chen, "Deep video prior for video consistency and propagation." IEEE Transactions Pattern Analysis and Machine Intelligence, vol. 45, no. pp. 356 371, 2022.
[58] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su, "Tensorf: Tensorial radiance fields," in European conference computer vision, pp. 333-350, Springer, 2022.
[59] S. Fridovich-Keil, G. Mcanti, F. R. Warburg, B. Recht, and A. Kanazawa, "K-planes: Explicit radiance fields in space, time, and appearance," in Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 12479-12488, 2023.
[60] R. Ranftl, K. Lasinger, D. Hafner, K. Schindler, and V. Koltun, "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer," IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 3. pp. (623-1637, 2020.
[61] Z. Wang, A. C. Bovik, R. Sheikh, and Simoncelli, "Image quality assessment: from error visibility to structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612, 2004.
[62] R. Zhang, Isola, A. A. Efros, F. Shechtman, and O. Wang. "The unreasonable effectiveness of deep features as a perceptual metrie," in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 595.595.222 2018.
[63]
J. Gilles and N. B. Ferrante, "Open turbulent image set (otis)," Pattern Recognition Letters, vol. 86, pp. 38-41, 2017.
[64] N. Anantrasirichai, "Atmospheric turbulence removal with complex-valued convolutinall neural network," Puttern Recognition Letters, vol. 171, pp. 69-75, 2023.
[65] D. Qin, R. K. Saha, W. Chung, S. Jayasuriya, J. Ye, and N. Li, "Unsupervised moving object segmentation with atmospheric turbulence," in European Conference Computer Vision, pp. 18-37, Springer, 2025.
[66] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint 1412:v1412.6980) 2014.
[67] W.-S. Lai, J.B.. Huang, O. Wang, F. Shechtman, E. Yumer, and M.-H. Yang, "L.carning blind video temporal consistency," in Proceedings of the European conference computer vision (ECCV), pp. 170-185, 2018.
[68] M. Ruder, A. Dosovitskiy, and T. Brox, "Artistic style transfer for videos." in Pattern Recognition: 38th German Conference, GCPR 2016, Hannover, Germany, September 12-15, 2016, Proceedings 38, pp. 26 36. Springer, 2016.
[69] Z. Li, R. Tucker, N. Snavely, and A. Holynski, "Generative image dynamics," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14142-24155, 2024.
[70] B. D. Lucas and T. Kanade, "An iterative image registration technique with application to stereo vision." in IJCAI'81: 7th international joint conference Artificial intelligence, vol. 2, pp. 674-679, 1981.
A Appendix / supplemental material
A.1 MLP Network Details
Our network architecture consists of two MLPs: a content MLP and a deformation MLP. The content MLP has fully-connected layers, with an input dimension Q1 hidden dimension of 128, and an output dimension of 2, representing Az and Ay. The deformation MLP comprises 6 fully-connected layers, with an input dimension of Q2, a hidden dimension of 256, and an output of 3 channels representing RGB intensity.
A.2 Position Encoding
Position encoding for spatial and temporal indices is embedded within the trainable feature map, as these indices are trainable. We directly use y. and to query the corresponding feature tensors from the feature maps. Notably, the temporal feature map, neighboring features are shared across multiple frames, with each frame weighted differently due explicit regularization.
14

---

First Frume
Patch
Now exiting
End Frame
comperaiin
Slice
y
Reul Turbulence
Original
Original
DATUM
DATUM
Figure 11: Mitigation capability of our method without using base restoration methods for pre- processing,
A.3 Additional Results Mitigation Capability without Base Restoration Methods.
Additional results highlighting our method's mitigation capability independently of base restoration techniques are presented in Figure I]
15

---

NeurIPS Paper Checklist
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope?
Question:
Answer:[Yes Yes]
Justification:
As shown in sections I 2 S
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section discusses the limitation our method
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

This paper does not provide theoretical results. Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
This paper provides detailed explanation of proposed method and experimental
setup in sections 4 and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: We have a website to host all open-source resources, including code, data, and results. The website link is attached to the abstract of the main paper. We will organize code base and make it easy for users to use.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
public/guidess Sotiddde/oodeSuuissss..oooo oolliicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //ipps.ccppblic//uu.dde sss ssinisolicyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All those experiment details are reported in section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We didnt include error bars calculation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
the resource needed for our work
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research is conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
[ discusses the positive influence this work could bring related research fields. For social-wise impact, the influence is difficult for us to figure out.
Justification:
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work doesn't 't involve any possibility of being misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets used in this paper are open-source.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All related documents well-studied.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Justification: We don't have experiments related to human subject.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We don't have experiments related to human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "N5tYR5Hcf2", "reviewer": "Reviewer_DNNj", "summary": "This paper presents an implicit neural representation (INR) framework for taking a pre-trained supervised video atmospheric turbulence mitigation (ATM) model and regularizing its output to be more temporally consistent. The main components are (1) an INR called the temporal deformation field; and (2) a subsequent INR called the spatial content field to output RGB intensity at a pixel in the video (at a certain time). These two INRs are trained on the output of a pre-trained ATM model, and regularized using a disparity loss (with MiDas pre-trained network) for temporal consistency, and a similarity loss for the content of the video. Experiments are conducted on real-world datasets with comparison to state-of-the-art ATM models recently proposed as well as some simulated ablation studies.", "questions": "1. Can the authors explain (1) how the feature maps M, N, C are novel compared to other INR video representations? Is it the use of the Hadamard product, and the two stage architecture? If so, an ablation study of comparing the Hadamard product of M and N to just a 3D spatial feature map directly is warranted. (2) I would be good to visualize these features after optimization (what do they look like), and what information/interpretability can be gleaned from them? \n\n2. There are missing details: what is the details of the MLP layers, any positional encoding? These should be added to a supplemental file. \n\n3. I am interested if L_temp is the key factor that improves the method, or its a minor improvement. Showing a qualitative example as discussed earlier would be beneficial here. \n\n4. Can there be a discussion about issues involving camera shake? I assume the method would require stabilized videos first, and if there is any residual motion leftover, this would cause errors in the reconstruction. \n\n5. What is the wall clock time of the method? How long does it take (in actual seconds) from start to finish? The paper only states 80 epochs, I'm curious how long that actually takes. \n\n6. In line 134 - it says an additional enhancement module is applied after S_field. This was never discussed again. How important is this module? What's the performance with and without the module?", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ Method can improve a variety of existing state-of-the-art ATM models, and the use of INRs with different feature representations used as inputs seems like an original contribution (at least to this application field, if not video representation in general)\n\n+ The use of KLT tracker for visualizing the temporal variability across the frames is a good visualization and helps show the improvement of the method in a qualitative way\n\n+ Supplemental videos on the website show the method stabilizing video in the presence of turbulence\n\n+ Extensive quantification of the method shown in Table 2 to illustrate the effectiveness of ConVRT", "weaknesses": "- There is little detail about the spatial feature map M, temporal feature map N, canonical spatial feature map C. What does it mean to call these feature maps, and why are they chosen the way they are? For instance, why the Hadamard product for M and N, and not just learning a 3D feature map at that place instead directly? I also don't see how the C map is \"canonical\" to me in any obvious way (for instance, you could change the dimensions of Q1, Q2 and I don't see why that couldn't work in the method?). \n\n- The method seems to be focused primarily on fixing errors for supervised ATM methods. However, some of the classical approaches such as [Mao 2020] that utilize lucky frames, would they have this problem of temporal variability? I'm not necessarily asking for a comparison or new experiments, but it would be good to discuss if this problem primarily is for supervised methods. \n\nReference: Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan, ‘‘Image Reconstruction of Static and Dynamic Scenes through Anisoplanatic Turbulence’’, IEEE Transactions on Computational Imaging, vol. 6, pp. 1415-1428, Oct. 2020\n\n- Table 3 is a very modest improvement . Does the Ltemp really help? A qualitative example would really help clear up that this L_temp is working (show a figure with and without Ltemp).\n\n- How would the method handle issues such as camera shake (common in long-range videos that are shot with high optical zoom)? \n\nMinor suggestions:\n- Line 187 - TurbNet, shouldn't it be whatever ATM method you are comparing with? \n- Line 205 - One shouldn't be capitalized\n- Line 109 - unresolved citation\n- Table 1 - more stylistic, but I don't think its necessary to put down the venue into the table. We shouldn't judge methods based on their venue, but on the content of the method itself, so having the citation alone is enough to let readers draw their own conclusions about the papers. I would remove this column from the table. \n- Table 3 - there is a typo in the PSNR_Img column where the lower number is bolded rather than the higher one", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**`R1-Q1 / R1 - weakness 1`** : **Visualization of Canonical Spatial Field and Representation Field Design**\n\nThe canonical spatial field C serves as a base spatial representation, containing all spatial content of the video. We can obtain a canonical image by deriving it from this field without applying delta x and y. This canonical spatial field functions similarly to a key frame in video compression, providing a reference from which other frames in the sequence can be derived. For more detailed results and discussion on this topic, please refer to our global response to **`shared question D`** \n\nOur feature maps M and N, combined with the Hadamard product, provide an efficient representation strategy for 3D feature volumes represented by INR. This tensor decomposition strategy is commonly used to parameterize 3D volumes in INR, enhancing their ability to represent 3D signals while reducing the number of required parameters [1][2][3]. The Hadamard product in our two-stage architecture efficiently combines spatial and temporal information, offering a balance between computational efficiency and representational power, particularly suited for video ATM tasks.\n\n\n**`R1-Q3 / R1 - weakness 3`**: **More ablation study on loss function**\n\nBoth representation and loss design are effective in our method. The neural representation design plays a more crucial role, bringing a 0.95dB improvement in performance, while the L_temp loss contributes an additional 0.15dB improvement, as demonstrated in our ablation study. For more details, please refer to our response to **`R2-Q3`**.\n\n\n**`R1-Q4 / R1-weakness 4`** : **Camera shake**\n\nThank you for the interesting question. Please see our global response to **`shared question B`**.\n\n\n**`R1-Q5`**: **Wall clock time**\n\nWe conducted experiments on a 25-frame video with 540x540 resolution. Our total running time consists of two parts: the preprocessing using the ATM model (DATUM) and our test-time optimization approach. For comparison, the running time for other unsupervised test-time optimization methods (Li's [2] and Mao’s [1] methods) only includes the methods themselves, as they do not rely on preprocessing.\nAs shown in the **`Table`** below, even with the additional DATUM preprocessing time, our method is more than 15 times faster than Mao’s method and around 30 times faster than Li's. Our method's efficiency stems from combining the advantages of supervised models with test-time optimization. We use faster supervised methods to address major issues in video turbulence mitigation and apply the slower test-time optimization only to residual temporal inconsistencies. This approach makes our method significantly more efficient than pure test-time optimization methods that must solve the entire problem independently.\n\n\n| Method             | Running Time (min) |\n| :----------------: | :------:     |\n| Mao                |  165         |\n| Li               |  300         |\n| DATUM + Ours         |   10         | \n\n\n**`R1-Q2`**: **Missing details about MLP** \n\nOur network architecture consists of two MLPs: a content MLP with 3 fully-connected layers (input: Q1, hidden: 128, output: 2) and a temporal MLP with 8 fully-connected layers (input: Q2, hidden: 128, output: 3 RGB). We encode position information in the optimizable Canonical Spatial Feature Map C and Spatial Feature Map M, both with dimensions H x W matching the image. For each RGB image location, we query the corresponding Spatial Feature Map M location, perform a Hadamard product, and input to the MLP. In the Temporal Feature Map, multiple frames share neighboring features with different weights due to explicit regularization. We will provide additional illustrations in the revision to further clarify these concepts.\n\n**`R1-Q6`**: **The missing enhanced module**\n\nThe “enhanced module” refers to a network feature that ablation studies caused us to remove. Referencing it in the manuscript was a typo—apologies for the confusion. This reference will be removed.\n\n\n**`R1 - weakness 2`**: **Add comparison with Mao’s method**\n\nOur method substantially outperforms Mao et al. [4] and [5] on dynamic video content. This is because their approaches are only effective for static scene turbulence removal and fail when applied to videos containing moving objects. For qualitative comparison and discussion, please see our global response to **`shared question C`**\n\n**`R1 - weakness 6`**: **Term involved in the loss function** \n\nAt line 187, the loss is calculated between the output of our method and the output of the base method (TurbNet, TMT, DATUM, etc.). We will clarify this description in the camera-ready version.\n\n**`R1 - weakness 5/7/8/9`**: **Typos**\n\nThank you for pointing out the typos. We will fix them in the camera-ready version.\n\n\n[1] Chen, Anpei, et al. \"Tensorf: Tensorial radiance fields.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Fridovich-Keil, Sara, et al. \"K-planes: Explicit radiance fields in space, time, and appearance.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[3] Zhou, Haowen, et al. \"Fourier ptychographic microscopy image stack reconstruction using implicit neural representations.\" Optica 10.12 (2023): 1679-1687.\n\n[4] Mao, Zhiyuan, Nicholas Chimitt, and Stanley H. Chan. \"Image reconstruction of static and dynamic scenes through anisoplanatic turbulence.\" IEEE Transactions on Computational Imaging 6 (2020): 1415-1428.\n\n[5] Li, Nianyi, et al. \"Unsupervised non-rigid image distortion removal via grid deformation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OIcW9k1tA7", "reviewer": "Reviewer_EL8T", "summary": "This paper introduced ConVRT, a novel method for video atmospheric turbulence mitigation. \nThis paper has a good structure and is well-written.", "questions": "1. Test cases with visualization are so limited, even in the supplemental material. Please show more real-world cases to show the effectiveness of the algorithm.\n2. As the representative unsupervised method, \"Unsupervised non-rigid image distortion removal via grid deformation,\" it is important to compare with it, no matter from the algorithm or qualitative result.\n3. Lack of ablation studies to prove the effectiveness of the proposed algorithm.\n4. Can the method deal with the video with moving objects?\n5. No matter whether it is for a single image-based or multiple frames-based turbulence mitigation, most existing algorithms can deal with them very well. With the help of the diffusion model, the resulting image can be refined further. It means that it could generate a good final result in a short time.  Then, what is the advantage of your algorithm?\n6. It is important to visualize the temporal and spatial field to verify the algorithm's effectiveness.", "limitations": "Same with question", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper proposed a new method to deal with turbulence mitigation.", "weaknesses": "1. limited real-world case visualization\n2. limited proof of algorithm effectiveness\n3. limited comparison with classic algorithm", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**`R2-Q1 & R2-Q4`**: **Capability of moving object on more cases**\n\nYes, our method can handle moving objects. More than half of the cases in our main paper and supplementary materials are dynamic videos. This is evident as the lines in X-t Slice or Y-t Slice are not perfectly vertical or horizontal, indicating the objects are moving along the temporal dimension. Please also view the html file in the supplement which includes video reconstructions of our scenes.\n\nWe have also added additional experimental results, with large object motion, in our global response to **`shared question A`**. Our approach can effectively reconstruct scenes with significant object motion.\n\n**`R2-Q2`**:  **Comparison with Li’s method** \n\nWe have added comparisons with Li's [1] in our global response to **`shared question C`**.  Our approach handles videos with moving objects, whereas Li's method cannot.\n\n[1] Li, Nianyi, et al. \"Unsupervised non-rigid image distortion removal via grid deformation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n**`R2-Q3`**: **More ablation study.**\n\nWe conducted an ablation study on temporal resolution (T) and the presence of L_temp, as shown in **`Table below`**, with corresponding qualitative results in **`attached file Figure 4`**.\n\nThe first row of the Table shows TurbNet's performance. Equipping TurbNet with our method yields a 0.76 dB improvement in PSNR, demonstrating the effectiveness of our representation field design in regularizing irregular turbulence motion. Decreasing T from 15 to 5 (implying stronger regularization) further improves PSNR by 0.94 dB, SSIM by 0.03, and PSNR_slicext by 1.28 dB. Adding our proposed loss function achieves additional performance gains. \n\n**`Attached file Figure 4`** illustrates qualitative results. Removing the proposed loss function and relaxing regularization leads to noisier and more jittery X-t slices compared to our full method\n\n\n| Method Name | Resolution of T| Our Feature Maps  | L_temp | PSNR | SSIM | PSNRx-t |\n|--------------|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|:------------------:|\n| TurbNet      |                    |               |        |    23.46  |   0.68   |   26.08 |\n| Ours            | 15               | ✔️            |        |    24.22  |   0.69   |   26.63 |\n| Ours            | 8               | ✔️            |        |    24.88  |  0.70    |   27.46 |\n| Ours            | 5               | ✔️            |        |   25.16   |   0.72   |   27.91 |\n| Ours            | 5               | ✔️             | ✔️      |  25.29    |   0.73   |   27.99 | \n\n\n**`R2-Q5`**: **Advantage of the proposed method over diffusion model.**\n\nOur method in video ATM offers several advantages over diffusion models: It is generalizable, physics-grounded, temporally consistent, and free from hallucinations.\n\nDiffusion models can learn incredibly accurate priors on images and, more recently, videos. Long term, they could be an effective method to mitigate turbulence. However, they also come with severe limitations. Diffusion models require extensive training data and often suffer from hallucination biases inherent in that data. These models are purely data-driven and do not consider domain knowledge or the physics of turbulence. In contrast, our method addresses residual temporal inconsistency issues from existing physics-based turbulence removal models. By observing turbulence motion features, we design the mitigation based on temporal motion regularity, achieving effective results without any training. Moreover, our method can be applied to any existing physics-based ATM method, making it more grounded in the physics of turbulence rather than relying solely on training data quality.\n\nTo date, there has been no exploration of diffusion models for video-based ATM. The only diffusion model method, PiRN, for image-based ATM is a closed-source approach trained on limited synthetic image datasets. There is no evidence it can reconstruct temporally consistent dynamic videos. In contrast, our method significantly improves temporal consistency. \n\n**`R2-Q6`**: **Visualization of the temporal and spatial field.** \n\nPlease see our global response to the **`shared question D`**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "L8Zb0wdSL1", "reviewer": "Reviewer_7Q3b", "summary": "This paper proposes a method for improving the temporal consistency of turbulence-affected videos. The proposed method uses neural representations (MLP layers) to separately model the spatial and temporal deformations caused by air turbulence and is able to improve the temporal consistency of restoration results. It seems that the proposed method needs to be used in conjunction with another base turbulence restoration method. The prosed method (by combining with several other SOTA approaches) is evaluated on existing real turbulent video dataset, and a small dataset collected by the authors. Temporal consistency, especially for videos with moving objects, is apparently improved after applying the proposed method.", "questions": "- It seems that motions in demonstrated results are quite slight and slow. Is the method also robust to large motions? How is this related to turbulence strength? \n- I'm interested at knowing more details about the airport dataset acquired by the authors, like size of the dataset, type of scenes, turbulence strength, etc.", "limitations": "Two limitations are discussed: 1. length of input video cannot be too long, and 2. processing time is sort of long (running time of the method is not reported). Turbulence strength and motion scale could be also discussed, if they are limiting factors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written and easy to follow.\n- The MLP-based network for mitigating spatial and temporal deformations is new and seems to be effective, especially on maintaining temporal consistency. \n- Qualitative evaluation is performed on videos with moving objects and have shown apparent improvement.\n- A small real-world dataset was captured and used for testing, but details about the dataset are not clear.", "weaknesses": "- It seems that the method is an add-on approach for regularizing the temporal consistency of videos restored by another turbulence restoration methods. Since turbulent degradation is more complex than temporal inconsistency and spatial distortions (e.g., there might be blurriness and color aberration beyond the spatial and temporal deformations), being able to only handle these two types of artifacts seems quite limited for turbulence mitigation. \n- The quantitative evaluation results (Table 2) are confusing. This table shows metric scores for using the proposed method as a stand-alone (i.e., using the original turbulent video as input). However, the \"no base results\" are not demonstrated in any visual comparison figures (not even in supplementary materials). It would be useful to see visual comparison between \"no base results\" and others. What really confuses me about this table is that the metric scores of the original turbulent images (denoted as \"ori\") are even better than processed results in many cases (for example, its PSNR is higher than DATUM results for the HeatChamber dataset, and there many such cases). But according to visual results, most processed results have apparent improvement. Besides, after applying the proposed method, some metric scores become much worse (for example, the slice_tv scores for most datasets and method combinations). There should be some discussions explaining the metric results. \n- The paper missed some relevant prior works (see below). These works either use MLP for modeling turbulent distortions or use similar idea to enforce temporal consistency and they should be discussed and compared with the proposed method.\nLi et al., \"Unsupervised Non-Rigid Image Distortion Removal via Grid Deformation,\" ICCV 2021.\nThapa et al. \"Learning to Remove Refractive Distortions from Underwater Images,\" ICCV 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**`R3-weakness 1`**:  **Forward Model of Turbulence.**\n\nIndeed mitigating turbulence requires overcoming color aberrations, blurriness, and deformations and other effects. Fortunately, existing techniques have successfully addressed many of these challenges and arguably the largest limitation of existing approaches is that their results lack temporal consistency. Our method addresses this challenge.\n\nRecent advancements in the field, from TMT (2023) to DATUM (2024), have leveraged increasingly large datasets created by advanced simulators and more complex networks. While these improvements have significantly enhanced sharpness in real-world cases, as evidenced in **` main paper Figures 1`** and **`main paper Figure 5`** of our main paper, poor temporal consistency remains a glaring issue. This inconsistency is not only noticeable to the human eye but also detrimental to downstream tasks such as video segmentation.\n\nThis persistent problem suggests that video-based ATM methods struggle to improve temporal consistency relative to other types of degradation. The primary challenge lies in poor generalization to the random and complex deformations caused by real-world turbulence.\n\nOur method addresses this critical gap by providing an effective refinement solution for video-based ATM methods when applied to real-world videos. We believe that tackling the temporal inconsistency issue represents a significant contribution to the video ATM field at this stage, as it targets a key weakness in current state-of-the-art approaches\n\n\n**`R3-weakness 2`**: **Qualitative result of ours trained on the distorted video (No Base Method).**\n\nEven without a base method to provide partially restored frames, our approach effectively improves temporal consistency by a significant margin, as demonstrated in **` attached file Figure 1`**. For clear evidence of this improvement, please observe the X-t slice presented there.\n\n\n**`R3-weakness 2`**: **Table 2 is confusing. Why do video-based ATM methods have worse PSNR and PSNR_x-t compared to image-based methods on the HeatChamber dataset?**\n\nApologies for any confusion caused by the content in the main table. Below is a detailed explanation of our results. We will clarify in the revision.\n\nWhen reconstructing static scenes, image-based reconstruction methods have a significant advantage over video-based methods. This is because image-based methods effectively impose a prior assumption that the scene is completely static, while video-based methods do not. Consequently, it's not surprising that image-based methods perform well on the static HeatChamber dataset.\n\nWithout this strong static prior, it's understandable that video-based methods are less effective on static datasets. However, this situation reverses for dynamic scenes, where the static scene prior would be invalid. In these cases, image-based methods perform much worse than video-based methods.\n\n**`R3-weakness 2`**: **Why do some metric scores (tv_slice) appear much worse?**\n\nThis is a typo; a lower slice_tv indicates better performance. We will fix it in the camera-ready version.\n\n**`R3-weakness 3`**: **Missings comparisons two unsupervised methods**\n\nAt present, no open source implementation of [3] is available. We have provided comparisons with Mao’s [1] Li's [2] in our global response to the **` shared question C`**. Our approach handles videos with moving objects, whereas Li's and Mao's methods cannot.\n\n[1] Mao, Zhiyuan, Nicholas Chimitt, and Stanley H. Chan. \"Image reconstruction of static and dynamic scenes through anisoplanatic turbulence.\" IEEE Transactions on Computational Imaging 6 (2020): 1415-1428.\n\n[2] Li, Nianyi, et al. \"Unsupervised non-rigid image distortion removal via grid deformation.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3] Thapa, Simron, Nianyi Li, and Jinwei Ye. \"Learning to remove refractive distortions from underwater images.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n\n**`R3-Q1`** : **Capability on large motion**. \n\nYes, our method can handle moving objects with large motion. For the qualitative results, please see our global response to the **` shared question A `** .", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K23YgwPkO7", "reviewer": "Reviewer_VLLH", "summary": "The study focuses on Video Atmospheric Turbulence Mitigation (ATM), which aims to restore videos that are affected by distortions caused by atmospheric turbulence. Specifically, the proposed ConVRT introduces a neural video representation that decouples spatial and temporal information, allowing targeted regularization of the network's temporal representation capability. Also, this paper integrates supervised and self-supervised learning, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data.", "questions": "Please refer to the detailed comments!", "limitations": "Please refer to the detailed comments!", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The study introduces a novel framework called ConVRT for video ATM, which addresses the challenge of maintaining temporal consistency in turbulence mitigation. It proposes a neural video representation that decouples spatial and temporal information, allowing targeted regularization and effective mitigation of turbulence-induced temporal frequency variations. Furthermore, the study integrates supervised and self-supervised learning, improving the temporally consistent mitigation of ATM methods on diverse real-world data.", "weaknesses": "1.\tAs a neural representation method, ConVRT is designed to handle video clips with a limited number of frames, making it challenging to handle larger video sequences and more significant motions without compromising accuracy.\n2.\tThe related work section lacks completeness. More test-time methods should be carefully reviewed. Also, no SOTA methods were involved in the experimental section for comparison, which is unacceptable.\n3.\tIn addition to VRT, more SOTA transformer-based methods, as well as the recent-popular diffusion-based models, should be added for analysis.\n4.\tConVRT employs test-time optimization, which can be computationally intensive. Therefore, model efficiency should be discussed.\n5.\tSome intermediate results should be given.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**`R4-Q1`**: **Capability of handling longer video sequences and significant motion**.\n\nOur method effectively handles videos with significant motion. We've included additional experimental results and a detailed analysis in our global response to **`shared question A`**. This demonstrates our approach's robustness across various dynamic scenarios\n\n**`R4-Q2`**: **The related work section lacks completeness. More test-time methods should be carefully reviewed. Also, no SOTA methods were involved in the experimental section for comparison, which is unacceptable.**.\n\t\nWe'd like to highlight that our experimental section includes a comprehensive setup that SOTA and advanced methods in the related fields:\n\n* TMT (TCI’23) [1]: The pioneering video-based ATM method utilizing transformer architecture and temporal attention.\n\n* DATUM (CVPR’24)  [2]: The current SOTA video-based ATM method, trained on an advanced turbulence simulator dataset, achieving 10x faster inference time and superior performance compared to TMT.\n\n* TurbNet (ECCV’22) [3]: An effective image-based method trained on an advanced turbulence simulator.\n\n* VRT (TIP’24) [4]: A notable video method for general deblurring.\n\nComprehensive qualitative and quantitative results are presented in  Figure 1, Figure 5 and Table 1 of the main paper, with additional comparisons shown in **`attached file Figure 1`**. These results demonstrate our method's significant performance improvements in addressing the residual distortion (temporal inconsistency) that persists in current SOTA methods.\n\nTable 1 and subsection 1.1 in the main paper present a comprehensive taxonomy of current ATM methods, analyzing their strengths and limitations relative to our approach. \n\nWe've added two more unsupervised and test-time optimization-based turbulence removal methods for comparison and discussion in our global response to **`shared question C`**.\n\n[1] Zhang, Xingguang, et al. \"Imaging through the atmosphere using turbulence mitigation transformer.\" IEEE Transactions on Computational Imaging (2024).\n\n[2] Zhang, Xingguang, et al. \"Spatio-Temporal Turbulence Mitigation: A Translational Perspective.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[3] Mao, Zhiyuan, et al. \"Single frame atmospheric turbulence mitigation: A benchmark study and a new physics-inspired transformer model.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[4] Liang, Jingyun, et al. \"Vrt: A video restoration transformer.\" IEEE Transactions on Image Processing (2024).\n\n\n\n**`R4-Q3`** : **In addition to VRT, more SOTA transformer-based methods, as well as the recent-popular diffusion-based models, should be added for analysis**.\n\nRegarding the comparison with diffusion-based models, we've prepared a comprehensive analysis highlighting our method's unique advantages. Please see our response to **`R2-Q5`** for an in-depth discussion on this topic. \n\n**`R4-Q4`**: **Discussion about Efficiency**.\n\nWe appreciate your suggestion on our model's efficiency. Our global response to **`R1-Q5`** provides a detailed breakdown of our method's computational performance, including comparisons with other test-time optimization approaches for turbulence removal.\n\n**`R4-Q5`**: **Intermediate results**.\n\nThese results are compiled in our global response to **`shared question D`** for a clearer understanding of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
**`Shared Question A `** : **How does the proposed method handle large object motion?**

Since our method relies on moderating the temporal regularity of motion in the video, it is natural to ask whether it can distinguish between large object motion and turbulence motion. We provided additional experimental results on large motion cases from the Augmented URG-T Dataset [1], featuring videos with both strong turbulence and fast-moving objects.

As shown in **`attached file Figure 1`**, our approach remains effective for scenes with both large object motion and strong turbulence. This is evidenced by the long KLT trajectories in our restored videos, which successfully capture the original large object motion. Our approach achieves this because its MLP implicitly penalizes temporally irregular motion caused by turbulence while effectively fitting large but regular motion due to the scene itself. This behavior is illustrated in the X-t slices presented in Figure 1; turbulence motion appears irregular (jittering regions), while large object motion, despite being significant, maintains a smooth shape.

Fundamentally, the difference between large object motion and turbulence is that large object motion results in large optical flow magnitude, but the optical flow direction in a local region remains regular. In contrast, turbulence motion causes irregular optical flow directions. Our method (implicitly) allows large optical flows as long as they are locally regular in direction, while surpassing irregular optical flows in a local region. This is why our method can handle large object motion while mitigating turbulence.   

[1] Saha, Ripon Kumar, et al. "Turb-Seg-Res: A Segment-then-Restore Pipeline for Dynamic Videos with Atmospheric Turbulence." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024


**`Shared Question B `** : **What is the relationship between turbulence, camera shake, and object motion?**

As discussed in **`Shared Question A`**, in the X-t slice (a way to visualize optical flow over time), turbulence motion appears as an irregular jittering shape due to irregular optical flow direction over time, while object motion, even if large, results in a smooth shape in the X-t slice. Similarly, as shown in the X-t slice **`Figure 2 in attached file`**, camera shake also manifests as an irregular jittering in a local region. Accordingly, by mitigating irregular motion between frames our method can simultaneously compensate for both turbulence and camera shake. 

We simulated Brownian motion to represent camera shake/translation and used MiDas [2]  depth to simulate the effect of decreasing translational optical flow inversely proportional to scene depth due to camera motion. In **`Figure 2`**, we present results for scenes with camera shake alone and with a combination of heavy turbulence and camera shake. Our methods effectively mitigate camera shake since it exhibits irregular motion patterns over time. Additionally, for combined camera shake and turbulence, our methods enhance temporal consistency, even when residual camera shake and temporal inconsistencies remain after performing DATUM.   

[2] Ranftl, René, et al. "Towards robust monocular depth estimation: Mixing datasets for zero-shot cross-dataset transfer." IEEE transactions on pattern analysis and machine intelligence 44.3 (2020): 1623-1637



**`Shared Question C`** : **Comparison with Unsupervised and Test-time Optimization Baselines.**

We have added additional baselines with two popular unsupervised/test-optimization methods: Mao’s [3] (only designed for grayscale frames) and Li’s [4]. As shown in **`attached file Figures 3`**, both methods are designed for static scenes and fail to capture object motion while removing turbulence. This is evident as both methods replace moving human body parts with an averaged static background. 

These unsupervised and test-time optimizations are only effective for static scene turbulence removal and fail with videos containing moving objects. To our knowledge, our method is the only unsupervised and test-time optimization approach designed to address temporal consistency in video turbulence mitigation involving moving objects. This is the key difference and novelty of our method compared to other unsupervised and test-time optimization turbulence mitigation methods.


[3] Mao, Zhiyuan, Nicholas Chimitt, and Stanley H. Chan. "Image reconstruction of static and dynamic scenes through anisoplanatic turbulence." IEEE Transactions on Computational Imaging 6 (2020): 1415-1428.

[4] Li, Nianyi, et al. "Unsupervised non-rigid image distortion removal via grid deformation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.



**`Shared Question D`** :  **Feature Visualization**.

We visualize the canonical image by inputting the canonical spatial feature map into the content MLP without applying $\Delta x$ and $\Delta y$, as shown in the **`attached file Figure 4`**. The canonical image contains most of the video's content, providing a base representation from which other frames can be derived. Consequently, the canonical spatial field functions similarly to a key frame in video compression, serving as a reference for other frames in the sequence to query information.

While directly visualizing the N Temporal Feature Map (dimension [T_res, Q1]) is challenging, we demonstrate its effects through ablation studies. We control the strength of regularization by adjusting T_res. As shown in Table R2-Q3, decreasing T_res achieves better regularization on temporal inconsistency

Meta-review: This paper presents a novel method for improving the temporal consistency of videos affected by atmospheric turbulence, utilizing implicit neural representations (INRs) to model temporal and spatial deformations separately. The proposed framework introduces two key INRs: the temporal deformation field and the spatial content field, both of which are trained on the outputs of a pre-trained atmospheric turbulence mitigation model. The method effectively employs disparity and similarity losses to ensure temporal consistency and maintain content fidelity. Experimental results on both real-world and custom datasets demonstrate significant improvements in temporal consistency, particularly for videos with moving objects, when integrated with state-of-the-art turbulence mitigation models.

The paper is generally well-written, and the majority of reviewers support its acceptance. While reviewer VLLH raised concerns regarding the method's capability to handle large motion and moving objects, the AC believes these concerns are mitigated by the challenging nature of the problem and the promising results already demonstrated in the paper. The ability of the proposed method to effectively address the issue of moving objects has been substantiated by the experiments. Therefore, the AC recommends accepting this paper.
Decision: Accept (poster)