Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations

Authors: haoming  cai
jingxi  chen
brandon y feng
weiyun  jiang
mingyang  xie
kevin  zhang
cornelia  fermuller
yiannis  aloimonos
ashok  veeraraghavan
christopher a metzler
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:14:39

Abstract: Atmospheric turbulence, caused by random fluctuations in the atmosphere's refractive index, introduces complex spatio-temporal distortions in imagery captured at long range. Video Atmospheric Turbulence Mitigation (ATM) aims to restore videos affected by these distortions. However, existing video ATM methods, both supervised and self-supervised, struggle to maintain temporally consistent mitigation across frames, leading to visually incoherent results. This limitation arises from the stochastic nature of atmospheric turbulence, which varies across space and time. Inspired by the observation that atmospheric turbulence induces high-frequency temporal variations, we propose ConVRT, a novel framework for consistent video restoration through turbulence. ConVRT introduces a neural video representation that explicitly decouples spatial and temporal information into a spatial content field and a temporal deformation field, enabling targeted regularization of the network's temporal representation capability. By leveraging the low-pass filtering properties of the regularized temporal representations, ConVRT effectively mitigates turbulence-induced temporal frequency variations and promotes temporal consistency. Furthermore, our training framework seamlessly integrates supervised pre-training on synthetic turbulence data with self-supervised learning on real-world videos, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data. More information can be found on our project page: https://convrt-2024.github.io/

Main paper: 
 introduction: 
 Atmospheric turbulence poses a significant challenge in long-range imaging applications, causing unique distortions in captured videos. These turbulence-distorted videos suffer from spatially-varying and time-varying degradations, including blur and warping effects, due to the random fluctuations of the refractive index in the atmosphere. These distortions significantly hinder the performance of computer vision applications like object detection, recognition, and surveillance systems by obscuring the true shapes, edges, and visual details of objects. Therefore, this work focuses on Video Atmospheric Turbulence Mitigation (ATM), aiming to recover videos degraded by these atmospheric distortions.Mathematically, the process of capturing video through atmospheric turbulence can be modeled by the following equation '24) and TMT (3)(TCI '23) , designed for video ATM, fail to maintain temporal consistency in real-world atmospheric turbulence. For instance, they produce flickering artifacts on a static pole.EQUATIONwhere B t and T t are blur and tilt process at t time stamp. • denotes the application of tilt followed by blur (1) .As described by Equation ( 1), the key challenge arises from the stochastic nature of atmospheric turbulence, which varies across space and time, making temporally consistent video restoration difficult. Figure 1 illustrates the challenge of temporal inconsistency by showing a static scene captured with a stationary camera and object. Despite the static setup, the atmospheric turbulence introduces erratic movements of the stationary traffic cone across frames, causing flickering artifacts in the resulting video sequence. Notably, even state-of-the-art methods like DATUM (2) , designed for video turbulence mitigation, fail to produce temporally consistent mitigation results, result in flickering video. This underscores the critical need for novel solutions tailored to address the challenge of temporal consistency in video atmospheric turbulence mitigation.Table 1 : Comparison of recent supervised (S), selfsupervised (SS), and hybrid (S+SS) learning approaches for image and video ATM. degradation in images and videos, various methodologies have been developed. Current state-of-the-art methods can generally be categorized into supervised and self-supervised learning manner. Supervised learning techniques in ATM use turbulence simulators to generate paired training data (clean and distorted images/video) that can be used for training (14; 15; 16; 1; 17; 18) . Fig. 3(A ) and the supervised learning section of Table 1 depict methods that achieve significant results based on large amounts of paired data. Despite the continual evolution of simulators, the persistent gap between simulated and real-world atmospheric turbulence poses challenges for this design in handling unseen real-world data. In videos, this drawback is further amplified, leading to issues like temporally inconsistent mitigation. To address this temporal inconsistency issue, in addition to better simulators, enlarging the dataset and model capacity are necessary, which substantially increases the computational costs of training.Self-supervised learning approaches for ATM employ internal learning techniques to leverage data priors such as lucky images, internal data distributions, or blind degradation estimation, as depicted in Fig. 3 (B) and self-supervised learning section of Table 1 . A key advantage of these methods is their test-time optimization capability, allowing them to adapt to any test data. However, to date these approaches have not been used to enforce temporal consistency in video ATM. Furthermore, because they don't exploit accurate learned image priors, the performance of self-supervised methods in real-world turbulence mitigation often falls short of supervised learning approaches. This paper develops a hybrid algorithm that can consistently mitigate real-world atmospheric turbulence across video frames. Our pipeline can leverage the knowledge encoded in pre-trained models while leveraging test-time optimization to adapt to the complexities of real-world turbulence. As shown in Fig. 3 (C) and the final section of Table 1 , our pipeline leverages the strengths of both self-supervised learning and simulation-based pre-training.Our work is motivated by the insight, illustrated in Figure 2 , that state-of-the-art ATM methods struggle to remove the temporal distortions introduced by turbulence. That is, while existing methods are reasonably effective at removing the spatial distortions (e.g., blur) introduced by turbulence, they do not effectively remove the temporal distortions.To address this challenge, we develop an approach that explicitly decouples spatial and temporal information. This method leverages the low-pass filtering properties of neural networks to reduce turbulence-induced degradations. Specifically, we propose a self-supervised method called ConVRT (Consistent Video Restoration through Turbulence). ConVRT forms a neural representation of the reconstucted video that explicitly decouples spatial and temporal information: The video is represented with a spatial content field and a temporal deformation field. This decoupling allows ConVRT to effectively regularize the temporal information while preserving spatial information and fine details.Through extensive evaluations, we demonstrate that ConVRT substantially improves temporally consistency while also marginally improving per-frame restoration quality. 
related work: 
 Implicit neural representations. Our work leverages a coordinate-based implicit neural representation (INRs), which has been commonly adopted to model 2D images or 3D videos as multilayer perceptions (MLPs). INRs take 2D pixel coordinates (x, y), or 3D pixel coordinates with temporal encoding, (x, y, t) and output the corresponding pixel values. These INRs demonstrate exceptional performance when fitting images (19; 20; 21; 22; 23; 24; 25) , videos (26; 22) , 3D shapes (27; 28; 26; 29; 30; 31) , and optical components (32) . Not only they are able to represent these 2D or 3D signals, but they also show strong priors for solving inverse problems, such as image super resolution (33) , phase retrieval (34) , and reducing optical aberration (35; 36; 37; 38) .Neural video representation. Our work aligns closely with the evolving field of neural video representation (39; 40; 41; 42) . While there are existing approaches (43; 44; 42; 45) that seek to represent a video into decomposed layers, these primarily focus on clean videos and are not applicable to videos with severe degradation turbulence. Our work extends the application of neural video Atmospheric turbulence mitigation. Attempts to mitigate atmospheric turbulence (46; 47) have applied optical flow (9; 48), B-spline grid (49) , and diffeomorphism (50) to unwarp each distorted image and then fuse and combine these registered distorted images into a clean and sharp image. The fusion is usually modeled as patch-wise stitching (9) or blind deconvolution (51) . Recent development of high-performance GPUs and fast turbulence simulators (16; 18; 17; 15; 14) leads to new progress in turbulence mitigation (15; 5; 11; 7; 12) . However, previous efforts tend to overlook the importance of temporal consistency on the reconstructed video. Our method, ConVRT, is specifically designed to restore temporal consistency with on test-time optimization of a neural video representation.Blind Video Restoration via deep video prior. Supervised video restoration methods (52; 53; 54; 55) have made significant advancements but are constrained by the need for paired data, which increases the value of blind video restoration. One promising direction involves leveraging deep priors. The deep video prior (DVP) and DVP-based blind video consistency methods (56; 57) use convolutional neural networks (CNNs) to learn image operators that exploit the implicit priors in CNNs to remove video artifacts. These approaches have demonstrated impressive results in tasks such as colorization and white-balancing. However, turbulence mitigation presents a more complex challenge compared to these common degradations, involving spatially and temporally varying blur and tilt. This complexity raises unexplored questions for these types of methods 3 MethodQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Section 5 discusses the limitation of our method Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
overview of the pipeline: 
 The framework of our method, ConVRT, is presented in Figure 4 . In this subsection, we provide a high-level overview covering the design inspiration, the video representation mechanism, and the training process.Design Inspiration. As discussed in Section 1.2, the core design logic of ConVRT is to apply temporal-wise regularization in video representation learning. For the representation, our method is inspired by a series of works on tensor decomposition, commonly used to parameterize 3D volumes in implicit neural representations (INR). These approaches enhance the ability to represent 3D signals while reducing the number of required parameters (33; 58; 59) . Building upon this, we developed the ConVRT method.Video Representation. ConVRT represents videos using two main components: the 3D Spatial-Temporal Deformation Field (T field ) and the 2D Spatial Content Field (S field ). The process begins with T field , which receives the pixel location (x, y, t) as input, where (x, y) are spatial coordinates and t is the temporal frame index. T field outputs deformation offsets (∆x, ∆y), indicating changes in the pixel's spatial position across frames relative to a canonical frame. These offsets are then used by S field . S field queries the Canonical Spatial Feature Map (C) at the modified location (x + ∆x, y + ∆y) to retrieve the corresponding feature from a trainable feature map. This feature is subsequently processed by an MLP to predict the RGB intensity values for the pixel location (x, y, t).Training Overview. During training, the trainable parameters include all feature maps and the Multi-Layer Perceptrons (MLP) described below. The loss function measures the difference between the predicted RGB intensity values and the corresponding pixel colors in the restored video, obtained using any ATM method. Since no ground-truth data is available, ConVRT is designed to overfit each partially restored video; however, its limited capacity for capturing temporal information prevents it from overfitting to turbulence artifactsWe represent the input video's spatial-temporal features using two main components: the Spatial Feature Map and the Temporal Feature Map. The Spatial Feature Map (M ) acts as a dictionary for spatial features, with dimensions R H×W ×Q1 , where H is the frame height, W is the frame width, and Q 1 is the number of spatial feature channels. Each pixel coordinate (x, y) serves as a key to retrieve the corresponding spatial feature vector M x,y ∈ R Q1 . The Temporal Feature Map (N ) functions as a dictionary for temporal features, with dimensions R Tres×Q1 . Here, T res is the regularized temporal resolution and Q 1 is the number of temporal feature channels.To construct the spatial-temporal feature vector V x,y,t at a specific pixel location (x, y, t), we first query the Spatial Feature Map M using the pixel coordinates (x, y), extracting the spatial feature vector M x,y ∈ R Q1 . Next, we query the Temporal Feature Map N using the time coordinate t, extracting the temporal feature vector N t ∈ R Q1 . These vectors are then combined using the Hadamard product, which performs element-wise multiplication, to form the spatial-temporal feature vector:EQUATIONThis Hadamard product effectively combines the spatial and temporal features, creating a compact and efficient representation of the video's spatial-temporal characteristics. This V x,y,t is then fed into a compact MLP, referred to as the deformation MLP. The details of the deformation MLP are provided in the supplementary material. The deformation MLP outputs the offsets (∆x, ∆y) necessary for warping the canonical spatial feature map.To regularize the temporal representation capability of the Temporal Feature Map (N ), we constrain its dimensions to R Tres×Q1 , where T res is much smaller than the total number of video frames (T ). Consequently, multiple neighboring frames share the same temporal feature. For example, frames at t -1, t, and t + 1 may query the same temporal feature N t due to the reduced temporal resolution. Additionally, we define the deformation MLP with a reduced number of parameters. Both regularizations decrease the representation capacity of the temporal features, promoting smoother and more consistent temporal dynamics across frames, as inspired by our motivation experiment.The Spatial Content Field focuses on accurately representing the spatial details of each video frame. Unlike the Spatial Feature Map (M ) used in the Temporal Deformation Field, we initialize a new optimizable feature map, denoted as the Canonical Spatial Feature Map (C), with dimensions R H×W ×Q2 , where H is the frame height, W is the frame width, and Q 2 is the number of spatial feature channels specific to this field.Each pixel coordinate (x, y) is adjusted by the deformation offsets (∆x, ∆y), resulting in new coordinates (x + ∆x, y + ∆y). These adjusted coordinates are then used to query the Canonical Spatial Feature Map (C), retrieving the spatial feature vector C x+∆x,y+∆y ∈ R Q2 . These spatial features are processed by a Content MLP, which transforms the spatial feature vector C x+∆x,y+∆y into the final RGB intensity values for the corresponding pixel. The details of the Content MLP are provided in the supplementary material. This transformation ensures that the spatial details of the video frame are accurately captured and represented.Temporal Consistency Regularization. To ensure temporal stability across video frames, we use a disparity estimation network (MiDas ( 60)) to calculate pixel-wise disparities. These disparities serve as weights for the predicted warp (one of D field 's outputs), helping to maintain spatial consistency over time. The loss is defined as:EQUATIONwhere Disparity(I) measures the pixel-level disparity, and ∥Predicted Warp∥ 1 enforces sparsity in the grid changes. The design of L temp minimizes the L1 norm of the predicted warp, conditioned by 1 -Disparity(I), to prioritize consistency in far regions based on the depth information. This focused approach on temporal consistency significantly reduces the propagation of turbulence-induced distortions, ensuring a smooth transition between frames.Similarity Loss. The Similarity Loss Term is given by:EQUATIONwhere λ mse , λ ssim , and λ lpips are weights for each term. This loss term assesses the fidelity of the predicted output compared to the outputs of arbitrary ATM methods, incorporating Mean Squared Error (MSE), Structural Similarity Index Measure (SSIM) (61) , and Learned Perceptual Image Patch Similarity (LPIPS) (62) . This multifaceted approach ensures a comprehensive evaluation of reconstruction quality.Overall Loss. The overall loss combines the similarity loss with temporal consistency and semantic enhancement:EQUATION4 ExperimentsQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] This paper does not provide theoretical results. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
datasets and training details: 
 We adopt several real-world datasets for evaluation, including the OTIS (63), HeatChamber (5), subset of BVI-CLEAR dataset (64), TSR-WGAN dataset (4) and DOST (65) . We trained the ConVRT model individually on each video clip with a learning rate of 2 × 10 -3 , using the Adam optimizer (66) . For each video clip, the batch size equals to the number of frames in that clip. The spatial resolution of both the trainable spatial feature map and the canonical spatial feature map matches the original frame resolution after square cropping. The temporal resolution parameter T res was set to 5, with parameters Q 1 and Q 2 configured to 128 and 256, respectively. More details about the network settings are provided in the supplementary material. Training was conducted on a single RTX A6000.We selected VRT (52) , TMT(3), and DATUM(2) as the base video methods for ConVRT due to their state-of-the-art performance in video restoration and video ATM. TurbNet( 5) is selected for base image method. We also directly applied ConVRT to the original video without the base methods to assess its standalone performance. To evaluate the consistency of turbulence removal in videos, we employed four metrics for quantitative evaluation and two interframe-related methods for qualitative assessment.Temporal Consistency and Per-frame Quality. We used PSNR and SSIM to measure the per-frame reconstruction quality. Following (67), we utilized the average warp error to quantify the temporal consistency of the restored video. The warp error between two consecutive frames is defined as:EQUATIONwhereV (i)t+1 is the warped frame by optical flow at time t + 1 and M (i) t ∈ 0, 1 is the occlusion mask estimated by the methods proposed in (68) . The average warp error across the entire video sequence is calculated as: In addition to the warp loss, inspired by ( 69), we also calculate the Total Variation loss of the X-t slice, Slice tv , and Total Variation of the optical flow, Flow tv , to quantitatively measure whether the temporal variation of time slices in restored videos are small.EQUATIONKLT Trajectories. We employed the KLT tracker (70) to track feature points and plot their trajectories, as shown in Figure 5 . KLT tracking is directly based on image gradient information, such that common issues in turbulence restoration, i.e., blurriness, artifacts, and temporal inconsistency, are reflected in the tracked trajectories. Smooth and coherent trajectories indicate temporally consistent restoration, while erratic or discontinuous trajectories suggest the presence of artifacts or inconsistencies.x-t Slice. We plotted x-t slices to visualize the motion of a row of pixels, as illustrated in Figure 5 . If the video restoration is temporally consistent, the x-t slice plot will exhibit smooth and continuous curves. In contrast, non-smooth or jagged curves in the x-t slice indicate temporal inconsistencies or artifacts in the restored video.Qualitative and Quantitative Improvements on Existing Methods. Figure 8 : Illustration of camera shake simulation using Brownian Motion. In the Y-t slice plots, we observe similarities between camera shake and turbulence. The plots also demonstrate the effectiveness of our approach in handling both camera shake alone and in combination with turbulence. Qualitative Synthetic Cases. Similarly, on synthetic video, As shown in Figure 6 , our method enhances temporal turbulence removal when applied to baseline methods. The video dynamics generated by ConVRT closely resemble the ground-truth videos, effectively smoothing out atmospheric turbulence. The improved KLT trajectories further demonstrate this temporal consistency.Quantitative Results. We evaluated the performance of our proposed ConVRT method across realworld datasets containing both static and dynamic scenes, as shown in Table 2 . ConVRT demonstrates consistent improvements across models and most of datasets, underscoring its broad applicability. On the HeatChamber dataset, which provides real-world paired data through a controlled heating mechanism, we calculated PSNR values to further substantiate ConVRT's effectiveness. onVRT consistently improves PSNR, demonstrating robust enhancement of temporal consistency, especially given PSNR's sensitivity to pixel misalignment.Regularized temporal resolution T res is critical for ensuring temporal consistency. Lowering it results in smoother transitions but loses fine details, while a higher value preserves details but increases the risk of flickering. We conducted an ablation study on the impact of T res and L temp , as shown in Table 3 , with qualitative results in Figure 7 . These results demonstrate the effectiveness of our representation field design in regularizing irregular turbulence motion. : Mitigation capability of our method without using base restoration methods for preprocessing. The scene is an elephant raising its head.Why It Works. Our method's effectiveness stems from two key factors. First, it leverages the distinct differences in motion patterns, particularly the regularity of optical flow directions within a short time window, between regular object movement and atmospheric turbulence, as discussed in Section 1.2. This distinction also enables our method to handle camera shake, which shares similar irregular patterns with turbulence. The results of this capability are illustrated in Figure 8 . Second, our method includes a robust video representation that overcomes limitations in unsupervised methods for static scenes. As illustrated in Figure 9 , these methods often struggle on moving objects, blending them into static backgrounds. In contrast, our approach preserves object integrity across frames, making it well-suited for video-based turbulence mitigation.Mitigation Capability without Base Restoration Methods. Even without a base restoration method to provide partially restored frames, our approach could improve temporal consistency, as shown in Figure 10 . However, we recommend combining our method with other restoration techniques. This allows user to benefit from the sharpness improvements offered by supervised methods, while also taking advantage of the temporal consistency improvements provided by ConVRT.Visualizing Trainable Feature Maps. We visualize the canonical image by inputting the canonical spatial feature map into the content MLP without applying ∆x and ∆y, as shown in the Figure 7 . The canonical image contains most of the video's content, providing a base representation from which other frames can be derived. Consequently, the canonical spatial field in our video representation functions similarly to a key frame in video compression, serving as a reference for other frames in the sequence to query information.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: This paper provides detailed explanation of proposed method and experimental setup in sections 3, 4 and supplementary.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
limitations: 
 While ConVRT offers significant improvements in video atmospheric turbulence mitigation, there are two limitations. First, as a neural representation method, ConVRT's performance depends on accurate video representation and currently optimized to capture motion with precision in short clips. Extending this to longer sequences and more complex motions is a potential area for future exploration. Second, ConVRT processes a 25-frame video at 540x540 resolution in approximately 10 minutes, including DATUM as base method. Although much faster than Mao's (165 minutes) and Li's (300 minutes) methods, there is still room for improving computational efficiency, especially for larger or more complex sequences.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We have a website to host all open-source resources, including code, data, and results. The website link is attached to the abstract of the main paper. We will organize code base and make it easy for users to use. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: 1 discusses the positive influence this work could bring to related research fields. For social-wise impact, the influence is difficult for us to figure out. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
conclusion: 
 In this paper, we present ConVRT, a novel approach aimed at enhancing temporal consistency in video ATM tasks. ConVRT uses a dual-field approach-Temporal Deformation Field and Spatial Content Field-to accurately capture spatial information while regularizing temporal information, focusing on regular object motion rather than irregular turbulence. Combined with any ATM method, ConVRT leads to visibly improved temporal consistency. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work doesn't involve any possibility of being misused.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: All assets used in this paper are open-source.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: A.1 MLP Network DetailsOur network architecture consists of two MLPs: a content MLP and a deformation MLP. The content MLP has 4 fully-connected layers, with an input dimension of Q 1 , a hidden dimension of 128, and an output dimension of 2, representing ∆x and ∆y. The deformation MLP comprises 6 fully-connected layers, with an input dimension of Q 2 , a hidden dimension of 256, and an output of 3 channels representing RGB intensity.Position encoding for spatial and temporal indices is embedded within the trainable feature map, as these indices are trainable. We directly use x, y, and t to query the corresponding feature tensors from the feature maps. Notably, in the temporal feature map, neighboring features are shared across multiple frames, with each frame weighted differently due to explicit regularization. Guidelines:Patch• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: This paper presents a novel method for improving the temporal consistency of videos affected by atmospheric turbulence, utilizing implicit neural representations (INRs) to model temporal and spatial deformations separately. The proposed framework introduces two key INRs: the temporal deformation field and the spatial content field, both of which are trained on the outputs of a pre-trained atmospheric turbulence mitigation model. The method effectively employs disparity and similarity losses to ensure temporal consistency and maintain content fidelity. Experimental results on both real-world and custom datasets demonstrate significant improvements in temporal consistency, particularly for videos with moving objects, when integrated with state-of-the-art turbulence mitigation models.

The paper is generally well-written, and the majority of reviewers support its acceptance. While reviewer VLLH raised concerns regarding the method's capability to handle large motion and moving objects, the AC believes these concerns are mitigated by the challenging nature of the problem and the promising results already demonstrated in the paper. The ability of the proposed method to effectively address the issue of moving objects has been substantiated by the experiments. Therefore, the AC recommends accepting this paper.

Reviews: review: paper_summary: paper_summary: This paper presents an implicit neural representation (INR) framework for taking a pre-trained supervised video atmospheric turbulence mitigation (ATM) model and regularizing its output to be more temporally consistent. The main components are (1) an INR called the temporal deformation field; and (2) a subsequent INR called the spatial content field to output RGB intensity at a pixel in the video (at a certain time). These two INRs are trained on the output of a pre-trained ATM model, and regularized using a disparity loss (with MiDas pre-trained network) for temporal consistency, and a similarity loss for the content of the video. Experiments are conducted on real-world datasets with comparison to state-of-the-art ATM models recently proposed as well as some simulated ablation studies.
strength_weakness: strength_weakness: - There is little detail about the spatial feature map M, temporal feature map N, canonical spatial feature map C. What does it mean to call these feature maps, and why are they chosen the way they are? For instance, why the Hadamard product for M and N, and not just learning a 3D feature map at that place instead directly? I also don't see how the C map is "canonical" to me in any obvious way (for instance, you could change the dimensions of Q1, Q2 and I don't see why that couldn't work in the method?). 

- The method seems to be focused primarily on fixing errors for supervised ATM methods. However, some of the classical approaches such as [Mao 2020] that utilize lucky frames, would they have this problem of temporal variability? I'm not necessarily asking for a comparison or new experiments, but it would be good to discuss if this problem primarily is for supervised methods. 

Reference: Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan, ‘‘Image Reconstruction of Static and Dynamic Scenes through Anisoplanatic Turbulence’’, IEEE Transactions on Computational Imaging, vol. 6, pp. 1415-1428, Oct. 2020

- Table 3 is a very modest improvement . Does the Ltemp really help? A qualitative example would really help clear up that this L_temp is working (show a figure with and without Ltemp).

- How would the method handle issues such as camera shake (common in long-range videos that are shot with high optical zoom)? 

Minor suggestions:
- Line 187 - TurbNet, shouldn't it be whatever ATM method you are comparing with? 
- Line 205 - One shouldn't be capitalized
- Line 109 - unresolved citation
- Table 1 - more stylistic, but I don't think its necessary to put down the venue into the table. We shouldn't judge methods based on their venue, but on the content of the method itself, so having the citation alone is enough to let readers draw their own conclusions about the papers. I would remove this column from the table. 
- Table 3 - there is a typo in the PSNR_Img column where the lower number is bolded rather than the higher one
questions: questions: 1. Can the authors explain (1) how the feature maps M, N, C are novel compared to other INR video representations? Is it the use of the Hadamard product, and the two stage architecture? If so, an ablation study of comparing the Hadamard product of M and N to just a 3D spatial feature map directly is warranted. (2) I would be good to visualize these features after optimization (what do they look like), and what information/interpretability can be gleaned from them? 

2. There are missing details: what is the details of the MLP layers, any positional encoding? These should be added to a supplemental file. 

3. I am interested if L_temp is the key factor that improves the method, or its a minor improvement. Showing a qualitative example as discussed earlier would be beneficial here. 

4. Can there be a discussion about issues involving camera shake? I assume the method would require stabilized videos first, and if there is any residual motion leftover, this would cause errors in the reconstruction. 

5. What is the wall clock time of the method? How long does it take (in actual seconds) from start to finish? The paper only states 80 epochs, I'm curious how long that actually takes. 

6. In line 134 - it says an additional enhancement module is applied after S_field. This was never discussed again. How important is this module? What's the performance with and without the module?
limitations: limitations: Yes
paper_summary: paper_summary: This paper introduced ConVRT, a novel method for video atmospheric turbulence mitigation. 
This paper has a good structure and is well-written.
strength_weakness: strength_weakness: 1. limited real-world case visualization
2. limited proof of algorithm effectiveness
3. limited comparison with classic algorithm
questions: questions: 1. Test cases with visualization are so limited, even in the supplemental material. Please show more real-world cases to show the effectiveness of the algorithm.
2. As the representative unsupervised method, "Unsupervised non-rigid image distortion removal via grid deformation," it is important to compare with it, no matter from the algorithm or qualitative result.
3. Lack of ablation studies to prove the effectiveness of the proposed algorithm.
4. Can the method deal with the video with moving objects?
5. No matter whether it is for a single image-based or multiple frames-based turbulence mitigation, most existing algorithms can deal with them very well. With the help of the diffusion model, the resulting image can be refined further. It means that it could generate a good final result in a short time.  Then, what is the advantage of your algorithm?
6. It is important to visualize the temporal and spatial field to verify the algorithm's effectiveness.
limitations: limitations: Same with question
paper_summary: paper_summary: This paper proposes a method for improving the temporal consistency of turbulence-affected videos. The proposed method uses neural representations (MLP layers) to separately model the spatial and temporal deformations caused by air turbulence and is able to improve the temporal consistency of restoration results. It seems that the proposed method needs to be used in conjunction with another base turbulence restoration method. The prosed method (by combining with several other SOTA approaches) is evaluated on existing real turbulent video dataset, and a small dataset collected by the authors. Temporal consistency, especially for videos with moving objects, is apparently improved after applying the proposed method.
strength_weakness: strength_weakness: - It seems that the method is an add-on approach for regularizing the temporal consistency of videos restored by another turbulence restoration methods. Since turbulent degradation is more complex than temporal inconsistency and spatial distortions (e.g., there might be blurriness and color aberration beyond the spatial and temporal deformations), being able to only handle these two types of artifacts seems quite limited for turbulence mitigation. 
- The quantitative evaluation results (Table 2) are confusing. This table shows metric scores for using the proposed method as a stand-alone (i.e., using the original turbulent video as input). However, the "no base results" are not demonstrated in any visual comparison figures (not even in supplementary materials). It would be useful to see visual comparison between "no base results" and others. What really confuses me about this table is that the metric scores of the original turbulent images (denoted as "ori") are even better than processed results in many cases (for example, its PSNR is higher than DATUM results for the HeatChamber dataset, and there many such cases). But according to visual results, most processed results have apparent improvement. Besides, after applying the proposed method, some metric scores become much worse (for example, the slice_tv scores for most datasets and method combinations). There should be some discussions explaining the metric results. 
- The paper missed some relevant prior works (see below). These works either use MLP for modeling turbulent distortions or use similar idea to enforce temporal consistency and they should be discussed and compared with the proposed method.
Li et al., "Unsupervised Non-Rigid Image Distortion Removal via Grid Deformation," ICCV 2021.
Thapa et al. "Learning to Remove Refractive Distortions from Underwater Images," ICCV 2021.
questions: questions: - It seems that motions in demonstrated results are quite slight and slow. Is the method also robust to large motions? How is this related to turbulence strength? 
- I'm interested at knowing more details about the airport dataset acquired by the authors, like size of the dataset, type of scenes, turbulence strength, etc.
limitations: limitations: Two limitations are discussed: 1. length of input video cannot be too long, and 2. processing time is sort of long (running time of the method is not reported). Turbulence strength and motion scale could be also discussed, if they are limiting factors.
paper_summary: paper_summary: The study focuses on Video Atmospheric Turbulence Mitigation (ATM), which aims to restore videos that are affected by distortions caused by atmospheric turbulence. Specifically, the proposed ConVRT introduces a neural video representation that decouples spatial and temporal information, allowing targeted regularization of the network's temporal representation capability. Also, this paper integrates supervised and self-supervised learning, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data.
strength_weakness: strength_weakness: 1.	As a neural representation method, ConVRT is designed to handle video clips with a limited number of frames, making it challenging to handle larger video sequences and more significant motions without compromising accuracy.
2.	The related work section lacks completeness. More test-time methods should be carefully reviewed. Also, no SOTA methods were involved in the experimental section for comparison, which is unacceptable.
3.	In addition to VRT, more SOTA transformer-based methods, as well as the recent-popular diffusion-based models, should be added for analysis.
4.	ConVRT employs test-time optimization, which can be computationally intensive. Therefore, model efficiency should be discussed.
5.	Some intermediate results should be given.
questions: questions: Please refer to the detailed comments!
limitations: limitations: Please refer to the detailed comments!

Comments: title: Follow-up Request
Further discussion with Reviewer EL8T (denoted as R2)
Solve most of my conserns
Further discussion with Reviewer DNNj (denoted as R1)
Response to Author Rebuttal
Response to Author Rebuttal
comment: Thank you for letting us know our rebuttal addresses all your concerns. We will incorporate all your suggestions in the camera-ready version (if it is accepted), to help readers better understand our method and its contribution to the ATM field. Thank you again for your valuable time and comments
We greatly appreciate your positive evaluation and recognition. All your suggesions will be included in the revised version. Thank you again for your valuable time and review.
Thank you for your thoughtful review and for recognizing the potential of our paper.

In the revised version, we will include both quantitative and qualitative comparisons with unsupervised and test-time optimization methods ( Li et al. & Mao et al). Additionally, we will expand our discussion section to incorporate your insightful questions and our exchange, helping readers better understand our contribution in the atmospheric turbulence mitigation field.

Again, we greatly appreciate your valuable time and insightful suggestions.
My concerns are mostly addressed by the rebuttal. I would support the acceptance of this paper, given its strength on handling moving objects and improving the temporal consistency of turbulence restoration. In the final version, the author should clarify quantitative results and include comparisons with the methods by Li et al. and Mao et al.
I have no more questions. Thanks
Would you mind elaborating on your concern? 
As illustrated in Figure 1 of the response, our approach can effectively reconstruct scenes with substantial motion: Over the course of the video the person moves across nearly half the field-of-view (see the long diagonal band in the x-t slice).
Thank you for your further explanation.

Applying diffusion models as add-ons to ATM methods presents challenges with hallucination and may worsen temporal consistency. Current restoration-focused diffusion models still struggle with hallucination. Using these models on ATM video outputs has a high likelihood of exacerbating temporal inconsistency, as turbulence-induced random local motion in each frame may result in inconsistent texture restoration, potentially degrading video quality. We will include a small figure in our revised version to further illustrate the limitations of the diffusion model as adds-on.

However, your suggestion points to a valuable but unexplored direction for future ATM research. To effectively use diffusion models as add-ons to refine current ATM methods, future work might consider developing a specialized turbulence module within video-based diffusion models. Our work could inspire such developments, potentially including a regularized temporal attention module.

We appreciate your response. Please let us know if you have any further questions or concerns. We'll respond promptly to your comments.
I am satisfied with these revisions and will raise my rating to weak accept for this paper. The main justification behind my rating is the experimental results shown in the paper (KLT tracking, enhanced temporal consistency) including some moving scenes in the supplemental html page.
For 'Advantage of the proposed method over diffusion model', I mean that it could be an further improve the result of some algorithms turbulence mitigation result.
We greatly appreciate your prompt response. Thank you for your follow-up question and suggestions. 

**`Suggestion 2 & Further Concern`** **More details about the feature map M and N**

Yes, M (Spatial Feature Map), N (Temporal Feature Map), and C (Canonical Spatial Feature Map) are all optimizable during training. M and N are learnable parameters with shapes [H,W,Q1] and [T_res,Q1], respectively. The optimizable parameters in our pipeline include all these feature maps (M,N,C) and two MLPs (content and deformation MLP), as mentioned in line 136 of the main paper. We will emphasize the details of each component by adding more legends and markers to the pipeline figure in the revision.

Regarding the visualization of M (spatial feature map) and N (temporal feature map), we will include visualizations of the deformation grid (delta x and delta y) and M in the revised version. The delta x and delta y serve as a sampling grid, indicating how each frame is sampled from the canonical field (similar to a keyframe in video compression). This visualization will help clarify the product of M and N for specific x, y, and t coordinates

**`Suggesion 1`** **Acknowledge the tensor decomposition**

Thank you for your suggestion. In the revised version, we will reorganize the related work section about INR and add a new subsection to the methods part, explaining the inspiration behind our method's design. 

**`Suggesion 3`** **Description of effectiveness brought by the loss function.**

Thank you for your considerate suggestion. We will add the qualitative results from our ablation study on loss functions, as shown in the **`attached file figure 4 `**, to the revised paper. This addition will help readers better understand the effectiveness of each component. Additionally, we will revise our language to more clearly explain the effectiveness of each component. We appreciate your feedback, which will improve the clarity and comprehensiveness of our paper

**`Suggesion 4`** **Running Time Breakdown**

Thank you for the suggestion. In the revised version, we will include a detailed runtime discussion, breaking down the time for DATUM (0.5 mins) and our method (9.5 mins). Our method is the first refinement strategy for supervised video-based ATM methods struggling with temporal consistency. Therefore, running time efficiency compared to supervised methods is not our primary focus in this work. However, we agree that discussing the runtime of each method, particularly with the breakdown for DATUM, is crucial. This information will help readers understand the computational cost of incorporating our method into existing approaches.


We appreciate your timely response to our rebuttal. Please let us know if you have further concerns. We will respond to your comments as quickly as we can
I appreciate the author's rebuttal.  However, I am still concerned about the capability for large motion and moving objects.

Therefore, I have decided to maintain my original rating of "Borderline reject".
Thank you for the rebuttal, it satisfied most of my concerns. 

One remaining question: I am still confused about M and N. Are they learned feature maps (optimized)? Perhaps you can visualize them as you did C, so the reader has a better understanding of what these maps look like (probably good for supplemental material). 

A few follow-ups for the final camera-ready paper (if accepted): 

1) I think it is important that the paper acknowledge that the Hadamard product decomposition is inspired by other papers (such as [1-3]) by explicitly putting those references and sentence into a revised version of the paper. 

2) Visualizing C in the main paper greatly assists the reader in understanding what this map is. I would endeavor to include it in one of your figures, or as a small additional figure. Your text description should also be incorporated so that the reader can more clearly understand C. Also if M and N are able to be visualized (see question above), it would be good to point readers to understand those feature maps as well. 

3) If the loss function only adds 0.15dB, please revise the language in that section to say a "modest" improvement. It is not a major difference, and do not want to overclaim the contributions of that one loss function. If you want to keep the original language, then I still think a qualitative figure with and without the loss function is needed to convince readers the 0.15dB is significant. 

4) For the wall-clock latency time, I think it would be good to breakdown how much time was due to DATUM (which is not your method), and how much was due to your optimizations. That gives the readers some understanding that if they add your module to their base ATM method, this is the expected overhead. Please add this discussion to either the main paper or the supplemental material.