Title: Temporally Consistent Atmospheric Turbulence Mitigation with Neural Representations
Venue: NeurIPS 2024
Accepted: yes
Keywords: Atmospheric Turbulence Mitigation
Abstract: Atmospheric turbulence, caused by random fluctuations in the atmosphere's refractive index, introduces complex spatio-temporal distortions in imagery captured at long range. Video Atmospheric Turbulence Mitigation (ATM) aims to restore videos affected by these distortions. However, existing video ATM methods, both supervised and self-supervised, struggle to maintain temporally consistent mitigation across frames, leading to visually incoherent results. This limitation arises from the stochastic nature of atmospheric turbulence, which varies across space and time. Inspired by the observation that atmospheric turbulence induces high-frequency temporal variations, we propose ConVRT, a novel framework for consistent video restoration through turbulence. ConVRT introduces a neural video representation that explicitly decouples spatial and temporal information into a spatial content field and a temporal deformation field, enabling targeted regularization of the network's temporal representation capability. By leveraging the low-pass filtering properties of the regularized temporal representations, ConVRT effectively mitigates turbulence-induced temporal frequency variations and promotes temporal consistency. Furthermore, our training framework seamlessly integrates supervised pre-training on synthetic turbulence data with self-supervised learning on real-world videos, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data. More information can be found on our project page: https://convrt-2024.github.io/


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe study focuses on Video Atmospheric Turbulence Mitigation (ATM), which aims to restore videos that are affected by distortions caused by atmospheric turbulence. Specifically, the proposed ConVRT introduces a neural video representation that decouples spatial and temporal information, allowing targeted regularization of the network's temporal representation capability. Also, this paper integrates supervised and self-supervised learning, significantly improving the temporally consistent mitigation of ATM methods on diverse real-world data.\n\nSTRENGTHS:\nThe study introduces a novel framework called ConVRT for video ATM, which addresses the challenge of maintaining temporal consistency in turbulence mitigation. It proposes a neural video representation that decouples spatial and temporal information, allowing targeted regularization and effective mitigation of turbulence-induced temporal frequency variations. Furthermore, the study integrates supervised and self-supervised learning, improving the temporally consistent mitigation of ATM methods on diverse real-world data.\n\nWEAKNESSES:\n1.\tAs a neural representation method, ConVRT is designed to handle video clips with a limited number of frames, making it challenging to handle larger video sequences and more significant motions without compromising accuracy.\n2.\tThe related work section lacks completeness. More test-time methods should be carefully reviewed. Also, no SOTA methods were involved in the experimental section for comparison, which is unacceptable.\n3.\tIn addition to VRT, more SOTA transformer-based methods, as well as the recent-popular diffusion-based models, should be added for analysis.\n4.\tConVRT employs test-time optimization, which can be computationally intensive. Therefore, model efficiency should be discussed.\n5.\tSome intermediate results should be given.\n\nQUESTIONS:\nPlease refer to the detailed comments!\n\nLIMITATIONS:\nPlease refer to the detailed comments!", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for improving the temporal consistency of turbulence-affected videos. The proposed method uses neural representations (MLP layers) to separately model the spatial and temporal deformations caused by air turbulence and is able to improve the temporal consistency of restoration results. It seems that the proposed method needs to be used in conjunction with another base turbulence restoration method. The prosed method (by combining with several other SOTA approaches) is evaluated on existing real turbulent video dataset, and a small dataset collected by the authors. Temporal consistency, especially for videos with moving objects, is apparently improved after applying the proposed method.\n\nSTRENGTHS:\n- The paper is generally well-written and easy to follow.\n- The MLP-based network for mitigating spatial and temporal deformations is new and seems to be effective, especially on maintaining temporal consistency. \n- Qualitative evaluation is performed on videos with moving objects and have shown apparent improvement.\n- A small real-world dataset was captured and used for testing, but details about the dataset are not clear.\n\nWEAKNESSES:\n- It seems that the method is an add-on approach for regularizing the temporal consistency of videos restored by another turbulence restoration methods. Since turbulent degradation is more complex than temporal inconsistency and spatial distortions (e.g., there might be blurriness and color aberration beyond the spatial and temporal deformations), being able to only handle these two types of artifacts seems quite limited for turbulence mitigation. \n- The quantitative evaluation results (Table 2) are confusing. This table shows metric scores for using the proposed method as a stand-alone (i.e., using the original turbulent video as input). However, the \"no base results\" are not demonstrated in any visual comparison figures (not even in supplementary materials). It would be useful to see visual comparison between \"no base results\" and others. What really confuses me about this table is that the metric scores of the original turbulent images (denoted as \"ori\") are even better than processed results in many cases (for example, its PSNR is higher than DATUM results for the HeatChamber dataset, and there many such cases). But according to visual results, most processed results have apparent improvement. Besides, after applying the proposed method, some metric scores become much worse (for example, the slice_tv scores for most datasets and method combinations). There should be some discussions explaining the metric results. \n- The paper missed some relevant prior works (see below). These works either use MLP for modeling turbulent distortions or use similar idea to enforce temporal consistency and they should be discussed and compared with the proposed method.\nLi et al., \"Unsupervised Non-Rigid Image Distortion Removal via Grid Deformation,\" ICCV 2021.\nThapa et al. \"Learning to Remove Refractive Distortions from Underwater Images,\" ICCV 2021.\n\nQUESTIONS:\n- It seems that motions in demonstrated results are quite slight and slow. Is the method also robust to large motions? How is this related to turbulence strength? \n- I'm interested at knowing more details about the airport dataset acquired by the authors, like size of the dataset, type of scenes, turbulence strength, etc.\n\nLIMITATIONS:\nTwo limitations are discussed: 1. length of input video cannot be too long, and 2. processing time is sort of long (running time of the method is not reported). Turbulence strength and motion scale could be also discussed, if they are limiting factors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

comments: SUMMARY:
This paper introduced ConVRT, a novel method for video atmospheric turbulence mitigation. 
This paper has a good structure and is well-written.

STRENGTHS:
This paper proposed a new method to deal with turbulence mitigation.

WEAKNESSES:
1. limited real-world case visualization
2. limited proof of algorithm effectiveness
3. limited comparison with classic algorithm

QUESTIONS:
1. Test cases with visualization are so limited, even in the supplemental material. Please show more real-world cases to show the effectiveness of the algorithm.
2. As the representative unsupervised method, "Unsupervised non-rigid image distortion removal via grid deformation," it is important to compare with it, no matter from the algorithm or qualitative result.
3. Lack of ablation studies to prove the effectiveness of the proposed algorithm.
4. Can the method deal with the video with moving objects?
5. No matter whether it is for a single image-based or multiple frames-based turbulence mitigation, most existing algorithms can deal with them very well. With the help of the diffusion model, the resulting image can be refined further. It means that it could generate a good final result in a short time.  Then, what is the advantage of your algorithm?
6. It is important to visualize the temporal and spatial field to verify the algorithm's effectiveness.

LIMITATIONS:
Same with question
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-26

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an implicit neural representation (INR) framework for taking a pre-trained supervised video atmospheric turbulence mitigation (ATM) model and regularizing its output to be more temporally consistent. The main components are (1) an INR called the temporal deformation field; and (2) a subsequent INR called the spatial content field to output RGB intensity at a pixel in the video (at a certain time). These two INRs are trained on the output of a pre-trained ATM model, and regularized using a disparity loss (with MiDas pre-trained network) for temporal consistency, and a similarity loss for the content of the video. Experiments are conducted on real-world datasets with comparison to state-of-the-art ATM models recently proposed as well as some simulated ablation studies.\n\nSTRENGTHS:\n+ Method can improve a variety of existing state-of-the-art ATM models, and the use of INRs with different feature representations used as inputs seems like an original contribution (at least to this application field, if not video representation in general)\n\n+ The use of KLT tracker for visualizing the temporal variability across the frames is a good visualization and helps show the improvement of the method in a qualitative way\n\n+ Supplemental videos on the website show the method stabilizing video in the presence of turbulence\n\n+ Extensive quantification of the method shown in Table 2 to illustrate the effectiveness of ConVRT\n\nWEAKNESSES:\n- There is little detail about the spatial feature map M, temporal feature map N, canonical spatial feature map C. What does it mean to call these feature maps, and why are they chosen the way they are? For instance, why the Hadamard product for M and N, and not just learning a 3D feature map at that place instead directly? I also don't see how the C map is \"canonical\" to me in any obvious way (for instance, you could change the dimensions of Q1, Q2 and I don't see why that couldn't work in the method?). \n\n- The method seems to be focused primarily on fixing errors for supervised ATM methods. However, some of the classical approaches such as [Mao 2020] that utilize lucky frames, would they have this problem of temporal variability? I'm not necessarily asking for a comparison or new experiments, but it would be good to discuss if this problem primarily is for supervised methods. \n\nReference: Zhiyuan Mao, Nicholas Chimitt, and Stanley H. Chan, ‘‘Image Reconstruction of Static and Dynamic Scenes through Anisoplanatic Turbulence’’, IEEE Transactions on Computational Imaging, vol. 6, pp. 1415-1428, Oct. 2020\n\n- Table 3 is a very modest improvement . Does the Ltemp really help? A qualitative example would really help clear up that this L_temp is working (show a figure with and without Ltemp).\n\n- How would the method handle issues such as camera shake (common in long-range videos that are shot with high optical zoom)? \n\nMinor suggestions:\n- Line 187 - TurbNet, shouldn't it be whatever ATM method you are comparing with? \n- Line 205 - One shouldn't be capitalized\n- Line 109 - unresolved citation\n- Table 1 - more stylistic, but I don't think its necessary to put down the venue into the table. We shouldn't judge methods based on their venue, but on the content of the method itself, so having the citation alone is enough to let readers draw their own conclusions about the papers. I would remove this column from the table. \n- Table 3 - there is a typo in the PSNR_Img column where the lower number is bolded rather than the higher one\n\nQUESTIONS:\n1. Can the authors explain (1) how the feature maps M, N, C are novel compared to other INR video representations? Is it the use of the Hadamard product, and the two stage architecture? If so, an ablation study of comparing the Hadamard product of M and N to just a 3D spatial feature map directly is warranted. (2) I would be good to visualize these features after optimization (what do they look like), and what information/interpretability can be gleaned from them? \n\n2. There are missing details: what is the details of the MLP layers, any positional encoding? These should be added to a supplemental file. \n\n3. I am interested if L_temp is the key factor that improves the method, or its a minor improvement. Showing a qualitative example as discussed earlier would be beneficial here. \n\n4. Can there be a discussion about issues involving camera shake? I assume the method would require stabilized videos first, and if there is any residual motion leftover, this would cause errors in the reconstruction. \n\n5. What is the wall clock time of the method? How long does it take (in actual seconds) from start to finish? The paper only states 80 epochs, I'm curious how long that actually takes. \n\n6. In line 134 - it says an additional enhancement module is applied after S_field. This was never discussed again. How important is this module? What's the performance with and without the module?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}