$\textit{One Stone Three Birds:}$ Three-Dimensional Implicit Neural Network for Compression and Continuous Representation of Multi-Altitude Climate Data

Authors: Alif Bin Abdul Qayyum, Xihaier Luo, Nathan Urban, Xiaoning Qian, Byung-Jun Yoon
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
Keywords: Multi-modal representation learning, continuous super-resolution, dimensionality reduction, cross-modal prediction, scientific data compression

Abstract:
Wind energy stands out as a promising clean and renewable energy alternative, not only for its potential to combat global warming but also for its capacity to meet the ever-growing demand for energy. However, analysis of wind data to fully harness the benefits of wind energy demands tackling several related challenges: 
(1) Current data resolution is inadequate for capturing the detailed information needed across diverse climatic conditions;
(2) Efficient management and storage of real-time measurements are currently lacking;
(3) Extrapolating wind data across spatial specifications enables analysis at costly-to-measure, unobserved points is necessary.
In response to these challenges, we introduce a modality-agnostic learning framework utilizing implicit neural networks. Our model effectively compresses a large volume of climate data into a manageable latent codec. It also learns underlying continuous climate patterns, enabling reconstruction at any scale and supporting modality transfer and fusion. Extensive experimental results show consistent performance improvements over existing baselines.

Decision: reject
Reviewer scores: 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730140300344

thread_type:
official_review

<thread_object>
creator: reviewer_hthi

<content>
summary:
This paper designed a deep learning model to achieve scalable multi-modality learning, data dimensionality reduction, and continuous super-resolution for climate data. It addresses three emergent issues related to wind power generation: insufficient resolution of wind data, lack of storage of wind data, and high cost of wind data gathering at specific locations.

strengths:
originality: It provides a new architecture for continuous super-resolution, which makes multi-modality learning more scalable.

quality: It provides rather comprehensive experiments to support its architecture modification and superiority.

clarity: The paper is readable to catch its main idea, but to get deep understanding of technique details requires effort.

significance: It addresses wind power generation challenges claimed in the paper.

weaknesses:
Not enough justification of its architecture modification even though experimentally supported.

The notation system is extremely hard to follow. 

The paper requires extra effort on the writing. 

Your model is capable of arbitrary scale super-resolution after training. The model should be tested for zero-shot super-resolution which means to test on a different upsampling factor from the one it was trained on.

Your model is a super-resolution model. But there is not extensive comparison to other super-resolution models.

questions:
Line 60: the framework name is very confusing.
Line 134: It is hard to read and understand. It requires to rephrase.  
Figure 1: The figure is too busy and hard to digest. For example, there are many dashed lines intersecting
L289: You did not split a part of your data into validation set. Then how did you choose your model hyper parameters?
L289: Why would you choose a rather small dataset for your method? Is it enough to train your model? 
L296: why are training data and test data constructed differently?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730469556625

thread_type:
official_review

<thread_object>
creator: reviewer_ztio

<content>
summary:
Authors introduce a multi-modal architecture inspired by the implicit neural representation literature for cross-modal super-resolution. Their architecture's design is motivated by its usefulness for wind profile prediction, which tackles the following three points: 
1. **Resolution**: They build a continuous representation which allows their implicit super-resolution decoder to infer the wind field at any given coordinate.
2. **Data storage**: Their 3D Encoder Implicit Neural Network (INN) allows them to downsample the input low-resolution wind field that is later used for continuous reconstruction, which provides memory savings and efficient storage.
3. **Generalization**: Their achitecture allows for modality transfer using ther 3D Decoder INN which transfers the downsampled input wind field at one altitude level to a different altitude with the same resolution as the input, allowing them to later continuously reconstruct the higher-resolution wind profile.

strengths:
- The paper is well motivated and presents three challenges in the face of wind energy deployment, which guided the design of their proposed architecture.
- Extensive experiments for demonstrating the strength of the architecture for super-resolution, compression and modality transfer
- OOD experiments for cross-region wind profile prediction
- Extensive ablation experiments motivate the utility of introduced architectural components

weaknesses:
- Previous works have introduced architectures for doing similar things albeit in the related PDE solving and Neural Operator literature and the authors neither cite them nor do they compare against (at least one of) them:
  - [MeshfreeFlowNet: A Physics-Constrained Deep Continuous Space-Time  Super-Resolution Framework (SC20: International Conference for High Performance Computing, Networking, Storage and Analysis 2020)](https://arxiv.org/pdf/2005.01463.pdf)
  - [MAgNet: Mesh Agnostic Neural PDE Solver (Advances in Neural Information Processing Systems 2022)](https://arxiv.org/pdf/2210.05495.pdf)
  - [Fourier Neural Operator for Parametric Partial Differential Equations (International Conference On Learning Representations 2020)](https://arxiv.org/pdf/2010.08895.pdf)
 To name a few

questions:
# Questions
- In line 302, it's said that the model is optimized using the L1 loss, however, it's not clear what exactly are the inputs of this loss. It's a loss between what and what?
# Comments
- The paper doesn't contain visualizations of the upscaled frames. Regression learns the mean of the conditional distribution which leads to a loss of the high-frequency information that is usually present in the wind profile (especially with turbulence).
- It's a bit hard to parse the PSNR/SSIM figure since there's no space between them, it would be better to have a small vertical space between them.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604173268

thread_type:
official_review

<thread_object>
creator: reviewer_pjzg

<content>
summary:
This paper presents an implicit neural network-based approach to analyzing wind data, addressing key challenges like limited data resolution (super-resolution downscaling task), high storage demands (data compression), and efficient extrapolation to unmeasured locations (spatial extrapolation).

strengths:
- Meaningful motivation and nice problem settings - addressing several challenges (various resolutions, high-dimension, unobserved areas) in climate data.
- Nice application of Kolmogorov-Arnold Network (KAN) to climate domain.
- Fluent paper representation and enough experiments.

weaknesses:
- Lack of state-of-the-art ML baselines for comparison. 
- Several claims are not precise (please see the questions below).
- No available codes.

questions:
- You claim your work is **continuous** super-resolution, how finer is it to say "continuous"?
- I believe there is some related work using diffusion models and generative adversarial networks (GANs) for downscaling tasks. You may want to consider them as ML baselines. The current comparison against ML baselines is relatively weak.
- As you mentioned in the related work, data sources could be text, images, audio, and video. This is why it is called multi-modal learning (generally in the AI domain). In your work, you call the data at different altitudes as different modalities. It is confusing for users, especially for ICLR readers.
- You claim your model enables reconstruction at any scale. Do you use one model or train separate models for different scales?
- PSNR, SSIM, and CR are used as evaluation metrics. It would be better to provide the details on how to compute them? Any equations? Adding them in the appendix also helps if you have a limited-space problem.
- In Figure 4, it is surprising to see the model performance does not drop as the super-resolution scale increases. Could you please provide any possible reasons?
- In section 5.6, you train the ML model on one region and test it on other regions. This is actually an out-of-distribution (OOD) problem. You may want to add a short discussion on it (FYI). And, what is the meaning of the $x$-axis in Figure 6? Why are there some values between 1 and 2 or between 2 and 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732071200077

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision of Manuscript

comment:
We thank all reviewers for their thoughtful and valuable comments. We have revised our manuscript accordingly. The highlights of the major changes are discussed here:

_Revised Problem Statement \& Methodology Section:_ We have updated our **Problem Statement** and **Methodology** section. For better readability, we have added a **Preliminaries** section that discusses some preliminaries on LIIF, Gaussian Adaptive Attention, KAN, Three-dimensional Encoding, and defines why we interpret multi-altitude data as multi-modal data.  

_Revised Experimental Setup:_ We have updated the **Experimental Setup** section according to the reviewers' comments for better clarity.

_Additional Experimental Results:_ We have compared our proposed LIIF-KAN with more super-resolution models, including:
 - HiNOTE ([1] ICML 2024)
 - SRNO ([2] CVPR 2023)
 - DIINN ([3] WCAV 2023)
 - LTE-KAN ([4] CVPR 2022)

along with the previous baselines. As our problem statement specifically requires continuous reconstruction, we only focus on existing continuous super-resolution models as baselines.

[1] Luo, X., Qian, X. and Yoon, B.J., Hierarchical Neural Operator Transformer with Learnable Frequency-aware Loss Prior for Arbitrary-scale Super-resolution. In Forty-first International Conference on Machine Learning.

[2] Wei, M. and Zhang, X., 2023. Super-resolution neural operator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18247-18256).

[3] Nguyen, Q.H. and Beksi, W.J., 2023. Single image super-resolution via a dual interactive implicit neural network. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4936-4945).

[4] Lee, J. and Jin, K.H., 2022. Local texture estimator for implicit representation function. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1929-1938).

We have also added extra experimental results on altitude-based spatial generalization test. Due to space constraints, we have moved the section on spatial generalizability section to Appendix Section D.

We have also updated Figure 3 with out-of-distribution super-resolution scales.

_Revised Baseline Experimental Result:_ We thank the reviewers for suggesting additional baseline experiment with Fourier Neural Operators. We chose Geometry-Informed Neural Operator [5] as the additional baseline experiment due to its suitability for this specific problem statement, and updated the manuscript with results of this additional baseline experiment. 

[5] Li, Zongyi, et al. "Geometry-informed neural operator for large-scale 3d pdes." Advances in Neural Information Processing Systems 36 (2024).

_Neat figures:_ We have updated all the figures in the revised manuscript with hopefully improved figures for more visibility and readability.

_Notation update:_ We have added a list of notations in the Appendix Section B to make the notations easy to follow.

_Description of metrics:_ We have added the details of the evaluation metrics in Appendix section C of the revised manuscript.

If you have additional thoughts in mind, we would greatly appreciate your suggestions.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732522147320

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers,

As the author-reviewer discussion period is approaching the deadline soon, we kindly request you to review our responses to your comments, concerns and suggestions. If you have further questions or comments, we will do our best to address them before the discussion period ends. If our responses have resolved your concerns, we would greatly appreciate it if you could update your evaluation of our work accordingly.

Thank you once again for your valuable time and thoughtful feedback.

Regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a new architecture for analyzing wind data. The model is able to superresolve the wind fields to any resolution (through continuous representation), functions as a data compressor (through downsampling layers), and allows for modality transfer to different altitude levels. The paper presents experiments demonstrating the model's strengths in the above three tasks, ablations motivating the components of the architecture.

Strengths: new architecture design that is well motivated for the problem, experiments and ablations demonstrating the effectiveness of the architecture.

Weaknesses: the contributions seem incremental  - while the compression results are positive, the inclusion of other baselines like GINO show that other architectures can also show good performance on the other tasks. Another weakness is the dense presentation of material in the paper. 

An analysis on the impact of the performance improvement over other models (say GINO) will be useful. Showing careful visual predictions that highlight the downscaling advantages can also be useful. The authors could additionally consider metrics like power spectra plots that show what wavenumbers are adequately reconstructed by their model and what other models miss. Superresolution in physical tasks may require a deeper analysis on what is being reconstructed rather than averaged metrics.