SSIF: Physics-Inspired Implicit Representations for Spatial-Spectral Image Super-Resolution

Authors: Gengchen Mai, Zeping Liu, Ni Lao, Weiwei Sun, Yuchi Ma, Jiaming Song, Chenlin Meng, Hongxu Ma, Jinmeng Rao, Qian Cao, Nemin Wu, Zhangyu Wang, Stefano Ermon
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Neural Implicit Function, Spatial-Spectral Super Resolution, Spectral Encoding

Abstract:
Existing digital sensors capture images at fixed spatial and spectral resolutions (e.g., RGB, multispectral, and hyperspectral images), and generating super-resolution images with different resolution settings requires bespoke machine learning models. Spatial Implicit Functions (SIFs) partially overcome the spatial resolution challenge by representing an image in a spatial-resolution-independent way. However, they
still operate at fixed, pre-defined spectral resolutions. To address this challenge, we propose Spatial-Spectral Implicit Function (SSIF), a neural implicit model that represents an image as a function of both continuous pixel coordinates in the spatial domain and continuous wavelengths in the spectral domain. This continuous representation across spatial and spectral domains enables a single model to learn from a diverse set of resolution settings, which leads to better generalizability. This representation also allows the physical principle of spectral imaging and the spectral response functions of sensors to be easily incorporated during training and inference. Moreover, SSIF does not have the equal spectral wavelength interval requirement for both input and output images which leads to much better applicability. We empirically demonstrate the effectiveness of SSIF on two challenging spatial-spectral super-resolution benchmarks. We observe that SSIF consistently outperforms state-of-the-art baselines even when the baselines are allowed to train separate models at each spatial or spectral resolution. We show that SSIF generalizes well to both unseen spatial and spectral resolutions. Moreover, due to its physics-inspired design, SSIF performs significantly better at low data regime and converges faster during training compared with other strong neural implicit function-based baselines.

Decision: reject
Reviewer scores: 5, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730277567997

thread_type:
official_review

<thread_object>
creator: reviewer_xer1

<content>
summary:
This paper utilizes an implicit representation framework to address both spatial and spectral degradation in hyperspectral restoration tasks. Various structural variants are proposed based on different sampling strategies, achieving promising results in both in-distribution and out-of-distribution scenarios.

strengths:
The motivation of this study is articulated clearly, beginning with the physical imaging process of hyperspectral sensors to design spectral super-resolution. Experiments confirm the effectiveness of this design. Spatial super-resolution has already been widely adopted in previous work, and this paper combines both approaches to achieve dual-domain super-resolution.

weaknesses:
1、This paper has undergone multiple revisions over an extended period, but it still requires the addition of experiments of 2024. \
2、Implicit representations have become quite advanced in the task of spatial super-resolution, and the approach presented in this paper is relatively straightforward. However, this work lacks a certain level of innovation in the spatial dimension of super-resolution.\
3、This paper still contains typographical errors, such as in section A.5 'SSIF Model Variants,' where 'Both' on line 978 should be corrected to 'both.' Additionally, while the paper presents numerous experiments, some, such as those in section A.14, lack significance. For instance, the careful selection of only two points does not effectively demonstrate the model's superiority.

questions:
1、How are the starting and ending points, $\lambda_{i,s}$ and $\lambda_{i,e}$, determined for each wavelength interval?" \
2、How are the points within the wavelength interval selected? Does the choice between random and non-random selection significantly affect the results?\
3、To investigate how the selection of points affects the results, would it be possible to adopt a coarse-to-fine approach, similar to NeRF, to re-sample the points?\
4、Why is the variance in a Gaussian distribution expressed in this form $\sigma_i = \frac{\lambda_{i,e} - \lambda_{i,s}}{6}$?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655596580

thread_type:
official_review

<thread_object>
creator: reviewer_iyg4

<content>
summary:
This paper proposes the Spatial-Spectral Implicit Function (SSIF), a physics-inspired neural implicit model that represents an image as a continuous function of both pixel coordinates in the spatial domain and wavelengths in the spectral domain. The authors validate SSIF on two challenging benchmarks, demonstrating its superior performance.

strengths:
1. This article presents a relatively novel approach to HSI spatial-spectral super-resolution, attempting to achieve it from the perspective of continuous physical space.
2. SSIF demonstrates good efficiency under low data conditions and converges more quickly during the training process.

weaknesses:
1. The description of the methods section in this paper is insufficient.	
2. This article's main innovation lies in combining the physical principles of spectral imaging with the spectral response functions of sensors to achieve HSI spatial-spectral super-resolution. Unfortunately, the authors do not provide sufficient explanation and analysis in the paper.
3. The experimental section lacks the latest relevant methods.

questions:
1. The physical principles of spectral imaging in SSIF’s model design lack deep analysis of their physical property. It is unknown whether the learned neural network reflects physical properties.
2. The author should provide some relevant implicit function-based methods for comparison.
3. The paper seems like a combination of INR, SwinIR, and ciaoSR in HSI spatial-spectral tasks. Could you elaborate on any challenges faced when designing SSIF? This clarification would help in understanding the complexity and novelty of your methodology. Actually, as mentioned in the weaknesses, the explanation of certain blocks in this paper lacks soundness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715095367

thread_type:
official_review

<thread_object>
creator: reviewer_tqqy

<content>
summary:
This work proposed Spatial-Spectral Implicit Function (SSIF), which generalizes neural implicit representations to the spectral domain as a physics-inspired architecture by incorporating sensors’ physical principles of spectral imaging.This method is quite appealing, but there are still many issues that need to be resolved.

strengths:
1.The figures used to analyze the algorithm are very striking, as shown in Figure 1.
2.Diverse methods were used to analyze the performance of each band in the images.

weaknesses:
1.Table 1 lacks a consistent number of decimal places; for example, 9.3, 13.2, and 27.3 have one decimal place, while the remaining values have two.
2. The latest experiment only includes one comparison algorithm for 2023; a comparison algorithm for 2024 should be added.
3.The authors conducted experiments on arbitrary channels with 𝐶>31 in CAVE, but the total number of channels in CAVE is only 31. My suggestion is to see if it is possible to recover the channel count from 𝐶<31 back to 31, and then perform comparative experiments.
4.Figure 19 could benefit from the addition of an error plot to better illustrate the comparison results

questions:
1.Is the comparison algorithm from the most recent year?
2.Channel analysis should be conducted under the guidance of the ground truth (GT); in other words, I would like to see the channels reshaped from C<31 to 31
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719463417

thread_type:
official_review

<thread_object>
creator: reviewer_nbsh

<content>
summary:
This paper introduces a neural implicit model that expresses an image as a function of continuous pixel coordinates in the spatial domain and continuous wavelengths in the spectral domain, achieving simultaneous super-resolution in both dimensions—an approach that has not been explored before. The study demonstrates substantial innovation, is well-supported by theory, and includes extensive comparative and ablation experiments. In summary, this paper presents impressive innovations and demonstrates substantial effort. However, *Issue 1* raise some doubts about the reliability of the experimental results. Moreover, since the performance in spectral super-resolution is more appealing, the related experiments need improvement (such as Issue 3). If the concerns I raised are addressed, I would consider raising the score.

strengths:
This paper introduces a neural implicit model that expresses an image as a function of continuous pixel coordinates in the spatial domain and continuous wavelengths in the spectral domain, achieving simultaneous super-resolution in both dimensions—an approach that has not been explored before. The study demonstrates substantial innovation, is well-supported by theory, and includes extensive comparative and ablation experiments.

weaknesses:
1.	Using the CAVE dataset as an example, CAVE contains 31 spectral bands, meaning it lacks ground truth (GT) data for cases where C > 31. Therefore, how was the PSNR for C > 31 in Figure 3 calculated? If the GT is obtained through interpolation, wouldn't the model's performance be limited to only approximating the performance of the interpolation?
2.	The paper explains how \(C\) is selected, but how is the channel dimension \(c\) of the HR-MSI (with a shape of \(H \times W \times c\)) determined? In Figure 3, the spectral bands from 1 to 7 are considered out-of-distribution. Is this because \(c\) is chosen from the range of 8 to 31? Was the downsampling performed by simply truncating consecutive spectral bands, or was it uniformly sampled across the range? The lack of clarity in this explanation makes it unclear why \(C_{\min} = 8\) was chosen instead of 1.
3.	For the out-of-distribution spectra, I would prefer to see a visual representation of the results, but this was not included in the paper.
4.	Since \(\Lambda\) can be specified, is it possible to achieve spectral compression, such as converting an HSI image into an RGB image?

questions:
1.	Using the CAVE dataset as an example, CAVE contains 31 spectral bands, meaning it lacks ground truth (GT) data for cases where C > 31. Therefore, how was the PSNR for C > 31 in Figure 3 calculated? If the GT is obtained through interpolation, wouldn't the model's performance be limited to only approximating the performance of the interpolation?
2.	The paper explains how \(C\) is selected, but how is the channel dimension \(c\) of the HR-MSI (with a shape of \(H \times W \times c\)) determined? In Figure 3, the spectral bands from 1 to 7 are considered out-of-distribution. Is this because \(c\) is chosen from the range of 8 to 31? Was the downsampling performed by simply truncating consecutive spectral bands, or was it uniformly sampled across the range? The lack of clarity in this explanation makes it unclear why \(C_{\min} = 8\) was chosen instead of 1.
3.	For the out-of-distribution spectra, I would prefer to see a visual representation of the results, but this was not included in the paper.
4.	Since \(\Lambda\) can be specified, is it possible to achieve spectral compression, such as converting an HSI image into an RGB image?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732609407277

thread_type:
official_comment

<thread_object>
creator: reviewer_iyg4

<content>
title:
Rebuttal

comment:
Thank you for ICLR's message regarding the ongoing discussion. I'd like to have a quality discussion, but the author seems to have given up on this. I regret this.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes Spatial-Spectral Implicit Function (SSIF), a neural implicit model that represents an image as a function of both continuous pixel coordinates in the spatial domain and continuous wavelengths in the spectral domain. The motivation of this study is articulated clearly. However, I agree with the reviews’ comments that the description of the methods section in this paper is insufficient. It is unclear what advantages the proposed method offers compared to other hyperspectral image super-resolution methods based on implicit functions. 
Based on the reviewer's average score, I believe that this paper is not yet ready for publication at ICLR.