PIN: Prolate Spheroidal Wave Function-based Implicit Neural Representations

Authors: Dhananjaya Jayasundara, Heng Zhao, Demetrio Labate, Vishal M. Patel
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Prolate Spheroidal Wave Functions, Implicit Neural Representations, MLPs

Abstract:
Implicit Neural Representations (INRs) provide a continuous mapping between the coordinates of a signal and the corresponding values. As the performance of INRs heavily depends on the choice of nonlinear-activation functions, there has been a significant focus on encoding explicit signals within INRs using diverse activation functions. Despite recent advancements, existing INRs often encounter significant challenges, particularly at fine scales where they often introduce noise-like artifacts over smoother areas compromising the quality of the output. Moreover, they frequently struggle to generalize to unseen coordinates. These drawbacks highlight a critical area for further research and development to enhance the robustness and applicability of INRs across diverse scenarios. To address this challenge, we introduce the Prolate Spheroidal Wave Function-based Implicit Neural Representations (PIN), which exploits the optimal space-frequency domain concentration of Prolate Spheroidal Wave Functions (PSWFs) as the nonlinear mechanism in INRs. Our experimental results reveal that PIN excels not only in representing images and 3D shapes but also significantly outperforms existing methods in various vision tasks that require INR generalization, including image inpainting, novel view synthesis, edge detection, and image denoising.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729129096658

thread_type:
official_review

<thread_object>
creator: reviewer_bt7n

<content>
summary:
This paper argues that the Gaussian and Gabor wavelet activation functions cannot achieve the optimal space-frequency trade-off, and thus may not effectively capture distant relevance. Motivated by this, the paper proposes using the Legendre polynomial numerical estimation of the Prolate Spheroidal Wave Function (PSWF) as the activation function, which has been proven in previous work to offer the optimal space-frequency trade-off. The authors demonstrate improvements through experiments on various tasks, including image regression, neural radiance fields, and so on.

strengths:
1. The motivation presented in the paper is compelling and provides a solid foundation for the proposed approach.  
2. This paper offers a insightful critique of previous works, identifying limitations in existing activation functions and highlighting the need for improvement.  
3. The experiments encompass a diverse range of tasks, allowing for a more comprehensive and thorough comparison. This diversity enhances the validity of the findings and demonstrates the effectiveness of the proposed method across different domains.

weaknesses:
1. The exact implementation of the numerical estimation of the Prolate Spheroidal Wave Function lacks clarity in the main text. For instance, the approximation order is not explicitly stated, and the impact of the approximation order on the computational complexity (in terms of space/time consumption) and the model's performance is not clearly explained.
2. The experiments in tasks, such as Occupancy Field Representation and Neural Radiance Field, appear to be conducted on a subset of the entire dataset, which may undermine the persuasiveness of the results.
3. The presentation could be improved by refining certain details, such as employing the vector graphics and employing the \autoref{} command for figure citations to ensure consistency and clarity.

questions:
1. Could you clarify the approximation order used for the Prolate Spheroidal Wave Function (PSWF)? Additionally, does this approximation affect the theoretical properties of the PSWF? The paper mentions that PSWF possesses infinite support in space, but it remains unclear whether the finite-order approximation may potentially diminish this property.
2. I observed that a bandwidth parameter \(c\) governs the frequency. Could you provide guidance on how to select this parameter effectively? Does it influence model performance, and if so, what strategies should users follow to determine an optimal value, especially for tasks in continuous domains like Neural Radiance Fields (NeRF), where there might be no frequency bounds, unlike the image regression scenario?
</content>
</thread_object>

</thread>
<thread>
creation date:
1729592764491

thread_type:
official_review

<thread_object>
creator: reviewer_1n58

<content>
summary:
This paper introduces the use of prolate spheroidal wave functions (PSWF) for implicit neural representation (INR). By employing PSWF as the activation function in INR, the method excels not only in representing images and 3D shapes but also significantly outperforms existing approaches in various vision tasks that rely on INR generalization, including image inpainting, novel view synthesis, edge detection, and image denoising.

strengths:
1. Extensive experiments across various vision tasks demonstrate the effectiveness of PSWF.

2. A comprehensive theoretical analysis highlights the advantages of using PSWF.

weaknesses:
1. When comparing different INR methods, do you ensure that the same parameters are used? Could you provide the specific parameters for each INR method?

2. I am curious about the decoding complexity of the PSWF-based INR. Could you provide the decoding speed or time for the different INR methods?

3. Besides the vision tasks mentioned in the paper, can PSWF also improve performance in image super-resolution?

questions:
I noticed that the authors use initial INR methods as their baselines. However, there are several approaches aimed at enhancing the expressivity and generalizability of INR, including improvements in training strategies [1] and input signals [2]. Could PSWF be applied to these methods to further enhance the representation performance of INR?


[1] Improved Implicit Neural Representation with Fourier Reparameterized Training, CVPR 2024

[2] Disorder-invariant implicit neural representation, CVPR 2023.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730529519675

thread_type:
official_review

<thread_object>
creator: reviewer_yext

<content>
summary:
The paper proposes a novel Implicit Neural Representation (INR) utilizing Prolate Spheroidal Wave Functions (PSWFs) to improve performance and generalization in computer vision tasks. By leveraging the optimal space-frequency domain concentration of PSWFs, the proposed method addresses the noise artifacts over smoother areas and poor generalization of existing INRs, demonstrating superior results in image inpainting, novel view synthesis, edge detection, and image denoising.

strengths:
1. The paper clearly explains the limitations of current INRs and proposes a novel activation function (PSWFs) for INR to overcome these limitations.
2. The localization and expressivity properties of PIN have been theortically proven.
3. The results validate the effectiveness of the proposed INR across various tasks. The authors not only show the good representation ability of PIN for various signals, such as image, Occupancy Filed, and NeRF, but also show that PIN has a very good performance on image Image Inpainting.

weaknesses:
1. The activation function seems to be rather computationally heavy, making it necessary to report its speed in applications. 
2. As I have reviewed the previous submission of this paper, I notice that the Fig.4 is replaced with a scene with better results. I wonder how these new results are obtained and why these results are not attached in previous submission? Are they obtained by tuning parameters for each scene specifically?
3. Since the first NeurIPS submission of this paper, several new INRs have been proposed, including FINER (and its extension, FINER++) and H-SIREN. However, these new INRs are not cited or compared within the current manuscript.

questions:
See Weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730604948310

thread_type:
official_review

<thread_object>
creator: reviewer_8ygc

<content>
summary:
The paper proposes Prolate Spheroidal Wave Function-based Implicit Neural Representations (PIN), an effective representation inspired by Prolate Spheroidal Wave Functions (PSWFs). The proposed PIN outperforms other INR baselines in various reconstruction tasks.

strengths:
- The proposed PIN outperforms other INR baselines in various reconstruction tasks, including image reconstruction, image inpainting, and occupancy field and NeRF.
- Detailed ablation studies are conducted.

weaknesses:
- The metrics provided in the paper are only evaluated on a few individual examples. I appreciate the evaluation on the Kodak Lossless
True Color Image Dataset. However, it only contains 24 images. It is ideal to perform larger-scale evaluations, e.g. calculating the mean PSNR for a thousand images. For example, one may consider using the DIV2K dataset [1]. 

[1] Agustsson E, Timofte R. Ntire 2017 challenge on single image super-resolution: Dataset and study[C]//Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 2017: 126-135.

questions:
- I am interested in the runtime of the proposed PIN. Will the new formulation hurt the speed of INRs? Is there a trade-off between quality and runtime?
- How is the experiment conducted for the Image Inpainting task? Are the missing pixels marked as black and fed into the INR? Or are those coordinates masked out and not used during training? Are the pixel masks provided to the INR?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732729240080

thread_type:
official_comment

<thread_object>
creator: area_chair_yjye

<content>
title:
Please take a look at the authors' rebuttal and start a discusssion if needed

comment:
Dear Reviewers,

Thanks for your contributions in reviewing this paper.

As the author-reviewer discussion deadline is approaching, please could you take a look at the authors' rebuttal (if not yet) and see if it addressed your concerns or if you have any further questions. Please feel free to start a discussion.

Thanks,

AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
In this paper, the authors presented a new activation function for implicit neural representations (INRs) -- Prolate Spheroidal Wave function (PSWF). Motivated by the challenges faced by existing INRs and their struggle to generalise to unseen coordinates, the authors introduced the PSWF-based INRs, termed PIN, leveraging the optimal space-frequency domain concentration of PSWFs. Experimental evaluations over a few different vision tasks (including image inpainting, novel view synthesis, edge detection, and image denoising) show the effectiveness of the proposed PIN. The strengths of this paper include:
- The proposed method was well-motivated, with a clear and solid foundation for the approach.
- The paper did a good job in analysing and explaining the limitations of existing INRs, which could provide insights for following research in this direction.
- The proposed method was backed up with a comprehensive theoretical analysis.
- An extensive experimental analysis covering several vision tasks, showing the effectiveness and validity of the proposed method.

The weaknesses of this paper include:
- The rationale for employing INRs to address low-level problems (as raised by a reviewer during 2nd phase discussion, details see below). Forward training-based models typically offer better generalisation capabilities.
- Potential issues with the infinitely differentiable property of cubic spline that was used for the PSWF in this paper (also confirmed in the 2nd phase).
- Insufficient evaluations, computational complexity concern, issues with the results, and missing comparison to recent related works.

Most of the concerns/weaknesses were well addressed in the rebuttal phase and the reviewers also acknowledged that. Overall, this paper presented an interesting idea with insights into INRs, and the AC think this would be of interest to a group of audience in ICLR. As a result, the AC is happy to recommend an Accept, but the authors are **highly suggested to incorporate the further provided evidence and clarifications during the discussions to the final version, and please also merge the Appendix (currently in a separate file), which has some essential analysis and results, to the end of the main paper.**