Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to perform signal processing of a signal represented via a neural field directly, without the need to first sample the underlying MLP. To this end, the authors propose to express operators on top of the neural field as functions of the first- and higher-order derivatives of the MLP. Though the output is *not* a new, monolithic MLP parameterizing the output of the operator, but instead a linear combination of derivative MLPs, the output nevertheless retains the "resolution-invariant" property of the underlying neural field representation. 

strengths_and_weaknesses: # Originality
The paper is original and the proposed approach novel. To the best of my knowledge, no prior work on directly computing convolutional operators on top of neural fields without sampling exists. The treatment of how to approximate any shift-equivariant and certain group-invariant kernels as the linear combination of first- and higher-order gradients of a neural field is similarly novel. The authors propose and demonstrate a set of applications, ranging from image processing to a simple convolutional neural network, that have also not been tackled in this way before. All in all, this paper is highly original and was joyful to read.

# Quality
The paper is of high quality. Figures are informative to a degree that a single pass through the paper already provides high-level understanding of the method. A detailed read-through yields cool insights, such as the derivation of universal approximation of any convolutional filter, though that is, as the authors remark, not a new result. The quality of the experiments is similarly high, with relevant experiments that show-case the use-case of the proposed method - in particular, I enjoyed the low-level image processing tasks and image inpainting.

# Clarity
The paper is very clear, especially the figures serve to lay out the key contributions of the paper.

# Significance
The practical significance of the proposed approach seems to be limited for now: First, it requires one to already have access to a set of neural fields fit to the signal at hand. Second, the cost of evaluating the resulting network can be rather prohibitive, as it requires evaluating the stack of all derivative networks to decode a single pixel.

However, I am not overly concerned with the practical significance of the approach. In my opinion, the key significance of this paper lies in educating the neural field community about the fact that operators on neural fields can be expressed as functions of the derivative networks. Further, I am convinced that this paper will inspire work that will push this idea further, maybe eventually to the point where we will be able to compute functions of neural fields directly in the weight space.

questions: Could the authors comment on the effective (not theoretical) receptive field of the operators that can be parameterized by the proposed approach? In particular, SIREN is known to be a highly local INR: the directions in weight space to change the colors of pixels even a small distance apart are basically orthogonal (increasingly so with higher frequencies). It would seem that this "local support" also somehow impacts the receptive field of a convolutional operator expressed via the gradients of the SIREN?

Do the authors have a way in mind in which they might lift the limitation that their approach requires a-priori fitting of all the neural fields? Note that I do not expect this to be the case, any insight is simply appreciated!

Minor comments:
- line 49: "are loaded from the INR being processing..." - this sentence does not make sense to me. Maybe being -> during?
- line 51: "even though we are not able to (perform) surgery"

limitations: The processing remains expensive, in that the output of the proposes INSP-Net is not a single neural field, but rather, a linear combination of derivative operators, each of which is expensive to query. 

The present effort still requires recovering all neural fields of all the signals upfront. 

The authors certainly discuss these limitations, but I would have expected a "limitations" section where these issues are made explicit.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Motivated by recent advances in implicit neural representation (INR) for images, this paper seeks to develop an approach to modify learned INR *"without decoding”*. Specifically, the paper proposes training an additional network which operates on the high-order spatial derivatives of the original INR previously fitted for some image. This extra neural network takes in the original INR prediction, as well as its high-order derivatives w.r.t. the input coordinates (hence the spatial derivative); the objective of this network depends on the specific task, and the paper discusses tasks such as denoising, deblurring, inpainting, and classification.
strengths_and_weaknesses: Strengths:

**Clear and reasonable motivations.**
INR has achieved great success in accurately fitting images, and it is therefore well-motivated to explore possibly modifying the learned INR for further downstream tasks. It is also admirable that the paper points out the potential benefits of improving INR based on ideas from the field of digital signal processing

**Variety of tasks.**
This paper presents a wide range of applications, such as edge detection, blurring, deblurring, denoising, inpainting, and classification. It is helpful for the community to be aware of the huge potential of INR of images.

Weaknesses:

**Experimental design.**

* The title suggests a general framework of signal processing, and the teaser figure (Fig. 1) shows a 3D mesh and explicitly states “geometric processing”, but the paper is actually only about image processing. 

* The paper has no experiment on geometric processing on signed distance fields, despite the fact that DeepSDF [10] is one of the most prominent examples of successful INR applications. 

* The paper has no experiment on signal processing of time series signals (e.g. audio or speech), which is arguably the most representative data domain for the entire field of digital signal processing. The SIREN paper [27] that this paper is based on has covered audio signals, so it is not a foreign type of data for INR.

* It is completely fine if a paper only focuses on images, but the title and its first figure should at least appropriately reflect the content and scope.

**Validation rigor.**
* The paper does indeed cover a lot of image processing applications, but it lacks sufficient rigor in validating the proposed approach.
The paper presents quantitative metrics (PSNR/SSIM) for five selected images only (Figures 4 to 7). Those provided in the supplement do not contain any quantitative metric, and even if they do, the paper should still provide average results across a large dataset to avoid cherry picking.
* In the denoising/deblurring/inpainting experiments, the INR is first fitted against the noisy/blurry/masked image before then processed by the task-specific network. The results seem to suggest that this task-specific network can successfully denoise/deblur/inpaint from the corrupted input image. However, the paper does not show the decoded image from the original INR. It is well-known that INR itself can at least do denoising [30], so it is not clear how much the proposed training actually improves from the INR simply fitted against the corrupted input image.
* In the classification experiment, it is not clear how only training a 2-layer depthwise CNN is a “fair comparison”. The paper does not provide the number of network parameters, nor does it explain why it does not consider training a small MLP.
* Across all experiments, it is not clear what number k, or how many high-order derivatives, should one use. The paper presents no analysis on the impact of having more high-order derivatives. It is not even clear if one really needs high-order derivatives.

**Literature review.** \
*Some related papers are not correctly interpreted.*
* While discussing “existing approaches" for “editing on INRs”, the paper mentions (Line 36 to 37) that “another main direction” and only references three works, [27], [28], and [29]. However, [28] and [29] are both from 2017, well before the recent introduction of INR, whose main characteristic is the “coordinate-to-value mapping” (Line 25). In fact, [28] makes no use of neural networks, and does not even mention the phrase “neural network” at all; [29] is about using GAN to generate 3D voxels, hardly related to INR either.
* While introducing FFM (Line 89 to 98) and Eq. (2), the notation seems to suggest that “positional encoding” is merely a sine function (Line 97 to 98), and ignores the fact that FFM [30] introduces the Gaussian mapping and uses it by default.
* Section 4.2 discusses the supposedly related works on “editing the reconstructed 3D scenes”. This paper only focuses on image processing, so the relevance of this subsection is really weak.

*Some highly related papers are not included.*
* Using the derivatives of INR has been explored by [a] (with similar computational graph discussion) in the context of faster INR-based rendering. 
* Using INR for downstream applications (e.g. classification) has been explored by [b].
* Using INR for image and video compression has been also explored by [c] (with a PDE-based positional encoding) in the context of light fields.
* [d] introduces a technique that significantly accelerates the fitting of INR on 2D images.
* [e] introduces a technique for improving the representation efficiency of INR of images via reducing the parameter count.

* These works contain far higher relevance to this paper, compared to the 7 referenced papers on 3D shapes (there’s no 3D shape in this paper) and the other 7 referenced papers about 3D GANs

[a] Autoint: Automatic integration for fast neural volume rendering. Lindell et al. CVPR 2021. \
[b] From data to functa. Dupoint et al. ICML 2022. \
[c] Signet: Efficient Neural Representation for Light Fields. Feng et al. ICCV 2021. \
[d] Learned Initializations for Optimizing Coordinate-Based Neural Representations. Tancik et al. CVPR 2021. \
[e] Meta-learning Sparse Implicit Neural Representations" Lee et al. NeurIPS 2021. 


questions: * Why is the title about general signal processing but the paper only deals with images? 
* What happened to the geometric processing and the 3D shape shown in Fig. 1?
* For Fig. 4 to 7, why not show the initial decoded images after fitting the INR on the corrupted images? How much does the extra network actually help?
* Isn’t the INSP-Net with the “A” kernel just a fully connected MLP (Line 119)? Why would stacking multiple INSP-Net be called INSP-ConvNet? Is it because MLP is equivalent to 1D Convolution? By this logic, isn’t the original INR also a “ConvNet”?
* What does “modify an INR without explicit decoding” mean? Doesn’t the proposed method need to decode the INR (Eq. 3) by evaluating the “acquired INR” on every single pixel “x”? Isn’t that decoding already?
* If we need to evaluate every pixel “x” with the INR, before feeding it to the extra network, what benefit does it bring over simply using the decoded image?
* Why not show the inference speed / network run-time? How much slower would it be due to computing the high-order derivatives?

In particular, it would be critical to elaborate the "without decoding" claim, clarify why the proposed method is not also decoding from the INR, and how the proposed INSP-ConvNet really is **convolutional** instead of just **multiple MLPs**. These would determine the validity of the main claims and conceptual contributions of this paper, no matter how good or bad the experimental results are.
limitations: The authors have not really addressed them. Discussion about at least the limitation of a potentially slower run-time should be expected in the revision.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper studies implicit (and continuous) neural representations (INRs) of data defined on discrete lattices. In particular, the authors present a way of performing different operations on these INRs directly without the need for "decoding" them by parametrizing these operators with learnable operators on these representations. The authors demonstrate these ideas on Image denoising, deblurring (and blurring), inpainting and classification.
strengths_and_weaknesses: + This reviewer likes the general idea of this paper. While INR are becoming increasingly popular, it's not obvious how one can perform operations on these continuous elements that are similar to the discrete operations defined over lattices. This paper proposes the first solution to do this (to the best of my knowledge).
+ The method is intuitive and demonstrated on a series of image processing tasks, from low to higher level.

- While the general idea is intuitive, the text is not very clear. There are several typos and general English usage issues that are cumbersome while reading (see below).
- While this demonstrates that these operations can be *approximated* on INRs, the quality of the solutions is not very satisfying. As a result, the advantages of working on the INR domain are not very appealing.
questions: - In Eq. (1), did the authors mean \arg\min instead of \arg\max?
- Notation is confusing, as the authors use m and M to denote several things: m is the dimension of the discrete input lattice; e.g. m=2 for images, and M denotes the number of discrete measurements of the continuous function. However, line 117 denotes $x\in R^m$ - shouldn't it be $x\in R^M$? At the same time, Eq. (3) then makes use of $M$ to define the dimension of the domain of the operator $\mathcal A$, which is independent of $m$ and $M$ and depends on the order of the computed derivatives.
- One of the main theses of this paper is that common signal processing operations can be instead defined on INRs, which the authors attempt to demonstrate. However, all of these operators (with the exception of edge detection") are defined only as parametric functions learned from data in a supervised way - as opposed to the original way of defining these operators, "from scratch". This should be clarified, because while gradients can be obtained "in closed form", these operators cannot.
- This reviewer believes that while the idea proposed in this work is nice, the results are far from satisfying: 

i) the denoising performance seems quite poor, and the 2 db difference with DnCNN looks very strange given the (inverse) difference in SSIM. Indeed, the INSP-Net image looks considerably more noisy. Moreover, while it's fine comparing with DnCNN, this method is by now 5 years old, and much better algorithm exist (see the methods used for comparison in [1] for a good subset of them). Generally speaking, the denoising performance does not look good and appears far below normal current methods (even below previous methods based on non-local means, dictionary learning, and more).

ii) Likewise, the deblurring performance is also not very satisfying: note the considerable noise introduced in the sky area shown on Fig. 5. The same holds for inpainting (very high levels of noise are introduced in the sky).

iii) Importantly, image processing results are just demonstrations and numbers are presented for only those individual examples - no numbers are reported over the entire dataset. 

iv) I also appreciate the comparison on MNIST and CIFAR-10, but such low numbers are concerning. Recall that simple linear models (e.g. PCA +SVM) can better than 87%. 

- In line 230, the authors mention that "nonlinear activations and normalizations are naturally element-wise functions", but the authors never state what these are. What are they?
- Throughout the paper, a supervised learning setting is never presented, and so labels are never defined. Thus, it's unclear what the authors mean by "semantic labels y_i" in line 235. Only later in the experimental section one understands what labels the authors are referring to.
- I appreciate the result in Theorem 1, although it's novelty is very limited: indeed, differential operators are shift invariant, and any function defined on norms of vectors is rotation invariant. 
- It is unclear why the authors mention that the differential operators form a *wavelet* basis - do these operators satisfy the properties that wavelet must satisfy?
- In line 204, the authors comment that their operators can simulate *exact* convolutions. This seems incorrect to me, as the exact equivalence holds for an infinite sequence, and thus only a finite approximation is possible in this framework.
- Lastly, there are many errors that make reading quite difficult. A non exhaustive list (just the first page) includes:
  - "Implicit Neural Representations (INR) encoding continuous multi-media data..." -> encodes?
  - what are "agnostic" parameters? (line 4)
  - ".. can be computed analytically and invariant to translation" -> invariant to translation, line 11.
  - "presented" -> represented (line 23)
  - "study to encoder" -> study how to encode?
  - what do the authors mean by "parameter stored" in line 33?



[1] Tu, Zhengzhong, et al. "Maxim: Multi-axis mlp for image processing." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.
limitations: The authors did not address negative societal impacts nor the limitations of their contribution.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper introduces a novel signal processing framework named INSP-Net that directly operates on INRs without explicit decoding. The INSP-Net can process INRs by combining high-order differential operators. The authors prove any continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators and empirically validate the effectiveness of the proposed method across both low- and high-level vision tasks.
strengths_and_weaknesses: Strength: 

1. The idea is novel. The proposed method attempts to directly process INRs by utilizing high-order differential operators. Theoretically, the authors prove continuous convolution filter can be uniformly approximated by a linear combination of high-order differential operators. Besides, extensive experiments are conducted to validate the effectiveness across various vision tasks.
2. The paper is well organized and easy to be understood.

Weakness:

1. From experiments, it seems that INSP-Net operating on INRs cannot exhibit superiority against convolution operating on pixels. For example, edge detection results of other filters seems contain more detailed textures than that of INSP-Net. In Figures 4 and 5, although the PSNR score is higher, the processed images still contain obvious noise or other artifacts. As for image classification, why is Depthwise CNN used as baseline rather typical CNN? 

2. There are some typos in the paper, e.g. Line 77 “sought to to”

questions: See weakness.
limitations: The authors points out some future works without adequately discussing limitations and negative societal impact.
review_title: null

Meta-review: metareview: The paper proposes a framework to perform signal processing tasks on a signal represented with an implicit neural representation directly in the representation space, without the need to instantiate the signal. 

After the rebuttal period, all reviewers recommend acceptance. 

In particular reviewer 1Yx6, an expert on the topic, finds the idea original, and the quality and clarity of the paper to be high. The reviewer finds that while for now the significance is limited (since working as proposed in the representation space is computationally prophibitavely expensive), this is not a major issue, since the the paper is likely to inspire work that will push the idea further. 
Reviesers edqT and qGk1 also liked the general idea of the paper and find the proposed method to directly perform operations on the representation space of implicit neural representations to be novel and interesting

Reviewer Uh51 initially identified a few issues regarding experimental design, the validation, and on the included literature. The main concern regarding the experimental design of the reviewer was that the paper focuses on images (and not signal processing tasks more broadly), which I don't consider a shortcoming, due to the importance of image processing tasks. The concerns on the validation issues have been addressed as well, and the reviewer raised their score. 

I recommend acceptance of the paper. 

confidence: Certain
Decision: Accept