Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a patch-based framework for solving inverse problems. The proposed approach is able to achieve reasonable performance on several tasks.
strength_and_weaknesses: Strength:
1. The paper is well-written. The proposed approach is simple and easy to follow.
2. The empirical results are good. The model is able to achieve comparable performance while using fewer parameters than state-of-the-art models on several inverse problem tasks.

Weakness:
1. Section 3 is hard to follow. For instance, what does "sample discrete images from a powerful convolutional generator" mean? Do you mean to generate an image using the pretrained generator? What does "discrete image" mean? Does it have a fixed resolution?
2. In section 3.1, it would be better to define the input of ST formally (e.g., coordinate, image) instead of just mentioning its dimension. 
3. In section 3.2, does it mean given a pretrained generative model, you apply FunkNN to the output from the generated samples of the model? Why not directly apply it to existing low-resolution images so that the quality of the results will not be bounded by the pretrained generator? 
4. The notation is section 4.1 is hard to follow. For instance, a new model IN$_\theta$ is introduced. What is this model doing? It's better to keep the notation consistent. The parameter $j$ is also not formally defined, what are $u$ and $x$?
5. The structure of the paper can be improved. For instance, some results are presented in section 4, but there is another section called Experimental results (section 5). 
6.  Besides the super-resolution results (table 1), there is no comparison with state-of-the-art baselines for the other tasks.

clarity,_quality,_novelty_and_reproducibility: The main approach is simple and easy to follow. However, the method and experiment sections are very hard to follow, with undefined notations and unexplained experimental settings. The contribution of the paper is mainly on the empirical side, and thus more thorough comparisons with state-of-the-art approaches (both quantitatively and qualitatively) on each task are needed. The reproducibility is also unclear.
summary_of_the_review: This work proposes a simple approach to solving inversion problems. However, the presentation of the paper can be improved, and more empirical results and comparisons with state-of-the-art baselines are needed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a method for image super-resolution that resembles an implicit function, but has strong inductive biases in the underlying architecture. To predict the super-resolved value for a particular coordinate, they interpolate a small 2D patch centered at that coordinate from the original image (reminiscent of RoI-Align from the segmentation literature), and pass this through a CNN.
They show competitive super-resolution results on natural image datasets, and show that their method produces plausible spatial derivatives that be used in a medical imaging applications.
Compared to other super-resolution methods, theirs seems to be much more efficient in terms of computational complexity and parameter count.
strength_and_weaknesses: Strengths:
1. The proposed method is well-motivated and clearly explained. 
2. The method is sensible and has a number of desirable properties.

Weaknesses:
The main weakness is that the experimental evaluation could be more extensive. For example:
1. In "Related Work", the authors discuss GAN-based super-resolution methods, but do not include any such methods as baselines in their experiments.
2. I can imagine that the proposed method could be successfully applied to density estimation. For example, as a cascaded density model similar to what [1] does with diffusion models.

I wonder if more emphasis on directions like these would make the paper more compelling to a broader audience. In contrast, the discussion about OOD generalization feels a bit weak (proposed method not substantially better than baseline), and the image derivative experiments feel a bit niche (I personally do not find these particularly exciting, and imagine that much of the generative modeling community feels similarly).


[1] Cascaded Diffusion Models for High Fidelity Image Generation. Ho et al, https://arxiv.org/abs/2106.15282.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. The proposed approach is moderately novel and well-executed. The authors do not provide code, but I think the paper could be reproduced from the details in the text.
summary_of_the_review: I do think that this paper would be stronger if the authors focused on slightly different directions (see "Weaknesses" section), but overall I feel that it is a solid incremental contribution in super-resolution.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces FunkNN, a super-resolution architecture built on a two-stage process to model images as a continuous function.

In order to predict the color value for a given (x, y) location, first a Spatial Transformer with a bilinear kernel is used to extract a PxP patch from the image whose coordinates are centered on the point (x, y), and then a CNN is used to predict the color value at the chosen point. In a sense, one could tell that FunkNN is built by using a CNN to sharpen an initial bilinear upscaling.

The model as a whole allows to both compute spatial gradients in the image, and back-propagate gradient information back to the low-resolution image. This makes it possible to combine FunkNN with any off-the-shelf low-resolution generative model. The paper illustrates this by using such models as a prior over in images in problems of high-resolution image reconstruction from partial data: reconstructing an image from its 20% highest intensity spatial gradients, or in the context of limited-view CT reconstruction.

On the super-resolution task, FunkNN is shown to be competitive to LIIF, a state of the art model, on x2, x4 and x8 tasks, while being a much smaller model in terms of parameter count.
strength_and_weaknesses: **Strengths:**

The paper proposes a simple and elegant model, showing the possibility to build continuous super-resolution methods with *relatively* small models. The efficiency of FunkNN is demonstrated on multiple different tasks, illustrating the various possible applications of the model. The construction of the model is well motivated.

The core idea of the paper is in my opinion the fact that it delegates the "interpolation" part of the problem to a spatial transformer which achieves bilinear interpolation over the whole patch, rather than trying to train a NN to do it. It does seem like a good inductive prior to give to the CNN for the final prediction.

**Weaknesses:**

First of all, I want to say that I am not deeply familiar with the problem of super-resolution, and I refined my understanding of it using the references from the paper. My attention went in particular to the LIIF model (reference Chen et al. 2021 from the paper), which seems to me to have quite a lot of similarities with the proposed FunkNN.

The models are clearly different:
- LIIF passes the whole low-resolution image in a CNN encoder, then extracts a small patch from that encoded representation, and uses a NN to interpolate to the target pixel value from that small patch.
- FunkNN extracts a small patch from the low-resolutions and at the same time interpolates it to align its pixel grid to the target coordinate using a spatial transformer, and then use a CNN to predict the target pixel value from that aligned patch.

However, as far as I can tell, the main properties of FunkNN advanced by the paper (being able to compute spatial gradients of the image and being able to back-propagate gradients to the low-resolution input) are shared by LIIF. This means that for all the inverse problems described in section 4, on which the performance of FunkNN is illustrated, it would be possible to make a performance comparison with LIIF.

Furthermore, FunkNN like LIIF represents the image prediction as a continuous function of pixel values. This means that, like LIIF, FunkNN could theoretically be able to generalize to scale factors different than the ones it was trained on. LIIF for example is trained on x2, x3 and x4 factors, and shown to generalize well up to x30 from that. No such attempt at generalization is made on FunkNN.

The paper points the large number of parameters of LIIF. From what I understand, it comes mostly from the large CNN encoder LIIF uses, that works on 48x48 patches. However, nothing in the structure of LIIF would prevent using a much smaller encoder working on smaller patches (for example 8x8 like FunkNN in the experiments). Doing so could probably bring LIIF to a number of parameters comparable with FunkNN.

The core of my concern is thus this: while conceptually different (interpolation done by a NN vs. done by the spatial transformer), LIIF and FunkNN are two models with very similar sets of capabilities. I thus find it a lack in the paper that the two models are not compared in a more in-depth way: the inverse problems FunkNN is evaluated on should be accessible to LIIF, and the arbitrary scaling of LIIF should be accessible to FunkNN. I find this especially lacking given that, from a conceptual point of view, it seems to me that the good inductive bias of FunkNN has the potential to make it very competitive with LIIR on all those tasks.
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written presents the proposed model and test setting with all the necessary details to reproduce its experiments.

The paper take advantages of previous work (spatial transformers) and re-combines it in a novel and well-justified way for the task of super-resolution.
summary_of_the_review: I believe this is overall a decent paper. The proposed model is well-justified, simple and elegant, and experimentally shown to be applicable to several relevant tasks.

However I think that to truly be a good paper it is lacking an in-depth comparison with other models of the literature that can be applied to the same tasks (I focused on LIIR from the references of the paper, but it's entirely possible other models could also be applicable here).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper introduces FunkNN, a continuous super resolution module for image-like data. Given a low resolution image and an image coordinate x (which may not lie on the low resolution grid), FunkNN returns the predicted RGB value at that coordinate, effectively leading to a continuous image representation, where pixel values can be probed at arbitrary coordinates in R^2. To achieve this, the authors propose to use a spatial transformer which takes as input an image and a coordinate and extracts a (linearly interpolated) patch centered on this coordinate. The patch is then passed through a small CNN which returns the predicted RGB value at the coordinate location. The authors perform various interesting experiments on image super resolution and inverse problems involving image derivatives.
strength_and_weaknesses: **Strengths**

- The paper tackles an interesting and useful problem.
- The proposed model has various interesting properties. For example, the model can be trained with very low memory since it only requires patches and the continuous nature of the model allows for easy estimation of derivatives.
- The inverse problem applications and experiments are interesting.
- The out of distribution experiments are interesting. It is nice to see that a model trained on CelebAHQ can generalize to LSUN bedrooms.
- The figures are nice and help with the general understanding of the paper.


**Weaknesses**

- In general, the discussion of related work is lacking and the paper is not very well situated in the literature. In particular, the proposed model is very closely related to LIIF which is briefly mentioned as a baseline. However, from reading the FunkNN paper this is not clear. It was not until I had a look at the LIIF paper that I realised how closely related these two ideas are (using local information to upsample images at arbitrary resolutions). The differences need to be discussed in more detail and related works should be properly acknowledged. In addition, generative models of continuous images are briefly mentioned but not discussed in detail (and a reference to [1] is missing). Further, while works like Chen & Zhang 2019 and Dupont et al 2021 are mentioned as performing poorly for grid-based generative tasks, it is worth noting that these models are also much more general and can be applied easily on various data modalities.
- As the proposed method is a super resolution module and not a generative model, I believe it makes more sense to focus on super resolution in the related work section instead of generative models (which currently takes up much more space). In general, the model does not feel very well situated in the super resolution literature.
- There is very little quantitative evaluation of the model. For example, for the inverse problem experiments there is not a single quantitative metric (even though e.g. just measuring reconstruction error should be simple) and only a handful of quanlitative comparisons. Further, there is little discussion of what baselines could be relevant for these experiments.
- The authors discuss generative models throughout the paper, but do not include any experiments or evaluation for this. For example, it would be interesting to see how FunkNN applied to some low res generative model would compare to other generative models of continuous images.
- While the writing is generally clear, there are quite a few typos and some missing information. For example, in Table 1 it is not clear on which dataset the model was evaluated. Is the performance averaged across all examples on some test subset of CelebAHQ?


**Nitpicks**

- It seems like a lot of citations are missing brackets. I would recommend using `\citep{}`.
- The authors often refer to "Implicit Neural Networks". However, it seems like the two dominant terms in the literature now are "Implicit Neural Representations" or "Neural Fields", so I would go with one of these options to be consistent with the literature.

[1] Adversarial Generation of Continuous Images, Skorokhodov et al, CVPR 2021
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The paper is generally quite clearly written and the figures are nice. A few details are missing in the results section.

**Quality**

The paper is fairly high quality. The method and experiments are both interesting and the paper is quite well written.

**Novelty**

The proposed method is reasonably novel, but is closely related to exisiting methods.

**Reproducibility**

The details and explanations provided in the paper seem to be sufficient for reproducing the results in the paper.
summary_of_the_review: Overall, I think this is a fairly good paper, which may be of interest to the ICLR community. The method and experiments are interesting. However, I do not think the paper is very well situated in the literature and also lacks some experimental evaluation. I therefore believe the paper in its current state is borderline, but I am willing to update my scores if some of the concerns are addressed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper examines continual signal generation through a signal-processing perspective. They propose FunkNN, a CNN that reconstructs continuous images at arbitrary coordinates and, when combined with a discrete generative model, it can act as a "prior in continuous ill-posed inverse problems." The authors further examine results for high-quality continuous image generation, out-of-distribution performance, and inverse problems.

The reviewers generally found the paper well-written overall, the idea intuitive and with sensible properties, and with good empirical results. There are quite a few gaps in the writing that is pointed out (e.g., related work, lack of clarity in parts of Section 3) and the lack of comparison to state-of-the-art baselines outside the superresolution task. The writing details are easily fixable (and have been according to the authors during rebuttal), and several new baselines have been added during the rebuttal.

All reviewers leaned toward accept and I agree with their consensus.

Note: There was an ethics flag by Reviewer kWFU on potential bias during superresolution. I think this is something independent of the specific paper's contributions and should be handled when applying the method to certain datasets.
justification_for_why_not_higher_score: Focus is somewhat niche, so harder to see its interest to a broader audience. It's not clear the extent to which the authors fully resolved the reviewers' shared concerns about writing clarity and baselines (unfortunately, little participated in the discussion).
justification_for_why_not_lower_score: The reviewers generally found the paper well-written overall, the idea intuitive and with sensible properties, and with good empirical results.
Decision: Accept: poster