FunkNN: Neural Interpolation for Functional Generation
Authors: AmirEhsan Khorashadizadeh
Anadi Chaman
Valentin Debarnot
Ivan Dokmanić
Venue: ICLR 2023 poster (2023)

Abstract: Can we build continuous generative models which generalize across scales, can be evaluated at any coordinate, admit calculation of exact derivatives, and are conceptually simple? Existing MLP-based architectures generate worse samples than the grid-based generators with favorable convolutional inductive biases. Models that focus on generating images at different scales do better, but employ complex architectures not designed for continuous evaluation of images and derivatives.
We take a signal-processing perspective and treat continuous signal generation as interpolation from samples. Indeed, correctly sampled discrete images contain all information about the low spatial frequencies. The question is then how to extrapolate the spectrum in a data-driven way while meeting the above design criteria. Our answer is FunkNN---a novel convolutional network which learns how to reconstruct continuous images at arbitrary coordinates and can be applied to any image dataset. Combined with a discrete generative model it becomes a functional generator which can act as a prior in continuous ill-posed inverse problems. We show that FunkNN generates high-quality continuous images and exhibits strong out-of-distribution performance thanks to its patch-based design. We further showcase its performance in several stylized inverse problems with exact spatial derivatives.

Published as a conference paper at ICLR 2023
FUNKNN: NEURAL INTERPOLATION FOR FUNCTIONAL GENERATION
AmirEhsan Khorashadizadeh", Anadi Chaman', Valentin Debarnot" and Ivan Dokmanić* "Department of Mathematics and Computer Science, University of Basel i Coordinatee Science Laboratory, University of Illinois at Urbana-Champaign
ABSTRACT
Can we build continuous generative models which generalize across scales, can be evaluated at any coordinate, admit calculation of exact derivatives, and are conceptually simple? Existing MLP-based architectures generate worse sam- ples than the grid-based generators with favorable convolutional inductive bi- ases. Models that focus on generating images at different scales do better, but employ complex architectures and derivatives. We take a signal-processing perspective and treat continuous image generation as interpolation from samples. Indeed, correctly sampled dis- crete images contain all information about the low spatial frequencies. The ques- tion is then how to extrapolate the spectrum in a data-driven way while meet- ing the ahove design criteria. Our answer is FunkNN-a new convolutional net- work which learns how to reconstruct continuous images at arbitrary coordinates and can be applied to any image dataset. Combined with discrete generative model it becomes a functional generator which can act as a prior in continuous ill-posed inverse problems. We show that FunkNN generates high-quality con- tinuous images and exhibits strong didt---distrbbution performance thanks to its patch-based design. We further showcase its performance in several stylized in- verse problems with exact spatial derivatives. Our implementation is available at https: / / thub com swg.swiiigiggoo wwwnng-rreeeeee resea rch rch/FunkNrs
designed for continuous evaluation of images
INTRODUCTION
Deep generative models are effective image priors in applications from ill-posed inverse problems (Shah & Hegde, 2018; Bora et al., 2017) to uncertainty quantification (Khorashadizadeh et al., 2022) and variational inference (Rezende & Mohamed, 2015). Since they approximate distributions of im- ages sampled on discrete grids they can only produce images at the resolution seen during training. But natural, medical, and scientific images inherently continuous. Generating continuous im- ages would enable a single trained model to drive downstream applications that operate at arbitrary resolutions. If this model could also produce exact spatial derivatives, it would open the door to generative regularization of many challenging inverse problems for partial differential equations (PDEs).
There has recently been considerable interest in learning grid-free image representations. Implicit neural representations (Tancik et al., 2020; Sitzmann et al., 2020; Martel al., 2021; Saragadam et al., 2022) have been used for mesh-free image representations in various inverse problems (Chen al., 2021; Park al., 2019; Mescheder al., 2019; Chen & Zhang, 2019; Vlašié et al., 2022; Sitzmann et al., 2020). An implicit network fo(x), often multi-layered perceptron (MLP), directly approximates the image intensity spatial coordinate RDD. While fo(x) only represents single image, different works incorporate a latent code in fe(x, z) to model distributions of continuous images.
These approaches perform well on simple datasets but their performance complex data like hu- man faces is far inferior to that conventional grid based generative models based on convolutional neural networks (CNNs) (Chen & Zhang, 2019; Dupont et al., 2021; Park et al., 2019). This is in fact true even when evaluated at resolution they were trained on. One reason for their limited perfor- mance is that these implicit models use ML.s which are not well-suited for modelling image data.

---

Published as a conference paper at ICLR 2023
Generative model
Axed resolusion image
FunKNN
Position IR
#E
Patch extraction
Scatial mansformer
0(x)) R
The proposed architecture. The generative model (orange) produces a resouutno image that is differentiably used by FunkNN to produce the image intensity at any location (blue).
Figure 1:
In addition, unlike their grid-based counterparts, these implicit generative models suffer from a sig- nificant overhead in the form of separate encoding hyper-network (Chen & Zhang, 2019; Chen et al., 2021; Dupont et al., 2021) or parameter training (Park et al., 2019) to obtain latent codes z.
In this paper, we alleviate the above challenges with mesh- free convolutional image generator that can faithfully learn the distribution of continuous image functions. The key feature of the pro- posed framework takes a discrete image at any resolution and super-resolves it to generate image intensities at arbi- trary spatial coordinates. As shown Figure 1), our approach combines traditional discrete image generator with FunkNN, resulting in deep generative model that can produce images at arbitrary coordinates or resolution. FunkNN can be combined with any off-the-shelf pre-trained image gen- erator (or trained jointly with one). This includes the highly successful GAN architectures (Karras et al., 2019; 2020), normalizing flows (Kingma & Dhariwal, 2018; Kothari et al., 2021) or score- matching generative models (Song & Ermon, 2019; Song et al., 2020). It naturally enables us to learn complex image distributions.
novel patch-based continuous super-resolution Network-FukKNN- which
Unlike prior works (Chen & Zhang, 2019; Chen et al., 2021; Dupont et al., 2021; Park et al., 2019), FunkNN neither requires a large encoder to generate latent codes nor does it use any MLPs. This is possible thanks FunkNN's 's unique way of integrating the coordinate with image features. The key idea is that resolving image intensity at coordinate x should only depend on its neighborhood. Therefore, instead of generating code for the entire image and then combining it with using an MLP, FunkNN simply crops a patch around x in the low-resolution image obtained from a traditional generator. This window is then provided to small convolutional neural network that generates the image intensity at x. The window cropping is performed in differentiable manner with spatial transformer network (Jaderberg et al., 2015).
We experimentally show that FunkNN reliably learns to resolve images to resolutions much higher than those seen during training. In fact, performs comparably to state-of-the-art continuous super- resolution networks (Chen et al., 2021) despite having only fraction of the latter's 's trainable param- eters. Unlike traditional learning- belong to image distributions different from those seen while training. This is a benefit of patch- based processing which reduces the chance of overfitting on global image features. In addition, we show that our overall generative model framework can produce high quality image samples at any resolution. With the continuous, differentiable map between spatial coordinates and image intensi- ties, access spatial image derivatives at arbitrary coordinates and use them solve inverse problems.
methods, our approach can also super-resolve images that
IMPLICIT NEURAL REPRESENTATIONS FOR CONTINUOUS IMAGE REPRESENTATION
Let E (R denote a continuous signal of interest with 1 channels and u be its discretized version supported along a fixed grid with n x n e N elements. For simplicity we consider the case with D = 2. Our discussion naturally extends higher dimensions.
2

---

Published as a conference paper at ICLR 2023
An implicit neural representation approximates by fe R - IR parameterized by weights 0 RK standard approaches, the weights are obtained by solving
? Σ llx(x.) - x(x))
0- (u) arg min
where u(x;) is the image target value sampled at spatial coordinate x,, This way, implicit neural representations can provide continuous interpolation for u. Notice, however, that the properties of this interpolation scheme depend on the choice of the network architecture and do not exploit the statistics of given images.
From (I), we can observe that the obtained weights (u) apply only to a single image u, thereby making [x(x) incapable of providing interpolating representations for a class of im- ages. Some prior works address this by incor- porating a low dimensional latent code z in the implicit network. The new representation then is given by the map x + fx(x, z). where each represents different image. Note that this framework often entails an additional overhead relate the images with the latent codes. For example, (Chen & Zhang, 2019) and (Dupont al., 2021) jointly train fo with an encoder network to generate the latent codes for images. (Park al., 2019), on other hand, optimize the codes as trainable parameters. Figure 2 shows a general illustration of implicit gener- ative networks.
Existing approaches
FunkNN
Direct optimizations
enscoder
Low resolntion
8
Latent code
Hyper network. concatenation,
Conditionning process x E. R
Patch extraction
MLP
CNN
The prior approaches to model continuous im- age distributions often exhibit poor perfor- mance in comparison to their grid-based gen- erative counterparts (Chen & Zhang, 2019; Dupont et al., 2021). This is due to their re- liance MLP networks which priori do not possess the right inductive biases model images.
Figure 2: Conceptual difference between FunkNN and existing implicit neural representations.
OUR APPROACH
To address the challenges discussed Section 2 we propose convolutional image generator that produces continuous images. As shown in Figure the proposed framework relies FunkNN, our new continuous super-resolution network which interpolated images to arbitrary resolution. Our approach first samples discrete fixed-resolution images from expressive discrete convolutional generator. This is followed by continuous interpolation with FunkNN. We combine FunkNN with any state-of-te of-te-of-the-ar pre- are-trained image learn complex image distributions.
The key advantage of FunkNN is its architectural simplicity and interpretability from signal pro- cessing perspective. The main idea that super-resolving sharp feature from low-resolution image at spatial coordinate x should only use information from the neighborhood of x. This is because high frequency structures like edges are spatially localized and have small correlations with pixels far-off spatial locations in the low-resolution image. FunkNN, therefore, selects small patch around coordinate x and passes to a small CNN to regress image intensity at x. Figure 2 illustrates the differences between FunkNN and prior approaches. In the following sections, we provide a formal description of FunkNN and our continuous image generative framework.
3.1 FUNKNN: CONTINUOUS SUPER-RESOLUTICN NETWORK
Given a low-resolution image u, E ddxdxe FunkNN estimates the image intensity at a spatial coordinate IR2 by two-step process: i) first extracts a p patch E Rexpxe from around
3

---

Published as a conference paper at ICLR 2023
Spatial Tranformee
reselution
Continuoss and diftematialle patch extraction
1
Bilbner ancerpllation at point RR
CNN
0
31
+
oi RGB value fled grid.
A(Rva): RGB value quey
Figure 3: FunkNN architecture. Given location (z, y). the Spatial Transformer extracts patch in the discrete image and a CNN produces the intensity at the query position.
the coordinate x, and then ii) a CNN maps the patch to the estimated intensity. More formally, if we denote ST (x, u) + to be patch extraction module and CNN₂ RPxxxc - R to be a CNN block that estimates pixel intensity at x from w, then FunkNN can be characterized as
FunkNN, CNN, ST.
While we could extract a patch in the ST module by trivially cropping it from the image, doing so would not give access to derivatives image intensities with respect to spatial coordinates, property of importance for PDE-based problems. We address this by modelling ST with spa- tial transformer network (Jaderberg et al., 2015). As illustrated in Figure 3, spatial transformers map the low-resolution image and input coordinate to the cropped patch in continuous manner, therehy preserving differentiability. These derivatives can, in fact, he exactly computed using auto- differentiation in PyTorch (Paszke et al., 2019) and TensorFlow (Abadi al., 2016). We evaluate the accuracy of these derivatives in Figure 13 in the Appendix E. This experiment clearly shows the effectivity of FunkNN for approximating the first-order and second-order derivatives, crucially important for solving PDE-based inverse problems. Another advantage of the ST module over triv- ial cropping is that it allows us to learn the effective receptive field of the patches during training, thereby providing better performance. For further details, please refer to the Appendix A.2.
3.1.1 TRAINING FUNKNN
Let (Um)igmgM M denote dataset of M discrete images with the maximum resolution X We generate low-resolution images (Um,,)1SmmsM from this dataset where Ug.m E Rdxxxx and d n. FunkNN is then trained to super-resolys (umm with (um) as targets using patch- frame- work. In particular, from each low-resolution image, small p p sized patches centered at randomly selected coordinates are extracted and provided to FunkNN which then regresses the corresponding intensity the coordinates in the high-resolution image. We optimize the weights of the CNN as follows.
n ΣΣ ΣDN
arg min OERK
Funk Urrm) Um (x:)
Note that since FunkNN takes patches as inputs, it can be used simultaneously train on images with different sizes. In addition, it allows us train with mini-batches on both images and sampling locations, thereby enabling memory-efficient training.
We consider three strategies to train our model, namely (i) single, (ii) continuous and (iii) factor. In single mode, FunkNN is trained with input images fixed resolution, eg. d= n.. The continuous mode uses low-resolution input images at different scales, i.e. d 2, where the scale can vary between [smin |_ax]] In factor mode, the super-resolution factor, 8, between the low and high- resolution images is always fixed and the low-resolution size d is varied in the range dman and dmax- The high-resolution target size then correspondingly chosen as ds x ds. At test time, we can then this model to super-resolve images by factor in hierarchical manner. For instance, with
4

---

Published as a conference paper at ICLR 2023
(0) of size d x d, can iteratively generate high-resolution images (u 121 of input image Uhr
sizes (d:)22 satisfying uhr Fun ( and d; =s'd.
3.2 GENERATIVE NETWORK
To generate continuous samples, we can combine FunkNN with any fixed-resolution convolutional generator. This flexibility allows our overall model to learn complex continuous image distributions, and to produce images and their spatial derivatives arbitrary resolutions. More formally, let G R- + Rdxdxe be generative prior low-resolution images of size d x d that takes as input a latent code of size L. Then, G' can be combined with FunkNN, to yield a continuous image generator FunkNN, o G.
The choice fixed-sized generator we use prior to FunkNN depends on the downstream application. Generative adversarial networks (GANs) (Karras et al., 2019) and normalizing flows (Kingma & Dhariwal, 2018) are popular high-quality image generators. However, the former exhibits challenges when used generative prior for inverse problems (Bora et al., 2017; Kothari et al., 2021) and the latter have very large memory requirements and are expensive to train (Whang et al., 2021; Asim al., 2020; Kothari al., 2021). Injective encoders (Kothari al., 2021) were recently proposed to alleviate these challenges by using a small network and latent space. However, since we do not necessarily require injectivity in our applications of focus, we use simpler architecture inspired from (Kothari et al., 2021) for faster training. In particular, we use a conventional auto-encoder to obtain a latent space for our training images and use a low-dimensional normalizing flow to learn this latent space distribution. The standard practice of using mean squared loss as suggested in (Brehmer & Cranmer, 2020; Kothari et al., 2021) results in loss of high frequency components in our reconstructions. We remedy this by using perceptual loss (Johnson et al., 2016). Further details on network architecture and training are given in Appendix A. worth noting that generative modelling is not aimed, we can train FunkNN independently of any generator as shown in Section 5.1.
CONTINUOUS GENERATIVE MODELS AND SOLVING INVERSE PROBLEMS
Continuous generative models built using FunkNN can be used for image reconstruction resolu- tions much higher than those seen during training. This is particularly useful when solving ill-posed inverse problems commonly found in experimental sciences like seismic (Bergen et al., 2019), med- ical (Lee et al., 2017) and 3-D molecular imaging (Arabi & Zaidi, 2020). Due to difficulty and high cost of measurement acquisition, obtaining training data is challenging in these applications. There- fore, new models can not readily be trained when faced with measurements acquired at resolution different from the ones previously seen during training.
We consider three inverse problems: reconstructing an image given its (i) first derivatives, (ii) when a fraction of derivatives are known, and (iii) limited- computed tomography (CT). In the following, we assume that we have low-resolution generative prior on d x d-pixel images, G R Raxdxe, which takes as input latent code z (0, Tdy).
4.1 INVERTING DERIVATIVES
We first apply FunkNN to reconstruct continuous image from its spatial derivatives Vxu(x,) in coordinates fx, ) Related ill- ill-posed reconstructions appear in different domains such as com- puter vision (Park et al., 2019) or obstacle scattering (Vlasić et al., 2022). We use the exact spatial derivatives provided by FunkNN and a pre-trained generative model to obtain the latent code z" that gives output aligned with the given derivatives,
z = arg min Σ I FunkNNNNoo - Vxu(x,) A||2112
We also found that updating the weights 0 of the generative model further helps in producing better reconstruction suggested in (Hussein et al., 2020). The exact procedure is detailed in Appendix B. The estimated reconstruction is given by sampling the trained FunkNN network on the high- resolution grid: ă i(x;) Fun G(z")).
5

---

Published as a conference paper at ICLR 2023
4.2 SPARSE DERIVATIVES
We now make the previous problem even more ill-posed by observing not the spatial derivatives on dense grid, but only 20% of them that have the highest intensity. It amounts only sample the gradient on the n < n² locations that gives the largest value [[yx"(x In order to account for the missing low-ampliudd gradient, we add an extra votal-varattion regularization term. Thus, we aim at solving
arg min (x G(z)) vxx(x)) A \z)/VG
(4)
using the same process as the previous setting (see Appendix B.1).
4.3 LIMITED-VIEW CT
In CT, we observe projections of a 2-dimensional image u (RR2 through the (discrete angle) Radon operator A(a): L²2 - (IR)' where a {ar,...,ar) ar) denotes the / viewing direc- tions with 0; E [0, 2T). Finally, the discrete observation given by
v. (A(a:)))(x) Th
where x = [x]..... is the n X sampling grid and v); A( Id,,²) is random perturbation. the view directions correspond to a fine sampling of the interval [0, 2T),, then the adjoint operator A", namely the filtered back-projection (FBP) operator, allows to recover the original signal even in the presence of noise. For application-specine reasons it is not always possible to observe the full viewing direction set, leading limited-view CT. Here sample (a:) uniformly between -70 degrees and +70 degrees. This incomplete set of observations makes the estimation of the signal from (5) strongly ill ill-posed. We address limited-view CT using the FunkNN framework:
arg min (l.K)))nk G(z)) -v:l2 Alzz|12
(6)
5
EXPERIMENTAL RESULTS
5.1 SUPER-RESOLLUTION
We first evaluate the continuous, arbitrary-scale super-resolution performance of FunkNN. We work on the CelebA-HQ (Karras et al., 2017) dataset with resolution 128x 128, and we train the model using three training strategies explained in section 3.1.1, single, continuous factor. More details about network architecture and training details are provided in Appendix A.2.
The model never sees training data at resolutions 256 256 or higher. Figure shows the perfor- mance of FunkNN trained using factor strategy, which indicates that FunkNN yields high-quality image reconstructions at much higher resolutions than it was trained on. This experiment shows that FunkNN learns accurate super-resolution operator and generalizes resolutions not seen during training. Table shows that our model demonstrates comparable performance to the state-of-the-ar baseline LIFFEDSR (Chen et al., 2021) but with considerably smaller network; model only has 140K trainable parameters while LIIF uses 1.5M parameters. Additionally, we provide a com- parison with LIIF by pruning its trainable parameters reach 140k. The results suggest that while FunkNN obtain comparable results to the original LIIF architecture, reducing its parameters to be closer to FunkNN deteriorates the results. This experiment emphasizes the capacity of FunkNN to perform similarly to state-of oftit-----hhe-rrt method wile being conceptually simpler. It is worth mentioning that on the same Nvidia 1000 GPU, the average time to process 100 images of size 128 128 1.97 seconds for LIIF, 0.81 seconds for LIIF with 140k parameters and 0.7 seconds for FunkNN.
5.2 ROBUSTNESS TO OUT-OF-DISTRIBUTTTNN SAMPLES
One exciting aspect of the proposed model is its strong generalization to out-of-distribiie (OOD) data-a boon obtained from FunkNN's pateh- operation. We showcase it by evaluating FunkNN

---

Published as a conference paper at ICLR 2023
FuniNN
1034-11024 512x512 256x206
Figure 4: FunkNN model only trained over CelebA- AAA images of maximum resolution 128 128, but it can reliably reconstruct high-quality images in higher resolutions.
Bilinear Interpolation
U- Net
LIFF
FunkNN
Ground truth
Figure 5:
Image super-resolution on 256 x 256 LSUN images with the model trained on 128 x 128 images from CelebA-HQ data.
originally trained on CelebA-HQ (Karras et al., 2017) at resolution 128x 128 (Section 5.1), to super- resolve images from the LSUN-bedroom (Yu al., 2015) dataset which are structurally very differ- ent from the training data. Moreover, we super-resolve images from size 128 to 256, which were seen training time. Figure indicates that the proposed model can faithfully super-resolve OOD images at new resolutions. We compare our approach with U-Net (Ronneberger et al., 2015) which has shown promising results for super-resolution; however, it can only be trained to map be- tween two fixed resolutions. We trained U-Net to superresolve from 128 x 128 to 256 x 256; it was thus trained on larger images than other methods. Nevertheless, Table shows that while the U-Net performs well on inlier samples, it gives rather bad results for OOD data. We also report the shift-invariant SNRs (Weiss et al., 2019) in Table 2 to compensate for the fact that U-Net is unable to recover the correct amplitudes for OOD samples.
Table 1: Performance of different methods over super-resolution in SNR (dB) the SNRs are aver- aged over 100 test samples of CelebA-HQ (inliers) or LSUN-hedroom (outliers).
128 - 256
128 + 512
128 - 1024
128 + 256 (OOD)
Truinable purams
Bilincur Interpolation
27.5
22.4
U-Net (Ronneberger el at., 2015)
31.3
19.9
8800K
LIF-EDSR (Chen 2021)
31.6
28.0
23.9
28.0
1500K
LIF-EDSR (Chen al., 2021) (140K puram.)
31.6
27.5
23.8
26.7
140K
FaukNN (siugle)
31.2
26.3
22.5
27.2
140K
FankNN (comtinuous)
30.7
27.7
23.5
140K
FankNN (factor)
32.6
28.2
24.2
26.6
140K
5.3 INVERSE PROBLEMS
We solve equation 3 using the generative framework described in Section 4 trained over CelebA- HQ samples to reconstruct an image from its spatial derivatives. Experimental settings are provided in Appendix B.I. Figure 6 illustrates FunkNN's performance, which provides high-quality spatial

---

Published as a conference paper at ICLR 2023
Intalization
Ground truth derivatives
Reconstructed derlvatlees
Ground buth
Recorstructed
Figure 6: Reconstructing image from its spatial derivatives using the generative prior; FunkNN could faithfully reconstruct the image details.
reconstructions from the observation of the spatial derivatives. Notice that Problem (3) is ill-posed and it is not possible to accurately estimate mean intensity of the original image only based on the derivatives. Despite the different color scales, the unknown image is accurately estimated by FunkNN.
We solve equation with the same procedure but over LoDoPaB-CT samples (Leuschner et al., 2021). We compare FunkNN with the continuous GAN proposed by (Dupont al., 2021). We solve the problem described in Equation (4) but replacing our model by their GAN. The training procedure is described in Appendix A.3. Figure 7 shows that FunkNN significantly outperforms the continuous GAN; while we observe an accurate estimation of the image gradients by GAN (first row, second column), its reconstruction (second row, second column) lacks lot of details that were well retrieved by our method. The poor reconstruction of GANs as prior for solving inverse problems has already been observed in (Whang al., 2021; Kothari al., 2021) and is confirmed by this experiment. This example of highly under-determined system shows that the proposed FunkNN architecture can leverage existing prior in addition to super-resolving both an image and its derivative.
SNRs are reported in the bottom- corner of each reconstruction.
FunkNN
GT derivatives
Dupont
PunkNN
1R
GT image
Dupoar et al.
PunkNN
19.1d
9.0a
11 113888
Figure 7: Solving PDE-based inverse problem with sparse derivatives.
J5
Figure Limited-view CT reconstruction.
We repeat the same procedure and solve equation 6 to reconstruct image from its limited-view sinogram. The precise experimental setting is relegated to Appendix C. Figure shows FunkNN's reconstructions compared to FBPs which confirms the power of the learned low-resolution prior in solving Problem (5).
RELATED WORK
REPPE-RESOOLTION
Various learning based approaches have been proposed the years for image super-resolution. dictionary learning (Yang et al., 2010) and random forests (Schulter et al., 2015) have been used to learn mapping between fixed low and high-resolution image pairs. More recently,
8

---

Published as a conference paper at ICLR 2023
deep learning methods based on convolutional neural networks have provided of-te-of-the-ar per- formance on super-resolution tasks (Dong et al., 2014; 2015; 2016; Kim et al., 2016; Lim et al., 2017; Lai et al., 2017). Inspired by the popular U-Net (Ronneberger et al., 2015), (Liu et al., 2022) proposed a multi-scale convolutional sparse coding based approach that performed comparably to U-Net on various tasks including super-resolution. Furthermore, (Jia al., 2017) learned the inter- polation kernel to design an implicit mapping from low to high resolution samples.
(Bora et al., 2017) and (Kothari et al., 2021) have used generative models as priors for image super- resolution using an iterative process. In addition, conditional generative models have been used address ill-posedness of super-resolution (Ledig al., 2017; Wang et al., 2018; Lugmayr al., 2020: Khorashadizadch et al., 2022). While the prior works mentioned above exhibit excellent super-resolution performance, they all have a limitation- these models are restricted to be used only at the image resolution seen during training. On the contrary, we focus on super-resolving images to arbitrary resolution.
In an attempt towards building continuous super-resolution models, (Hu et al., 2019) propose new upsampling module, MetaSR, that adjusts its upscale weights dynamically to generate high resolution images. While this approach exhibits promising results in-distribution training scales, it has limited performance of out of distribution factors. The closest to our work is the network LIIF (Chen et al., 2021) that also uses local neighbourhoods around spatial coordinates for super- resolution. However, unlike FunkNN that directly operates on small image patches, LIF first uses a large convolutional encoder generate feature maps for the image. Image intensity coordinate x is then obtained by passing this coordinate along with features in its neighbourhood to an MLP. Note that FunkNN's approach of combining image features with spatial coordinates is architecturally much simpler than that of LII.. It also requires less parameters, is slightly faster to evaluate and performs equivalently well.
6.2 GENERATIVE MODELS FOR CONTINUOUS SIGNALS
Generative models based on MLPs have been proposed in recent years to learn continuous image distributions. (Chen & Zhang, 2019) use implicit representations learn 3D shape distributions but lack expressivity to represent complex image data. (Dupont et al., 2021) propose adverserial training based on MI.ss and Fourier features (Tancik al., 2020) to represent different data modalities like faces, climate data and 3D shapes. While their approach is quite versatile, it still performs poorly compared to convolutional image generators (Radford et al., 2015; Karras et al., 2019; 2020).
As step towards building expressive continuous image generators, (Xu et al., 2021) and (Ntavelis et al., 2022) used positional encodings to make style-GAN generator (Karras et al., 2019) shift and scale invariant, thereby resulting in arbitrary scale image generation. (Skorokhodov et al., 2021; Anokhin et al., 2021) proposed continuous GANs for effectively learning representations of real- world images. However, these models do not provide access derivatives with respect spatial coordinates, making them unsuitable for solving inverse problems.
In related line of work, (Ho al., 2022) propose combine cascade of a diffusion and fixed-size super-resolution models to generate images at higher resolution. Unlike FunkNN, they use fixed-size super-resolution blocks so their approach does not generate continuous images.
7 CONCLUSION
Recent continuous generative models rely on fully connected layers and lack the right inductive biases to effectively model images. In this paper, we address continuous generation two steps. (i) Instead of generating continuous images directly, we first use convolutional generator pre-trained to sample discrete images from the desired distribution. (ii) We then use FunkNN to continuously super-resolve the discrete images given by the generative model. Our experiments demonstrate performance comparable to ofte--o-thh--ae art networks for continuous super-resolution despite using only fraction of trainable parameters.

---

Published as a conference paper at ICLR 2023
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, et al. Tensorflow: system for large- scale machine learning. In /2th {USENIX} symposium on operating systems design and imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Ivan Anokhin, Kirill Demochkin, Taras Khakhulin, Gleb Sterkin, Victor Lempitsky, and Denis Ko- rzhenkov. Image generators with condttinalll-indeeenneen pixel synthesis. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14278 --28-,447778 2021.
Hossein Arabi and Habib Zaidi. Applications of artificial intelligence and deep learning in molecular imaging and radiotherapy. European Journal of Hybrid Imaging, 4(1):1-23, 2020.
Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand. Invertible generative models for inverse problems: mitigating representation error and dataset bias. In International Conference Machine Learning, pp. 409. PMLR, 2020.
Karianne J. Bergen, Paul A. Johnson, Maarten V. de Hoop. and Gregory C. Beroza. Machine learning for data-driven discovery in solid earth geoscience. Science, 363(6433) 2019. doi: 10. 00.11166scienceeaaa0022.
Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis. Compressed sensing using genera- tive models. In International Conference on Machine Learning, pp. 537-546. PMLR, 2017.
Johann Brehmer and Kyle Cranmer. Flows for simultaneous manifold learning and density estima- tion. Advances in Neural Information Processing Systems, 33:442-453, 2020.
Yinbo Chen, Sifci Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8628-8638 8638, 2021.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Laurent Dinh, Jascha Sohh-Dickstein, and Samy Bengio. Density estimation using real nvp. arXiv preprint 1605 08803, 2016.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Learning a deep convolutional network for image super-resolution. European conference on computer vision, pp. 184-199. Springer, 2014.
Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. Image super-resolution using deep convolutional networks. IEFE transactions on pattern analysis and machine intelligence, 38(2): 295-307, 2015.
Chao Dong, Chen Change Loy, and Xiaoou Tang. Accelerating the super-resolution convolutional neural network. In European conference computer vision, pp. 391-407. Springer, 2016.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models as distributions of func- tions. arXiv preprint 1rX:v:2102.04776, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Jonathan Ho, Chitwan Saharia, William Chan, David Fleet, Mohammad Norouzi, and Tim Sali- mans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res., 23: 2022.
Xuecai Hu, Haoyuan Mu, Xiangyu Zhang, Zilei Wang, Tieniu Tan, and Jian Sun. Meta-sr: A magnification arbitrary network for sesortiesolution. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1575-1584, 2019.
10

---

Published as a conference paper at ICLR 2023
Shady Abu Hussein, Tom Tirer, and Raja Giryes. image-adaptin gan based reconstruction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3121-3129, 2020.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
Xu Jia, Hong Chang, and Tinne Tuytelaars. Super-resolution with deep adaptive image resampling. arXiv preprint arXiv:/7/2.06466. 2017.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119 2020.
Robert Keys. Cubic convolution interpolation for digital image processing. IEEE transactions on acoustics, speech, and signal processing, 29(6)11133-1166 1981.
AmirEhsan Khorashadizadeh, Konik Kothari, Leonardo Salsi, Ali Aghababaei Harandi, Maarten de Hoop, and Ivan Dokmani's c. Conditional injective flows for bayesian imaging. arXiv preprint arXiv:2204.07664, 2022.
Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. Accurate image super-resee super-resolution using very deep convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1646-1654, 2016.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6998 2014.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018.
Konik Kothari, AmirEhsan Khorashadizadeh, Maarten de Hoop, and Ivan Dokmanić. Trumpets: Injective flows for inference and inverse problems. In Uncertainty in Artificial Intelligence, pp. 1269-1278. PMLR, 2021.
Wei-Sheng Lai, Jia-Bin Huang, Narendra Ahuja, and Ming-Hsuan Yang. Deep laplacian pyramid networks for fast and accurate super-resolution. In Proceedings of the IEEE conference on com-
puter vision and pattern recognition, pp. 624-632, 2017.
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang. et al. Photo-realistic sin- gle image super-resolution using generative adversarial network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 46900 2017.
June-Goo Lee, Sanghoon Jun, Young- Won Cho. Hyunna Lee, Guk Bac Kim, Joon Beom Sco, and Namkug Kim. Deep learning in medical imaging: general overview. Korean journal of radiology, 18(4))570-584 2017.
Johannes Leuschner, Maximilian Schmidt, Daniel Otero Baguer, and Peter Maass. Lodopab-ct, benchmark dataset for low-dose computed tomography reconstruction. Scientific Data, 8(1):1-12, 2021.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep resid- ual networks for single image super-resolun resoluton. In Proceedings of the IEEE conference on com- puter vision and pattern recognition workshops, pp. 136-144, 2017.
11

---

Published as a conference paper at ICLR 2023
Tianlin Liu, Anadi Chaman, David Belius, and Ivan Dokmanié. Learning multiscale convolutional dictionaries for image reconstruction. IEEE Transactions on Computational Imaging, 8:425-437, 2022.
Andreas Lugmayr, Martin Danelljan, Luc Van Gool, and Radu Timofte. Srîlow: Learning the super- resolution space with normalizing flow. In European conference on computer vision, pp. 715-732. Springer, 2020.
Julien NP Martel, David Lindell, Connor Z Lin, Eric R Chan, Marco Monteiro, and Gordon Wetzstein. Acorn: Adaptive coordinate networks for neural scene representation. arXiv preprint arXiv:2105.02788, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc-
cupancy networks:
Learning reconstruction in function space. In Proceedings tfe IEEE/CVF conference on computer vision and paltern recognition, pp. 4460-4470, 2019.
Evangelos Ntavelis, Mohamad Shahbazi, lason Kastanis, Radu Timofte, Martin Danelljan, and Luc Van Gool. Arbitrary-scale image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11533-11542, 2022.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 165-174, 2019.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 2015.
Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In Interna-
tional conference on machine learning, pp. 1530-1538. PMLR, 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234 241. Springer, 2015.
Vishwanath Saragadam, Jasper Tan, Guha Balakrishnan, Richard Baraniuk, and Ashok Veer- araghavan. Miner: Multiscale implicit neural representations. arXiv preprint 03X5522202.03532, 2022.
Samuel Schulter, Christian Leistner, and Horst Bischof. Fast and accurate image upscaling with super-resolution forests. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3791-3799, 2015.
Viraj Shah and Chinmay Hegde. Solving linear inverse problems using gan priors: An algorithm with provable guarantees. In 20/8 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. (609--411 IEEE, 2018.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Newral Information Processing Systems, 33:7462-7473, 2020.
Ivan Skorokhodov. Savva Ignatyev, and Mohamed Elhoseiny. Adversarial generation of continuous images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 10753-10764, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
12

---

Published as a conference paper at ICLR 2023
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Process- ing Systems, 33:7537-7547, 2020.
Tin Vlašić, Hieu Nguyen, and Ivan Dokmanić. Implicit neural representation for mesh-free inverse obstacle scattering. arXiv preprint arXiv:2206.02027. 2022.
Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu, Chao Dong. Yu Qiao, and Chen Change Loy. Esrgan: Enhanced super-resolution generative adversarial networks. In Proceedings of the European conference on computer vision (ECCV) workshops, pp. 0-0, 2018.
Pierre Weiss, Paul Escande, Gabriel Bathie, and Yiqiu Dong. Contrast invariant snr and isotonic regressions. International Journal of Computer Vision, 127(8):11441116,, 2019.
Jay Whang, Qi Lei, and Alex Dimakis. Solving inverse problems with a flow-based noise model. In International Conference Machine Learning, pp. 11146-11157, PMLR, 2021.
Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy. Positional encoding spatial inductive bias in gans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13569-13578, 2021.
Jianchao Yang, John Wright, Thomas S Huang, and Yi Ma. Image super-resolution via sparse representation. IEEE transactions on image processing, 19(11):2861-288 2010.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans loop. arXiv preprint arXiv:1506.03335,, 2015.
13

---

Published as a conference paper at ICLR 2023
(a) Ground truth test samples
(b) AE reconstructions
Figure 9: AE performance CelebA-HQ in resolution 128 x 128.
A NETWORK ARCHITECTURES
A.1 GENERATIVE MODEL ARCHITECTURE
The generative networks used to address inverse problems described in Section are trained in two-steps: i) train auto-encoder (AE) neural network to map the training samples lower dimensional latent code, ii) we train a normalizing flow model to learn the distribution of the com- puted latent codes by using maximum likelihood (ML) loss function.
A.1.1 AUTO-ENCODER ARCHITECTURE
The AE network is a succession of an encoder and a decoder neural networks. The encoder maps images low dimensional latent code, L-dimensional vector. It is succession of 6 Conv+ReLu blocks (one convolution layer followed by a ReLu activation unit). Between each Conv+ReLu blocks, there is a down-sampling operation. The decoder network transforms the computed latent code to the image samples. The decoder has the same structure as the encoder network, except that the down-sampling operations
replaced by up-sampling layers. In order increase the expres- sivity of the model, we equip our model with local skip connections in both encoder and decoder, as suggested in (He al., 2016). The latent code dimension is 512 for training data in resolution 128 x 128 for both RGB and gray-scale images. The AE model has around 40M and 1 IM trainable parameters for limited-view CT and CelebA HQ experiments.
Training strategy While we can train the AE model using only the MSE loss (Brehmer & Cranmer, 2020: Kothari al., 2021), 1), it often suppresses the high-frequency components of the reconstructed image. To alleviate this issue, we combine the perceptual loss proposed in (Johnson et al., 2016) with regular MSE loss which significantly improves the quality of the reconstructions.
Experimental Result: We train AE model over 40000 images of the LoDoPaB-CT (Leuschner et al., 2021) and 30000 images of CelebA-HQ (Karras et al., 2017) datasets in resolution 128 x 128. We train the model over CT and CelebA-HQ images for 600 and 200 iterations. Figures 10 and display the output image produce by the AE on the test dataset. Notice that very small details are lost by the AE. However, the final reconstruction contains the most important details. We show that this is enough to learn an informative prior and solve ill posep inverse problems. We also show that it is possible go beyond the quality of reconstruction given by the trained AE, Section 4.3.
14

---

Published as a conference paper at ICLR 2023
(a) Ground truth test samples
(b) AE reconstructions
Figure 10: AE performance on LoDoPaB-CT in resolution 128 128.
(a) CT
(b) CelchA-HQ
Figure 11: Generative model performance on CT and CelebA-HQ samples in resolution 128 x 128.
A.1.2 NORMALIZING FLOW
We use RealNVP (Dinh et al., 2016) architecture with fully-connected layers for scale and bias subnetworks with two hidden layers of dimension 1024 and 512. The model comprises five masked affine coupling blocks and activation normalization. As mentioned earlier, we train the model using ML loss. The flow model has 1OM trainable parameters.
Experimental Results: As soon the flow model is trained, we take samples from the Gaussian distribution and feed the flow model to generate new latent codes. Decoder then takes the gen- erated latent codes and produce new images. Figure 11 demonstrates the generated samples by our generative model (flows + AE) for CT and CelebA-HQ dataset.
A.2 FUNKNN ARCHITECTURE
We crop patch with size p 9 from the low-resolution image using spatial transformer with bicubic interpolation kernel and reflection mode for the border pixels. We two trainable parameters defined the size of the patch with respect to low-resolution image, i.e. the location of
15

---

Published as a conference paper at ICLR 2023
Figure 12: CT images generated by (Dupont et al., 2021)
the 9 pixels around the query pixel are estimated during training. We use eight convolutional layers with filter size and channel dimension 64 and ReLu activation functions. We use two maxpooling layers with size (2,2) inside to reduce the feature size. This downsampling layer if applied between succesive applications of convolutio++ReLu layers, so that the size of the input devided by 4 in total in each direction. We use skip connections between each convolution layers. We then use four fully-connected layers with latent dimension 64 with skip connections between each Linear+Relu block. Finally, following inspiration from residual neural network, we add to the output the intensity of the pixel central to the input patch. We also use mini-batches with 512 pixels and 64 objects for each training iteration. All the models trained using Adam optimizer (Kingma & Ba, 2014) with learning rate 10-4
A.3 DUPONT ET AL. ARCHITECTURE
We train the continuous generative model proposed in Dupont et al. (Dupont et al., 2021) over the 40000 images of the LoDoPaB-CT dataset. We train their model over 100 epochs using the default parameters given in their public repository. We visually asses the quality of the generated sample in Figure 12.
B INVERSE PROBLEMS: EXPERIMENTAL SETTINGS
B. PDE INVERSE PROBLEM
The training strategy of the networks detailed in Appendix A for two datasets used in Section 5.3. The low-resolution images are composed of d x d = 128 x 128 pixels and the high-resolutions target images are of size
256 256. We use et al., 2021) and optimize for 2500 iterations using Adam optimizer over Problem (3) with X = and on Problem (4) with À and X2 1 Then, we run 1000 iterations of stochastic gradient descent to optimize the weights of the auto-encoder of the generative model suggested in (Hussein al., 2020).
as the initialization as suggested by (Whang
C LIMITED-VIEW CT
The training strategy of the networks is detailed in Appendix A. The low-resolution images are composed of d 128 128 pixels and the high-resolutions target images of size n x 256 x 256. We use z as the initialization. The estimated solution is obtained by running 5000 iterations of stochastic gradient descent using Adam optimization algorithm Problem (5) with
16

---

Published as a conference paper at ICLR 2023
À 0. The observations given by equation (5) are degraded by additive Gaussian noise such that the SNR of each projection 30dB.
D SUPER-RESOLUTION
Table 2: Performance of different methods over super-resolution in scale-invariant SNR (dB)
128 - 256
128 -
512
128 -> 1024
128 -+ 256 (OOD)
Bilinear Interpolation
27.5
24.9
22.4
22.9
U-Net (Ronneberger ol, 2015)
LHF-EDSR (Chen et al., 2021)
31.6
28.0
24.0
28.0
LIF--DDR (Chen et al., 2021) (140k param.)
31.6
27.6
23.8
26.7
FunkNN (single)
31.4
26.4
22.6
27.2
FunkNN (continaous)
23.5
FunkNN (factor)
32.6
28.2
24.2
26.6
E ACCURACY OF THE DERIVATIVES PROVIDED BY FUNKNN
order to quantify the quality of FunkNN derivatives, we train our network on images with intensity given by a Gaussian functional:
g(z, v) exp (-(x (x:
x0 (y yo) )2
20
where the position of the center (æo, yo) is chosen random in the field of view and the standard deviation a is chosen uniformly at random in [0.1, 0.4]. We use cubic convolutional interpolation kernel Keys (1981) in the spatial transformer of the FunkNN trained using the training procedure described in A.2. In the first row of Figure 13, from left to right, we display the test function the norm of its gradient and its Laplacian sampled a discrete grid. in the second row, display the same quantity estimated by the trained FunkNN as well as the SNR between the estimation and the ground truth.
Image
First derivative
Laplacian
Ground truth
FunkNN
72.1dB
33.8dB
18
Figure 13: The derivatives produce by FunkNN on images of a Gaussian. The true derivatives can be computed analytically.
17


Reviews:
[{"review_id": "MdnwlgT87UV", "reviewer": "Reviewer_kWFU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a patch-based framework for solving inverse problems. The proposed approach is able to achieve reasonable performance on several tasks.", "summary_of_the_review": "This work proposes a simple approach to solving inversion problems. However, the presentation of the paper can be improved, and more empirical results and comparisons with state-of-the-art baselines are needed.", "strength_and_weaknesses": "Strength:\n1. The paper is well-written. The proposed approach is simple and easy to follow.\n2. The empirical results are good. The model is able to achieve comparable performance while using fewer parameters than state-of-the-art models on several inverse problem tasks.\n\nWeakness:\n1. Section 3 is hard to follow. For instance, what does \"sample discrete images from a powerful convolutional generator\" mean? Do you mean to generate an image using the pretrained generator? What does \"discrete image\" mean? Does it have a fixed resolution?\n2. In section 3.1, it would be better to define the input of ST formally (e.g., coordinate, image) instead of just mentioning its dimension. \n3. In section 3.2, does it mean given a pretrained generative model, you apply FunkNN to the output from the generated samples of the model? Why not directly apply it to existing low-resolution images so that the quality of the results will not be bounded by the pretrained generator? \n4. The notation is section 4.1 is hard to follow. For instance, a new model IN$_\\theta$ is introduced. What is this model doing? It's better to keep the notation consistent. The parameter $j$ is also not formally defined, what are $u$ and $x$?\n5. The structure of the paper can be improved. For instance, some results are presented in section 4, but there is another section called Experimental results (section 5). \n6.  Besides the super-resolution results (table 1), there is no comparison with state-of-the-art baselines for the other tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "MZuP_ts-bky", "reviewer": "Reviewer_BzrP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces FunkNN, a super-resolution architecture built on a two-stage process to model images as a continuous function.\n\nIn order to predict the color value for a given (x, y) location, first a Spatial Transformer with a bilinear kernel is used to extract a PxP patch from the image whose coordinates are centered on the point (x, y), and then a CNN is used to predict the color value at the chosen point. In a sense, one could tell that FunkNN is built by using a CNN to sharpen an initial bilinear upscaling.\n\nThe model as a whole allows to both compute spatial gradients in the image, and back-propagate gradient information back to the low-resolution image. This makes it possible to combine FunkNN with any off-the-shelf low-resolution generative model. The paper illustrates this by using such models as a prior over in images in problems of high-resolution image reconstruction from partial data: reconstructing an image from its 20% highest intensity spatial gradients, or in the context of limited-view CT reconstruction.\n\nOn the super-resolution task, FunkNN is shown to be competitive to LIIF, a state of the art model, on x2, x4 and x8 tasks, while being a much smaller model in terms of parameter count.", "summary_of_the_review": "I believe this is overall a decent paper. The proposed model is well-justified, simple and elegant, and experimentally shown to be applicable to several relevant tasks.\n\nHowever I think that to truly be a good paper it is lacking an in-depth comparison with other models of the literature that can be applied to the same tasks (I focused on LIIR from the references of the paper, but it's entirely possible other models could also be applicable here).", "strength_and_weaknesses": "**Strengths:**\n\nThe paper proposes a simple and elegant model, showing the possibility to build continuous super-resolution methods with *relatively* small models. The efficiency of FunkNN is demonstrated on multiple different tasks, illustrating the various possible applications of the model. The construction of the model is well motivated.\n\nThe core idea of the paper is in my opinion the fact that it delegates the \"interpolation\" part of the problem to a spatial transformer which achieves bilinear interpolation over the whole patch, rather than trying to train a NN to do it. It does seem like a good inductive prior to give to the CNN for the final prediction.\n\n**Weaknesses:**\n\nFirst of all, I want to say that I am not deeply familiar with the problem of super-resolution, and I refined my understanding of it using the references from the paper. My attention went in particular to the LIIF model (reference Chen et al. 2021 from the paper), which seems to me to have quite a lot of similarities with the proposed FunkNN.\n\nThe models are clearly different:\n- LIIF passes the whole low-resolution image in a CNN encoder, then extracts a small patch from that encoded representation, and uses a NN to interpolate to the target pixel value from that small patch.\n- FunkNN extracts a small patch from the low-resolutions and at the same time interpolates it to align its pixel grid to the target coordinate using a spatial transformer, and then use a CNN to predict the target pixel value from that aligned patch.\n\nHowever, as far as I can tell, the main properties of FunkNN advanced by the paper (being able to compute spatial gradients of the image and being able to back-propagate gradients to the low-resolution input) are shared by LIIF. This means that for all the inverse problems described in section 4, on which the performance of FunkNN is illustrated, it would be possible to make a performance comparison with LIIF.\n\nFurthermore, FunkNN like LIIF represents the image prediction as a continuous function of pixel values. This means that, like LIIF, FunkNN could theoretically be able to generalize to scale factors different than the ones it was trained on. LIIF for example is trained on x2, x3 and x4 factors, and shown to generalize well up to x30 from that. No such attempt at generalization is made on FunkNN.\n\nThe paper points the large number of parameters of LIIF. From what I understand, it comes mostly from the large CNN encoder LIIF uses, that works on 48x48 patches. However, nothing in the structure of LIIF would prevent using a much smaller encoder working on smaller patches (for example 8x8 like FunkNN in the experiments). Doing so could probably bring LIIF to a number of parameters comparable with FunkNN.\n\nThe core of my concern is thus this: while conceptually different (interpolation done by a NN vs. done by the spatial transformer), LIIF and FunkNN are two models with very similar sets of capabilities. I thus find it a lack in the paper that the two models are not compared in a more in-depth way: the inverse problems FunkNN is evaluated on should be accessible to LIIF, and the arbitrary scaling of LIIF should be accessible to FunkNN. I find this especially lacking given that, from a conceptual point of view, it seems to me that the good inductive bias of FunkNN has the potential to make it very competitive with LIIR on all those tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "MIX2KnatPE", "reviewer": "Reviewer_Zsei", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for image super-resolution that resembles an implicit function, but has strong inductive biases in the underlying architecture. To predict the super-resolved value for a particular coordinate, they interpolate a small 2D patch centered at that coordinate from the original image (reminiscent of RoI-Align from the segmentation literature), and pass this through a CNN.\nThey show competitive super-resolution results on natural image datasets, and show that their method produces plausible spatial derivatives that be used in a medical imaging applications.\nCompared to other super-resolution methods, theirs seems to be much more efficient in terms of computational complexity and parameter count.", "summary_of_the_review": "I do think that this paper would be stronger if the authors focused on slightly different directions (see \"Weaknesses\" section), but overall I feel that it is a solid incremental contribution in super-resolution. ", "strength_and_weaknesses": "Strengths:\n1. The proposed method is well-motivated and clearly explained. \n2. The method is sensible and has a number of desirable properties.\n\nWeaknesses:\nThe main weakness is that the experimental evaluation could be more extensive. For example:\n1. In \"Related Work\", the authors discuss GAN-based super-resolution methods, but do not include any such methods as baselines in their experiments.\n2. I can imagine that the proposed method could be successfully applied to density estimation. For example, as a cascaded density model similar to what [1] does with diffusion models.\n\nI wonder if more emphasis on directions like these would make the paper more compelling to a broader audience. In contrast, the discussion about OOD generalization feels a bit weak (proposed method not substantially better than baseline), and the image derivative experiments feel a bit niche (I personally do not find these particularly exciting, and imagine that much of the generative modeling community feels similarly).\n\n\n[1] Cascaded Diffusion Models for High Fidelity Image Generation. Ho et al, https://arxiv.org/abs/2106.15282.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EoTffIpxBO5", "reviewer": "Reviewer_9a5t", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces FunkNN, a continuous super resolution module for image-like data. Given a low resolution image and an image coordinate x (which may not lie on the low resolution grid), FunkNN returns the predicted RGB value at that coordinate, effectively leading to a continuous image representation, where pixel values can be probed at arbitrary coordinates in R^2. To achieve this, the authors propose to use a spatial transformer which takes as input an image and a coordinate and extracts a (linearly interpolated) patch centered on this coordinate. The patch is then passed through a small CNN which returns the predicted RGB value at the coordinate location. The authors perform various interesting experiments on image super resolution and inverse problems involving image derivatives.", "summary_of_the_review": "Overall, I think this is a fairly good paper, which may be of interest to the ICLR community. The method and experiments are interesting. However, I do not think the paper is very well situated in the literature and also lacks some experimental evaluation. I therefore believe the paper in its current state is borderline, but I am willing to update my scores if some of the concerns are addressed.", "strength_and_weaknesses": "**Strengths**\n\n- The paper tackles an interesting and useful problem.\n- The proposed model has various interesting properties. For example, the model can be trained with very low memory since it only requires patches and the continuous nature of the model allows for easy estimation of derivatives.\n- The inverse problem applications and experiments are interesting.\n- The out of distribution experiments are interesting. It is nice to see that a model trained on CelebAHQ can generalize to LSUN bedrooms.\n- The figures are nice and help with the general understanding of the paper.\n\n\n**Weaknesses**\n\n- In general, the discussion of related work is lacking and the paper is not very well situated in the literature. In particular, the proposed model is very closely related to LIIF which is briefly mentioned as a baseline. However, from reading the FunkNN paper this is not clear. It was not until I had a look at the LIIF paper that I realised how closely related these two ideas are (using local information to upsample images at arbitrary resolutions). The differences need to be discussed in more detail and related works should be properly acknowledged. In addition, generative models of continuous images are briefly mentioned but not discussed in detail (and a reference to [1] is missing). Further, while works like Chen & Zhang 2019 and Dupont et al 2021 are mentioned as performing poorly for grid-based generative tasks, it is worth noting that these models are also much more general and can be applied easily on various data modalities.\n- As the proposed method is a super resolution module and not a generative model, I believe it makes more sense to focus on super resolution in the related work section instead of generative models (which currently takes up much more space). In general, the model does not feel very well situated in the super resolution literature.\n- There is very little quantitative evaluation of the model. For example, for the inverse problem experiments there is not a single quantitative metric (even though e.g. just measuring reconstruction error should be simple) and only a handful of quanlitative comparisons. Further, there is little discussion of what baselines could be relevant for these experiments.\n- The authors discuss generative models throughout the paper, but do not include any experiments or evaluation for this. For example, it would be interesting to see how FunkNN applied to some low res generative model would compare to other generative models of continuous images.\n- While the writing is generally clear, there are quite a few typos and some missing information. For example, in Table 1 it is not clear on which dataset the model was evaluated. Is the performance averaged across all examples on some test subset of CelebAHQ?\n\n\n**Nitpicks**\n\n- It seems like a lot of citations are missing brackets. I would recommend using `\\citep{}`.\n- The authors often refer to \"Implicit Neural Networks\". However, it seems like the two dominant terms in the literature now are \"Implicit Neural Representations\" or \"Neural Fields\", so I would go with one of these options to be consistent with the literature.\n\n[1] Adversarial Generation of Continuous Images, Skorokhodov et al, CVPR 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster