Versatile Neural Processes for Learning Implicit Neural Representations
Authors: Zongyu Guo
Cuiling Lan
Zhizheng Zhang
Yan Lu
Zhibo Chen
Venue: ICLR 2023 poster (2023)
Keywords: Implicit Neural Representations
Neural Processes
Variational Inference

Abstract: Representing a signal as a continuous function parameterized by neural network (a.k.a. Implicit Neural Representations, INRs) has attracted increasing attention in recent years. Neural Processes (NPs), which model the distributions over functions conditioned on partial observations (context set), provide a practical solution for fast inference of continuous functions. However, existing NP architectures suffer from inferior modeling capability for complex signals. In this paper, we propose an efficient NP framework dubbed Versatile Neural Processes (VNP), which largely increases the capability of approximating functions. Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost while providing high modeling capability. At the decoder side, we hierarchically learn multiple global latent variables that jointly model the global structure and the uncertainty of a function, enabling our model to capture the distribution of complex signals. We demonstrate the effectiveness of the proposed VNP on a variety of tasks involving 1D, 2D and 3D signals. Particularly, our method shows promise in learning accurate INRs w.r.t. a 3D scene without further finetuning.

Published as a conference paper at ICLR 2023
VERSATILE NEURAL PROCESSES FOR LEARNING IM- PLICIT NEURAL REPRESENTATIONS
Zongyu Guo'; Cuiling Lan? Zhizheng Zhang", Yan Lu Zhibo Chen'
l University of Science and Technology of China, Miicosooft Research Asia
1 [guozy@mail., chenzhi boe ustc. edu..eduu.co en
2 (culan, zhizzhang, yanlu) emicrosoft.com
ABSTRACT
Representing a signal continuous function parameterized by neural network (a.k.a. Implicit Neural Representations, INRs) has attracted increasing attention functions
recent years. Neural Processes (NPs), which model the distributions conditioned on partial observations (context set), provide a practical solution for fast inference of continuous functions. However, existing NP architectures suffer from inferior modeling capability for complex signals. In this paper, we propose an efficient NP framework dubbed Versatile Neural Processes (VNP), which largely increases the capability of approximating functions. Specifically, we introduce a bottleneck encoder that produces fewer and informative context tokens, relieving the high computational cost while providing high modeling capability. At the decoder side, we hierarchically learn multiple global latent variables that jointly model the global structure and the uncertainty of function, enabling model to capture the distribution of complex signals. We demonstrate the effectiveness of the proposed VNP variety of tasks involving ID, 2D and 3D signals. Particularly, our method shows promise in learning accurate INRs w.r.t. a 3D scene without further finetuning. Code is available here
INTRODUCTION
recent line of research on learning representations is to model a signal (e.g., image, 3D scene) continuous function that map the input coordinates into the corresponding signal values. By parameterizing continuous function with neural networks, such implicitly defined representations, i... implicit neural representations (INRs), offer many benefits over conventional discrete (e.g.. grid-based) representations, such as the compactness and memory-efficiency (Sitzmann et al.) 2020b) Tancik et al., 2020 Mildenhall et al., 2020: Chen et al., (021a). Charaaterziingpaaameeennnn signal by corresponding set of network parameters generally requires re- re-training the neural network, which is computationally costly. practice, at test time, desired to have models that support fast adaptation to partial observations of new signal without finetuning.
In fact, the Neural Processes (NPs) family (Tha et al. 2022) supports such merit. It meta-learns the implicit neural representations of probabilistic function conditioned on partial signal observations. During test-time inference, enables the prediction of the function values at target points within a single forward pass. Naturally, given partial observations of signal, there exists uncertainty inside its continuous function since there are many possible ways to interpret these observations (i.e., context set). The NP methods (Garnelo et al.) 8018a by learn to map a context set of observed input-output pairs to a conditional distribution over functions (with uncertainty modeling). However, it has been observed that NPs are prone to underfit the data distribution. Following the spirits of variational auto-encoders (Kingma& & Welling 2014), the work of (Garnelo et al.) 2018b) introduces global latent variable better capture the uncertainty in the overall structure of the function, which still suffers from the inferior capability for modeling complex signals. Attentive Neural Processes (ANP) attention
(Kim et al. 2019) can further alleviate this issue, which leverages the permutation-invant mechanism |Vaswani al.) (017)) reweight the context points and the target predictions. However, taking each context point as a token, ANP has troubles processing complex signals that requires
Work done during an internship Microsoft Research Asia.

---

Published as a conference paper at ICLR 2023
Partial Signal Observations
Encoder
Predicted Target Signals
1D
* M
A As
Tokenite
Nattomboo
20
Set
Contest
Yokens
Modulated MLP Dlock
Moduloted Block
3D
x
lg
x
Zig
Context Set Dr (x..
(X,
Hierarchical Lotent Variabies
Po (r,
Figure 1: The proposed Versatile Neural Processes framework contains bottleneck encoder and a hierarchical latent modulated decoder. The input context set is first encoded into fewer and informative context tokens by set tokenizer followed by self-attention blocks, which provide powerful network capability with tolerable complexity. The decoder consists of cross- coss-attention modules and multiple modulated MLP blocks, enhancing the model expressiveness for complex signals.
abundant context points as condition (e.g., image with high resolution), where the computational cost very expensive. Moreover, for complex signals, modeling the global structure and uncertainty of the function with a single latent Gaussian variable may be suboptimal. It is worthwhile to explore an efficient framework to excavate the potential of NPs in modeling complex signals.
In this paper, we propose Versatile Neural Processes (VNP), an efficient and flexible framework for meta-learning of implicit neural representations. Figure I shows the framework of VNP. Specifically, VNP consists of a bottleneck encoder and hierarchical latent modulated decoder. The bottleneck encoder powered by the set tokenizer and self-attention blocks encodes the set of context points into fewer and informative context tokens, refraining from high computational cost especially on complex signals while attaining higher modeling capability. At the decoder, we hierarchically learn multiple latent Gaussian variables for jointly modeling the global structure and uncertainty of the function distributions. Particularly, we sample from the latent variables and use them to modulate the parameters of the MLP modules. Our VNP has high expressiveness to complex signals (e.g.. 2D images and 3D scenes) and significantly outperforms existing NPs approaches on 1D synthetic data.
We summarize our main contributions below:
We propose Versatile Neural Processes (VNP) that is capable of learning accurate INRs for approximating the function of complex signal.
We introduce a bottleneck encoder produce compact yet representative context tokens, facilitating the processing of complex signals with tolerable computational complexity.
We design a hierarchical latent modulated decoder that can better capture and describe the structure and uncertainty of functions through the joint modulation from the multiple global latent variables.
We implement the VNP framework on ID, 2D, and 3D signals respectively, demonstrating the state-of-the-art performance a variety of tasks. Particularly, our method shows promise in learning accurate INRs of 3D scenes without further finetuning.
RELATED WORK
Implicit Neural Representations (INRs). INRs aim parameterizing signal by differentiable neural network, i.e., learning a continuous mapping function w.r.t. the signal (Stanley) 2007] Sitzmann al. 2020b Tancik et al. 2020). In the seminal work CPPN (Stanley) (007). a neural network trained to learn the implicit function that fits signal, e.g., an image. Given any spatial position identified by 2D coordinate, the model that acts as a function, outputs the color value of this position. Such continuous representations, as powerful paradigm, have wide range of applications such as
image super-resolution (Chen et al. 2021b), modeling shapes (Chen & Zhang 2019 Park et al. 2019)
and textures (Oechsle et 2019). 3D scene reconstruction (Mildenhall al. 20200 Martin Brualla
et al. 2021] Niemeyer & Geiger 202T), and even lossy compression (Dupont et al.) 2021 2022
Schwarz & Teh 2022). Most of these methods require re-training the neural network model/overfit
2

---

Published as a conference paper at ICLR 2023
new signal, which is computationally costly (Sitzmann al. 2020b Tancik et al. 2020 Mildenhall
jet al. 2020 Chen et al.) 20 Dupont et al.) 2021).
In practice, it is desired to have models that support fast adaptation to signal, i.e., approaching the continuous function of this signal without abundant steps of optimization in inference. Some works (Chen et al., 2021b Lee et al.) [2021] Sitzmann al. 2020a) adopt standard gradient- based meta-learning algorithms to learn the initial weight parameters the network (Finn et al. (017).
However, they still require a few gradient decent steps to fit for the new signals.
Neural Processes (NPs), Neural Processes actually can learn the continuous function conditioned on partial observations of signal, enabling fast adaptation inference). The series of NPs methods (Garnelo et al., 20 8a b) provide probabilistic solutions in predicting continuous functions from partial observations. NPs approximate the distributions in the function space, which formulates Stochastic Processes (Ross et al. in function realization. line of researches on neural processes has been introduced, targeting at
new signal (not requiring finetuning in
1996), by introducing stochasticity
improving the prediction accuracy (Kim et al. 2019: Lee et al. 2020, Wang & Van Hoof 2020 Volpp et al. 2021), preserving the stationarity of stochastic processes (Gordon et Foong al. 22020), and
generalizing to observation noise (Kim et al.) 2022). The work of Neural Processes (Garnelo et al.
2018b) learns a latent variable distribution to capture the global uncertainty in the overall structure of the function, which is optimized with variational inferece (Kingma & Welling 2014). Attentive Neural Processes (ANP) leverages the attention mechanism to enhance the representation of each context point and alleviate the underfitting problem (Kim et al.) 2019). Transformer Neural Processes (TNP) (Nguyen & Grover 2022) similarly takes each context point as a token and leverages transformer architecture approximate the function. However, for complex signals that requiring abundant context points as condition (e.g., image with high resolution), the computational complexities of ANP and TNP are very high which are quadratic with respect the number of context points.
It is desired to have framework that can effectively approximate the functions of complex signals. In this work, we introduce a strong NP framework, Versatile Neural Processes (VNP), which leverages informative context tokens and explores the hierarchical global latent variables for modulation, leading to superior approximation of function distributions.
REVISITING THE PROBLEM FORMULATION OF NPs
Neural Processes (NPs) (Garnelo et al. 2018b a) are class of methods that approximate the probabilistic distribution of continuous functions conditioned on partial observations. Suppose we have a labeled context set De (Xc, Yc) := (x, yik sampled from continuous function with inputs x; and outputs y, NPs target at predicting arbitrary and finite target points Dr (Xr, Yr) == (x,, Ye ee) by learning the input-output mapping function f. Given some signal observations (a set of context points), many possible functions may match well to these observations and thus naturally there exists function uncertainties. The conditional distributions of targets points can be modeled as:
Ps XT. D.)
II II N(y; My (x, oz) D.))-
To generate coherent function predictions and better model the function distributions, Garnelo et al. (Garnelo et al. (0188) introduce the (Latent) Neural Process by encoding the global structure and uncertainty of the function into a latent Gaussian variable via Bayesian inference:
z ~ po NT, Dc); Po.0 (Yr/XT 2) II ( Myy; XT, Dc), oz XT, Dc)). (2)
Due to the intractable log- likeliillooodd some previous works adopt amortized variational inference (Kingma & Welling, 2014), which is also used to optimize our proposed framework. We can derive the evidence lower bound (ELBO) on log po (o (Xrl z). where the ELBO can be viewed as a
combination of the reconstruction term (the first term) and the KI. term (the second term):
(loo po (Yr XT, Do) D l (z) (z)XT, Dc)]].
Here, s) and 9 refer to the parameters of conditional prior encoder and the decoder, similar to conditional VAEs (Sohn et al. 2015: Ivanov et al. 2011). g (z) Dr)) denotes the posterior distribution of the latent variable given the ground truth target points, which is only used in training and not accessed in inference.
3

---

Published as a conference paper at ICLR 2023
x hg
R
Y
Context Tobens
100
Pa
MIPs Pusing
Paoking Module
MLD
Meerarchical Variables
Dr (Xy.
96 A
ky)
(a) Pipeline of the decoder.
(b) Modulated MLP block.
Figure 2: Diagram of the decoder with the hierarchical latent modulated MLPs.
So far, we have revisited the problem formulation of NPs. Since NPs approximate the distribution over functions with some function observations, they also be regarded as probabilistic, continuous, conditional generative models, where the generated result a continuous function.
4 VERSATILE NEURAL PROCESSES
We propose an efficient NP framework, Versatile Neural Processes (VNP), which provides much improved capability in approximating the function of various signal. Figure provides overview of our proposed VNP. VNP is generic that can be used to generate the functions for ID, 2D or 3D signals. For test-time inference, the inputs are set of context points Dc (Xc, Yc) and the coordinates of target points Xr. The outputs are the predicted values of target points Yr. The continuous function is parameterized by the network weights. The proposed VNP consists of bottleneck encoder that efficiently encodes the context points into fewer informative context tokens, refraining from high computational cost especially on complex signals. At the decoder side, cross- attention layers are utilized to exploit the contexts relevant to the given target followed by a stack of modulated MLP blocks. Particularly, we introduce multiple global latent variables to jointly modulate the MLP parameters, which facilitates the modeling of complex signal.
4.1 BOTTLENECK ENCODER IN VNP
Some NP methods encode the context points to produce local feature representations target coordinates. The early work (Garnelo et al. 2018b) uses simple MLP layers to learn features of the context set and suffers from underfitting problem. Attentive NP (Kim et al.) (209)) and Transformer NP (Nguyen & Grover) (022) enhances the feature representation by introducing point-wise self- attention with each context point taken as a token. However, when the number of context points large (e.g. in order to model image signal with many details), the computational burden is heavy and such a framework is impractical.
We address this issue by simply introducing set tokenization module (set tokenizer) followed by self-attention layers. The set tokenizer instantiated by set convolution layer (Zaheer et al. 2017) which transforms the neighboring context points token. Taking 2D image an example, with the kernel size of k x & and stride of k; the set tokenizer can reduce the number of sample points by 222 times (assume the image resolution is integer multiple of k). By using the set tokenizer, the complexity of attention layers is reduced from the first term to the second term as below:
N2 + Le O(L.NZ + L.NoNN) + O(L, h4
Nc 12
Nr),
(4)
where L,, and L. are the number of the self-attention layers and the cross- coos-attention layers, No and Nr are the number of context points and target points. The difference between the set convolution and the conventional convolution is that the former can handle missing points (e.g.. in the application of inpainting) and the data that live "off the grid" (e.g. time series data that observed irregularly at any time). The combination of set tokenizer and attention would empower our framework the flexibility in processing 3D signals, which is infeasible for ConvNP (Gordon et al.) Foong et al. 2020) because ConvNP requires preserve all the 3D grids, which is very inefficient in sparse 3l) space.
4

---

Published as a conference paper at ICLR 2023
4.2 HIERARCHICAL LATENT MODULATED MLPs
In the previous NP methods (Garnelo et al. 2018b; Kim et al. 2019). they learn a single global latent Gaussian variable from the observed context set model the distributions of a function. However, potentially limits the expressiveness of the model. Intuitively, increasing the dimension of the latent variables may increase such flexibility, but in practice, this not sufficient (Lee et al., 2020).
We design a hierarchical latent modulated decoder in order to better model the complex distribution of function. We sequentially learn multiple global latent variables modulate the parameters of MLP blocks. Figure F shows the details of the designed hierarchical structure.
The decoder consists of Le cross- atssentttnntoon blocks and LK modulated MLP blocks. With the target location Xr as query and the context tokens from encoder as the keys and values, the rrossaatenntion blocks output the target location features Yo E I where T is the number of target coordinates and d is the feature dimension. The sequential modulated MLP blocks enable the exploitation of hierarchical global latent variables for the approximation of complex signal. The output of the final (L modulated MLP block goes through two MLP layers to estimate the probability of Yr.
Figure 2b illustrates the detailed network structure of modulated MLP block. We build Modulated MLP block by stacking two modulated MLP layers and two unmodulated MLP layers. This block refines - to output features Y%, which is the input of the next Modulated MLP block. At the heart of each block is a latent variable z E IR 1xd, modeled by Gaussian distribution. The generation process of zu (of the th block) can be formulated as follows:
(5) where MLPs refer to two MLP layers and an intermediate ReLU activation layer. The prediction results Y from the previous block (i.e., the (A: 1)th block) are used for calculating the condi- tional prior distribution of 2k, i.e., P (Zk <k Dc, X-). During training, the conditional posterior distribution 4o (Zz) Yok; Dc, Dr) can be calculated as well, by incorporating the ground truth target signal values Yr:
Ps (2x Do, XT) = -N (P 222) + MLPs (AvgPool (MLPs( YR-1))),
(2 (Eck, Dc, Dc,DT)= = NN 22. < MLPs (AvgPool (MLP>(( (( Dr))))-
(6)
Marked with dashed lines, Yr only participates in training and cannot be accessed during inference.
After sampling the low-dimensional latent variable 2k, we use the modulated fully-connected (ModFC) layer (Karras et al. 2020, 202T) adjust the parameters of modulated MLP layers, taking sampled realization of Zk as the style vector input. Unlike previous NP methods that concatenate the latent variable every target coordinate, modulating the MLP parameters from the low-dimensional latent variables is more flexible way to represent continuous functions (Sitzmann) jt al. 2020b). Please see Appendix A for more details for the mechanism of the modulated MLP.
We model the function representations by multiple latent variables (Z1, Zz. ZLx ). The KL term Eq. 3 which measures the mismatch between the approximated distribution and ground truth distribution, takes the hierarchical format as
Lx
DKL = Σ Dhw. De, Xr)Il
(Z Do
+ DKL DDK
where go(zex/Dos Dr) II:-1 (s(2, YY Dc, DT) is the approximate posterior of Zzkk This decomposed KL term and the reconstruction term formulate the objective:
= E rugo (81.2 (Dr) log po (orooorrra (Yr|zi:Lg) Xr, Dc)] + 3- DKL,
(8)
where 3 denotes weight to balance the importance between the two terms. In our experiments on 2D and 3D signals, we multiply the KI. term with a small weight / to better capture the uncertainty of function distributions (Higgins et al.). In addition, we emphasize that although there are some prior works designing hierarchical architecture for VAES (Sønderby et al. 2016 Vahdat & Kautz) 2020 Child 2021 or double latent variable models for NP (Wang & Van Hoof 2020), our proposed hierarchical architecture is different in principle. Here, every latent variable is a low-dimensional vector obtained after average pooling. Therefore, the designed hierarchical architecture deal with arbitrary target coordinates and thereby can be used for boosting the performance of approximating the global structure of continuous functions.
5

---

Published as a conference paper at ICLR 2023
RBF kernel GP
Matern kernel
Parameters
context
target
context
target
CNP
1.023-00033
0.019::.0155
0.935=0.036
0..124:..010
0.99 M
BANP
1.380-10.000
0.267:10.001
1.380:10.002
0.072:10.002
1.58 M
ConvNP
1.382+000001
0.275++0.001
1.383+00001
0.081+0.000
1.97 M
Stacked ANP
1.381=000001
0,400:10.004
1.38110.001
0.183110002
1.52
Stacked ANP +
1.301=000001
0.406-+0.006
1.381+000001
0.188++0.014
2.31 M
HNP
1.374+0.002
0.561+0.003
1.377+0.001
0.336:10.000
1.66 M
HNP-Mod
1.379=00001
0.627=000200
1.371#0.004
0.370
1.96 M
VNP
1.377=00.004
0.651+0.001
1.376:0.004
0.439:10.007
2.29 M
Table 1: The test log likelihood (larger is better) on the synthetic ID regression experiment. The proposed VNP outperforms previous methods by a large margin. Both the pre- pre-processing transformer and the hierarchical structure improve the expressiveness of function representations. Stacked ANP + means the enhanced version of Stacked ANP with more channels.
EXPERIMENTS
The proposed Versatile Neural Process (VNP), as an efficient meta-learner of implicit neural repre- sentations, can be implemented into a variety of tasks. We evaluate the effectiveness of VNP on 1D) function regression Suubscction[5.1 2D image completion and superresolution (subsection 5.2) and view synthesis for 3D scenes (subsection 5.3) respectively.
5.1 ID SIGNAL REGRESSION
We implement the proposed VNP to learn the implicit neural representations for ID signal regression. This classical 1D) regression aims at predicting the function values at given target locations, condi- tioned on several observations of the samples from the function. Following (Kim et al.) (001)), we measure the performance by considering the context set likelihood and target set likelihood, which reflects the context reconstruction error and target prediction error, respectively.
Settings. We train the models synthetic functions drawn from prior function distributions synthesized with different kernels (RBF, Matern) by following (Gordon et al. Kim et al. 2022). For the evaluated methods, we employ importance weighted sampling (Burda et al. al.||2011 (2016] to evaluate the log likelihood, where the last four methods in Table arre measured by sampling latent variables from the posterior distribution and calculating the tighter ELBO with importance weighted sampling. For fair comparison, we manage to keep the network size comparable with that of other methods, which is completed by adjusting channel number. Please refer to Appendix B for detailed settings.
Comparison with the State-of-the-Arts We compare method with conditional neural process (CNP) (Garnelo et al., 2018a), (stacked) attentive neural process (ANP) (Kim et al., 2019) (with stacked self-attention layers), bootstrapping attentive neural process (BANP) (Lee et al. (020), and convolutional neural process (ConvNP) (Foong et al. 2020). Table [ shows the results in terms of log likelihood. Most of the models can provide satisfactory reconstruction results for context points, except CNP which suffers from underfitting problem (Garnelo et al. (0188)). On the context points, ours also achieves comparable performance. However, the prediction results of other methods on the unseen target points are much worse than that on the context sets. In contrast, our final model, VNP, outperforms previous approaches by large margin the target points on both of the function distributions. Note that although BANP attempts to increase the expressiveness of function representations by using latent variable bootstrapping from the perspective data resampling.
Ablation Study. We conduct group of ablation study on this ID regression task to investigate the effectiveness of different components. Based on ANP, we first build a Hierarchical Neural Process (HNP) with hierarchical latent variables. Note that similar to ANP, the implemented HNP still concatenates every target coordinate with the global latent variables. We observe significant improvements in Table by comparing HNP with ANP, demonstrating that the hierarchical global latent variables boost the performance of function approximation. Secondly, the modulated MLP layer for implicit function parameterization can be further equipped in HNP, referred to as HNP. Mod. Compared with the concatenation of latent variable, using low- now-dimensional latent variables to modulate the network parameters enables more flexible function approximating. Based on HNP-Mod,

---

Published as a conference paper at ICLR 2023
number Modulated ML.P blocks
0
2
4
6
8
6 6/single z
context
1.375=0.001
1.376:10.001
1.376+0.001
1.376=00001
1.37610.001
1.376=0.001
larget
0.076=0.001
0.336:10.028
0.30110.001
0.409-0.007
0.43510.028
0.245
Table 2: Ablation study about the detailed hierarchical structure Matern kernel. Here also provide the results of test log- log-likelihood (larger is better).
laige
large
(a) Matem kernel. ANP our VNP.
(b) RBF kemel.. ANP vs. our VNP.
Figure 3: Visualizations of the ID regression results. VNP delivers diverse function prediction results,
while ANP (Kim et al. 2011) tends to underestimate the function variances at some target locations.
we finally add the bottleneck encoder to learn conditional network inputs that are adaptive to target locations, building final model VNP. In Table II we can find this powerful proceesssingg encoder can further improve the performance, because the set tokenizer (set convolution here) can preserve locality to learn more appropriate features.
In T we further ablate on the detailed hierarchical structures in our model. As we can observe, when we increase the number of Modulated MLP blocks, the prediction performance (at target points) is improved until saturated with 6 blocks (i.e., Lx = 6). In addition, the column with "6 single z" means we use exactly the same network structures as Lı 6, but instead only impose the KL constraint to the final latent variable Zo- It is found that the performance will also drop dramatically, which verifies that the gains come from the hierarchical design instead of the increased network capacity. This ablation study guides us to set the number of Modulated MLP blocks as 6 to compare with other methods, which can keep balance between the performance and the complexity.
Visualizations. We visualize the obtained function distributions from stacked ANP (Kim et al. 2019) and our VNP, by sampling the latent variables 20 times. Given partial observations of a signal, there exists uncertainty on the continuous function since there are many possible ways to interpret these observations (i.e., context set). An excellent Neural Process model should be able to model such uncertainty through the fitted functions. In other words, the generated functions conditioned on the context set should approximate the data distribution and cover the target set points. As shown in Figure 3 we can that the generated functions from the stacked ANP cannot predict the target points accurately, e.g., those regions marked by orange. In contrast, our method provide good approximations for the continuous functions, where the distribution covers the groundtruth function.
5.2 2D IMAGES COMPLETION AND SUPERRESOLUTION
2D image can be modeled by a continuous function that maps the 2D pixel coordinates the color values. We implement our VNP framework for learning the continuous function of 2D images, which supports tasks such as image completion and super-resolution to arbitrary size.
Settings. We conduct experiments on CelebA dataset (Liu et al. 2015), mainly with the resized resolution of 64 x 64. Due to the limited representation capability or the high requirement on computation resources, most previous NP methods (Kim et al. 2019 Lee et al. (020) are in general trained and evaluated on relatively low resolution such as 32 x 32. Our framework enables the processing of complex signal with higher resolution. We train single model that supports image completion and super-resolution tasks. During training, we control the context ratio as 0.03, which means 3% pixels are taken context set. The target ratio is set as 0.15 and the target has partial intersection with the context set. More detailed experimental settings can be found in Appendix B.
Visualizations. We compare VNP with stacked ANP Kim al. 2019) tested with three different context ratios respectively as shown in Figure 4 Our VNP generates diverse and realistic image completion results with fine details. In comparison, the results of Stacked ANP are blurred.
7

---

Published as a conference paper at ICLR 2023
Context Ratio
3%
10%
50%
3%
50%
NLL (lower betler) context ratin 0.03
CelebA64
Context
Stacked
2.988
VNP, ks= Lx =1
2.994
Sample1
VNP, ks 1, Lx
2.953
VNP, ks 2, Lx
2.952
Sample2
VNP, k= Lx =6
2.964
Table Quantitative results measured
Stacked ANP
VNP
by Eq 8 on the test set. Lower is better. ks is the kernel size set tokenizer. LK is the number of Modulated MLP blocks.
Figure 4: Qualitative comparisons with Stacked ANP (Kim et al. 2019). Our VNP presents diverse and realistic image completion results.
Original
Context
Completion
resolution
5.2
(a)
(b)
Figure 5: Visualization of VNP results on CelebA64 (Luu et al. 2015). (a) Image Completion. (b)
Superresolution arbitrary size. (c) Handling image completion and superresolution simultaneously.
CelebA64
CelebA178
GFLOPs
Context Ratio
Context Ratio
0.05
0.25
1.0
0.05
0.25
Stacked ANP
17.1
39.8
198.5
190.0
867.2
12150
Our VNP
27.6
27.6
27.6
204.5
204.5
204.5
Table 4: Comparing our VNP with stacked ANP (Kim et al. by FLOPs. The bottleneck encoder plays an important role in reducing the computational cost. The statistic in gray is estimated since we have an upper bound limitation of GPU memory.
2019)
about the complexity, measure
In Figure 5 we show that the proposed VNP achieves satisfactory results for image completion and superresolution on CelebA64 dataset. As unified framework, with single model, it can support image completion, superresolution, and the concurrent of completion superresolution.
Influence of Patch/Kernel Size. We conduct an ablation study to investigate the influence of the patch/kernel size in our set tokenizer. We set the stride the same as the kernel size. The results are shown in Table It is observed that using larger kernel size does not degrade the performance obviously. In addition, using hierarchical structure brings gains.
Complexity Comparisons. Thanks framework for complex signal modeling, where the number of context points as condition is usually large. For comparison, 0.25 on CelebA64 and CelehA178 respectively. Here, the kernel size of set tokenizer is on CelebA64 and 10 CelebA178. The computational complexity comparisons in terms of GFLOPs are shown in Table 4 Since we use set tokenizer to reduce the number of tokens and process the context set in image grids, the GFLOPs of VNP would be smaller than that of Stacked ANP, especially when the context ratio is high.
bottleneck encoder design, the proposed VNP is practical
calculate the GFLOPs of stacked ANP and VNP when target ratio
5.3
VIEW SYNTHESIS ON 3D SCENE
Implicit neural representations excel at representing 3D signals. However, since 3D signals are usually much more complex, most previous works (Mildenhall al. 2020: Martin-Brunt al.
8

---

Published as a conference paper at ICLR 2023
Input
NeRF-VAE
VNP
GT
Input
NeRF-VAE
VNP
GT
h
h
h
H
(a) Cars.
(b) Chairs.
Figure 6: Novel view synthesis on ShapeNet (Chang al. 2015) Objects. Our VNP presents more
realistic prediction results than the blurry results of NeRF- VAE Kosiorek et al. 2021
PSNR (dB)
Cars
Lamps
Chairs
Learned Init (Chen et al. 2021b)
22.80
22.35
18.85
Deterministic
Trans-INE (Chen & Wang (022)
23.78
22.76
19.66
NeRF-VAE (Kosiorek et al. 2021
21.79
21.58
17.15
Probabilistic
Our VNP
24.21
24.10
19.54
Table 5: The quantitative results one-shot novel view synthesis. We compare the proposed VNP with both deterministic and probabilistic methods. All of these methods are able to produce INRs representing the 3D scenes within few steps.
2021) require expensive optimization to fit the neural networks to represent the scenes. We focus on the task view synthesis this section to evaluate our proposed VNP.
Settings. We follow the spirits of NeRF MMildenhall et al., 2020) that fits network to map the world coordinate into the corresponding RGB values and volume density. We use bottleneck encoder (tokenization from the image patches) to calculate the adaptive input of the MLPs, queried by target world coordinates. Then the hierarchically learned global latent variable will modulate the parameters of MLPs predict the RGB values and volume density. Note that there is extra volume rendering process inside each decoding block before the pooling module, because it requires transferring the compute the latent distribution of Zg. We conduct experiments on ShapeNet (Chang et al. 2015) objects, including three sub-datasets: cars, lamps, and chairs. More details on the network structures and hyper parameters can be found Appendix
world coordinate the image coordinate
Comparisons. Our VNP model can generate the implicit neural representations w.E.f the previously unseen scene single forward pass. this 3D task, one prior work NeRF-VAL (Kosiorek et al.) 202T) can also generate the scene from the randomly sampled latent variable. We make quantitative and qualitative comparisons with NeRF-VAE.
addition, we compare with (Chen et al., (021b) and (Chen & Wang 2022) which can also learn implicit neural representations of the previously unseen scene, although they are not in the family of probabilistic models and thus cannot model function distributions. As shown in Table 5 method achieves the best performance in terms of Peak Signal-to-Nois. Ratio (PSNR) on Cars and Lamps, and is comparable with Trans- -NN (Chen) & Wang 2022 Chairs. The visualizations in Figure 6a show that VNP produces much better novel-view prediction results than NARF-VAF, which is also a probabilistic generative model.
6 CONCLUSION AND DISCUSSION
The Neural Processes family provides an efficient way for learning implicit neural representations by approximating the function distribution when only partial observation of signal is given. We propose an efficient NP framework, Versatile Neural Processes (VNP), that largely increases the capability of approximating functions. Our bottleneck encoder and hierarchical latent modulated decoder enables strong modeling capability to the complex signals. Through comprehensive experiments, effectiveness of the proposed VNP on ID. 2D and 3D signals. Our work demonstrates the potential of neural process promising solution for efficient learning of INRs in complex 3DD scenes.
show the

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work was supported in part by NSFC under Grant U1908209, 62021001.
REFERENCES
Yuri Burda, Roger B. Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In 4th
International Conference Learning Representations, ICLR 2016, 2016.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint ar2iv:151..03012, 2015.
Hao Chen, Bo He, Hanyu Wang, Yixuan Ren, Ser Nam Lim, and Abhinav Shrivastava. Nerv: Neural representations for videos. volume 34, 2021a.
Yinbo Chen and Xiaolong Wang. Transformers as meta-leamers for implicit neural representations. arXiv preprint arXiv:2208.02800) 2022.
Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8628 8638, 2021b.
Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5939-5948, 2019.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Emilien Dupont, Adam Golinski, Milad Alizadeh, Yee Whye Teh, and Arnaud Doucet. COIN: COmpression with implicit neural representations. In Neural Compression: From Information Theory Applications Workshop @ ICLR 2021, 2021.
Emilien Dupont, Hrushikesh Loya, Milad Alizadeh, Adam Golinski, Whye Teh, and Arnaud Doucet. Coin++: Neural compression across modalities. Transactions Machine Learning Research, 2022(11), 2022.
Chelsea Finn, Pieter Abbecl, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning. pp. 1126-1135. PMLR, 2017.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284-8299 2020.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1713, PMLR, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Tch. Neural processes. arXiv preprint arXhv:1807.01622 2018b.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In 8th International Conference on Learning Representations, ICLR 2020.
Irina Higgins, Loic Matthey, Arka Pal, Christopher P. Burgess, Xavier Glorot, Matthew M. Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017.
Oleg Ivanov, Michael Figurnov, and Dmitry P. Vetrov. Variational autoencoder with arbitrary conditioning. In 7th International Conference Learning Representations, ICLR 2019, 2019.
10

---

Published as a conference paper at ICLR 2023
Saurav Jha, Dong Gong, Xuesong Wang, Richard E Turner, and Lina Yao. The neural process family: Survey, applications and perspectives. arXiv preprint 0rXiv:2209.00517, 2022.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality stylegan. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. $110-81199 2020.
Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. volume 34, 2021.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In 7th International Conference on Learning Representations, ICLR 2019, 2019.
Mingyu Kim, Kyeong Ryeol Go, and Se- Young Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In 10th International Conference on Learning Representations, ICLR 2022, 2022.
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. 2nd International
Conference on Learning Representations, ICLR 2014, 2014.
Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Sona Mokrá, and Danilo Jimenez Rezende. Nerf-vae: A geometry aware 3d scene generative model. In International Conference on Machine Learning, pp. 5742-5752. PMLR, 2021.
Jacho Lee, Jihoon Tack. Namhoon Lee, and Jinwoo Shin. Meta-learning sparse implicit neural representations. volume 34, pp. 11769-11780, 2021.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Tch. Bootstrap- ping neural processes. volume 33, 6606-6615, 2020.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Ricardo Martin-Brualla, Noha Radwan, Mehdi SM Sajjadi, Jonathan Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
7210-7219, 2021.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In European conference on computer vision, pp. 405-421. Springer, 2020.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In International Conference on Machine Learning, pp. 16569-16594. PMLR, 2022.
Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1153-11164, 2021.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4531-4540, 2019.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 165-1744 2019.
Sheldon M Ross, John J Kelly, Roger Sullivan, William James Perry, Donald Mercer, Ruth M Davis, Thomas Dell Washburn, Earl V Sager, Joseph B Boyce, and Vincent L Bristow, Stochastic processes, volume 2. Wiley New York, 1996.
11

---

Published as a conference paper at ICLR 2023
Jonathan Schwarz and Yee Whye Teh. Meta-learning sparse compression networks. Transactions Machine Learning Research, 2022. ISSN 2835-8856.
Vincent Sitzmann, Eric Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta-leaming signed distance functions. volume 33, pp. 10136-10147, 2020a.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein, Implicit neural representations with periodic activation functions. volume 33, pp. 7462-7473, 2020b.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. volume 28, 2015.
Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8(2):131-162, 2007.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Kell Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features networks learn high frequency functions in low dimensional domains. volume 33, pp. 7537-7547, 2020.
Arash Vahdat and Jan Kautz. Nvae: A deep hierarchical variational autoencoder. volume 33, pp. 19667-19679, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. volume 30, 2017.
Michael Volpp, Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference Machine Learning, pp. 10018-10028. PMLR, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. volume 30, 2017.
12


Reviews:
[{"review_id": "wgDO-sQ1X10", "reviewer": "Reviewer_HSYx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an auto-encoder based pipeline to achieve versatile neural process, which consists of a bottleneck encoder and modulated MLP based decoder. The bottleneck encoder aims to produce fewer and informative context tokens while the decoder hierarchically learns multiple global latent variables and the uncertainty of a function to improve the capability in modeling complex signals. While the developed method follows the existing paradigms quite a bit, it shows practical improvements compared to chosen baselines under three different tasks.", "summary_of_the_review": "Although this paper yields some interesting empirical results compared to some baselines, the proposed method builds mainly on top of previous techniques without enough technical contributions. Additionally, it is also weak in the experimental part, including lack of evaluations over challenging datasets and optimization-based methods. However, I am not an expert in this area, so I will not object if the other reviewers tend to accept and think it has already met the bar of ICLR.", "strength_and_weaknesses": "Strengths\n\n(1) This paper is generally well-written and easy to follow, except some points. I can understand most statements easily. For some improvement suggestions, please see the weakness part below.\n\n(2) The authors present consistly improved results over chosen baselines under three different tasks.\n\n(3) Most of the technical elements are built on existing literatures, providing solid effectiveness foundations.\n\nWeakness\n\n(1) The novelty of this submission seems to be limited. For example, using self-attention layers && set tokenizer (in Sec. 4.1) and hierarchical latent codes (in Sec. 4.2) to improve results is not new. Although the combinations of these strategies produce appealing results, the authors do not show specific design insights, making the proposed method look a bit ad-doc.\n\n(2) There are no discussions on the limitation and failure cases.\n\n(3) The current developed framework is more complex than previous methods. Given some practical improvements, I am curious about how much time it needs for the training and testing? Please report the training and inference speed as well as corresponding experimental environment for more comprehensive evaluations.\n\n(4) For the baseline comparisons, why do not the authors make evaluations over optimization-based implicit representation modeling methods (e.g. SIREN, Fourier Feature Network)? I look forward to seeing these empirical comparision results.\n\n(5) In terms of experimental results, the used datasets are too simple. How about the results produced by NeRF's synthetic datasets, the complex shape sets used in SIREN and NGLoD, et al.? Please use more complex datasets to prove the technical advantages.\n\n(6) Some minor presentation issues:\n\nA. In Sec. 4.2, the functionality of cross-attention is unclear. Why should we use the cross-attention here?\n\nB. In Sec. 4.2, why multiplying the KL term with a small weight can better capture the uncertainty of function distributions?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "uR0QcU5ysBL", "reviewer": "Reviewer_vL4F", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an encoder-decoder architecture with strong modeling capability on complex signals, which is capable of capturing the distribution of 1D, 2D and 3D signals efficiently. The main contributions are:\n1. a framework that increases the capability of approximating complex signals (from 1D signal to 2D images and 3D shapes).\n2. an efficient encoder that can significantly reduce computational cost.\n3. a hierarchical decoder that learns multiple global latent variables for better approximation of global structure of a continuous function.", "summary_of_the_review": "Given the above-mentioned strength (effectiveness over previous methods, extensive analysis, etc), my suggestion is that the paper is good for a publication.", "strength_and_weaknesses": "**Strength**  \n1. The proposed encoder structure is efficient in modeling complex signals especially for 2D and 3D data, compared with previous works (Attentive NP, Transformer NP, ConvNP, etc.)\n2. The authors conduct extensive experiments on 1D, 2D and 3D data separately to demonstrate the effectiveness of the proposed framework on modeling different signals, compared with previous SOTA methods.\n3. The authors also perform sufficient ablation studies to validate the design choices.\n\n**Questions**\n1. In fig.3 the results of VNP are pretty diverse, does this mean that the prediction is unstable/noisy?\n2. For image completion comparison, are you using the same resolution(64 or 32?) for a fair comparison? Does the blurred results come from low-resolutions instead of a inferior method?\n\n**Writing**\n1. Typo: fig.4 \"Our HINP\"\n2. I would suggest to ensure the tab/fig is on the same page as the text that mentions them. This will make reading easier. For example, fig.2, tab.2, fig.4.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "6Ar0Mpw6MC", "reviewer": "Reviewer_L3M4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present new architectural improvements to the Neural Process family, arguing that the costly self-attention mechanism designed to overcome context underfitting introduced in the work on Attentive Neural Processes (Kim et al., 2019) makes the method family unsuitable for large signals. Instead, the authors propose the use of a set tokenizer (Zaheer et al., 2017) which summaries several neighbouring context points into individual tokens, does reducing the complexity of the ANP attention mechanism. A second innovation are significant architectural changes to the decoder, introducing hierarchical latent variables, modulated MLP blocks as well as other changes. The authors evaluate their method on NP tasks and compare to a wide variety of other NP architectures. ", "summary_of_the_review": "Overall an interesting idea with the potential for publication, but currently unsuitable due to serious Clarity & Quality issues. Will consider raising my score if the authors significantly improve those aspects during a rebuttal. ", "strength_and_weaknesses": "`Strengths:`\n- Neural Processes are interesting techniques for meta-learning INRs due to their reduction of the high memory cost of training, making this an interesting topic to investigate. Reducing the computational complexity of the ANP's self-attention is a valuable contribution.\n- Results seem overall strong (Table 1, Figure 3), fewer FLOPs are required as stated (Table 3).\n\n`Weaknesses:`\n- While the Set Tokenizer is shown as a key innovation in the presented work, reducing the complexity of the attention mechanism, little insight is provided in terms of the cost that is paid by the summarisation of multiple context points. Table 1 shows that as expected, context reconstruction quality does decrease, but it is unclear how that decrease depends on the kernel size and stride (Section 4.1). No complexity analysis (in terms of $\\mathcal{O}$ notation) is provided. \n- Due to a large number of architectural and algorithmic changes, the authors introduce a large number of additional hyperparameters to be tuned by a partitioner (number of hierarchical blocks/number of latent variables, $\\beta$ (KL objective), kernel width/stride of set transformer). Reading between the lines, the method appears to suffer from potential instability \"We only modulate some of the MLPs to preserve the training stability.\" \n- The manuscript in its current form has several clarity and quality issues making it currently unsuitable for publication. The submission appears rushed and with insufficient attention to writing.\n\n`Questions:`\n- To which extent are the improved results in Table 1 explained by the significantly larger models? What happens when baseline architectures are scaled to provide an equal comparison in terms of #parameters? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5b-Px-YeCF", "reviewer": "Reviewer_vtKS", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new architecture for Neural Processes named Versatile Neural Process (VNP). This new architecture is build as an improvement over previous Attentive Neural Process (ANP) with two main modifications:\n1. The context points are pre-processed by a encoder module built using set convolutions and self-attention layers. This module summarizes the (potentially large) set of context points into a much smaller set of context tokens. This allows the model to scale to much larger sets of context points than ANP.\n2. The decoder part of the NP is augmented with a hierarchy of random variables (as shown by (Child 2021), this has a large impact on expressiveness to VAE-like models). These variables are used to module the prediction in an iterative way.\n\nThese two modifications together are empirically shown to significantly improve the capacity of VNP to model complex and varied functional distributions, as well as computationally scale to large inputs such as super-resolution tasks.", "summary_of_the_review": "This paper introduces a new iteration on the design of NPs which significantly improves the capacity of the model to capture variability in the function distribution as well as its scalability with regard to the number of context points.\n\nWhile the application cases for NPs remains rather nebulous to me, I believe this is a good paper.", "strength_and_weaknesses": "**Strengths:**\n- The proposed architecture and its training procedure is described with extensive details.\n- The proposed VNP is empirically compared to many other models from the literature on various tasks.\n  - The improvement on 1D tasks are impressive, showing how VNP is able to capture the variability of the function distribution much more effectively than previous models.\n  - The comparison of computational cost on the image reconstruction task clearly shows the gain from the added encoder, making the prediction cost almost independent on the input size.\n\n**Weaknesses:**\n- The tasks used to asses the model quality are artificial problems. This is an issue shared by the other NP models from the literature, but as the model family is maturing it would be interesting to asses its applicability to real world problems. After reading the paper I don't have a clear view of when a practitioner would want to use a VNP.\n  - In particular, in the context of the super-resolution task, VNP is compared to other NP models, but not to the general state of the art of super-resolution models, which also contains models able to do arbitrary-scale super-resolution (such as *Learning Continuous Image Representation with Local Implicit Image Function*, Chen, Liu & Wang, 2021, cited in the paper).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster