Rényi Neural Processes
Authors: Xuesong Wang
He Zhao
Edwin V. Bonilla
Venue: ICML 2025 oral (2025)
Keywords: neural processes
Rényi divergence
Primary area: deep_learning->generative_models_and_autoencoders
TL;DR: Using Rényi divergence for robust inference of neural processes

Abstract: Neural Processes (NPs) are deep probabilistic models that represent stochastic processes by conditioning their prior distributions on a set of context points. Despite their advantages in uncertainty estimation for complex distributions,  NPs enforce parameterization coupling between the conditional prior model and the posterior model. We show that this coupling amounts to prior misspecification and revisit the NP objective to address this issue.  More specifically, we propose Rényi Neural Processes (RNP), a method that replaces the standard KL divergence with the Rényi divergence, dampening the effects of the misspecified prior during posterior updates. We validate our approach across multiple benchmarks including regression and image inpainting tasks, and show significant performance improvements of RNPs in real-world problems. Our extensive experiments show consistently better log-likelihoods over state-of-the-art NP models.

Rényi Neural Processes
Xuesong Wang'
He Zhao! Edwin V. Bonilla!
Abstract
its parameters with an approximate posterior model. In this paper we show that this parameter sharing amounts to prior misspecification and can yield biased estimate of the pos- terior variance and deteriorate predictive performance (Can- non et al., 2022; Knoblauch et al., 2019). Such misspec- ification can be worsened under noisy context sets (Jung et al., 2024; Liu et al., 2024). Other cases of prior mis- specification encompass domain shifts (Xiao et al., 2021), out-of-distribution predictions (Malinin & Gales, 2018) and adversarial samples (Stutz al., 2019).
Neural Processes (NPs) are deep probabilistic models that represent stochastic processes by con- ditioning their prior distributions on a set of con- text points. Despite their advantages in uncer- tainty estimation for complex distributions, NPs enforce parameterization coupling between the conditional prior model and the posterior model. We show that this coupling amounts to prior misspecification and revisit the NP objective to address this issue. More specifically, we pro- pose Rényi Neural Processes (RNP), a method that replaces the standard KL divergence with the Rényi divergence, dampening the effects of the misspecified prior during posterior updates. We validate our approach across multiple bench- marks including regression and image inpainting tasks, and show significant performance improve- ments of RNPs in real-world problems. Our ex- tensive experiments show consistently better log- likelihoods over state-of-the-art NP models.
To address the prior misspecification caused by parameter- ization coupling in vanilla NPs, several studies have been proposed to relax the constraint (Wang et al., 2023; Wang & Van Hoof, 2022; Wu et al., 2018; Wicker et al., 2021). For instance, the prior and the posterior models can share partial parameters instead of the entire network (Rybkin et al., 2021; Liu et al., 2022); hierarchical latent variable models are also used where both models share the same global latent variable and induce prior or posterior-specific distribution parameterization (Shen et al., 2023; Requeima et al., 2019; Kim et al., 2021; Lin et al., 2021).
Instead of modifying the specifics of the NP model or its underlying architectures, in this paper offer new in- sight into prior misspecification in NPs through the lens of robust divergences (Futami et al., 2018), which seek to learn an alternative posterior without changing the parameters of interests. Instead of minimizing the standard KL diver- gence between the prior and posterior distributions, robust divergences are theoretically guaranteed to produce bet- ter posterior estimates under prior misspecification (Verine et al., 2024; Regli & Silva, 2018). The Rényi divergence (Li & Turner, 2016; Van Erven & Harremos, 2014b), for in- stance, introduces an additional parameter a to control how the prior distribution can regularize the posterior updates. This parameter allows us to reduce the regularization effects of the misspecified prior (Knoblauch et al., 2019), thereby mitigating the biased estimates of the posterior variance, avoiding oversmoothed predictions (Alemi et al., 2018; Hig- gins et al., 2017), and achieving performance improvements.
1. Introduction
Neural processes (NPs) (Garnelo et al., 2018b) strive to represent stochastic processes via deep neural networks with desirable properties in uncertainty estimation and flexi- ble feature representation. The vanilla NP (Garnelo et al., 2018b) predicts the distribution for unlabelled data given any set of observational data as context. The main advan- tage of NPs is learn a set-dependent prior distribution, where the KL divergence is minimized between posterior distribution conditioned on target set with new data and the prior distribution conditioned on the context set (Kim et al., 2019; Jha et al., 2022; Bruinsma et al., 2023).
However, as the parameters of the conditional prior are un- known, NP proposes coupling scheme where the model that parameterizes the prior distribution is forced to share
CSIRO's Data61, Australia. Correspondence to: Xuesong Wang <xuesong, wang @ d.ta6l.csinn .esiro.au>.
In light of this, we propose Rényi Neural Processes (RNPs) that focus on improving neural processes with a more robust objective. RNP minimizes the Rényi divergence between the posterior distribution defined on the target set and the true
Proceedings of the nnnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Renyi Neural Processes
posterior distribution given the context and target sets. We prove that RNP connects the common variational inference and maximum likelihood estimation objectives for train- ing vanilla NPs via the hyperparameter through which RNP provides the flexibility to dampen the effect of the mis- specified prior and empower the posterior model for better expressiveness.
The approximate posterior distribution for VI-based meth- ods is usually chosen as (a(2) = go (z (C, TT). As the param- eters of the conditional prior p. (z) (z) Yc) are unknown, NPs couple its parameters with the approximate posterior P& (z) X Yc) Ri go(z|Xc. Yc). We now replace the no- tation of the approximate posterior with to be consistent with the ML objective:
Our main contributions are summarized as:
- -vvi(0, 4)) R Be (x) log (Yr) =)-
DKL (x (z, (x)))))
We identify that the parameter coupling between the con- ditional prior and the approximate posterior inherent to NPs amounts to prior misspecification.
The KL term in Eq is sometimes referred to as the consis- tency regularizer (Wang et al., 2023; Foong et al., 2020), which assumes the parameter coupling between the condi- tional prior and the posterior. This assumption, as will show later, is the source of the inference suboptimality of vanilla NPs in the existence of finite context data.
2. We introduce a new objective RNP that addresses this misspecification and that unifies the variational inference (VI) and maximum likelihood estimation (MLE) objectives.
3. We show that our RNP method can be applied to sev- eral SOTA NP families without changing the models, and provides generalization performance improvements over competing approaches
The Rényi divergence (RD) (Van Er- ven & Harremos, 201 Rényi, 1961) is defined on two distributions with hyperparamter a E (0, +00) and a o 1:
Rényí Divergences:
Da (a(()))lp((z)
log (()))(() p(z)'=odz dz
(4)
2. Preliminaries
p(z)
1-a
log E. (x(())
9(4)
Neural processes are family of deep probabilistic models that represent stochastic pro- cesses (Wang & Van Hoof, 2020; Lee et al., 2020). Let f: : : > Y be a function sampled from a stochastic pro- cess p(f) where each fz maps some input features x to an output y and Durin and iess are meta-tasks induced by dif- ferent fusin and frest during meta- veta-training and meta-testing. For specific task ID,, we split the data further into con- text set C (Xe, Yc) == {(xm: ) and target set T (Xr. Yr) == {(x..,,) } = \C Our goal is to predict the target labels given the target inputs and the observable context set: p(Y
Neural Processes:
Note that the RD is closely related to the KL divergence in that if a + 1 then (ullp) + DxL((lll)) (Van Erven & Harremos, 2014 In other words, choosing a close to for variational inference would result in a posterior as close to the standard VIs. Changing the KL divergence to RD can induce robust posterior via the hyperparameter a, as the influence of the density ratio l determines how much we can penalize the posterior with the prior. With the flexibility of choosing o. the model can adjust the de- gree of prior penalization. When the prior is misspecified, choosing the RD can lead a more robust posterior that focuses more on improving the likelihood and less on prior regularization (Futami et al., 2018; Regli & Silva, 2018).
NPs (Garnelo et al., 2018b) introduce latent vari- able z to parameterize the conditional distribution p(fIC) and define the model as p(Yr|Xx, Xc, (Yr/XTT (2|xx, Yc)dz where 0 and & are net- work parameters of the likelihood and prior (also known as recognition) models, respectively. Due to the intractable likelihood, two types of objectives including the variational inference (VT) and maximum likelihood (MI.) estimation have been proposed to optimize the parameters (Foong et al., 2020; Nguyen & Grover, 2022; Bruinsma et al., 2023; Guo et al., 2023):
3. Rényi Neural Processes
In this section we describe our Rényi Neural Process (RNP) framework, a simple yet effective strategy which provides a more robust way to learn neural processes without changing the model. We start by analyzing the main limitation of the standard neural process objective and present a motivating example. We illustrate case where the prior is misspecified and describe our new objective with the RD to mitigate this.
- -Lvr(0, do. s0) = Eraa (By.te) log 2)) DKL. (20(2)llp Yc))]
3.1. Motivation: Standard neural processes and prior misspecification
Definition 3.1. (Prior misspecification (Huang et al., 2024)) Let q. Yz) be distribution associated with a map X x -- Z parameterized by y and p(z|X, Y) is the ground truth distribution. Then, (4 X, Y), E () defines a
(Lm = Eo (Epp)alXe..o)) lor ] (2)
'Our code is published at https: httees:/ggtttubbch
esi ro- wwwmm//eennnnuuraaae
2

---

Renyi Neural Processes
0.0
NP Plum
BNN
Irdh pred ciive mean
cantiad HGC +10
predectve
Truth
Garges -10
-BC
-1.0
-1
-1.0
(a) Posterior distributions.
(h) Predictions of ANP using Lvi
(c) Predictions using CANT
Figure 1. An illustrative example. (a) Two 2D Gaussian posteriors q(x(C) conditioned on the context set using CNP VI ohjective and our
CaNP. (b) Predictive results on a GP regression dataset using the VI objective. (c) Results obtained with our RNP objective.
set of distributions on the space Z induced by the model. The prior model is misspecified if V ① (2((2)) * YY)
predictions as shown in Fig (c)). We will now introduce our new neural process learning method, which can be applied not only to the VI objective, but also the ML objective where SOTA NPs were trained with.
This translates to NPs as the approximate prior model q (x Yc) in Eq 3 can not recover the ground truth prior p(z) Xc, Yc) for any parameterization of o. We now show how this definition can assist us to analyze how mis- specified prior model can hinder neural processes learning.
3.2. Proposed Method: Neural processes with the new objectives
New objective for VI-based NPs. The main issue of NPs is the prior approximation q, (z) Ye) wrt the true prior Pc Yc). In this case, the posterior variance may be critically overestimated some regions and underestimated in others. We therefore seek to obtain an alternative pos- terior distribution to alleviate this prior misspecification. Depending on whether the original NP framework is trained using the VI or the ML objective, we can revise the objec- tive by minimizing the RD instead of KLD on two distri- butions. More specifically, in the case of VI-NPs where the inference of the latent variable g()) is of interest, the RD between the the approximated posterior distribution q (z XT, Yr,
Proposition 3.2. Due to the coupling parameterization of in Eq 3, the NP prior model can be misspecified, which is unable to bound the ground truth marginal likelihood Xx,
Detailed proof can be found in Supp A.4. We state the main finding here: the prior term in the ELBO gradient estimate of posterior parameters is approximated in Lv1:
(2,,) V & log P() & Eg (2)CTT V sp log u (2]C). Due to this parameter coupling, we now have a "Iearned" prior model 9. (z((z) that can move towards posterior sam- ples z 4p C, T) which can lead biased estimate of the posterior and peculiar optimization dynamics. We will later show that by using objective, this biased term is scaled by the data likelihood such that the gradient is smaller if the posterior is not likely to generate the data, therefore dampening the effects of prior misspecification.
9 (z, T) and the true poste- rior p(z) Xr, Yr, Xc, Yc) = p(z C, T) is minimized:
min Da (go T) 0,0
Po Xr (2 1-a Cx,
RO max 8,0 1-
log (el.,)
Illustrative example: We first use Fig to compare the posteriors and predictions of neural processes using objectives. The objective of this example is to compare how the vanilla NP and our to-be-proposed RNP behave when the prior is misspecified. Both prior models are potentially misspecifted due to the coupling parameterization scheme. We delay the introduction of the formulation of CANP in the next section, which is unnecessary for the illustration.
where the details can be found in A.5. Eq 5 an approximation obtained by replacing p(z]Xc, Yc) with qe (2) Yc). We can approximate the intractable expec- tation with Monte Carlo:
1 Lavr(0, p) I Boan log 1
K
po Nr, Zu (2) 1-a 4: (2x [C
(6)
)
The resulted posteriors in Fig ((a) show that RNPs ob- tain a much smaller variance estimate, suggesting that the DKL. (2)T 2. (z|C)) is too strong in the vanilla NPs. As result, vanilla NPs produce oversmoothed predictions (Fig 1(b) with a large variance that underfits the data). In contrast, our RNP dampens the impacts of prior misspecifi- cation and produces smaller variance but more expressive
where Z g (z|C,T)
Tuning E (0, 1) can mitigate prior misspecification in RNPs. We will present how RNP can handle prior misspec- ification through the gradients of the parameters of the en- coder networks &o (more details can be found in Supp A.8):
3

---

Renyi Neural Processes
the new RNP objective essentially reweights the samples based on their likelihood. We define polyn, z Xn; C) po(yn, z Xx, (z((C)( and use Po(z]Xc, Yc) 2 p(z|C). Note that this prior can still be misspecified in NN parame- ters of ip and the family of distributions we choose for the approximation distribution p. Then Eq 11 can be approxi- mated by Monte Carlo:
K Σ
wk
VoLANP=
v log wk
k=1
k=
wk
Po(Yr Xx, go (2+|(C 9: (2 TT) T³)
where wkk-
Zt ~ (T (8)
Note that V log wh corresponds the gradient of the Lvz objective. As Daudel al. (2023) pointed out, Rényi varia- tional inference amounts to importance weighted varia- tional inference where the weight wa is scaled by the power of (1 a). Therefore, the prior gradient inside the log term V log (z/() is also scaled by the weight Σt wi where high-liken gighllikelihod samples are given bigger weights whereas low-likelihood samples are given smaller weights. As result, even if the prior misspecified, as long as the the weight is small, it will affect posterior update. We can control how much we trust the likelihood weight to guide the posterior by tuning the a E (0, 1). When a 1 and k- = 1 the RNP will recover the LVι behavior. When = 0 the RNP recovers the maximum likelihood behavior as introduced next.
CRNPML(0,P)) Bo (a 1).V
1
Σ log(
(12)
K E Poly BR, xn)) where Zu ~ Pz(()C K
weg
One advantage of ML-based method is that we do not need to estimate the density of the samples from the prior model. Hence, reparametcrization tricks can be applicd to obtain samples from non-standard distributions: 2 si (Xc, Yc, ck), ck ~ (O I) with neural network S.-
3.3. Properties of RNPs
We generalize the theorem from Rényi variational inference (Li & Turner, 2016) RNPs:
New objective for ML-baaed NPs. As the goal of NPs is to maximize the predictive likelihood instead of inferring the latent distribution, another type of NPs directly parame- terizes the likelihood model without explicitly defining the latent variable z. Following Futami et al. (2018), we can rewrite the maximum likelihood estimation as minimizing the KLD between the empirical distribution p(y x, C) and
Theorem 3.3. (Monotonicity (Li & Turner, 2016))
CANP continuous and non-increasing with respect to the hyper-parameter a.
Proposition 3.4. (Unification of the objectives)
CML. = LANP, 2 P.N 2 LRN = Lv1-
the model distribution p(y x, C, B):
Theorem 3.3 and Proposition 3.4 (Proof see Supp A.7) bridge the two commonly adopted objectives via our RNP framework. We show that these objectives are bounded by log p(Yr|/X,, z) (), Yc). Although it is not the marginal due the approximation q(, s p(z)Xcc Yc), the gradient of the RNP is more robust against misspecified priors and hence improve the posterior update.
(M)(() = max EE... log Xr,
1
max Re
Po Xn. C)
N
23 min Epran DKL (plylx, (ylxoly)x,))))))))))) C)..
where p(y(x, x,, C) is the empirical distribution defined as 1 > 0(y,yn) vn) where yn are samples from the unknown distribution p" (y(y/x C). Replacing the KLD with RD gives:
N
3.4. Prior, posterior and likelihood models
One main advantage of RNP is do not need to change the parameters of interests of the original NP models. Therefore, for the likelihood model pe Nr/,X,,2 z) we can adopt simple model architectures like NPs (Garnelo et al., 2018b) 8b) which assume independence between target points pe Po Yr|/Xr, 17,2 2) = I Po (y (yn|xn, 2). The distri- bution of each target point is then modeled as Gaussian po po 2) A (hyp (Xn, z), Diag (h.(Xn, z))), and the decoder networks h1 and hi map the concatenation of the input feature x, and z to the distribution parameters.
min Enrn De (ply xx, C) (po (y)xx C))
N
se min Enra N a
Σ
log "(x,lx., C) + Const
(10)
CRNPML(0) = Brrin (a 1)N
Σ log log 1 -- "yn (X., C)
1-0 (frolven C)dz Cidz)
EErnn (a 1).V
Σ log
(11)
The prior model 4 (z)X Yc) is more interesting as it is a set-conditional distribution and we are supposed to sample from it and evaluate the density of the samples.
where details can be found in A.6. Note that a = corresponds to the maximum likelihood estimation and
4

---

Renyi Neural Processes
One feasible solution is define a parametric distribu- tion DeepSet (Zaheer et al., 2017). For instance, go (z Yc) = N (gn (h(C)) (ia (h(C)) where ((C) = [
Under the existing NP setting, more members were intro- duced with different inductive biases in the model (Jha et al., 2022; Bruinsma et al., 2023; Dutordoir et al., 2023; Jung et al., 2024; Vadeboncoeur et al., 2023). For instance, at- tentive neural processes (Kim et al., 2019) incorporated dependencies between observations with attention neural works. Convolutional neural processes (Foong et al., 2020; Huang et al., 2023) assume translation equivariance among data points. These two methods explicitly defined the latent variable which requires density estimation. Recent works such as transformer neural processes (Nguyen & Grover, 2022) and neural diffusion processes (Dutordoir et al., 2023) turn to marginal likelihood maximization and do not have the latent distribution.
|c| Em m=1 h(xmm is a DeepSet function on the context set C. In practice diagonal Gaussian distri- butions worked well with high dimensional latent vari- ables z. ANPs (Kim et al., 2019) incorporate dependen- cies between context points q, (z(C) ( Vi:m) using self-attention networks. But one can consider more flexible distributions such as conditional normaliz- ing flows (Luo et al., 2023) for sample and density esti- mation. As previously stated, the posterior distribution is defined by coupling its parameters with the prior. There- fore the posterior (z) XT, Yr, Xc, Yc) in the DeepSet case can be represented as (z(z|(Xg N (hC,, T)), Diag(go (h(C, To apply stochastic gradient descent over the parameters of the posterior, we applied the reparameterization trick to obtain samples Zi:- ga (M(C, T))= * e + 9a (M(C,T) E ~ (0,
Xc,
Other neural processes that claim to provide exact (Markou et al., 2022) or tractable inference (Lee et al., 2023; Wang et al., 2023) have been introduced. Stable neural pro- cesses (Liu et al., 2024) argued that NPs are prone to noisy context points and proposed a weighted likelihood model that focuses on subsets that are difficult to predict, but do not focus regularizing the posterior distribution. Compared to variational inference based methods, non-VI Vo predictions can be less robust to noisy inputs in the data (Futami et al., 2018). 1). It is also challenging to incorporate prior knowl- edge (Zhang et al., 2018) into these neural processes, which could be beneficial when no data is observed for the task. Several works introduced strategies to incorporate explicit priors in the function space rather than low-dimensional la- tent variables, we refer the readers to (Ma et al., 2019), (Ma & Hernández-LLoatt, 2021), (Rodríguez-Santana al., 2022) and (Rudner et al., 2022) for their formulations.
3.5. Inference with Rényi Neural Processes
During inference time, as we cannot access the ground truth for the target outputs YT, we use the approximate prior (z)[Xc, Yc) instead of the posterior distribution (2) Yr, Xo, Yc) to estimate the marginal distribution: P(Yr)Xr,Xe Yc) f po(Yr|Xr,,)
Yo)dz
(13)
1 2 K Σ po (Yr|/XT 2x), 2s (x,
We now provide the pseudo code for Rényi Neural Pro- cesses in Supp Algorithm 1. In addition to vanilla neural processes, our framework can also be generalized to other neural process variants as shown in the experiments section.
Robust divergences. Divergences in variational inference can be viewed as an regularization on the posterior distri- bution via the prior distribution. The commonly adopted KL divergence which minimizes the expected density ratio between the posterior and the prior is notorious for under- estimating the true variance of the target distribution (Regli & Silva, 2018). Several other divergences have since been proposed to focus on obtaining robust posterior when the imput and output features are noisy or when there are outliers in the dataset.
4. Related Work
Neural processes family. Neural processes (Garnelo et al., 2018b) and conditional neural processes (Gameeo et al., 2018a) were initially proposed for the meta learning scheme where they make predictions given few observations as context. Both of them use deepset models (Zaheer et al., 2017) to map a finite number of data points to a high dimen- sional vector and their likelihood models assume indepen- dencies among data points. The main difference is whether estimating likelihood maximization directly or introducing the latent variable and adopting variational inference frame- work. Since their introduction, Neural Processes have seen widespread adoption in various fields, including continu- ous learning (Jha et al., 2024), spatial temporal forecast- ing (Wang et al., 2021) 1) and weather prediction (Allen et al., 2025). We refer the readers to this survey (Jha et al., 2022) for more details.
Examples of robust divergences include Rényi diver- gence (Lee & Shin, 2022), beta and alpha divergences (Fu- tami et al., 2018; Regli & Silva, 2018) which require ad- ditional parameters to control the density ratio so that the posterior can focus more on mass covering, mode seeking abilities or is robust against outliers based on prior knowl- edge. (Santana et al., 2022) utilized a divergence in implicit processes and observed robust prediction against model misspecification. f-divergence (Cheng et al., 2021; Wan et al., 2020) variational inferences provide unification of different divergences under a general definition of con-
5

---

Renyi Neural Processes
vex function f but would require the specification of the function f as well its dual function. Generalized varia- tional inference (Knoblauch et al., 2019) suggested that any form of divergence can be used to replace the KL objective when the model is misspecified. As far as we understand, we are the first to analyse the limitations of NPs from the perspective of prior misspecifications. and facilitate robust divergence to enable better NP learning.
5.1. Predictive performance under parameterization- related prior misspecification
Here we investigate prior misspecificatiion caused by poor parameterization of Ppoor in (z(z)C) which can happen in NPs because their posterior and prior models are forced to couple the parameters. We focus on the predictive per- formance, i..., test log- big-liielhhood, of both the context and target sets across different datasets. Specifically, we adopted the VI-based RNP objective to train NPs, ANPs and VNPs as their model designs include the prior models. We used the ML-based RNP objective train TNP and BANP be- cause the TNP objective was originally defined using ML only and the ML objective sigmificantly outperformed the VI objective for BANPs. We set a = 0.7 to train for VI-based RNPs and analogously a = 0.3 for ML-based baselines. However, we show in section 5.3 that, in fact, these values can be set optimally via cross-validation. To put the baseline models in the spectrum, VI solutions (using the KLD), and a = 0 corresponds to the maximum likelihood solutions.
5. Experiments
Datasets and training details. We evaluate the proposed method on multiple regression tasks: 1D regression (Gar- nelo et al., 2018a; Gordon et al., 2019; Kim et al., 2019; Nguyen & Grover, 2022), image inpainting (Gordon et al., 2019; Nguyen & Grover, 2022). ID regression includes three Gaussian Process (GP) regression tasks with different kernels: RBF, Matern 5/2 and Periodic. Image inpainting involves 2D regression on three image datasets: MNIST, SVHN and CelebA. Given some pixel coordinates x and intensities y as context, the goal is to predict the pixel value for the rest of image. More details about the training setups can be found in A.3.
corresponds to the standard
Table shows the mean test log- log-likelihood + one standard deviation using different random seeds for each method. We see that RNP consistently improved log- likelihhodd over the other two objectives and ranked the highest for all the baselines. RNP also consistently achieved better likelihood on TNP-D and VNP which generally outperform other base- line models across datasets. Some prominent improvements were achieved in harder tasks in ID regression such as ANP Periodic and BA-NP Periodic where the vanilla NP objec- tives underperform. As previously illustrated in Fig (bb and Fig I(c), RNP improves predictive performance by mitigating the oversmoothed predictions periodic data, This could suggest that misspecified prior model in the vanilla NP objective imposes an unjustifiable regularization on the posterior and hinder the expressiveness of the pos- terior and consequently predictive performance. RNP also significantly improved test likelihood of BA-NP and VNP image inpainting tasks, demonstrating the superiority of RNPs on higher dimensional data.
Baselines. We first validate our approach on state-of-the the art NP families: neural processes (NP) (Garnelo et al., 2018b), attentive neural processes (ANP) (Kim et al., 2019), Bayesian aggregation neural processes (BA-NP) (Volpp et al., 2021), transformer neural processes with diagonal covariances (TNP-D) (Nguyen & Grover, 2022). and versa- tile neural processes (VNP) (Guo et al., 2023). For VNPs, they chose different parameterizations for the prior and pos- terior models, which can be used to validate if our objective is superior than simply decoupling the two models. We generalize the NP objective to RNP using Eq 12 or Eq 6 depending on whether the baseline model infer the latent dis- tribution p(z). The methods are considered as a special case of a= 1 of RNP if the baseline model uses the VI objective or a = 0 if the baseline model uses the ML-objective. The number of samples K for the Monte Carlo is 32 for training and 50 for inference. Our experiments aim to answer the following research questions:
5.2. Predictive performance under context-related prior misspecification
(1) How does RNPs perform under parameterizatiio--elatte prior misspecification? To achieve this, We compare RNPs with SOTA NP frameworks on predictive performance. (2) How does RNPs perform under context-related prior mis- specification? Here we introduce various types of in the context set such as domain shift and noisy context. (3) How to select the optimal a values? We also carry out ablation studies investigating how to select the optimal a values, the number of MC samples and the number of context points for our RNP framework.
Here we designed another set of experiments where the prior model q(z) CC) is clearly misspecified due to poor context data Cpoor. We first tested the framework under noisy con- texts and then utilized domain shift datasets such as Lotka- Volterra and EMNIST (more detailed settings can found in section A.9). Supp Table shows test liglliieldhood under noisy context where we inject noises to the context labels. RNPs still outperformed baseline VI methods despite the general deterioration of context corruption.
Table 2 shows the test loge-iikldood for the high-performing
6

---

Renyi Neural Processes
Table Test set log- likelihood 1. The bold results indicate significant improvements of
RNP objective with p value 0.05.
Model
Set
Objective
RBF
Matern S/2
Periodic
MNIST
SVIN
CelebA
Avg rank
Lvi
0.69=0.01
0.56=0.02
-0,4940.01
0.99:0.01
3.24#0.02
1.71=0.04
2.3
context
LMI.
0.68=0.02
0.55=0.02
-0.48+0.03
1.0000.01
3.22+0.03
1.70+0.03
2.5
NP (Gamelo et al., 2018b)
Lunple)
0.78+0.01
0.66#0.01
+0.49+0.00
1.01+0.02
3.26+0.01
1.72+0.05
1.2
Lvi
0.26=0.01
0.09+0.02
+0.61+..00
0.90=0.01
3.08=0.01
1.45#0.03
target
Cuz
0.28=0.02 02
0.11
-0.61+0.01
0.92=0.01
3.07=0.02
1.47=0.02
LaMP())
0.33+0.01
0.1620.01
-0.62+0.00
0.91
3.09+0.01
1.45+0.03
Lvi
1.38=0.00
1.38=0.00
0.65=0.04
1.3800.00
4.1400.00
3,9240.07
1.3
context
Cur.
1.38+0.00
1.38+0.00
0.63+0.03
1.38+0.00
4.14+0.01
3.86+0.07
ANP (Kim et al., 2019)
Cuyplo)
1.38=0.00
1.38+0.00
1.22+0.02
1.38+0.00
4.14+0.00
3,97=0.03
1.0
Lvi
0.80=0.00
0.64+0.00
00001
106=0.01
3.65+0.01
2.24#0.03
1.7
larget
Cur
0.80#0.00
0.64+0.00
0.889+0.02
1.04+0.01
3.65+0.01
2.23+0.03
LayP(o)
0.84=0.00
0.67
-0,57+0.01
1.05+0.01
3.61+0.02
2.24+0.02
Lyr
1.43+0.03
T. 1.0+0008
0.65+0.02
0.81+0.84
2.76+0.55
1.65#0.01
context
Cuz
1.30=0.09
0.69+0.05
-0770+0.07
3.62+0.06
4.87=0.05
2.02+0.02
BA-NP (Volpp et al., 2021)
Luyplo)
1.45=0.04
1.010000
0..4100.02
3.85+0.09
4.88=0.05
2.00=0.02
Lvi
1.1950.03
0.79
0.89+0.01
0,2400644
2.60H1550
1.30HL0TT
larget
Luz
1.12+0.08
0.53+0.04
0.91+0005
3.56+0.06
4.29+0.04
1.63+0001
LayPle)
1.22+0.04
0.79+0.03
0.77+00.02
3.79:0.09
4.31+0.02
1.61+0002
1.2
CML
2.58=0.01
2.57=0.01
0,52+0.00
1.73+0.11
10.63=0.12
4.61=0.27
TNP-D (Nguyen Grover, 2022)
context
Dayplo)
2.59=0.00
2.59=0.00
+0,52+0.00
18
10.72=0.08 08
4.66+0.23
1.0
LML
1.3840.01
T.0300000
0.59+0.00
1.630007
6.69
2.45
1.8
larget
CaNP(e)
1.1+0.000
1.0400.00
+0.59+0.00
1.67+0.07
6.71+0.04
2.46+0.06
Lvi
1.37=0.00
1.37+0.00
1,2340.03
16000.110
0,80#0.00
0.08+0.03
2.0
VNP (Guo et al., 2023)
context
CuyPlo)
1.38=0.00
1.38=0.00
1.32+0.01
3.63+0.39
4.00=0.06
2.65+0.06
1.0
Lvi 0,90=0.02
0.70+0.03
00,49+0.00
1.59:0.10
0,80+0.00
0.08=0.03
target
LANP(e)
0.92=0.01
0.71+0.03
-0.48:0.00
3.62+0.37
3.8920.06
2.49=0.06
1.0
Lyrar Cardedt
Hale comtes
Lyla co dect
contee
2a
d1 25
de
05
14
15
25
-10
-20 d0
45
12
15
26
-10
-50
20
10
22
(a) ANP Lv1
(b) ANP LANP
(c) TNP-D LME
(d) TNP-D CANP
Figure 2. Prior misspecification experiment. Both models are trained on simulated Lotka-Volteraa data and tested on the real-world
Hare-Lynx dataset.
baseline TNP-D on two misspecified cases where tasks are generated from different distributions during the meta training and meta testing phase. For the 1D regression task, the model is trained using the Velka-Volterraaaeo dataset which is generally used for prey- peey-predator simulations. The dynamics is controlled by two-variable ordinary differ- ential equations: ă = 0Iæ= - Ozry. y = = + fıæy where & and y correspond to the populations of the prey and predator respectively. The parameters are chosen as 0, = 1, = 0.01, 0g = 0.5, 04 = 0.01 following (Gordon al., 2019). The number of context points is randomly sampled M ~ (15, 100), and the number of target points is N ~ (15, 100- M). We choose 20,000 functions for training, and sample another 1,000 functions for evaluation.
across multiple datasets as the impact of misspecified con- texts is alleviated via the divergence. The results in table 2 show that RNP significantly outperformed the ML objective both the training and testing data, highlighting the ro- bustness of our objective. Fig 2 shows the prediction results the Lynx dataset, where the RNP achieves better uncer- tainty estimate and tracks the seasonality of the data more efficiently than the MI. objective. We also tested TND-D on the Extended MNIST dataset with 47 classes that include letters and digits. We use classes 0-10 for meta training and hold out classes 1 1146 for meta testing under prior misspec- ification. Table 2 shows that RNP performed slightly worse on the EMNIST training task but significantly outperformed the MI. objective on the test set (last column), which demon- strates the superior robustness of the new objective under misspecification.
We then test the model on a real-world Hare-Lynx dataset which tracks the two species populations over 90 years. The input and output features were normalized via z-score nor- malization. Our method in table shows outperformance
7

---

Renyi Neural Processes
Table Loglikelihood f under prior misspecifiiation using TNP-D. The bold results indicate significant improvements with p.00.05.
Daaa Voltera)
Misspee Diett
Dinin EMNIST
Misspec Dee (class 11-46)
Objective
(Hare-Lymn Lynx)
(class 0-10)
context
larget
context
target
context
target
context
target
LML. LANP
3.09+0.22
1.98=0.11 2.12#0.06
-0.59+0.4 .47
4,44+0.41
1.54+0.05
1.56+0.07 1.4740.12
0.03=0.97 0.96#0.18
-0.20=0.57
3.32+0.15
0.1740.31
-3.63+0.09
1.52+0.08
0.70+0.15
0.04
1.41
3
shou
0.2
0.82
1,40
0.1
2-
1.39
0.80
-0.1
Best
-0.2
0.70
0.0
0.5
1.0
1.5
2.5
0.0
0.5
ANP
3.0
155
0.0
0.5
TNPD
3.0
155
R-NP
R-ANP
0
RVVNP
R-TNPD
(a) RBF. The red line indicates the best results, and the blue line corresponds to the KL results.
50
45 60
75
90
No. MC samples
No. context points
(a) MC samples
(b) Number of context points
1.64
0.90
1.350
1.62
Figure 4. Ablation study. We investigated how MC sample sizes and the number of context points affect test log-likelihood.
1.60
- 0.65
1.000
1.58
0.975
3. Gradually decrease (with granularity according to computational constraints) to O. The intuition is inspired by KL. annealing for VAE models (Bowman et al., 2016), which starts with strong prior penalization and gradually reduces the prior penalization and focuses more on model expressivity. Our results in Supp Table 5 show that this automatic a tuning strategy still outperformed the baselines.
0.80
Best
1.58
0.950
1.54
00
65
3.0
1.5
2.0
0.0
85
1.0
1.5
2.2
20
35
12
15
20
ANP
TNPO
(b) MNIST.
Figure 3. Hynerparameter(()) tuning. cross-validation is used to select the optimal a E 2).
5.4. Ablation studies
5.3. How to select the optimal a values?
Effects of Monte Carlo samples on likelihood. As both RNP (Eq 6) and RNP-ML (Eq 12) require MC ap- proximations, we investigate the effects of the number of MC samples K on predictive likelihood. We set K {1, 8, 16, 32, 50} for optimizing the RNP objective during training and use K = 50 for inference. Note that K = corresponds to the deterministic NPs (conditional NPs).
We have shown that choosing 0.7 for the VI objec- tives and 0.3 for the MI. objectives provides signif- icant performance improvements over the competing ap- proaches. One may also use the prior knowledge to select the based on the understanding of the misspecification. Nevertheless, we recognize that in other scenarios such as very different datasets and/or models, this default value may not work as well as in our experiments. For this purpose, have found that cross-validation is an effective tool for finding near-optimal values (Futami et al., 2018). We hyper- searched the a values from 0 to 2 with an interval of 0.1. As shown in Fig 3(a) and Fig 3(b), the optimal solutions are model and dataset specific. Similarly, (Rodríguez-Sanntn Santana & Hernández-Lobar 2022) tuned a for general Bayesian inference and showed that the selection of is relevant to different error metrics. Lastly, since cross-valinn validaioi is com- putationally expensive, adopted the following heuristics:
Fig 4(a) shows both the context and target log-likelihood for three methods: NP, ANP and TNP-D on the RBF dataset. As expected, increasing the number of MC samples improves the LL mean and also reduces the variance for all the meth- ods with K = 50 achieving the highest LL and the smallest variance. In practice, we set K = 32 to balance perfor- mance and memory efficiency. However, takes as few as samples during training to gain better estimates, and one can choose to increase the number of samples during infer- ence for better predictive performance or to reduce it for scalability and real-time applications. We also reported the wall clock time between standard NPs and RNPs in Supp Table and no significant differences were found with our objective. Our computational complexity is linear to the number of MC samples, which is also comparable to the VI objective.
1. start with value close to 1, which corresponds to stan- dard KL minimization.
Only consider 0 < a < since otherwise we may violate the conditions for the divergence between two Gaussians.
8

---

Renyi Neural Processes
Effects of the number of context points likelihood. We study the effect of the context points the target likelihood. During training the number of context points is sampled from (3, 50) and vary the number of context points from 5 to 95 at an interval of 10 for evaluation. The results of RBF in Fig 4(b) shows that increasing the context set size leads to improved LL for all the methods. Most methods (e.g., NP, ANP, VNP) plateaued after the number of contexts increases to more than 45, whereas TNP-D still shows unsaturated performance improvement with the increased context size.
their valuable insights on the theory discussions. We would like to thank the anonymous reviewers for the constructive feedback and advice.
Impact Statement
The social impact of our work to improve the robustness of generative models and to enable reliable machine learning algorithms. Negative impacts of our model include recov- ering facial photos from incomplete images for fraudulent scams.
Different prior and posterior parameterization. To vali- date if a simple strategy of decoupling the prior and poste- rior coupling works for NPs, we compared the RNP with the separate
References
parameterization for NPs and ANPs. The results in Supp Table showed that RNP still outperformed this baseline with much fewer parameters. The theoretical justification of the baseline could be the vi- olation of the consistency property of NPs where different parameterizations of á and &o make the KL term non-zero and the marginal is no longer consistent.
Alemi, A., Poole, B., Fischer, I., Dillon, J., Saurous, R. A., and Murphy, K. Fixing broken elbo. In International conference on machine learning, pp. 169-168. PMLR, 2018.
Allen, A., Markou, S., Tebbutt, W., Requeima, J., Bruinsma, W. P., Andersson, T. R., Herzog, M., Lane, N. D., Chantry, M., Hosking, J. S., et al. End-to-end data-driven weather prediction. Nature, pp. 1-8, 2025.
6. Conclusion
We have proposed the Rényi Neural Process (RNP), NP framework designed to mitigate prior misspecification in neural processes. RNPs bridge the variational inference and maximum likelihood estimation objectives in vanilla NPs through the use of the Rényi divergence. We have shown the superiority of our generalized objective in improving predic- tive performance by selecting optimal values. We have ap- plied our framework to multiple state-of-the-ar NP models and observed consistent log- likeliiooodd improvements across benchmarks, including ID regression, image inpainting, and real-world regression tasks. A limitation of our framework lies in drawing multiple samples with Monte-Carlo, which sacrifices some computational efficiency in exchange for better predictive performance, due to the infcasibility of computing analytical solutions the divergence. To fur- ther improve our efficiency, we suggest to adopt variance gradient
Bowman, S., Vilnis. L., Vinyals, O., Dai, A., Jozefowicz, R., and Bengio, S. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, pp. 10-21, 2016.
Bruinsma, W. P., Markou, S., Requiema, J., Foong, A. Y., Andersson, T. R., Vaughan, A., Buonomo, A., Hosking, J. S., and Turner, R. E. Autoregressive conditional neural processes. arXiv preprint arXiv:2303. /4468, 2023.
Cannon, P., Ward, D., and Schmon, S. M. Investigating the impact of model misspecification in neural simulation- based inference. arXiv preprint wxxi:2209900882 2022.
Cheng. H., Liu, X., Pereira, L., Yu, Y., and Gao, J. Poste- rior differential regularization with f-divergence for im- proving model robustness. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 1078-1089 2021.
reduction methods such as double-reparameetn estimator that could require fewer MC samples. Addition- ally, for attention- based NPs, the computational complexity of self-attention is O(N2), which impedes real-time appli- cations. We propose to adopt efficient attention mechanisms, e.g.. Nyströmformer which uses low rank approximation of the attention matrix and has the complexity of O(N).
Daudel, K., Benton, J., Shi, Y., and Doucet, A. Alpha-divergence variational inference meets importance weighted auto-encoders: Methodology and asymptotics. Journal of Machine Learning Research, 24(243): 83, 2023.
Acknowledgements
This work was supported by resources and expertise pro- vided by CSIRO IMT Scientific Computing. We would like to thank our colleague Rafael Oliveira from CSIRO's Data 61 and Kian Ming Chai from DSO National Laboratories for
Dutordoir, V., Saul, A., Ghahramani, Z., and Simpson, F. Neural diffusson processes. In International Conference on Machine Learning, pp. 8990-9012. PMLR, 2023.

---

Renyi Neural Processes
Foong, A., Bruinsma, W., Gordon, J., Dubois, Y., Requeima, J., and Turner, R. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33: 2020.
Kim, D., Cho, S., Lee, W., and Hong, S. Multi-task neural processes. arXiv preprint arXiv:21 10. 14953, 2021.
Kim, H., Mnih, A., Schwarz, J., Garnelo, M., Eslami, A., Rosenbaum, D., Vinyals, O., and Teh, Y. W. Attentive neural processes. arXiv preprint arXiv:1901.057000 2019.
Futami, F., Sato, I., and Sugiyama, M. Variational inference based on robust divergences. In International Confer- ence on Artificial Intelligence and Statistics, pp. 313-822. PMLR, 2018.
Knoblauch, J., Jewson, J., and Damoulas, T. Generalized variational inference: Three arguments for deriving new posteriors. arXiv preprint arXiv: 1tt::9944 1904.02063, 2019.
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Saxton, D., Shanahan, M., Teh, Y. W., Rezende, D., and Eslami, S. A. Conditional neural processes. In Interna- tional conference on machine learning, pp. 1704-1713. PMLR, 2018a.
Lee, H., Yun, E.,, Nam, G., Fong, E., and Lee, J. Mar- tingale posterior neural processes. arXiv preprint arXiv:2304.09431, 2023.
Lee, J., Lee, Y., Kim, J., Yang. E.,, Hwang. S. J., and Teh, Y. W. Bootstrapping neural processes. Advances in neural information processing systems, 33:6606-6611 2020.
Garnelo, M., Schwarz, J., Rosenbaum, D., Viola, F., Rezende, D. J., Eslami, S., and Teh, Y. W. Neural pro- cesses. arXiv preprint arXiv: 10X.:v8877.01222 2018b.
Lee, K. and Shin, J. Rényicl: Contrastive representation learning with skew rényi divergence. Advances in Neural Information Processing Systems, 35:6463-6477, 2022.
Gordon, J., Bruinsma, W. P., Foong, A. Y., Requeima, J.,
Dubois, Y., and Turner, R. E. Convolutional conditional neural processes. arXiv preprint arXiv: 0910. 13556, 2019.
Li, Y. and Turner, R. E. Rényí divergence variational infer- ence. Advances in neural information processing systems, 29, 2016.
Guo, Z., Lan, C., Zhang, Z., Lu, Y., and Chen, Z. Versatile neural processes for learning implicit neural representa- tions. arXiv preprint arXiv:2301.08S8
2023.
Lin, X., Wu, J., Zhou, C.,, Pan, S., Cao, Y., and Wang, B. Task adak-ade neural process for user cold-start recom- mendation. In Proceedings of the Web Conference 2021, pp. 1306-1316, 2021.
Higgins, I., Matthey, L.., Pal, A., Burgess, C. P., Glorot, X., Botvinick, M. M., Mohamed, S., and Lerchner, A. beta- vae: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 3, 2017.
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficiee fine- tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1950-1965, 2022.
Huang, D., Haussmann, M., Remes, U., John, S., Clarté, G., Luck, K., Kaski, S., and Acerbi, L. Practical equiv- ariances via relational conditional neural processes. Ad- vances in Neural Information Processing Systems, 36: 29201-29238. 2023.
Liu, H., Jing, L., and Yu, Neural processes with stability. Advances in Neural Information Processing Systems, 36, 2024.
Huang. D., Bharti, A., Souza, A., Acerbi, L., and Kaski, S. Learning robust statistics for simulation-basee infer- ence under model misspecification. Advances in Neural Information Processing Systems, 36, 2024.
Luo, L.,, Li, Y.-F., Haffari, G., and Pan, S. Normalizing flow-based neural process for few-shot knowledge graph completion. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 900-910, 2023.
Jha, S., Gong, D., Wang, X., Turner, R. E., and Yao, L. The neural process family: Survey, applications and perspec- tives. arXiv preprint arXiv:2209.005/ 17, 2022.
Jha, S., Gong, D., Zhao, H., and Yao, L. Npel: Neural processes for uncertainty- aware continual learning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Ma, C. and Hernández-Lobato, J. M. Functional variational inference based on stochastic process generators. Ad- vances in Neural Information Processing Systems, 34: 21795-21807, 2021.
Jung, M. C., Zhao, H., Dipnall, J., and Du, L. Beyond unimodal: Generalising neural processes for multimodal uncertainty estimation. Advances in Neural Information Processing Systems, 36, 2024.
Ma, C., Li, Y., and Hermández-Lobato, J. M. Variational im- plicit processes. In International Conference on Machine Learning, pp. 4222-4233 PMLR, 2019.
10

---

Renyi Neural Processes
Malinin, A. and Gales, M. Predictive uncertainty estima- tion via prior networks. Advances in neural information processing systems, 31, 2018.
Vadeboncoeur, A., Kazlauskaite, I., Papandreou, Y., Cirak, F., Girolami, M., and Akyildiz, O. D. Random grid neural processes for parametric partial differential equations. In International Conference on Machine Learning, pp. 34759-34778. PMLR, 2023.
Markou, S., Requeima, J., Bruinsma, W. P., Vaughan, A., and Turner, R. E. Practical conditional neural pro- cesses via tractable dependent predictions. arXiv preprinl arXiv:2203.087755 2022.
Van Erven, T. and Harremos, P. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Infor- mation Theory, 60(7):3797-3820, 2014a.
Nguyen, T. and Grover, A. Transformer neural processes: Uncertainty-awant meta learning via sequence modeling. arXiv preprint arXiv:2207.0411 07019, 2022.
Van Erven, T. and Harremos, P. Rényi divergence and kullback-leibler divergence. IEEE Transactions on Infor- mation Theory, 60(7):3797-3820, 2014b.
Regli, J.-B. and Silva, R. Alpha-beta divergence for varia- tional inference. arXiv preprint 1r05 2018.
Verine, A., Negrevergne, B., Pydi. M. S., and Chevaleyre, Y. Precision-recht divergence optimization for generative modeling with gans and normalizing flows. Advances in Neural Information Processing Systems, 36, 2024.
Rényi, A. On measures of entropy and information. In Pro-
ceedings of the fourth Berkeley symposium on mathemati- cal statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pp. 547-562. University of California Press, 1961.
Volpp, M., Flürenbrock, F., Grossberger, L., Daniel, C., and
Neumann, G. Bayesian context aggregation for neural processes. In ICIR, 2021.
Requeima, J., Gordon, J., Bronskill, J., Nowozin, S., and
Turner, R. E. Fast and flexible multi-task classification using conditional neural adaptive processes. Advances in neural information processing systems, 32, 2019.
Wan, N., Li, D., and Hovakimyan, N. F-divergence varia- tional inference. Advances in neural information process- ing systems, 33:17370-17379, 2020.
Sedriguuz-Santaane S. and Hernández- D. Adversar- ial a-divergence minimization for bayesian approximate inference. Neurocomputing, 471:260-277 274, 2022.
Wang, Q. and Van Hoof, H. Doubly stochastic variational in- ference for neural processes with hierarchical latent vari- ables. In International Conference on Machine Learning, pp. 100180 10028. PMLR, 2020.
Modriguez-Santanne S., Zaldivar, B., and Hernandez- Lobato, D. Function- space inference with sparse implicit pro- cesses. In International Conference on Machine Learning, pp. 18723-18740. PMLR, 2022.
Wang, Q. and Van Hoof, Learning expressive meta- representations with mixture of expert neural processes. Advances in neural information processing systems, 35: 26242-26255, 2022.
Rudner, T. G., Chen, Z., Teh, Y. W., and Gal, Y. Tractable function-space variational inference in bayesian neural networks. Advances in Neural Information Processing Systems, 1122686-22695 2022.
Wang. Q., Federici, M., and van Hoof, H. Bridge the in- ference gaps of neural processes via expectation maxi- mization. In The Eleventh International Conference on Learning Representations, 2023.
Rybkin, O., Daniilidis, K., and Levine, S. Simple and effective vae training with calibrated decoders. In Inter- national conference on machine learning, pp. 9179-9189. PMLR, 2021.
Wang, X., Yao, L.,, Wang, X., Paik, H.-Y., and Wang, S. Global convolutional neural processes. In 2021 IEEE International Conference on Data Mining (ICDM), pp. 699-708, 2021. doi: 10. 10910000055122.
Santana, S. R., Ortega, L. A., Hermández-Lobato D., and Zaldivar, B. Correcting model bias with sparse implicit processes. arXiv preprint arXiv:2207. 10673, 2022.
.00081.
Wicker, M., Laurenti, L., Patane, A., Chen, Z., Zhang, Z., and Kwiatkowska, M. Bayesian inference with certifi- able adversarial robustness. In International Conference Artificial Intelligence and Statistics, pp. 2431-2439, PMLR, 2021.
Shen, J., Zhen, X., Wang, Q., and Worring, M. Episodic multi-task learning with heterogeneous neural processes. Advances in Neural Information Processing Systems, 36: 75214-75228, 2023.
Stutz, D., Hein, M., and Schiele, B. Disentangling adversar- ial robustness and generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6976-6987, 2019.
Wu, A., Nowozin, S., Meeds, E.,, Turner, R. E., Hernandez- Lobato, M., and Gaunt, A. L. Deterministic variational inference for robust bayesian neural networks. arXiv preprint / 2018

---

Renyi Neural Processes
Xiao, Z., Shen, J., Zhen, X., Shao, L.,, and Snoek, C. A bit more bayesian: Domain-invariant learning with uncer- tainty. In International Conference on Machine Learning, pp. 11351-11361 PMLR, 2021.
Zaheer, M., Kottur, S., Ravanbakhsh, S., Poczos, B., Salakhutdinov, R. R., and Smola, A. J. Deep sets. Ad- vances in neural information processing systems, 30, 2017.
Zhang, C., Bütepage, J., Kjellström, H., and Mandt, S. Ad- vances in variational inference. IEEE transactions pattern analysis and machine intelligence, 4108):2008- 2026, 2018.
12

---

Renyi Neural Processes
A. Appendix
A.1. Pseudocode
Algorithm 1 Rényi Neural Processes
Input: Context inputs X and outputs Yo. Target inputs Xy, and target outputs Yr during training
Output: Target distribution p(Yr Xr, Xc, Yc)
Training:
for epoch=1 to max max_maa_eeaeeeee eepoch do
Sample a context set (Xc, Yc) and a target set (xr.Yr)
Obtain the posterior distribution with the encoding network: Oe XT, Yr. Xc; Yc) Obtain the approximated prior distribution with the encoding network: qo (2/ Yc) Sample 2 Zy 9p (z Yr, Xc,Yo Yc)
Construct the likelihood model with the decoding network: NY, 24)
Compute the objective LANP using Eq 6 or Eq 12
Update the encoder parameters with V.L (Eq 8) and the decoder parameters with VoL end for
Inference:
Construct the approximated prior 9, (z| Yc) Sample y1,..., 3 q: (z) (z) Ye) Predict the target distribution pa (Yr Zk)
Estimate the log- ligllikelhoodd of the target outputs using Eq 13.
A.2. Notation
The notations used in the article is summarized in Table 3.
Table 3. Notation
Name
Description
f
function sample from stochastic process
X'
input space
y
output space
latent space
x
input features
y
output features
latent variable representing stochasticity of the functional sample f
X
R RMD context inputs
Yo
RMX1 context outputs
XT
RNDD target inputs
YT
NX target outputs
M
number samples in the context set, indexed by
N
number samples in the target set, indexed by dimension of input features
DE
D
dimension of input features notation for Context and Target parameters the posterior model q( X, Y) parameters in the likelihood model p(Y z, X) number of samples for Monte Carlo approximation
C.T
13

---

Renyi Neural Processes
A.3. Training details
For ID regression tasks, given a function f sampled from a GP prior with varying scale and length and context set generated by such function, goal is to predict the target distribution. The number of context points is randomly sampled M (3, 50), and the number of target points is N ((3, 50 Md) (Nguyen & Grover, 2022). We choose 100,000 functions for training, and sample another 3,000 functions for testing. The input features were normalized to 2- 2].
Image inpainting involves 2D regression on three image datasets: MNIST, SVHN and CelebA. Given some pixel coordinates x and intensities y as context, the goal to predict the pixel value for the rest of image. The number of context points for inpainting tasks is M U3, 200) and the target point count N [333 200 M). The input coordinates were normalized to [-1,1] and pixel intensities were rescaled [-0.5, 0.5]. All the models be trained using single GPU with 16GB memory.
A.4. Proof of Proposition 3.2
The true ELBO of neural processes without prior approximation can be written as:
ELBO= = En (m Nr. Nr, pog Xx. (zlgl/((((C))) - log 9. 9. (z,C, 1o log (zggp(z(())
DEL (g Yr, Xc, Yc>ll((//Xe,)))))
(14a)
Tx [log po (Y
[Xr (Yr
(14b)
log PYr Xx. (X,, (z)(
+ log CCz[C)
(14c)
90 (2(2/C [
9o (2)()
log Po (o Xr, (z|CC)
p(z|C) log
(MCCT)
yg (x/C,T)
(14d)
9 (z C, T)
8z(2|01 (C)
C)) 9 (z C)
=-cv (x(CT) log
(14e)
Therefore, the VI objective will only recover the true ELBO only when (z|C) p(z|C), i.e. the ground truth posterior is identified and the prior model is well-specified. When (z(C) * p(z|C), which is usually the case, the model is optimizing an objective which not necessarily the true ELBOO.
Furthermore, we will derive the gradients of the true ELBO and the approximate ELBO Lyr w.r.t the parameters of the posteriors using the re-parameterization trick:
(2 V,ELBO- = V. (Ea.talc.)) log (0((7)Xr,21))(21))
(15a)
E. E. [logg(Yrr /Xr, (c()) + (og)( log (gx(c))), go(c) -(c,,,
(15b)
E. V. log' (x()))(()))))) (e))) V. log (o)))()((e))
(15c)
As we do not know the form of conditional prior CC) in Eq 15, the gradient is difficult to estimate. Therefore, the VI provides a way to approximate the gradient with
VoLvI V. (Tan) log Po(Yr(XTT, 2)q, (x(C)
(16a)
(x C, T)
E. Vog 1X lo - V. (ogg)(a
(16b)
Hence, now we have a "learned" prior model q,p (z) (C) that can move towards posterior samples z ~ T). However, as there no guidance to regularize (z)(C) to be close to the true prior, this prior approximation can be misspecified, which can inevitably cause biased estimate of the posterior.
14

---

Renyi Neural Processes
A.5. Derivation of LRNP (eq
min Da 0,
Yr, (oo Xo,
(17)
p(z Vr, Xc; Yc)
1-a
= =min
log E 1r 8, 0-1
(By definition)
(18)
90 9p Xx, Yr, Xc, Yc)
pz, Yr XT, Xc, Yc)
1-a
max
log E. Yr.
-Const.(Spli marginal likelihood)
(19)
0,p
a
2 2 Xx, Yr, Xc, Yc)
(2, Yr XT, Xc, Yc)
max
log E 1o X Nxx
(Equivalence of removing the constant)
(20)
0,P
a
4 2 Xr, Yr, Xc Yc)
Pa T X7. 2) 2) Xc, Yc) 90 (z XT Yr, Xc.
max
log E Nr. Ne..ee))
(21)
0,0
A.6. Derivation of CRNPML (eq 12)
start by rewriting the ML objective (Eq 9) as minimizing the KL divergence:
(0) = max Braa. log po (Y
(22)
9
max EDan
logpo C)
+ Const (Average likelihood for stabilized training)
(23)
s max Eram 0
ply|x, C) log C)dy( Definition of the empirical distribution)
(24)
max Eran
ply|x, C) [log (y x, C)- ((ygf(yxx.)) C) plygg((y) X C)] dy
(25)
min E Erai
[KLL (f(y xx C)lppo(yxx,,.mmmo)))))))))))x))) C)) (Definition of KLD and removing the constant without 0)
(26)
We now replace the KLD with RD
min Eo... [plp(y/x Cy),,) C))
(27)
min E
log 1r (y(1,x,))) (y|x, Cjdy (Definition of RD)
(28)
min Eo
log
N (yn|/xn C) (Definition of the empirical distribution)
(29)
min
E
log Σ Po
(yn Xn: ( C) + Consil Split the non-O term)
(30)
0
N
min E
log
Po Σ
1--0 "ynlxnn, (Average likelihood for stabilized training)
(31)
(a 1)N
n=l
N
1
(/ 2/x,, C)dz
min E ED
log
(32)
(a
1)N
A.7. Theoretical relationships between the LVI, LML and CRNP objectives. 3.4
Our Rényi objective unifies the common three objectives for NPs: LVI, Last (maximum likelihood estimation), and LCNP (conditional NPs or deterministic NPs).
15

---

Renyi Neural Processes
po (T (2) 1-a
LRNP: log E (e,,
(33a)
(p(2|XT, YT, Xc,
-Lviaa-+ + 1): (a log
(33b)
(x
LMLla= 0) log B, (e) (V7 Xr,2)
(33c)
-CNN (a = 0 and (z| Yc) ((x (o
(33d)
Proof of the ML objective:
Po(Yr[XT: (2/x( (z(2| Y1, Xo,
LRNP(a=0) = log F (nl
(34a)
=log / (ple//T: Yr, Xc, Yc)
Po XT. 2 q. [2) X.. Yc)
(34b)
10 (p(2|XT, YT, Xc, Yc)
log / ( Yc) log Eg (x) Vet. LML
(34c)
Proof of the NPVI objective: Next will prove that CANPAT+1 LVI. Applying Theorem 5 from (Van Erven & Harremos, 2014a) to the new posteior and prior, we have:
D (xr Yr, Xc, Yc)]lp(4[Xr: Yr, Xo, Yc))
(35a)
<xl( Yr, Xc, Yc)l plz Xr, Yr, Xo, Yc))
(35b)
(z Xr, Yr, Xc, Ye (c Yc, Xx)
Yy. log
(35c)
pzz, Yr Xr, Xc,
plz, Xx, Xe, Yc) (1. Xy log
Censt
(35d)
4, (2/ Yr, Xc, Yo)
= (el (logg(Yr/z,,...oo XT) + log Pa Xc, Yo) - log 1rn Lvr
(35e)
A.8. Gradients of Objectives for Rényi Neural Processes
Xr (2)o(((C) 1-a
VeLLNN= 1-
Vip log By Byppe c,m)
(36a)
2o (z C,C,
pa (TrYl (z (C)
Vi (o Xr. z 2) (2) (z (2( T)
E (x((.T)
Eg
(36b)
1
Go (z C, T)
pa (r z) (z C, T)
1-a)
-1
Pa poo(Yr(XTr: 2 2) (z C,
1-a
pa Xr. (x(CC)
E. ((IC,T)
( log
2o (z/C, T)
(36c)
w. Vooloo (Y Xr. (2)C)
(36d)
(2 [C T)
(u(Yr(Xr,,)).. 2) )u
1-a
pa Po Xr. z) (x)() 1-a
(36c)
(z([,, T)
no (z T)
Suppose wa Pa Ny Mu (Mu (C) where zn ~ (2(C, Te have VoLRNP- Σk=1
log Wk
(06 (C,T)
Σx
In a simple case of a Bayesian linear regression model y = 8 + e with two dimensional inputs and one dimensional output, the analytical solutions of the posterior is vatilvariate Gaussian p(0 x, y) ~ N(u, E). Suppose we use factorized
16

---

Renyi Neural Processes
Table 4. Test log- higeliikl with noisy contexts.
Model
Obj
RBF
Matern 5/2
Periodic
MNIST
SVHN
CelebA
Cvi
-0.53 + 0.01
-0.56 0.01
-0.74
0.76 + 0.02
2.81
0.04
0.89
NP
CANP
-0.45 +: 0.01
-0.50 = 0.01
-0.73 + 0.00
0.83 + 0.02
2,98 + 0.01
1.16 + 0.02
Cvr
-2.43 +: 0.19
-2.15
0.18
-0.99
0.01
0.90 + 0.02
2.83
0.06
1.55
0.05
ANP
Canr
-2.29 + 0.12
-2.11 = 0.15
-1.20 + 0.03
0.96 + 0.01
3.11 + 0.02
1.82 + 0.03
Table 5. RNP results with automatic tuning a values
Model
Set
Setting
RBF
Matern 5/2
Periodic
MNIST
SVHN
Lvs
0.69+0.01
0.56+0.02 02
-0.49=0.01
0.99=0.01
3.24+0.02
context
LRNP
0.75+0.02
0.61
-0.49:0.00
1.01
3,26#0.01
NP
Lvi
0.26+0.01
0.09#0.02
-0.6140.00
0.90#0.01
3.08=0.01
target
CRNP
0.31=0.01
0.13+0.01
-0.61#0.00
0.92+0.01
3.10=0.01
Cvi
1.38+0.00
1.38:0.00
0.65+0.04
1.38+0.00
4.1420.00
context
LANP
1.38+0.00
1.38:0.00
0.97#0.11
1.38
4.14#0.00
ANP
Lvi
0.8140.00
0.6420.00
-0.9140.02
1.0620.01
3.6520.01
target
CRNP ador
0.83+0.01
0.6620.01
-0.71=0.05
1.0620.01
3.65=0.01
Lvi
2.58+0.01
2.57
-0.5240.00
1.73+0.11
10.63=0,12
context
LdNP
2.58+0.01
2.57+0.01
-0.5240.00
1.94
10.73+0.57
TNPD
LvI
1.38+0.01
1.03+0.00
-0.5920.00
1.63+0.07
6.69#0.04
target
LANP
1.3940.00
1.03+0.00
-0.5940.00
1.56#0.02
6.71+0.24
Gaussian as an approximate posterior (0) = II; g(0,)
with g(01) = NN(1, ),
we can show the variance of the
approximate posterior factorized Gaussian varies by alpha: = Po E11 where Pa
2a (2x 1) + v
4a(1- 4a(1-a)Ef
En
is non-decreasing in
A.9. Prior misspecification settings
We consider two scenarios of prior misspecification: g(z[Cood, (p) and qZz[C, (Poa)). For ( (p), design the experiments with the following setting: keep the target data (Xr, Yr) clean and corrupt the context data with noise ic (1 3) yc + B C,c ~ (0, The noise level B is set as 0.3 for both training and testing, and the marginal predictive distribution is p(Y,|XT, Xc, Yc) and we report the test target set log-likelihood in Table 4.
For bad parameterization q(z C (Poaa). We adopted domain shift datasets since p(z) Dirain) and p(z (tes)) do not come from the same distribution. Therefore, the prior model is suboptimal when conditioned on the training parameters (z|(C, Ptrain).
A.10. Tuning of a.
We can start with training the model with the KL objective (a - 1) then gradually decrease (with granularity according to computational constraints) to The intuition inspired by KI. annealing for VAE models Bowman et (2016), which starts with strong prior penalization (close 1) reduce the posterior variance quickly and gradually reduces the prior model expressiveness. The results presented in Table 5. Our heuristics still
penalization (close to 0) and focuses managed to outperform the baselines across multiple datasets and methods.
A.11. Additional Results.
We provided some qualitative results of the baseline NPs well their corresponding RNPs in Fig 5, 6, 7. Given the same context sets, we compare the results obtained by the original method the left with the RNPs the right. The number of context is chosen to be relatively small for RBF and Matern and more context points are selected for periodic to capture the periodicity of the function. RNPs tend to predict smaller variances for different function samples. While most methods struggle with the periodic dataset, ANP and VNP using our RNP objective significantly improved the baseline models.
Table 6 compares RNP with a simple baseline model using separate parameters of the prior and posterior models.
17

---

Renyi Neural Processes
-1
(a) NP Lvs
(b) NP CANP
grouns trut prd othe mean
contost -5 get 11
grounc trUD)
n content +15
predictive moan
n Parger
-2
(c) ANP LVi
(d) ANP CRNP
cantast naaget =10
trat pred cliv
cantent parget =13
prdd cihe
(c) BA-NP LVi
(f) BA-NP LANT
groane
-1
(g) TNPD CML
(h) TNPD LRNP
grOLIN bruth
n contect +15 trret-10
ground buth pred clie moon
contest #15
1
presktive meat
target =10
2
(i) VNP LVI
ii VNP LANT
Figure 1D regression RBF dataset.
18

---

Renyi Neural Processes
0.4
0.-
-02-
-02-
(a) NP Lvs
(b) NP CANP
ground tuuh
co TExt +15
ground truth
n conTENt +15 nagget -10
credicttve mean
Earget -10
predictive mean
02
00
00-
-044
-0.4
(c) ANP LVi
(d) ANP CRNP
carteed HD0
ground cantast
sredictive moan
Carget =10
predictvve
target
021
0.2
-0--
-02
-0.4
-0.4
(c) BA-NP LVI
(f) BA-NP LANT
0.4
0.6
0.0
0.-
-0.2
-02-
(g) TNPD CML
(h) TNPD LANP
groand truth predictive mean
nomoatt -15
21ourd trut
cor taot -13
bagget -10
sredictive mean
1 Sarget =10
0,2
02-
-0.4
-0.4-
22
(i) VNP LVI
(i) VNP LANT
Figure 1D regression Matern datasct.
19

---

Renyi Neural Processes
h
-1
(a) NP Lv3
(b) NP CANP
gIouTE truo prd rther mean
grouNI TIUD)
n CONTEONE WOC
predictive mownn
n Pargec
(c) ANP LVI
(d) ANP CRNP
cantast
THAR clive
pred cihe
Darge-115
(c) BA-NP LVI
(f) BA-NP LANT
-1
TNPD LVs
(h) TNPD CANP
GrOLId Sruth
connect -50 Dorget--10
ground brts pred claee
CONTENE HOO
presktvverre
(i) VNP LVi
I VNP LANT
Figure regression Periodic dataset.
20

---

Renyi Neural Processes
Table Simple baseline comparison. In the setting column Separate for the NP frameworks and optimised using the VI objective
the prior and posterior models are parametcrised separately
Model
Set
Setting
RBF
Matern 5/2
Periodic
MNIST
SVHN
Separate PQ
0.56+#0002
0.41=0.01
-0.50=0.00
1.00=0.03
3.21=0.01
context
P
0.78=0.01
0.66#0.01
-0,4920.00
1.01+0.02
3.26#0.01
NP
Separate PQ
0.1820.01
0.01=0.00
-0.61=0.00
0.90=0.02
3.05#0.01
target
CRNP(a)
0.33+0.01
0.16#0.01
-0.62+0.00
0.91+0.01
3.09:0.01
Separate PQ
1.38=0.00
1.38=0.00
-0.170..25
1.38++.00
4.14:0.00
context
CRNP(a)
1.38=0.00
1.38=0.00
1.22+0.02
1.38++.00
4.14+0.00
ANP
Separate PQ
0.80+0.01
0.6320.01
-0.7040.02
1.0640.00
3.6620.01
target
CRNP(O)
0.84+0.00
0.67=0.00
-0,57=0.01
1.05+0.01
3.61
Table 7. Wall clock time (in seconds) comparison. 32 MC samples were chosen during training to be consistent with the results in Table 1.
Model
Setting
RBF
Matern 5/2
Periodic
MNIST
SVHN
Lvi
1028
1015
1067
3231
4620
NP
CRNPIA)
1108
1000
1048
3058
4780
Lvi
1585
1620
1721
3770
5391
ANP
CRNP(a)
1712
1693
1671
3605
5444
Table 7 added wall clock time comparison between objective and the VI objective.
21


Reviews:
[{"review_id": "psvrH8lWN7", "reviewer": "Reviewer_aNVZ", "summary": "The paper replaces the Kullback–Leibler (KL) divergence in the standard neural processes (NPs) with the Renyi divergence to mitigate the issue of prior misspecification. The proposed Renyi neural process (RNP) has a tuning parameter $\\alpha>0$ that penalizes the misspecified prior and unifies the variational inference ($\\alpha=1$) and maximum likelihood estimation ($\\alpha=0$) in the same framework. The reason of prior misspecification in NP is explained and the mitigation by RNP is well illustrated. The paper further investigates the robustness of Renyi divergence applied to other state-of-the-art (SOTA) NP algorithms and demonstrates advantages of Renyi divergence in NP applications.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our innovation of unifying the objectives and carefully reviewing details including supplementary materials and numerical results. \n\n# Additional evaluation metrics.\n\nWe have additionally reported the relative errors for two baseline methods NP and ANP on three GP regression datasets. Our objective still generally outperformed baseline models on this new metric. We will incorporate more results in the  version. \n\n| Method |   Set   | Objective |      RBF      |   Matern 5/2  |    Periodic   |\n|:------:|:-------:|:---------:|:-------------:|:-------------:|:-------------:|\n|   NP   | context | L\\_VI     | 2.56±0.12     | 5.79±0.73     | 2.88±0.09     |\n|        |         | L\\_ML     | 2.47±0.14     | 4.63±1.42     | 2.87±0.08     |\n|        |         | L\\_RNP    | **2.09±0.10** | **4.48±1.16** | **2.77±0.10** |\n|        |  target | L\\_VI     | 4.80±0.40     | 3.59±0.12     | 6.42±0.35     |\n|        |         | L\\_ML     | 4.52±0.36     | 3.33±0.07     | 6.42±0.43     |\n|        |         | L\\_RNP    | **4.23±0.35** | 3.40±0.16     | **6.22±0.25** |\n|   ANP  | context | L\\_VI     | 0.17±0.01     | 0.28±0.03     | 2.76±0.39     |\n|        |         | L\\_ML     | 0.18±0.01     | 0.24±0.03     | 2.92±0.38     |\n|        |         | L\\_RNP    | 0.17±0.01     | 0.25±0.06     | **0.52±0.08** |\n|        |  target | L\\_VI     | 2.51±0.06     | 2.42±0.05     | 8.81±0.43     |\n|        |         | L\\_ML     | 2.55±0.07     | 2.40±0.04     | 8.27±0.86     |\n|        |         | L\\_RNP    | **2.09±0.05** | **1.89±0.03** | **6.15±0.31** |\n\n\n# The effect of $\\alpha$ values on training.\n\n$\\alpha > 1$ usually impedes training as it focuses too much on improving the mass-covering of the posterior, resulting in an overestimate of the posterior variance (Please refer to Fig 3 in the paper for details).  Regarding the convergence rate, [1] showed that under mild regularity assumptions, it is bounded by $\\sqrt{n}$ with n being the number of samples rather than by $\\alpha$ values.\n\n[1] Jaiswal, P., Rao, et al, 2020. Asymptotic Consistency of $\\alpha $-Renyi-Approximate Posteriors. JMLR, 21(156).\n\n\n# Missing clarifications and typos.\n    \n-  **Smoothness in Fig 1**. \nThe difference between L_RNP and L_NP on smoothness and correlation strength is only controlled by $\\alpha$. Due to the misspecified prior regularization in standard KL, vanilla ANPs (Fig 1b) struggle with over-smoothed predictions. Our objective dampens such regularization and encourages a more expressive likelihood model that better fits the data.\n\n- **RNP advantage in Fig 2**.\n    RNP indeed did not impose a significant advantage over VI as shown in Fig 2 (a) and (b). However, our main claim in Fig 2 is that RNP achieves better uncertainty estimate than the ML objective. \n    \n- **Typos and missing captions**. \nWe thank the reviewer for pointing out the typos in the captions of Fig 1, which we will correct in the revision.  We will also add explanations for Fig 5, 6, 7 in the supplementary and fix the typos in sec 4 related work.", "claims_and_evidence": "The claims are well explained and there is enough numerical evidence to support the claims.", "methods_and_evaluation_criteria": "The paper includes comprehensive numerical studies. However, they are focused only on the objective function. It would be nice to include other metrics, e.g. relative error comparing against the truth, in one of the comparisons.", "theoretical_claims": "Yes. The proofs appear correct.", "experimental_designs_or_analyses": "Most of the experiments are well designed. I do have following questions:\n\n\n1. Figure 1: what controls the smoothness and the correlation strength of RNP, all by $\\alpha$? BTW, the labels in caption are wrong: '(c)' should be '(b)' and '(d)' should be '(c)'.\n2. Figure 2: What is the advantage of RNP (b) over VI (a) in ANP? The estimates with similar smoothness and both miss the truth between 0.5 and 1.\n3. How does $\\alpha$ affect the training? Is there any particular value that poses challenges in training, e.g. large gradients, slow convergence?", "relation_to_broader_scientific_literature": "The paper provides  a good contribution to neural process algorithms by exploring a divergence more robust to prior misspecification.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "The application of Renyi divergence to neural process is novel. The contribution that unifies VI and ML is appreciated.", "other_comments_or_suggestions": "Two sentences on page 5 under \"Robust divergence\" seem incomplete:\n\n\"KL divergence ... between the posterior...\" between the posteriors or between the posterior and the prior?\n\"Several other ... features are noise or the existence of outlier.\"?", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "bLYPw4qlNZ", "reviewer": "Reviewer_Fmj6", "summary": "The paper introduces Rényi Neural Processes (RNPs), a modification of Neural Processes (NPs) that replaces the standard Kullback-Leibler (KL) divergence with the Rényi divergence to mitigate prior misspecification. The authors argue that parameter coupling between the prior and posterior in traditional NPs leads to biased variance estimates and propose RNPs as a more flexible alternative. The method is tested on regression and image inpainting tasks, showing improved log-likelihood performance compared to state-of-the-art NP variants.\n\n\n_(for any missing input on any of the fields, please refer to the **Strengths and Weaknesses** or the **Other comments or suggestions** sections)_", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our \"well-motivated theoretical derivations\" and \"strong empirical results\".\n# Incremental novelty\nWe would like to clarify to the reviewer that our work goes beyond using RD for prior misspecification. We are the first to identify prior misspecification in the realm of NPs due to the parameter coupling between the prior conditional and the approximate posterior, and our theoretical and empirical analysis justify the use of robust divergences. Secondly, we introduce a new objective that unifies the VI and MLE objectives for NPs.\n# Adaptive $\\alpha$ value selection\nPlease kindly refer to the comments for the Reviewer GkqA for this and f-divergence related questions.\n# Results on uncertainty calibration\nWe added the results of continuous ranked probability scores (CRPS) w.r.t different α values for NP and ANP on the RBF dataset. CRPS is a commonly used to measure the forecast accuracy and lower scores indicate better calibration. The results showed that the CRPS calibration improved when α increases from 0 to 0.9.\n| α | NP            | ANP           |\n|----------|---------------|---------------|\n| 0.0      | 0.1427±0.0016 | 0.0978±3e-4 |\n| 0.3      | 0.1452±0.0029 | 0.0942±2e-4 |\n| 0.7      | 0.1370±0.0018 | 0.0827±5e-4 |\n| 0.9      | 0.1321±0.0014 | 0.0815±5e-4 |\n| 1.0      | 0.1484±0.0023 | 0.0965±3e-4 |\n# Theoretical discussions\nWe sincerely appreciate the reviewer's thoughtful feedback. While some of the questions raised by the reviewer extend beyond the immediate scope of this paper and address broader aspects of NP research, we recognize their importance and agree these points warrant further investigation. We are delighted to make every effort to clarify their concerns.\n- **Robust Bayesian inference and PAC-Bayes**\n\nIndeed, the literature on robust Bayesian inference is rich and we thank the reviewer for pointing out the need to discuss this. In particular, the relation to PAC-Bayes is fascinating. In short, dealing with model misspecification from a Bayesian perspective appears in the literature under several disguises, for example,  Gibbs posteriors, tempered posteriors and fractional posteriors [1,2]. These approaches essentially weigh the likelihood to temper its influence. Connections with PAC-Bayesian bounds controlling generalization in statistical learning have also been explored previously, more notably in [2]. These connections are extended  by explicitly analyzing variational approximations of PAC-Bayes/Gibbs posteriors [3].\nWe will expand on this in the final version. However, it is important to emphasize that our focus is on neural processes but these approaches remain exciting directions for future work.\n\n[1] Grunwald, P, et al. Inconsistency of Bayesian inference for misspecified linear models, and a proposal for repairing it. Bayesian Analysis, 2017. \n\n[2] Bhattacharya, A., et al. Bayesian fractional posteriors. The Annals of Statistics, 2019.\n\n[3] Alquier, P., et al. On the properties of variational approximations of Gibbs posteriors. JMLR, 2016.\n- **Theoretical guarantees of RNP**\n\nThe theoretical soundness of our work can be built on two parts: the frequentist consistency properties of the vanilla NPs using the KL divergence, and the relaxation of assumptions with our Rényi divergence. For the first part, we kindly refer the reviewer to section 3.3 -3.5 of the thesis [4] for prediction map approximation theory of NPs. Briefly, in the limit of infinite data, a variational family of prediction maps can recover the mean map of the true NP and the sum of the variance map and observation noise under some regularity assumptions. In the case of limited data, more assumptions including the input size, the compactness of the variational family, the boundedness of the data, and the boundedness of the stochastic process are required to guarantee the consistency. For the second part, some milder regularity conditions than KL divergence can be made for consistency [5], including a uniformly bounded prior distribution and a locally asymptotically normal likelihood model. \n\n[4] Bruinsma, W. (2022) Convolutional Conditional Neural Processes. Apollo - University of Cambridge Repository.\n    \n[5] Jaiswal, P., Rao, et al, 2020. Asymptotic Consistency of α-Rényi-Approximate Posteriors. JMLR, 21(156).\n- **Data-dependent priors**\n\nWe agree with the reviewer that it might sound un-Bayesian to have data-dependent priors, but in the case of NPs it is not only valid but necessary when viewed through the lens of hierarchical Bayes or meta-learning: the prior is being drawn from a hyper-prior. When we condition on context data, we're effectively doing Bayesian inference over a latent variable that defines the function.\n# Fixing typos and adding references\nWe will add the recommended references as well as [6] to the main text. We will also simplify some equation notations.\n\n[6] Rodríguez-Santana et al, 2022. Function-space Inference with Sparse Implicit Processes. ICML.", "claims_and_evidence": "- The parameter coupling in standard NPs leads to prior misspecification and degraded performance.\n  - *Evidence*: Theoretical derivations and empirical evaluations demonstrate that standard NPs overestimate posterior variance, leading to oversmoothed predictions.\n- Using Rényi divergence provides a tunable mechanism to reduce the impact of prior misspecification.\n  - *Evidence*: The proposed RNPs consistently outperform standard NPs and other NP variants across multiple benchmarks.\n- RNPs improve generalization without modifying model architectures.\n  - *Evidence*: The method is applied to existing NP models (ANP, VNP, TNP-D) with measurable improvements.", "methods_and_evaluation_criteria": "- The primary modification is substituting KL divergence with Rényi divergence in the NP objective.\n- Evaluations focus on predictive log-likelihood, testing the approach on 1D Gaussian process regression and image inpainting (MNIST, SVHN, CelebA).\n- Comparison against NP, ANP, VNP, and other baselines.\n- Additional experiments assess robustness under prior misspecification (e.g., noisy context points, domain shifts).", "theoretical_claims": "- \"New insight into prior misspecification in NPs through the lens of robust divergences\".\n- Proves that the standard NP prior model is misspecified due to parameter coupling.\n- Establishes that Rényi divergence generalizes KL divergence and allows tuning of prior penalization.\n- Demonstrates that RNPs unify variational inference (VI) and maximum likelihood (ML) approaches, bridging two common objectives.", "experimental_designs_or_analyses": "- Well-structured with comprehensive baselines.\n- Includes ablation studies on hyperparameter tuning ($\\alpha$ selection) and Monte Carlo sample size.\n- Tests both parameterization-induced and context-induced prior misspecification.\n- Considers real-world applications (Hare-Lynx dataset for time series forecasting).", "relation_to_broader_scientific_literature": "- Builds on extensive prior work in Neural Processes, Variational Inference, and robust divergences. It seems to cover the most relevant literature for the proposed method. \n- Connects with literature on robust Bayesian inference and alternative divergences (e.g., $\\alpha$-divergence, f-divergence).", "essential_references_not_discussed": "- A deeper discussion on the connection to PAC-Bayes approaches and other Bayesian robustness techniques could strengthen the theoretical grounding.\n- Explicit comparisons with f-divergence-based variational inference methods would be useful.\n- Regarding the choice of $\\alpha$, the authors could consider mentioning [1], which discusses the impact of $\\alpha$ in variational inference. \n- Related to the previous work, [2] seems strongly related to this contribution since they show that more flexible models in combination with robust divergences may fix prior misspecification issues. Not essential, although maybe worth mentioning here.\n- Some references to robust inference outside the NP community could provide a broader perspective. As a suggestion, the authors may consider mentioning other works that make use of robust divergences in Bayesian inference, like [3] (which, for instance, also mentions NPs as a particular case of their model).\n\n[1] Rodríguez-Santana, et al. \"Adversarial $\\alpha$-divergence minimization for Bayesian approximate inference.\" Neurocomputing 471 (2022): 260-274.\n\n[2] Santana et al. \"Correcting Model Bias with Sparse Implicit Processes.\" ICML 2022 Workshop \"Beyond Bayes: Paths Towards Universal Reasoning Systems\" arXiv preprint arXiv:2207.10673 (2022).\n\n[3] Ma, et al. (2019, May). \"Variational implicit processes\". In International Conference on Machine Learning (pp. 4222-4233). PMLR.", "other_strengths_and_weaknesses": "**Strengths:**\n- Well-motivated theoretical derivations to bolster the theoretical claims made in the text. The supplementary work provides a great deal of detail on the needed calculations needed to understand the method.\n- The approach is clearly motivated and well-explained, with a strong connection to prior work. \n- The implementation of the method is straightforward and could be integrated into existing NP frameworks. \n- The proposed approach provides strong empirical results across multiple benchmarks.\n- The paper is well-structured and written, making it easy to follow.\n\n**Weaknesses:**\n- Although the empirical results are strong, I fear that the initial idea of the paper might be a bit incremental. It is aided by the extensive theoretical derivations provided, but nonetheless, the core idea of using Rényi divergence to fix prior misspecification is not really that novel (similar efforts with similar empirical results have been achieved, e.g. see [1]).\n- There is no clear way to choose the optimal value of  $\\alpha$ for a given problem since it is dataset-dependent. This could be a limitation in practice, requiring cross-validation.", "other_comments_or_suggestions": "- Investigating adaptive $\\alpha$ selection methods could reduce the need for cross-validation.\n- Notation can sometimes be cumbersome (e.g. Eqs 3, 5 and 6). Maybe it would be beneficial to simplify it for readability.\n\n\nEDIT: I increased my initial score after reading the authors' response.", "questions_for_authors": "1. In general, what are the implications inside Bayesian modelling of constructing the prior in a data-dependent fashion? Does this not imply a form of overfitting? Maybe a discussion on this and its relationship with extensions of the Bayesian inference framework as in the (already mentioned in the paper) Generalized Variational Inference.\n2. Following the previous question, what theoretical guarantees can be provided on the inference results for the proposed method? In particular, I wonder about the soundness of the inference process and the quality of the uncertainty estimates.\n3. How does the choice of $\\alpha$ affect uncertainty calibration in practice?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OTSutIvVje", "reviewer": "Reviewer_3i1T", "summary": "The paper presents a novel approach for training neural processes. By replacing the conventional KL divergence with the Renyi divergence, this allows the model to adapt when confronted with a misspecified prior, therefore enabling more robust inference. This paradigm is somewhat analogous to the utilisation of a hierarchical prior. Experiments are conducted on a mix of regression and image tasks, showing improvement in log likelihood performance over existing techniques.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's efforts in helping us refine the details and acknowledge the original literature. We agree that a rigorous analysis and self-sufficient figures would strengthen the soundness of our work. \n\n# Significance tests in Table 1 and 2.\n\nAll the results presented in the paper were reported with error bars (i.e., standard deviations). As the reviewer suggested, we performed two‐sample t-tests between our RNP and the second best method and highlighted the bold results with significant improvements of **p value $<$ 0.05**. We observed that in Table 1 we significantly improved RBF, Matern and Periodic datasets for most of the methods. Here we show the updated Table 2. In Table 2, the ML objective is no longer significantly superior than our RNP objective on the  D\\_train EMNIST dataset.\n\n| Objective | D_train  (Lotka-Volterra) |               | Misspec D_test  (Hare-Lynx) |                | D_train EMNIST  (class 0-10) |           | Misspec D_test  (class 11-46) |               |\n|:---------:|:-------------------------:|:-------------:|:---------------------------:|:--------------:|:----------------------------:|:---------:|:-----------------------------:|:-------------:|\n|           |          context          |     target    |           context           |     target     |            context           |   target  |            context            |     target    |\n|    L_ML   |         3.09±0.22         |   1.98±0.11   |          -0.59±0.47         |   -4.44±0.41   |           1.54±0.05          | 1.56±0.07 |           0.03±0.97           |   -0.20±0.57  |\n|   L_RNP   |         3.32±0.15         | **2.12±0.06** |          -0.17±0.31         | **-3.63±0.09** |           1.52±0.08          | 1.47±0.12 |           0.96±0.18           | **0.70±0.15** |\n\n# Missing details for the Hare Lynx dataset.\n\nWe apologize for the missing details. Table 2 in the paper reported the results for both species as we treat them as a 2-dimensional input system with feature correlations (see sec 5.2). We left out the Hare plots in Fig 2 originally because when we tried to plot both the Lynx and Hare results in a single graph, the uncertainty intervals of two species were heavily overlapped and impeded the visibility. We will add the Hare results using different colors and opacity in the final version. We will also add more descriptions of this dataset in sec 5.2 and supplementary section for better clarification. \n\n# Relationship to hierarchical prior models. \n\nWe thank the reviewer for raising this interesting point. We believe  our approach to misspecification in neural processes based on the Rényi divergence is fundamentally different to hierarchical approaches. By introducing additional latent variables in a hierarchical fashion, one aims to have a more flexible marginal prior model. For example, under some mild conditions, mixture models are known to be able to approximate any continuous density. However, usually, that additional flexibility comes at the cost of more complex inference. Our approach, maintains the original prior but deals with model misspecification through dampening with the $\\alpha$ parameter. As pointed out by Reviewer Fmj6, it is more closely related to robust Bayesian inference methods based on Gibbs/tempered posteriors. We will discuss this in the final version.\n\n# Updating figures, captions, typos and references.\n    \n- **Captions of Fig 3 and Fig 4.** We will add captions for them to be self-sufficient. Fig 3:  cross-validation is used to select the optimal $\\alpha \\in [0, 2]$. Fig 4: We investigated how MC sample sizes and the number of context points affect test log-likelihood.\n\n- **Uncertainty quantification in Fig 3.** We have actually plotted the uncertainty intervals but some of them were occluded by thicker lines of mean values (see Fig 3 (a) TNPD $\\alpha = 1.5$ for better visibility). We will change the color and the thickness of the intervals to make them more evident.  \n\n- **$\\alpha$ range and dynamic plot range in Fig 3**. Fig 3 showed that $\\alpha >1$ impedes NP training due to an overestimate of the posterior variance (please check the comment for Reviewer aNVZ ``The effect of $\\alpha$ values on training'' for more details). Therefore, we recommend tuning $\\alpha \\in (0, 1)$ in the text. As suggested by the reviewer, we will highlight the results using the KL objective which corresponds to $\\alpha = 1$ in the plot and change the plot into a dynamic range.\n\n- **Adding reference**. We will add the original Rényi divergence paper for reference.\n\n- **Correcting typos**. We will fix the typo in the limitation discussion and proof read the paper.", "claims_and_evidence": "The claims made are generally well supported by the empirical evidence", "methods_and_evaluation_criteria": "Yes, a suitable selection of benchmarks and NP architectures are shown.", "theoretical_claims": "No, while some theoretical background is given, the key results here are empirical.", "experimental_designs_or_analyses": "The experiments span a good range of datasets and in most cases are presented alongside suitable uncertainty estimates.\n\nIn Tables 1 and 2, currently only the numerically highest performing log likelihood is set in bold, but in many cases there is no statistically significant difference to the second strongest method. I would recommend applying bold only to results that outperform at a statistically significant level (and explicitly state that chosen level). This is particularly significant in Table 2 where the L_ML is bolded twice when in neither case it is significantly superior.\n\nThe captions are a little sparse, for example Figure 3 simply reads \"Hyperparameter tuning\" and Figure 4 is \"Ablation study\". I'd recommend ensuring that the captions to the tables and figures are self sufficient.\n\nFigure 3 is somewhat lacking in uncertainty quantification, presumably this is showing the outcome for only a single split. It's therefore not clear how much of the functional form is stochastic in nature.", "relation_to_broader_scientific_literature": "This work seeks to build on previous studies in the development of Neural Processes.", "essential_references_not_discussed": "When introducing the Renyi divergence on page 2, it would be appropriate to cite the paper which first proposed it, which was Renyi 1961.", "other_strengths_and_weaknesses": "A novel, well motivated and clearly presented paper \n\nI think the main weakness relates to how some of the experimental results are missing details. It's not clear to me why Figure 2 shows just the Lynx data but not the Hare. Do the results in Table 2 relate to the data from both species or is it also just for the Lynx?", "other_comments_or_suggestions": "Figure 3 seems to illustrate alpha going up to 2.0 but in the text it is mentioned that it is restricted to alpha<1. Perhaps this figure will be more informative if we can highlight (as a horizontal dotted line for example) the vanilla alpha=1 value. I would also recommend maintaining a similar dynamic range on the y axis, between different panels, as this will illustrate eg that the vanilla NP on RBF is much more sensitive to the choice fo alpha.\n\nIt might be of interest to comment on the relation between using Renyi divergence and using a hierarchical prior. In that the latter one would explicitly evaluate a range of priors, while in the Renyi case the alternative forms of the prior are implicit.\n\nAnd a small typo:\n\"The limitations of our framework lie in drawing multiple samples....\" presumably ought to read\n\"A limitation of our framework lies in drawing multiple samples....\"", "questions_for_authors": "No further questions at this stage!", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8IUa0HWSDe", "reviewer": "Reviewer_GkqA", "summary": "This paper identifies an important issue in Neural Processes (NPs) where prior misspecification can appear from the fact that the conditional prior and posterior share parameters, which can lead to  and degrade uncertainty estimates. The authors propose Rényi Neural Processes (RNPs) as a solution, replacing the KL divergence in the NP objective with the Rényi divergence. This modification introduces a tunable hyperparameter $\\alpha$, which allows adjusting the influence of the prior and helps mitigate misspecification.\n\nOne of the key contributions of this work is showing how RNPs can bridge variational inference (VI) and maximum likelihood (ML) objectives. The paper also provides both theoretical support and empirical results. Experiments on datasets like MNIST, SVHN, and CelebA show improved log-likelihoods and more reliable predictions, particularly in cases where the prior is misspecified.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the effectiveness and theoretical rigor of our work. We appreciate their effort in helping us improve the efficiency and practicality of the work.\n\n# Computational costs. \n\nAs shown in Supp Table 7 of the paper, we have already compared the wall clock time between our RNP and the VI objectives and no significant differences were observed. Our computational complexity is linear to the number of MC samples, which is also comparable to the VI objective. We will add more clarifications for efficiency in sec 5.4. \n\n# Hyperparameter tuning of $\\alpha$ and adaptive $\\alpha$ tuning. \n    \nSince cross-validation can be computationally expensive, we have suggested the heuristics in sec 5.3 to gradually anneal $\\alpha$ from 1 to 0 without hyperparameter tuning. We have already shown in Supp Table 5 that our RNP consistently outperformed competing approaches. We will elaborate further on this heuristic in sec 5.3.\n\n# Scalability on larger datasets or real-time applications.\n\nWe would like to clarify that NPs are generally scalable to large datasets due to minibatch training and fast inference of the framework, and we have tested all our NP models on a sizeable image dataset CelebA. To further improve our efficiency, we suggest to adopt variance reduction methods such as control variates and Rao-Blackwellization [1] that could require fewer MC samples.  Additionally, for attention-based NPs, We suggest to use efficient attention mechanisms, e.g., Nyströmformer [2] which uses low rank approximation of the attention matrix. We will leave the efficiency improvement for our future work. \n\n[1] Ranganath, R. et al, 2014, Black box variational inference. In AISTAT. PMLR.    \n[2] Xiong, Y. et al, 2021, Nyströmformer: A nyström-based algorithm for approximating self-attention. In AAAI.\n\n# $f$-divergence results. \nWe added the comparing results using $f$-variational bound in [3]. More specifically, by specifying a convex function $f$ and its dual $f^*$, $f$-divergence connects several divergences including KL and Rényi divergences. Based on eq (8) in [3], we have the objective \n $L_f(\\phi, \\theta) = \\mathbb{E}_{q(z; \\phi)}[f^*(\\frac{p(z, Y_T|X_T, C; \\theta)}{q(z; \\phi)})] \\geq f^* (p(Y_T|X_T, C))$.\n\nWe chose the posterior $q(z;\\phi)= q(z|C, T; \\phi)$ like NPs do and compared our RNP objective with two functions.\n    $\\chi^2$ divergence [3]:  $f(u)= \\frac{1}{u} - u,   f^*(t)=t^2 -1$. and Jeffery divergence [4]:  $f(u) = (u-1)\\log u,    f^*(t)=(t-1)\\log t$. \n\n| Method |   Set   | Objective |      RBF      |   Matern 5/2  |  Periodic  |\n|:------:|:-------:|:---------:|:-------------:|:-------------:|:----------:|\n|   NP   | context | $\\chi^2$  | 0.64±0.01     | 0.52±0.03     | -0.49±0.01 |\n|        |         | Jeffery   | -0.00±0.00    | -0.01±0.01    | -0.59±0.01 |\n|        |         | RNP       | **0.78±0.01** | **0.66±0.01** | -0.49±0.00 |\n|        |  target | $\\chi^2$  | 0.21±0.02     | 0.07±0.01     | -0.68±0.00 |\n|        |         | Jeffery   | -0.23±0.00    | -0.27±0.00    | -0.61±0.00 |\n|        |         | RNP       | **0.33±0.01** | **0.16±0.01** | -0.62±0.00 |\n\nThe results provide additional support that the RNP objective improves the baseline models. We also tested f-divergence minimization approaches based on the bound of Nguyen et al [5] such as that proposed in [4]. However, as with many GAN-type objectives, they led to unstable training. Finally, unlike our approach, it is unclear how f-divergence minimization methods proposed in the literature can be extended to  improve robustness of maximum likelihood-based NPs. \n\n[3] Wan, N. et al, 2020, F-divergence variational inference. NeurIPS.\n\n[4] Nowozin, S. et al, 2016, f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS.\n\n[5] Nguyen, X. et al, 2010, Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory.\n\n# Intuition or visualizations for RNP posterior estimation.\n  \nIn Fig 1 (a) we have visualized the two posterior distributions obtained by our RNP and VI objectives. The VI objective produced an overestimate of the posterior variance, indicating a strong regularization from the prior model. As a result, it fitted the periodic data with a large variance and over-smoothed mean functions as shown in Fig 1 (b), whereas RNP dampens the prior regularization and obtained a posterior with much smaller variance and therefore encourages the likelihood model to be more expressive as illustrated in Fig 1 (c).", "claims_and_evidence": "The main claims in the paper are:\n\n1. Prior misspecification is an issue in NPs, and appears because of the parameter-sharing assumption.\n2. Using Rényi divergence instead of KL divergence mitigates this issue by allowing more flexibility in how the prior affects the posterior.\n3. The proposed approach unifies VI and ML objectives, offering a more general training framework for NPs.\n4. RNPs consistently outperform standard NPs across a variety of tasks and settings.\n\nThese claims are well-supported both theoretically and experimentally. The paper presents detailed derivations and proofs, showing how the Rényi divergence modifies the posterior updates.", "methods_and_evaluation_criteria": "The authors test RNPs on:\n1. 1D regression using different kernels.\n2. Image inpainting tasks on MNIST, SVHN, and CelebA.\n3. Ablation study with prior misspecification.\n\nThe experiments also include ablation studies to analyze the impact of Monte Carlo sampling, the number of context points, and the choice of $\\alpha$. Overall, the methodology is well thought out and effectively demonstrates the strengths of RNPs.", "theoretical_claims": "The paper demonstrates that replacing KL divergence with Rényi divergence allows for more robust posterior updates while still maintaining a connection to standard NP objectives. One of the most interesting theoretical insights is that RNPs naturally interpolate between VI and ML objectives. By adjusting $\\alpha$, the model can shift between behavior similar to KL-based variational inference ($\\alpha \\approx 1$) and maximum likelihood estimation ($\\alpha \\approx 0$).", "experimental_designs_or_analyses": "The experiments are well designed and include a good mix of standard benchmarks. The main findings are:\n\n1. RNPs achieve better log-likelihoods than baseline NPs.\n2. RNPs improve performance in the presence of prior misspecification, particularly in cases with noisy or distribution-shifted contexts.\n3. Tuning $\\alpha$ appropriately leads to noticeable gains, but default values (e.g., $\\alpha = 0.7$ for VI, $\\alpha = 0.3$ for ML) seem to work well across tasks.\n\nOne strength of the experimental setup is the inclusion of ablation studies that analyze different aspects of RNPs, including Monte Carlo sampling and the number of context points.", "relation_to_broader_scientific_literature": "This work builds on the Neural Processes (NP) family and connects it with ideas from robust divergence measures in variational inference. It references key prior work, including:\n\n1. The original NP papers by Garnelo et al.\n2. Extensions like Attentive NPs (ANP) and Transformer NPs (TNP)\n3. Research on robust divergences like Rényi divergence and $\\alpha$-divergence\n\nThe contribution is well-motivated, but the paper could benefit from a deeper discussion of alternative robust divergences (e.g., $\\alpha$-divergence, $f$-divergence) and why Rényi divergence is particularly well suited for NPs.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n1. Identifies a misspedification problem in standard NPs and provides a simple but effective solution.\n2. Theoretical contributions are rigorous and clearly explained.\n3. Extensive experiments on multiple datasets and settings.\n4. Ablation studies provide useful insights into the behavior of the method.\n\nWeaknesses:\n1. Computational cost. Monte Carlo sampling adds overhead, which could be an issue for large-scale applications.\n2. Hyperparameter tuning. Tuning $\\alpha$ is important, and the method might require careful tuning in some cases.", "other_comments_or_suggestions": "Consider adding more intuition or visualizations to help explain why Rényi divergence improves posterior estimation. Also, a discussion on scalability would be useful, can this method be made more efficient for larger datasets?", "questions_for_authors": "1. Did you try other robust divergence, for example, $f$-divergence? If so, how did they compare?\n2. Would RNPs work on larger datasets or real-time applications? Are there any known limitations?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that this paper addresses an important topic and is well motivated. There is a consensus that the claims are both valuable and well supported by extensive experimentation. 
The authors addressed all the questions and issues raised by the reviewers in the rebuttal.

From the reviews it seems that the paper can make an important contribution to the community by:
1. Identifying gaps in current practice of NPs and provide a proposed solution.
2. The theory of the proposed method is well explained.
3. The method can be used in practice by integrating it into existing NP frameworks.

Some reviewers raised the concern that the method itself (using Rényi divergence to fix prior misspecification) is not novel but rather its integration within the NP framework.
Decision: Accept (oral)