Rényi Neural Processes

## Abstract

Abstract Neural Processes (NPs) are deep probabilistic models that represent stochastic processes by conditioning their prior distributions on a set of context points. Despite their obvious advantages in uncertainty estimation for complex distributions, NPs enforce parameterization coupling between the conditional prior model and the posterior model, thereby risking introducing a misspecified prior distribution. We hereby revisit the NP objectives and propose Rényi Neural Processes (RNP) to ameliorate the impacts of prior misspecification by optimizing an alternative posterior that achieves better marginal likelihood. More specifically, by replacing the standard KL divergence with the Rényi divergence between the model posterior and the true posterior, we scale the density ratio p q 𝑝 𝑞 \frac{p}{q} divide start_ARG italic_p end_ARG start_ARG italic_q end_ARG by the power of (1- α 𝛼 \alpha italic_α ) in the divergence gradients with respect to the posterior. This hyper parameter α 𝛼 \alpha italic_α allows us to dampen the effects of the misspecified prior for the posterior update, which has been shown to effectively avoid oversmoothed predictions and improve the expressiveness of the posterior model.
Our extensive experiments show consistent log-likelihood improvements over state-of-the-art NP family models which adopt both the variational inference or maximum likelihood estimation objectives. We validate the effectiveness of our approach across multiple benchmarks including regression and image inpainting tasks, and show significant performance improvements of RNPs in real-world regression problems where the underlying prior model is misspecifed.

## 1 Introduction

Neural processes (NPs) (Garnelo et al., ) strive to represent stochastic processes via deep neural networks with desirable properties in uncertainty estimation and flexible feature representation. The vanilla NP (Garnelo et al., ) predicts the distribution for unlabelled data given any set of observational data as context. The main advance of NPs is to learn a set-dependent prior distribution, where the KL divergence is minimized between a posterior distribution conditioned on a target set with new data and the prior distribution conditioned on the context set (Kim et al., ; Jha et al., ; Bruinsma et al., ). However, as the parameters of such conditional prior are unknown, NP proposes a coupling scheme where the prior model that parameterizes the distribution is forced to share its parameters with an approximate posterior model. As the model parameter space induced by neural networks is usually large, enforcing coupling on such a space could lead to prior misspecification which could consequently produce a biased estimate of the posterior variance and deteriorate predictive performance (Cannon et al., ; Knoblauch et al., ). Such misspecification can be worsened under noisy context set (Jung et al., ; Liu et al., ). Other cases of prior misspecification encompass domain shifts (Xiao et al., ), out-of-distribution predictions (Malinin & Gales, ) and adversarial samples  (Stutz et al., ).

To address the prior misspecification caused by parameterization coupling in vanilla NPs, several studies have been proposed to relax the constraint (Wang et al., ; Wang & Van Hoof, ; Wu et al., ; Wicker et al., ). For instance, the prior and the posterior models can share partial parameters instead of the entire network (Rybkin et al., ; Liu et al., ); hierarchical latent variable models are also utilized where both models share the same global latent variable and induce prior or posterior-specific distribution parameterization (Shen et al., ; Requeima et al., ; Kim et al., ; Lin et al., ). In this paper, we offer a new insight of handling prior misspecification in NPs through the lens of robust divergence (Futami et al., ), which seeks to learn an alternative posterior without changing the parameters of interests. Instead of minimizing the standard KL divergence between the prior and posterior distributions, robust divergences are theoretically guaranteed to produce better posterior estimates under prior misspecification (Verine et al., ; Regli & Silva, ).
The Rényi divergence (Li & Turner, ; Van Erven & Harremos, ), for instance, introduces an additional parameter $\alpha$ to control how the prior distribution can regularize the posterior during the posterior updates. By scaling the density ratio $\frac{p}{q}$ by the power of (1-$\alpha$) in the divergence gradients with respect to the posterior, this hyper parameter $\alpha$ allows us to reduce the regularization effects of the misspecified prior (Knoblauch et al., ), thereby mitigating the biased estimates of the poster variance, avoiding oversmoothed predictions (Alemi et al., ; Higgins et al., ), and achieving performance improvements. Due to the constraint of parameterization coupling in NPs, such settings can give rise to a potentially critical case of prior misspecification and the need for more robust divergences in NP learning.

In light of this, we propose Rényi Neural Processes (RNPs) that focus on improving neural processes with a more robust objective. RNP minimizes the Rényi divergence between the posterior distribution defined on the target set and the true posterior distribution given the context and target sets. We prove that RNP connects the common variational inference and maximum likelihood estimation objectives for training vanilla NPs via the hyperparameter $\alpha$, through which RNP provides the flexibility to dampen the effect of the misspecified prior and empower the posterior model for better expressiveness. Our main contributions are summarized as:

1. We introduce a new objective for neural processes that unifies the variational inference and maximum likelihood estimation objectives via the Rényi divergence. With the optimal $\alpha$, we can achieve the better the marginal likelihood than both objectives by restricting the impact of prior misspecification and facilitating better posterior expressiveness.

2. We show that Rényi neural processes can be applied to several state-of-the-art neural process family models in a simple yet effective manner without changing the model and provide consistent log-likelihood improvements.

3. We validate the effectiveness of Rényi neural processes on extensive experiments including regression, image inpainting, and real-world regression problems where the underlying prior model is misspecified.

Figure: Figure 1: Graphical Model.
Refer to caption: x1.png

## 2 Preliminaries

Neural Processes:
Neural processes are a family of deep probabilistic models that represent stochastic processes (Wang & Van Hoof, ; Lee et al., ). Let $f_{\tau}:{\mathbb{X}}\rightarrow{\mathbb{Y}}$ be a function sampled from a stochastic process $p(f)$ where each $f_{\tau}$ maps some input features $\mathbf{x}$ to an output $\mathbf{y}$ and ${\mathbb{D}}_{\text{train}}$ and ${\mathbb{D}}_{\text{test}}$ are meta-tasks induced by different $f_{\text{train}}$ and $f_{\text{test}}$ during meta-training and meta-testing. For a specific task ${\mathbb{D}}_{\tau}$, we split the data further into a context set ${\mathbb{C}}:(X_{C},Y_{C}):=\{(\mathbf{x}_{m},\mathbf{y}_{m})_{m=1}^{M}\}$ and a target set ${\mathbb{T}}:(X_{T},Y_{T}):=\{(\mathbf{x}_{n},\mathbf{y}_{n})_{n=1}^{N}\}={
\mathbb{D}}_{\tau}\backslash{\mathbb{C}}$. Our goal is to predict the target labels given the target inputs and the observable context set: $p(Y_{T}|X_{T},X_{C},Y_{C})$. NPs (Garnelo et al., ) introduce a latent variable $\mathbf{z}$ to parameterize the conditional distribution $p(f|{\mathbb{C}})$ and define the model (see Fig 1) as $p(Y_{T}|X_{T},X_{C},Y_{C})=\int p_{\theta}(Y_{T}|X_{T},\mathbf{z})p_{\varphi}(
\mathbf{z}|X_{C},Y_{C})d\mathbf{z}$ where $\theta$ and $\varphi$ are network parameters of the likelihood and prior (also known as recognition) models, respectively. Due to the intractable likelihood, two types of objectives including the variational inference (VI) and maximum likelihood (ML) estimation have been proposed to optimize the parameters (Foong et al., ; Nguyen & Grover, ; Bruinsma et al., ; Guo et al., ):

$$ $\displaystyle-\mathcal{L}_{VI}(\theta,\phi,\varphi)=\mathbb{E}_{{\mathbb{D}}_{ \text{train}}}\left[\mathbb{E}_{q_{\phi}(\mathbf{z})}\log p_{\theta}(Y_{T}|X_{ T},\mathbf{z})-D_{\mathrm{KL}}\left(q_{\phi}(\mathbf{z})\|p_{\varphi}(\mathbf{ z}|X_{C},Y_{C})\right)\right]$ (1) $\displaystyle-\mathcal{L}_{ML}(\theta,\varphi)=\mathbb{E}_{{\mathbb{D}}_{\text {train}}}\left[\mathbb{E}_{p_{\varphi}(\mathbf{z}|X_{C},Y_{C})}\log p_{\theta} (Y_{T}|X_{T},\mathbf{z})\right]$ (2) $$

The approximate posterior distribution for VI-based method is usually chosen as $q_{\phi}(\mathbf{z})=q_{\phi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})$. As the parameters of the conditional prior $p_{\varphi}(\mathbf{z}|X_{C},Y_{C})$ are unknown, NPs couple its parameters with the approximate posterior
$p_{\varphi}(\mathbf{z}|X_{C},Y_{C})\approx q_{\phi}(\mathbf{z}|X_{C},Y_{C})$. We now replace the notation of the approximate posterior with $\varphi$ for ML consistency:

$$ $-\mathcal{L}_{VI}(\theta,\varphi)\approx\mathbb{E}_{{\mathbb{D}}_{\text{train} }}\left[\mathbb{E}_{q_{\varphi}(\mathbf{z})}\log p_{\theta}(Y_{T}|X_{T}, \mathbf{z})-D_{\mathrm{KL}}\left(q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C })\|q_{\varphi}(\mathbf{z}|X_{C},Y_{C})\right)\right]$ (3) $$

The KL term in Eq 3 is sometimes referred to as the consistency regularizer (Wang et al., ; Foong et al., ), which encourages the target set to be subsumed into the context set. This assumption, as will show later, is the source of the inference suboptimality of vanilla NPs in the existence of finite capacity/data.

Rényi Divergences:
The Rényi divergence (RD) (Van Erven & Harremos, ) is defined on two distributions with a hyperparamter $\alpha\in(0,+\infty)$ and $\alpha\neq 1$:

$$ $D_{\alpha}(q(\mathbf{z})\|p(\mathbf{z}))=\frac{1}{\alpha-1}\log\int q(\mathbf{ z})^{\alpha}p(\mathbf{z})^{1-\alpha}d\mathbf{z}=\frac{1}{\alpha-1}\log\mathbb{ E}_{q(\mathbf{z})}\left[\frac{p(\mathbf{z})}{q(\mathbf{z})}\right]^{1-\alpha} \text{.}$ (4) $$

Note that the RD is closely related to the KL divergence in that if $\alpha\rightarrow 1$ then $D_{\alpha}(q\|p)\rightarrow D_{\mathrm{KL}}(q||p)$ (Van Erven & Harremos, ). In other words, choosing $\alpha$ close to 1 would result in a posterior as close to the standard VIs. Changing the KL divergence to RD can induce a robust posterior via the hyperparameter $\alpha$. To see this, consider the gradients of the RD wrt the posterior parameters:

$$ $\frac{\partial}{\partial\varphi}\mathcal{D}_{\alpha}\left(q_{\varphi}(\mathbf{ z})\|p(\mathbf{z})\right)=\frac{\alpha}{\alpha-1}\frac{\int{\left[\frac{p( \mathbf{z})}{q_{\varphi}(\mathbf{z})}\right]}^{1-\alpha}\frac{\partial q_{ \varphi}(\mathbf{z})}{\partial\varphi}d\mathbf{z}}{\mathbb{E}_{q_{\varphi}( \mathbf{z})}\left[\frac{p(\mathbf{z})}{q_{\varphi}(\mathbf{z})}\right]^{1- \alpha}}$ (5) $$

where the influence of the density ratio $\frac{p}{q}$ on the gradient is scaled by the power of ${(1-\alpha)}$ as opposed to the unscaled ratio $\frac{p}{q}$ of the standard KL divergence (Regli & Silva, ). This ratio determines how much we can penalize the posterior with the prior. With the flexibility of choosing $\alpha$, the model can reduce the prior penalty less than significantly than the standard KL. Note that when $\alpha\in(0,1)$, $(\frac{p}{q})^{1-\alpha}<\frac{p}{q}$ when $\frac{p}{q}>1$ and $(\frac{p}{q})^{1-\alpha}>\frac{p}{q}$ when $\frac{p}{q}<1$, which means less penalty will be applied to the overestimate region of the prior where $\frac{p}{q}>1$. When the prior is misspecified, choosing the RD can lead to a more robust posterior that focuses more on improving the likelihood and less on reducing the divergence (Futami et al., ; Regli & Silva, ).

## 3 Rényi Neural Processes

In this section we describe our Rényi Neural Process (RNP) framework, a simple yet effective strategy which provides a more robust way to learn neural processes without changing the model.
We start by analyzing the main limitation of the standard neural process objective and present a motivating example. We illustrate a case where the prior is misspecified and describe our new objective with the RD to mitigate this.

### 3.1 Motivation: Standard neural processes and prior misspecification

We first consider a formal definition of prior misspecification.

###### Definition 3.1 .

(Prior misspecification (Huang et al., ))
Let $\eta_{\#}\mathbb{Q_{\varphi}}$ be a pushforward of a probability measure $\mathbb{Q}_{\varphi}$ parameterized by $\varphi$ under the map $\eta$: $\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{Z}$. Then, {$\eta_{\#}\mathbb{Q_{\varphi}:\varphi\in\Upphi}$} defines a set of distributions on the space $\mathcal{Z}$ induced by the model, and $\eta_{\#}\mathbb{P}$ is the pushforward of the ground truth measure ${\mathbb{P}}$. The prior model is misspecified if $\forall\varphi\in\Upphi,\eta_{\#}\mathbb{Q}_{\varphi}\neq\eta_{\#}\mathbb{P}$

The definition suggests that when the prior model $\eta_{\#}\mathbb{Q}_{\varphi}$ is misspecified, there exists no optimal parameter $\varphi^{*}$ that can represent the true prior $\eta_{\#}\mathbb{P}$. We now show how this definition can assist us to analyze how a misspecified prior model can hinder neural processes learning.

###### Proposition 3.2 .

Due to the prior approximation in Eq 3, the prior model is misspecified, meaning that $\forall\varphi\in\Upphi,\eta_{\#}\mathbb{Q}_{\varphi}(X_{C},Y_{C})\neq\eta_{\#
}\mathbb{P}(X_{C},Y_{C})$, and the resulting $\mathcal{L}_{VI}$ is no longer a valid bound for the marginal likelihood $p(Y_{T}|X_{T},X_{C},Y_{C})$.

Detailed proof can be found in Supp A.1. Proposition 3.2 challenges the common assumption $p(\mathbf{z}|X_{C},Y_{C})\approx q(\mathbf{z}|X_{C},Y_{C})$ that most NPs make. Such assumption, as we will show next, may result in a biased estimate of the posterior variance which hinders NP training.

Illustrative example: Fig LABEL:fig:_motivation_example provides an example of a misspecified prior for neural processes. The objective of this example is to compare how the vanilla NP and our proposed RNP behave when the prior is misspecified.
We delay the introduction of the formulation of $\mathcal{L}_{RNP}$ in the next section, which is unnecessary for the illustration.

Let ${\mathbb{D}}_{\text{train}}$ and ${\mathbb{D}}_{\text{test}}$ denote datasets (meta-tasks) generated by two Gaussian Processes (GPs) with different kernels. We first train an NP model on ${\mathbb{D}}_{\text{train}}$ using the vanilla NP objective: $\varphi_{NP_{D_{\text{train}}}},\theta^{*}=\min_{\varphi,\theta}\mathcal{L}_{
VI}(\varphi,\theta;{\mathbb{D}}_{\text{train}})$.
With that, we then fix the likelihood parameter $\theta^{*}$ in order to isolate the effect of the likelihood model on the predictive performance from the prior model and obtain another posterior model using the RNP objective: $\varphi_{RNP_{D_{\text{train}}}}=\min_{\varphi}\mathcal{L}_{RNP}(\varphi,
\theta^{*};{\mathbb{D}}_{\text{train}})$.
Now consider evaluating both models using ${\mathbb{D}}_{\text{test}}$ by conditioning on these parameters and the test context set ${\mathbb{C}}_{\text{test}}$, then $q(\mathbf{z}|{\mathbb{C}}_{\text{test}},\varphi_{NP_{D_{\text{train}}}})$ and $q(\mathbf{z}|{\mathbb{C}}_{\text{test}},\varphi_{RNP_{D_{\text{train}}}})$ can be viewed as misspecified prior models since their parameters were optimized on ${\mathbb{D}}_{\text{train}}$ which looks quite different from ${\mathbb{D}}_{\text{test}}$.
The resulting posteriors in Fig LABEL:fig:motivation_posterior show that RNPs obtaine a much smaller variance estimate, suggesting that the consistency regularizer $D_{\mathrm{KL}}\left(q_{\varphi}(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})\|q_{
\varphi}(\mathbf{z}|{\mathbb{C}})\right)$ is too strong in the vanilla NPs that it forces the posterior to produce a higher variance. As a result, vanilla NPs produce oversmoothed predictions that underfit the data as shown in Fig LABEL:fig:ANP_Periodic_ANP_VI.
In contrast, our RNP dampens the impacts of prior misspecification via the RD divergence and our predictive model respect the expressiveness of the posterior model better and produce superior predictive means as well as uncertainty estimate shown in Fig LABEL:fig:ANP_Periodic_ANP_RNP.

Hence, we argue that using RNP is beneficial when existing NP models have only limited capacity/data which could induce prior misspecification. We will now introduce our new neural process learning method, which can be generally applied to NPs using either VI or ML objectives.

###### Definition 3.1 .

###### Proposition 3.2 .

### 3.2 Proposed Method: Neural processes with the new objectives

New objective for VI-based NPs.
The main issue of NPs is the prior approximation $q_{\varphi}(\mathbf{z}|X_{C},Y_{C})$ wrt the true prior $p(\mathbf{z}|X_{C},Y_{C})$. In this case, the posterior variance may be critically overestimated in some regions and underestimated in others. We therefore seek to obtain an alternative posterior distribution to alleviate this prior misspecification. Depending on whether the original NP framework is trained using the VI or the ML objective, we can revise the objective by minimizing the RD instead of KLD on two distributions.
More specifically, in the case of VI-NPs where the inference of the latent variable $q(\mathbf{z})$ is of interest, the RD between the the approximated posterior distribution $q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})$ and the true posterior $p(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})$ is minimized.

$$ $\displaystyle\min_{\theta,\varphi}D_{\alpha}\left(q_{\varphi}(\mathbf{z}|X_{T} ,Y_{T},X_{C},Y_{C})\|p(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\right)$ (6a) $\displaystyle\approx\max_{\theta,\varphi}\frac{1}{1-\alpha}\log\mathbb{E}_{q_{ \varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p_{\theta}(Y_{T}|X_{T },\mathbf{z})q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T} ,Y_{T},X_{C},Y_{C})}\right]^{1-\alpha}$ (6b) $$

where details can be found inA.3. Eq 6b is an approximation obtained by replacing $p(\mathbf{z}|X_{C},Y_{C})$ with $q_{\varphi}(\mathbf{z}|X_{C},Y_{C})$. We can approximate the intractable expectation with Monte Carlo:

$$ $-\mathcal{L}_{RNP}(\theta,\varphi)=\frac{1}{1-\alpha}\mathbb{E}_{{\mathbb{D}}_ {\text{train}}}\left[\log\frac{1}{K}\sum_{k=1}^{K}\left[\frac{p_{\theta}(Y_{T} |X_{T},\mathbf{z}_{k})q_{\varphi}(\mathbf{z}_{k}|X_{C},Y_{C})}{q_{\varphi}( \mathbf{z}_{k}|X_{T},Y_{T},X_{C},Y_{C})}\right]^{1-\alpha}\right],\quad\mathbf {z}_{k}\sim q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\text{.}$ (7) $$

Prior, posterior and likelihood models. One main advantage of RNP is we do not need to change the parameters of interests of the original NP models. Therefore, for the likelihood model $p_{\theta}(Y_{T}|X_{T},\mathbf{z})$ we can adopt simple model architectures like NPs (Garnelo et al., ) which assume independence between target points $p_{\theta}(Y_{T}|X_{T},\mathbf{z})=\prod_{n=1}^{N}p_{\theta}(\mathbf{y}_{n}|
\mathbf{x}_{n},\mathbf{z})$. The distribution of each target point is then modelled as Gaussian $p_{\theta}(\mathbf{y}_{n}|\mathbf{x}_{n},\mathbf{z})=\mathcal{N}\left(h_{\mu}(
\mathbf{x_{n}},\mathbf{z}),\text{Diag}\left(h_{\sigma}(\mathbf{x_{n}},\mathbf{
z})\right)\right)$, and the decoder networks $h_{\mu}$ and $h_{\sigma}$ map the concatenation of the input feature $\mathbf{x}_{n}$ and $\mathbf{z}$ to the distribution parameters.
We can also consider more complex models like ANPs (Kim et al., ) which incorporate dependencies between target points $p_{\theta}(Y_{T}|X_{T},\mathbf{z})=\prod_{n=1}^{N}p_{\theta}(\mathbf{y}_{n}|
\mathbf{x}_{1:n},\mathbf{z})$ using attention networks.

The prior model $q_{\varphi}(\mathbf{z}|X_{C},Y_{C})$ is more interesting as it is a set-conditional distribution and we are supposed to sample from it and evaluate the density of the samples. One feasible solution is to define a parametric distribution on a DeepSet (Zaheer et al., ).
For instance, $p_{\varphi}(\mathbf{z}|X_{C},Y_{C})=\mathcal{N}\left(g_{\mu}(h({\mathbb{C}})),
\text{Diag}(g_{\sigma}(h({\mathbb{C}}))\right)$ where $h({\mathbb{C}})=\frac{1}{|{\mathbb{C}}|}\sum_{m=1}^{|{\mathbb{C}}|}h(\mathbf{x
}_{m},\mathbf{y}_{m})$ is a DeepSet function on the context set ${\mathbb{C}}$. In practice diagonal Gaussian distributions worked well with high dimensional latent variables $\mathbf{z}$. But one can consider more flexible distribution such as conditional normalising flows (Luo et al., ) for sample and density estimation. As previously stated, the posterior distribution is defined by coupling its parameters with the prior. Therefore the posterior $q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})$ in the DeepSet case can be represented as $p_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})=\mathcal{N}\left(g_{\mu}(h({
\mathbb{C}},{\mathbb{T}})),\text{Diag}(g_{\sigma}(h({\mathbb{C}},{\mathbb{T}})
)\right)$.

New objective for ML-based NPs.
As the goal of NPs is to maximize predictive likelihood instead of inferring the latent distribution, another type of NPs directly parameterize the likelihood model without explicitly defining the latent variable $\mathbf{z}$. For instance, TNP (Nguyen & Grover, ) models $\log p(Y_{T}|X_{T},X_{C},Y_{C})=\sum^{N}_{n=1}\log p_{\theta}(\mathbf{y}_{n}|
\mathbf{x}_{1:n},\mathbf{y}_{1:n-1},{\mathbb{C}})$ in an autoregressive manner and each step a transformer encoder network parameterizes the output mean and standard deviations $p_{\theta}(\mathbf{y}_{n}|\mathbf{x}_{1:n},\mathbf{y}_{1:n-1},{\mathbb{C}}):
\mathcal{N}(\mu_{\mathbf{y}_{n}},\Sigma_{\mathbf{y}_{n}})$. We can rewrite it as $p_{\theta}(\mathbf{y}_{n}|\mathbf{x}_{1:n},\mathbf{y}_{1:n-1},{\mathbb{C}})=
\int p(\mathbf{y}_{n}|\mathbf{z}_{n})p(\mathbf{z}_{n}|\mathbf{x}_{1:n},\mathbf
{y}_{1:n-1},{\mathbb{C}})d\mathbf{z}_{n}$ where $p(\mathbf{z}_{n}|\mathbf{x}_{1:n},\mathbf{y}_{1:n-1},{\mathbb{C}})=\delta(
\mathbf{x}_{1:n},\mathbf{y}_{1:n-1},{\mathbb{C}})$ implicitly defines a $\delta$ distribution using the transformer.

Following Futami et al. (), we can rewrite the maximum likelihood estimation as minimizing the KLD between the empirical distribution $\hat{p}(\mathbf{y}_{n}|\mathbf{x}_{n},{\mathbb{C}})$ and the model distribution $p(\mathbf{y}_{n}|\mathbf{x}_{n},{\mathbb{C}},\theta)$:

$$ $\displaystyle-\mathcal{L}_{ML}(\theta)=\max_{\theta,\varphi}\mathbb{E}_{{ \mathbb{D}}_{\text{train}}}\log p_{\theta}(Y_{T}|X_{T},{\mathbb{C}})$ (8) $\displaystyle\equiv\max_{\theta}\mathbb{E}_{{\mathbb{D}}_{\text{train}}}\left[ \frac{1}{N}\sum^{N}_{n=1}\log p_{\theta}(\mathbf{y}_{n}|\mathbf{x}_{n},{ \mathbb{C}})\right]\approx\min_{\theta}\mathbb{E}_{{\mathbb{D}}_{\text{train}} }\left[D_{\mathrm{KL}}(\hat{p}(\mathbf{y}|\mathbf{x},{\mathbb{C}})\|p_{\theta} (\mathbf{y}|\mathbf{x},{\mathbb{C}}))\right]$ (9) $$

where $\hat{p}(\mathbf{y}|\mathbf{x},{\mathbb{C}})$ is the empirical distribution defined as $\frac{1}{N}\sum_{n=1}^{N}\delta(\mathbf{y},\mathbf{y}_{n})$ where $y_{n}$ are samples from the unknown distribution $p^{*}(\mathbf{y}|\mathbf{x},{\mathbb{C}})$. Replacing the KLD with RD we get:

$$ $\displaystyle\min_{\theta}\mathbb{E}_{{\mathbb{D}}_{\text{train}}}D_{\alpha} \left(\hat{p}(\mathbf{y}|\mathbf{x},{\mathbb{C}})||p_{\theta}(\mathbf{y}| \mathbf{x},{\mathbb{C}})\right)\approx\min_{\theta}\frac{1}{N}\sum_{n=1}^{N} \frac{1}{\alpha-1}\log p_{\theta}^{1-\alpha}(\mathbf{y}_{n}|\mathbf{x}_{n},{ \mathbb{C}})+Const$ (10) $\displaystyle\mathcal{L}_{RNPML}(\theta)=\frac{1}{(\alpha-1)N}\sum_{n=1}^{N} \log p_{\theta}^{1-\alpha}(\mathbf{y}_{n}|\mathbf{x}_{n},{\mathbb{C}})=\frac{1 }{(\alpha-1)N}\sum_{n=1}^{N}\log\left(\int p_{\theta}(\mathbf{y}_{n},\mathbf{z }|\mathbf{x}_{n},{\mathbb{C}})d\mathbf{z}\right)^{1-\alpha}$ (11) $$

Note $\alpha=0$ corresponds to maximizing likelihood estimation and the new RNP objective essentially reweights samples based on their likelihood. We define $p_{\theta}(\mathbf{y}_{n},\mathbf{z}|\mathbf{x}_{n},{\mathbb{C}})=p_{\theta}(
\mathbf{y}_{n},\mathbf{z}|\mathbf{x}_{n})p(\mathbf{z}|{\mathbb{C}})$ and use $p_{\varphi}(\mathbf{z}|{\mathbb{C}})\approx p(\mathbf{z}|{\mathbb{C}})$ . Then Eq 10 can be approximated with Monte Carlo:

$$ $\displaystyle\mathcal{L}_{RNPML}(\theta,\varphi)\approx\frac{1}{(\alpha-1)N} \sum_{n=1}^{N}\log\left(\frac{1}{K}\sum_{k=1}^{K}p_{\theta}(\mathbf{y}_{n}| \mathbf{z}_{k},\mathbf{x}_{n})d\mathbf{z}\right)^{1-\alpha},\quad\mathbf{z}_{k }\sim p_{\varphi}(\mathbf{z}|{\mathbb{C}})$ (12) $$

One advantage of ML-based method is that we do not need to estimate the density of the samples from the prior model. Therefore, reparameterization tricks (Kingma et al., ) can be applied to obtain samples from flexible prior models: $\mathbf{z}_{k}=s_{\varphi}({\mathbb{C}},\epsilon),\epsilon\sim\mathcal{N}(
\mathbf{0},I)$ where $s_{\varphi}$ is a neural network.

## 4 Properties of RNPs

We generalize the theorem from Rényi variational inference  (Li & Turner, ) to RNPs:

###### Theorem 4.1 .

(Monotonicity (Li & Turner, ))

$\mathcal{L}_{RNP}$ is continuous and non-increasing with respect to the hyper-parameter $\alpha$.

###### Proposition 4.2 .

(Unification of the objectives)

$\mathcal{L}_{ML}=\mathcal{L}_{RNP,\alpha=0}\geq\mathcal{L}_{RNP,\alpha\in(0,1)
}\geq\mathcal{L}_{RNP,\alpha\rightarrow 1}=\mathcal{L}_{VI}$.

Theorem 4.1 and Proposition 4.2 (Proof see Supp A.4) suggest that these objectives are bounded by $\log\int p(Y_{T}|X_{T},\mathbf{z})q(\mathbf{z}|X_{C},Y_{C})$. However, it is not the marginal likelihood due to the approximation $q(\mathbf{z}|X_{C},Y_{C})\approx p(\mathbf{z}|X_{C},Y_{C})$, and the training of NPs using these objectives does not guarantee the validity of the approximation. Therefore, the optimal $\alpha$ that achieves the best likelihood might not always correspond to 0, i.e., the maximum likelihood objective does not always guarantees the best marginal likelihood, and we can still improve the test likelihood using our RNP objective. We show in our experiment that benefits can be readily obtained by tuning $\alpha$ on a validation set.

Tuning $\alpha\in(0,1)$ can mitigate prior misspecification in RNPs. We have shown that prior misspecification can be handled by improving the mass-covering ability in NPs. We will present how RNP can achieve this objective through the gradients of the parameters of the encoder networks $\varphi$ (more details can be found in Supp A.5):

$$ $\displaystyle\nabla\varphi\mathcal{L}_{RNP}=\nabla\varphi\left(\frac{1}{\alpha -1}\log\mathbb{E}_{q_{\varphi}(\mathbf{z}|{\mathbb{C}},{\mathbb{T}})}[\frac{p( Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|{\mathbb{C}})}{q_{\varphi}( \mathbf{z}|{\mathbb{C}},{\mathbb{T}})}]^{1-\alpha}\right)$ (13a) $\displaystyle=-\frac{1}{1-\alpha}\frac{A}{\int p(Y_{T}|X_{T},\mathbf{z})^{1- \alpha}q_{\varphi}(\mathbf{z}|{\mathbb{C}})^{1-\alpha}q_{\varphi}(\mathbf{z}|{ \mathbb{C}},{\mathbb{T}})^{\alpha}d\mathbf{z}}\quad,$ $\displaystyle A=\int p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}\left[\alpha\left( \frac{q_{\varphi}(\mathbf{z}|{\mathbb{C}})}{q_{\varphi}(\mathbf{z}|{\mathbb{C} },{\mathbb{T}})}\right)^{1-\alpha}\nabla_{\varphi}q_{\varphi}(\mathbf{z}|{ \mathbb{C}},{\mathbb{T}})+(1-\alpha)\left(\frac{q_{\varphi}(\mathbf{z}|{ \mathbb{C}},{\mathbb{T}})}{q_{\varphi}(\mathbf{z}|{\mathbb{C}})}\right)^{ \alpha}\nabla_{\varphi}q_{\varphi}(\mathbf{z}|{\mathbb{C}})\right]d\mathbf{ \mathbf{z}}$ (13b) $$

The influence of the density ratio $\frac{q(\mathbf{z}|{\mathbb{C}})}{q(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})}$ and $\frac{q(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})}{q(\mathbf{z}|{\mathbb{C}})}$ on the gradients is now scaled by $[\frac{q(\mathbf{z}|{\mathbb{C}})}{q(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})}]^{
1-\alpha}$ and $[\frac{q(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})}{q(\mathbf{z}|{\mathbb{C}})}]^{\alpha}$ respectively. Tuning $\alpha\in(0,1)$ offers the flexibility of how much we intend to penalize the gradients on the different regions of the posterior $q(\mathbf{z}|{\mathbb{T}},{\mathbb{C}})$ and $q(\mathbf{z}|{\mathbb{C}})$. Additionally, the gradient is scaled by $p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}$, and a greater gradient descent can be achieved on a lower likelihood.

###### Theorem 4.1 .

###### Proposition 4.2 .

## 5 Inference with Rényi Neural Processes

During inference time, as we cannot access the ground truth for the target outputs $Y_{T}$, we use the approximate prior $q(\mathbf{z}|X_{C},Y_{C})$ instead of the posterior distribution $q(\mathbf{z}|X_{T},Y_{T})$ to estimate the marginal distribution:

$$ $p(Y_{T}|X_{T},X_{C},Y_{C})=\int p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}( \mathbf{z}|X_{C},Y_{C})d\mathbf{z}\approx\frac{1}{K}\sum_{k=1}^{K}p_{\theta}(Y _{T}|X_{T},\mathbf{z}_{k}),\mathbf{z}_{k}\sim q_{\varphi}(\mathbf{z}|X_{C},Y_{ C}).$ (14) $$

Figure: Algorithm 1 Rényi Neural Processes

We now provide the pseudo code for Rényi Neural Processes in Algorithm 1. In addition to vanilla neural processes, our framework can also be generalized to other neural process variants as shown in the experiments section.

## 6 Related Work

Neural processes family. Neural processes (Garnelo et al., ) and conditional neural processes (Garnelo et al., ) were initially proposed for the meta learning scheme where they make predictions given a few observations as context. Both of them use deepset models (Zaheer et al., ) to map a finite number of data points to a high dimensional vector and their likelihood models assume independencies among data points. The main difference is whether estimating likelihood maximization directly or introducing the latent variable and adopting variational inference framework. Under the existing NP setting, more members were introduced with different inductive biases in the model (Jha et al., ; Bruinsma et al., ; Dutordoir et al., ; Jung et al., ; Vadeboncoeur et al., ). For instance, attentive neural processes (Kim et al., ) incorporated dependencies between observations with attention neural works. Convolutional neural processes (Foong et al., ; Huang et al., ) assume translation equivariance among data points. These two methods explicitly defined the latent variable which requires density estimation. Recent works such as transformer neural processes (Maraval et al., ) and neural diffusion processes (Dutordoir et al., ) turn to marginal likelihood maximization and do not have the latent distribution. More neural processes which claim to provide exact (Markou et al., ) or tractable inference (Lee et al., ; Wang et al., ) are introduced. Stable neural processes (Liu et al., ) argued that NPs are prone to noisy context points and proposed a weighted likelihood model that focuses on subsets that are difficult to predict, but do not focus regularizing the posterior distribution. Compared to variational inference based methods, non-VI predictions can be less robust to noisy inputs in the data (Futami et al., ). It is also challenging to incorporate prior knowledge (Zhang et al., ) into these neural processes, which could be beneficial when no data is observed for the task.

Robust divergences. Divergences in variational inference can be viewed as an regularization on the posterior distribution via the prior distribution. The commonly adopted KL divergence which minimizes the expected density ratio between the posterior is infamous for underestimating the true variance of the target distribution (Regli & Silva, ). Several other divergences have since been proposed to focus on obtaining a robust posterior when the input and output features are noisy or the existence of outliers. Examples of robust divergences include Rényi divergence (Lee & Shin, ), beta and alpha divergences (Futami et al., ; Regli & Silva, ) and f-divergences (Cheng et al., ). They provide the extra parameters or flexible functions to control the density ratio so that the posterior can focus more on mass covering, mode seeking abilities or is robust against outliers. Generalized variational inference (Knoblauch et al., ) suggested that any form of divergence can be used to replace the KL objective when the model is misspecified. As neural processes facilitate a posterior to approximate the prior distribution, we are able to control the density ratios of two posterior distributions with robust divergences for better predictions.

## 7 Experiments

Datasets and training details.
We evaluate the proposed method on three tasks: 1D regression (Garnelo et al., ; Gordon et al., ; Kim et al., ; Nguyen & Grover, ), image inpainting (Gordon et al., ; Nguyen & Grover, ) and tabular regression (Pace & Barry, ). 1D regression includes three Gaussian Process (GP) regression tasks with different kernels: RBF, Matern 5/2 and Periodic. Given a function $f$ sampled from a GP prior with varying scale and length and a context set generated by such function, our goal is to predict the target distribution. The number of context points is randomly sampled $M\sim\mathcal{U}(3,50)$, and the number of target points is $N\sim\mathcal{U}(3,50-M)$ (Nguyen & Grover, ). We choose 100,000 functions for training, and sample another 3,000 functions for testing. The input features were normalized to $[-2,2]$.
Image inpainting involves 2D regression on three image datasets: MNIST, SVHN and CelebA. Given some pixel coordinates $\mathbf{x}$ and intensities $\mathbf{y}$ as context, the goal is to predict the pixel value for the rest of image. The number of context points for inpainting tasks is $M\sim\mathcal{U}(3,200)$ and the target point count is $N\sim\mathcal{U}(3,200-M)$. The input coordinates were normalized to $[-1,1]$ and pixel intensities were rescaled to $[-0.5,0.5]$. All the models can be trained using a single GPU with 16GB memory.

Baselines. We first validate our approach on state of the art NP families: neural processes (NP) (Garnelo et al., ), attentive neural processes (ANP) (Kim et al., ), Bayesian aggregation neural processes (BA-NP) (Volpp et al., ), transformer neural processes with diagonal covariances (TNP-D) (Nguyen & Grover, ), and versatile neural processes (VNP) (Guo et al., ). We generalize the NP objective to RNP using Eq 12 or Eq 7 depending on the baseline. The methods are considered as a special case of $\alpha=1$ of RNP if the baseline model uses the VI objective or $\alpha=0$ if the baseline model uses the ML-objective. The number of samples $K$ for the Monte Carlo is 32 for training and 50 for inference. Our experiments aim to answer the following research questions: (1) Can RNPs achieve better performance over existing NP frameworks?
(2) How does the model perform under prior misspecifications? (3) How to select the optimal $\alpha$ values? We also carry out ablation studies investigating how to select the optimal $\alpha$ values, the number of MC samples and the number of context points for our RNP framework.

### 7.1 Predictive performance

We compare the test log-likelihood on both the context and target sets across different datasets. We set $\alpha=0.7$ for VI-based baselines,
as $\alpha=1$ corresponds to the standard VI solution (using the KLD).
Analogously, we set $\alpha=0.3$ for ML-based baselines,
since $\alpha=0$ corresponds to the maximum likelihood solution. However, we show in section 7.3 that, in fact, these values can be set optimally via cross-validation.

Table 1 shows the mean test log-likelihood $\pm$ one standard deviation using 5 different random seeds for each method. We see that RNP consistently improved log-likelihood over the other two objectives and ranked the highest for all the baselines. RNP also consistently achieved better likelihood on TNP-D and VNP which generally outperform other baseline models across datasets. Some prominent improvements were achieved in harder tasks in 1D regression such as ANP Periodic and BA-NP Periodic where the vanilla NP objectives underperform. As previously illustrated in Fig LABEL:fig:ANP_Periodic_ANP_VI and Fig LABEL:fig:ANP_Periodic_ANP_RNP, RNP improves predictive performance by mitigating the oversmoothed predictions on periodic data. This could suggest that a misspecified prior model in the vanilla NP objective imposes an unjustifiable regularization on the posterior and hinder the expressiveness of the posterior and consequently predictive performance. RNP also significantly improved test likelihood of BA-NP and VNP on image inpainting tasks, demonstrating the superiority of RNPs on higher dimensional data.

**Table 1: Test set log-likelihood $\uparrow\uparrow$. The best performance results for each dataset are in bold.**
| Model | Set | Objective | RBF | Matern 5/2 | Periodic | MNIST | SVHN | CelebA | Avg rank |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| NP<br>(Garnelo et al., ) | context | $\mathcal{L}_{VI}$ | 0.69±0.01 | 0.56±0.02 | -0.49±0.01 | 0.99±0.01 | 3.24±0.02 | 1.71±0.04 | 2.3 |
| $\mathcal{L}_{ML}$ | 0.68±0.02 | 0.55±0.02 | -0.48±0.03 | 1.00±0.01 | 3.22±0.03 | 1.70±0.03 | 2.5 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 0.78±0.01 | 0.66±0.01 | -0.49±0.00 | 1.01±0.02 | 3.26±0.01 | 1.72±0.05 | 1.2 |  |  |
| target | $\mathcal{L}_{VI}$ | 0.26±0.01 | 0.09±0.02 | -0.61±0.00 | 0.90±0.01 | 3.08±0.01 | 1.45±0.03 | 2.3 |  |
| $\mathcal{L}_{ML}$ | 0.28±0.02 | 0.11±0.02 | -0.61±0.01 | 0.92±0.01 | 3.07±0.02 | 1.47±0.02 | 1.8 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 0.33±0.01 | 0.16±0.01 | -0.62±0.00 | 0.91±0.01 | 3.09±0.01 | 1.45±0.03 | 1.7 |  |  |
| ANP<br>(Kim et al., ) | context | $\mathcal{L}_{VI}$ | 1.38±0.00 | 1.38±0.00 | 0.65±0.04 | 1.38±0.00 | 4.14±0.00 | 3.92±0.07 | 1.3 |
| $\mathcal{L}_{ML}$ | 1.38±0.00 | 1.38±0.00 | 0.63±0.03 | 1.38±0.00 | 4.14±0.01 | 3.86±0.07 | 1.7 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 1.38±0.00 | 1.38±0.00 | 1.22±0.02 | 1.38±0.00 | 4.14±0.00 | 3.97±0.03 | 1.0 |  |  |
| target | $\mathcal{L}_{VI}$ | 0.81±0.00 | 0.64±0.00 | -0.91±0.02 | 1.06±0.01 | 3.65±0.01 | 2.24±0.03 | 1.7 |  |
| $\mathcal{L}_{ML}$ | 0.80±0.00 | 0.64±0.00 | -0.89±0.02 | 1.04±0.01 | 3.65±0.01 | 2.23±0.03 | 2.2 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 0.84±0.00 | 0.67±0.00 | -0.57±0.01 | 1.05±0.01 | 3.61±0.02 | 2.24±0.02 | 1.3 |  |  |
| BA-NP<br>(Volpp et al., ) | context | $\mathcal{L}_{VI}$ | 1.43±0.03 | 1.04±0.08 | -0.65±0.02 | 0.81±0.84 | 2.76±0.59 | 1.65±0.01 | 2.3 |
| $\mathcal{L}_{ML}$ | 1.30±0.09 | 0.69±0.05 | -0.70±0.07 | 3.62±0.06 | 4.87±0.05 | 2.02±0.02 | 2.3 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 1.45±0.04 | 1.01±0.04 | -0.41±0.02 | 3.85±0.09 | 4.88±0.05 | 2.00±0.02 | 1.3 |  |  |
| target | $\mathcal{L}_{VI}$ | 1.19±0.03 | 0.79±0.09 | -0.89±0.01 | 0.24±0.64 | 2.60±0.50 | 1.30±0.01 | 2.5 |  |
| $\mathcal{L}_{ML}$ | 1.12±0.08 | 0.53±0.04 | -0.91±0.05 | 3.56±0.06 | 4.29±0.04 | 1.63±0.01 | 2.3 |  |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 1.22±0.04 | 0.79±0.03 | -0.72±0.02 | 3.79±0.09 | 4.31±0.02 | 1.61±0.02 | 1.2 |  |  |
| TNP-D<br>(Nguyen & Grover, ) | context | $\mathcal{L}_{ML}$ | 2.58±0.01 | 2.57±0.01 | -0.52±0.00 | 1.73±0.11 | 10.63±0.12 | 4.61±0.27 | 1.8 |
| $\mathcal{L}_{RNP(\alpha)}$ | 2.59±0.00 | 2.59±0.00 | -0.52±0.00 | 1.81±0.12 | 10.72±0.08 | 4.66±0.23 | 1.0 |  |  |
| target | $\mathcal{L}_{ML}$ | 1.38±0.01 | 1.03±0.00 | -0.59±0.00 | 1.63±0.07 | 6.69±0.04 | 2.45±0.05 | 1.8 |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 1.41±0.00 | 1.04±0.00 | -0.59±0.00 | 1.67±0.07 | 6.71±0.04 | 2.46±0.06 | 1.0 |  |  |
| VNP<br>(Guo et al., ) | context | $\mathcal{L}_{VI}$ | 1.37±0.00 | 1.37±0.00 | 1.23±0.03 | 1.60±0.10 | 0.80±0.00 | 0.08±0.03 | 2.0 |
| $\mathcal{L}_{RNP(\alpha)}$ | 1.38±0.00 | 1.38±0.00 | 1.32±0.01 | 3.63±0.39 | 4.00±0.06 | 2.65±0.06 | 1.0 |  |  |
| target | $\mathcal{L}_{VI}$ | 0.90±0.02 | 0.70±0.03 | -0.49±0.00 | 1.59±0.10 | 0.80±0.00 | 0.08±0.03 | 2.0 |  |
| $\mathcal{L}_{RNP(\alpha)}$ | 0.92±0.01 | 0.71±0.03 | -0.48±0.00 | 3.62±0.37 | 3.89±0.06 | 2.49±0.06 | 1.0 |  |  |

### 7.2 Prior misspecification

We now investigate the performance of our RNP method under prior misspecification. Table 2 shows the test log-likelihood for the high-performing baseline TNP-D on two misspecified cases where tasks are generated from different distributions during the meta training and meta testing phase. For the 1D regression task, the model is trained using the Lotka-Volterra dataset which is generated by a two-variable dynamical system for prey-predator simulations (Gordon et al., ). We then test the model on a real-world Hare-Lynx dataset which tracks the two species populations over 90 years. The input and output features were normalized via z-score normalization. The experiment is designed to test prior model misspecification since $p(\mathbf{z}|{\mathbb{D}}_{\text{train}})$ and $p(\mathbf{z}|{\mathbb{D}}_{\text{test}})$ do not come from the same distribution. The results in table 2 show that RNP significantly outperformed the ML objective on both the training and testing data, highlighting the robustness of our objective. Fig LABEL:fig:misspec_Lotka_Voltterra_HareLynx shows the prediction results on the Lynx dataset, where the RNP achieves better uncertainty estimate and tracks the seasonality of the data more efficiently than the ML objective. We also tested TND-D on the Extended MNIST dataset with 47 classes that include letters and digits. We use classes 0-10 for meta training and hold out classes 11-46 for meta testing under prior misspecification. Table 2 shows that RNP performed slightly worse on the EMNIST training task but significantly outperformed the ML objective on the test set (last column), which demonstrates the superior robustness of the new objective under misspecification.

**Table 2: Loglikelihood $\uparrow\uparrow$ under prior misspecification using TNP-D.**
| Objective | ${\mathbb{D}}_{\text{train}}$<br>(Lotka-Volterra) | Misspec ${\mathbb{D}}_{\text{test}}$<br>(Hare-Lynx) | ${\mathbb{D}}_{\text{train}}$ EMNIST<br>(class 0-10) | Misspec ${\mathbb{D}}_{\text{test}}$<br>(class 11-46) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| context | target | context | target | context | target | context | target |  |
| $\mathcal{L}_{ML}$ | 3.09±0.22 | 1.98±0.11 | -0.59±0.47 | -4.44±0.41 | 1.54±0.05 | 1.56±0.07 | 0.03±0.97 | -0.20±0.57 |
| $\mathcal{L}_{RNP}$ | 3.32±0.15 | 2.12±0.06 | -0.17±0.31 | -3.63±0.09 | 1.52±0.08 | 1.47±0.12 | 0.96±0.18 | 0.70±0.15 |

### 7.3 Hyper-parameter tuning

How to select the optimal $\alpha$ values? We have already shown choosing some $\alpha$ can obtain better log-likelihood across different datasets. We now demonstrate that one can find the optimal task-specific $\alpha$ value to further improve the performance. Following (Futami et al., ), we use cross validation for $\alpha$ selection. We hyper-searched the $\alpha$ values from 0 to 2 with an interval of 0.1. For $\alpha=1$, we use the vanilla NP objective. As shown in Fig LABEL:fig:_alpha_tuning_RBF and Fig LABEL:fig:_alpha_tuning_MNIST, the optimal solutions are data-specific and model specific, but $\alpha\in(0,1)$ generally improves the NP objective more than $\alpha>1$. Empirically, we found $\alpha=0.7$ for Eq 7 and $\alpha=0.3$ for Eq 12 keep a good balance between performance improvement and prior regularization.

### 7.4 Ablation studies

Effects of Monte Carlo samples on likelihood.
As both RNP (Eq 7) and RNP-ML (Eq 12) require MC approximations, we investigate the effects of the number of MC samples $K$ on predictive likelihood. We set $K\in\{1,8,16,32,50\}$ for optimizing the RNP objective during training and use $K=50$ for inference. Note that $K=1$ corresponds to the deterministic NPs (conditional NPs). Fig LABEL:fig:MC_samples shows both the context and target log-likelihood for three methods: NP, ANP and TNP-D on the RBF dataset. As expected, increasing the number of MC samples improves the LL mean and also reduces the variance for all the methods with $K=50$ achieving the highest LL and the smallest variance. In practice, we set $K=32$ to balance performance and memory efficiency.

Effects of the number of context points on likelihood. We study the effect of the context points on the target likelihood. During training the number of context points is sampled from $\mathcal{U}(3,50)$ and we vary the number of context points from 5 to 95 at an interval of 10 for evaluation. The results of RBF in Fig LABEL:fig:_ncontext shows that increasing the context set size leads to improved LL for all the methods. Most methods (e.g., NP, ABP, VNP) plateaued after the number of contexts increases to more than 45, whereas TNP-D still shows unsaturated performance improvement with the increased context size.

## 8 Conclusion

In this paper, we propose the Rényi Neural Process (RNP), a new NP framework designed to mitigate prior misspecification in neural processes. We bridge the commonly adopted variational inference and maximum likelihood estimation objectives in vanilla NPs through the use of Rényi divergence and demonstrate the superiority of our generalized objective in improving predictive performance by selecting optimal $\alpha$ values. We apply the framework to multiple state-of-the-art NP models and observe consistent log-likelihood improvements across benchmarks, including 1D regression, image inpainting, and real-world regression tasks. Our framework can be further extended to improve the robustness of contextual inferences, such as prompt design in large language models.
Limitations of our framework lie in drawing multiple samples with Monte-Carlo which scarifies the computational efficiency in exchange for better performance and infeasibility to compute analytical solutions on the divergence.

## Appendix A Appendix

### A.1 Proof of Proposition 3.2

The true variational objective of neural processes without prior approximation can be written as $\mathcal{L}_{VI}$:

$$ $\displaystyle\mathcal{L}_{VI}=\mathbb{E}_{q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C} )}\log p(Y_{T}|X_{T},\mathbf{z})-\mathcal{KL}(q(\mathbf{z}|X_{T},Y_{T},X_{C},Y _{C})||p(\mathbf{z}|X_{C},Y_{C}))$ (15a) $\displaystyle=\mathbb{E}_{q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}[\log p(Y_{T}| X_{T},\mathbf{z})+\log p(\mathbf{z}|X_{C},Y_{C})-\log q(\mathbf{z}|X_{T},Y_{T} ,X_{C},Y_{C})$ $\displaystyle\phantom{={}}\phantom{={}}\phantom{={}}+\log q(\mathbf{z}|X_{C},Y _{C})-\log q(\mathbf{z}|X_{C},Y_{C})]$ (15b) $\displaystyle=\mathbb{E}_{q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}[\log p(Y_{T}| X_{T},\mathbf{z})-\mathcal{KL}(q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})||q( \mathbf{z}|X_{C},Y_{C}))]$ $\displaystyle\phantom{={}}\phantom{={}}\phantom{={}}+\mathbb{E}_{q(\mathbf{z}| X_{T},Y_{T},X_{C},Y_{C})}\log\frac{p(\mathbf{z}|X_{C},Y_{C})}{q(\mathbf{z}|X_{ C},Y_{C})}$ (15c) $\displaystyle=\mathcal{L}_{NPVI}+\mathbb{E}_{q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_ {C})}\log\frac{p(\mathbf{z}|X_{C},Y_{C})}{q(\mathbf{z}|X_{C},Y_{C})}$ (15d) $$

### A.2 Gradients of the Divergence in Neural Processes

We derive the gradients of the KL divergence term in the NP loss. As the the approximated prior also shares the parameters of the encoder network, the result is different from standard VI. Suppose we abbreviate $(X_{T},Y_{T})$ as $T$ and $(X_{C},Y_{C})$ as $C$.

$$ $\displaystyle\nabla_{\varphi}\mathcal{L}_{NPVI}=\nabla_{\varphi}\left(\mathbb{ E}_{q_{\varphi}(\mathbf{z}|C,T)}\left[\log p_{\theta}(Y_{T}|X_{T},\mathbf{z})- \log q_{\varphi}(\mathbf{z}|C,T)+\log q_{\varphi}(\mathbf{z}|C)\right]\right)$ (16a) $\displaystyle=\int\left(q_{\varphi}(\mathbf{z}|C,T)\nabla_{\varphi}\log\frac{p _{\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}( \mathbf{z}|C,T)}+\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C,T)\left[\log\frac{p_ {\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf {z}|C,T)}\right]\right)d\mathbf{z}$ (16b) $\displaystyle=\int q_{\varphi}(\mathbf{z}|C,T)\nabla_{\varphi}\log p_{\theta}( Y_{T}|X_{T},\mathbf{z})+q_{\varphi}(\mathbf{z}|C,T)\nabla_{\varphi}\log q_{ \varphi}(\mathbf{z}|C)-q_{\varphi}(\mathbf{z}|C,T)\nabla_{\varphi}\log q_{ \varphi}(\mathbf{z}|C,T)$ $\displaystyle+\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C,T)\left[\log\frac{p_{ \theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf{ z}|C,T)}\right]d\mathbf{z}$ (16c) $\displaystyle=\int\frac{q_{\varphi}(\mathbf{z}|C,T)}{q_{\varphi}(\mathbf{z}|C) }\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C)-\nabla_{\varphi}q_{\varphi}(\mathbf {z}|C,T)+\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C,T)\left[\log\frac{p_{\theta} (Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf{z}|C,T) }\right]d\mathbf{z}$ (16d) $\displaystyle=\int\left(\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C)\frac{q_{ \varphi}(\mathbf{z}|C,T)}{q_{\varphi}(\mathbf{z}|C)}+\nabla_{\varphi}q_{ \varphi}(\mathbf{z}|C,T)\left[\log\frac{p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{ \varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf{z}|C,T)}\right]\right)d\mathbf{z}$ (16e) $$

Details of the gradient over the parametric VIs can be referred to (Sjölund, ). The first term updates the approximate conditional prior so that $\frac{q_{\varphi}(\mathbf{z}|C,T)}{q_{\varphi}(\mathbf{z}|C)}$ is maximized . The second therm updates the posterior so that $\log\frac{p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|C)}{q_{
\varphi}(\mathbf{z}|C,T)}$ is maximized.

### A.3 Derivation of RNP (eq 7 )

$$ $\displaystyle\min_{\theta,\varphi}D_{\alpha}\left(q_{\varphi}(\mathbf{z}|X_{T} ,Y_{T},X_{C},Y_{C})\|p(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\right)$ (17) $\displaystyle=\min_{\theta,\varphi}\frac{1}{\alpha-1}\log\mathbb{E}_{q_{ \varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p(\mathbf{z}|X_{T},Y_ {T},X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\right]^{1- \alpha}(\text{By definition})$ (18) $\displaystyle=\max_{\theta,\varphi}\frac{1}{1-\alpha}\log\mathbb{E}_{q_{ \varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p(\mathbf{z},Y_{T}|X_ {T},X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\right]^{1- \alpha}+Const.(\text{Split marginal likelihood})$ (19) $\displaystyle=\max_{\theta,\varphi}\frac{1}{1-\alpha}\log\mathbb{E}_{q_{ \varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p(\mathbf{z},Y_{T}|X_ {T},X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\right]^{1- \alpha}(\text{Equivalence of removing the constant})$ (20) $\displaystyle\approx\max_{\theta,\varphi}\frac{1}{1-\alpha}\log\mathbb{E}_{q_{ \varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p_{\theta}(Y_{T}|X_{T },\mathbf{z})q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T} ,Y_{T},X_{C},Y_{C})}\right]^{1-\alpha}$ (21) $$

### A.4 Theoretical relationships between the NPVI, NPML and RNP objectives. 4.2

Our Rényi objective unifies the common three objectives for NPs: $\mathcal{L}_{NPVI},\mathcal{L}_{NPML}$ (maximum likelihood estimation), and $\mathcal{L}_{CNP}$ (conditional NPs or deterministic NPs).

$$ $\displaystyle\mathcal{L}_{RNP}:\frac{1}{1-\alpha}\log\mathbb{E}_{q_{\varphi}( \mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p_{\theta}(Y_{T}|X_{T},\mathbf {z})q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_ {C},Y_{C})}\right]^{1-\alpha}$ (22a) $\displaystyle\mathcal{L}_{NPVI}(\alpha=1):\quad\mathbb{E}_{q_{\varphi}(\mathbf {z}|X_{T},Y_{T},X_{C},Y_{C})}\log\left[\frac{p_{\theta}(Y_{T}|X_{T},\mathbf{z} )q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C} ,Y_{C})}\right]\quad$ (22b) $\displaystyle\mathcal{L}_{NPML}(\alpha=0):\log\mathbb{E}_{q_{\varphi}(\mathbf{ z}|X_{C},Y_{C})}p_{\theta}(Y_{T}|X_{T},\mathbf{z})$ (22c) $\displaystyle\mathcal{L}_{CNP}\left(\alpha=0\text{ and }q_{\varphi}(\mathbf{z} |X_{C},Y_{C})=\delta(\varphi(X_{C},Y_{C})\right):\log p_{\theta}(Y_{T}|X_{T}, \varphi(X_{C},Y_{C}))$ (22d) $$

Proof of the NPML objective:

$$ $\displaystyle\mathcal{L}_{RNP(\alpha=0)}=\log\mathbb{E}_{q_{\varphi}(\mathbf{z }|X_{T},Y_{T},X_{C},Y_{C})}\left[\frac{p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{ \varphi}(\mathbf{z}|X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{ C})}\right]$ (23a) $\displaystyle=\log\int q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\left[ \frac{p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}{q _{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\right]$ (23b) $\displaystyle=\log\int p_{\theta}(Y_{T}|X_{T},\mathbf{z})q_{\varphi}(\mathbf{z }|X_{C},Y_{C})=\log\mathbb{E}_{q_{\varphi}(\mathbf{z}|X_{C},Y_{C})}p_{\theta}( Y_{T}|X_{T},\mathbf{z})=\mathcal{L}_{NPML}$ (23c) $$

Proof of the NPVI objective:
Next we will prove that $\mathcal{L}_{RNP,\alpha\rightarrow 1}=\mathcal{L}_{NPVI}$. Applying Theorem 5 from (Van Erven & Harremos, ) to the new posteior and prior, we have:

$$ $\displaystyle\mathcal{D}_{\alpha\rightarrow 1}\left(q_{\varphi}(\mathbf{z}|X_{ T},Y_{T},X_{C},Y_{C})||p(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\right)$ (24a) $\displaystyle=\mathcal{KL}\left(q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C} )||p(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})\right)$ (24b) $\displaystyle=\mathbb{E}_{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\log \frac{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})p(Y_{T}|X_{C},Y_{C},X_{T} )}{p(\mathbf{z},Y_{T}|X_{T},X_{C},Y_{C})}$ (24c) $\displaystyle=-\mathbb{E}_{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})} \log\frac{p(\mathbf{z},Y_{T}|X_{T},X_{C},Y_{C})}{q_{\varphi}(\mathbf{z}|X_{T}, Y_{T},X_{C},Y_{C})}+Const$ (24d) $\displaystyle\equiv-\mathbb{E}_{q_{\varphi}(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C} )}\log p(Y_{T}|\mathbf{z},X_{T})+\log p(\mathbf{z}|X_{C},Y_{C})-\log q(\mathbf {z}|X_{T},Y_{T},X_{C},Y_{C})=\mathcal{L}_{NPVI}$ (24e) $$

### A.5 Gradients of Objectives for Rényi Neural Processes

$$ $\displaystyle\nabla\varphi\mathcal{L}_{RNP}=\nabla\varphi\left(\frac{1}{\alpha -1}\log\mathbb{E}_{q_{\varphi}(\mathbf{z}|C,T)}[\frac{p(Y_{T}|X_{T},\mathbf{z} )q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf{z}|C,T)}]^{1-\alpha}\right)$ (25a) $\displaystyle=\frac{1}{\alpha-1}\frac{\int\nabla\varphi\left[p(Y_{T}|X_{T}, \mathbf{z})^{1-\alpha}q_{\varphi}(\mathbf{z}|C)^{1-\alpha}q_{\varphi}(\mathbf{ z}|C,T)^{\alpha}\right]d\mathbf{z}}{\int p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}q _{\varphi}(\mathbf{z}|C)^{1-\alpha}q_{\varphi}(\mathbf{z}|C,T)^{\alpha}d \mathbf{z}}$ (25b) $\displaystyle=\frac{1}{\alpha-1}\frac{\int p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha }\nabla\varphi\left[q_{\varphi}(\mathbf{z}|C)^{1-\alpha}q_{\varphi}(\mathbf{z} |C,T)^{\alpha}\right]d\mathbf{z}}{\int p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}q_{ \varphi}(\mathbf{z}|C)^{1-\alpha}q_{\varphi}(\mathbf{z}|C,T)^{\alpha}d\mathbf{ z}}$ (25c) $\displaystyle=-\frac{1}{1-\alpha}\frac{A}{\int p(Y_{T}|X_{T},\mathbf{z})^{1- \alpha}q_{\varphi}(\mathbf{z}|C)^{1-\alpha}q_{\varphi}(\mathbf{z}|C,T)^{\alpha }d\mathbf{z}}$ $\displaystyle A=\int p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}\left[\alpha\left( \frac{q_{\varphi}(\mathbf{z}|C)}{q_{\varphi}(\mathbf{z}|C,T)}\right)^{1-\alpha }\nabla_{\varphi}q_{\varphi}(\mathbf{z}|C,T)+(1-\alpha)\left(\frac{q_{\varphi} (\mathbf{z}|C,T)}{q_{\varphi}(\mathbf{z}|C)}\right)^{\alpha}\nabla_{\varphi}q_ {\varphi}(\mathbf{z}|C)\right]d\mathbf{\mathbf{z}}$ (25d) $$

### A.6 Additional Results.

We added the RMSE results in Table 3 and listed the notations used in the paper in Table 4.

**Table 3: Test set RMSE $\downarrow\downarrow$ (R- stands for Rényi-divergence)**
| Method | RBF | Matern 5/2 | Periodic | MNIST | SVHN | CelebA |
| --- | --- | --- | --- | --- | --- | --- |
| NP | 0.194 (0.005) | 0.220 (0.003) | 0.416 (0.000) | 0.084 (0.002) | 0.074 (0.001) | 0.118 (0.000) |
| R-NP | 0.185 (0.003) | 0.214 (0.003) | 0.417 (0.000) | 0.084 (0.002) | 0.072 (0.002) | 0.118 (0.000) |
| ANP | 0.104 (0.000) | 0.129 (0.000) | 0.386 (0.001) | 0.066 (0.002) | 0.042 (0.000) | 0.095 (0.000) |
| R-ANP | 0.104 (0.000) | 0.129 (0.000) | 0.389 (0.001) | 0.063 (0.000) | 0.042 (0.000) | 0.097 (0.001) |

**Table 4: Notation**
| Name | Description |
| --- | --- |
| $f$ | function sample from a stochastic process |
| $\mathcal{X}$ | input space |
| $\mathcal{Y}$ | output space |
| $\mathcal{Z}$ | latent space |
| $\mathbf{x}$ | input features |
| $\mathbf{y}$ | output features |
| $\mathbf{z}$ | latent variable representing stochasticity of the functional sample $f$ |
| $X_{C}$ | $\mathbb{R}^{M\times D}$ context inputs |
| $Y_{C}$ | $\mathbb{R}^{M\times 1}$ context outputs |
| $X_{T}$ | $\mathbb{R}^{N\times D}$ target inputs |
| $Y_{T}$ | $\mathbb{R}^{N\times 1}$ target outputs |
| $M$ | number of samples in the context set, indexed by $m$ |
| $N$ | number of samples in the target set, indexed by $n$ |
| $D_{x}$ | dimension of input features |
| $D_{y}$ | dimension of input features |
| $C,T$ | notation for Context and Target |
| $\beta$ | noise level for misspecified prior |
| $\epsilon$ | added noise for noisy context |
| $\varphi$ | parameters in the posterior model $q(\mathbf{z}|X,Y)$ |
| $\theta$ | parameters in the likelihood model $p(Y|\mathbf{z},X)$ |
| $\eta$ | pushward of a measure $\mathbb{P}:\mathcal{X}\times\mathcal{Y}\rightarrow\mathcal{Z}$ |
| $K$ | number of $\mathbf{z}$ samples for Monte Carlo approximation |

Figure: Figure 13: Local $\alpha$ divergence
Refer to caption: x2.png

Alternative objective using a prior-based “local” RD. Recall that
in standard variational inference the main goal is to minimize the KL divergence between the approximate posterior and the true posterior $\mathcal{KL}(q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}|\mathbf{x}))$. Since the true posterior is unavailable, VI instead maximizes the ELBO, which is the sum of an expected log likelihood term and the negative KL divergence between the approximate posterior and the prior $\mathcal{KL}(q(\mathbf{z}|\mathbf{x})||p(\mathbf{z}))$.
In contrast, our approach minimizes the RD directly, as it involves the logarithm of an expectation instead of the expectation of a logarithm.
However, it is possible to maximize an objective analogous to the ELBO:
$\mathbb{E}_{q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})}\log p(Y_{T}|X_{T},\mathbf{z
})-\mathcal{D}_{\alpha}(q(\mathbf{z}|X_{T},Y_{T},X_{C},Y_{C})||p(\mathbf{z}|X_
{C},Y_{C}))$ that replaces the prior-based KL divergence with the RD. We will refer to this objective as “local”.

Fig 13 shows the test log-likelihood difference between two objectives using the optimal $\alpha$ values. Our method outperforms the local Rényi divergence on three GP regression tasks. By comparing the gradient over the posterior parameters of the two objectives, our objective adds a scaling factor $p(Y_{T}|X_{T},\mathbf{z})^{1-\alpha}$ to the “local ” RD objective, which imposes a larger gradient penalty on less confident predictions and therefore focus more on samples with low likelihoods. This further improves the mass covering behaviour of the RNP and leads to better log-likelihood values.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: - The authors mention that the proposed framework can be further extended to improve the robustness of contextual inferences, such as prompt design in large language models. Could the authors provide more details or insights into how their method could be adapted or applied to such tasks?

- The parameter coupling between the posterior and prior in neural processes is by design to share parameters, but it's not a hard constraint. Isn't a simple baseline approach to mitigate prior misspecification to separately parameterize the posterior and prior models? Could the authors provide a comparison with this simple baseline to better demonstrate the advantages of their proposed method?

Overall, naively combining existing ideas may not constitute a sufficient contribution for a top-tier conference like ICLR, which expects a higher level of novelty and significance.
soundness: 3
strengths: - The paper provides a thorough theoretical analysis and derivations for the proposed Rényi Neural Process objective, establishing a solid foundation for the proposed method.
- The experiments are comprehensive, covering various datasets and tasks, including regression, image inpainting, and real-world regression problems with prior misspecification.
- The authors conduct extensive ablation studies and hyperparameter tuning, demonstrating the robustness and effectiveness of the proposed method.
summary: The paper proposes RNPs, a new framework that aims to mitigate the issue of prior misspecification in neural processes (NPs). NPs are deep probabilistic models that represent stochastic processes by conditioning prior distributions on context points. However, the parameterization coupling between the prior and posterior models in NPs can lead to a misspecified prior, resulting in biased posterior estimates and degraded performance.

To address this, the authors propose optimizing an alternative posterior using the Rényi divergence between the model posterior and the true posterior, instead of the standard KL divergence. The Rényi divergence introduces a hyperparameter α that scales the density ratio between the posterior and prior, dampening the effects of the misspecified prior. The proposed RNP objective unifies the variational inference and maximum likelihood estimation objectives for training NPs via α, allowing better marginal likelihood and posterior expressiveness.
weaknesses: - Limited novelty: The core idea of using an alternative divergence measure (Rényi divergence) to mitigate the effects of prior misspecification is not entirely novel. Previous works in the domain of robust variational inference have explored the use of other divergences, such as α-divergences and f-divergences, to address similar issues. The authors could provide a more comprehensive discussion of how their work relates to and differentiates from these earlier efforts.
- Hyperparameter sensitivity: The choice of the hyperparameter α plays a crucial role in the performance of the proposed method. While the authors provide some guidelines for tuning α, a more comprehensive analysis of the sensitivity of the method to different values of α and strategies for automatic tuning could further enhance the practical utility of the proposed framework.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 4
questions: 1, How does the choice of Renyi divergence impact the trade-off between predictive performance and computational efficiency in RNP compared to traditional Neural Processes?

2, What strategies can be employed to effectively tune the additional parameters introduced in RNP to ensure optimal model performance across different applications?
soundness: 3
strengths: RNP consistently achieves better log-likelihood and mitigates over-smoothing in predictions, particularly in challenging tasks like periodic data regression and higher-dimensional image inpainting. By utilizing Renyi divergence, RNP enhances the expressiveness of the posterior distribution, leading to more reliable uncertainty quantification compared to traditional Neural Processes.
summary: The paper introduces Renyi Neural Process, a framework that addresses the limitations of standard Neural Processes (NPs) by mitigating prior misspecification through a new objective function. RNP enhances uncertainty estimation and robustness in learning by applying Rényi divergence instead of traditional KL divergence, leading to improved performance across various benchmarks. The authors demonstrate consistent log-likelihood improvements in tasks such as 1D regression and image inpainting while acknowledging the trade-off between computational efficiency and performance due to Monte Carlo sampling.
weaknesses: The use of Renyi divergence requires additional computations, particularly due to the need for Monte Carlo sampling to estimate the divergence. This can lead to longer training times and higher resource consumption compared to standard Neural Processes, which may limit scalability in large datasets or real-time applications. RNP introduces extra parameters that control the behavior of the divergence, which can make the model sensitive to hyperparameter tuning. Improper selection of these parameters may lead to suboptimal performance, requiring extensive experimentation to find the best configuration for specific tasks. The complexity of the RNP framework, particularly with the incorporation of robust divergences, can make it harder to interpret the model's decisions and the underlying relationships in the data. This lack of interpretability may hinder its adoption in fields where understanding model behavior is crucial, such as healthcare or finance.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: ### Q1: Gradient of Renyi divergence is potentially misleading
Is the argument here potentially incorrect/misleading since the the gradient needs to also be taken with respect to the $q_\varphi(\mathbf{z\mid X_C, Y_C) \approx p(\mathbf{z})$?

### Q2: In A.3, doesn’t the logic from Eq 20 to 21 require that the prior is well specified?
Does this equality hold without this assumption?
soundness: 4
strengths: The authors identify an important modeling assumption in NP that can be a source of failure for these models and suggest an easy to implement and intuitive remedy.

The experimental section is well laid out, and I appreciated that the authors explicitly connected each experimental subsection to an empirical question, e.g., in lines 397-400. The empirical results themselves are extensive. The improvements over strong baselines, such as TNP, are compelling.
summary: This paper aims to address a key potential weakness in latent variable neural processes (NP), namely, that the true prior $p(\mathbf{z} \mid X_C, Y_C)$ can be approximated by a parametric model $q_\phi(\mathbf{z} \mid X_C, Y_C)$ which shares parameters with the approximate posterior model. When the prior is “misspecified”, as defined rigorously in the text, the variational bound used to optimize NP is no longer valid and the authors argue leads to worse data fitting. They remedy this issue by replacing the KL divergence in the lower bound with a Rényi divergence, whose hyperparameter $\alpha$ enables more expressive posteriors. On a variety of experiments commonly used in the NP literature, the authors demonstrate that existing NP models benefit from the Rényi divergence variation of the objective.
weaknesses: The list below roughly follows the order of appearance in the manuscript. In my opinion W2, W6, W10 are the most important among these.

### W1: The claims at the end of the introduction seem to overlap quite a bit.
For example, in contribution 1, the authors already discuss the improved likelihoods, which seems like it is more a part of the empirical contributions of 2 and 3. Additionally, again, in contribution 2 the reference to “consistent log-likelihood improvements” is actually the main point of the contribution of the third bullet.

### W2: The effects of the misspecified prior can be made more rigorous
Currently, the authors claim that KL divergence leads to:
- Biased estimation of the posterior variance
- Less expressive models that underfit

However, analyzing these claims analytically would be quite helpful ,e.g., what is the analytical form of this bias in estimating the posterior variance and is there a way to make the “underfitting” more rigorous. Currently, the main analysis is a single motivating example. For example, Figure2b is quite anecdotal and only shows a single sample.

I recognize that this is not a small ask, but I believe it will significantly improve the quality of Section 2 and the overall motivation.

### W3: Notation of Definition 3.1 seems detached from the rest of the paper.
How does $\eta$ relate to the rest of the notation introduced in this work? Does it correspond to a shared parametric form of $p_\varphi$ and $q_\varphi$? If so, this needs to be spelled out.

### W4: Proposition 3.2 is missing an “if”
Shouldn’t there be an “if” in lines 173 and 174:
> …, **if** the prior model is misspecified,...

Currently the proposition reads as if misspecification is inevitable, which, in my understanding, is not necessarily true.

### W5: I think the equation for $\mathcal{RNP}$ is imprecise
You are taking a Monte Carlo estimate of an expectation that relies on the parameters over which you are optimizing (presumably via stochastic gradient descent). This would require some methodology along the lines of the reparameterization trick which should be explicitly stated here.

### W6: Prior misspecification does not seem related to the ML-based objective
It is not clear to me from the current exposition why prior misspecification should affect the ML-based version of the NP objective.

### W7: Lines 242-247 seem out of place / irrelevant
I do not understand how these lines relate to the rest of this section. The generalization / recasting of TNP-A as an implicit latent variable model with Diracs is not used in the rest of the exposition / derivation to the best of my understanding.

### W8: The tabular regression experiments (Pace & Barry, 1997) appear to be missing
These results are not in the experimental results sections nor in any of the appendix sections

### W9: The details for the differential equations (e.g. Hare-Lynx and Lotka-Volterra) should be added to Section 7 before the Baselines paragraph.
A similar level of detail to that provided for GP/inpainting experiments should be added for these DE experimental setups as well.

### W10: The improvements seen in Table 1 (i.e., where misspecification is presumably not a problem) are not well explained.
Why do we see gains in the standard GP / inpainting experiments from RNP if there is not necessarily a misspecification issue here?

### W11: The additional compute overhead from the Monte Carlo approximation should be quantified.
Both for the main results and in the ablation analysis, the authors should have a secondary axis or some way of conveying how the actual compute overhead and how it grows in $K$.

### W12: The graphs are quite small and difficult to read without extensively zooming.
The legends are quite difficult to make out. 

Additionally, Figure3b has a different line color for some reason and Figures 4a and 4b are missing a legend for the red dot indicator.

### W13: Typos / Grammatical Errors
Below I list the minor typos/errors that I noticed through a preliminary read:
- Line 035: “advance” should perhaps be “advantage”?
- Line 061: “poster variance” should be “posterior”
- Line 074: “achieve the better…” should be “achieve a better”
- Lines 149-150: The wording is confusing here “the model can reduce the prior penalty less than significantly than the standard KL”
- Line 152: “overestimate” should be “overestimated”
- Line 216: There is a missing space between “inA.3”
- Line 233: $p_\varphi$ should be $q_\varphi$, I believe.
- Line 239: $p_\varphi$ should be $q_\varphi$, I believe.
- Line 269: $d\mathbf{z}$ should be removed from equation 12.
- Lines 404-405: These lines are written in a slightly confusing way. It sounds like the $\alpha$ value is what the baseline corresponds to. But the baselines use $\alpha = 0$ / $\alpha=1$ right?
- Lin 783: Should be $\alpha \rightarrow 1$ in the parentheses
- Table 1,Table 2, and Table 3 captions have duplicated $\uparrow$’s and $\downarrow$’s

### W14: Move Table 4 to the start of the appendix.
Not a real weakness, just a suggestion. Feel free to ignore it.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: * In Table 1, are you using ML or VI objectives for your RD-based loss functions?
* I am not very sure how and why you are calculating the log-likelihood on the context set. What data are you conditioning on when predicting the context set?
* On TNP-D, the results using $L_{RNP}$ are always better than using $L_{ML}$ in Table 1, why on the EMNIST dataset (Table 2), the results become worse?
soundness: 2
strengths: * The proposed method is simple yet effective, and can be easily deployed to other NP models by simply replacing the corresponding objective functions. This makes the method highly practical.
* The paper is overall well-written, with thorough literature review and clear, intuitive figures that effectively convey the key concepts.
* The paper employs a comprehensive set of baselines covering both CNP and NP variants, including the current state-of-the-art TNP method. The evaluation is also conducted on two real-world datasets, showing its practical applicability. Besides, the authors provide guidance for choosing $\alpha$, along with several ablation studies.
* They provide theoretical analysis connecting VI and ML objectives through the $\alpha$ parameter, and prove how RNP helps with prior misspecification.
* Their approach to rewriting the ML estimation as minimizing the KLD between the empirical distribution and the model distribution, and then applying RD is an interesting perspective.
summary: The paper introduces Rényi Neural Processes (RNP), a simple method that replaces KL divergence (KLD) with Rényi divergence (RD) to improve the neural process (NP) when there is prior misspecification. They propose RD-based objective functions for both commonly used NP objectives - Variational Inference (VI) in latent NP and Maximum Likelihood (ML) in conditional NP. The authors also provide theoretical analysis and proofs to support their proposed objective functions. In the experimental section, they implement RNP across multiple NP variants and demonstrate consistent improvements over the original models.
weaknesses: I am still trying to understand the definition of prior misspecification in your paper, it seems to lack clarity and consistency, and there appears to be a disconnect between the theoretical motivation and experimental validation.

Based on the introduction and the definition of RD, prior misspecification is presented as the mismatch between context and target sets, which commonly occurs when collecting context data with additional noise or other uncontrollable factors.
However, the toy examples and experimental setups seem to address a different type of prior misspecification - the mismatch between training and test distributions ($D_{train}$ and $D_{test}$), which is more akin to domain shift problems.

These are arguably two distinct scenarios, and the paper doesn't clearly explain:

* How these two types of misspecification relate to each other
* Why RD would help with the domain shift scenario
* Whether these can be unified under a single framework of prior misspecification

If I have misunderstood any points, I would appreciate clarification from the authors, and I am willing to increase my score if my concerns are fully addressed.

Some other points:
* In Section 7.1, the experiments are conducted on seemingly well-specified datasets, yet the RD-based methods still outperform the baselines. Is there any justification explains these improvements? 
* Line 229, ANP uses attention that learns to attend to the contexts relevant to the given target, to my understanding they didn't incorporate dependencies between target points, maybe you are talking about GNP[1] which models the target points jointly?
* Line 242, TNP has three versions, you are talking about TNP-A (autoregressive version), for example, TNP-D doesn't do any autoregressive prediction. And I think you are using TNP-D in the experiment section.
* Line 311, isn't the posterior distribution be $p(z|X_t, Y_t, X_c, Y_c)$ as you mentioned in eq.3?
* Line 352, TNP has an inappropriate reference, Maraval et al., 2024 only apply TNP to the BO setting.

[1] Markou, S., Requeima, J., Bruinsma, W., Vaughan, A., & Turner, R. E. Practical Conditional Neural Process Via Tractable Dependent Predictions. In International Conference on Learning Representations.

Meta-review: The  paper introduces Neural Renyi processes, where the neural processes objectives are revisited with Renyi divergences. Authors claim that the Renyi losses overcome prior misspecification, i,e when the target distribution does not belong to the parametrized family , i,e an identifiability problem. 

Reviewers raised several concerns, one regarding interpretability (Reviewer eR5B), and another one on the link between the prior mispecification and Renyi divergence (Reviewer 6B6t), and lack of novelty (Reviewer T3rx). Reviewer JR2F provided a detailed review that has improved the paper greatly. 

Given the large variance in the scores AC and reviewers discussed the paper, and I read the paper in details. I am not concerned with novelty or with the interpretability of the method. One thing I am still missing as I read the paper is how the prior misspecification is linked  to the use of Renyi divergence. 

The authors should provide a theoretical insight about this claim their work falls within the framework of [a], as it amounts to an importance sampling , this is largely studied in this paper theoretically.  The other aspect is that Renyi divergence minimization is rather a great alternative to KL when the target is heavy tailed or does not have defined moments see [b]. A rigorous proof for Renyi minimization alleviating the misspecification prior would strengthen the paper, or in terms of better targeting the tails as in [b] would bridge more the algorithmic part and the motivation of the work. 
 

[a] https://www.jmlr.org/papers/volume24/22-1160/22-1160.pdf

[b] https://hal.science/hal-04506761/document

Decision: Reject