Rényi Neural Processes

Authors: Xuesong Wang, He Zhao, Edwin V. Bonilla
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Neural processes, variational inference, meta learning, robust divergence

Abstract:
Neural Processes (NPs) are deep probabilistic models that represent stochastic processes by conditioning their prior distributions on a set of context points. Despite their obvious advantages in uncertainty estimation for complex distributions,  NPs enforce parameterization coupling between the conditional prior model and the posterior model, thereby risking introducing a misspecified prior distribution. We hereby revisit the NP objectives and propose Rényi Neural Processes (RNP) to ameliorate the impacts of prior misspecification by optimizing an alternative posterior that achieves better marginal likelihood. More specifically, by replacing the standard KL divergence with the Rényi divergence between the model posterior and the true posterior, we scale the density ratio $\frac{p}{q}$ by the power of (1-$\alpha$) in the divergence gradients with respect to the posterior. This hyper parameter $\alpha$ allows us to dampen the effects of the misspecified prior for the posterior update, which has been shown to effectively avoid oversmoothed predictions and improve the expressiveness of the posterior model.
Our extensive experiments show consistent log-likelihood improvements over state-of-the-art NP family models which adopt both the variational inference or maximum likelihood estimation objectives. We validate the effectiveness of our approach across multiple benchmarks including regression and image inpainting tasks, and show significant performance improvements of RNPs in real-world regression problems where the underlying prior model is misspecifed.

Decision: reject
Reviewer scores: 3, 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730215967955

thread_type:
official_review

<thread_object>
creator: reviewer_6b6t

<content>
summary:
The paper introduces Rényi Neural Processes (RNP), a simple method that replaces KL divergence (KLD) with Rényi divergence (RD) to improve the neural process (NP) when there is prior misspecification. They propose RD-based objective functions for both commonly used NP objectives - Variational Inference (VI) in latent NP and Maximum Likelihood (ML) in conditional NP. The authors also provide theoretical analysis and proofs to support their proposed objective functions. In the experimental section, they implement RNP across multiple NP variants and demonstrate consistent improvements over the original models.

strengths:
* The proposed method is simple yet effective, and can be easily deployed to other NP models by simply replacing the corresponding objective functions. This makes the method highly practical.
* The paper is overall well-written, with thorough literature review and clear, intuitive figures that effectively convey the key concepts.
* The paper employs a comprehensive set of baselines covering both CNP and NP variants, including the current state-of-the-art TNP method. The evaluation is also conducted on two real-world datasets, showing its practical applicability. Besides, the authors provide guidance for choosing $\alpha$, along with several ablation studies.
* They provide theoretical analysis connecting VI and ML objectives through the $\alpha$ parameter, and prove how RNP helps with prior misspecification.
* Their approach to rewriting the ML estimation as minimizing the KLD between the empirical distribution and the model distribution, and then applying RD is an interesting perspective.

weaknesses:
I am still trying to understand the definition of prior misspecification in your paper, it seems to lack clarity and consistency, and there appears to be a disconnect between the theoretical motivation and experimental validation.

Based on the introduction and the definition of RD, prior misspecification is presented as the mismatch between context and target sets, which commonly occurs when collecting context data with additional noise or other uncontrollable factors.
However, the toy examples and experimental setups seem to address a different type of prior misspecification - the mismatch between training and test distributions ($D_{train}$ and $D_{test}$), which is more akin to domain shift problems.

These are arguably two distinct scenarios, and the paper doesn't clearly explain:

* How these two types of misspecification relate to each other
* Why RD would help with the domain shift scenario
* Whether these can be unified under a single framework of prior misspecification

If I have misunderstood any points, I would appreciate clarification from the authors, and I am willing to increase my score if my concerns are fully addressed.

Some other points:
* In Section 7.1, the experiments are conducted on seemingly well-specified datasets, yet the RD-based methods still outperform the baselines. Is there any justification explains these improvements? 
* Line 229, ANP uses attention that learns to attend to the contexts relevant to the given target, to my understanding they didn't incorporate dependencies between target points, maybe you are talking about GNP[1] which models the target points jointly?
* Line 242, TNP has three versions, you are talking about TNP-A (autoregressive version), for example, TNP-D doesn't do any autoregressive prediction. And I think you are using TNP-D in the experiment section.
* Line 311, isn't the posterior distribution be $p(z|X_t, Y_t, X_c, Y_c)$ as you mentioned in eq.3?
* Line 352, TNP has an inappropriate reference, Maraval et al., 2024 only apply TNP to the BO setting.

[1] Markou, S., Requeima, J., Bruinsma, W., Vaughan, A., & Turner, R. E. Practical Conditional Neural Process Via Tractable Dependent Predictions. In International Conference on Learning Representations.

questions:
* In Table 1, are you using ML or VI objectives for your RD-based loss functions?
* I am not very sure how and why you are calculating the log-likelihood on the context set. What data are you conditioning on when predicting the context set?
* On TNP-D, the results using $L_{RNP}$ are always better than using $L_{ML}$ in Table 1, why on the EMNIST dataset (Table 2), the results become worse?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730416216122

thread_type:
official_review

<thread_object>
creator: reviewer_jr2f

<content>
summary:
This paper aims to address a key potential weakness in latent variable neural processes (NP), namely, that the true prior $p(\mathbf{z} \mid X_C, Y_C)$ can be approximated by a parametric model $q_\phi(\mathbf{z} \mid X_C, Y_C)$ which shares parameters with the approximate posterior model. When the prior is “misspecified”, as defined rigorously in the text, the variational bound used to optimize NP is no longer valid and the authors argue leads to worse data fitting. They remedy this issue by replacing the KL divergence in the lower bound with a Rényi divergence, whose hyperparameter $\alpha$ enables more expressive posteriors. On a variety of experiments commonly used in the NP literature, the authors demonstrate that existing NP models benefit from the Rényi divergence variation of the objective.

strengths:
The authors identify an important modeling assumption in NP that can be a source of failure for these models and suggest an easy to implement and intuitive remedy.

The experimental section is well laid out, and I appreciated that the authors explicitly connected each experimental subsection to an empirical question, e.g., in lines 397-400. The empirical results themselves are extensive. The improvements over strong baselines, such as TNP, are compelling.

weaknesses:
The list below roughly follows the order of appearance in the manuscript. In my opinion W2, W6, W10 are the most important among these.

### W1: The claims at the end of the introduction seem to overlap quite a bit.
For example, in contribution 1, the authors already discuss the improved likelihoods, which seems like it is more a part of the empirical contributions of 2 and 3. Additionally, again, in contribution 2 the reference to “consistent log-likelihood improvements” is actually the main point of the contribution of the third bullet.

### W2: The effects of the misspecified prior can be made more rigorous
Currently, the authors claim that KL divergence leads to:
- Biased estimation of the posterior variance
- Less expressive models that underfit

However, analyzing these claims analytically would be quite helpful ,e.g., what is the analytical form of this bias in estimating the posterior variance and is there a way to make the “underfitting” more rigorous. Currently, the main analysis is a single motivating example. For example, Figure2b is quite anecdotal and only shows a single sample.

I recognize that this is not a small ask, but I believe it will significantly improve the quality of Section 2 and the overall motivation.

### W3: Notation of Definition 3.1 seems detached from the rest of the paper.
How does $\eta$ relate to the rest of the notation introduced in this work? Does it correspond to a shared parametric form of $p_\varphi$ and $q_\varphi$? If so, this needs to be spelled out.

### W4: Proposition 3.2 is missing an “if”
Shouldn’t there be an “if” in lines 173 and 174:
> …, **if** the prior model is misspecified,...

Currently the proposition reads as if misspecification is inevitable, which, in my understanding, is not necessarily true.

### W5: I think the equation for $\mathcal{RNP}$ is imprecise
You are taking a Monte Carlo estimate of an expectation that relies on the parameters over which you are optimizing (presumably via stochastic gradient descent). This would require some methodology along the lines of the reparameterization trick which should be explicitly stated here.

### W6: Prior misspecification does not seem related to the ML-based objective
It is not clear to me from the current exposition why prior misspecification should affect the ML-based version of the NP objective.

### W7: Lines 242-247 seem out of place / irrelevant
I do not understand how these lines relate to the rest of this section. The generalization / recasting of TNP-A as an implicit latent variable model with Diracs is not used in the rest of the exposition / derivation to the best of my understanding.

### W8: The tabular regression experiments (Pace & Barry, 1997) appear to be missing
These results are not in the experimental results sections nor in any of the appendix sections

### W9: The details for the differential equations (e.g. Hare-Lynx and Lotka-Volterra) should be added to Section 7 before the Baselines paragraph.
A similar level of detail to that provided for GP/inpainting experiments should be added for these DE experimental setups as well.

### W10: The improvements seen in Table 1 (i.e., where misspecification is presumably not a problem) are not well explained.
Why do we see gains in the standard GP / inpainting experiments from RNP if there is not necessarily a misspecification issue here?

### W11: The additional compute overhead from the Monte Carlo approximation should be quantified.
Both for the main results and in the ablation analysis, the authors should have a secondary axis or some way of conveying how the actual compute overhead and how it grows in $K$.

### W12: The graphs are quite small and difficult to read without extensively zooming.
The legends are quite difficult to make out. 

Additionally, Figure3b has a different line color for some reason and Figures 4a and 4b are missing a legend for the red dot indicator.

### W13: Typos / Grammatical Errors
Below I list the minor typos/errors that I noticed through a preliminary read:
- Line 035: “advance” should perhaps be “advantage”?
- Line 061: “poster variance” should be “posterior”
- Line 074: “achieve the better…” should be “achieve a better”
- Lines 149-150: The wording is confusing here “the model can reduce the prior penalty less than significantly than the standard KL”
- Line 152: “overestimate” should be “overestimated”
- Line 216: There is a missing space between “inA.3”
- Line 233: $p_\varphi$ should be $q_\varphi$, I believe.
- Line 239: $p_\varphi$ should be $q_\varphi$, I believe.
- Line 269: $d\mathbf{z}$ should be removed from equation 12.
- Lines 404-405: These lines are written in a slightly confusing way. It sounds like the $\alpha$ value is what the baseline corresponds to. But the baselines use $\alpha = 0$ / $\alpha=1$ right?
- Lin 783: Should be $\alpha \rightarrow 1$ in the parentheses
- Table 1,Table 2, and Table 3 captions have duplicated $\uparrow$’s and $\downarrow$’s

### W14: Move Table 4 to the start of the appendix.
Not a real weakness, just a suggestion. Feel free to ignore it.

questions:
### Q1: Gradient of Renyi divergence is potentially misleading
Is the argument here potentially incorrect/misleading since the the gradient needs to also be taken with respect to the $q_\varphi(\mathbf{z\mid X_C, Y_C) \approx p(\mathbf{z})$?

### Q2: In A.3, doesn’t the logic from Eq 20 to 21 require that the prior is well specified?
Does this equality hold without this assumption?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730511566418

thread_type:
official_review

<thread_object>
creator: reviewer_er5b

<content>
summary:
The paper introduces Renyi Neural Process, a framework that addresses the limitations of standard Neural Processes (NPs) by mitigating prior misspecification through a new objective function. RNP enhances uncertainty estimation and robustness in learning by applying Rényi divergence instead of traditional KL divergence, leading to improved performance across various benchmarks. The authors demonstrate consistent log-likelihood improvements in tasks such as 1D regression and image inpainting while acknowledging the trade-off between computational efficiency and performance due to Monte Carlo sampling.

strengths:
RNP consistently achieves better log-likelihood and mitigates over-smoothing in predictions, particularly in challenging tasks like periodic data regression and higher-dimensional image inpainting. By utilizing Renyi divergence, RNP enhances the expressiveness of the posterior distribution, leading to more reliable uncertainty quantification compared to traditional Neural Processes.

weaknesses:
The use of Renyi divergence requires additional computations, particularly due to the need for Monte Carlo sampling to estimate the divergence. This can lead to longer training times and higher resource consumption compared to standard Neural Processes, which may limit scalability in large datasets or real-time applications. RNP introduces extra parameters that control the behavior of the divergence, which can make the model sensitive to hyperparameter tuning. Improper selection of these parameters may lead to suboptimal performance, requiring extensive experimentation to find the best configuration for specific tasks. The complexity of the RNP framework, particularly with the incorporation of robust divergences, can make it harder to interpret the model's decisions and the underlying relationships in the data. This lack of interpretability may hinder its adoption in fields where understanding model behavior is crucial, such as healthcare or finance.

questions:
1, How does the choice of Renyi divergence impact the trade-off between predictive performance and computational efficiency in RNP compared to traditional Neural Processes?

2, What strategies can be employed to effectively tune the additional parameters introduced in RNP to ensure optimal model performance across different applications?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667393880

thread_type:
official_review

<thread_object>
creator: reviewer_t3rx

<content>
summary:
The paper proposes RNPs, a new framework that aims to mitigate the issue of prior misspecification in neural processes (NPs). NPs are deep probabilistic models that represent stochastic processes by conditioning prior distributions on context points. However, the parameterization coupling between the prior and posterior models in NPs can lead to a misspecified prior, resulting in biased posterior estimates and degraded performance.

To address this, the authors propose optimizing an alternative posterior using the Rényi divergence between the model posterior and the true posterior, instead of the standard KL divergence. The Rényi divergence introduces a hyperparameter α that scales the density ratio between the posterior and prior, dampening the effects of the misspecified prior. The proposed RNP objective unifies the variational inference and maximum likelihood estimation objectives for training NPs via α, allowing better marginal likelihood and posterior expressiveness.

strengths:
- The paper provides a thorough theoretical analysis and derivations for the proposed Rényi Neural Process objective, establishing a solid foundation for the proposed method.
- The experiments are comprehensive, covering various datasets and tasks, including regression, image inpainting, and real-world regression problems with prior misspecification.
- The authors conduct extensive ablation studies and hyperparameter tuning, demonstrating the robustness and effectiveness of the proposed method.

weaknesses:
- Limited novelty: The core idea of using an alternative divergence measure (Rényi divergence) to mitigate the effects of prior misspecification is not entirely novel. Previous works in the domain of robust variational inference have explored the use of other divergences, such as α-divergences and f-divergences, to address similar issues. The authors could provide a more comprehensive discussion of how their work relates to and differentiates from these earlier efforts.
- Hyperparameter sensitivity: The choice of the hyperparameter α plays a crucial role in the performance of the proposed method. While the authors provide some guidelines for tuning α, a more comprehensive analysis of the sensitivity of the method to different values of α and strategies for automatic tuning could further enhance the practical utility of the proposed framework.

questions:
- The authors mention that the proposed framework can be further extended to improve the robustness of contextual inferences, such as prompt design in large language models. Could the authors provide more details or insights into how their method could be adapted or applied to such tasks?

- The parameter coupling between the posterior and prior in neural processes is by design to share parameters, but it's not a hard constraint. Isn't a simple baseline approach to mitigate prior misspecification to separately parameterize the posterior and prior models? Could the authors provide a comparison with this simple baseline to better demonstrate the advantages of their proposed method?

Overall, naively combining existing ideas may not constitute a sufficient contribution for a top-tier conference like ICLR, which expects a higher level of novelty and significance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732254684602

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary

comment:
We would like to thank all the reviewers for their constructive feedback and the time and effort applied in reviewing our manuscript. We provide individual responses to each review, but we also address some of the main common points here.

# Motivation of RNPs and analytical analysis of prior misspecifications.

 We analytically compared the gradients of our RNP objective with the gradients using the VI objective to show how $\alpha$ would affect the posterior parameter updates.
    
 # Sensitivity analysis and tuning of the hyperparameter $\alpha$.

 We would like to emphasize that we have carried out a sensitivity analysis on the hyperparameter $\alpha$ in the main paper. The main conclusion across all datasets and models is that $\alpha=0.7$ provides significant performance improvements over the competing approaches. Hence, we recommend this value as default to practitioners and researchers. (See also the additional clarification in section 7.1.) Nevertheless, we recognize that in other scenarios such as very different datasets and/or models, this default value may not work as well as in our experiments. For this purpose, we have found that cross-validation is an effective tool for finding near-optimal values. Since cross-validation is computationally expensive, we have also suggested heuristics to set $\alpha$. See below for details. 
    
 # MC sampling efficiency.

 We would like to clarify that our RNP framework *consumes the same resource as the standard NPs* since they usually require multiple samples to better estimate the likelihood model as well. Our computational overhead comes from non-analytical solutions for the divergence, whereas our ablations in Fig 5 showed that it takes as few as eight samples during training to gain better estimates, and one can choose to increase the number of samples during inference for better predictive performance or to reduce it for scalability and real-time applications. 

  # Changes in the paper.

 We have crystallized our contributions in section 1. We have improved clarifications on prior misspecifications and the reparameterization tricks used for the VI-based objectives in section 3.1. We have revised the formulation of the ML-objective and in section 3.2 and the gradient update in section 4. More experimental setups are elaborated on the prior misspecification datasets in section 7.2. We have fixed typos throughout the paper and polished the plots with better resolutions and bigger legends.

 # Adding new results.

We have added two tables in the rebuttal to compare our RNP with a baseline model suggested by Reviewer T3rx (Supp Table 4)  and to compare MC efficiency suggested by Reviewer eR5B via wall clock time comparison between RNP and standard Neural processes (Supp Table 5).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The  paper introduces Neural Renyi processes, where the neural processes objectives are revisited with Renyi divergences. Authors claim that the Renyi losses overcome prior misspecification, i,e when the target distribution does not belong to the parametrized family , i,e an identifiability problem. 

Reviewers raised several concerns, one regarding interpretability (Reviewer eR5B), and another one on the link between the prior mispecification and Renyi divergence (Reviewer 6B6t), and lack of novelty (Reviewer T3rx). Reviewer JR2F provided a detailed review that has improved the paper greatly. 

Given the large variance in the scores AC and reviewers discussed the paper, and I read the paper in details. I am not concerned with novelty or with the interpretability of the method. One thing I am still missing as I read the paper is how the prior misspecification is linked  to the use of Renyi divergence. 

The authors should provide a theoretical insight about this claim their work falls within the framework of [a], as it amounts to an importance sampling , this is largely studied in this paper theoretically.  The other aspect is that Renyi divergence minimization is rather a great alternative to KL when the target is heavy tailed or does not have defined moments see [b]. A rigorous proof for Renyi minimization alleviating the misspecification prior would strengthen the paper, or in terms of better targeting the tails as in [b] would bridge more the algorithmic part and the motivation of the work. 
 

[a] https://www.jmlr.org/papers/volume24/22-1160/22-1160.pdf

[b] https://hal.science/hal-04506761/document