Title: Learning to Generalize: An Information Perspective on Neural Processes
Venue: NeurIPS 2025
Accepted: yes
Keywords: Neural Processes

Information-Theoretic

Generalization
Abstract: Neural Processes (NPs) combine the adaptability of neural networks with the efficiency of meta-learning, offering a powerful framework for modeling stochastic processes. However, existing methods focus on empirical performance while lacking a rigorous theoretical understanding of generalization. To address this, we propose an information-theoretic framework to analyze the generalization bounds of NPs, introducing dynamical stability regularization to minimize sharpness and improve optimization dynamics. Additionally, we show how noise-injected parameter updates complement this regularization. The proposed approach, applicable to a wide range of NP models, is validated through experiments on classic benchmarks, including 1D regression, image completion, Bayesian optimization, and contextual bandits. The results demonstrate tighter generalization bounds and superior predictive performance, establishing a principled foundation for advancing generalizable NP models.

Learning to Generalize: An Information Perspective on Neural Processes

Abstract

Neural Processes (NPs) combine the adaptability of neural networks with the efficiency of meta-learning, offering a powerful framework for modeling stochastic processes. However, existing methods focus on empirical performance while lacking a rigorous theoretical understanding of generalization. To address this, we propose an information-theoretic framework to analyze the generalization bounds of NPs, introducing dynamical stability regularization to minimize sharpness and improve optimization dynamics. Additionally, we show how noise-injected parameter updates complement this regularization. The proposed approach, applicable to a wide range of NP models, is validated through experiments on classic benchmarks, including 1D regression, image completion, Bayesian optimization, and contextual bandits. The results demonstrate tighter generalization bounds and superior predictive performance, establishing a principled foundation for advancing generalizable NP models.

1 Introduction

Gaussian processes (GPs) are widely recognized as a robust framework for modeling distributions over functions [37]. Their appeal lies in the consistent probabilistic reasoning enabled by Bayesian inference, which facilitates data-efficient modeling. Despite their strengths, GPs are unsuitable for certain problems. For instance, a function exhibiting a single, unknown discontinuity is a classic example of a distribution that GPs fail to represent [34].

To address such limitations, researchers have turned to neural network-based generative models. Notable advancements in this area include meta-learning techniques like Neural Processes (NPs) [14, 15] and models based on variational autoencoders (VAEs) [33, 11]. These approaches leverage extensive small dataset training to transfer knowledge effectively across tasks during prediction. Neural networks (NNs) offer additional advantages by offloading computational intensity to the training phase, simplifying predictions, and freeing the model from Gaussianity constraints.

Building upon the foundational work of Conditional Neural Processes (CNPs) and Neural Processes (NPs) [14, 15], numerous studies have enhanced NPs by integrating advanced mechanisms. Attentive Neural Processes (ANPs) [27] incorporated attention mechanisms to better model long-range dependencies, while Transformer Neural Processes (TNPs) [36] leveraged self-attention for improved scalability. Convolutional Neural Processes (ConvCNPs) [10] adapted convolutional architectures to excel in spatial data tasks, and Neural Diffusion Processes (NDPs) [7] introduced diffusion mechanisms to enhance uncertainty estimation. NPCL [22] extended NPs to continual learning, enabling

sequential task adaptation, and multimodal approaches like [25] generalized NPs for uncertainty estimation across multiple modalities.

While these advancements significantly enhance the experimental performance of NPs, they primarily focus on task-specific improvements through architectural innovations and meta-learning strategies [20, 16]. However, the generalization capability of NPs-a critical hallmark of their design-has received limited theoretical exploration. Existing works prioritize performance optimization, often neglecting the theoretical interpretability of NPs. For instance, many methods emphasize improving inductive biases or increasing model complexity without addressing the theoretical underpinnings of NPs' generalization ability [17, 10]. This gap underscores the need for a formal theoretical foundation to better understand and generalize NPs across broader domains and tasks, paving the way for more principled and interpretable neural process models [44].

Information theory provides a powerful framework for analyzing and quantifying uncertainty, offering rigorous tools to evaluate the generalization of models [18, 35]. This paper introduces an information-theoretic framework to analyze and improve the generalization bounds of Neural Processes (NPs). Building on this analysis, we propose a novel approach that incorporates dynamical stability regularization, which explicitly minimizes sharpness through the trace and Frobenius norm of the Hessian matrix. Additionally, we demonstrate how noise-injected parameter updates complement this regularization by smoothing the optimization trajectory and improving gradient coherence. These insights are broadly applicable across NP variants and validated through extensive experiments on benchmarks such as 1D regression, image completion, Bayesian optimization, and contextual bandits.

The main contributions of this paper are as follows:

We conduct an information-theoretic analysis of NP generalization bounds, providing a rigorous perspective on their generalization capabilities.

We propose a dynamical stability regularization framework, complemented by noise-injected parameter updates, to enhance generalization by minimizing sharpness and improving optimization dynamics.

Extensive experiments on classic NP tasks validate our approach, demonstrating tighter generalization bounds and superior performance compared to existing methods.

2 Related Work

In this section, two relevant areas are briefly reviewed: neural processes and information-theoretic learning.

Neural Processes Neural Processes (NPs) are probabilistic models designed to meta-learn distributions over functions, offering a data-driven alternative to Gaussian Processes (GPs) [14, 15]. Unlike GPs, which rely on manually specified priors, NPs employ an encoder-decoder architecture to learn a family of functions, efficiently capturing uncertainty using neural networks. Conditional Neural Processes (CNPs) [14], the earliest variant, introduced a deterministic encoder-decoder framework but assumed independence among predictive outputs. Latent Neural Processes (NPs) [15] addressed this by introducing a global latent variable for uncertainty modeling. Subsequent works have enhanced expressiveness, scalability, and uncertainty modeling. Extensions of NPs have addressed specific limitations and broadened their applicability. Attentive Neural Processes (ANPs) [27] incorporated attention mechanisms to better model long-range dependencies and heteroscedastic uncertainty. Convolutional Neural Processes (ConvCNPs) [10] introduced translation-equivariant architectures, excelling in spatial and structured data tasks. Transformer Neural Processes (TNPs) [36] leveraged self-attention for scalability and sequence modeling, while Neural Diffusion Processes (NDPs) [7] used diffusion mechanisms to improve robustness and uncertainty estimation. Neural Processes for Uncertainty-Aware Continual Learning (NPCL) [22] extended NPs to sequential task adaptation without catastrophic forgetting. Additionally, multimodal extensions [25] generalized NPs to handle multi-sensor fusion and uncertainty estimation. These advancements highlight the versatility of NPs and their adaptability across diverse applications.

Information-Theoretic Learning Information-theoretic learning (ITL) leverages principles from information theory to analyze and optimize learning algorithms. Unlike traditional approaches based on measures like VC-dimension [39] or uniform stability [4], ITL characterizes learning using mutual information and related quantities [50, 38]. By framing generalization error as the mutual information between inputs and outputs, ITL captures dependencies among data distribution, hypothesis space, and learning algorithms, addressing challenges in uncertainty quantification, robustness, and data

efficiency. This framework provides valuable insights into the generalization capabilities of modern machine learning models. Recent advances have expanded ITL's application to generalization analysis and enhancement. Harutyunyan et al. [18] proposed improved generalization bounds by focusing on the mutual information between predictions and the training set, offering practical estimates of the generalization gap in deep learning. Neu et al. [35] extended ITL to stochastic gradient descent (SGD), deriving tighter bounds based on local gradient statistics and sensitivity along the optimization path. These advances demonstrate ITL's utility in understanding complex learning dynamics and improving theoretical guarantees for generalization.

3 Preliminaries

This section introduces the foundational concepts and mathematical tools necessary for understanding the methods and analyses presented in this paper, including information theory, neural processes, and dynamical stability.

Information Theory Let P X denote the marginal distribution of the random variable X . For the Markov chain X → Y , the conditional distribution (or Markov transition kernel) is denoted by P Y | X , and the notation X ⊥ Y indicates independence between X and Y [6]. The cumulant generating function (CGF) of a random variable X is defined as ψ X ( λ ) ≜ log E [ e λ ( X -E [ X ]) ] , where λ is a real number. A random variable X is σ -subgaussian if its CGF satisfies ψ X ( λ ) ≤ λ 2 σ 2 2 for all λ ∈ R [43].

The mutual information between X and Y is defined as I ( X ; Y ) ≜ KL ( P X,Y ∥ P X P Y ) , where P X,Y is the joint distribution of X and Y , and P X P Y is the product of their marginals [6]. The disintegrated mutual information between X and Y given U is I U ( X ; Y ) ≜ KL ( P X,Y | U ∥ P X | U P Y | U ) , where P X,Y | U is the conditional joint distribution given U . The conditional mutual information is then defined as I ( X ; Y | U ) ≜ E U [ I U ( X ; Y )] , the expectation of the disintegrated mutual information over the distribution of U [1].

Neural Processes Neural Processes (NPs) model the conditional predictive distribution of target values y T at target points X T based on a context set D C , expressed as P ( y T | X T , D C ) [15]. For deterministic NPs (CNPs), the conditional distribution is simplified as P ( y T | X T , D C ) = P ( y T | X T , r C ) , where r C is an aggregated feature of D C [14]. Probabilistic NPs introduce a latent variable z to capture uncertainty, modeling P ( y T | X T , D C ) = ∫ P θ ( y T | X T , z ) P θ ( z |D C ) d z . Training maximizes the evidence lower bound (ELBO): E z ∼ P θ ( z |D C ) [log P ( y | X )] -KL [ P θ ( z | X , D C ) || P θ ( z |D C )] [15]. During meta-training, NPs learn from tasks sampled from an environment τ , a probability measure over task distributions µ , where each task involves dividing data into context and target sets [20]. A meta-dataset consists of m datasets D 1: m = ( D 1 , . . . , D m ) , each sampled independently from µ n,τ , the mixture distribution induced by τ [9]. The meta learner A outputs a meta-parameter θ = A ( D 1: m ) ∼ P θ |D 1: m , representing shared knowledge across tasks [3]. The meta risk is defined as R τ ( θ ) ≜ E D∼ µ n,τ ,µ ∼ τ [ E D C [ R µ ( θ )]] , where R µ ( θ ) ≜ -E ( x,y ) ∼ µ log P ( y | x, D C ) [3]. The empirical meta risk is given by R D 1: m ( θ ) ≜ 1 m ∑ m j =1 R D j ( θ ) , with R D j ( θ ) ≜ -1 n ∑ n i =1 log P ( y i | x i , D C j ) . The meta generalization error is then defined as gen NPs meta ( τ, A ) ≜ E θ, D 1: m [ R τ ( θ ) -R D 1: m ( θ )] , which quantifies the difference between true and empirical meta risks [3]. By learning a meta-parameter θ shared across tasks, NPs efficiently adapt to new tasks, leveraging shared information in the environment.

Dynamical Stability Dynamical stability refers to the robustness of the stochastic gradient descent (SGD) optimization process to small perturbations in the parameter space and has been shown to play a critical role in both optimization and generalization [49, 42]. Consider the parameter update rule θ s +1 = θ s -η ∇ R s ( θ s ) , where η is the learning rate, and ∇ R s ( θ s ) is the stochastic gradient. The sensitivity of the optimization dynamics can be quantified by the spectral norm ∥ J s ∥ 2 of the Jacobian matrix J s = ∂θ s +1 ∂θ s , which governs how small perturbations δ s at step s propagate through subsequent iterations [48, 40, 47]. When ∥ J s ∥ 2 < 1 , perturbations decay over time, leading to stable training. Importantly, this stability imposes an implicit regularization effect, as SGD inherently biases the optimization process towards flatter minima with lower curvature, measured by the Hessian H = ∇ 2 R ( θ ) [21, 26]. Flatter minima are associated with better generalization due to their robustness to noise and parameter variations [19, 8]. Moreover, the Lyapunov exponents, which quantify the exponential rates of divergence or convergence of nearby trajectories in parameter space, provide a formal link between stability and generalization. Negative Lyapunov exponents imply stable dynamics and a tendency towards generalizable solutions [42, 49]. This perspective highlights the central role of dynamical stability in understanding the implicit biases of SGD and its impact on both the optimization landscape and model generalization [47, 41].

4 Methodology

This section investigates the generalization properties of Neural Processes (NPs) by integrating information-theoretic principles with optimization dynamics. First, we quantify the generalization error of NPs using mutual information (MI), capturing uncertainties from data and task distributions. Next, we introduce a risk-aware dynamical stability regularization term R dyn, addressing sharpness and curvature to improve generalization. Finally, we propose an optimization-aware noise injection strategy to enhance stability and guide parameter updates toward flatter minima.

4.1 Quantifying Generalization with Information Theory

Understanding the generalization capabilities of Neural Processes (NPs) requires quantifying the uncertainties inherent in the learning process. By leveraging mutual information (MI), the generalization error can be decomposed into components reflecting different sources of uncertainty, offering insights into the behavior of NPs across diverse tasks. While MI provides a rigorous framework to analyze generalization, standard approaches often neglect the impact of model complexity, which is particularly critical in the hierarchical nature of meta-learning. This section lays the foundation for incorporating complexity constraints in future analysis.

To analyze generalization bounds, we consider the relationship between true risk and empirical risk un-

0.6

0.6

0.4

0.4

Generalization Error

0.2

0.2

0.0

0.0

0.2

0.2

0.4

0.4

-2e4

0

2e4

4e4

6e4

8e4

10e4

12e4

14e4

der a given task distribution. Let µ represent an unknown distribution on the instance space X × Y . A meta-dataset consists of m datasets, denoted as D 1: m = ( D 1 , . . . , D m ) , where each dataset D j consists of n independent samples drawn from µ . The full dataset is thus D = D 1: m = {D j } j m =1 . For a hypothesis θ , the true risk is defined as R µ ( θ ) ≜ E X × Y ∼ µ [ ℓ ( θ, X × Y )] , and the empirical risk as R D ( θ ) ≜ 1 mn ∑ m j =1 ∑ n i =1 ℓ ( θ, x i,j , y i,j ) , where ℓ : Θ × X × Y → R is the loss function. A learning algorithm A maps the dataset D to a randomized hypothesis θ = A ( D ) ∼ P θ |D , and its generalization error is defined as gen ( µ, A ) ≜ E θ, D [ R µ ( θ ) -R D ( θ )] .

Theorem 4.1. If the loss ℓ ( θ, X × Y ) is σ -subgaussian for each θ ∈ Θ with respect to X × Y ∼ µ , the generalization error of a learning algorithm A satisfies the bound | gen ( µ, A ) | ≤ √ 2 σ 2 mn I ( θ ; D ) , where I ( θ ; D ) is the mutual information between the dataset D and the hypothesis θ . For meta-learning tasks, where µ ∼ τ , the meta-generalization error satisfies:

The mutual information I ( θ ; D 1: m ) can be computed based on the joint distribution P ( θ, D 1: m ) and the marginal P ( θ ) ; detailed derivations are provided in the appendix. This bound indicates that minimizing the dependence of θ on the data (via I ( θ ; D 1: m ) ) improves the generalization performance.

Although the MI framework elegantly quantifies generalization, it does not explicitly account for the hypothesis space complexity or the solution sharpness. For NPs, these factors are especially significant as a result of their hierarchical structure. The hypothesis space in NPs, determined by the meta-parameters θ , encodes both task-level priors and task-specific adaptations. This space is inherently high-dimensional, making it prone to overfitting, particularly when the meta-dataset D 1: m is small or lacks diversity.

θ 1

···

θ S

𝜉𝜉

1

···

𝜉𝜉

𝑆𝑆

θ 0

Optimization-

Aware Noise

Strategy

𝐁𝐁

𝟏𝟏

:

𝐦𝐦

𝟏𝟏

···

𝐁𝐁

𝟏𝟏

:

𝐦𝐦

𝐒𝐒

Context Points

Target Points

Task

T

𝐷𝐷1

:

𝑚𝑚

Risk-Aware

Dynamical Stability

Regularization

𝑅𝑅

𝑑𝑑𝑑𝑑𝑛𝑛

1

···

𝑅𝑅

𝑑𝑑𝑑𝑑𝑛𝑛

𝑆𝑆

The sharpness of the learned solution is related to the curvature of the loss surface, quantified by the Hessian H = ∇ 2 R ( θ ) . A high Hessian

trace implies sensitivity to parameter perturbations, adversely affecting generalization, especially across diverse task distributions. As illustrated in Figure 1, the generalization error of NPs deteriorates with increasing Hessian trace.

Based on the above analysis, we focus on improving NPs' generalization from both risk-aware dynamical stability regularization and optimization-aware noise injection learning strategy, the whole learning framework is given in Figure 2.

4.2 Risk-Aware Dynamical Stability Regularization

Inspired by previous works that demonstrate the critical role of sharpness and curvature in optimization and generalization [26, 23, 45], we introduce a dynamical stability regularization (DSR) term R dyn that incorporates model complexity constraints into the generalization bound. Specifically, sharpness of the solution has been linked to poor generalization, motivating the need to explicitly regularize the curvature of the loss landscape [19, 21, 51]. Moreover, recent studies have explored how Lyapunov stability and noise in optimization dynamics can implicitly bias solutions towards flat minima, further supporting the necessity of stability-based regularization [40, 49].

Definition 4.2. The dynamical stability regularization term R dyn quantifies the complexity of the hypothesis space by leveraging properties of the Hessian matrix H , which represents the second-order derivatives of the loss function with respect to the parameters θ :

where Tr ( H ) (the trace of the Hessian) measures the solution sharpness by summing the eigenvalues of H , and ∥ H ∥ F (the Frobenius norm) captures the overall curvature of the loss landscape. The hyperparameters λ 1 and λ 2 control the relative importance of these two components, with empirical values typically set as λ 1 ∈ [0 . 01 , 0 . 1] and λ 2 ∈ [0 . 001 , 0 . 01] to maintain an appropriate balance between the two terms. By penalizing sharpness and curvature, R dyn aligns the theoretical bounds with practical observations in optimization.

Theorem 4.3. By incorporating R dyn into the information-theoretic framework, the refined metageneralization error bound for Neural Processes (NPs) is given by:

where I ( θ ; D 1: m ) is the mutual information between the meta-parameter θ and the dataset D 1: m , R dyn is the dynamical stability regularization term, and α > 0 is a scaling factor that controls the influence of R dyn. The proof of this theorem is provided in the Appendix.

Including R dyn in the generalization bound has notable effects. Penalizing high Tr ( H ) encourages the model to select flatter minima, which are associated with better generalization. Additionally, constraining the Frobenius norm ∥ H ∥ F prevents overfitting to high-curvature regions. This regularization complements the implicit bias of optimization algorithms, such as gradient descent, which naturally steer solutions toward regions of lower sharpness.

The role of R dyn depends on the training regime. When the number of tasks m is large ( m → ∞ ) with fixed samples per task n , the mutual information term I ( θ ; D 1: m ) dominates the bound, and R dyn constrains the sharpness across tasks. Conversely, when n grows ( n →∞ ) while m remains fixed, the generalization error decreases naturally as task-level uncertainty reduces, making R dyn less critical.

From a practical perspective, incorporating R dyn into the generalization bound provides actionable insights for optimizing Neural Processes. Maximizing R dyn during training improves generalization, and tuning the hyperparameters λ 1 and λ 2 balances sharpness reduction and complexity control. Ultimately, R dyn bridges the gap between theoretical generalization bounds and practical challenges, leading to tighter bounds and a deeper understanding of the factors influencing generalization in NPs.

4.3 Optimization-Aware Noise Injection Learning Strategy

Motivated by the well-established relationship between noise injection and generalization [19, 41, 46], we propose an optimization-aware noise injection strategy tailored for meta-learning tasks. Noise has been shown to play a critical role in escaping sharp minima [26], smoothing the loss landscape [21], and improving generalization through implicit regularization [47]. Furthermore, recent studies highlight the importance of integrating noise injection into gradient-based optimization to ensure stability and guide parameter trajectories towards flatter regions of the loss landscape [24, 49].

Building upon these insights, we explicitly incorporate noise into the parameter update rule to enhance both the stability and generalization of meta-parameter optimization. The overall training process of NPs, incorporating noise injection and dynamical stability regularization, is illustrated in Figure 2. The pseudocode for the entire algorithm can be found in the Appendix.

To explicitly incorporate the effect of noise into the learning dynamics, we modify the standard gradient-based parameter update rule by introducing an isotropic Gaussian noise term ξ s at each iteration s . The parameter update rule for meta-parameters θ at iteration s is formally given by:

where ˜ R D T i ( θ s -1 ) + R dyn is the empirical risk computed on the target set D T i of task i , η s is the learning rate at iteration s , and ξ s ∼ N (0 , σ 2 s I k ) is an isotropic Gaussian noise term with variance σ 2 s and dimensionality k . This update rule consists of two key components: the gradient descent step, which minimizes the empirical risk, and the noise injection step, which perturbs the parameter updates. The variance σ 2 s can be adjusted over iterations to control the strength of the perturbation.

The introduction of noise into the parameter updates serves two primary purposes. First, it provides implicit regularization by encouraging the model to explore flatter regions of the loss landscape and favor minima with lower sharpness. This aligns with the objective of maximizing R dyn, as described in Section 4. Second, noise facilitates escaping sharp minima, which are common in high-dimensional parameter spaces and often lead to poor generalization. By perturbing the trajectory of the updates, noise helps the optimization process avoid these undesirable regions.

To achieve the desired regularization effect without destabilizing the optimization, the variance σ 2 s of the noise is carefully scaled with respect to the learning rate η s . A common choice for this scaling is σ 2 s = η s /γ , where γ > 0 is a scaling factor that controls the relative strength of the noise. This formulation ensures that the noise magnitude decreases as the learning rate decreases, allowing the optimization to stabilize in later iterations.

The noise-injected parameter update rule in Eq. 4 directly influences the dynamical stability regularization term R dyn by affecting the Hessian properties of the loss landscape. Specifically, the perturbation introduced by ξ s reduces the likelihood of converging to regions with high Tr ( H ) , thus implicitly maximizing the sharpness of the solution. Furthermore, noise smooths the optimization trajectory, reducing the overall curvature of the solution as measured by ∥ H ∥ F . Together, these effects demonstrate the critical role of noise injection in improving both the stability and generalization performance of meta-parameter optimization. While introducing noise and regularization incurs additional computational overhead, the performance gains justify these costs (detailed analysis in Appendix).

5 Deeper Analysis

In this section, we establish a theoretical foundation for understanding how noise and dynamical stability regularization affect the generalization performance of NPs. By analyzing the interplay between noise, sharpness, and mutual information, we demonstrate their influence on generalization bounds. Additionally, we show how noise injection mitigates gradient incoherence, a key challenge in meta-learning, to improve stability and generalization.

5.1 Theoretical Analysis

We analyze how noise-injected parameter updates affect the dynamical stability of the optimization process and its impact on generalization performance. Specifically, we focus on their effects on sharpness, curvature (captured by the dynamical stability regularization term R dyn), and mutual information I ( θ ; D 1: m ) . This analysis highlights the role of noise in reducing both R dyn and I , thereby improving the generalization of NPs.

To begin, we linearize the parameter updates around the optimal solution θ ∗ , which minimizes the true risk R µ ( θ ) . Let ˜ θ s = θ s -θ ∗ represent the deviation from θ ∗ at iteration s . The linearized noise-injected parameter update rule is:

where H = ∇ 2 [ ˜ R D T i + R dyn ]( θ ∗ ) is the Hessian of the empirical risk, and ξ s ∼ N (0 , σ 2 s I k ) is isotropic Gaussian noise. This form separates the contributions of optimization dynamics ( I -η s H ) and noise ξ s to the evolution of ˜ θ s .

Noise affects sharpness, measured by the trace of the Hessian Tr ( H ) , by perturbing the optimization trajectory and encouraging exploration of flatter regions. This perturbation has a dual effect on the loss landscape: it locally reduces the curvature around specific minima while simultaneously increasing the system's overall stability by promoting robust, flat solutions. Consequently, the effective Hessian trace in the presence of noise satisfies Tr ( H noise ) ≥ Tr ( H ) · (1 + η s σ 2 s ) , where a higher variance of noise σ 2 s results in a greater enhancement of dynamic stability. Similarly, the Frobenius norm ∥ H ∥ F , which provides a comprehensive measure of the curvature of the loss landscape, satisfies ∥ H noise ∥ F ≥ ∥ H ∥ F · √ 1 + η s σ 2 s , indicating that noise enhances the system's ability to maintain stability in the face of parameter perturbations. These results suggest that noise injection encourages the optimization process to explore flatter regions of the loss landscape, develop higher dynamic stability, and improve generalization performance by enhancing robustness to parameter perturbations.

By enhancing both stability metrics, noise effectively amplifies the dynamical stability regularization term R dyn, which appears in the denominator of our generalization bound. Specifically, substituting the enhanced trace and Frobenius norm into the definition of R dyn yields:

This increase in R dyn directly tightens the generalization bound by increasing the denominator term 1 + α · R dyn, resulting in improved generalization capabilities for Neural Processes.

As the noise variance σ 2 s decreases over iterations ( σ 2 s = η s /γ ), the optimization stabilizes near a flat minimum, with the stability benefits retained as enhanced R dyn. Noise also reduces the mutual information I ( θ ; D 1: m ) , which quantifies the dependency between model parameters θ and training data D 1: m . High mutual information indicates potential overfitting, as parameters become overly dependent on specific data. The reduction is given by:

This ensures I noise ( θ ; D 1: m ) remains non-negative and significantly lower than the original mutual information. Excessive noise or high learning rates, however, may destabilize optimization, limiting these benefits. By jointly reducing I ( θ ; D 1: m ) and amplifying R dyn through increased Tr ( H ) and ∥ H ∥ F , noise-injected updates tighten the generalization bound:

where the second inequality follows from I noise ( θ ; D 1: m ) ≤ I ( θ ; D 1: m ) and R dyn, noise ≥ R dyn . This interplay between noise, mutual information, and dynamical stability is crucial for achieving tighter generalization bounds and improving meta-learning performance.

5.2 Gradient Incoherence

Gradient incoherence, caused by inconsistencies between gradients computed on the context set ( D C i ) and the target set ( D T i ), affects the stability and generalization of Neural Processes (NPs). This issue can lead to suboptimal updates and hinder learning. Noise-injected parameter updates mitigate this problem by smoothing the loss landscape and reducing gradient mismatches.

Definition 5.1. The gradient incoherence for a task i at iteration s is defined as the ℓ 2 -norm of the difference between the gradients on the full dataset D i and the target set D T i :

where ∇ ˜ R D i ( θ s -1 ) + R dyn and ∇ ˜ R D T i ( θ s -1 ) + R dyn are gradients of the empirical risk over D i and D T i , respectively. The overall gradient incoherence across tasks and iterations is given by GI = 1 mS ∑ m i =1 ∑ S s =1 ϵ θ i,s , where m is the number of tasks and S the number of iterations. Minimizing GI ensures more consistent updates, improving generalization.

Noise reduces gradient incoherence by perturbing the optimization trajectory. The noise ξ s introduced during updates disrupts gradient alignment, exponentially decreasing incoherence as noise variance σ 2 s increases. Formally, the reduction can be expressed as ϵ θ i,s ≤ ϵ θ i,s (0) · exp( -η 2 s σ 2 s γ ) , where ϵ θ i,s (0) is the initial incoherence. Larger noise promotes exploration and reduces mismatches early in training, while smaller noise stabilizes updates in later stages. This reduction stabilizes parameter

Gen-NP

NP

Original

Task

CNP

Gen-CNP

NP

Gen-NP

ANP

(a) 1D Regression Results on NP

(b) Image Completion Results on CelebA Datasets

context 30

target 10

context 30

target 10

2.0

1.5

1.5

1.0

0.5

0.5

0.0

0.0

-2.0-1.5-1.00.50.0

0.5

1.0

1.5

-2.0-1.5-1.00.50.0

0.5

1.0

1.5

updates, prevents oscillations caused by inconsistent gradients, and improves alignment with true risk minimization. Additionally, noise-induced coherence complements the regularization effect of R dyn by further reducing the sharpness and curvature of the loss landscape.

The effectiveness of noise depends on scaling its variance σ 2 s with the learning rate η s . Setting σ 2 s = η s /γ balances noise magnitude across iterations, ensuring exploration in early training and stability later. Empirical results in Section 6 confirm that noise-injected updates significantly reduce GI, leading to improved stability and generalization across tasks.

6 Experiments

The proposed Generalization Neural Processes (Gen-NPs) are evaluated and compared with other methods in the NP family on tasks such as regression, image completion, Bayesian optimization, and contextual bandits. These tasks are widely used to benchmark NP-based models, as demonstrated in prior works. The comparison includes Conditional Neural Processes (CNPs) [14], Neural Processes (NPs) [15], Attentive Neural Processes (ANPs) [27], Bootstrapping Neural Processes (BNPs) [30], and Transformer Neural Processes (TNPs) [36]. In addition to standard task performance, we analyze the gradient incoherence (GI) introduced in Section 5.2, highlighting how Gen-NPs improve optimization dynamics and generalization capabilities compared to other methods. The experimental setup is consistent across all methods, and the implementation leverages the official codebase of TNPs. Due to space limitations, extensive experimental results, including detailed images and tables, are provided in Appendix C.

6.1 1-D Regression

We evaluate Gen-NPs on a 1-D regression task, training models on RBF kernel functions sampled from a Gaussian Process (GP) and testing on unseen RBF, Matérn 5/2, and Periodic kernels. Metrics include log-likelihood (LL) for predictive accuracy and uncertainty, and gradient incoherence (GI) for optimization stability. Results are averaged over five random seeds with standard deviations reported. 1-D regression results on NP model are visually illustrated in Fig. 3(a), where Gen-NP demonstrate a superior ability to capture the underlying character-

istics of the data compared to baseline models. As summarized in Table 1, Gen-NPs consistently outperform baseline models across all kernels and metrics. For example, Gen-NPs achieve higher LL values on Matérn 5/2 and Periodic kernels, while also demonstrating reduced GI, indicating improved optimization dynamics and generalization. These results validate the effectiveness of the proposed general recipe. Additional figures and tables, including detailed results for MAE, RMSE, and variability across random seeds, are provided in Appendix C.2.

6.2 Image Completion

We evaluate Gen-NPs on image completion tasks using CelebA [32] and EMNIST [5], formulated as a 2-D regression problem where pixel coordinates are inputs and intensities are outputs [15]. CelebA is downsampled to 32 × 32 , while EMNIST uses 10 training classes (0-9) and evaluates generalization on unseen classes (10-46). Figure 3(b) shows partial results on CelebA, where

Gen-NPs demonstrate superior performance in image completion. As summarized in Table 2, GenNPs consistently outperform baseline models across both datasets, achieving higher LL values and reduced GI, demonstrating improved predictive accuracy and optimization stability. Visualizations in Appendix C.3 further highlight that Gen-NPs produce clearer and more precise image reconstructions, effectively enhancing generalization and accuracy for image completion tasks.

6.3 Bayesian Optimization

We evaluate the effectiveness of Gen-NPs in Bayesian optimization (BO) tasks, where the goal is to maximize a black-box function f ( x ) accessible only through evaluations without gradient information [12, 28, 2, 29]. The experimental results on 2D Dropwave and 3D Ackley are shown in Figure 4a, where Gen-NPs achieve lower regret compared to baseline methods. Results, detailed in Appendix C.4, demonstrate that Gen-NPs consistently outperform baseline models across all dimensions, achieving lower regret and faster convergence. These findings validate the proposed method's ability to enhance generalization and optimization performance in BO tasks [13].

6.4 Ablation Study

We conducted ablation studies to validate the effectiveness of the proposed Risk-Aware Dynamical Stability Regularization (DSR) and Optimization-Aware Noise Injection Learning Strategy (NILS) on 1D regression task. Table 9 presents the results of the original method, Gen-NPs with only DSR, Gen-NPs with only NILS, and the full Gen-NPs with both modules included.

As shown in Table 9 and Appendix, these results highlight the significance of incorporating DSR for improving dynamical stability and NILS for robust optimization. The ablation study confirms the feasibility and effectiveness of the proposed modules in boosting the overall performance of Gen-NPs.

6.5 Comparison with Stability Neural Processes

We compare the stability-based generalization error (SGE) [4, 31] with our proposed Gen-NPs on 1D regression task, emphasizing the differences in their noise introduction mechanisms and evaluation methods. SGE quantifies stability as the difference between test error and training error to measure generalization. Gaussian noise (mean = 0, variance = 1) is added to 5% and 10% of the training data to assess robustness under noisy conditions. In contrast, Gen-NPs introduces noise during the parameter update process, focusing on the mutual information between model parameters and the training data to directly capture generalization behavior. Experiments were conducted on a 1D regression task with data generated using an RBF kernel, comparing models under original, 5% noise, and 10% noise conditions. Results, shown in Appendix C.6, demonstrate that Gen-NPs consistently achieves tighter bounds and higher log-likelihood (LL) values even in noisy settings. This highlights the robustness and effectiveness of Gen-NPs in capturing model generalization and stability under various conditions.

Dropwave (2D)

Ackley (3D)

1.7

5.5

1.6

5.0

4.5

1.5

Regret

4.0

1.4

3.5

1.3

3.0

1.2

2.5

0

20

40

Iterations

60

80

100

0

20

40

Iterations

60

100

NP

Gen-NP

ANP

CNP

Gen-CNP

(a) Results of Bayesian Optimization

CNP

BNP

ANP

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

Original (Original)

Original (Gen-*)

Noise (+5%) (Original)

Noise (+5%) (Gen-*)

Noise (+10%) (Original)

Noise (+10%) (Gen-*)

(b) Comparison of Log-Likelihood Across Different Noise Levels

7 Conclusion

This paper proposes an information-theoretic framework to enhance the generalization capabilities of neural processes (NPs) by addressing both parameter optimization and regularization. Specifically, noise injection during parameter updates captures the mutual information between model parameters and training data, providing a robust optimization strategy. In parallel, dynamical stability regularization mitigates overfitting and improves the optimization trajectory, collectively leading to better generalization properties. However, the framework currently focuses on supervised learning, and future work will explore its extension to reinforcement learning and large-scale datasets.

References

Robert B. Ash. Information Theory . Courier Corporation, 2012.

Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G. Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian optimization. Advances in Neural Information Processing Systems (NeurIPS) , 33:21524-21538, 2020.

Jonathan Baxter. A model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR) , 12:149-198, 2000.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In Proceedings of the International Joint Conference on Neural Networks (IJCNN 2017) , pages 2921-2926. IEEE, 2017.

Thomas M. Cover. Elements of Information Theory . John Wiley & Sons, 1999.

Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. In Proceedings of the International Conference on Machine Learning (ICML 2023) , pages 8990-9012. PMLR, 2023.

Gintare Karolina Dziugaite and Daniel Roy. Entropy-sgd optimizes the prior of a pac-bayes bound: Generalization properties of entropy-sgd and data-dependent priors. In International Conference on Machine Learning , pages 1377-1386. PMLR, 2018.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the International Conference on Machine Learning (ICML 2017) , pages 1126-1135. PMLR, 2017.

Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems (NeurIPS) , 33:82848295, 2020.

Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilistic time series imputation. In International Conference on Artificial Intelligence and Statistics , 2020.

Peter I. Frazier. A tutorial on bayesian optimization. arXiv preprint , arXiv:1807.02811, 2018.

Jacob Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew G. Wilson. Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems (NeurIPS) , 31, 2018.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In Proceedings of the International Conference on Machine Learning (ICML 2018) , pages 1704-1713. PMLR, 2018.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, SM Ali Eslami, and Yee Whye Teh. Neural processes. In Proceedings of the ICML Workshop on Theoretical Foundations and Applications of Deep Generative Models , 2018.

Hassan Gharoun, Fereshteh Momenifar, Fang Chen, and Amir H Gandomi. Meta-learning approaches for few-shot learning: A survey of recent advances. ACM Computing Surveys , 56(12):1-41, 2024.

Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta-learning probabilistic inference for prediction. In Proceedings of the International Conference on Learning Representations (ICLR 2018) , 2018.

Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Informationtheoretic generalization bounds for black-box learning algorithms. Advances in Neural Information Processing Systems , 34:24670-24682, 2021.

Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation , 9(1):1-42, 1997.

Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) , 44(9):5149-5169, 2021.

Stanisław Jastrz˛ ebski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623 , 2017.

Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npcl: Neural processes for uncertainty-aware continual learning. Advances in Neural Information Processing Systems , 36, 2024.

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations , 2020.

Haotian Ju, Dongyue Li, and Hongyang R Zhang. Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In International Conference on Machine Learning , pages 10431-10461. PMLR, 2022.

Myong Chol Jung, He Zhao, Joanna Dipnall, and Lan Du. Beyond unimodal: Generalising neural processes for multimodal uncertainty estimation. Advances in Neural Information Processing Systems , 36, 2024.

Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-batch training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representations, ICLR 2017 , 2017.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In Proceedings of the International Conference on Learning Representations (ICLR 2019) , 2019.

Jungtaek Kim. Benchmark functions for bayesian optimization, 2020.

Jungtaek Kim and Seungjin Choi. Bayeso: A bayesian optimization framework in python. Journal of Open Source Software (JOSS) , 8(90):5320, 2023.

Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. Advances in Neural Information Processing Systems (NeurIPS) , 33:6606-6615, 2020.

Huafeng Liu, Liping Jing, and Jian Yu. Neural processes with stability. Advances in Neural Information Processing Systems , 36, 2024.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August , 15(2018):11, 2018.

S. Mishra, S. Flaxman, T. Berah, M. Pakkanen, H. Zhu, and S. Bhatt. Pivae: Encoding stochastic process priors with variational autoencoders. arXiv preprint arXiv:2002.06873 , 2020.

Radford M. Neal. Regression and classification using gaussian process priors. In J. M. Bernardo, J. O. Berger, J. W. Dawid, and A. F. M. Smith, editors, International Conference on Learning Representations . Oxford University Press, 1998.

Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Informationtheoretic generalization bounds for stochastic gradient descent. In Conference on Learning Theory , pages 3526-3545. PMLR, 2021.

Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Proceedings of the International Conference on Machine Learning (ICML 2022) , pages 16569-16594. PMLR, 2022.

Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning . MIT Press, 2006.

Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory , 66(1):302-323, 2019.

Stephan R Sain. The nature of statistical learning theory, 1996.

Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning , pages 9058-9067. PMLR, 2020.

Samuel L Smith, Benoit Dherin, David Barrett, and Soham De. On the origin of implicit regularization in stochastic gradient descent. In International Conference on Learning Representations , 2022.

Samuel L Smith and Quoc V Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations , 2018.

Martin J. Wainwright. High-Dimensional Statistics: A Non-Asymptotic Viewpoint , volume 48. Cambridge University Press, 2019.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and S Yu Philip. Generalizing to unseen domains: A survey on domain generalization. IEEE transactions on knowledge and data engineering , 35(8):8052-8072, 2022.

Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. Advances in Neural Information Processing Systems , 36:1024-1035, 2023.

Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and a probabilistic perspective of generalization. Advances in neural information processing systems , 33:4697-4708, 2020.

Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning , pages 10367-10376. PMLR, 2020.

Lei Wu, Chao Ma, et al. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems , 31, 2018.

Lei Wu and Weijie J Su. The implicit regularization of dynamical stability in stochastic gradient descent. In International Conference on Machine Learning , pages 37656-37684. PMLR, 2023.

Aolin Xu and Maxim Raginsky. Information-theoretic analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems (NeurIPS) , 30, 2017.

Huanrui Yang, Xiaoxuan Yang, Neil Zhenqiang Gong, and Yiran Chen. Hero: Hessian-enhanced robust optimization for unifying and improving generalization and quantization performance. In Proceedings of the 59th ACM/IEEE Design Automation Conference , pages 25-30, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Please refer to Section 1 Introduction and Abstract.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Please refer to the Appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to the Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The paper clearly explains how to reproduce the algorithm. We will open source the code as much as possible.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We will provide them in the supplemental material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Most experiment setting and details are in the Appendix C.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted experiments in multiple models and multiple data.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See the Experimental section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: Our research complies with the NeurIPS Code of Ethics in all respects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to the Appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets we use are licensed.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We will open source the data and code as soon as possible.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: Our core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

with equality for Q = P X .

Proof

Since KL ( P X ∥ Q ) ≥ 0 , the equality exists only when Q = P X , which concludes the proof.

A.2 Conditional Mutual Information and Its Variational Form

Let X , Y , and Z be random variables. For all Z -measurable probability measures Q on the space of X , with equality for Q = P X | Z .

Proof

Since KL ( P X | Z ∥ Q ) ≥ 0 , the equality exists only when Q = P X | Z , which concludes the proof.

Appendix

The Technical Appendix is organized into five key sections. In Section A , we present essential foundational concepts and lemmas. Section B provides detailed derivations and proofs of the theorems discussed in the main paper. Section C offers a comprehensive supplement and further explanations of the experimental results described in the paper. Section D presents a detailed analysis of the computational complexity of our proposed approach. Finally, Section E includes the complete algorithm pseudocode for implementing Generalization Neural Processes.

A Lemma

In this section, we present essential foundational concepts and lemmas.

A.1 Variational Form of Mutual Information

Let X and Y be two random variables. For all probability measures Q defined on the space of X , we have

A.3 Mutual Information Under Conditioning

Let X , Y , and Z be random variables. For all Z -measurable probability measures Q defined on the space of X ,

with equality for Q = P X | Z .

A.4 Kullback-Leibler Divergence and its Representation

Let P and Q be two probability measures defined on a set X . Let g : X → R be a measurable function, and let E X ∼ Q [exp( g ( X ))] ≤ ∞ . Then

A.5 Data Processing Inequality

Given random variables X,Y,Z,V , and the Markov Chain:

then we have

For Markov chain we have

Proof. Since

and with the Markov Chain, we have X ⊥ Z | Y , therefore

In addition, I ( X ; Y | Z ) ≥ 0 , so I ( X ; Z ) ≤ I ( X ; Y ) .

with I ( Y ; Z | X ) ≥ 0 , we have I ( X ; Z ) ≤ I ( Y ; Z ) .

Similarly, for the second Markov chain, we have X ⊥ Y | V , therefore

So we have I ( X ; Z | V ) ≤ I ( X ; Y | V ) , the rest proof is similar and omitted.

A.6 Conditional Independence and Information Bounds

Given random variables X,Y,Z 1 , Z 2 , and the graph model:

then we have

Proof. Apply chain rule, we get:

From the graph model, we have Y ⊥ Z 1 , Y ⊥ Z 2 and ( X,Y ) ⊥ Z 1 | Z 2 . Hence

the last equality is obtained with Y ⊥ Z 1 , Z 2 , and since I ( Y ; Z 2 | X,Z 1 ) ≥ 0 , we get

Consequently, we have I ( X ; Y | Z 1 ) ≤ I ( X ; Y | Z 2 ) , conclude the proof.

A.7 Donsker-Varadhan Representation of Mutual Information

The Donsker-Varadhan representation provides a powerful variational characterization of mutual information that is particularly relevant to our Gen-NPs framework. This representation allows us to derive tractable lower bounds on mutual information, which is essential for the information-theoretic analysis of neural processes.

Let X and Y be random variables with joint distribution P X,Y . The Donsker-Varadhan representation states that:

where the supremum is taken over all measurable functions T for which the expectations exist, and P X ⊗ P Y denotes the product of marginal distributions.

Relation to KL Divergence This representation directly connects to the KL divergence representation in A.4, as mutual information is the KL divergence between the joint distribution and the product of marginals: I ( X ; Y ) = KL ( P X,Y ∥ P X ⊗ P Y ) .

Application to Gen-NPs In our Gen-NPs framework, we leverage this representation to analyze the mutual information between model parameters θ and training data S . By setting X = θ and Y = S , we can derive:

Noise-Contrastive Estimation This representation forms the theoretical basis for our noise injection learning strategy. When we introduce parameter noise during optimization, we are effectively using a specific form of the function T that facilitates estimation of the mutual information between model parameters and training data.

Moreover,

Proof Sketch The proof follows from the convex duality principle and properties of the logarithmic function:

Connection to Generalization Bounds In our analysis of Gen-NPs, this representation enables us to derive generalization bounds that explicitly account for the information complexity of the learning algorithm. Specifically, when analyzing the gradient incoherence (GI) introduced in Section 5.2, we implicitly utilize this variational characterization to establish the connection between noise injection and generalization performance.

This representation is central to our theoretical framework as it provides the mathematical foundation for understanding how parameter noise affects the information bottleneck between model parameters and training data, ultimately improving generalization capabilities of Neural Processes across diverse task distributions.

B Theorem Proof

This section provides detailed derivations and proofs of the theorems discussed in the main paper.

B.1 Proof of Theorem 1

If the loss ℓ ( θ, X × Y ) is σ -subgaussian for each θ ∈ Θ with respect to X × Y ∼ µ , the generalization error of a learning algorithm A satisfies the bound | gen ( µ, A ) | ≤ √ 2 σ 2 mn I ( θ ; D ) , where I ( θ ; D ) is the mutual information between the dataset D and the hypothesis θ . For meta-learning tasks, where µ ∼ τ , the meta-generalization error satisfies:

The mutual information I ( θ ; D 1: m ) can be computed based on the joint distribution P ( θ, D 1: m ) and the marginal P ( θ ) ; detailed derivations are provided in the appendix. This bound indicates that minimizing the dependence of θ on the data (via I ( θ ; D 1: m ) ) improves the generalization performance.

Proof Let θ ∈ Θ be a random variable representing the meta-parameter learned from the datasets D 1: m , and let ˆ θ ∈ Θ be an independent copy of θ such that ˆ θ is independent of the datasets D 1: m . The distribution of ˆ θ is the marginal distribution P θ , which is averaged over the possible datasets D 1: m drawn from the environment τ .

The mutual information I ( θ ; D 1: m ) quantifies the dependency between the meta-parameter θ and the observed datasets D 1: m . Specifically, it measures how much information about θ is gained by observing the datasets D 1: m . This dependency impacts the meta generalization error, as the error is influenced by the extent to which θ captures relevant information from the datasets while avoiding overfitting.

To express the meta generalization error, consider the function:

For any λ ∈ R , let

Moreover,

Since ( θ, D i ) , i = 1 , . . . , m are mutually independent given τ , and D 1 , . . . , D m are independent, we have

Hence,

Since ˆ θ ⊥ ⊥ D 1: m , we have that

Hence,

Therefore,

If we use Equations (2) and (3), then Equation (1) becomes

Since this inequality is also valid when λ is negative, this implies that we also have

Consequently,

Since ℓ ( θ, Z ) is σ -subgaussian, we have that

is σ √ nm -subgaussian. Hence,

Thus, we have which gives the theorem.

B.2 Proof of Theorem 2

Theorem B.1. By incorporating R dyn into the information-theoretic framework, the refined metageneralization error bound for Neural Processes (NPs) is given by:

where I ( θ ; D 1: m ) is the mutual information between the meta-parameter θ and the dataset D 1: m , R dyn is the dynamical stability regularization term, and γ > 0 is a scaling factor that controls the influence of R dyn. The proof of this theorem is provided in the Appendix.

Proof Let θ ∈ Θ be a random variable representing the meta-parameter learned from the datasets D 1: m , and let ˆ θ ∈ Θ be an independent copy of θ such that ˆ θ is independent of the datasets D 1: m . The distribution of ˆ θ is the marginal distribution P θ .

We define the effective mutual information I eff ( θ ; D 1: m ) as:

where R dyn = λ 1 · E [ Tr ( H )] + λ 2 · E [ ∥ H ∥ F ] quantifies the complexity of the hypothesis space through the trace and Frobenius norm of the Hessian matrix H .

To express the meta generalization error, consider the function:

For any λ ∈ R , let

By using the value of λ that minimizes the r.h.s. of the above equation, we have

Returning to Equation (4), we have for λ > 0 :

Hence, we also have

Then, Equations (5) and (6) together imply that

For the standard mutual information, we have:

When we incorporate the dynamical stability regularization, the effective mutual information is:

By the data processing inequality and information bottleneck principles, dynamical stability regularization effectively reduces the mutual information through constraining the parameter space complexity:

Continuing with the proof, we have:

Since ( θ, D i ) , i = 1 , . . . , m are mutually independent given τ , and D 1 , . . . , D m are independent, we have

Hence,

Since ˆ θ ⊥ ⊥ D 1: m , we have that

Hence,

Therefore,

Combining these results, we get:

Since this inequality is also valid when λ is negative, we also have:

Consequently,

Since ℓ ( θ, Z ) is σ -subgaussian, we have that

Now, considering the influence of R dyn on the effective mutual information, we have:

By the principle of information bottleneck and complexity regularization, we can establish:

is σ √ nm -subgaussian. Hence,

Substituting this, we get:

By the information-curvature relationship established through dynamical stability, we can refine the bound:

Optimizing for λ , we set λ = √ 2 nm · I ( θ ; D 1: m ) σ 2 , which gives:

However, considering the impact of R dyn on the mutual information through the lens of information bottleneck theory and complex system dynamics, we can establish that:

Therefore, the final bound becomes:

This completes the proof.

C Experiments

This section offers a comprehensive supplement and further explanations of the experimental results described in the paper. Here we use TNP-A as TNP.

C.1 Hardware and Software Configuration

To ensure the reproducibility and reliability of the experiments conducted in this study, we detail the hardware and software environments used.

GPU Model(s):

-

Model: NVIDIA RTX A4000

-

Count: 8 GPUs

Memory per GPU: 16 GB

CPU Model(s):

Model: Intel(R) Xeon(R) Platinum 8358P

-

Core Count: 32 cores

· Operating System:

OS: Ubuntu 20.04 LTS

Kernel Version: 5.15.0-113-generic

· Relevant Software Libraries and Frameworks:

CUDA: Version 11.8

-

cuDNN: Version 8.6

PyTorch: Version 2.0.0

Scikit-learn: Version 1.5.0

-

NumPy: Version 1.26.3

-

Pandas: Version 2.2.2

C.2 1-D Regression

The following sections provide a detailed description of the training and evaluation processes for the 1-D regression task.

Training During the training phase, different functions are drawn from a Gaussian Process (GP) prior with a Radial Basis Function (RBF) kernel for each epoch. These functions are represented as f i ∼ GP ( m,k ) , where the mean function is m ( x ) = 0 and the covariance function is k ( x, x ′ ) = σ 2 f exp ( -( x -x ′ ) 2 2 ℓ 2 ) . The GP hyperparameters ℓ and σ f are randomized for each function, providing a diverse set of training samples. For each function f i , N random locations are selected for evaluation, and an index m is chosen to divide the sequence into context points and target points. The parameters are set as follows: ℓ ∼ U [0 . 6 , 1 . 0) , σ f ∼ U [0 . 1 , 1 . 0) , B = 16 , N ∼ U [6 , 50) , and m ∼ U [3 , 47) .

Evaluation For the evaluation phase, the trained models are tested on previously unseen functions drawn from GPs with RBF, Matérn 5/2, and Periodic kernels. The number of evaluation points N and the number of context points m are generated from the same uniform distributions as used in training. The evaluation set includes 48,000 functions for each kernel type. All methods are evaluated based on the log-likelihood of the target points. Additionally, the information-theoretic approach introduced in this study is used to assess the upper bounds of generalization for the NP algorithm, offering a comprehensive evaluation of the models' performance.

Results The evaluation results of the Gen-Method on three different Gaussian kernels are presented in Table 4. The table showcases the performance metrics including Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Log-likelihood. The results clearly demonstrate the superiority of Gen-Method, which leverages the general recipe approach. Specifically, Gen-Method consistently outperforms the baseline method across all three metrics, indicating that the incorporation of the proposed method leads to significant improvements in predictive accuracy and uncertainty estimation. Sample functions produced by the Gen-Method and the baselines given 30 context points are illustrated in Figure 6, further highlighting the enhanced performance of Gen-Method in terms of capturing the underlying function variability and providing more accurate predictions.

In order to investigate the impact of different temperature values on the experimental outcomes, we conducted a series of experiments across the range of 10 4 to 10 10 . The experimental results are illustrated in Figure 5. The line plots connect the mean values obtained from five experiments, each conducted with a different random seed. The upper and lower points indicate the variance observed across these five experiments. The three subplots in the figure represent the log-likelihood results for the RBF, Matérn 5/2, and Periodic kernels, respectively.

Matern 5/2 Kernel

PeriodicKernel

0.31

0.08

1.40

0.30

0.07

1.42

Log-Likelihood

0.29

0.06

1.44

0.28

0.05

0.27

1.46

0.04

0.26

1.48

103

104

105

106

107

108

109

1010

1011

103

104

105

106

107

108

109

1010

1011

103

104

105

106

107

108

109

1010

1011

Temperature

Temperature

Temperature

C.3 Image Completion

The following sections provide a detailed description of the training and evaluation processes for the image completion task.

Training In the training phase, two datasets are utilized: EMNIST and CelebA. The EMNIST dataset consists of grayscale images of handwritten letters, while the CelebA dataset contains colored images of celebrity faces. Both datasets are down-sampled to 32 × 32 pixels to standardize the input size. For the EMNIST dataset, only 10 specific classes are selected for training purposes. During training, random subsets of pixels are chosen as context and target points, where the number of total points N is sampled from a uniform distribution U [6 , 200] and the number of context points m is sampled from U [3 , 197] . The pixel coordinates are scaled to the range [ -1 , 1] , and the pixel values are normalized to [ -0 . 5 , 0 . 5] to ensure consistency across the training process.

Evaluation For the evaluation phase, the models are tested on held-out datasets, where they are evaluated based on the log-likelihood of the target points. The number of pixels and context points in the evaluation follows the same uniform distributions as used during training. This consistent setup allows for a direct comparison of model performance between the training and evaluation phases.

Results Table 5 and Table 6 present the results of the Gen-NPs method on the CelebA and EMNIST datasets, respectively, showcasing the performance metrics such as Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Log-Likelihood. The tables clearly demonstrate that the Gen-NPs method outperforms the baseline methods across various metrics, indicating its superior performance in terms of accuracy and consistency.

Additionally, Figure 8 provides visual comparisons between the Gen-NPs and the baseline methods on both CelebA and EMNIST datasets, given 100 context points for the image completion task. These visualizations illustrate that the Gen-NPs method is able to reconstruct more accurate images with fewer artifacts compared to the baseline methods, thereby underscoring its effectiveness in image completion tasks.

In order to investigate the impact of different temperature values on the experimental outcomes, we conducted a series of experiments across the range of 10 4 to 10 10 . The experimental results are illustrated in Figure 7. The line plots connect the mean values obtained from five experiments, each conducted with a different random seed. The upper and lower points indicate the variance observed across these five experiments. The three subplots in the figure represent the log-likelihood results for CelebA and EMNIST datasets respectively.

C.4 Bayesian Optimization

The following sections provide a detailed description of the training and evaluation processes for the Bayesian optimization task.

Training In the training phase, the 1D scenario follows the approach outlined in Section 4.1. For multi-dimensional input x , training data is generated using the method proposed by NPs, where multivariate Gaussian Processes (GPs) with a Radial Basis Function (RBF) kernel are employed. In the 2D scenario, the number of total points N is sampled from a uniform distribution U [60 , 128] , and the number of context points m is sampled from U [30 , 98] . Similarly, in the 3D scenario, N is sampled from U [128 , 256] , and m is sampled from U [64 , 192] . This training setup ensures that the models are

well-prepared to handle the complexities of Bayesian optimization tasks in both one-dimensional and multi-dimensional spaces.

Evaluation For the evaluation phase, in the 1D scenario, the objective functions are generated from Gaussian Processes with RBF, Matérn 5/2, and Periodic kernels. In multi-dimensional settings, various benchmark functions from the optimization literature are employed, with the Bayesian optimization process implemented using a comprehensive framework that includes both the optimization and acquisition functions. Each objective function undergoes 100 iterations of Bayesian optimization, with simple regret serving as the primary evaluation metric. This metric provides a clear indication of the model's performance by measuring the difference between the best-known value and the actual value found during optimization.

Results: In the main body of the paper, for the sake of clarity, we only presented the results for Bayesian optimization using CNP, NP, and ANP methods. However, to provide a more comprehensive and detailed comparison, Figures 9 to 11 showcase the results for all methods, including their Genenhanced variants, across different dimensions and benchmark functions. Each method's performance is individually illustrated for 1D, 2D, and 3D Bayesian optimization tasks, providing a clearer visualization of the differences.

Figure 9 focuses on the 1D Bayesian optimization tasks, where results for different kernel functions (RBF, Matérn 5/2, and Periodic) are presented for each method. Figure 10 expands the comparison to 2D tasks, demonstrating the performance across various benchmark functions like Ackley, Dropwave, and Michalewicz. Finally, Figure 11 extends the analysis to 3D tasks, further illustrating the effectiveness of each method on more complex functions such as Cosine and Rastrigin.

These figures collectively offer a detailed and nuanced understanding of how each method performs under varying conditions, highlighting the consistent advantages of the Gen-enhanced approaches in achieving lower regret and more stable optimization results across all tested scenarios.

C.5 Contextual bandits

The study compares Gen-NPs with baselines using the wheel bandit framework. This framework involves a unit circle segmented into a low-reward zone (colored blue) and four high-reward zones of different colors. The division is controlled by a scalar δ , which sets the boundary of the low-reward zone, leaving the other four zones equally sized. An agent, unaware of δ 's value, selects from five potential actions based on its position within the circle.

When the agent's position ∥ X ∥ is less than or equal to δ , it is located in the low-reward zone. The optimal choice here is action k = 1 , rewarding the agent with r ∼ N (1 . 2 , 0 . 012) . All other actions yield r ∼ N (1 . 0 , 0 . 012) . Conversely, if ∥ X ∥ > δ , indicating presence in a high-reward zone, the agent should choose from actions k = 2 -5 . These choices can grant a significant reward of r ∼ N (50 . 0 , 0 . 012) , with all non-optimal choices returning N (1 . 0 , 0 . 012) , except for k = 1 .

1

1

1

0

0

-1

-1

-1

0

1

-1

0

1

0

1

1

0

0

-1

-1

-1

-1

0

1

-1

0

1

-1

0

1

Training A dataset is created by generating B different wheel problem instances { δ i } B i =1 , where δ values are uniformly distributed between 0 and 1. For each instance, N points are sampled to assess and select m points as context for training, with each point being a pair ( X,r ) of coordinates and the corresponding reward. The goal is to learn to predict reward values based on X . Parameters are set with B = 8 , N = 562 , and m = 512 .

Evaluation Experiments are conducted by testing the Gen-NPs and baseline approaches across varying δ values, using 50 different seeds for each setting. Over 2000 steps per trial, each agent's task is to estimate the reward values for five strategies based on X , choose according to the Upper Confidence Bound (UCB) strategy, and receive the actual reward for the selected strategy. Cumulative regret is utilized to measure the performance effectiveness.

Results As shown in Table 7, for cumulative regret, the Gen versions of CNP, NP, ANP, BNP, and TNP generally achieve lower regret across all δ settings, demonstrating their superior capability in handling the complex decision-making scenarios introduced by the wheel bandit framework. Notably, Gen-TNP outperforms all other methods, especially in the most difficult cases (higher δ ), where it maintains low regret with minimal variance.

Similarly, Table 8 presents the simple regret outcomes, further confirming the advantage of Genenhanced methods. The simple regret is notably lower for Gen versions, indicating more effective exploration and exploitation of the reward landscape, which is crucial in achieving optimal decisionmaking.

The visual depiction of the wheel bandit problem with varying δ values is provided in Figure 12. This figure illustrates the segmentation of the reward zones within the unit circle, helping to contextualize the challenge faced by the models in predicting optimal actions based on incomplete and uncertain information.

Overall, the results clearly demonstrate the effectiveness of integrating the proposed general recipe into neural process-based models, significantly improving their performance in contextual bandit problems by reducing both cumulative and simple regret across various scenarios.

C.6 Ablation Study

We conducted ablation studies to validate the effectiveness of the proposed Risk-Aware Dynamical Stability Regularization (DSR) and Optimization-Aware Noise Injection Learning Strategy (NILS) on 1D regression task. Table 9 presents the results of the original method, Gen-NPs with only DSR, Gen-NPs with only NILS, and the full Gen-NPs with both modules included.

As shown in Table 9 and Appendix, these results highlight the significance of incorporating DSR for improving dynamical stability and NILS for robust optimization. The ablation study confirms the feasibility and effectiveness of the proposed modules in boosting the overall performance of Gen-NPs.

C.7 Comparison with Stability Neural Processes

To provide a comprehensive analysis, we conducted additional experiments to compare stability-based generalization error (SGE) with our proposed Gen-NPs, emphasizing the differences in their noise introduction mechanisms and evaluation methods.

While SGE quantifies generalization as the difference between test error and training error, our approach focuses on gradient incoherence (GI) and directly modeling the parameter-data mutual information through controlled noise injection. To test robustness, we added Gaussian noise (mean = 0, variance = 1) to a random selection of 5% and 10% of the training data. The experiments were performed on a one-dimensional regression task with data generated using an RBF kernel.

Table 10 presents detailed results for CNP, ANP, and BNP models under original, 5% noise, and 10% noise conditions. Gen-NPs consistently achieves higher log-likelihood (LL) values and lower gradient incoherence (GI) compared to baseline methods across all noise levels. Notably, even as noise increases, Gen-NPs maintains better performance relative to standard NPs, demonstrating its enhanced robustness.

The key difference lies in how noise is utilized: Stability Neural Processes approaches add noise to the training data to measure stability, while Gen-NPs introduces noise strategically during the parameter update process. This fundamental difference enables Gen-NPs to better capture the mutual information between model parameters and training data, leading to improved generalization capacity under various conditions.

D Computational Complexity Analysis

In this section, we analyze the computational complexity and overhead introduced by our GenNPs approach compared to standard NP methods. While noise injection and dynamical stability regularization enhance model performance, they also introduce additional computation. Here we quantify these costs and demonstrate that the performance benefits outweigh the computational overhead.

Theoretical Complexity Analysis Let d denote the model parameter dimension, n the number of context points, m the number of tasks, and b the batch size.

For the dynamical stability regularization (DSR) term computation, we avoid explicitly forming the full Hessian matrix, which would require O ( d 3 ) operations. Instead, we utilize efficient Hessianvector product approximations, reducing the complexity to O ( bd 2 ) . The noise injection component

has minimal overhead of O ( d ) for sampling from a Gaussian distribution and adding the noise to the parameter updates.

Empirical Evaluation We measured the actual computational overhead across different tasks using the same hardware setup for all experiments. Table 12 summarizes these findings.

Task-Specific Training Time Analysis We further analyzed the training time across different tasks and architectures to provide a comprehensive view of the computational overhead.

Cost-Benefit Analysis While Gen-NPs introduce approximately 10% additional training time, the consistent performance improvements across all tasks and metrics easily justify this minimal overhead. The most computationally intensive component is the DSR term, specifically the computation of Hessian-related properties. However, this overhead is only present during training; inference time remains virtually identical to standard NP methods.

For tasks requiring high accuracy and reliable uncertainty quantification, such as Bayesian optimization and medical image completion, the 7-9% improvement in log-likelihood represents a significant practical advantage that substantially outweighs the modest 10% increase in training resources.

Furthermore, we found that in practice, Gen-NPs often require fewer training iterations to reach a target performance level compared to standard NPs, which can fully offset the per-iteration computational overhead in end-to-end training scenarios. This favorable performance-to-cost ratio makes Gen-NPs particularly attractive for practical applications where generalization and reliable uncertainty estimation are critical.

Implementation Considerations To minimize the computational overhead while maintaining performance benefits, we recommend:

Using stochastic approximations of the Hessian trace and Frobenius norm when applicable

Gradually decreasing the frequency of DSR computation during later training stages

Implementing the DSR term computation with efficient auto-differentiation libraries that optimize Hessian-vector products

For very large models, considering a reduced-precision implementation of the DSR component

These optimizations can further reduce the computational gap between standard NPs and Gen-NPs while preserving the generalization benefits of our approach.

E Algorithm Pseudocode

We present the complete algorithm for Generalization Neural Processes (Gen-NPs) that integrates both the Risk-Aware Dynamical Stability Regularization (DSR) and Optimization-Aware Noise Injection Learning Strategy (NILS) components. Algorithm 1 provides a comprehensive pseudocode implementation that practitioners can follow to apply our method to various Neural Process variants.

Algorithm 1 Generalization Neural Processes (Gen-NPs)

Require: Task environment τ , initial learning rate η 0 , inverse temperature γ , number of tasks per batch B , number of iterations S , DSR coefficients λ 1 ∈ [0 . 01 , 0 . 1] , λ 2 ∈ [0 . 001 , 0 . 01]

1: Randomly initialize θ 0

2: for s ← 1 to S do

3: Sample a batch of tasks {D i } B i =1 ∼ τ

4: Initialize task gradients and DSR term: G ( θ s -1 ) = 0 , R dyn = 0

5: for each task i ∈ [ B ] do

6: Randomly split task D i into context set D C i and target set D T i

7: Calculate task-specific empirical risk ˜ R D T i ( θ s -1 )

8: Calculate task-specific gradient g i ( θ s -1 , D C i , D T i )

9: Estimate Hessian trace Tr ( H i ) and Frobenius norm ∥ H i ∥ F using efficient approximations

10: Update DSR term: R dyn + = 1 B ( λ 1 · Tr ( H i ) + λ 2 · ∥ H i ∥ F )

11: end for

12: Aggregate gradients over all tasks: G ( θ s -1 ) = 1 B ∑ B i =1 g i ( θ s -1 , D C i , D T i )

13: Calculate gradient of DSR term: ∇ R dyn

14: Update learning rate to η s

15: Calculate Gaussian noise variance σ 2 s = η s γ

16: Sample Gaussian noise ξ s ∼ N (0 , σ 2 s I k )

17: Update parameter θ s = θ s -1 -η s ( G ( θ s -1 ) + ∇ R dyn ) + ξ s

18: end for

(a) Sample functions produced by CNP and Gen-CNP.

(b) Sample functions produced by NP and Gen-NP.

(c) Sample functions produced by ANP and Gen-ANP.

(d) Sample functions produced by BNP and Gen-BNP.

context30

context 30

context 30

2.0

context 30

target 10

target 10

target 10

target 10

S'

1.0

1.5

1.0

1.0

0.0

3.5

0.0

0.5

0.5

00

1.0

0.5

-1.5-1.00.50.00.5101.52.0

context 30

context 30

2.0

context 30

context 30

target 10

target 10

target 10

target 10

2.0

1.5

1.5

10

0.5

0.0

0.5

0.0

0.5

0.0

0.5

1.0

-1.5-1.0-0.50.00.51.01.5

-2.0-1.5-1.0-0.50.00.51.01.52.0

target 10

context 30

2.0

context 30

target 10

2.0

。

context 30

target 10

2.0

target 10

context 30

1.5

1.5

10

1.0

0.5

0.0

0.0

0.5

0.5

-2.0-1.5-1.0-0.50.00.51.01.52.0

context30

target 10

2.

context30

target 10

1.5

context 30

target 10

2.0

context 30

target 10

2.0

1.0

0

1.5

0.5

0.5

1.0

0.0

0.5

0.5

1.0

-1.5-1.0-0.50.0

0.51.0

1.5

2.0

-1.5-1.0-0.50.00.51.01.52.0

context 30

target 10

2.0

。

context 30

target 10

context 30

target 10

2.0

context 30

target 10

15

1.0

1.0

1.0

0.5

S°

0.0

0.5

0.5

0.5

0.0

0.0

00S0-0'T-

0.51.0

1.5

2.0

(e) Sample functions produced by TNP and Gen-TNP.

CelebA

Emnist(0-9)

Emnist(10-46)

2.205

0.80

0.57

2.200

0.79

0.56

0.78

Log-Likelihood

2.195

0.77

0.55

2.190

0.76

0.54

2.185

0.75

0.53

2.180

0.74

0.73

0.52

2.175

103

104

105

106

107

108

109

1010

1011

103

104

105

106

107

108

109

1010

1011

103

104

105

106

107

108

109

1010

1011

Temperature

Temperature

Temperature

Original

Task

CNP

Gen-CNP

NP

Gen-NP

ANP

BNP

TNP

Gen-TNP

Original

Task

CNP

Gen-CNP

NP

Gen-NP

ANP

BNP

TNP

Gen-TNP

Original

Task

CNP

Gen-CNP

NP

Gen-NP

ANP

BNP

TNP

Gen-TNP

0.40

RBF - NP

NP

0.40

NP

0.40

Periodic - NP

NP

0.35

Gen-NP

0.35

Gen-NP

0.35

Gen-NP

0.30

0.30

0.30

0.25

0.25

0.25

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

0.00

0.00

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Iterations

Iterations

Iterations

0.40

RBF - ANP

0.40

Matern 5/2 - ANP

0.40

Periodic - ANP

ANP

ANP

0.35

0.35

0.35

0.30

0.30

0E'0

0.25

0.25

0.25

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

S0'0

0.00

0.00

0.00

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

Iterations

Iterations

Iterations

0.40

RBF - BNP

0.40

Matern 5/2 - BNP

0.40

Periodic - BNP

BNP

BNP

BNP

0.35

0.35

0.35

0.30

0.30

0.30

0.25

0.25

0.25

0.15

0.15

0.15

0.10

0.10

0.10

s0°0

50°0

S0°0

0.00

0.00

0.00

20

40

Iterations

60

80

100

0

20

40

Iterations

60

80

100

20

40

Iterations

60

80

100

RBF - CNP

Matern 5/2 - CNP

0.40

0.40

0.40

CNP

CNP

CNP

0.35

Gen-CNP

0.35

Gen-CNP

0.35

Gen-CNP

0.30

0.30

0.30

0.25

0.25

0.25

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

0.00

0.00

0

20

40

Iterations

60

80

100

0

20

40

Iterations

60

80

100

0

20

40

Iterations

60

80

100

0.40

RBF - TNP

0.40

Matern 5/2 - TNP

0.40

Periodic - TNP

TNP

TNP

TNP

0.35

Gen-TNP

0.35

Gen-TNP

0.35

Gen-TNP

0.30

0.30

0.30

0.25

0.25

0.25

0.15

0.15

0.15

0.10

0.10

0.10

0.05

0.05

0.05

0.00

0.00

0.00

0

20

40

60

80

100

20

40

60

80

100

0

20

40

60

80

100

Iterations

Iterations

Iterations

NP - Ackley (2D)

NP - Dropwave (2D)

NP - Michalewicz (2D)

Gen-NP

NP

0.70

Gen-NP

NP

1.6

Gen-NP

NP

0.65

1.4

0.60

Regret

0.55

1.2

0.50

0.45

1.0

0.40

20

40

60

80

100

0

20

40

60

80

100

8'0

0

20

40

60

80

100

ANP - Ackley (2D)

ANP - Dropwave (2D)

ANP - Michalewicz (2D)

ANP

0.7

ANP

1.6

一

ANP

0.6 -

1.4

0.5

1.2

Reg

1.0

0.4

2

0.8

0.3

0.6

0.2

0.4

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

BNP - Ackley (2D)

BNP - Dropwave (2D)

BNP - Michalewicz (2D)

BNP

0.70

BNP

1.6

BNP

0.65

0.60

1.4

Regret

0.55

1.2

0.50

0.45

1.0

0.40

0.35

0.8

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

CNP- Dropwave (2D)

CNP - Michalewicz (2D)

CNP

Gen-CNP

0.7

Gen-CNP

CNP

1.6

—CNP

Gen-CNP

0.6

1.4

0.5

1.2

0.4 -

1.0

0.3

0.8

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

TNP - Ackley (2D)

TNP - Dropwave (2D)

TNP - Michalewicz (2D)

Gen-TNP

TNP

0.7

Gen-TNP

TNP

1.6

Gen-TNP

TNP

0.6

1.4

1.2

Regret

0.5

1.0

0.4

0.8

0.3

0.6

0.2

0.4

D

20

40

Iterations

60

80

100

0

20

40

Iterations

60

80

100

0

20

40

Iterations

60

80

100

521

NP - Ackley (3D)

NP -Cosine (3D)

NP - Rastrigin (3D)

5.5

Gen-NP

NP

1.7

Gen-NP

NP

58

Gen-NP

NP

56

1.6

5.0

1.5

52

1.4

4.0

1.3

50

3.5

1.2

48

0

20

40

60

80

100

20

40

60

80

100

20

40

60

80

100

ANP - Ackley (3D)

ANP - Cosine (3D)

ANP - Rastrigin (3D)

5.0

5.5

—ANP

1.8

1.6

1.4

ANP

55

ANP

4.5

1.2

1.0

45

3.5

0.6

40

3.0

0.4

2.5

0.2

35

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

BNP - Ackley (3D)

BNP - Cosine (3D)

BNP - Rastrigin (3D)

5.5

BNP

BNP

58

BNP

5.0

1.6

4.5

54

1.4

52

3.5

1.2

3.0

1.0

2.5

2.0

44

0

20

40

60

80

100

0

20

40

60

80

100

20

40

60

80

0T

CNP - Cosine (3D)

CNP - Rastrigin (3D)

5.5

一

CNP

Gen-CNP

1.7

CNP

Gen-CNP

58

一

CNP

Gen-CNP

1.6

56

5.0

1.5

54

1.4

4.0

3.5

50

1.1

3.0

48

0

20

40

60

80

100

20

40

60

80

100

0

20

40

60

80

100

TNP - Ackley (3D)

TNP - Cosine (3D)

TNP - Rastrigin (3D)

5.5

Gen-TNP

TNP

1.75

Gen-TNP

TNP

58

Gen-TNP

TNP

1.50

5.0

4.5

1.25

54

1.00

52

0.75

3.5

50

0.50

3.0

0.25

48

2.5

0.00

46

0

20

40

Iterations

60

80

100

20

40

Iterations

60

80

100

20

40

Iterations

60

80

100

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an information-theoretic approach to enhance generalization in Neural Processes (NPs). It combines dynamical stability regularization (based on second-order curvature metrics) with an optimization-aware noise-injection strategy during training. Through theoretical analysis and experiments on standard benchmark tasks, the authors claim improved generalization and optimization stability by guiding the model toward flatter minima.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is well structured and accessible even to readers without extensive theoretical backgrounds such as myself. The theoretical concepts (e.g., mutual information and dynamical stability) are well explained. The approach is carefully motivated through rigorous theoretical discussions linking mutual information, generalization bounds, and curvature of the loss landscape. Literature on Neural Processes and information-theoretic generalization bounds is thoroughly reviewed, providing valuable context for their contributions. Next, I will list the weaknesses:\n\nNoise injection into stochastic gradient descent is well-studied in machine learning optimization (e.g., stochastic gradient Langevin dynamics and various noise-based regularizers). While the authors clearly articulate their approach, the novelty of explicitly injecting noise into the parameter updates is unclear. The authors do not sufficiently differentiate their proposed noise injection strategy from existing methods. The paper would significantly benefit from explicitly clarifying its novelty—perhaps through comparison to classical methods such as SGLD or entropy-SGD—and outlining how it improves upon or differs from these well-established approaches.\n\nA core theoretical argument made by the authors is that their method favors flatter minima, leading to better generalization. However, this claim lacks direct empirical validation. The paper does not provide explicit demonstrations—such as visualizing or analyzing synthetic loss landscapes, Hessian eigenvalue spectra, or trajectories during optimization that could convincingly illustrate how the introduced noise injection explicitly leads to flatter minima. Providing visualizations or constructing simplified synthetic experiments to highlight this connection explicitly would significantly strengthen the authors' key argument.\n\nThe authors scope their work within the context of Neural Processes and meta-learning frameworks, but the justification for this remains unclear. The connection between meta-learning and the relevance of flatter optima is not explained well. It is not evident why the proposed regularization and noise injection approach should be particularly suitable or beneficial for NPs. Clarifying this connection, and potentially extending discussions or experimental results to conventional training regimes, would improve the significance and clarity of the contribution. Moreover, the paper does not adequately compare its performance to widely used meta-learning benchmarks or state-of-the-art meta-learning algorithms (e.g., MAML, Reptile, ANIL, Bayesian meta-learning methods, or gradient-based meta-learning frameworks).  \n\nA significant limitation is the absence of explicit comparisons against well-known optimization or regularization strategies, which are commonly used to promote flatter minima. For instance, methods like learning-rate scheduling, batch-size reduction, weight decay, or dropout are standard baselines that naturally encourage flatter solutions and improved generalization. Without benchmarking their method against these conventional approaches, the paper cannot convincingly demonstrate the unique advantages of its approach. Including comparisons against such widely adopted techniques would clarify whether the proposed method provides meaningful additional value beyond existing, simpler methods.\n\nThe authors employ Hessian-based sharpness measures (e.g., Hessian trace and Frobenius norm) to quantify curvature. However, these metrics are known to be sensitive to reparameterization (the same model expressed differently can produce different curvature values). For example, recently proposed reparameterization-invariant sharpness metrics, which measure changes in the loss relative to probability distributions rather than parameter-space perturbations, could provide more robust insights into solution flatness and generalization. The absence of these modern invariant metrics limits the validity and interpretability of their sharpness analysis. Employing or comparing to reparameterization-invariant metrics would considerably strengthen the theoretical and empirical claims.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes the generalization gap of Neural Processes (NPs), a class of meta-learning models that operate over task distributions. Starting from an information-theoretic upper bound on the generalization error—formulated via the mutual information between the task dataset and model parameters—the authors introduce two mechanisms to tighten this bound. First, a dynamical stability regularizer penalizes sharp minima through the trace and Frobenius norm of the Hessian of the loss, encouraging convergence to flatter, more stable regions of the loss landscape. Second, noise injection is applied to the parameter updates during training, which reduces the mutual information between parameters and data while further promoting flat minima. Together, these mechanisms lead to solutions that are less sensitive to task-specific data and more stable under training dynamics. The enhanced training scheme is evaluated on multiple NP variants (CNP, NP, ANP, TNP, BNP) and shows consistent improvements in predictive performance and Bayesian Optimization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed algorithm is agnostic to NP model structure and has been validated on various architectures (CNP, ANP, TNP, etc.) as well as downstream tasks such as Bayesian Optimization.\n- The paper is generally well-written and mathematically consistent.\n\nWeaknesses\n- The rational of the two approaches: While not necessarily a short come, the link between the two approaches, namely the Hessian regularization and noise injection in gradient descent, is not structurally derived but instead motivated by their joint effect on the generalization bound. This is reasonable, but it raises the question of whether the particular mechanisms are canonical or somewhat arbitrary. For instance, one could consider using dropout or other stochastic regularization to reduce mutual information and hence tighten the bound.\n- Incremental via algorithm wise: Both proposed techniques—curvature regularization and noise injection—are established tools in the literature, and the paper's contribution is primarily in reinterpreting them within a refined theoretical framework rather than in algorithmic novelty.  \n- While not a limitation, the claim of tighten bound through noise injection training could be empirically validated if a mutual information quantification can be approximated and compared.\nMinor writing issues:\n    - (DSR) spacing error in line 204.\n    - Line 214: Froubinius norm form typo?\n\nQUESTIONS:\n- how is the $\\lambda_1$ and $\\lambda_2$ range choosen as l216, is it (also the hyperparameter $\\sigma_s$) problem dependent?\n- The performance improvement in terms of LL and GI is rather small to me, while it seems most of the BO/contextual bandit benchmark the performance difference is pretty obvious, could the author provide some comments on this discrepancy?\n\nLIMITATIONS:\nI didn't find any description of limitations which is claimed to be in the Appendix. I think a proper discussion should be included (e.g., considering aforementioned weakness unless further clarification)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is a decent contribution to the Neural Processes (NPs) research. Although numerous architectural variants exist (ANPs, TNPs, ConvCNPs), there's limited theoretical understanding of their generalization properties.\n\nThe key contributions appear to be:\n1. Information-theoretic bounds: Derives generalization bounds for NPs using mutual information between parameters and training data.\n2. Dynamical stability regularization: Introduces R_dyn = λ₁·E[Tr(H)] + λ₂·E[||H||_F] to penalize sharp minima.\n3. Noise-injected updates: Adds Gaussian noise to parameter updates (not data), which reduces mutual information and increases dynamical stability.\n\nSTRENGTHS AND WEAKNESSES:\nTo me, this paper is a solid contribution to the NP literature. I believe it should be accepted into the conference.\n\nI believe the strengths of the paper are as follows:\n1. Good theoretical framework for NP generalization. This appears to solve a genuine gap in the literature\n2. General applicability, the methods discussed seem to work across all NP variants without architectural modifications\n3. Clean execution, the paper is well-written with thorough related work\n4. Solid experimental validation\n\nI believe the following are weaknesses:\n1. The theory is not as novel as it could be. Each component -- MI bounds, sharpness regularization and noise injection -- are well established.\n2. The improvements are not ground breaking\n3. There appears to be computational overhead from the Hessian calculations (although I have not thought about this deeply and how it scales relative to the other parts of the setup).\n\nQUESTIONS:\nI don't have too many questions, although there are a few that come to mind.\n1. What's the computational cost of calculating the Hessian? How much time does this take, how does this calculation scale compared to others in the setups?\n2. How sensitive are the results to the hyperparameters?\n3. Does this theory extend to other meta-learning algorithms?\n4. I may have misread, but I think you claim that noise increases Tr(H) which improves stability. But higher curvature usually means worse generalization. What's going on here? Are there competing forces?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-13"}