Learning to Generalize: An Information Perspective on Neural Processes
Authors: Hui Li
Huafeng Liu
Shuyang Lin
Jingyue Shi
Yiran Fu
Liping Jing
Venue: NeurIPS 2025 poster ()
Keywords: Neural Processes
Information-Theoretic
Generalization
Primary area: deep_learning

Abstract: Neural Processes (NPs) combine the adaptability of neural networks with the efficiency of meta-learning, offering a powerful framework for modeling stochastic processes. However, existing methods focus on empirical performance while lacking a rigorous theoretical understanding of generalization. To address this, we propose an information-theoretic framework to analyze the generalization bounds of NPs, introducing dynamical stability regularization to minimize sharpness and improve optimization dynamics. Additionally, we show how noise-injected parameter updates complement this regularization. The proposed approach, applicable to a wide range of NP models, is validated through experiments on classic benchmarks, including 1D regression, image completion, Bayesian optimization, and contextual bandits. The results demonstrate tighter generalization bounds and superior predictive performance, establishing a principled foundation for advancing generalizable NP models.

Learning to Generalize: An Information Perspective on Neural Processes
Hui Li Li3,, Huafeng Liu
Shuyang Lin Lin 12,3,
Jingyue Shi
Yiran Fu Ful.2.3, Liping Jing
State Key Laboratory of Advanced Rail Autonomous Operation, Beijing, China
School of Computer Science and Technology, Beijing Jiaotong University, Beijing. China 3 Beijing Key Laboratory of Traffic Data Mining and Embodied Intelligence, Beijing, China fhuili97, hfliul, sylin1, jingyueshi, yiranfu, 1pjing/0bjtu edu.
Abstract
Neural Processes (NPs) combine the adaptability of neural networks with the effi- ciency of meta-learming, offering a powerful framework for modeling stochastic processes. However, existing methods focus on empirical performance while lack- ing a rigorous theoretical understanding of generalization. To address this, we propose an information teheoretic framework to analyze the generalization bounds NPs, introducing dynamical stability regularization to minimize sharpness and improve optimization dynamics. Additionally, we show how noise-injected parame- ter updates complement this regularization. The proposed approach, applicable a wide range of NP models, is validated through experiments classic benchmarks, including ID regression, image completion, Bayesian optimization, and contextual bandits. The results demonstrate tighter generalization bounds and superior predic- tive performance, establishing a principled foundation for advancing generalizable NP models.
1 Introduction
Gaussian processes (GPs) are widely recognized as a robust framework for modeling distributions over functions [37]. Their appeal lies the consistent probabilistic reasoning enabled by Bayesian inference, which facilitates data-efficient modeling. Despite their strengths, GPs are unsuitable for certain problems. For instance, function exhibiting a single, unknown discontinuity is a classic example of distribution that GPs fail to represent
To address such limitations, researchers have turned to neural network-based generative models. Notable advancements in this area include meta-learning techniques like Neural Processes (NPs) and models based on variational autoencoders (VAEs) BB [TI These approaches leverage extensive small dataset training to transfer knowledge effectively across tasks during prediction. Neural networks (NNs) offer additional advantages by offloading computational intensity to the training phase, simplifying predictions, and freeing the model from Gaussianity constraints.
14 15
Building upon the foundational work of Conditional Neural Processes (CNPs) and Neural Processes (NPs) [5), numerous studies have enhanced NPs by integrating advanced mechanisms. Atten- tive Neural Processes (ANPs) (27 incorporated attention mechanisms better model long-range dependencies, while Transformer Neural Processes (TNPs) 36) leveraged self-attention for improved scalability. Convolutional Neural Processes (ConyCNPs) I0 adapted convolutional architectures to excel in spatial data tasks, and Neural Diffusion Processes (NDPs) [7] introduced diffusion mecha- nisms to enhance uncertainty estimation. NPCL 22 extended NPs to continual learning, enabling
"Corresponding authors: Liping Jing; 'These authors contributed equally to this work. Codes:
https:
//github. h/g.g/tbb.ccm//leeeeeeeeeeeeeeee http:ccmm//ee0097//eeee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sequential task adaptation, and multimodal approaches like estimation across multiple modalities.
[25 generalized NPs for uncertainty
While these advancements significantly enhance the experimental performance of NPs, they primarily focus task-specific improvements through architectural innovations and meta-learning strate- gies (20 [6]. However, the generalization capability of NPs-a critical hallmark of their design-has received limited theoretical exploration. Existing works prioritize performance optimization, often neglecting the theoretical interpretability of NPs. For instance, many methods emphasize improving inductive biases or increasing model complexity without addressing the theoretical underpinnings of NPs' generalization ability [0] This gap underscores the need for a formal theoretical foundation better understand and generalize NPs across broader domains and tasks, paving the way for more principled and interpretable neural process models [44].
Information theory provides powerful framework for analyzing and quantifying uncertainty, of- fering rigorous tools to evaluate the generalization of models [8 [35]. This paper introduces an inforeiaion-thherrttee framework analyze and improve the generalization bounds of Neural Pro- cesses (NPs). Building on this analysis, we propose a novel approach that incorporates dynamical stability regularization, which explicitly minimizes sharpness through the trace and Frobenius norm of the Hessian matrix. Additionally, we demonstrate how noise-injected parameter updates complement this regularization by smoothing the optimization trajectory and improving gradient coherence. These insights are broadly applicable across NP variants and validated through extensive experiments on benchmarks such as ID regression, image completion, Bayesian optimization, and contextual bandits.
The main contributions of this paper are as follows:
We conduct an information-theorete analysis of NP generalization bounds, providing a rigorous perspective on their generalization capabilities.
We propose a dynamical stability regularization framework, complemented by noise-injected parameter updates, to enhance generalization by minimizing sharpness and improving optimization dynamics.
Extensive experiments on classic NP tasks validate our approach, demonstrating tighter generaliza- tion bounds and superior performance compared to existing methods.
2 Related Work
In this section, two relevant areas are briefly reviewed: neural processes and information- -theoretic learning.
Neural Processes Neural Processes (NPs) are probabilistic models designed to meta-learn distribu- tions over functions, offering data-driven alternative Gaussian Processes (GPs) 14 Unlike GPs, which rely on manually specified priors, NPs employ an encoder-decoder architecture to learn a family of functions, efficiently capturing uncertainty using neural networks. Conditional Neural Processes (CNPs) [14], the earliest variant, introduced a deterministic encoder-decoder framework but assumed independence among predictive outputs. Latent Neural Processes (NPs) [15] addressed this by introducing a global latent variable for uncertainty modeling. Subsequent works have enhanced expressiveness, scalability, and uncertainty modeling. Extensions of NPs have addressed specific limitations and broadened their applicability. Attentive Neural Processes (ANPs) [27] incorporated attention mechanisms to better model long-range dependencies and heteroscedastic uncertainty. Convolutional Neural Processes (ConvCNPs) [10] introduced wwwsslaaioo-euvvrriiaa architectures, excelling in spatial and structured data tasks. Transformer Neural Processes (TNPs) (36) leveraged self-attention for scalability and sequence modeling, while Neural Diffusion Processes (NDPs) 7 used diffusion mechanisms to improve robustness and uncertainty estimation. Neural Processes for Uncertainty-Awe Aware Continual Learning (NPCL) [22] extended NPs to sequential task adaptation without catastrophic forgetting. Additionally, multimodal extensions [25] generalized NPs to handle multi-sensor fusion and uncertainty estimation. These advancements highlight the versatility of NPs and their adaptability across diverse applications.
cetimaatioo-thhoreeee learning (ITL) leverages principles from information theory to analyze and optimize learning algorithms. Unlike traditional approaches based on measures like VC-dimension (39] or uniform stability 4,, ITL characterizes learning using mutual information and related quantities [50, 38 By framing generalization error as the mutual information between inputs and outputs, ITI. captures dependencies among data distribution, hypothesis space, and learning algorithms, addressing challenges in uncertainty quantification, robustness, and data
Information- Citormation-Theorttee Learning

---

efficiency. This framework provides valuable insights into the generalization capabilities of modern machine learning models. Recent advances have expanded ITL's application to generalization analysis and enhancement. Harutyunyan et al. [8] proposed improved generalization bounds by focusing on the mutual information between predictions and the training set, offering practical estimates of the generalization gap in deep learning. Neu et al. [35] extended ITTL to stochastic gradient descent (SGD), deriving tighter bounds based on local gradient statistics and sensitivity along the optimization path. These advances demonstrate ITL's utility in understanding complex learning dynamics and improving theoretical guarantees for generalization.
3 Preliminaries
This section introduces the foundational concepts and mathematical tools necessary for understanding the methods and analyses presented this paper, including information theory, neural processes, and dynamical stability.
Information Theory Let Px denote the marginal distribution of the random variable X. For the Markov chain X - Y, the conditional distribution (or Markov transition kernel) is denoted by Pyjx, and the notation X J. Y indicates independence between X and Y [6] The cumulant generating function (CGF) of a random variable X is defined as t/x (A) & log Blea ([x((xRX))) where À is a real number. A random variable X is o-subgaussian if its CGF satisfies tix (A) S x²0= for all X E R 43
The mutual information between X and Y is defined as I(X; Y) A KL[Px.y PI Py), where Px.y is the joint distribution of and Y, and PxPy the product of their marginals (o.. The disintegrated mutual information between X and Y given U is 1 (X;Y) A k [lPxxuPriu) where Px.y u the conditional joint distribution given U. The conditional mutual information is then defined as T(X; Y U) & Be(T" (X;Y),, the expectation of the disintegrated mutual information over the distribution of
Neural Processes Neural Processes (NPs) model the conditional predictive distribution of target values y" at target points X based on a context set DC, expressed as P(y [x DC) [5]. For deter- ministic NPs (CNPs), the conditional distribution is simplified as P(y" X "
D = xx" (x,x)) r))),
where r is an aggregated feature of DC [14. Probabilistic NPs introduce a latent variable z to capture uncertainty, modeling Ply" P Poly"" IX", .z) Pz( dz. Training maximizes the evidence lower bound (ELBO): E (ajDE, During meta-training, NPs learn from tasks sampled from environment т, a probability measure over task distributions μu, where each task involves dividing data into context and target sets [20]. A meta-dataset consists of datasets Di.m. (Di..... D..), each sampled independently from pn,r, the mixture distribution induced by r M The meta learner A outputs a meta-parameter Palpu.m representing shared knowledge across tasks [3]. The meta risk is defined as (+) $ ED- [Enc (0))), where (0) & -B( log P(ule, DC) 3 The empirical meta risk is given by RD (0) _ 1 Σn= (0),
P(y X)] - K [L[ X
D [[5]
0 = A(D
(0). with RD, (0) A 1 > log P(ui Df). The meta generalization error is then defined as genmeta
NPs (T) e De (R-(0) RD (0)), which quan- tifies the difference between true and empirical meta risks 3. By learning a metaa-parameter 0 shared across tasks, NPs efficiently adapt new tasks, leveraging shared information in the environment.
Dynamical Stability Dynamical stability refers to the robustness of the stochastic gradient descent (SGD) optimization process small perturbations in the parameter space and has been shown to play critical role in both optimization and generalization 49 42). Consider the parameter update rule 0.41 6_- v/ R. (f.), where 7 is the learning rate, and (.. is the stochastic gradient. The sensitivity of the optimization dynamics can be quantified by the spectral norm || of the Jacobian matrix J, = 18,11 which governs how small perturbations 8, at step propagate through subsequent iterations 48 2id 40 47). When J. < 1, perturbations decay over time, leading to stable
training. Importantly, this stability imposes an implicit regularization effect, as SGD inherently biases the optimization process towards flatter minima with lower curvature, measured by the Hessian H V²)(0) [21 (6). Flatter minima are associated with better generalization due to their robustness to noise and parameter variations [9 g). Moreover, the Lyapunov exponents, which quantify the exponential rates of divergence or convergence of nearby trajectories in parameter space, provide a formal link berween stability and generalization. Negative Lyapunov exponents imply stable dynamics and a tendency towards generalizable solutions 42 49 This perspective highlights the central role of dynamical stability in understanding the implicit biases of SGD and its impact on both the optimization landscape and model generalization
3

---

4 Methodology
This section investigates the generalization properties of Neural Processes (NPs) by integrating cermattion-thorrete principles with optimization dynamics. First, we quantify the generalization error of NPs using mutual information (MD, capturing uncertainties from data and task distributions. Next, we introduce a risk-aware dynamical stability regularization term Rays, addressing sharpness and curvature to improve generalization. Finally, we propose an optimization-wwan noise injection strategy to enhance stability and guide parameter updates toward flatter minima.
4.1 Quantifying Generalization with Information Theory
Understanding the generalization capabilities of Neu- ral Processes (NPs) requires quantifying the uncer- tainties inherent in the learning process. By lever- aging mutual information (MI), the generalization error can be decomposed into components reflecting different sources of uncertainty, offering insights into the behavior of NPs across diverse tasks. While MI provides rigorous framework analyze generaliza- tion, standard approaches often neglect the impact of model complexity, which is particularly critical in the hierarchical nature of meta- meta-learming. This sec- tion lays the foundation for incorporating complexity constraints in future analysis.
000
04
Catern Emnrr
02
2.0
00
0122
205
664
1001
1204
Figure 1: Generalization error of Neural Pro-
To analyze generalization bounds, consider the relationship between true risk and empirical risk un- der given task distribution. Let ş represent an unknown distribution on the instance space X x Y. A meta-dataset consists of m datasets, denoted as Di.m = (Di...., Dm), where each dataset D, consists of n independent samples drawn from μr. The full dataset thus D Di.m [D,} For hypothesis 0, the true risk is defined R,, & 1RxxY [E(,, X Y)], and the empirical risk as (0) _ >x= E(O, vi.. Vi.)), where & : e x X x Y - TR is the loss function. learning algorithm A maps the dataset to randomized hypothesis 0 A(D) Payp, and its generalization error is defined as gen(u, A) A E. p Ry (0)]-
cesses (NPs) varies with the Hessian trace
Theorem 4.1. If the loss 8 x Y) is o-subgaussian for each 0 O with respect to X X ~ μl, the 2n
generalization error of a learning algorithm A satisfies the bound igen(p, A)l S v mn 1(;; D), where I(0:D) is the mutual information between the dataset D and the hypothesis 0. For meta-learning tasks, where μ ~ T, the meta- generalization error satisfies:
1202 mn
NPs gen (T, AT)
I(0; D1.m).
The mutual information [(e; Dr.m) can be computed based on the joint distribution P(0, Da.m.) and the marginal P(0); detailed derivations are provided in the appendix. This bound indicates that minimizing the dependence of on the data (via I(0; D1.m)) improves the generalization performance.
Although the MI framework elegantly quantifies generalization, it does not explic- itly account for the hypothesis space com- plexity or the solution sharpness. For NPs, these factors are especially significant as a result of their hicrarchical structure. The hypothesis space in NPs, determined by the meta--araamtter 0, encodes both task-level priors and task-specific adaptations. This space is inherently nigh-dimensional, mak- ing it prone overfitting, particularly when the meta-dataset Di.m is small or lacks di- versity.
Dum
Contest Points
Targe: Points Task
@o
Skk-foome Dynemiza Statiity Degulanration
Injecton Learning Strttgy
82
8
Figure 2: The training process of Gen-NPs
The sharpness of the learned solution
related to the curvature of the loss surface, quantified by the Hessian H 4²R(0). A high Hessian

---

trace implies sensitivity to parameter perturbations, adversely affecting generalization, especially across diverse task distributions. As illustrated in Figure [ the generalization error of NPs deteriorates with increasing Hessian trace.
Based on the above analysis, focus on improving NPs' generalization from both risk-aware dynamical stability regularization and optimization-awarc noise injection learning strategy, the whole learning framework is given in Figure
4.2 Risk-Aware Dynamical Stability Regularization
Inspired by previous works that demonstrate the critical role of sharpness and curvature in optimization
and generalization 126 23 45 we introduce dynamical stability regularization (DSR) term Ray that incorporates model complexity constraints into the generalization bound. Specifically, sharpness of the solution has been linked to poor generalization, motivating the need to explicitly regularize the curvature of the loss landscape [19] 21 Morcover, recent studies have explored how Lyapunov stability and noise in optimization dynamics can implicitly bias solutions towards flat minima, further supporting the necessity of stability- based regularization [40 49].
Definition 4.2. The dynamical stability regularization term Raya quantifies the complexity of the hypothesis space by leveraging properties of the Hessian matrix H, which represents the second-order derivatives of the loss function with respect to the parameters 0:
Rayn A1 B[Tr(H)) + A2 [|]]]
where (r(HI) (the trace of the Hessian) measures the solution sharpness by summing the eigenvalues of H, and || (the Frobenius norm) captures the overall curvature of the loss landscape. The hyperparameters Å and A2 control the relative importance of these two components, with empirical values typically set
[0.01, 0.1] and λ [0.001, 0.01] to maintain an appropriate balance between the two terms. By penalizing sharpness and curvature, Rdyn aligns the theoretical bounds with practical observations in optimization.
Theorem 4.3. By incorporating Raw into the infornation-theoretti framework, the refined meta- generalization error bound for Neural Processes (NPs) is given by:
2a² (0:Dimm) (T, ADSR) S
gen gen meta
NPs
(3)
nm 1 + a Ram
where 1 (0; Di:m) the mutual information between the meta-parameter 0 and the dataset Dimma Ramm is the dynamical stability regularization term, and a > 0 is scaling factor that controls the inftuence of Rayn- The proof of this theorem is provided in the Appendix.
Including Raya in the generalization bound has notable effects. Penalizing high Tr(H) encourages the model to select flatter minima, which are associated with better generalization. Additionally, con- straining the Frobenius norm [| prevents overfitting to high-curvatur regions. This regularization complements the implicit bias of optimization algorithms, such as gradient descent, which naturally steer solutions toward regions of lower sharpness.
The role of Raya depends the training regime. When the number of tasks m is large (m + oo) with fixed samples per task n, the mutual information term [(;; Di.m) dominates the bound, and Rays constrains the sharpness across tasks. Conversely, when grows (n + oc) while m remains fixed, the generalization error decreases naturally as task-level uncertainty reduces, making Raya less critical.
From a practical perspective, incorporating Rayn into the generalization bound provides actionable insights for optimizing Neural Processes. Maximizing Rayn during training improves generalization, and tuning the hyperpurameters A and A balances sharpness reduction and complexity control. Ultimately, Rasa bridges the gap between theoretical generalization bounds and practical challenges, leading to tighter bounds and a deeper understanding of the factors influencing generalization in NPs.
4.3 Optimization-Awar Noise Injection Learning Strategy
Motivated by the eslabestabl relationship between noise injection and generalization [9|1 46) we propose an optimization-awanl noise injection strategy tailored for meta-learning tasks. Noise has been shown to play a critical role in escaping sharp minima (26), smoothing the loss landscape (2]] and improving generalization through implicit regularization [47. Furthermore, recent studies highlight the importance of integrating noise injection into gradient-based optimization to ensure stability and guide parameter trajectories towards flatter regions of the loss landscape [24 49)
5

---

Building upon these insights, we explicitly incorporate noise into the parameter update rule to enhance both the stability and generalization of meta-pat parameter optimization. The overall training process of NPs, incorporating noise injection and dynamical stability regularization, is illustrated in The pseudocode for the entire algorithm can be found in the Appendix.
Figure 2
To explicitly incorporate the effect of noise into the learning dynamics, we modify the standard gradient- parameter update rule by introducing an isotropic Gaussian noise term 5 at each iteration s. The parameter update rule for meta-parameters 0 at iteration is formally given by:
0" = 00=00-1- 7/s V RRD? (0- + Rayn] + E"
where Roy (8*-))+ + Rayn is the empirical risk computed the target set D.' of task i, 7h is the learning rate at iteration s, and 5* ~ (0, Is)) an isotropic Gaussian noise term with variance of and dimensionality k. This update rule consists of two key components: the gradient descent step, which minimizes the empirical risk, and the noise injection step, which perturbs the parameter updates. The variance of can be adjusted over iterations control the strength of the perturbation.
The introduction of noise into the parameter updates serves two primary purposes. First, provides implicit regularization by encouraging the model to explore flatter regions of the loss landscape and favor minima with lower sharpness. This aligns with the objective of maximizing Rayn: as described in Section 4. Second, noise facilitates escaping sharp minima, which are common in high-dimensional parameter spaces and often lead to poor generalization. By perturbing the trajectory of the updates, noise helps the optimization process avoid these undesirable regions.
To achieve the desired regularization effect without destabilizing the optimization, the variance of of the noise is carefully scaled with respect to the learning rate 7/.- A common choice for this scaling of = n//77, where y > is a scaling factor that controls the relative strength of the noise. This formulation ensures that the noise magnitude decreases as the learning rate decreases, allowing the optimization to stabilize in later iterations.
The noise-injected parameter update rule in Eq. directly influences the dynamical stability reg- ularization term Rayn by affecting the Hessian properties of the loss landscape. Specifically, the perturbation introduced by 5* reduces the likelihood of converging to regions with high Tr(H), thus implicitly maximizing the sharpness of the solution. Furthermore, noise smooths the optimization trajectory, reducing the overall curvature of the solution as measured by || Together, these effects demonstrate the critical role of noise injection in improving both the stability and generalization performance of meta- meta--arammtter optimization. While introducing noise and regularization incurs additional computational overhead, the performance gains justify these costs (detailed analysis in Appendix).
5 Deeper Analysis
In this section, we establish a theoretical foundation for understanding how noise and dynamical stability regularization affect the generalization performance of NPs. By analyzing the interplay between noise, sharpness, and mutual information, we demonstrate their influence on generalization bounds. Additionally, we show how noise injection mitigates gradient incoherence, a key challenge in meta-learning, to improve stability and generalization.
5.1 Theoretical Analysis
We analyze how noise-injected parameter updates affect the dynamical stability of the optimization process and its impact generalization performance. Specifically, we focus their effects on sharpness, curvature (captured by the dynamical stability regularization term Raya), and mutual information (0; ).... This analysis highlights the role of noise in reducing both Rayn and 1, thereby improving the generalization of NPs.
To begin, we linearize the parameter updates around the optimal solution 0", which minimizes the true risk R,, Let 8' = 8" - 8" represent the deviation from 8¹ at iteration s. The linearized noise-injected parameter update rule is:
getl = H1 E"
(5)
where H [fi + (0")
) is the Hessian of the empirical risk, and 5* ~ NO, (3 is isotropic Gaussian noise. This form separates the contributions of optimization dynamics (T - 17, H) and noise ç" to the evolution of 0

---

Noise affects sharpness, measured by the trace of the Hessian Tr(H). by perturbing the optimization trajectory and encouraging exploration of flatter regions. This perturbation has a dual effect on the loss landscape: it locally reduces the curvature around specific minima while simultaneously increasing the system's overall stability by promoting robust, flat solutions. Consequently, the effective Hessian trace in the presence of noise satisfies mise) 2 Tr(I)- (1 + n,02). where a higher variance of noise of results in a greater enhancement of dynamic stability. Similarly, the Frobenius norm ||l which provides a comprehensive measure of the curvature of the loss landscape, satisfies | 2 ||H]| / n.o indicating that noise enhances the system's 's ability to maintain stability in the face of parameter perturbations. These results suggest that noise injection encourages the optimization process to explore flatter regions of the loss landscape, develop higher dynamic stability, and improve generalization performance by enhancing robustness to parameter perturbations.
By enhancing both stability metrics, noise effectively amplífies the dynamical stability regularization term Rayn, which appears in the denominator of our generalization bound. Specifically, substituting the enhanced trace and Frobenius norm into the definition of Rayn yields:
Rayn, noise 2 A1 ([) (1 + nTo + A2 [[( V1+
This increase in Rdyn directly tightens the generalization bound by increasing the denominator term 1 + Reyn. resulting in improved generalization capabilities for Neural Processes.
As the noise variance of decreases over iterations (o² = na/7), the optimization stabilizes near a flat minimum, with the stability benefits retained enhanced Raya. Noise also reduces the mutual information [(; Di.m), which quantifies the dependency between model parameters 0 and training data 1.... High mutual information indicates potential overfitting, parameters become overly dependent on specific data. The reduction is given by:
nosie (0; D Dism) S max (o, (0; D Dim) - -oo03 R (0)11))
This ensures Inoise (0; Dim) remains non neg-nigte and significantly lower than the original mutual information. Excessive noise or high learning rates, however, may destabilize optimization, limiting these benefits. By jointly reducing [(;; Di.m) and amplifying Ray through increased Tr(H) and || lle, noise-injected updates tighten the generalization bound:
202 Lnoise (0; D 11:m)
202 1(0; Di.m)
gen
Gen NPs
(T, A)) S V num + a Rayn, noise
V mum 1 + C - Rayn
(8)
where the second inequality follows from Incss D1.m) S I(0; D1.)) and Reya. 2 Ram- This interplay between noise, mutual information, and dynamical stability is crucial for achieving tighter generalization bounds and improving meta-learning performance.
5.2 Gradient Incoherence
Gradient incoherence, caused by inconsistencies between gradients computed on the context set (DC) and the target set (D(),, affects the stability and generalization of Neural Processes (NPs). This issue can lead to suboptimal updates and hinder learning. Noise-injected parameter updates mitigate this problem by smoothing the loss landscape and reducing gradient mismatches.
Definition 5.1. The gradient incoherence for a task at iteration s is defined as the fn-norm of the difference between the gradients on the full dataset D; and the target set DT:
E. = | (OTB,,(0'11) + Rayml - V[Rp (0--1) + Rayml
(9)
where V VRD (8-11) + Rave and VinT (8-11) + Rayn are gradients of the empirical risk over D. and D.. respectively. The overall gradient incoherence across tasks and iterations is given by GI m 1= S <.... where m the number of tasks and S the number of iterations. Minimizing GI ensures more consistent updates, improving generalization.
Noise reduces gradient incoherence by perturbing the optimization trajectory. The noise g" introduced during updates disrupts gradient alignment, exponentially decreasing incoherence as noise variance
n²03 of increases. Formally, the reduction can be expressed as <. S <f,,(0) exp(= -?00- ). where ()) the initial incoherence. Larger noise promotes exploration and reduces mismatches early in training, while smaller noise stabilizes updates in later stages. This reduction stabilizes parameter
7

---

Original
Task
CNP
Gen-CNP
Gen-NP
ANP
Gen-NNP
NP
Gen-NP
wpw
Regression Results
(b) Image Completion Results CelebA Datasets
Figure 3: Part results of ID Regression and Image Completion.
updates, prevents oscillations caused by inconsistent gradients, and improves alignment with true risk minimization. Additionally, noise-induced coherence complements the regularization effect of Rays by further reducing the sharpness and curvature of the loss landscape.
The effectiveness of noise depends scaling its variance of with the learning rate 7/.- Setting of = ///yy balances noise magnitude across iterations, ensuring exploration in early training and stability later. Empirical results in Section 6 confirm that noise-injected updates significantly reduce GI, leading to improved stability and generalization across tasks.
Experiments
The proposed Generalization Neural Processes (Gen-NPs) are evaluated and compared with other methods in the NP family tasks such regression, image completion. Bayesian optimization, and contextual bandits. These tasks are widely used to benchmark NP-hased models, as demonstrated in prior works. The comparison includes Conditional Neural Processes (CNPs) 14], Neural Processes (NPs) [15. Attentive Neural Processes (ANPs) (27), Bootstrapping Neural Processes (BNPs) 30), and Transformer Neural Processes (TNPs) 36 In addition standard task performance, analyze the gradient incoherence (GI) introduced in Section 5.2, highlighting how Gen-NPs improve optimization dynamics and generalization capabilities compared to other methods. The experimental setup is consistent across all methods, and the implementation leverages the official codebase of TNPs. Due to space limitations, extensive experimental results, including detailed images and tables, are provided in Appendix C.
6.1 1-D Regression
We evaluate Gen-NPs on a 1-D regression task, training models on RBF kernel functions sam- pled from a Gaussian Process (GP) and testing on unseen RBF, Matérn 5/2, and Periodic ker- nels. Metrics include log-likelihood (LL) for predictive accuracy and uncertainty, and gradi- ent incoherence (GI) for optimization stability. Results are averaged over five random seeds with standard deviations reported. 1-D regres- sion results on NP model are visually illustrated in Fig. Bj), where Gen-NP demonstrate supe- rior ability to capture the underlying character- istics of the data compared to baseline models. As summarized in Table I Gen-NPs consistently outperform baseline models across all kernels and metrics. For example, Gen-NPs achieve higher LL values Matérn 5/2 and Periodic kernels, while also demonstrating reduced GI, indicating improved optimization dynamics and generalization. These results validate the effectiveness of the proposed general recipe. Additional figures and tables, including detailed results for MAE, RMSE, and variability across random seeds, are provided in Appendix C.2.
Table 1: Comparison of Gen-NPs with the basclines on
LL and GI of the target points on various GP kernels.
Method
RBF-LL Matérn-LL
Periodic-LLL.
GI
CNP
0.265
0.045.20.014
1.435
.800000027
Sen-CNPO.2866
20.00000110 20.00
-1.188
#0.025
0.830
20.042
NP 0.240 022
0.051
=0.019
1.145
0.490+0
Gen-NP 00.270
0.073
-1.125
0.470
ANP 0.806 oos
(1630
20.004
5.320
280
0.973g0.04
Gen-ANPP..812:00 200
0.63610.002
5.0028
200
10-501
BNP D.389+ 017
0.
185
0.970+0.01o
0.1000000000..0 o0.07
Gen-BNP 20.000
0.200.0.00s
-0.46620.000
0.148
20.48:0.00m
TNP 1.650
onG
1.218
00.00
2.320
0.750 0.008
Gen 1.662 o0.02 1.226+0.008
2.010
0.730+0.000
6.2 Image Completion
We evaluate Gen-NPs on image completion tasks using CelebA 32 and EMNIST [.] formulated as a 2-D regression problem where pixel coordinates are inputs and intensities are outputs [15] CelebA downsampled to 32 32, while EMNIST uses training classes (0-9) and evaluates generalization on unseen classes (10-46). Figure b)) shows partial results CelebA, where
8

---

Table 2: Comparison of Gen-NPs with the baselines LL and GI of the target points on CelebA and EMNIST.
CelebA
EMNIST
Method
LL
G1
Seen-LL
Unseen-LL
G1
CNP
2. 2.160
1.399
0.737
0,485
0.466
Gen-CNP
2.188
2.390
0.786
0.556
0.410=0.035
NP
2..011
0.69-10.034
0.79510.002
0.584
0.187
Gen-NP
2.52440.008 00.64
0.814=0.0006
0.603.00.00
40.010
ANP
2.9214:0.00 1.989
0:0000
0.884 0.000
0.526
Gen-ANP
2.064
1.816=0.025
00004
0.886+00.004
0,468=00.034
BNP
20.00
22.835
0.870
0.716=0.012
0.2822000228
Gen-BNP
2.776.0.000 22.565
2:0.68
0.905 000008
0.764 0776440.000 0.000
0.279 000016
TNP
4. 1..04
94.1011:2:22
1.550.10.004
1.419
80.995 666595
Gen-TNP
4.409
92.364_33.650
1.555_0.00m
1..03
65.874_.4.382
Gen-NPs demonstrate superior performance in image completion. As summarized in Table 2 Gen- NPs consistently outperform baseline models across both datasets, achieving higher I.I. values and reduced GI, demonstrating improved predictive accuracy and optimization stability. Visualizations in Appendix C.3 further highlight that Gen-NPs produce clearer and more precise image reconstructions, effectively enhancing generalization and accuracy for image completion tasks.
6.3 Bayesian Optimization
We evaluate the effectiveness of Gen-NPs in Bayesian optimization (BO) tasks, where the goal is to maximize black-box function fx) accessible only through evaluations without gradient information [2] 28, 29]. The experimental results on 2D Dropwave and 3D Ackley are shown in Figure 4a where Gen-NPs achieve lower regret compared to bascline methods. Results, detailed in Appendix C.4, demonstrate that Gen-NPs consistently outperform bascline models across all dimensions, achieving lower regret and faster convergence. These findings validate the proposed method's ability to enhance generalization and optimization performance in BO tasks 13
6.4 Ablation Study
We conducted ablation studies to validate the effectiveness of the proposed Risk-Aware Dynamical Stability Regularization (DSR) and Optimization Aware Noise Injection Learning Strategy (NILS) the results of the original method, Gen-NPs with only DSR,
on ID regression task. Gen-NPs with only NILS, and the full Gen-NPs with both modules included.
Table Ablation study results comparing the original method.
Method
LL (RBF)
G1 (RBF)
LL (Periodic)
G1 (Periodic)
Original CNP
0. g.2
0.880
-1.435
1.312+t.063
Gen-CNP (with DSR only)
0.2
0.85820.030
-1.428
1.20220.045
Gen-CNP (with NILS only)
0.219
0. 0.005
-1.423 0.002
1.151
Full Gen-CNP (DSR NILS)
0.286+0.010
0.830+0.042
-1.418
1.112+0.039
As shown in Table 9 and Appendix, these results highlight the significance of incorporating DSR for improving dynamical stability and NILS for robust optimization. The ablation study confirms the fcasibility and effectiveness of the proposed modules in boosting the overall performance of Gen-NPs.
6.5 Comparison with Stability Neural Processes
We compare the stability-based generalization error (SGE) 4 31 with our proposed Gen-NPs on ID regression task, emphasizing the differences in their noise introduction mechanisms and evaluation methods. SGE quantifies stability as the difference between test eтror and training error to measure generalization. Gaussian noise (mean O, variance 1) is added to 5% and 10% of the training data to assess robustness under noisy conditions. In contrast, Gen-NPs introduces noise during the parameter update process, focusing on the mutual information between model parameters and the training data to directly capture generalization behavior. Experiments were conducted on ID regression task with data generated using an RBF kernel, comparing models under original, 5% noise, and 10% noise conditions. Results, shown in Appendix C.6, demonstrate that Gen-NPs consistently achieves tighter bounds and higher log-likelihood (LL) values even in noisy settings. This highlights the robustness and effectiveness of Gen-NPs in capturing model generalization and stability under various conditions.

---

Original (Orighaa)
Original (Gen-r)
Noise (+5%) (Original)
Netse 1-5%) (Gen-1)
Neise in 30%1 (Original)
[-10%) (Gee-*)
Gen-IP
Gen-AMP
CVP
Get-OUP
(a) Results Bayesian Optimization
(b) Comparison of Log- Log-Likelihood Across Differ- ent Noise Levels
Figure 4: Part results of Bayesian Optimization and Comparison with Stability Neural Processes
Conclusion
This paper proposes an information-theeortiic framework to enhance the generalization capabilities of neural processes (NPs) by addressing both parameter optimization and regularization. Specifically, noise injection during parameter updates captures the mutual information between model parameters and training data, providing robust optimization strategy. In parallel, dynamical stability regular- ization mitigates overfitting and improves the optimization trajectory, collectively leading to better generalization properties. However, the framework currently focuses on supervised learning, and future work will explore its extension to reinforcement learning and large-scale datasets.
8 Acknowledgements
This work was partly supported by The National Key Research and Development Program of China (2024YFE0202000; The National Natural Science Foundation of China under Grant (62406019, 62436001, 62536001, 62176020); Beijing Natural Science Foundation (4244096); Young Elite Scien- tists Sponsorship Program of the Beijing High Innovation Plan. The Joint Foundation of the Ministry of Education for Innovation team (0091B04225)) The State Key Laboratory of Rail Traffic Control and Safety (RCS2023K006); the Talent Fund of Beijing Jiaotong University (202XXKR0075).
10

---

References
[1] Robert B. Ash. Information Theory. Courier Corporation, 2012.
[2] Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G. Wilson, and Eytan Bakshy. Botorch: A framework for efficient monte-carlo bayesian opti- mization. Advances in Neural Information Processing Systems (NeurIPS), 33:21524-21538, 2020.
[3] Jonathan Baxter. model of inductive bias learning. Journal of Artificial Intelligence Research (JAIR), 12:149-198, 2000.
[4] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[5] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In Proceedings of the International Joint Conference on Neural Networks (IJCNN 2017), pages 2921-2926 IEEE, 2017.
[6] Thomas M. Cover. Elements of Information Theory. John Wiley & Sons, 1999.
[7] Vincent Dutordoir, Alan Saul, Zoubin Ghahramani, and Fergus Simpson. Neural diffusion processes. In Proceedings of the International Conference on Machine Learning (ICML 2023), pages 8990-9012, PMLR, 2023.
[8] Gintare Karolina Dziugaite and Daniel Roy. Entropy-sgd optimizes the prior of pac-bayes bound: Generalization properties of entropy-sgd and data-dependent priors. In International Conference on Machine Learning, pages 1377-1386. PMLR, 2018.
[9] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adapta- tion of deep networks. In Proceedings of the International Conference on Machine Learning (/CM), pages 1126-1135. PMLR, 2017.
[10] Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-leaming stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems (NeurIPS), 33:8284- 8295, 2020.
[11] Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. Gp-vae: Deep probabilistic time series imputation. In International Conference on Artificial Intelligence and Statistics, 2020.
[12] Peter I. Frazier. tutorial on bayesian optimization. arXiv preprint,
2018.
[13] Jacob Gardner, Geoff Pleiss, Kilian Q. Weinberger, David Bindel, and Andrew G. Wilson Gpytorch: Blackbox matrix-matrix gaussian process inference with gpu acceleration. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018.
[14] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In Proceedings of the International Conference on Machine Learning (ICML 2018), pages 1704 1713. PMLR, 2018
[15] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, SM Ali Eslami, and Yee Whye Teh. Neural processes. In Proceedings of the ICML Workshop Theoretical Foundations and Applications of Deep Generative Models, 2018.
[16] Hassan Gharoun, Fereshtch Momenifar, Fang Chen, and Amir H Gandomi. Meta-learning approaches for few-shot learning: A survey of recent advances. ACM Computing Surveys, 56(12): 2024.
[17] Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard E. Turner. Meta-learning probabilistic inference for prediction. In Proceedings of the International Conference Learning Representations (ICLR 2018), 2018.
11

---

[18] Hrayr Harutyunyan, Maxim Raginsky, Greg Ver Steeg, and Aram Galstyan. Information- theoretic generalization bounds for black-box learning algorithms. Advances in Neural Informa- lion Processing Systems, 34:24670-24683 2021.
[19] Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-42, 1997.
[20] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE Transactions on Puttern Analysis and Machine Intelligence (TPAMI), 44)94:144))4 2021.
[21] Stanishaw Jastrzpbski, Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos Storkey. Three factors influencing minima in sgd. arXiv preprint arXiv:1711.04623, 2017.
[22] Saurav Jha, Dong Gong, He Zhao, and Lina Yao. Npel: Neural processes for uncertainty-aware continual learning. Advances in Neural Information Processing Systems, 36, 2024.
[23] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2020.
[24] Haotian Ju, Dongyue Li, and Hongyang R Zhang, Robust fine-tuning of deep neural networks with hessian-based generalization guarantees. In International Conference on Machine Learning, pages 10431-10461. PMLR, 2022.
[25] Myong Chol Jung, He Zhao, Joanna Dipnall, and Lan Du Beyond unimodal: Generalising neural processes for multimodal uncertainty estimation. Advances in Neural Information Processing Systems, 36, 2024.
[26] Nitish Shirish Keskar, Jorge Nocedal, Ping Tak Peter Tang, Dheevatsa Mudigere, and Mikhail Smelyanskiy. On large-bate batge training for deep learning: Generalization gap and sharp minima. In 5th International Conference on Learning Representutions, ICLR 2017, 2017.
[27] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR 2019), 2019.
[28] Jungtack Kim. Benchmark functions for bayesian optimization, 2020.
[29] Jungtaek Kim and Seungjin Choi. Bayeso: A bayesian optimization framework in python. Journal of Open Source Software (JOSS), 8(90):5320, 2023.
[30] Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. Advances in Neural Information Processing Systems (NeurIPS), 33:6606-6615, 2020.
[31] Huafeng Liu, Liping Jing, and Jian Yu. Neural processes with stability. Advances in Neural Information Processing Systems, 36, 2024.
[32] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(2018):11, 2018.
[33] S. Mishra, S. Flaxman, T. Berah, M. Pakkanen, H. Zhu, and S. Bhatt. Pivae: Encoding stochastic process priors with variational autoencoders. arXiv preprint arXiv:2002.06873, 2020.
[34] Radford M. Neal, Regression and classification using gaussian process priors. In J. M. Bernardo,
J. O. Berger, J. W. Dawid, and A. F. M. Smith, editors, International Conference on Learning Representations. Oxford University Press, 1998.
[35] Gergely Neu, Gintare Karolina Dziugaite, Mahdi Haghifam, and Daniel M Roy. Information- theoretic generalization bounds for stochastic gradient descent. In Conference on Learning Theory, pages 3526-3545. PMLR, 2021.
12

---

[36] Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty aware meta learning via sequence modeling. In Proceedings of the International Conference on Machine Learning (ICMI. 2022), pages 16569-16594, PMLR, 2022.
[37] Carl Edward Rasmussen and Christopher KI Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
[38] Daniel Russo and James Zou. How much does your data exploration overfit? controlling bias via information usage. IEEE Transactions on Information Theory, 66(1):302-323, 2019.
[39] Stephan Sain. The nature of statistical learning theory, 1996.
[40] Samuel Smith, Erich Elsen, and Soham De. On the generalization benefit of noise in stochastic gradient descent. In International Conference on Machine Learning, pages 9058-9067. PMLR, 2020.
[41] Samuel Smith, Benoit Dherin, David Barrett, and Soham De. On the origin implicit regular- ization in stochastic gradient descent. In International Conference on Learning Representations, 2022.
[42] Samuel Smith and Quoe Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
[43] Martin J. Wainwright. High-Dimensional Statistics: Non-Asymptotic Viewpoint, volume 48. Cambridge University Press, 2019.
[44] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Yu Philip. Generalizing to unseen domains: survey on domain general- ization. IEEE transactions on knowledge and data engineering, 35(8):8052-80722 2022.
[45] Kaiyue Wen, Zhiyuan Li, and Tengyu Ma. Sharpness minimization algorithms do not only minimize sharpness to achieve better generalization. Advances Neural Information Processing Systems, 36:1024-1035, 2023.
[46] Andrew G Wilson and Pavel Izmailov. Bayesian deep learning and probabilistic perspective of generalization. Advances in neural information processing systems, 33:4697-4708, 2020.
[47] Jingfeng Wu, Wenqing Hu, Haoyi Xiong, Jun Huan, Vladimir Braverman, and Zhanxing Zhu. On the noisy gradient descent that generalizes as sgd. In International Conference on Machine Learning, pages 10367-10376, PMLR, 2020.
[48] Lei Wu, Chao Ma, et al. How sgd selects the global minima in over-parameteriized learning: A dynamical stability perspective. Advances in Neural Information Processing Systems, 31, 2018
[49] Lei Wu and Weijie Su. The implicit regularization of dynamical stability in stochastic gradient descent. In International Conference Machine Learning, pages 37656-37684. PMLR, 2023.
[50] Aolin Xu and Maxim Raginsky. Information-theoreti analysis of generalization capability of learning algorithms. Advances in Neural Information Processing Systems (NewrIPS), 30, 2017.
[51] Huanrui Yang, Xiaoxuan Yang, Neil Zhenqiang Gong, and Yiran Chen. Hero: Hessian- enhanced robust optimization for unifying and improving generalization and quantization performance. In Proceedings of the 59th ACM/IEEE Design Automation Conference, pages 25-30, 2022.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Please refer to Section Introduction and Abstract.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer to the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Justification:
Please refer to the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper clearly explains how to reproduce the algorithm. We will open source the code as much possible.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We will provide them in the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
iossiinnnoiyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppi.ccppulli//uuuiu wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Most experiment setting and details are in the Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We conducted experiments in multiple models and multiple data.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See the Experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: /tte:::///euis..eeeeeeeee httpp//s..ccppullcccctccssttttta
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics all respects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Please refer the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets we use are licensed.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification: We will open source the data and code as soon as possible.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification: Our core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non- standad components.
Please refer to our LLM policy hattps://neurips... 200ps//ee///0ooeeeeeee////2///// for what should or should not be described.
20

---

Appendix
The Technical Appendix is organized into five key sections. In Section A, present essential foundational concepts and lemmas. Section B provides detailed derivations and proofs of the theorems discussed in the main paper. Section C offers comprehensive supplement and further explanations the experimental results described in the paper. Section presents a detailed analysis of the computational complexity of our proposed approach. Finally, Section E includes the complete algorithm pseudocode for implementing Generalization Neural Processes.
A Lemma
In this section, we present essential foundational concepts and lemmas.
A.1 Variational Form Mutual Information
Let X and Y two random variables. For all probability measures Q defined on the space of X, we have
I(xY) [XL(((x)))))
with equality for Q = Px.
Proof
p(a,z)
I(XY)) KI(Px11)
p(z, v) log p(x)p(u)
dr dy
+ / p(n) log p(a) dr ((x)
/
p(z, v) log p(a.y)
dz dy
p(z)p(3)
/
p(r) dx dy
p( v) log
g(x)
p(alu) dz dy
p(z, v) log
(xx)
By [x Since KL((x(IQ) 2 0, the equality exists only when Q = Px. which concludes the proof.
A.2 Conditional Mutual Information and Its Variational Form
Let X, and Z be random variables. For all Z-measurable probability measures Q on the space of X,
(x;))) < Ey ( (KLxxxx())))
with equality for Q = -Pxzz-
Proof
XLLP) 2112)
// (( log p(x, v12) dady
((x)(((2)))
/ p(al=) log p(x|2)
dr
((x)
// p(a, v|== log p(a, ulz) dady ((=))(()(2)
/ʃ p( log
p(x| z)
dady
((x))
// p(a, vlz) log
p( z) dady
q(x)
= ( 2)11))
Since (L(Pxiz IQ) 0, the equality exists only when Q Pxiz, which concludes the proof.
21

---

A.3 Mutual Information Under Conditioning
Let X, Y, and Z be random variables. For all Z-measurable probability measures Q defined on the space of X.
T(X;YIZ) = B (X;Y) s By (xzx(KT((xxxxxx100)
with equality for Q Px|z-
A.4 Kullback-Leibler Divergence and its Representation
Let P and Q be two probability measures defined a set X. Let g X + IR be measurable function, and let (x-qlexp(g(XX)) s oc. Then
KL(PIIQ sup [Ex~rla(X]] log log_xxxolesp(((x))
A.5
Data Processing Inequality
Given random variables X, Y, Z, V, and the Markov Chain:
then have
( ((X;)), (1;:))1((:22
For Markov chain
V X + Y +4
we have
T(X:2]V ((XYII), T(X;4[V S ((Y
Proof. Since
T(X;Y22) T(X; - ( ((X;2)))
and with the Markov Chain, we Y therefore
((X:2I) H(XIV) H(XIV, 7) =0
In addition, I(X;YIZ) 2 0, so [XX; Z) - ((X;))
7(2; = [(Z;X)= ((ZVV]))
=I(Z;Y)+ + (((:)(()
=((;2))
with 1(;;2/7) 2|X) 2 0, we have 1(X; < I(Y; Similarly, for the second Markov chain, we have X 1 YIV. therefore
I(X; V) H(| V) H(||Y, V) = 0.
1( = [(X;Z(V + ((;YY[V,))
= =T(X;YIV) + T(X;
-I(X;
So we have I(X; S VV) the rest proof is similar and omitted.
A.6 Conditional Independence and Information Bounds
Given random variables X, Y, 71, Z2, and the graph model:
- +Y,
then have
I();Y < x1(x)112)
22

---

Proof.
Apply chain rule, we get:
=((: [1xx=_)))++((xx+122222
From the graph model, we have Y 1 Z1, Y 1 Z2 and (X,Y) 1 Zz. Hence
I(x; (2 Zi) = H(X 22. 21) -X (2,
(2) H(X Zz)
Moreover,
1(x;Y,2%/z)) -[Xx22124+++X+,,,,,,
[[);
=I(72xX|V,
(122;X|V,
the last equality is obtained with Y 1 Z, 22, and since T(Y; X Z1) 2 0, we get
[(X; Y, Z1).
Consequently, we have I(X;YIZı) I(X; conclude the proof.
A.7 Donsker- Donsker-Varadhan Representation Mutual Information
Varadhan
The Donsker- Varadhhan representation provides powerful variational characterization of mutual information that is particularly relevant our Gen-NPs framework. This representation allows us to derive tractable lower bounds on mutual information, which essential for the information-theoree analysis of neural processes.
Let X and Y be random variables with joint distribution Px,.. The Donsker-Varadhan representation states that:
1(x;Y)
sup -3
BPx, [T Y)) log Eox Epx (e (x)))
where the supremum is taken over all measurable functions for which the expectations exist, and Px @ Py denotes the product of marginal distributions.
Relation to KL Divergence This representation directly connects to the KI. divergence representa- tion in A.4, as mutual information is the KI. divergence between the joint distribution and the product of marginals: T(X;Y) = KL(Pxxy Px @ Py).
Application to Gen-NPs In our Gen-NPs framework, we leverage this representation analyze the mutual information between model parameters 0 and training data S. By setting = and
Y = S, we can derive:
T)0:5)
sup T:xxS-R
Era,s (T). - log Er,or T(0.s)
Estimation This representation forms the theoretical basis for our noise injec- tion learning strategy. When we introduce parameter noise during optimization, we are effectively using a specific form the function T that facilitates estimation of the mutual information between model parameters and training data.
23

---

Proof Sketch function:
The proof follows from the convex duality principle and properties of the logarithmic
1(XY)) KL(PXx, &
= = p(a, v) log p(z)p(y)
pla,y) dy
=sup { p(a,y) dx dy / p(x)p(u)(e do dy
sup [Exx. T(X. log Eexory (errxmn)
Connection to Generalization Bounds In analysis of Gen-NPs, this representation enables us to derive generalization bounds that explicitly account for the information complexity of the learning algorithm. Specifically, when analyzing the gradient incoherence (GI) introduced in Section 5.2, implicitly utilize this variational characterization to establish the connection between noise injection and generalization performance.
This representation central to our theoretical framework as it provides the mathematical foundation for understanding how parameter noise affects the information bottleneck between model parameters and training data, ultimately improving generalization capabilities of Neural Processes across diverse task distributions.
B Theorem Proof
This section provides detailed derivations and proof' of the theorems discussed in the main paper.
B.1 Proof of Theorem
If the loss ((0, X Y) is o-subgaussian for each 0 O with respect to X Y μ., the generalization error of a learning algorithm A satisfies the bound Igen(p, A)] V 2n [(0; D), where I(0; D) is the mutual information between the dataset D and the hypothesis 0. For meta-learning tasks, where j ~ r, the meta-geneeallzztion error satisfies:
20 [(;; D1:m).
NPs gen meta (T, A)
The mutual information I(0; Di.) can be computed based on the joint distribution P(0, Di.m) and the marginal P(0); detailed derivations are provided in the appendix. This bound indicates that minimizing the dependence of 0 on the data (via [00; D Di.)) improves the generalization performance.
Proof Let E @ be random variable representing the meta meta-pareee parameter Iearned from the datasets Di... and let 0 E θ be independent copy of 0 such that 0 is independent of the datasets Di... The distribution of is the marginal distribution Po. which is averaged over the possible datasets Di.m drawn from the environment
The mutual information [(;; Di:m) quantifies the dependency between the meta-parameter 0 and the observed datasets Dı.m- Specifically, it measures how much information about 0 is gained by observing the datasets D Dim- This dependency impacts the meta generalization error, as the error is influenced by the extent to which @ captures relevant information from the datasets while avoiding overfitting.
To express the meta generalization error, consider the function:
(0..D..m) def
Ep, (DD((0)
Σ Er. [e(0, Z:)].
For any X R, let
08 (2) del log Eg
x1 (C((A.D...) (s.m) -R, 11)
= logg Eg Bi.
Di [F(0. Dr.m)
24

---

Moreover,
I(0; Di:m) DRa. (Po PoPpom
sup { Di. (g(0, Di:m) )] - log R Dim [cald.Dmm)]
2 Drom Prom [f(e, D1m)) )} - log D (c)(0.)1m)) YAER
1
= XEg,
ED. [BD;
776
XE Dim m
Ep,
R2, (8)
V/ Dim (A). (I)
Since (O): i= I...., m are mutually independent given t, and Di...., Dm are independent, we have
m p(0 7).
p(0) 1:mm 7)
Hence,
XBa,Dism
Ro. (0)
1=1
Dimm
ED. [R],
m7
AB Dr:m
[ (0) (2)
Since i Dimm. have that
Da Pl.
Hence,
R2 (0) = EDE Eamp 610
[0)
Epur [0]
Therefore,
1 RD. (0) = XB3 [R](0]]
XB Piom m
ABo.Di.m [R)] (3)
If we use Equations (2) and (3), then Equation (1) becomes
- (f) Re (0)
I(00 D1.m)
th Di.. (A), E.ER. (4)
Since this inequality is also valid when X is negative, this implies that we also have
Ea. (0) Ro (0)
T(0: Dr.m) + VoD (-11], A1 0.
Consequently,
1 [(0: + V...i. (-1)], (-1)) AA >
NPs gen (7, A) x
Since É(0, Z) is o-subgausiian, we have that
(10 1=1
(1)
2_ Subgausssi... Hence, Vam
2z0²
V 0.P.. (A) S
VA E R.
2nm
25

---

Thus, we have
110; mm
gen meta (T, A) S X
YA > 0.
2nm
By using the value that minimizes the r.h.s. of the above equation, we have
NPs gen meta (T, A) 1 20 Drmm)
(5)
Returning to Equation (4), we have for À > 0:
B (R) RD (0)] 2 - [10: Dimm) + 4/ Di:m (a)
1
2202 1202 D1o
nmm
Hence, we also have
2 T Di.m)
gen
NPs
(T.
(6)
nm
Then, Equations (5) and (6) together imply that
10; Di.m)
Igen (T, S
NPs
which gives the theorem.
B.2 Proof of Theorem 2
Theorem B.1. By incorporating Ram into the information-therreic frameeeeeeerr framework, the refined meta- generalization error bound for Neural Processes (NPs) is given by:
202 I(0; (0;
gen
MPs (T, ADSR) S
nm + y Rayn
where [0; Dymm. is the mutual information between the meta mneaa-parameter and the dataset Vom- Rayn is the dynamical stability regularization term, and Y 0 scaling factor that controls the influence of Ray- The proof of this theorem is provided in the Appendix.
Proof Let 0 E θ be a random variable representing the meta-parameter learned from the datasets Dimm. and let 0 e θ be an independent copy of such that is independent of the datasets D1.m- The distribution of ê is the marginal distribution Pa-
We define the effective mutual information Letr( (e;00:Dpn 00;Di1a 11mm
1(0; D1m)
Lef (0; D D1.m) = 1+7 Rayn
where Rayn A [[r(H))++
+1 E H|]]] quantifies the complexity of the hypothesis space through the trace and Frobenius norm of the Hessian matrix H.
To express the meta generalization error, consider the function:
f(0, D 1:m) del ED. RRD (R)
1 Σ m
ED. (((0,2))
m7
For any À E R. let
V Dum (A) def log Eg Bi.m
c P H))
=log Eg D.
(2((0:Dem))
AB. Drmm
(1m))
26

---

For the standard mutual information, we have:
sup [(;; 1.mm
DxL (Pa.D.. Pa.
P
RR 10(0, 1=mm] - log RB Di.m
AEg Drom [f 1.m)) log Di. imm (AA(ORLmm) AA E R
AEo Diom
ED, [RD, (0)] A
Ep,
RD, [R) Vo. (1). (A).
When we incorporate the dynamical stability regularization, the effective mutual information is:
T(0: D1:m)
(8; Dimm)=
1+ 7 Rdyn
XIE Dum [f(0, 1imm) - log E8g Pimm
+ Raya
By the data processing inequality and information bottleneck principles, dynamical stability reg- ularization effectively reduces the mutual information through constraining the parameter space complexity:
[0; D1mm)
tree (0; D1mm) S
1 1 Rayn
Continuing with the proof, we have:
(1 + 7 Raya) Toff (0; D 11m) 2 Drmm [5 11m 1 log R. Dim
(
Since De), 1..... are mutually independent given T, and Di,..., Dm are independent, we have
p(0 l:m
II p())...
Hence,
1
1
XE Damm
RD. (0)
>EE.D.... Dr.m
Ep E 10, [RD (0)
m
771
AE Dim
[RD (0)
Since 0 11. Dimm. we have that
P Pappi. = Pa-
Hence,
R2 (0) ED Eirron PopD [Q)
Eper E. B@wp Ry, (0)
Therefore,
D Drom
Rp, (0) = XE R. (0)
1=1
AEE Dr... [R))
27

---

Combining these results, we get:
B Dimm [R-(0) RD (0) - (1 + Raya) Ten(0; (1;Dim) D. (2)
[(; D1:m) w/ 12 (2);
AA E RR
Since this inequality also valid when is negative, we also have:
1 [(0: Dimm) Vo.D (-1)]; AA
Eg RR (0) RDD... (0) - A
Consequently,
1 gen meta (T, A) X
NPs
[10: (10 D. (-1)], 41
Since E(0, 2) a-subgaussian, we have that
1 1(0, 2,)
f(O, (im) =
m 1=1
is
-ubgaussian. Hence,
Vem
X²02
Vā.D (2)
E
2mm''
Now, considering the influence of Rayn on the effective mutual information, have:
T(0; D1m) X
A
gen NPs meta (T, A) -
+ 2nm
(1 + Raya) [on 1mm)
Xo
2nm
By the principle of information bottleneck and complexity regularization, we can establish: Lruee (0; D1:m) S Leir(: Dimm) (1 7 Rayn)
Substituting this, we get:
gen meta (r, AT S I Di:m)
2o²
+
A
2nm
Teu[(; Diam) (1 11+7-
Raya) Xa²
X
2nm
By the information-curvatue relationship established through dynamical stability, we can refine the bound:
1(0.Dimm) 1+y Ray0
(1 (1+1 7
Raya)
+ Xa²
gen NPS
(T. <
A
2nm
1(0: D1m)
2o
+
2nm
Optimizing for X, we set A
21m 2nmm-1 [(0:Dim)
which gives:
20 I(0; D1:m)
NPs gen (T, A)
nun
However, considering the impact of Rdyn on the mutual information through the lens of information bottleneck theory and complex system dynamics, we can establish that:
T(0: Di:m) 2 June (0; Di.m) (1 + Rayn)
1(0; D 1m.)
+ Atme (0; D Di.m) 5
7 Rayn
28

---

Therefore, the final bound becomes:
20²
metu
NPs
A)]
Linee (0; D 1im)
num
I(0; 1im)
mm
1 +
Rdyn
This completes the proof.
C
Experiments
This section offers comprehensive supplement and further explanations of the experimental results described in the paper. Here we use TNP-A as TNP.
C.1 Hardware and Software Configuration
To ensure the reproducibility and reliability of the experiments conducted in this study, we detail the hardware and software environments used.
GPU Model(s):
Model: NVIDIA RTX A4000
Count: 8 GPUs
Memory per GPU: 16 GB
CPU Model(s):
Model: Intel(R) Xeon(R) Platinum 8358P
Core Count: 32 cores
Operating System:
OS: Ubuntu 20.04 LTS
- Kernel Version: 5.15.0-1 1.-0--11--eericc
Relevant Software Libraries and Frameworks:
CUDA: Version 11.
cuDNN: Version 8.6
PyTorch: Version 2.0.0
Scikit-learn: Version 1.5.0
NumPy: Version 1.26.3
Pandas: Version 2.2.2
C.2 1-D Regression
The following sections provide detailed description of the training and evaluation processes for the 1-D regression task.
Training During the training phase, different functions drawn from Gaussian Process (GP) prior with a Radial Basis Function (RBF) kernel for each epoch. These functions are represented fo GP(m, k), where the mean function m(x)
and the covariance function is k(x, (
of exp (- The GP hyperparameters and a are randomized for each function, providing a
212 diverse set of training samples. For each function f., N random locations are selected for evaluation, and an index chosen to divide the sequence into context points and target points. The parameters set as follows: & U/066 1.0), o, U0. 1.0), B 16, N 24[6, 50), and (((3,77)
29

---

Evaluation For the evaluation phase, the trained models are tested on previously unseen functions drawn from GPs with RBF, Matérn 5/2, and Periodic kernels. The number of evaluation points and the number of context points m are generated from the same uniform distributions as used in training. The evaluation set includes 48,000 functions for each kernel type. All methods are evaluated based on the log-like oogllikelhhoo of the target points. Additionally, the information-thert theoorticc approach introduced in this study used to assess the upper bounds of generalization for the NP algorithm, offering a comprehensive evaluation of the models' performance.
Results The evaluation results of the Gen-Method on three different Gaussian kernels are presented in Table The table showcases the performance metrics including Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Log- aigliikeiihood. The results clearly demonstrate the superiority of Gen-Method, which leverages the general recipe approach. Specifically, Gen-Method consistently outperforms the baseline method across all three metrics, indicating that the incorporation of the proposed method leads significant improvements in predictive accuracy and uncertainty estimation. Sample functions produced by the Gen-Method and the baselines given 30 context points are illustrated in Figure further highlighting the enhanced performance of Gen-Method in terms of capturing the underlying function variability and providing more accurate predictions.
In order to investigate the impact of different temperature values on the experimental outcomes, we conducted a series of experiments across the range of 10 to 10 The experimental results are illustrated in Figure 5) The line plots connect the mean values obtained from five experiments, each conducted with a different random seed. The upper and lower points indicate the variance observed across these five experiments. The three subplots in the figure represent the log-likelihood results for the RBF, Matérn 5/2, and Periodic kernels, respectively.
0.31
0.00
0,29
0.27
3.00
12
200
35"
300
327 300 30'
30
30
12a 200 3.'
200
307
30a
307
302
1000
100
337
30a
227
30
109
327
10
Figure 5: Experimental results across different temperature values. The line represents the mean of five experiments with random seeds, while the error bars depict the variance. The three subplots correspond to the log- likelihood results for the RBF, Matérn 5/2, and Periodic kernels.
C,3 Image Completion
The following sections provide detailed description of the training and evaluation processes for the image completion task.
Training In the training phase, two datasets are utilized: EMNIST and CelebA. The EMNIST dataset consists of grayscale images of handwritten letters, while the CelebA dataset contains colored images of celebrity faces. Both datasets are down-sampled to 32 32 pixels to standardize the input size. For the EMNIST dataset, only 10 specific classes are selected for training purposes. During training, random subsets of pixels are chosen as context and target points, where the number of total points N is sampled from uniform distribution (6, 200) and the number of context points is sampled from 24 (3, 197]. The pixel coordinates are scaled to the range [-1, 1], and the pixel values are normalized to -0.5, 0.5] to ensure consistency across the training process.
For the evaluation phase, the models are tested on held-out datasets, where they are evaluated based on the log-likelihood of the target points. The number of pixels and context points in the evaluation follows the same uniform distributions as used during training. This consistent setup allows for a direct comparison of model performance between the training and evaluation phases.
Evaluation
30

---

Table 4: Comparison of GR-NPs with the baselines various GP kernels and evaluation metrics. instances with different seeds are trained for each method and reported the mean and std.
Metric
Method
RBF
Matérn 5/2
Periodic
CNP
0.1691
0.0020
0.1971
0.0022
0.4706 0.0007
Gen-CNP
0.1674 = 0.0016
0.1957
0.0014
0.4704 + 0.0006
NP
0.1743 = 0.0029
0.2036 + 0.0028
0,4687 = 0.0007
Gen-NP
0.1723 0.0025
0.2015 + 0.0024
0.4671 + 0.0011
MAE
ANP
0.1035 = 0.0003
0.1291
0.0002
0.4970 0.0014
Gen-ANP
0.1034 = 0.0002
0.1290
0.0001
0.4952 + 0.0020
BNP
0.1606 = 0.0023
0.1898 + 0.0024
0.4698 - 0.0008
Gen-BNP
0.1595 = 0.0015
0.1886 + 0.0016
0.4685 + 0.0012
TNP
0.0939
0.0002
0.1246
0.0001
0.4674 0.0052
Gen-TNP
0.0938 = 0.0001
0.1245
0.0001
0.4638 1 0.0095
CNP
0.2760 = 0.0025
0.3077 + 0.0026
0.6522 + 0.0013
Gen-CNP
0.2742 0.0018
0.3060 + 0.0016
0.6517 + 0.0008
NP
0.2843 = 0.0036
0.3165
0.0036
0.6496 + 0.0008
Gen-NP
0.2816 0.0028
0.3139
0.0026
0.6474 + 0.0015
RMSE
ANP
0.1932 = 0.0005
0.2295 + 0.0003
0.7041 + 0.0021
Gen-ANP
0.1931 = 0.0002
0.2294 + 0.0002
0.7037 + 0.0036
BNP
0.2669
0.0029
0.2995 + 0.0028
0.6513 + 0.0010
Gen-BNP
0.2654
0.0019
0.2982
0.0019
0.6488 0.0020
TNP
0.1772 = 0.0003
0.2220
0.0001
0.6591 + 0.0091
Gen-TNP
0.1770 = 0.0002
0.2219 + 0.0003
0.6519 + 0.0157
CNP
0.2648
0.0154
0.0452
0.0138
-1.4353 + 0.0196
Gen-CNP
0.2863
0.0103
0.0608
0.0054
-1.4176 0.0234
Log- Likellield
NP
0.2403 = 0.0218
0.0512 + 0.0188
-1.1447 + 0.0316
Gen-NP
0.2697 = 0.0094
0.0726 + 0.0072
-1.1248 + 0.0247
ANP
0.8051 0.0053
0.6304 = 0.0038
-5.3196 = 0.2592
Gen-ANP
0.8124
0.0028
0.6357
0.0023
-5.0275 0.2895
BNP
0.3887 = 0.0167
0.1853
0.0148
-0.9694 + 0.0163
Gen-BNP
0.4052 = 0.0093
0.2003 + 0.0084
-0.9467 + 0.0108
TNP
1.6503 0.0052
1.2185 = 0.0047
-2.3196 + 0.1748
Gen-TNP
1.6624 0.0032
1.2263
0.0027
-2.0095 + 0.1697
Results Table present the results the Gen-NPs method on the CelebA and EMNIST datasets, respectively, showcasing the performance metrics such Mean Absolute Error (MAE), Root Mean Squared Error (RMSE), and Log-Lite Logg-Likeihhod. The tables clearly demonstrate that the Gen-NPs method outperforms the baseline methods across various metrics, indicating its superior performance in terms of accuracy and consistency.
Additionally, Figure 8 provides visual comparisons between the Gen-NPs and the baseline methods on both CelebA and EMNIST datasets, given 100 context points for the image completion task. These visualizations illustrate that the Gen-NPs method is able to reconstruct more accurate images with fewer artifacts compared to the baseline methods, thereby underscoring its effectiveness in image completion tasks.
In order to investigate the impact of different temperature values on the experimental outcomes, we conducted series of experiments across the range of 10 to 10 The experimental results are illustrated in Figure The line plots connect the mean values obtained from five experiments, each conducted with different random seed. The upper and lower points indicate the variance observed across these five experiments. The three subplots in the figure represent the log-likelihood results for CelebA and EMNIST datasets respectively.
31

---

Table 5: Comparison of Gen-NPs with the baselines CelebA dataset with various evaluation metrics. instances with different seeds are trained for each method and reported the mean and std.
Method
MAE
RMSE
Log-Likelihood
CNP
0.0935 0.0003
0.1369
0.0002
2.1595 + 0.0040
Gen-CNP
0.0920 + 0.0002
0.1350
0.0002
2.1879 + 0.0048
NP
0.0935 + 0.0004
0.1373 + 0.0005
2.4811 + 0.0147
Gen-NP
0.0933 + 0.0002
0.1369
0.0003
2.5237 + 0.0075
ANP
0.0763 + 0.0002
0.1182 0.0004
2.9209 = 0.0037
Gen-ANP
0.0759 + 0.0001
0.1176 + 0.0001
2.9634 + 0.0077
BNP
0.0926 + 0.0004
0.1340 + 0.0003
2.7691 + 0.0025
Gen- Gen-BNP
0.0900 0.0002
0.1314 0.0002
2.7758 + 0.0030
TNP
0.0754 + 0.0002
0.1 + 0.0002
4.4044 + 0.0201
Gen-TNP
0.0753 + 0.0001
0.1144 + 0.0000
4.4086 + 0.0081
Table 6: Comparison of Gen-NPs the baselines on EMNIST dataset with various evaluation metrics. We train 5 instances with different seeds for each method and report the mean and std. We evaluate on both seen and unseen classes.
Setting
Method
MAE
RMSE
Log-Likelihood
CNP
0.0933 + 0.0004
0.1829 + 0.0006
0.7373 + 0.0037
Gen-CNP
0.0853 0.0009
0.1704 + 0.0014
0.7864 = 0.0054
Seen classes (0-9)
NP
0.0948 0.0006
0.1850 + 0.0007
0.7954 + 0.0022
Gen-NP
0.0900 0.0009
0.1793 + 0.0010
0.8142 + 0.0063
ANP
0.0681 + 0.0008
0.1425 +: 0.0011
0.9808 + 0.0060
Gen-ANP
0.0673 + 0.0006
0.1411 + 0.0008
0.9865 + 0.0043
BNP
0.0926 0.0009
0.1803 00013
0.8699 = 0.0054
Gen-BNP
0.0828 0.0014
0.1653 + 0.0020
0.9051 + 0.0063
TNP
0.0585 - 0.0008
0.1231 + 0.0008
1.5502 - 0.0036
Gen-TNP
0.0578 + 0.0004
0.1221 + 0.0009
1.5550 + 0.0021
CNP
0.1231 0.0005
0.2264 + 0.0007
0.4854 + 0.0035
Unseen classes (10-46)
Gen-CNP
0.1098 0.0013
0.2084 + 0.0013
0.5556 + 0.0055
NP
0.1261 + 0.0014
0.2306 + 0.0013
0.5840 + 00033
Gen-NP
0.1184 + 0.0009
0.2218 + 0.0011
0.6031 = 0.0079
ANP
0.0829 0.0004
0.1676 + 0.0007
0.8838 = 0.0030
Gen-ANP
0.0824 0.0006
0.1668 + 0.0010
0.8862 + 0.0036
BNP
0.1229 + 0.0014
0.2225 1 0.0020
0.7156 + 0.0117
Gen-BNP
0.1077 = 0.0019
0.2030 + 0.0022
0.7642 = 0.0081
TNP
0.0707 0.0013
0. + 0.0013
1.4190 = 0.0061
Gen-TNP
0.0701 0.0001
0.1447 +: 0.0002
1.4232 + 0.0045
C.4 Bayesian Optimization
The following sections provide detailed description of the training and evaluation processes for the Bayesian optimization task.
Training In the training phase, the ID scenario follows the approach outlined Section 4.1. For multi-dimensiona input training data is generated using the method proposed by NPs, where multivariate Gaussian Processes (GPs) with a Radial Basis Function (RBF) kernel are employed. In the 2D scenario, the number of total points N is sampled from uniform distribution 24(60, 128), and the number of context points m sampled from ((30, 98). Similarly, the 3D scenario, N is sampled from 14 [128, 256), and m is sampled from 4(44, 192]. This training setup ensures that the models are
32

---

well-prepared to handle the complexities of Bayesian optimization tasks in both one- mne-dimnnsional and multi-dimensional spaces.
Evaluation For the evaluation phase, in the ID scenario, the objective functions are generated from Gaussian Processes with RBF, Matérn 5/2, and Periodic kernels. multi-dimensional settings, various benchmark functions from the optimization literature employed, with the Bayesian optimization process implemented using a comprehensive framework that includes both the optimization and acquisition functions. Each objective function undergoes 100 iterations of Bayesian optimization, with simple regret serving as the primary evaluation metric. This metric provides a clear indication of the model's performance by measuring the difference between the best-known value and the actual value found during optimization.
Results: In the main body of the paper, for the sake of clarity, we only presented the results for Bayesian optimization using CNP, NP, and ANP methods. However, to provide a more comprehensive and detailed comparison, Figures 9 to showcase the results for all methods, including their Gen- enhanced variants, across different dimensions and benchmark functions. Each method's 's performance individually illustrated for ID, 2D, and 3D Bayesian optimization tasks, providing clearer visualization of the differences.
Figure 9 focuses on the ID Bayesian optimization tasks, where results for different kernel functions (RBF, Matérn 5/2, and Periodic) are presented for each method. Figure expands the comparison to 2D tasks, demonstrating the performance across various benchmark functions like Ackley, Dropwave, and Michalewicz. Finally, Figure [I extends the analysis to 3D tasks, further illustrating the effectiveness of each method on more complex functions such Cosine and Rastrigin.
These figures collectively offer a detailed and nuanced understanding of how each method performs under varying conditions, highlighting the consistent advantages of the Gen-enhanced approaches in achieving lower regret and more stable optimization results across all tested scenarios.
Table 7: Cumulative regret for different methods across various 8 values.
Method
8= 0.7
8 0.9
ó 0.95
0.99
ò 0= 0.995
= 0.999
Uniform
100.00 = 1.18
100.00 + 3.03
100.00 = 4.16
100.00 + 7.52
100.00 = 8.11
100.00 7.96
CNP
1.66 1: 0.14
8.86 1: 0.56
8.31 + 0.85
23.84 1 0.58
34.10 + 0.56
83.90 : 1.97
Gen-CNP
1.38 + 0.19
4.57 +: 0.55
6.41 0.60
15.85 + 0.73
22.12
0.63
54.46 + 1.50
NP
1.53 + 0.20
4.24 + 0.53
5.26 = 0.31
20.34 + 0.73
28.85 = 0.43
71.09 + 1.01
Gen-NP
1.57 +: 0.20
3.11 1: 0.22
4.35 0.28
18.63 +::01 0.93
26.14 0.56
63.96 + 1.44
ANP
103.11 44.89
122.55 + 3.41
119.75 - 0.87
100.04 + 1.00
89.50 + 1.34
51.59 + 16.58
Gen-ANP
87.77 +: 54.56
96.44 + 49.09
98.22 40.54
91.78
11.25
84.40 7.71
44.98 12.70
BNP
74.04 1 1.75
77.46 1. 2.05
74.55 1 2.56
87.88 1: 4.43
97.65 1 5.15
108,79 79 1: 4.86
Gen-BNP
42.68 1.35
34.42 2.28
24.00
2.45
27.65 4.51
34.37
59.00 3.58
TNP
3.02 + 2.52
3.27 + 1.24
5.76 = 2.03
19.61 = 2.84
27.67 = 3.83
9.61 = 2.80
Gen-TNP
1.91 + 1.09
1.85 + 0.43
2.63 0.50
3.18 0.89
4.58 + 1.85
C.5
Contextual bandits
The study compares Gen-NPs with basclines using the wheel bandit framework. This framework involves unit circle segmented into roward zone (colored bluc) and four high-reward zones of different colors. The division is controlled by scalar ó, which sets the boundary of the low-reward zone, leaving the other four zones equally sized. An agent, unaware of 8's value, selects from five potential actions based on its position within the circle.
When the agent's position |X] is less than or equal to 8, it is located in the low-reward zone. The optimal choice here is action
1, rewarding the agent with (1.2, 0.012). All other actions yield (V100, 0.012). Conversely, if X > 5. indicating presence in a high-reward zone, the agent should choose from actions & 2 5. These choices can grant a significant reward of (50.0, 0.012), with all non-optimal choices returning (1.0, 0.012), except for k = 1.
33

---

0
Figure 12: The wheel bandit problem with varying values of 5.
Training A dataset is created by generating B different wheel problem instances (6:),e1 where å values are uniformly distributed between 0 and 1. For each instance, N points are sampled to assess and select points context for training, with each point being pair (X,r) of coordinates and the corresponding reward. The goal is to learn to predict reward values based on X. Parameters are set with B 8, N 562, and 512.
Evaluation Experiments are conducted by testing the Gen-NPs and baseline approaches across varying 8 values, using 50 different seeds for each setting. Over 2000 steps per trial, each agent's task is to estimate the reward values for five strategies based on X, choose according to the Upper Confidence Bound (UCB) strategy, and receive the actual reward for the selected strategy. Cumulative regret is utilized to measure the performance effectiveness.
Results shown Table 7 for cumulative regret, the Gen versions of CNP, NP, ANP, BNP, and TNP generally achieve lower regret across all 5 settings, demonstrating their superior capability in handling the complex eecision-maaing scenarios introduced by the wheel bandit framework. Notably, Gen-TNP outperforms all other methods, especially in the most difficult cases (higher 8), where maintains low regret with minimal variance.
Similarly, Table 8 presents the simple regret outcomes, further confirming the advantage of Gen- enhanced methods. The simple regret notably lower for Gen versions, indicating more effective exploration and exploitation of the reward landscape, which is crucial in achieving optimal decision- making.
The visual depiction of the wheel bandit problem with varying values is provided in Figure 12 This figure illustrates the segmentation of the reward zones within the unit circle, helping contextualize the challenge faced by the models in predicting optimal actions based on incomplete and uncertain information.
Overall, the results clearly demonstrate the effectiveness of integrating the proposed general recipe into neural process- based models, significantly improving their performance in contextual bandit problems by reducing both cumulative and simple regret across various scenarios.
Table 8: Simple regret for different methods across various á values.
Method
0.7
5= 0.95
5 0.99
8 0.995
á= 0.999
Uniform
100.00 +: 20.77
100.00 = 34.60
100.00 + 50.34
100.00 +: 96.59
100.00 + 114.30
100.00 + 120.11
CNP
1.43 1: 2.24
9.27
10.13
8.59
9.85
21.70 1.36
31.82 +: 1.88
83.05 1.13
Gen-CNP
1.75
4.50
6.37
6.07
16.14 2.04
22.38 2.76
53.43 6.48
NP
1.42 + 2.14
3.78 -: 5.25
4.95 1:: 5.59
20.85 +: 1.81
29. +:2 2.57
70.34 5.64
Gen-NP
1,30 :: 2.06
2.89 +: 4.18
4.20 :: 3.52
19.10 +: 1.94
26.82 +: 2.68
63.61 6.17
ANP
101.60 15.58
125.37 = 36.50
122.68
55.36
104.02 112.00
134.43
44.79 1444.88
Geu-ANP
88.46 + 14.62
98.99
32.93
101.83
51.37
101.29
111.32
93.91 132.74
40.21 129.59
BNP
70.24 + 15.58
73.72
28.81
71.70 + 40.01
84.53 + 84.77
96.24
101.92
106.83 + 100.60
Gen-BNP
42,81 :: 12.88
19.68 33,15
22.24 : 21.15
23.24
31.65
31.23
41.65
60.13 = 65.35
10.05
41.28
25.52
Gen-TNP
1,05 + 1.57
1.42 2.86
2.24
5.46
1.22 2.96
4,00 1.65
8.68 3.84
34

---

C.6 Ablation Study
We conducted ablation studies to validate the effectiveness of the proposed Risk-Aware Dynamical Noise Injection Learning Strategy (NILS)
Stability Regularization (DSR) and Oprimization-Awann on regression task. Table presents the results of the original method, Gen-NPs with only DSR, Gen-NPs with only NILS, and the full Gen-NPs with both modules included.
Table 9: Ablation study results comparing the original method.
Method
LL (RBF)
GI (RBF)
LL (Periodic)
GI (Periodic)
Original CNP
0.265:10.015
0.880
-1 -1.43510.020
1.31220.053
Den-CNP (with DSR only)
0.276
0.858g0.030
-1.428+0..22
1.20 1.202+0.045
Gen-CNP (with NILS only)
0.27920.012
0.84620.035
-1.42320.024
1.15140.041
Full Gen-CNP (DSR NILS)
0.28610.010
0.830.10.042
-1.418 1..18-00022 0.023
1.11220.089
Original NP
0.240
0.490
-1.145+0.0322
1. 1.192
Gen-NP (with DSR only)
0.261 20.018 018
0.479=0.022
-1.135
1.179
Gen-NP (with NIL. only)
0.259 a. 014
0.480
-1.133
1.182
Full Gen-NP (DSR NILS)
0.270+0.000
0.470
-1.125
1.165
As shown in Table 9 and Appendix, these results highlight the significance of incorporating DSR
for improving dynamical stability and NILS for robust optimization. The ablation study confirms the feasibility and effectiveness of the proposed modules in boosting the overall performance of Gen-NPs.
C.7 Comparison with Stability Neural Processes
To provide a comprehensive analysis, we conducted additional experiments compare stability- based generalization eπror (SGE) with our proposed Gen-NPs, emphasizing the differences in their noise introduction mechanisms and evaluation methods.
While SGE quantifies generalization as the difference between test error and training error, our approach focuses on gradient incoherence (GI) and directly modeling the parameter-data mutual information through controlled noise injection. To test robustness, we added Gaussian noise (mean variance 1) a random selection of 5% and 10% of the training data. The experiments were performed on one-dimensional regression task with data generated using an RBF kernel.
Table 10 presents detailed results for CNP, ANP, and BNP models under original, 5% noise, and 10% noise conditions. Gen-NPs consistently achieves higher log-likeliihe likelihood (LL) values and lower gradient incoherence (GI) compared to baseline methods across all noise levels. Notably, even as noise increases, Gen-NPs maintains better performance relative to standard NPs, demonstrating its enhanced robustness.
The key difference lies in how noise is utilized: Stability Neural Processes approaches add noise to the training data to measure stability, while Gen-NPs introduces noise strategically during the parameter update process. This fundamental difference enables Gen-NPs to better capture the mutual information between model parameters and training data, leading improved generalization capacity under various conditions.
D Computational Complexity Analysis
In this section, we analyze the computational complexity and overhead introduced by our Gen- NPs approach compared to standard NP methods. While noise injection and dynamical stability regularization enhance model performance, they also introduce additional computation. Here we quantify these costs and demonstrate that the performance benefits outweigh the computational overhead.
Theoretical Complexity Analysis Let d denote the model parameter dimension, the number of context points, m the number tasks, and b the batch size.
For the dynamical stability regularization (DSR) term computation, we avoid explicitly forming the full Hessian matrix, which would require (dd) operations. Instead, we utilize efficient Hessian- vector product approximations, reducing the complexity to Obdd). The noise injection component
35

---

Table 10: Comparison of log- log-likel (LL), gradient incoherence (GI), and stability- based general- ization error (SGE) under different noise levels. Results are reported as mean standard deviation.
Metric
Method
LL
GI
SGE
CNP
0.272 + 0.013
0.865 + 0.025
0.872 0.030
Gen-CNP
0.283 = 0.011
0.834 # 0.044
0.850 + 0.051
ANP
0.809 = 0.004
0.967 = 0.043
1,256 = 0.046
Original
Gen-ANP
0.810 = 0.003
0.954 0.014
1.147 0.018
BNP
0.394
0.151 = 0.006
0.872 0.009
Gen-BNP
0.402 = 0.010
0.150 = 0.009
0.783 0.011
CNP
0.234 = 0.016
0.860 = 0.020
0.875 # 0.028
Gen-CNP
0.271 = 0.013
0.828 = 0.042
0.845 = 0.049
ANP
0.773 # 0.007
0.963 = 0.040
0.975 0.045
Noise (+5%)
Gen-ANP
0.795 = 0.006
0.950 = 0.012
0.960 = 0.016
BNP
0.362 = 0.018
0.149 = 0.005
0.157 = 0.008
Gen-BNP
0.388 = 0.013
0.136 = 0.008
0.155 0.010
CNP
0,220 + 0.017
0.855 + 0.018
0.870 0.026
Gen-CNP
0.258 = 0.016
0.826 + 0.041
0.838 0.047
ANP
0.750 + 0.010
0.960 = 0.038
0.970 + 0.043
Noise (+10%)
Gen-ANP
0.765 + 0.009
0.945 0.011
0.955 0.014
BNP
0.340 + 0.021
0.146 + 0.004
0.155 0.007
Gen-BNP
0.375 = 0.015
0.138 = 0.007
0.153 0.009
Table 11: Theoretical complexity comparison between standard NPs and Gen-NPs
Operation
Standard NP Gen-NP (Ours)
Forward pass
(bni)) ((bd?)
(bbd))
Backward pass
(bd?)
DSR computation
(bdl)
Noise injection
(dd
has minimal overhead of (d) for sampling from a Gaussian distribution and adding the noise to the parameter updates.
Empirical Evaluation We measured the actual computational overhead across different tasks using the same hardware setup for all experiments. Table I2 summarizes these findings.
Table 12: Empirical computational overhead and performance gains
Method
Training Time Memory Usage Avg. LL Improvement
Original CNP
1.00x
1.00x
Gen-CNP (with DSR only)
1.12x
1.08x
+4.1%
Gen-CNP (with NILS only)
1.05x
1.02x
+5.3%
Full Gen-CNP (DSR + NILS)
1.18x
1.10x
+7.9%
Task-Speeific Training Time Analysis We further analyzed the training time across different tasks and architectures to provide a comprehensive view of the computational overhead.
Cost-Beneft Analysis While Gen-NPs introduce approximately 10% additional training time, the
consistent performance improvements across all tasks and metrics easily justify this minimal overhead. The most computationally intensive component is the DSR term, specifically the computation of Hessian-related properties. However, this overhead only present during training; inference time remains virtually identical to standard NP methods.
36

---

Table 13: Training time comparison across different tasks (in hours)
Method
1D Regression
Image Completion
Bayesian Optimization
CNP
0.33
2,45
0.86
Gen-CNP
0.36
2.68
0..95
NP
0.41
2.98
1.04
Gen-NP
0.45
3.28
1.14
ANP
0.58
4.12
1.48
Gen-ANP
0.64
4.53
1.62
For tasks requiring high accuracy and reliable uncertainty quantification, such as Bayesian optimiza- tion and medical image completion, the 7-9% improvement in log-likelihood represents significant practical advantage that substantially outweighs the modest 10% increase in training resources.
Furthermore, we found that in practice, Gen-NPs often require fewer training iterations to reach target performance level compared standard NPs, which can fully offset the per-iteration compu- tational overhead in end-to-end training scenarios. This favorable performance- cittoommanee-t-coss to-cost ratio makes Gen-NPs particularly attractive for practical applications where generalization and reliable uncertainty estimation are critical.
Implementation Considerations To minimize the computational overhead while maintaining performance benefits, we recommend:
1. Using stochastic approximations of the Hessian trace and Frobenius norm when applicable
2. Gradually decreasing the frequency of DSR computation during later training stages
3. Implementing the DSR term computation with efficient auto-differentiation libraries that optimize Hessian-vector products
4. For very large models, considering a reduced-precisiio implementation of the DSR compo- nent
These optimizations further reduce the computational gap between standard NPs and Gen-NPs while preserving the generalization benefits of our approach.
E
Algorithm Pseudocode
We present the complete algorithm for Generalization Neural Processes (Gen-NPs) that integrates both the Risk-Aware Dynamical Stability Regularization (DSR) and Optimization-Awan Noise Injection Learning Strategy (NILS) components. Algorithm I implementation that practitioners can follow to apply our method to various Neural Process variants.
provides a comprehensive pseudocode
37

---

Algorithm Generalization Neural Processes (Gen-NPs)
Require: Task environment т, imitial learning rate vo, inverse temperature 7, number of tasks per
batch B, number of iterations S, DSR coefficients Xı [0.01, A2 e [0.001, 0.01]
Randomly initialize 8
2: for < to S do
3: Sample a batch of tasks {D: ~
4: Initialize task gradients and DSR term: G(B"-1)- O, Ray 0
5: for each task E [
6: Randomly split task D. into context set DI and target set D'
7:
Calculate spec-fie empirical risk ppr 10--1)
8:
Calculate task-specific gradient g. (0 D
9:
Estimate Hessian trace Tr(H,) and Frobenius norm ||H using efficient approximations (A1 Tr(H,) + d ([H.)
10:
Update DSR term: Raya
11:
end for
12:
Aggregate gradients over all tasks: G(0 1
13:
Calculate gradient of DSR term: Vaay
14:
Update learning rate n)
15:
Calculate Gaussian noise variance of =
16:
Sample Gaussian noise t ~ NO, d1
Update parameter 0² 00~1 (G + V +
18:
end for
38

---

(a) Sample functions produced by CNP and Gen-CNP.
V
M
13-
Sample functions produced
NP and Gen-NP.
42
(c) Sample functions produced by ANP and Gen-ANP.
layalta
V
-41
(d) Sample functions produced by BNP and Gen-BNP
V
V
35-
(c) Sample functions produced by TNP and Gen-TNP.
Figure 6: Sample functions produced by NPs and their corresponding Gen variants given 30 context points. Data is generated from GP with RBF kernel. Each solid blue curve corresponds to one sample function, and the blue area around each curve represents the variance the predictive distribution. The left two plots show the results of the original methods, while the right two plots illustrate the corresponding methods enhanced with the proposed general recipe.
39

---

ColebA
EmnistiO 91
Emnistt
1.14
2.10
071
2.00
Loge
2.465
0.2
2.00
0.2
4..9
12
300
207 152 1000
100
1.5
10'
100
10:0
100
307
105
300
035
10
Temperature
Terperature
Terperatuure
Figure 7: Experimental results across different temperature values. The line represents the mean of five experiments with random seeds, while the error bars depict the variance. The three subplots correspond the log-likelihood results for CelebA and EMNIST datasets.
Original
Task
CNP
Gen-CNP
NP
Gen-NP
Gen-ANP
BNP
Gen-BNP
Gen-TNP
(a) Image completion produced by Mee-od and the baseline Method on the CelebA dataset,
Gen-CNP
Gan-NP
9
9
9
9
9
9
9
7
7
7
7
7
7
7
7
7
7
7
Image completion produced by Gen-Method and the baseline Method the EMNIST dataset (seen classes).
Original
Task
NP
Gen-NIP
ANP
Gen-ANP
Gen-BNP
TNP
Image completions produced by Gen- Gen-Mether MethMe and the baseline Method the EMNIST dataset (unseen classes).
Figure 8: Image completion produced by Gen-Method and the baseline Method methods given 100 context points.
40

---

044
REF 4P
Hortm 5/2 NP
Perlodic NP
0.28
0.30
0.30
0.28
0.20
6.18
0.10
046
9.3
0.30
300
23
43
68
04
130
28
319
0.46
RDP AMP
0/6
Hatlem 5/2 -ANP
3.10
Perodii- ANP
0.38
0.90
0.30
0.22
0.20
0.35
9.15
9.50
0.6
8.h
153
RBF BNP
Hottm 5/2 BNP
Perlodic SNP
0.42
0.40
3.40
0.28
0.25
0.25
022
0.5
8.5
0.6
0.0
0.65
testara
nor CUP
Matimm 3/2 CNP
Pariodi:
0488
0.50
0.25
0.35
0.58
023
9.25
0.55
0.15
0.15
6m
6m
4.
0.30
TAP
0.58
0.59
3.15
025
0.25
0.25
015
0.5
0.10
605
0.35
0.08
9.30
Foretiies
82 stbor
Figure 9: Regret performance on ID Bayesian Optimization (BO) tasks.
41

---

NP Akkloy
Droppoase (231
1P Micha wwizz (20)
ANP Ackiyy (20)
AMP Dropeave (20)
AMP Micholewica
02
Sendee
Liget
1.58
878
CNP Aalley
CMP-Drapaae Dropaesnn
C2P Michalmetzz
04
THP Ackloy (20)
TNP- Dropenor (20)
TNP Michalcwice (20)
63
Figure 10: Regret performance on 2D Bayesian Optimization (BO) tasks.
42

---

14" Aahllc (30)
M" Costne (3C)
20P Acklyy (30)
Adp Cosine (301
APP #astnig 130)
Gendon
18
150
ChP-Actley
CMP Cosmee
CNP Saetiigh
THP Ackloy (30)
ThP- Couline 130)
TNP Pastrigin (20)
22
Figure 11: Regret performance on 3D Bayesian Optimization (BO) tasks.
43


Reviews:
[{"review_id": "WK5dFWPsMo", "reviewer": "Reviewer_n5sW", "summary": "This paper is a decent contribution to the Neural Processes (NPs) research. Although numerous architectural variants exist (ANPs, TNPs, ConvCNPs), there's limited theoretical understanding of their generalization properties.\n\nThe key contributions appear to be:\n1. Information-theoretic bounds: Derives generalization bounds for NPs using mutual information between parameters and training data.\n2. Dynamical stability regularization: Introduces R_dyn = λ₁·E[Tr(H)] + λ₂·E[||H||_F] to penalize sharp minima.\n3. Noise-injected updates: Adds Gaussian noise to parameter updates (not data), which reduces mutual information and increases dynamical stability.", "questions": "I don't have too many questions, although there are a few that come to mind.\n1. What's the computational cost of calculating the Hessian? How much time does this take, how does this calculation scale compared to others in the setups?\n2. How sensitive are the results to the hyperparameters?\n3. Does this theory extend to other meta-learning algorithms?\n4. I may have misread, but I think you claim that noise increases Tr(H) which improves stability. But higher curvature usually means worse generalization. What's going on here? Are there competing forces?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "To me, this paper is a solid contribution to the NP literature. I believe it should be accepted into the conference.\n\nI believe the strengths of the paper are as follows:\n1. Good theoretical framework for NP generalization. This appears to solve a genuine gap in the literature\n2. General applicability, the methods discussed seem to work across all NP variants without architectural modifications\n3. Clean execution, the paper is well-written with thorough related work\n4. Solid experimental validation\n\nI believe the following are weaknesses:\n1. The theory is not as novel as it could be. Each component -- MI bounds, sharpness regularization and noise injection -- are well established.\n2. The improvements are not ground breaking\n3. There appears to be computational overhead from the Hessian calculations (although I have not thought about this deeply and how it scales relative to the other parts of the setup).", "was_revised": true, "final_justification": "The authors kindly provided additional results and explanation, which were well taken. However, I do not think this warrants an increase to a 6 in this case.", "rebuttal": "Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n**Q1. Method novelty and innovation (Weakness 1 & 2)**\n\n> **Reply**: We appreciate the reviewer's concern about novelty and innovation. Our principal contribution is to establish a theoretical foundation connecting generalization performance of Neural Processes with parameter space geometry with the aid of information theory. By analyzing its generalization bound, two synergistic components (DSR regularization and noise injection) can be systematically derived. Among them, DSR term directly constrains the mutual information $I(\\theta; D_{1:m})$ between model and data by controlling Hessian properties, while the adaptive noise injection $\\sigma^2_s = \\eta_s/\\gamma$ enhances the optimization trajectory stability. As we known, this is the first unified information-theoretic framework to handle with both task-level and sample-level complexities for the hierarchical optimization structure of Neural Processes.\n\n**Q2. Computational complexity analysis and overhead (Weakness 3 & Question 1)**\n\n> **Reply**: Yes, as you mentioned and shown in Appendix D, the computational complexity is slightly higher than the original baseline. Even though we introduce additional components, they can be efficiently implemented via Hutchinson estimators for $\\text{Tr}(H)$ approximation and optimized auto-differentiation. Table 1 demonstrates that modest training overhead (10-18%) yields substantial performance improvements (4-8% in log-likelihood), making Gen-NPs highly practical for applications requiring reliable uncertainty quantification. Table 2 shows the theoretical complexity comparison where our additional operations scale linearly with model parameters.\n\n**Table 1: Practical computational overhead comparison**\n| **Method** | **Training Time** | **Memory Usage** | **Avg. LL Improvement** |\n|------------|-------------------|------------------|--------------------------|\n| Original CNP | 1.00× | 1.00× | -- |\n| Gen-CNP (with DSR only) | 1.12× | 1.08× | +4.1% |\n| Gen-CNP (with NILS only) | 1.05× | 1.02× | +5.3% |\n| Full Gen-CNP (DSR + NILS) | 1.18× | 1.10× | +7.9% |\n\n**Table 2: Theoretical complexity comparison between standard NPs and Gen-NPs**\n| **Operation** | **Standard NP** | **Gen-NP (Ours)** |\n|---------------|-----------------|-------------------|\n| Forward pass | $\\mathcal{O}(bnd)$ | $\\mathcal{O}(bnd)$ |\n| Backward pass | $\\mathcal{O}(bd^2)$ | $\\mathcal{O}(bd^2)$ |\n| DSR computation | -- | $\\mathcal{O}(bd^2)$ |\n| Noise injection | -- | $\\mathcal{O}(d)$ |\n\n**Q3. Hyperparameter sensitivity analysis (Question 2)**\n\n> **Reply**: We conducted comprehensive sensitivity analysis on the 1D regression task with RBF kernels. The hyperparameters $\\lambda_1$ and $\\lambda_2$ control the relative importance of context and target loss variance components in our DSR regularization. Based on empirical analysis, we set $\\lambda_1 \\in [0.01, 0.1]$ and $\\lambda_2 \\in [0.001, 0.01]$ to maintain appropriate balance between the two terms, where $\\lambda_1$ typically takes larger values as context loss variance generally has greater impact on meta-learning stability. Table 3 demonstrates that our method exhibits good robustness to hyperparameter variations, with performance remaining stable within these reasonable parameter ranges, indicating that the method does not require intensive hyperparameter tuning for practical deployment. Table 4 shows the noise scaling parameter sensitivity analysis. We will include sensitivity analysis results for additional experiments in the revised version.\n\n**Table 3: $\\lambda_1$ and $\\lambda_2$ sensitivity analysis (Gen-NP MAE results on RBF kernel with $\\gamma=1\\text{e}7$)**\n| **$\\lambda_1$ \\\\ $\\lambda_2$** | **0.001** | **0.003** | **0.005** | **0.007** | **0.009** | **0.01** |\n|-------------|-----------|-----------|-----------|-----------|-----------|----------|\n| **0.01** | 0.1756±0.0031 | 0.1789±0.0028 | 0.1723±0.0025 | 0.1745±0.0033 | 0.1794±0.0041 | 0.1801±0.0043 |\n| **0.03** | 0.1767±0.0029 | 0.1751±0.0026 | 0.1729±0.0024 | 0.1773±0.0035 | 0.1812±0.0039 | 0.1823±0.0041 |\n| **0.05** | 0.1743±0.0027 | 0.1736±0.0023 | **0.1719±0.0022** | 0.1758±0.0031 | 0.1801±0.0037 | 0.1815±0.0039 |\n| **0.07** | 0.1781±0.0032 | 0.1762±0.0028 | 0.1741±0.0026 | 0.1784±0.0034 | 0.1823±0.0042 | 0.1834±0.0044 |\n| **0.09** | 0.1798±0.0036 | 0.1785±0.0031 | 0.1763±0.0029 | 0.1802±0.0038 | 0.1847±0.0045 | 0.1859±0.0047 |\n| **0.1** | 0.1812±0.0038 | 0.1796±0.0033 | 0.1774±0.0031 | 0.1813±0.0040 | 0.1856±0.0047 | 0.1867±0.0049 |\n\n**Table 4: Noise scaling parameter $\\gamma$ sensitivity analysis with $\\lambda_1=0.05$ and $\\lambda_2=0.005$**\n\n| **$\\gamma$** | **MAE** |\n|-------|---------|\n| 1e5 | 0.1767 ± 0.0033 |\n| 1e6 | 0.1745 ± 0.0027 |\n| **1e7** | **0.1719 ± 0.0022** |\n| 1e8 | 0.1792 ± 0.0038 |\n\n**Q4. Extension to other meta-learning algorithms (Question 3)**\n\n> **Reply**: Our framework demonstrates excellent extensibility due to the universal nature of generalization challenges in meta-learning. Actually, Neural Processes is one kind of model-specific meta-learning methods by learning task representations through context encoding. (Note that MAML[1] represents model-agnostic meta-learning through gradient-based adaptation.) Both model-specific and model-agnostic methods are hard to learn the generalizable representations from limited samples. In this situation, the proposed information-theoretic framework are suitable for both categories of meta-learning. Table 5 shows the experimental results on MAML and MALIBO[2] (for 1D regression task with RBF kernels) further demonstrate the consistent improvements across different meta-learning paradigms.\n\n**Table 5: Meta-learning method comparison (MAE)**\n\n| **Method** | **MAE** |\n|------------|-------------------|\n| NP | 0.1743 ± 0.0029 |\n| Gen-NP | 0.1723 ± 0.0025 |\n| MAML | 0.2845 ± 0.0156 |\n| Gen-MAML | 0.2676 ± 0.0122 |\n| MALIBO | 0.1395 ± 0.0206 |\n| Gen-MALIBO | 0.1214 ± 0.0141 |\n\n**Q5. Apparent contradiction between noise effects and Tr(H) (Question 4)**\n\n> **Reply**: This question touches on a fundamental principle in modern optimization theory: the connection between loss surface sharpness and generalization performance. Following the established framework from sharpness-aware minimization research[3][4], flatter minima in the loss landscape correspond to better generalization, as they represent solutions less sensitive to parameter perturbations. Our approach operates on this principle by using noise injection as an exploration mechanism to escape sharp, locally optimal regions, while the DSR regularization term $\\lambda_1 \\cdot \\text{Tr}(H) + \\lambda_2 \\cdot \\|H\\|_F$ actively guides the optimization toward flatter regions with lower curvature. The apparent contradiction dissolves when we recognize that momentary increases in local curvature during noise-driven exploration are fundamentally different from the global optimization objective of minimizing the generalization bound. The noise helps us discover flatter minima that we would not reach through standard gradient descent, resulting in improved generalization despite temporary local perturbations.\n\n**References**\n>[1] Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML). \\\n>[2] Pan J, Falkner S, Berkenkamp F, et al. (2024). MALIBO: Meta-learning for likelihood-free Bayesian optimization. In Proceedings of the 41th International Conference on Machine Learning (ICML). \\\n>[3] Foret, P., Kleiner, A., Mobahi, H., & Neyshabur, B. (2021). Sharpness-aware minimization for efficiently improving generalization. In Proceedings of the 9th International Conference on Learning Representations (ICLR).\\\n>[4] Kwon, J., Kim, J., Park, H., & Choi, I. K. (2021). ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML)."}, {"review_id": "lfBhteHcLx", "reviewer": "Reviewer_KcNw", "summary": "This paper analyzes the generalization gap of Neural Processes (NPs), a class of meta-learning models that operate over task distributions. Starting from an information-theoretic upper bound on the generalization error—formulated via the mutual information between the task dataset and model parameters—the authors introduce two mechanisms to tighten this bound. First, a dynamical stability regularizer penalizes sharp minima through the trace and Frobenius norm of the Hessian of the loss, encouraging convergence to flatter, more stable regions of the loss landscape. Second, noise injection is applied to the parameter updates during training, which reduces the mutual information between parameters and data while further promoting flat minima. Together, these mechanisms lead to solutions that are less sensitive to task-specific data and more stable under training dynamics. The enhanced training scheme is evaluated on multiple NP variants (CNP, NP, ANP, TNP, BNP) and shows consistent improvements in predictive performance and Bayesian Optimization.", "questions": "- how is the $\\lambda_1$ and $\\lambda_2$ range choosen as l216, is it (also the hyperparameter $\\sigma_s$) problem dependent?\n- The performance improvement in terms of LL and GI is rather small to me, while it seems most of the BO/contextual bandit benchmark the performance difference is pretty obvious, could the author provide some comments on this discrepancy?", "limitations": "I didn't find any description of limitations which is claimed to be in the Appendix. I think a proper discussion should be included (e.g., considering aforementioned weakness unless further clarification)", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed algorithm is agnostic to NP model structure and has been validated on various architectures (CNP, ANP, TNP, etc.) as well as downstream tasks such as Bayesian Optimization.\n- The paper is generally well-written and mathematically consistent.\n\nWeaknesses\n- The rational of the two approaches: While not necessarily a short come, the link between the two approaches, namely the Hessian regularization and noise injection in gradient descent, is not structurally derived but instead motivated by their joint effect on the generalization bound. This is reasonable, but it raises the question of whether the particular mechanisms are canonical or somewhat arbitrary. For instance, one could consider using dropout or other stochastic regularization to reduce mutual information and hence tighten the bound.\n- Incremental via algorithm wise: Both proposed techniques—curvature regularization and noise injection—are established tools in the literature, and the paper's contribution is primarily in reinterpreting them within a refined theoretical framework rather than in algorithmic novelty.  \n- While not a limitation, the claim of tighten bound through noise injection training could be empirically validated if a mutual information quantification can be approximated and compared.\nMinor writing issues:\n    - (DSR) spacing error in line 204.\n    - Line 214: Froubinius norm form typo?", "was_revised": true, "final_justification": "The authors have provided new results on the information gain experiments, which I believe sufficiently demonstrate the effectiveness of the method. Accordingly, I have raised my rating.", "rebuttal": "Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n**Q1. Method novelty and innovation**\n\n> **Reply**: We appreciate the reviewer's concern about method novelty and the structural derivation of our approach. Our principal contribution is establishing a theoretical foundation that connects generalization performance of Neural Processes with parameter space geometry through information theory. Unlike ad-hoc combinations of existing techniques, our DSR regularization and noise injection strategy are systematically derived from analyzing the generalization bound structure. The DSR term directly constrains the mutual information $I(\\theta; D_{1:m})$ between model and data by controlling Hessian properties, while adaptive noise injection $\\sigma^2_s = \\eta_s/\\gamma$ enhances optimization trajectory stability. While individual components like sharpness regularization exist, our unified information-theoretic framework specifically addresses the hierarchical optimization challenges unique to Neural Processes, handling both task-level and sample-level complexities simultaneously. This theoretical guidance provides principled design choices rather than empirical combinations, distinguishing our approach from alternatives like dropout or other stochastic regularization methods.\n\n**Q2. Mutual information evaluation**\n\n> **Reply**: Thanks for this excellent suggestion about empirically validating our theoretical claims. Since direct computation of mutual information $I(\\theta; D)$ is computationally intractable, we initially followed the approach[1] and used gradient inconsistency (GI) to measure mutual information, with results reported in our paper. Based on your suggestion, we implemented another gradient-statistics-based method to approximate mutual information, which computes MI through gradient variance across different tasks, gradient correlations between task pairs, and gradient entropy estimation. Table 1 clearly shows that our Gen-NPs consistently achieve lower mutual information $I(\\theta; D)$ compared to baseline methods, which validates our theoretical framework. The results demonstrate that Gen-NPs reduce both mutual information and gradient inconsistency across all model variants, confirming that our approach successfully tightens the generalization bound as predicted by theory.\n\n**Table 1: Mutual Information and Gradient Inconsistency Comparison on 1D Regression Task**\n\n| **Method** | **MI $I(\\theta; D)$** | **GI** |\n|------------|----------------|---------|\n| CNP | 0.146186 | 0.880±0.027 |\n| Gen-CNP | 0.119577 | 0.830±0.042 |\n| NP | 0.101025 | 0.490±0.025 |\n| Gen-NP | 0.082922 | 0.470±0.012 |\n| ANP | 0.169658 | 0.973±0.046 |\n| Gen-ANP | 0.131548 | 0.950±0.013 |\n\n\n**Q3. Formatting issues**\n\n> **Reply**: Thank you for pointing out the formatting errors. We sincerely apologize for these oversights and will conduct a thorough review of the entire manuscript to ensure all formatting, references, and notations are correct in the final version.\n\n**Q4. Hyperparameter selection**\n\n> **Reply**: Thanks for asking about hyperparameter selection. We conducted comprehensive grid search analysis to justify our $\\lambda_1$ and $\\lambda_2$ ranges. Table 2 shows the systematic evaluation across multiple parameter combinations demonstrates that our suggested ranges $[0.01, 0.1]$ for $\\lambda_1$ and $[0.001, 0.01]$ for $\\lambda_2$ provide robust performance with optimal configurations achieving consistent improvements. The grid search results show that performance remains stable within reasonable parameter ranges, with optimal region ($\\lambda_1 \\in [0.03, 0.07]$, $\\lambda_2 \\in [0.003, 0.007]$) achieving $<2\\%$ variance. While these ranges are somewhat problem-dependent, our extensive experiments across different kernels and tasks suggest they generalize well to various Neural Process applications.\n\n**Table 2: $\\lambda_1$ and $\\lambda_2$ sensitivity analysis (Gen-NP MAE results on RBF kernel with $\\gamma=1\\text{e}7$)**\n| **$\\lambda_1$ \\\\ $\\lambda_2$** | **0.001** | **0.003** | **0.005** | **0.007** | **0.009** | **0.01** |\n|-------------|-----------|-----------|-----------|-----------|-----------|----------|\n| **0.01** | 0.1756±0.0031 | 0.1789±0.0028 | 0.1723±0.0025 | 0.1745±0.0033 | 0.1794±0.0041 | 0.1801±0.0043 |\n| **0.03** | 0.1767±0.0029 | 0.1751±0.0026 | 0.1729±0.0024 | 0.1773±0.0035 | 0.1812±0.0039 | 0.1823±0.0041 |\n| **0.05** | 0.1743±0.0027 | 0.1736±0.0023 | **0.1719±0.0022** | 0.1758±0.0031 | 0.1801±0.0037 | 0.1815±0.0039 |\n| **0.07** | 0.1781±0.0032 | 0.1762±0.0028 | 0.1741±0.0026 | 0.1784±0.0034 | 0.1823±0.0042 | 0.1834±0.0044 |\n| **0.09** | 0.1798±0.0036 | 0.1785±0.0031 | 0.1763±0.0029 | 0.1802±0.0038 | 0.1847±0.0045 | 0.1859±0.0047 |\n| **0.1** | 0.1812±0.0038 | 0.1796±0.0033 | 0.1774±0.0031 | 0.1813±0.0040 | 0.1856±0.0047 | 0.1867±0.0049 |\n\n**Q5. Statistical significance of improvements**\n\n> **Reply**: We acknowledge that LL and GI improvements appear modest compared to Bayesian Optimization and Contextual Bandits results. To address whether these improvements are statistically meaningful, we conducted rigorous significance analysis using paired t-tests across 10 independent runs with different random seeds. Table 3 shows that all improvements are statistically significant with p-values well below 0.05, confirming these are reliable enhancements rather than random fluctuations. The key insight is that small numerical differences can be highly significant when experimental variance is low and improvements are consistent across runs. In regression tasks, even modest enhancements in likelihood and uncertainty quantification translate to substantial practical benefits for applications requiring reliable probabilistic predictions. The larger improvements in BO and Contextual Bandits reflect the different nature of these tasks, where our method's ability to escape sharp minima and find flatter solutions has more pronounced effects on optimization trajectories and decision-making performance.\n\n**Table 3: Statistical Significance Analysis (p-values from paired t-tests)**\n\n| **Comparison** | **RBF-LL** | **Matérn-LL** | **Periodic-LL** | **GI** |\n|----------------|-------------|---------------|-----------------|---------|\n| Gen-CNP vs CNP | 0.0241 | 0.0186 | 0.0354 | 0.0289 |\n| Gen-NP vs NP | 0.0193 | 0.0267 | 0.0421 | 0.0176 |\n| Gen-ANP vs ANP | 0.0312 | 0.0398 | 0.0445 | 0.0234 |\n| Gen-BNP vs BNP | 0.0287 | 0.0341 | 0.0383 | 0.0356 |\n| Gen-TNP vs TNP | 0.0365 | 0.0412 | 0.0487 | 0.0298 |\n\nAll p-values are well below 0.05, confirming statistical significance with high confidence levels.\n\n**References**\n>[1] Chen Q, Shui C, Marchand M. Generalization bounds for meta-learning: An information-theoretic analysis[J]. Advances in Neural Information Processing Systems, 2021, 34: 25878-25890."}, {"review_id": "00k07aH55z", "reviewer": "Reviewer_Wfzp", "summary": "The paper proposes an information-theoretic approach to enhance generalization in Neural Processes (NPs). It combines dynamical stability regularization (based on second-order curvature metrics) with an optimization-aware noise-injection strategy during training. Through theoretical analysis and experiments on standard benchmark tasks, the authors claim improved generalization and optimization stability by guiding the model toward flatter minima.", "questions": "Please see the weaknesses.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper is well structured and accessible even to readers without extensive theoretical backgrounds such as myself. The theoretical concepts (e.g., mutual information and dynamical stability) are well explained. The approach is carefully motivated through rigorous theoretical discussions linking mutual information, generalization bounds, and curvature of the loss landscape. Literature on Neural Processes and information-theoretic generalization bounds is thoroughly reviewed, providing valuable context for their contributions. Next, I will list the weaknesses:\n\nNoise injection into stochastic gradient descent is well-studied in machine learning optimization (e.g., stochastic gradient Langevin dynamics and various noise-based regularizers). While the authors clearly articulate their approach, the novelty of explicitly injecting noise into the parameter updates is unclear. The authors do not sufficiently differentiate their proposed noise injection strategy from existing methods. The paper would significantly benefit from explicitly clarifying its novelty—perhaps through comparison to classical methods such as SGLD or entropy-SGD—and outlining how it improves upon or differs from these well-established approaches.\n\nA core theoretical argument made by the authors is that their method favors flatter minima, leading to better generalization. However, this claim lacks direct empirical validation. The paper does not provide explicit demonstrations—such as visualizing or analyzing synthetic loss landscapes, Hessian eigenvalue spectra, or trajectories during optimization that could convincingly illustrate how the introduced noise injection explicitly leads to flatter minima. Providing visualizations or constructing simplified synthetic experiments to highlight this connection explicitly would significantly strengthen the authors' key argument.\n\nThe authors scope their work within the context of Neural Processes and meta-learning frameworks, but the justification for this remains unclear. The connection between meta-learning and the relevance of flatter optima is not explained well. It is not evident why the proposed regularization and noise injection approach should be particularly suitable or beneficial for NPs. Clarifying this connection, and potentially extending discussions or experimental results to conventional training regimes, would improve the significance and clarity of the contribution. Moreover, the paper does not adequately compare its performance to widely used meta-learning benchmarks or state-of-the-art meta-learning algorithms (e.g., MAML, Reptile, ANIL, Bayesian meta-learning methods, or gradient-based meta-learning frameworks).  \n\nA significant limitation is the absence of explicit comparisons against well-known optimization or regularization strategies, which are commonly used to promote flatter minima. For instance, methods like learning-rate scheduling, batch-size reduction, weight decay, or dropout are standard baselines that naturally encourage flatter solutions and improved generalization. Without benchmarking their method against these conventional approaches, the paper cannot convincingly demonstrate the unique advantages of its approach. Including comparisons against such widely adopted techniques would clarify whether the proposed method provides meaningful additional value beyond existing, simpler methods.\n\nThe authors employ Hessian-based sharpness measures (e.g., Hessian trace and Frobenius norm) to quantify curvature. However, these metrics are known to be sensitive to reparameterization (the same model expressed differently can produce different curvature values). For example, recently proposed reparameterization-invariant sharpness metrics, which measure changes in the loss relative to probability distributions rather than parameter-space perturbations, could provide more robust insights into solution flatness and generalization. The absence of these modern invariant metrics limits the validity and interpretability of their sharpness analysis. Employing or comparing to reparameterization-invariant metrics would considerably strengthen the theoretical and empirical claims.", "was_revised": true, "final_justification": "The issues that I raised were addressed well with additional experiments.", "rebuttal": "Thanks for your positive and valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n**Q1. Novelty of Noise Injection Strategy**\n\n> **Reply**: We establish a novel generalization bound for Neural Processes that connects their performance to the mutual information $I(\\theta; D_{1:m})$ between model parameters and training tasks. This bound reveals that tighter generalization requires controlling both the complexity of learned representations and their sensitivity to task variations. Based on this theoretical foundation, we systematically derive two complementary components: DSR regularization that directly constrains $I(\\theta; D_{1:m})$ through Hessian properties, and adaptive noise injection $\\sigma^2_s = \\eta_s/\\gamma$ that enhances optimization trajectory stability across task distributions. Unlike SGLD's thermodynamic sampling or entropy-SGD's diversity-seeking exploration, our noise injection is contextually modulated based on meta-learning requirements, targeting the uncertainty inherent in generalizing across diverse task families. This creates an adaptive mechanism that responds to meta-learning dynamics rather than applying uniform perturbations, making it particularly effective for few-shot learning scenarios where models must balance expressiveness with generalization across limited data.\n\n**Q2. Direct Empirical Validation of Flatter Minima**\n\n> **Reply**: We conducted additional experiments to directly demonstrate that our method achieves flatter minima. We computed Hessian-based curvature measures including Hessian Trace (sum of eigenvalues indicating overall curvature) and Maximum Eigenvalue (capturing the sharpest direction) using finite differences approximation around converged parameters on 1D regression with RBF kernels across 5 random seeds. Table 1 clearly shows that our Gen-variants consistently achieve significantly lower curvature values across all measures, providing direct empirical evidence that our method finds flatter minima. We will add intuitive visualization plots showing loss landscape contours and optimization trajectories in the revised version to provide clearer visual evidence of the flatter minima achieved by our approach.\n\n**Table 1: Hessian-based Curvature Measures**\n| **Method** | **Hessian Trace** | **Max Eigenvalue** |\n|------------|-------------------|-------------------|\n| CNP | 68.436 ± 15.220 | 1.847 ± 0.592 |\n| Gen-CNP | 50.750 ± 13.183 | 1.233 ± 0.314 |\n| NP | 45.651 ± 18.475 | 1.934 ± 0.476 |\n| Gen-NP | 16.581 ± 18.340 | 0.647 ± 0.387 |\n| ANP | 52.727 ± 22.869 | 1.545 ± 0.551 |\n| Gen-ANP | 36.898 ± 21.921 | 0.778 ± 0.329 |\n\n**Q3. Connection to Meta-Learning and Benchmark Comparisons**\n\n> **Reply**: The connection between meta-learning and flatter optima lies in the generalization hierarchy of meta-learning. Traditional learning optimizes for single-task performance, while meta-learning must generalize across task distributions. Flatter minima in meta-parameter space correspond to more robust representations that are less sensitive to task-specific variations, which is crucial for few-shot adaptation. Neural Processes must encode prior knowledge about function families while remaining flexible for new tasks, and our approach specifically targets this by regularizing learning dynamics toward solutions stable across task variations. To address benchmark comparisons, we extended our evaluation to other meta-learning methods, showing consistent improvements across different paradigms beyond just Neural Processes. We compared against MAML[1], a gradient-based meta-learning algorithm that learns optimal initialization parameters for rapid adaptation, and MALIBO[2], a representative Bayesian meta-learning method that uses probabilistic models to capture task uncertainty, demonstrating that our approach provides consistent benefits across diverse meta-learning frameworks.\n\n**Table 2: Meta-learning method comparison (MAE)**\n| **Method** | **MAE** |\n|------------|-------------------|\n| NP | 0.1743 ± 0.0029 |\n| Gen-NP | 0.1723 ± 0.0025 |\n| MAML | 0.2845 ± 0.0156 |\n| Gen-MAML | 0.2676 ± 0.0122 |\n| MALIBO | 0.1295 ± 0.0206 |\n| Gen-MALIBO | 0.1084 ± 0.0141 |\n\n**Q4. Comparison with Conventional Optimization Strategies**\n\n> **Reply**: Our approach operates on a different technical pathway from conventional strategies like learning rate scheduling, weight decay, and dropout - they can coexist without conflict since they target different aspects of optimization. Both our Gen-NP variants and baseline methods use identical conventional optimization setups including Adam optimizer with L2 regularization (weight_decay=1e-4), cosine annealing scheduling, and gradient dropout (rate 0.1). Since all methods share the same optimization foundation, the observed improvements directly result from our DSR regularization and adaptive noise injection components. The key difference is that conventional methods provide general regularization, while our approach offers task-distribution-aware regularization through adaptive noise scaling $\\sigma^2_s = \\eta_s/\\gamma$ that specifically targets meta-learning dynamics. This demonstrates that our method adds meaningful value beyond existing techniques by addressing the unique challenges of Neural Processes in few-shot learning scenarios.\n\n**Q5. Reparameterization-Invariant Sharpness Metrics**\n\n> **Reply**: You raise an excellent point about reparameterization sensitivity. To address this limitation, we implemented two reparameterization-invariant metrics as shown in Table 3. Effective Sharpness[3] normalizes perturbations by adaptive scaling factors ($\\hat{\\epsilon} = \\epsilon / \\max(|\\theta|, \\gamma)$ with $\\gamma = 1e-3$, $\\rho = 0.05$), while Distribution Sharpness directly measures output sensitivity to parameter perturbations ($\\epsilon = 0.01$). Both metrics are computed every 50 training steps using fresh data batches. Table 3 demonstrates that our Gen-variants achieve substantial reductions across both invariant metrics compared to baselines. This confirms that the observed flatness represents genuine improvements in loss landscape geometry rather than artifacts of parameter scaling, validating that our approach achieves meaningfully flatter minima robust to different measurement methodologies.\n\n**Table 3: Reparameterization-Invariant Sharpness Metrics**\n| **Method** | **Effective Sharpness** | **Distribution Sharpness** |\n|------------|------------------------|---------------------------|\n| CNP | 0.142 ± 0.086 | 0.587 ± 0.134 |\n| Gen-CNP | 0.094 ± 0.041 | 0.312 ± 0.098 |\n| NP | 0.115 ± 0.062 | 0.428 ± 0.115 |\n| Gen-NP | 0.057 ± 0.034 | 0.195 ± 0.073 |\n| ANP | 0.189 ± 0.071 | 0.516 ± 0.127 |\n| Gen-ANP | 0.108 ± 0.038 | 0.268 ± 0.084 |\n\n**References**\n>[1] Finn, C., Abbeel, P., & Levine, S. (2017). Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning (ICML). \\\n>[2] Pan J, Falkner S, Berkenkamp F, et al. (2024). MALIBO: Meta-learning for likelihood-free Bayesian optimization. In Proceedings of the 41th International Conference on Machine Learning (ICML). \\\n>[3] Kwon, J., Kim, J., Park, H., & Choi, I. K. (2021). ASAM: Adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In Proceedings of the 38th International Conference on Machine Learning (ICML)."}]

Author rebuttal:

Meta-review: The paper addresses the limited theoretical understanding of generalization in Neural Processes (NPs). It analyzes their generalization gap using an information-theoretic framework. The authors derive an upper bound on the generalization error based on the mutual information between model parameters and the training data. The paper then proposes methods to tighten this bound. The methods have strong theoretical justification, and the results are encouraging.

All reviewers have unanimously agreed to accept the paper, which presents an important study of neural processes. The initial scores were lower, and the authors took care to answer reviewers' questions. As a result, the confusions were resolved and the reviewers increased their scores.
Decision: Accept (poster)