Neural Processes with Stability
Authors: Huafeng Liu
Liping Jing
Jian Yu
Venue: NeurIPS 2023 poster (2023)
Keywords: Neural processes
stability
TL;DR: We provide theoretical guidelines for deriving stable solutions with high generalization by introducing the notion of algorithmic stability into NPs.

Abstract: Unlike traditional statistical models depending on hand-specified priors, neural processes (NPs) have recently emerged as a class of powerful neural statistical models that combine the strengths of neural networks and stochastic processes. NPs can define a flexible class of stochastic processes well suited for highly non-trivial functions by encoding contextual knowledge into the function space. However, noisy context points introduce challenges to the algorithmic stability that small changes in training data may significantly change the models and yield lower generalization performance. In this paper, we provide theoretical guidelines for deriving stable solutions with high generalization by introducing the notion of algorithmic stability into NPs, which can be flexible to work with various NPs and achieves less biased approximation with theoretical guarantees. To illustrate the superiority of the proposed model, we perform experiments on both synthetic and real-world data, and the results demonstrate that our approach not only helps to achieve more accurate performance but also improves model robustness.

Neural Processes with Stability
Huafeng Liu, Liping Jing, Jian Yu
Beijing Key Lab of Traffic Data Analysis and Mining, Beijing Jiaotong University The School of Computer and Information Technology, Beijing Jiaotong University (hfliul, lpjing, jianyu]ebjtu edu.
Abstract
Unlike traditional statistical models depending hand-specified priors, neural processes (NPs) have recently emerged as a class of powerful neural statistical models that combine the strengths of neural networks and stochastic processes. NPs can define flexible class of stochastic processes well suited for highly non-trivial functions by encoding contextual knowledge into the function space. However, noisy context points introduce challenges to the algorithmic stability that small changes in training data may significantly change the models and yield lower generalization performance. In this paper, we provide theoretical guidelines for deriving stable solutions with high generalization by introducing the notion of algorithmic stability into NPs, which can be flexible to work with various NPs and achieves less biased approximation with theoretical guarantees. To illustrate the superiority of the proposed model, we perform experiments on both synthetic and real-world data, and the results demonstrate that our approach not only helps achieve more accurate performance but also improves model robustness.
Introduction
Neural processes (NPs) 10 constitute a family of variational approximation models for stochastic processes with promising properties in computational efficiency and uncertainty quantification. Different from traditional statistical modeling for which a user typically hand-specifies a prior (e.g., smoothness of functions quantified by Gaussian distribution Gaussian process (25), NPs implicitly define a broad class of stochastic processes with neural networks in a data-driven manner. When appropriately trained, NPs can define flexible class of stochastic processes well suited for highly non-trivial functions that are not easily represented by existing stochastic processes.
NPs meta-learn distribution over predictors and provide way select inductive bias from data to adapt quickly to a new task. Incorporating the data prior into the model as an inductive bias, NPs reduce the model complexity and improve model generalization. Usually, NP predictor is described as predicting set of data (target set) given a set of labeled data (context set). However, the number of noise in data introduces challenges to the algorithmic stability. In NPs, models are biased to the meta-datasets (a dataset of datasets), so small changes in the dataset (noisy or missing) existing NPs
may significantly change the models. As demonstrated in previous work 99 10 14 cannot provide stable predictions under noisy conditions, which may introduce high training error variance, and minimizing the training error may not guarantee consistent error reduction on the test set, i.c. low generalization performance [2]. In this case, algorithmic stability and generalization performance have strong connections and an unstable NP model has low generalization performance.
stable model is one for which the learned solution does not change much with small changes in training (2). In general, heuristic techniques, such as cross-validation and ensemble learning, can be adopted to improve the generalization performance. Cross-validation needs to sacrifice the limited training data, while ensemble learning is computationally expensive on training sub- models. Recently, there are several improved NPs focused on considering model stability and
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

improving generalization performance empirically, such as hierarchical prior 127. stochastic attention mechanism [15], bootstrap [33. and Mixture of Expert [28]. However, most of them are unable to investigate the theoretical bound of the generalization performance of NPs. It is desirable to develop robust algorithms with low generalization error and high efficiency.
In this paper, we investigate NP-related models and explore more expressive stability toward general stochastic processes by proposing stable solution. Specifically, by introducing the notion of stability into NPs, we focus on developing theoretical guidelines for deriving a stable NPs solution. We propose method find out subsets that are harder to predict than average, which is a key step for constructing this optimization problem. Based on it, a new extension of NPs with stable guarantees is formulated, which can be flexible work with various NPs and achieves less biased approximation with theoretical guarantees. Considering the model adaptivity, an adaptive weighting strategy is proposed. To illustrate the superiority of the proposed stable solution, we perform experiments on synthetic 1D regression, system identification of physics engines, and real-world image completion tasks, and the results demonstrate that NPs with our stable solution are much more robust than original NPs.
2 Related Work
In this section, we briefly review two different areas which are highly relevant to the proposed method, neural processes, and algorithmic stability.
Neural Processes Neural processes are a well-known member of the stochastic process family by directly capturing uncertainties with deep neural networks, which are not only computationally efficient but also retain a probabilistic interpretation of the model [9] TO, [4] [3]. Starting with conditional neural processes (CNP) (9. there have been several follow-up works to improve NPs in various aspects (6] Vanilla CNP combines neural networks with the Gaussian process to extract prior knowledge from training data. NP [0) introduces a global latent variable to model uncertainty in variational manner. Considering the problem of underfitting in the vanilla NP, Attentive NP [14] introduces the attention mechanism to improve the model's reconstruction quality. []] introduced convolutional conditional neural process (CONVCNP) models translation equivariance in the data. Wang and Van Hoof (27] presented a doubly stochastic variational process (DSVNP), which combines both global and local latent variables. Lee et al. [18] extended NP using Bootstrap and proposed the bootstrapping neural processes (BNP). Kawano et al. 13 presented a group equivariant conditional neural process by incorporating group equivariant into CNPs in meta-learning manner. Wang and van Hoof (2 proposed to combine the Mixture of Expert models with NPs to develop more expressive exchangeable stochastic processes. Kim et al. [1] proposed a stochastic attention mechanism for NPs to capture appropriate context information. Although there are many NP variants to improve the model performance, those do not consider stability to yield high generalization performance.
Algorithmic Stability Stability, as known as algorithmic stability, is a computational learning theory of how a machine learning algorithm is perturbed by small changes to its inputs (2). Many efforts have been made to analyze various notions of algorithmic stability and prove that a broad spectrum B proved that l2 regularized learning algorithms are uniformly stable and able to obtain new bounds on generalization performance. [29] generalized [3]'s ss results and proved that regularized learning algorithms with strongly convex penalty functions on bounded domains. Hardt et al. [12] showed that parametric models trained by stochastic gradient descent algorithms uniformly stable. Li et al. [19] introduced the stability notation to low-rank matrix approximation. Liu et al. (22 proved that tasks in multi-task learning can act as regularizers and that multi-task learning in very general setting will therefore be uniformly stable under mild assumptions. This is the first work to investigate the stability of NPs from theoretical guidelines and derive NPs solutions with high stability.
of learning algorithms are stable in some sense 2 29 [2]
3 Preliminary
Let calligraphic letters (e.g., A) indicate sets, capital letters (e.g., A) indicate scalars, lower-case bold letters (e.g., a) denote vectors, and capital bold letters (e.g., A) indicate matrices. Suppose there is a dataset D = (X, y) = {(x;, yz) ) with N data points = X2,, xx]] E R NxD and corresponding labels y = U2, .NN] R... Considering an arbitrary number of data points DC = (xx.' = {(x,, M4 hec, where S {1, 2,.., N) is an index set defining context information, neural processes model the conditional predictive distribution of the target values y" {ys hye

---

at some target data points X {xi)iET based on the context Dc, i... P(y (",") Usually. target set is defined as [1, ... .N.. Only in CNP T 21,2, _N and Tnc 0. In this paper, we define T (1, ... N} for all NPs, i.e. conditional predictive distribution is P(y|X, II.,,
D.).
Fundamentally, there are two NP variants: deterministic and probabilistic. Deterministic NP [9] i.c. CNP, models the conditional distribution as P(y|X, DC = P =P(y'X,r'), r", where r I is an aggregated feature vector processed by a function that maps D into a finite-dimensional vector space in intpuuttttoonnivriiann way. In probabilistic NPs [0], a latent variable z E R is introduced to capture model uncertainty and the NPs infer Ps (x given context set using the reparameterizztion trick [16] and models such conditional distribution as (y(ys) x., DC) = / (M xx, D 2))Paxz (D)) and it is trained by maximizing an ELBO: (x|X.) [log [y - K ( y) (ol
To achieve fast prediction meta-learn a distribution over predictors. To perform meta-learning, we require a meta-dataset (dataset of datasets). We consider an unknown distribution μ on an instance space X' x y. and set of independent sample D = {xx, ) drawn from (x,, y/)
Meta Training NP Prediction
new context
test time, NPs
and D ~ n² Suppose meta-dataset contains M datasets DI.M = (Dm Jmd= with D., = (De. ) we assume that all M datasets drawn from a common environment T, which is a probability measure on the set of probability measures on X' x y. The draw of μ ~ T indicates the encounter of specific learning task μ in the environment т. For simplicity, we assume that each dataset has the same sample size N. Following the previous work related to multi-task learning (24 and meta learning [5]. The environment induces a measure J,N.r on (X x y) such that J.N, (A) By-- [n (A)), (A)), VA S (X x y) Thus a dataset Dm independently sampled from task μ encountered т, which is denoted as D,,. ~ J.Nr for E [M).
Suppose there exists meta parameter 0 indicating the shared knowledge among different tasks. In this case, a meta learning algorithm Ameta for NPs takes meta-datasets D I as input, and then outputs meta parameter Ameta (D1:2s) Popp When given new test dataset D, we can evaluate the quality of the meta parameter θ by the following true risk:
(0) = ED- Bu.. (0)
where R2. Ex,,,
un you log (a (x,, D.). Usually, T and μ are unknown, can only estimate the meta parameter 8 from the observed data Dr.M. In this case, the empirical risk w.r.t @ is:
M Ro (0) = 1/M) > Ea-P, Dipg Rom (0)
(2)
where RD.,, (0) -0 (1/N) E log Po (x) 2=).
NPs have various strengths: 1) Efficiency: meta-learning allows NPs to incorporate information from a new context set and make predictions with a single forward pass. The complexity is linear or quadratic in the context size instead of cubic as with Gaussian process regression; 2) Flexibility: NPs can define a conditional distribution of an arbitrary number of target points, conditioning an arbitrary number of observations; 3) Permutation invariance: the encoders of NPs use set property [2 to make the target prediction permutation invariant. Thanks to these properties, NPs are widely-used in lots of
tasks, c.g.. Bayesian optimization S recommendation [20 211. physics engines controlling (27) etc.
While there are many NP variants to improve the performance of NPs (9 10 14 13 15 28), those do
not take model's stability into consider account yet, which is the key the robustness of the model.
4 Problem Formulation
Stability of NP A stable learning algorithm has the property that replacing one element in the training set does not result in significant change to the algorithm's output 22] Therefore, if we take the training error as a random variable, the training error of a stable learning algorithm should have a small variance. This implies that stable algorithms have the property that the training errors are close to the testing error (2). Based on the defined risks, the algorithmic stability of approximate {y, her in NPs is defined as follows.
Definition 4.1. (Algorithmic Stability of Neural Processes) For any measure AN,- on (X' x >)) such that pN.+(A) (A) = ((rr(H((A)) VA C (X x V),,, sample M datasets DI.ss from ,N,, randomly, For given > we say that RoDLS (0) is 8-stable if the following holds:
P ([-(()) HD as (0) 1 c) 1 - d.
(3)
3

---

The above stability for NPs has the property that the generalization error is bounded, which indicates that minimizing the training error will have a high probability of minimizing the testing error. This new stability notion makes it possible to measure the generalization performance between different NP approximations. For instance, for any two meta-datasets D and Di.M from JN,,,, train NPs on Pla and ai are -t-stable and 5y-stable, respectively. Then Ret.ar (0) is more stable than Req... (0) if 81 < 62. This implies that Rot. (8) is close to R. (0) with higher probability than R (0), i.e. minimizing Roj..r (9) will lead to solutions that are of high probabilities with better generalization performance than minimizing Rpj., (0).
Based on the above analysis, we can see that the reliability of data points is crucial to the success of NPs and frail NPs are susceptible to noise.
Stability vs. Generalization Error data, incomplete and noisy introduces challenges to the algorithm stability, NP models are biased to the quality of context data and target data, so small changes in the
The sparsity of the
100%
0.25
Sttabiiiyy
80%
Generallzann teneralization
Coneralization
Stability
60%
0.15
training data (noisy) may significantly change the mod-
els. In this case, unstable solutions will introduce high training error variance, and minimizing the training er- may not guarantee consistent error reduction on the testing dataset, i.e., low generalization performance. In other words, the algorithm stability has a direct impact on generalization performance, and an unstable NP so- lution has low generalization performance. We take NPs with ID regression task as an example [9] to inves-
0.1
20%
0.05
0%
50
The number of replaced datasets
Figure 1: Stability vs. generalization error with different numbers of replaced noisy datasets.
tigate the relationship between generalization performance and stability of NPs. The total number of training and testing datasets is 200 and 100. We trained the NPs model with curves generated from the Gaussian process with RBF kernels by replacing the normal data dataset with noisy dataset, i.e. the number of replaced datasets is turned in {1, 10, 20, 50, 80, 100}. We quantify stability changes of NPs with the generalization error when the number of replaced datasets increases from 1 to 100. We compute the difference between training error and test error to measure generalization error. We define the difference between test error and training error as (0) RD v. (0), and compute P( (0) RD, A (0)] S e) with 100 different runs to measure stability. Here we choose in Definition 4.1 as 0.0015 to cover all error differences when the number of replaced datasets 1. As shown in Figure D the generalization error increases when the number of replaced points increases since the testing error becomes lower. On the contrary, the stability of NP decreases with the number of replaced points increases. This indicates that stability decreases with generalization error increases. This study demonstrates that existing NPs suffer from lower generalization performance due to low algorithmic stability. Therefore, it is important to develop stable solution for NPs that offers good generalization performance.
5
Method
In this section, inspired by the previous work 19 we present stable solution for NPs with stability and high generalization. Algorithmic stability provides an intuitive way to measure the changes in the outputs of learning algorithm when the input is changed. Various ways have been introduced to measure algorithmic stability. Following the definition of uniform stability [2], given a stable NP, the approximation results remain stable if the change of the datasets. For instance, we can remove subset of easily predictable data points from Dia to obtain D minimizing both DI:M and D' as together will be more stable than the solution of minimizing DI:M only. The following Theorem formally proves the statement.
desirable that the solution of
Theorem 5.1. Let DI:M (M 2 2) be a sampled meta-datant of measure HN.T- Let Dg E D be subset of the meta-dataset, which satisfy that v(x,, v.) E D.. log Po Po X,, DC) E RR (0). Let DILM = DI.M - D., then for any e > O and 1 >> wo > 0, 1 > wi > 0 (Wo + wj = 1), wo WoRDL (0) + un RD m (0) and RDD (0) are 8n St-ble and St-stalle, respectively, then d1 S 62.
Proof. Let's assume that (0) Rr a (0) E [- an] and R. (wo Rw (0) + wi Roy M [-a2, a2] are two random variables with zero mean, where a1 (..(Ht-(8) (0) RD M (0)) and
(0)

---

(u)( - (wolDD,ar (0) + Bog (0))). Based on Markov's 's inequality for any t > 0, we have
B (2.()) (0))
P(R-(0)- Re (0) 2)
(4)
etc
Based on Hoeffding's ' lemma P we have Ele*(R,(0)-Ro))
RD (1)) ed i.e. P(R,(0) Ro (0) Z
2 s) S efc Similarly, we have P(R, (0) RD (0) S -c] S
e dtr Combining those two
20
inequalities, we have (0) - RD (0) 2 c) - eic i.e.
2e 2uu
P( (0) RD (0) S c) 2 1-
(5)
ele
Similarly, we have
20 e¹
P ( (we Ro (0) + wn Roy (0)) sr) 2 1
(6)
In this case, the relationship between 01 and 02 is
42 = (0) RD I (0) + wi RR (0) Pp Roy 1s (0))
= suu (R- (0) - RD (0)) + wi sup P (0) - Ro (0)}
(7)
01 Austy RD n (0) RD (01}
Since v(x., ys) E D., - log Po(ys X,, DD) I RD (0). we have (1/NV) EN=1 log Po Xx, D.) S AD (0). Then, since Dr.M = D. Di... This means that sup (0) Roy (0)) S 0. Thus,
2 P93 2: dtc ..e. 81 S 82.
we have ag S 01. This turns out that
ofc S
Theorem 5.1 indicates that, if we remove a subset that is easier to predict than average from D1:Mr to form D... then wolld, (0) + wi wiRke (0) has higher probability of being close to Rr (0) than Royar (0). Therefore, minimizing wo (0) + us Rpi (0) will lead to solutions that have better generalization performance than minimizing RD (0).
However, Theorem 5.1 only proves that it is beneficial to remove an easily predictable dataset from DiAM to obtain Pi but does not show how many datasets we should remove from Dr.M. Actually, removing more datasets that satisfy - Log Ps (x)x,, D I RD (0) can obtain better Di.M. as shown
in following Theorem 5.2
Theorem 5.2. Let Dras (M 2 2) be a sampled mela-datasel of measure JN,.- Let DD1 and D22 be two subsets of Drac, which satisfy D=2 Da1 C D1.M. D.2 and Da satisfy that V(x,, ys) e D.L, - log Po (ollx, D I RD (0). Let DLAM = Dr.M- D.i and Di.M = Dr.M - -D,, then for any e > and 1 wo 0, 1 > wi > (wo + w1 = 1), WoRDD (0) + w WiR (0) and
WoRD (0) + un R (0) are 61 stable and 52-stable, respectively. then 81 S 82.
Proof. Let's assume that R. (0)-
(0)+w; (0)) I-a1, ai] and R., (0) (woRD, (0)+
wi Repg... (0)) E [g] are two random variables with 0 mean, where a1 sup R. (0) (wo 2 (0) + un Rol.,. (0)) and 0g = sup (0) (wo RD as (0) + ur Roj (0))).
Then, based on Markov's 's inequality and Hoeffding's 's lemma, we have
P( (+) - (wo Ro (0) + w. Ret (8) sc) 2 2c Paf
P
pla
(8)
2c Pa3
P
P (R+(()) - (wo Ro (0) + w R., MM (8) st) 2
pla
Since v(x,, 31) D=i, - log Po DC) S a (0) and Da2 C Da C DI.M, we have Roj
(0)
Rpg MM (0). Thus, we have sup 1.,. (0) Rop (0)) S since a 02 wisup (Rp (0)
1:M
Rog (0)) and a3 S 0g. Then, we can conclude that 81 S 82.
(EEteededd version of Markov's 's Inequality) Let x be real-valued non-negative random variable and (p(-) be a nondecreasing nonnegative function with (p()= > O. Then, for any e > 0, P(z 2 e) < E(p(z)]
v(c)
"(Hoeffding's Lemma) Let x be real- random variable with zero mean and p(z E (a, b)) = 1. Then. for any E IR, [] & exp (1 23 a)
5

---

Theorem 5.2
indicates that removing more data points that are easy to predict will obtain more stable
NPs. Therefore, it desirable to choose Di.Mv (i.e. DI.M - D.) as the whole set which is harder to predict than average, i.e. the whole set satisfying V1) Pii.s log Po(us xx, DD) < RD (0). Without loss of generality, we can extend Theorem b.2 considering several harder predicted sets to obtain more stable solution by minimizing them all together. In this case, we need to prove that the stability of a model with K subsets is better than the model with K - 1 subsets, as shown in Theorem 5.3
Theorem 5.3. Let D1.M (M 2 2) be sampled meta-dataset of measure J.N.-. Let D.I. D.,..... DaK C DIM be K subsets and satisfy V(x, vk) E <Dx (k E [K]. log Po DC) < RD, (0). Let Di = DIMM Da* for all k E [K], then for any c <> 0 and 1 > wa > 0 for all
k E [K] (wo + 1 +... + WK = 1), woRD,,s (0) Ek=1 ux Rot (0) and (wo + wx ( RD, (0)
2k=1 W. (0) are ó1 stable and 82 respectively, then d1 S 62.
I:M
Proof. For simplicity, we denote wo wolD (0) + Σx WeRD (0) as R1 and (wo + wx) RD (0) + E Wy Ry (0) as Rg. Let's assume that (0) Rı E [-a1,00 ai] and R- R₂ E [- 02 02] are
two random variables with O mean, where a2 = sup [t- (0) - R and 0g sup [Rr (0) - Ra). Then,
based on Markov's ' incquality and Hoeffding'ss s lemma, we have
(1 Ral ()5) 21 2
(9) Similar to the proof of Theorem 5.3 we have RoK (0) 2 RD (0), which indicates that sups RD (8) Rok
(0) Ri| S 22121
etc
ete
(0)) < 0. Since 02 = + wysuup RD (0) - Ro (0)}, we know that
a2 S aj. Thus we can conclude that 2 Aa2² I 26 2012
elc .i.. 81 62
etc
Based on Theorem 5.3 know that optimization DI:M and more than one hard predictable subsets of DI:M can achieve more stable prediction. However, how to select data points that are difficult to predict from PI.M is still a challenging problem, especially, since we need to select K subsets.
5.1 The Proposed Solution
According to the analysis of stability in NPs, we propose a stable solution for NPs to achieve model stability with the aid of hard predictable subsets selection. Specifically, we introduce a solution to obtain those hard predictable subsets based on only one set of casily predicted data points which can be broken into four steps:
(1) Selecting existing NP model (e.g., CNP [. NP 10 and training it with meta-dataset Di:M:;
(2) Selecting an easily predicted subset DD C Di.M, which satisfies that v(xi, v.) E D.,
log Po Xx, DC) S (
(3) Dividing D, into K non-overlapping subsets D.2,...... ,D. and satisfying _K_ Dak D.;
(4) Defining K subsets that are difficult to predict, i.e. D DI.M- Da for all & E [K]:
Thus, a new extension of NPs is given as L arg min We Ro (0) Σx
wy R Dt (0),
(10)
k=m
where Wo, "1,".. WK indicate the contributions of each component and satisfy Σk We 1.
The whole learning algorithm is given in Algorithm 1 From steps to 12, obtain K different hard predictable subsets, and the complexity of lines to 12 is O(MN). The complexity of line 11 is related the applied NP models (such CNP, NP, ANP, etc). Thus, the computational complexity of NPs with our stable solution is similar to the original NPs. As shown in Algorithm 1, we need to pre-train the base model select samples. In fact, the pre-trained model can not only be used for sample selection but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster.
Stability Guarantee Here we give a theoretical guarantee of the proposed stable solution.
Theorem 5.4. Let DaM (M 2 2) be a sampled meta-datasel of measure J., Let D. C Dr:M which satisfies that (Xi, 2)) E D., log Po x1, DC) S RD (0). By dividing D. into K subsets

---

Algorithm 1 Learning algorithm for stable NPs
Input: Meta-dataset DI.M, 3 > 0.5 a predefined probability for data selection. D, 0.
I: Train a NP model with parameter & based on RD (0);
2: for (x,, B/) DI: do
3:
randomly generate split parameter E [0,1];
4: if ((- log Pa (v:) xx,, DC) S RD (0)) & p S (8)
5: or ((- log (1 [x, D > RD (0) & S B) then
6:
D3 - D. Vi):
7: end if
8: end for
9: divide D. innoo Dx1, Da2. DaN with Das D.;
10: for = 1.. do
11:
DiE = D1:M Dak:
12: end for
13: update parameters by optimizing 0* arg ming we Ro (0) (0)+E*= Ex=1 W. jiMs (0):
Output: The learned optimal parameters 0*
Da,,... .DaK which satisfy that = Dak Dr.Ms Let D -D.M D. and Di.M D1.M Da for all [K], then for any > 0 and 1 > Wk > 0 for all K E [K]. (wo + w1 +... WK = 1), woRDD (0) + Σk= w.RBpt (0) and woRD (0) (1 wo Rog (0) are 81 -t-stalle and ö2-stable, respectively, then 51 82-
1:M
R= wx R R (0) as R2 and we RID (0) + (1 wo) Rog (0) R2. Let's assume that (0) R1 E [[a1,,a] a1l and (0) R₂ [-a2, 02) are two i. random variables with O mean, where (u)( R1} and 02 sup(R (0) R2}. Then, based
Proof. For simplicity, we denote we RD (0) Σ
on Markov's 's inequality and Hoeffding's 's lemma, we have
2e 2r3
P(R,, (0) Hil sc21-
P( Ra S < 2
(11)
etr
et
wk E [K], D. D. and v(x,, vi) E D., - log Po Xx,, D S RD (0), we have Ri (0) < Rpg. (0). By combining the above inequalities over all &s E [K]. we have
Σ k=1 we Ryp (0) Σx=
k= we Rep of (0) (1 wo) Rog (M).
(12)
Thus, su. R1 S (R Ro), i.c. an S az Thus we have S 82.
According to the above theorem, we can achieve model stability by selecting only one easily predicted subset.
Experiments
We started with learning predictive functions on synthetic datasets, and then high-dimensional tasks, e.g., system identification on physics engines, image completion, and Bayesian optimization, were performed to evaluate the properties of the NP-related models.
6.1 ID Regression
To verify the proposed stable solution, we combined the stable solution with different baseline NP classes (CNP (9) NP [T0]. ANP [4, .CovCCNP [I], ConvNP 6), and their bootstrapping ver- sions 118 and compared them on ID regression task. Among them, BCNP, BNP, BANP, BCO and NC are recently proposed stable strategies for NPs with Bootstrap Specifically, the stochastic process (SP) initializing with 0 mean Gaussian Process (GP) ()) GP ((.)))(())) -)) indexed in the interval x E [-2.0. 2.0.2 2.0] were used to generate data, where the radial basis function kernel and Matern Kernel were adopted for model-data mismatch scenario. More detailed information can be obtained in the Appendix. We investigated the model performance in terms of different noise settings. We introduced Gaussian noise (o, 1) and added noise to different proportions of the data, such as (0%, 5%, 10%, 15%). Table the average log-likelihoods comparison in terms of different noise proportions. The best result is marked bold. First, we can see that if we adopt the robust solution in baselines, the model achieves the best results on all the datasets, showing the effectiveness of the
7

---

Table I: Average Log- Log-likelihoods over all context and target points realizations from Synthetic Stochastic Process on the different percent of added noise. Here we set the context size to 20. (Mean + Std). Note that adding 'S' before the original model name is model with our stable solution.
Kernel
Method
Original
No
(00%)
context
target
context
target
context
target
context
target
CNP
0.8724
0.4334
0.8522sucems
0,4001
0.801420.um
0.3552
0.7152
0.2853_u.00
BCNP
0,9042=0.000
0.4589=6.000
0.877420.00
0.4278=00008
0.8316=0.006
0.3767#0.005
0.7487=0.00
0.3017-0,000
SCNP
0.9255 0..00
0.4733 2o.01
0.8935
0,4478.0.00
2o.517
0.3986.c.uu
0.7621
0.3279 00,000
NP
0.8215_0.004
0.3853+6.006
0.801 20.004
20.000
0.761
+0.000
0.3042=c.00s
0.6722
0.2435
BNP
0.8722-00004
0,4211
0.8321
0.387620.004
0.7922
o04
0.338946.00 002
0.7189
0.2776
SNP
0.8955_0.0008
0.4356
1.8567=0.00
0.4046-20.000
0.8165 +0.16540.004
0.3568
0.7356
0.2955
-0,000
RBF
ANP
1.2563_0.022
0.5763 004
1.2245200007
0.1742 0.034720.00
00g
0,487
a. 007
0.9821 20.005
0,4151
0004
BANP
1.2722_.0.001 0.5887 10.00
1.241 1,241 1:0000
0.5471
100000
0.
1886 10.000
0.4917
a0.000
1.0642...00
0.4327 0.4327_0.005 0.005
SANP
1 1.2831
0.5994.c.o0
on4
1
1.2564_0.004
0.5578_0.00
1.2052
on4
0,5025 o06
1.1243
0.4356
006.
ComCCNP
1,2631
0.,00
0.6421 10.000
1.2353
0.5415
0.1
20.000
0,493666.00 o0s
1.0241
10.000
0.4262
0.4262_0..05 0.005
BConvCNP
1.2761-0.004
0..531
1.247620.004
0.5533
0.1931
D 0.49862c.om
000
1.0716
0.4396_0..0
SComCCNP
1.3991
20.001
0.6793
10.00
0.5623
0.000s
1.2126
10.001
0.5096...00
ons
1.1331
0.006
0.4461
0.000
ConvNP
1,2874
0.033
0.6503
10.0M
1.2371
10000s
0.5451
0.1865
10.004
0,4965
0.9913
0..00
0,4335
0.000
BCowNN
1.2922_00
0.003
0.6627
10.000
1.2505
10.006
0.5583_0.004
0.1971
20.000
0.5025
10.007
1.0731 00231.0.000
0.4436
0.004
SComNNP
1,40366
0.002
0.6831
1o.0ma
1.2671
10.00s
0.5675
1.2188
1o.0rc
0.5157
1o.0ne
1.1389
0.4505
CNP
0.8531 -..000000000
0.2431
0.8231
10.005
0.2144_a.000
0.7761
0.1784
20.0000000 50.007
0,7052
10.005
0.1452
0.006
BCNP
0.8778
0.2762
10.000
0,8487
10.000
0.2477
0.8015
0,205
10.007
0.7378
0.000
0.1766
0..00
SCNP
0.8963_0.00
0.2953
0,8689
10.005
0.2658 -0007
0.8268
00.000
0.2258
10.000
0.7567
0.000
0.1936
0.005
NP
0.7643 0.015
0.204]
10.013
0.7342
0.1725
0.6892
10.094
0.1542
0.006 0.6255
0.1352
0.007
BNP
0.8156-0..06
0.2689
sc.007
=0008
0.2215
0.742 2c.00s
+0.007
=0.00
0.1828
0..000000000
SNP
0.8368
n.w0s
0.2844
10.000
0.8036
Locoss
00000 0.2483
0.7635
10.091
0.2325
0.066 0.6973
0.2016
0.005
Matern
ANP
1,2421
00,002
0.6360
0.0001
21784
0.162226.00
1..202
0.5274-0.008
BANP
1. 1.3456_0.008
0.651426.006
1.3125
0.611520.000
1227222..04
0.171
1771100.00
1.2236_00.00
0,5306_0.000
SANP
0.002 0..653
1.3461 s000s
0.6256=0.004
L3011
20.000
0.1782-20.04
1.2457 =0,005
0.5356-0.002
ConcCNP
1.211500.00
0.641822.004
1,222620.006
0.6085
11832+6.00
0.187 2.187140.007
1.1326_0.000
0,5151
BConvCNP
1..527.0.05
0.661626.0x
onc
100052
0.6235_n.007
1.2767 20 006
0.1952aa.0m ons
1.131520.000
0.5417-0.006
SComCCNP
1. 1.88220.008
1.6734
1.3364-0.04
0.6335_0.000
1,2331
0.2033
1.1521
0.5557
ConvXP
1.2746_0.03 0.6537
200
0.601520.005
20.000
D.19433 ons
1.1358g0.000
0.3397 20.000
BConNN
1. 1.3356_0.001
1.3305 16.080
10.00
0.6383 L..383
1,881
10.000
1..015
1.1415 10.000
0,5338
SC
1.8088_00009
0.6836.c.om
1.3435-000m
0.6417
1.2866.c.m1
0.2025.0.0m ons
1.1521.0.00s
0.5363_.0.00
stable solution. Besides, performances on all methods become less accurate in more complicated settings, while our solution has fewer effects. One interesting observation is that the improvements against the base model on CNP are less significant than NP and ANP. The possible reason that CNP only predicts points out of the context set.
To investigate the model's ability to address model-data mismatch scenar- ios, conducted experiments ID regression tasks with Periodic kernel. Following the setting of BANP and similar noise settings of our previous kernels, we list the results on both original data and noise(+155 data in Table 2 In this model-data mismatch data, stable versions still significantly outperform their corresponding orig- inal versions.
Table 2: Experiments on ID regression data with Periodic kernel.
Periodic
Original
Noise(-15%)
context
larget
context
target
ANP
0.5730=0.00s 44.2345+0.00m
0.352 #0.005
-5.321 =00000
ConvCNP
0.598320.005
-4,0215
0.3658=0.006
-4.5233
CoavNP
0.6125 -3,8952+0.000
0.3756+n.006
-4.3413=00008
BANP
0.625320.000 33.4113
000
0.3651 1g0 ooe
4.251120015
BComCCN
0.63421n.mm -3,4142.0.001
0.37122..000
4.1750 Lo
BComNNP
0.6355
-3,3627 000
0.3768 one
-.01 007
SANP
0.631510.m
-3.3317
0.3748
4.0515
o00
SComyCNP
s0.0zz
-3.1515pa.004
0.3866-00000
13.9551=0cm
SConNNP
0.6551 10.000
-3.1062,00.00
0.3981 10.000
-3.8895_o
6.2 Image Completion
Image completion can be regarded as 2D function regression task and be interpreted being generated from a stochastic process (since there are dependencies between pixel values). Following the setting in previous work [14], we trained the NPs on EMNIST [4] and 32 x 32 CELEBA (23 using the standard train/test split with up to 200 context/target points at training. Detailed experiment settings are given in the Appendix. We evaluated average likelihoodss over all points on realizations from image completion. Table 3 lists the comparisons between NPs with and without our stable solution in terms of original setting and noise setting, and the performance demonstrates the superiority of our stable solution.
6.3 System Identification Physics Engines
The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which detailed 7 27). The Cart-Pole swing-up task is standard benchmark for nonlinear control due to the non- conecriyyy in the dynamics, and the requirement for nonlinear
8

---

Table 3: Average Log-likelihoods over all context and target points EMNIST and CELEBA.
Dutaset
Method
Oniinnl
Noise 13%)
Naise[
10%)
[176
context
target
context
target
context
Target
context
target
CNP
0.9522eo.ca
0.8977 0.7515eo.00ub
0.6336e0.07
0.8212=0.00s
0.65666n.o0r 0,5784+0.00
0.5341-0.0.6
BCNP
0,9678
0.808820.00
2.901520.00s
2.
0.841520.00
0.608920.00 0.0
0.6788 000
0. 0.5715-0.000
SCNP
0,9716e0.00
0,334
+0.000
0.9251-0.00s
0.6971.0.00
0.8674
0.6343+0.007
0.69864c.0m
0.5877
0.5077-0.008
NI
2.0678
0.775020.000
0:901
200000
200000
0.8544
0.6455...oo our
0.7054
000
0.5865
BNP
0.9757 g000s
0.8388
00.
0.8116
200007
10
0.8759
Lo 007
0.6773
0.7451
Lo 00G
0.6237 00000
SNP
0.9847
0.862220.0c
0.8368
0.7844
0.8984
0.6984
0.7653
0.6456-0.004
EMNIST
ANP
2.1125
[0321
an 004
20..0
0.636610 000
0.9022
1.0 001
2.7053 oms
0.8454
000
0,7034_ 20.00
BANP
1.1355 1000
1.0615
10.005
1.0256
0.654910
0.9155
1.
10.020
0.8612
0.7515
SANP
g0000
10877.0.000
L.421
2002120.000
0.677610000
0.9321
20 002
0.7843 oce
0.8732
000
0.7657 -00 006
CoecCNP
1.1363
L0466 10.004
1.0252 20.000
0.6448
0.9116
0.21 15.0.000
0.8621
10 00G
0.7246
000004
BC
1.1425
L.0787 20.000
1001
0.6626:0.om
0.9252:0.com
0.7617-2..00
0.8717
0. 0.7627
SConv
1.1631 go00
1,094
10.001
LO563 20.004
6.677_00001
0.9356_0.om
0.7885 10.
0.8813
L 005
0.0092_0000 0.000
ConcN
1,1415
L056330.00
L0256
0.653610.000
0.9168
0.7171
0.5675
0.7368
NC
1.122400004
40.005
L.0415
0.668420.00s
0.9285-o.com
0.7762+0.000
0.8742
006
0. 0.7127 -0.005
SComNNP
1,1753 1o00
1,934
10.004
10.004 L.041
0.6837 10.002
0.9402
00.000
0.7925 L0.000
0.8923
10 000
0.7853 0.005
CNP
1.0523
0.7845
20.013
2017720.006
0.7438g0011
0.895620
009
0.7344
0.7677
BCNP
1.0452=0c0
0.8015+0.008
L0075=0.009
0.7726+0.00
:05
0,9351 1
0.8376=0.000 000
0.8015
40
0. 0.6816-0.008
SCNP
1.0525
0,8243
20.000
L4348
0. 0,7868
0.9562.0
0.8545
0.8344
0,7045
NP
Locot
0.876619.005
1.104__o.015
1.0355
1.0034
cos
10.061
0,8935
LG 000
6893 _0.006
BNP
1.17320000s
0.8901 20.000
L.137820.00
0.8678
1.041
000
0.8711
0.9256
0. 0,767 =0.000
CELEBA
ANP
SNP
1.1952
1
10.164
0.9062.1.00m 10.000
1,1565
1.137 20.004
0.9866
0.884610001
10000000000 000
1,0488
1,0542
0.0000
206
0.8956 10
10.000 0.8845
0.9425
0.9363
p. mm
1.0 004
0. 0.7985
0. 7,45 0.000
BANP
1,775
L1389
1. 11148_g..0
10015
1,0002:...mm
0.9489
0.8415
SANP
1,0994
10.004
L1688
Lo,00
L005330.001 10.001
10.000
10.001 0.9455
0.9655
10 000
0.8673 0.005
Comy
1,1697
1.169700.04
=00004
L10366
+0.004
L L1445
0.004720.005
1,5422
0.5971
+0.007
0.9521
40
0. 1.7563-0.00
RCansCNP
1.1822
L0467
10.004
1 10.005
1,271 10.000
1.068610.0
0.9317
10.000
0.9542
0,8527 _0.00
SConvCNI
11889...0m
1.015
20.000
1,1553
-0.0c4
L0478.0.00
1.0752
0,9485 +o.000
0,9693
4c 007
0.8714 0.000
Co
1.1767=0c04
100131
L
L0150e0.00
10059
0.9066ee..07
0.9573
0.7779
BConsNP
1.1822.0.co4
10.05
1.1573.0.004
1.035 10:005
1.0762.0.00
000
0.9461 10.000
0,9661
15 000
06366... 0.000
SConvNP
1.1867
1.0668
1.1846
10447+0.000
1.0863+0.00
0.9511+0.008
0.9743
0.8836
Table 4: Bayesian optimization experiments on data generated by different GP kernels.
BANP
SANP
ConVCNP
BCoayCNP
SCowCCNP
BCowNN
SCom<NP
RBF
0..12
0.1037 20.000
0.1166 11156.c..o om
0.001
Malern Periodic
0. 0.15182
0.1316
12121 20000
0. 0.1489 200
0.1301 zome
0.1216.0.00
0. 0,144620.1 mo
0. 0,1242
0.1284
0 1892-20007
0.1 0.1788
0.1672.0.0
1576
0.1487
0.1611
0.1498_
0.1446
controllers to successfully swing up and balance the pendulum. More detailed experimental settings given in the Appendix.
For each configuration of the simulator including training and testing environ- ments, we sampled 400 trajectories of the horizon 10 steps using random controller. During the testing process, 100 state transition pairs were randomly selected for each configuration of the environment, working the maximum context points to identify the configura- tion of dynamies. FigureZssows the pre- dictive Log-Likelihoods (LL) and Mean Average Error (MAE) on Cart-Pole State Transition Testing Dataset. We can see that our stable NPs achieve better likelihood and lower prediction error than the original ones. The variances on the stable one are consistently smaller than all original baselines.
eonggnall curr
0.08
0.04
0.02
(a) LL
(b) MAE
Figure 2: The predictive Log Likelihood (LL) and Mean Average Error (MAE) on Cart-Pole state transition testing dataset.
6.4 Bayesian Optimization
Following the setting in BANP [8] we conducted the Bayesian optimization experiment. Taking GP data with RBF, Matern, and Periodic prior functions examples, we gave the results of ANP, ConvCNP, Co.vNP, Boostraping versions, and our stable versions. To maintain consistent comparison, standardized the initializations and normalized the results. We reported the best simple regret, which represents the difference between the current best observation and the global optimum. As shown in Table 4 NPs.
we can see that our stable solutions consistently achieve lower regret than other

---

Table 5: Predator-prey model results.
Methed
ANP
BANP
SANP
ConCNNP
BComCNP
SComCNP
ConoNi
BOomNN
SComNP
2.500
26177 10.33
26003
1614
25925
278125
Simulzzed
L,035
1.844 E0.832
10
18824
19956
19228
L.9442...co
1.966
18412
1.795620.392
13000
2
1839229
1.3476ga
1.776
Real-torgee
1.802210000
..2827..0
-33444g00
-5.1526...94
A3115537.00
-5.2615
<5.1148
6.5 Predator-prey Models
Following [88 and [[I], we consider the Volktr-Vlteera model [30], which is used to describe the evolution of predator-ppi prey populations. We first trained the models using simulated data generated from Lotka- Volterra model and tested them real-world data (Hudson's 's Bay hare-lynx data), which quite different from the simulated data and can he considered as a mismatch scenario. Table 5 lists the results both simulated and real data. Similar to the previous observation, stable version still outperforms the original version. Among stable versions, SConvNP achieves the best performance.
6.6 Ablation Study
The key parameter in our stable solution is the number of hard predictable subsets K. Taking SANP as an example, we investigated the average tog-likelihood terms different K on the ID regression task, as shown in Figure 3 We can see that SANP performs better as K increases, reaches the best value around K 4, and then becomes stable performance as K grows larger. As proved in Theorem 5.3 optimization on DI.M and more than one hard predictable subset of PI.M can achieve more stable prediction. We also conducted different experiments to explore the impact of different wg. Taking the ID regression task with RBF-GP data for experiments, as shown Table 6. It can be seen from the table below that when different weights the model using stable strategy is better than the original model, and when the weight is set to be equal, its performance is optimal. In addition, when the value of wy (gg 2 1) is significantly different from wo, such as (0.625, 0. 125, O. 125, 0.125) and (0.0625, 0.31 25, 0.3125, 0.3125), its performance is more significantly reduced compared to (0.25, 0.25, 0.25, 0. 0.25), but still has a significant improvement compared to the original non-stable model.
example, we set K 3 and different wa
1:29
0.61
1.4
0.67
1.28
0.6
1.35
0.66
1.27
0.59
0.65
0.58
0.64
1.25
0.57
1.2
0.63
1.24
0.56
1.15
0.62
(a) RBF/context
(b) RBF/Target
(c) Matern/context
(d) Matern/target
Figure 3: Log-likelihood (SANP) comparisons with different K on ID regression task.
Table 6: likeliood comparisons with different uy on ID regression task.
Weights
(025025..220.255)
(0.0022222))
(0.62..012500 2...0.0.2)))
(0.10.330.330.33
2106250.11250031155....8
SCNP
(0.9255. 0.4125)
(0,9035, 0,39%
(1.149, 0.4086)
(0.8927, 0.3991)
SNP
(0.8955.00 0.3925)
(0.8737, 0.3817)
(0.8716. 0.3809)
(0.8831, 0,3859)
(0.8657, 0.3775)
SANP
(1.2831, 0.5215)
(1,2776. 0.5187)
(1,2738, 0.5196)
(1.2791, 0.5203)
(1.2712, 0.5193)
SConvCNP
(1.3991. 0.5996)
(1.3916. 0.5933)
(1.3841, 0.5915)
(1.3934, 0.5946)
(1.3854, 0.5919)
SConyNP
(1.4036, 0.6015)
(1.3991, 0.6004)
(1,3931, 0.5992)
(1,4012, 0.6015)
(1.3957, 0.5995)
Conclusion and Future Work
In this paper, we provided theoretical guidelines for deriving stable solutions for NPs, which can obtain good generalization performance. Experiments demonstrated the proposed stable solution can help NPs achieve more accurate and stable predictions. Although the theoretical analysis we give is based on regression models, it is still open to question whether this conclusion is appropriate for classification models. Therefore, we are interested in extending our theory, expecting it to apply to more different types of tasks.
10

---

Acknowledgement
This work was partly supported by the National Natural Science Foundation of China under Grant 62176020; the National Key Research and Development Program (3020AAA0106000; the Joint Foundation of the Ministry of Education (809 IB042235); the Beijing Natural Science Foundation under Grant L21 1016; the Fundamental Research Funds for the Central Universities (2019JBZ110); and Chinese Academy Sciences (OEIP-O-202004)
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint 107.00600 2016.
[2] Olivier Bousquet and André Elisseeff. Algorithmic stability and generalization performance. Advances Neural Information Processing Systems, pages 196-202. 2001.
[3] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[4] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) pages
IEEE, 2017.
[5] Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via pac-bayes and uniform stability. Advances in Neural Information Processing Systems, 34:2173-2186, 2021.
[6] Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-leariig stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284 23:8224-82995, 2020.
[7] Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data- Eaffc-effi Machine Learning workshop, ICML, volume 4, page 25, 2016.
[8] Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Gamelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint arXiv: /907, 2019.
[9] Marta Gamelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713. PMLR, 2018.
[10] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 2018.
[11] Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.1s: 2019.
[12] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225_ 1234. PMLR, 2016.
[13] Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. arXiv preprint 11X2:22022.0759 2021.
[14] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz. Marta Garnel,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.0517 057611, 2019.
[15] Mingyu Kim, Kyeongryeol Go, and Se- Young Yun. Neural processes with stochastic atten- tion: Paying more attention to the context dataset. In International Conference on Learning Representations, 2022.

---

[16] Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint drXiv:1312.6114, 2013.
[17] Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning, pages 2796-2804. PMLR, 2018.
[18] Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 6606-6615 Curran Associates, Inc., 2020.
[19] Dongsheng Li, Chao Chen, Qin Lv, Junchi Yan, Li Shang, and Stephen Chu, Low-rank matrix approximation with stability. In International Conference on Machine Learning, pages 295-303. PMLR, 2016.
[20] Xixun Lin, Jia Wu, Chuan Zhou, Shirui Pan. Yanan Cao, and Bin Wang. Task-adaptive neural process for user cold-start recommendation. In Proceedings of the Web Conference, pages 1306-1316, 2021.
[21] Huafeng Liu, Liping Jing, Dahai Yu, Mingjie Zhou, and Michael Ng. Learning intrinsic and extrinsic intentions for cold-start recommendation with neural stochastic processes. In Proceedings of the 30th ACM International Conference on Multimedia, pages 491-500, 2022.
[22] Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J Maybank. Algorithm- dependent generalization bounds for multi-task learning. IEEE transactions on pattern analysis and machine intelligence, 39(2):227-24 2016.
[23] Ziwei Liu, Ping Luo,, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
[24] Andreas Maurer, Massimiliano Pontil, and Bernardino Porere-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81)11-22, 2016.
[25] Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pages 63-71. Springer, 2003.
[26] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages
2017.
[27] Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning, pages 10018-1008... PMLR, 2020.
[28] Qi Wang and Herke van Hoof. Learning expressive meta-repreesentations with mixture of expert neural processes. In Advances in neural information processing systems, 2022.
[29] Andre Wibisono, Lorenzo Rosasco, and Tomaso Poggio. Sufficient conditions for uniform stability of regularization algorithms. 2009.
[30] Darren J Wilkinson. Stochastic modelling for systems biology. CRC press, 2018.
[31] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poezos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. arXiv preprint arXiv: 14, 2017.
[32] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In I. Guyon, U. V. Luxburg, S. Bengio. H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
12

---

A
Inductive biases
Here, we revisit some properties, which would help us understand NPs. First, we give a concept of Permutation Invariant Function which is the basic property of stochastic process, e.g., NPs.
Definition A.1. (Permutation Invariant Function) A function f(-) x TR T R mapping set of
data points {x, ) is Permutation Invariant Function if
X2, xN] f (f
(x(xx((.N))].
(13)
where 2; E IR. and the function output is d dimensional vector. Operation 11,2, .N] [T), "* NN] is permutation set over the order of elements in the set.
Definition A.2. (Permutation Equivariant Function) A function f() X RM mapping set 1. is Permutation Invariant Function if
of data points [x,
X
[ay Xag:
[=x] + fa = л f(xi,))
(14)
where the function output contains N elements keeping the order of inputs.
Permutation Equivariant Function keeps the order of elements in the output consistent with that in the input under any permutation operation æ. Permutation invariant functions are candidate functions for learning embeddings of set other order uncorrelated data structure } and the invariant property is easy to be verified. Here, we give a mean operation structure over the output
F(Xm(1,N))
N Σ
di (x.),
Σ N
N
(2)))
1
Σ
N
(D(xx))
(15)
B Model Architecture
We show the architectural details of the CNP, NP, and ANP models used for the 1D and 2D function regression experiments. The neural process aims to learn a stochastic process (random function) mapping target features x; prediction yı given the context Dc training data realization from the stochastic process), i.e., learning
N
log P (y x .DC) log (y)x, Dy =E s=1
y: [x D²)
(16)
Conditional neural process (CNP) 9 describes P (vilxi, DC) with a deterministic neural network taking DC to output the parameters of (yilx; D9). CNP consists of an encoder (-).(.), an aggre- gator fags() (-) and a decoder (:) :); the encoder summarizes DC and x, into latent representations [ri,..., reell E I via permutation-ine invvrriant neural network [31] where d is the number of latent dimensions, and aggregator summarizes the encoded context features to a single representa- tion r. and decoder takes as input the aggregated representations r and and output the single output-specific mean μ4; and variance of for the corresponding value of yi-
r; = fenc (x,, Ve), iec c
1
Σn. r
|c|
IEC
(17)
Φ = faez( (xe)(rC)
(Mi, σ;) = fdec(e, P (Mix,, = N (U:li, iET
where fene (-) and f-ee(-) are feed-forward neural networks. The decoder output μN and variance of are predicted mean and variance. We use Gaussian distribution N (gr Hi, a as predictive distribution. CNP is trained to maximize the expected likelihood EP (y4 x, DC)).
Neural process 10 further models functional uncertainty using global latent variable. Unlike CNP, which maps context into a deterministic representation F,, NP encoders context into Gaussian latent variable giving additional stochasticity in function construction. Following [4] we consider
13

---

an NP with both deterministic path and latent path, where the deterministic path models the overall skeleton of the function F,, and the latent path models the functional uncertainty:
r; Seae
Wi), iEC
ri
ICI
iEC
(18)
Φ = Sags (r)
(U2, 0z) 1 (DS), q(z)DD) N2z
(14, oi) = faoel z, x,), p (uilx, 2, D = N
iET
with (-) and fenc having the same structure feno(.) in Eq. 17p. In this scenario, the conditional distribution is lower bounded as:
N
P ( 2, DD) P(z|DC)
log P Xy DC) 2
Σ By(e/De)
log
(19)
y(((X,
We further approximate ((z (G) = P(z and train the model by maximizing this expected lower
bound over tasks. Furthermore, ANP introduces attention mechanisms into NP to resolve the issue of under-fitting.
The architectural details of the CNP, NP, and ANP are the same as in 14] Here we give the detailed architectures of the encoder and decoder of NPs.
B.1 Encoder without attention
Encoder focuses learning embeddings for each data point in the context set, and the basic component is multi-layer perceptron, which is defined by
MLP(I, din, dn, dout) LINEAR(da, dout) (RELU LINEAR(dA., da) ....) oLINEAR(d,, din)
x(I-2)
(20)
where / is the number of layers, d dy and dout are dimensinalities of inputs, hidden unites and outputs. Here RELU(-) is adapted as activation function.
The encoder in Vanilla CNP uses deterministic encoder which focuses on learning embeddings for each data point in context set.
r; = MMP((Ll,, d2 + dy, dr, du dn) vil), r -Err ri x= MLP(le2 dn dr)
(21)
where du and dy the dimensionalities of x, and y:-
To follow the encoder structure in NP, we introduce another encoder aligned with original deterministic encoder to permit the same number parameters, i.e.,
r
MLP(le1, de + dy, da, dn ([x;, vil)
Σn
Φ1 MLP(le2, da, ( (1)
c
iEC
MI.P(le1, de + dy, dn, dr ( [x Mol)
(22)
2
Σ
(2)
Φ2 MLP(1e2, dny dn (x
ri
iEC
Φ = [2]
14

---

The encoder in NP contains deterministic path and a latent path, i.e.,
(1) MLP(( de + dy: dh, dn ([x,s vil)
(1)
Σr
Dr.. & -= --LP((lae2 du, di (rc (2))
c
IEC
(2)
MI. de + dy. dr. (1x,,
(23)
o' - MLP(l dn, dn (re (2))
c
iEC
0, 0.1 0.9 SIGMOID(O'), z (1 diag( ))
In this case, the encoder outputs deterministic representation & and latent representation z.
B.2 Encoder with attention
The attention mechanism is widely used in NPs, Specifically, multi-head attention 26 is adapted, which is defined by
Q' [LINEAR (dgy dont ( (q))EE
(QU Rhead SPLIT( Q', Aheodd)
K' (LINEAR (ds; dont ) (k))
(K)
Phoad
SPLIT(K' (hhoad)
V' (LINEAR(d,.. dont (v)
(V'] Ryhaa SPLIT(VV Nhead))
H. SOFTMAX (Q((K)) dout)
V H CONCAT ((H,
1=11
RAcad)
H' LAVERNORM(Q' H)
((aA(dot))).... K. V) LAYERNORM(H'' RELU(LINEAR(douss dowt)))
(24)
where dy. de dy are the dimensionalities of query Q, key K. and value V, respectively. Thedd is the number head. Here Layer normalization (1) LAYERNORM(() adapted. is easy derive self-attention by setting Q- K = V, i.e.
SA(dou))) = (XXA X)
(25)
For CNP, the encoder with attention still contains two deterministic paths,
Sp MLP(Igh: duy dn, da.)
Q fqn((s), iET
K (fab(xi)): iEC
V SA(dA) de + dy, dr, dz) vil) biec)
Φ1 MHA(d))(Q,K,,
(26)
H SA(da) ((RELU o MIP(he1, de + dy. du, da ([x, wil )ee)
22 MLP(le. dn, dh)
Σ h
[C)
IEC
Φ [ei, و22
Similarly, encoder with attention in NP contains deterministic path and latent path, i.c.,
før = MLP(Iyt, dz, dr,dk) dn.)
Q fae(xi), iET
(xx((x))) iEC
(27)
V SA(dW) d2 dy. dr, (1x(, Hiee)
Φ = MIA(CN)(Q)KK.)
15

---

and
Sa(dh) ([RELU o MLP(1e1, du + dy dn. d,) Bill)ee)
M2 o=]] = MLP(Hia dn, dn)
h
LEC |
(28)
0, = 0.1 + 0.9 SIGMOID(O'),
2 ~ diag
B.3 Decoder
The decoder focuses on predicting output for target points based the encoder's outputs Φ. For target point (XT the decoder of CNP is defined by
[ns o MLP(ddee: 2dh + dz, dn, 2d,) iET 0; 0.1 0.9 SOFTPLUS(O))
(29)
~ o;)
Decoder of NP is defined by
[nt, of MLP(ddee: dn + d2 + dz, dn. [d,)lo,x,,00 iET
a; 0.1 + 0.9 SOFTPLUS(O))
(30)
C Implementation Details and Experiments
For CNP [9, BCNP 18]. and our SCNP, we apply the encoder with attention described in Eq (26)
and decoder described in Eq (29). For NP [00, ANP [14,, BNP [18]. BANP [88 and our SNP and
SANP models, we apply encoder with attention described in Eq (27 and 28 and decoder described
Eq 30
C.1
1D Regression
For synthetic ID regression experiments, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP/SNP/SANT refer to Appendix B The number of hidden units is d, 128 and latent representation d- = 128. The number of layers are L, = lde = 4a = Lap L. = 2.
We generate datasets for synthetic 1D regression. Specifically, the stochastic process (SP) initializes
with 0 mean Gaussian Process (GP) ()) ~ GP(O, -)) indexed in the interval x E [-2.0, 2.0), where the radial basis function kernel k(ær, x = o² exp(
x'|| 2211)// with U0.1, 1.0) and a (0.1, 0.6). Furthermore, GP with Matern Kernel is adopted for model-data mismatch scenario, which is defined as k(x, x²) = o²2( + /5d/1 + 5d2 /(3r)) exp( exp(-v/d/l) and = | x') with 1. 1.0) and a training, and testing for all models.
7(0.1,0.6) For fair comparison, set the same data generation,
We trained all models for 100, 000 steps with each step computing updates with batch containing 100 tasks. We used the Adam optimizer with an initial learning rate 10-4 and decayed the learning rate using Cosine annealing scheme for baselines. For SCNP/SNP/SANP we set K = 3. The size of the context was drawn as |C] ~ (3, 200). Testings were done for 3, 000 batches with each batch containing 16 tasks (48, 000 tasks in total).
We investigate the model stability from the size of the context set and the percent of added noise. First, we conduct experiments on different size of context set, i.e., |C| E (20. 50, 100, 200). Table shows the Average likelihoss performance comparison between different methods in terms of different context size. We can see that the performance becomes better with the increasing of [C] and NPs with stable solution still achieve better performance. Second, we investigate the model performance in terms of different noise setting. Here we introduce Gaussian noise A (0. 1) and add noise different proportions of the data, such as (0%, 5%, 10%, 15%). Table g lists the Average Log-likelihoods performance comparison in terms of different noise proportions.
16

---

Table 7: Average Log-likelihoods over all context and target points on realizations from Synthetic Stochastic Process on different size of context set. (Mean + Std).
Kernel
Method
contect
targel
contest
largel
contict
Uargel
sunticst
largel
CNF
0.8724
0.0334
0,3533
0.532
0.156.120.co
8.5783
BCNP
0.9015-00000
0.1579
0.9787
0.5215
1.1687
0.5716
0.1985.2..00
p.6086
SCNP
0,9255 10.000
0.4733.0.00
0.9944
10.000
0.5433 se.00s
10.003
1o.00s
0.5918
10.002
0.2111 10.004
0.6333 10.004
RDF
BNP
NP
0.8215
0.8714.0.004
1o.02
0.3853
0.9124
0.9212240.004
0.4234a0.033
0.4718
1,1426.0.000
1,0855
s0.000
0.4767
0.5269
00.000
1.1716.0.000
1.1225
0.5233 eo.000
0. 0.5698,,.00
0.8955.0.0 .0.000
0..356.o.mm
0.9866.0.004
o6.0s
1.1637 10,004
0.5434
00.00
10.000
p..933
ANP
1226120000
0.3763 001
1.3231
0.632226.00 .0a
D.680 000
12992 200
D. 0.732225.00 001
BANP
1.2715_0.000
0.5878.0.000
1,3325
0.6465
1.07780..00
0.6915
1.5115
0. 0,743
SANP
200000
go.994...m
1.3452
0.6877.0.00 o0s
1,489 20
0.7043
1.5285.0.00
0,7534
CSP
0.8531
L00,00
0,2131 +0.000
0.9123
20.000
L 20000
0.351 200
1.98 10981 200
0. 0,202220,00
BCNP
20.00
0.2788.2..0m
0,9-11
29
0, 0.326620.00
1,0752
0.3762
L124520000
0. 0.4326
SCNP
0.8963-n.o0
0.2953.9.com
0.9555
0.3467
1,0967
0.3967
1,167.
0.4556
Motern
20.000
0.2517 200.
0.9322 40.025
0.3155 20.
20.005
BNP
18052
10.000
0,2651
a0.007
s0.007
0,8678
10.000
0.3163 s0.004
0.9672
s0.000
0.3656
1.1052 10.005
0,4015 10.002
SNP
0.8368
0.
0.2844
0,8956.9
0,3326
0.9959
20.000
0.3849
11215
0.431
20.004
ANP
L0.00
0.6
0.6881
10.000
T22111
20.004
0,2351
1.3452.0.007
1.4056.0..00
x..015
10.204
g.7531
1.4966.0.00
p.790
SANP
1.3721 g0.00
0.6653
1.4322 10.000
0,7126
1.4633
0.7644.6.00
1,5153 206
0.8125 1.0000000
Table 8: Average Log-likelihoods over all context and target points on realizations from Synthetic Stochastic Process on different percent of added noise. Here we set the context size to 20. (Mean + Std). Note that adding 'S' before the original model name is model with our stable solution.
Kernel
Method
Onginal
Noise(+316)
Noise(100%)
1155
context
target
context
target
context
target
context
target
CNF
0.8724
0,4334
0.85222c.com
0,4001
0.801420.00
0,3552z0004
0.71225.005
0,2853
1.9042.0.009
0.4589
_.006
0.8774.c.00
0.4278_nox
0.3767 1000s
197077
0.3017
SCNP
0,9255
0.9255.0.00 10.005
0.4733
0.044
0,8938
14.0015
0.4478 8.006
0.8817
10.001
0.3986_n000
0,7621 Lo.00
0.3279
.0006
RBF
HNP
NP
0.821510.001
8722400.04
0.4211
0.3853_0036
00004
0.8011
0.8321
20.003
10
0,3511
0.387620034
0.7922.0.04
0.761 20
0,30422000s
0.3389
0.672225.00
0.7189
0,2435
0.2776-00007
SNP
0.8955
s0.003
0.4356
9.904
0.8567
20.003
0,4046 2000
0.8165 20.001
0.3568 g0006
0.7356
0,2955
000
ANP
1.2563
10.003
0.5763
1.2245
0.5347
0.1742
0.4871 0007
0.9821
0.4151
RANP
4.272240.04
0.5887-0000
1.241
0.5471
=0000
081886220000 006
0.491720000
1.064223.005
0.4327
SANP
20.00
0.5944.0004
1.2564
001
0.5578..0x 2004
1.2052
001
0.5025-000e
0.4356
-000
CNF
0.8331
0.2131
0.221
0,2114
0.7761
0.1784 10007
0.N05110.00
0.1122 00000
BCNP
0.877840.005
0,2762_0000
0.8487
0.24772000 20000
0.8015
007
0,2051
0.7378
0.1766-0000
SCNP
003 0.8933
0.2953_0006
0,8689 Lo on5
0,2658 20077
0,8268
0.225% L0000
20.01
0.1936
-00
Matem
NF
O. 0.761350000
0.2011200.6
0.73
20
0.1725=000s
0.6892
20.004
0.131220000
0,6235
0.13
BNP
0.8166
0,2689
0.7789
0.2315=aa05
0,7421
0,21
0.615
0,1828
10.005
0.0c5 0,2844
0,8036 Lo o03
.000
0.7635
10.001
0,2325 00006
0.6973.000
0,2016
ANP
2222 $0.002
0.6366
1.2113
0.6001 2000s
L. L.7744
001
0.562220000
L125249.00
U.5274
2000
BANP
1335653.003
0.6514_000r
1.3125
0.611520038
1.267220
0.5711=0006
1.223653.05
0,5306
SANP
+.322
0044 004
1.3611..000
0.6256-0004
1.3011-0.00
0.5782=0004
+..47
0.5356-0002
C.2 System Identification on Physics Engines
The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which detailed 27). The Cart-Pole swing-up task is standard benchmark for nonlinear control due to the non- non- con-liicar in the dynamics, and the requirement for nonlincar controllers to successfully swing up and balance the pendulum. A pendulum of length is attached to cart by a frictionless pivot. The system begins with the cart at position ze 0 and the pendulum hanging down: A. The goal is to accelerate the cart by applying horizontal force un at each time-step t to invert and then stabilize the pendulum's endpoint at the goal. There are some parameters that need to be known, such as cart mass m. pendulum mass mpp acceleration of gravity g = 9.82m s², time horizon T, time discretization At and ground friction coefficient fo. In this case, the Cart-Pole swing-up task aims to forecast the transited state [Ie, 0, x'y 0') in time step I + 1 based on the input as state action pair [Ere, 0, x', 0', a) in time step I.
For system identification task on physics engines, the neural architectures for CNP, NP, ANP, BANP and our RNP refer to Appendix B The number of hidden unites is dn = 32 and latent representation d = 32. The number of layers are 4e = lde = 4a = lyk = 4, = 2.
To generate variety of trajectories under random policy for this experiment, the mass mc and the ground friction coefficient fe are varied in the discrete choices m. E (0.3, 0.5, 0.6, 0.7} and fo {0.06, 0.08, 0.1, 0.12). Each pair of [me. fcl values specifies a dynamics environment, and we formulate all pairs of m. E (0.3, 0.5, 0.7) and Se E (0.08, 0.12) as training environments with the rest 16 pairs of configurations as the testing environments. For each configuration of the simulator including training and testing environments, we sample 400 trajectories of horizon as 10 steps using random controller, and more details refer to Supplementary material. During the testing process,
17

---

4 Goal
f
m2
Figure 4: Cart-Pole Dynamical Systems. The cart and the pole are with masses me and mp, and the length of the pole is I. And the configuration of the simulator is up to parameters of the cart-pole mass and the ground friction coefficient here with other peper-parameters fixed in this experiment.
Table 9: Average loglliiclshoods over all context and target points EMNIST and CELEBA.
Original
Noise( +33)
Noiser +10%;
1195)(+15%)
Datisee
Method
context
context
CNP
0352220003
g.8977
0'8336ga.117
0.8222_..om 0015
1.5754.0.000
075666550001
BCNP
0.9678
0.8058
c..0s a.8058.
0.9015
0.671 sou.co
0.8415 s0.007
0.6089 10.000
0.6788 0.6788
0.5715
-000
EMNNST
SCNP
0,8343
0.9251
0.6971.0.007
0.867420..00
0.6343:0007
0.6966.0.00
0.5877
KF
0,7756gm.
0.9011
0XS1
2.705420.000
0.3865
BNP
0.9757
0.81162a.cor
06255
0,8759 com
0,6773
0.745 20.00
0.6237.000
SNP
0.9847
0.838.9..om
a.784
40 406
0.6984.0.066
20-06
26456_0000
ANP
g.025
131321 game
0
0.381550.00 go.000
03X021
20
DC7085 20.066
0.444 $0.00
0,70344
BANP
1355
1,0615
1.0236x0.000
0 16.49
0.9155 10.004
0.7521
18.02
0,7515
SANP
1,087
0.677619.00
0
0.7843.0.00
0.732200.00
0.7657
0.7438
03895600.c0
20.011
00707700-018
D.
BCNP
LO452
10.00
0.8015
0.8015 c0.00s
1.0275
1.0275 s0.000
1.7726
0.9351 10.006
0.8376 10.000
0
0.8015
0.6816
CELEBA
SCNP
L,525
0,8243
1,0348
0, 0,7868 souace
0.9562.0..04
0.8545 10,006
0,8344 20.44.0.000
0,7045
KF
L
1,10032n.003
133552
10005
0.395520.000
0,689
BNP
L1732g4005
0.8901 on
1.1378aa.com
8678
0.8711 290
208
3,7671200m
SNP
11952
0.9
g.
0 1.8
1,0542
406
0.8956.0.04
0.9425 20-06
0,7985
ANP
2263330000
1.137799.04
0 036661..000
1011
0.8845
09263
0,7316
BANP
1.27512000
1,03922000
1,1488ga.00
11155
0400
2.
0.988200000
0,8415
SANP
1,0594
1,1685
0.9655
0.8673
100 state transition pairs are randomly selected for each configuration of the environment, working as the maximum context points to identify the configuration of dynamics.
C.3 Image Completion
Analogous to the ID experiments, we take random pixels of a given image at training as targets, and select subset of this contexts, again choosing the number of contexts and targets randomly (n ~ U U3, 200). m ~ n + [0, 200 n)). The x, are rescaled to [-1, 1] and the y are rescaled to [-0.5, 0.5]. We use a batch size of 16 for both EMNIST and CelebA, i.c. use 16 randomly selected images for each batch.
For image completion experiments on EMNIST and CelebA dataset, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP, SNP, and SANP refer to Appendix B The number of hidden unites is dn 128 and latent representation dę 128. The number of layers are
= Ldee = 4, lta Lob = Ly 5. hreod =
C.4 Uncertainty Measuring
Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. We further analyze the learned models using the framework introduced in (17) to quantify uncertainty by investigate the calibration error and sharpness of the models. By assuming predictive distribution Po x,, DC) Gaussian distribution A/ (vi μ4, of), can get the proba-
bilistic forecast (z,)). . More formally, m confidence levels S Pi < P2 < ... Pm I 1 are choose.
For each threshold Pj. compute the empirical frequency
lys Fyye) S Pj. E
6,
(31)
[T]
18

---

Table 10: Calibration error and sharpness of the models for ID regression experiments (Mean = Std).
RBF
Method
CAL
SHA
CAL
SHA
CNP
0.0724
0.0887 0.0887200.07
0.05142o.00s
0.0831 +0.010
BCNP
0.0015
00.05
0.0724+0.005 0ss
0.1152=00.00
SCNP
0.1225
0.1357.0.00s
0,0425
0.1325
NP
0.0615
0.0715
0.0343 015
0.0717
BNP
0.0871 +0.00G
0.1052=0.005
0.0325
0.0015
SNP
0.1155 0.1257 200005
0.0347 oos
0.0718go.oor
ANP
0.153220.002
0.0616go.no
0.0921 004
0.0871 =:0 00000
BANP
0,2353
0.0689
0.07752.0.00s
0.0741 10.000
SANP
0.2633
0.0741=0.006
0.0415
0.0667++.000
Table 11: Calibration error and sharpness of the models for system identification experiments (Mean + Std).
Method
CAL
SHA
CNP
0,0872:10.00
0.0415 0.000
BCNP
0.1051
0.0765
SCNP
0.1124:0.008
0.083310.000
NP
0.0821 1 003
1.0081
BNP
0.0952 003
0.061640.00
SNP
0.1001
001
0.0668.40.0m
ANP
0.0863 #0.001
0.0673+0.004
BANP
0. 0.1225aa..m a 0010
0.0815+0.005
SANP
0. 14.433 +0.000
0. 0.10944+..00
In this case, the calibration error is defined as a numerical score describing the quality of forecast calibration:
CAL (Fi, PITI. = Σ Wj (pj - -2₂)
(32)
here wj is weight and we wy
Sharpness is measured by using the variance var var(F.) of of the random variable whose CDF is F,.
Low-variance predictions are tightly centered around one value. A sharpness score can be defined by
SHA(FI, FIT)) = T
1
Σ of ET
(33)
We evaluated the CE and sharpness of CNP, NP, ANP, BCNP, NBP, BANP and corresponding stable versions SCNP, SNP, and SANP trained in the experiments. Table TO етrоr and sharpness score on ID regression, system identification, and mmae completion tasks. In several settings, models with our stable solution can achieve better calibration and sharpness but work worse in some settings, such as the calibration eтror being worse than NP and ANP in terms of ID regression tasks with Matern. The possible reason is that our method tends to produce conservative credible intervals, so become under-confident or less over-confident in some settings.
12 list the calibration
Table 12: Calibration error and sharpness of the models for image completion experiments (Mean + Std).
EMNIST
CELEBA
Method
CAL
SHA
CAL
SHA
CNP
0.0182+0.002
0,0574_0.002
0.025340.002
0.0743=0.001
BCNP
0.0415
0.0716_0.002
0.0412+t.002
0.0981 0.0981 10.8120.002
SCNP
0.0543 0.0846_0.002 002
0.0457+...000
0.113620.002
NP
0.016320.001
0.067120.007
0.0261++0.00
0.0711
BNP
0.0352+0.002
0,0815
0.0463
0.0981
SNP
0.044622.0002
0.0918_0.o0m
0.0532+1o.00
0. 0.104620000)
ANP
0.0156
0.0656
0.0261 +00.004
0,0815 +0.000
BANP
0.0412 002
0.0871 2011=00.002
10051340..00
0. 0.1125 003
SANP
0.0558
0,0954
0.0632+4..002
0.1265
19

---

0.51
1.26
0.58
1.25
0.64
1.25
0.57
1.2
0.63
1.15
(a) RBF/context
(b) RBF/Target
(c)
Matern/context
(d) Matern/target
Figure 5: The log-likelihood comparisons with different K on ID regression task.
(a) LL
(b) MAE
Figure 6: The LL and MAE comparisons with different K on system identification task.
110
1.0
(a) EMNIST
(b) CELERA
Figure 7: The LL comparisons with different K on image completion task.
C.5 Ablation Study
The key parameter our stable solution the number of hard predictive subsets K. Taking SANP as an example, we investigated the average log-likelihood in terms of different K on ID regression task, shown in Figure We can see that SANP performs better К increases, reaches the best value at around K 4, and then becomes stable in performance as K grows larger. As proved in Theorem 5.4 in the main manuscript, optimization on DC and more than one hard predicted subsets of DC can achieve more stable prediction. Similarly, we also conducted experiments to investigate the effect of K on System identification and image completion task and similar observations can be seen in Figure and
20


Reviews:
[{"review_id": "88Lf0CuGOf", "reviewer": "Reviewer_nxGu", "summary": "This work studies the algorithmic stability and generalization error of neural processes (NPs). Motivated by the stability analysis of NPs, a new method is proposed to train NPs that results in improved stability and better generalization performance with theoretical guarantees. The proposed method is based on the idea of removing easy datasets from the meta-dataset, which is essentially a weighted-training algorithm that down weights easy samples. The authors also empirically verify the usefulness of the proposed method on both synthetic and real-world datasets.", "questions": "See the weaknesses section. I will increase my score accordingly if some or all of the weakness points are addressed.", "limitations": "Limitations are not discussed in the paper.\n\nThis work does not have obvious potential negative societal impact.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of studying the algorithmic stability and generalization error of NPs is interesting and well-motivated.\n2. The proposed method for improving the robustness of NPs has theoretical guarantees and appears to be useful in practice.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. In the beginning of the proof for every theorem, the random variables $R_{\\tau}(\\theta)-(...)$ are assume to have zero mean with no justifications. It is unclear why they have zero means. The authors should justify these assumptions since all proofs are based on them.\n2. The proposed method is essentially a weighted-training algorithm that down weights easy datasets. According to the definition in the paper, a dataset is easy if the NLL of *every data point* in it is smaller than $R_{\\mathcal{D}_{1:M}}(\\theta)$ under a regularly trained NP. I can imagine that in practice each dataset will have some easy data points and some hard data points, but it is less likely that all data points in a data set are easy. The authors should discuss the applicability of the proposed method in this regard.\n3. The proposed method is computationally more expensive, since it requires 1) training a regular NP; 2) filtering out easy datasets according to some criteria; 3) reweighting each dataset; and 4) training the final NP. The extra computational costs will especially be enlarged when the NP is used for downstream tasks such as Bayesian optimization where one needs to constantly update the model as new data points/datasets arrive. The authors should discuss and compare the computational costs between the proposed method and regular NPs.\n4. It would be more convincing if the authors can show that the NPs trained by the proposed method can lead to improved performance in downstream tasks such as Bayesian optimization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n\n\n**Q1 The definition of the random variable with zero mean in theorems.**\n\n> The random variables we assume are the difference between the expected risk and the empirical risk, and considering the accuracy of the model, we expect the empirical risk to be as close as possible to the empirical risk, hence the zero-mean assumption.\n>\n> We will explain the setting of this assumption in more detail in subsequent versions.\n\n\n\n**Q2 The definition of easy datasets.**\n\n> We are not defining a dataset as a simple dataset, as shown in section 5.1, we are selecting samples that are easy to predict from M meta-datasets to form an easily predicted subset, thus we are not defining a certain dataset as an easy dataset.\n\n\n\n**Q3 Discuss and compare the computational costs between the proposed method and regular NPs.**\n\n> Thank you for your good suggestions. As shown in Algorithm 1, we need to pre-train the base model to select samples. In fact, the pre-trained model can not only be used for sample selection, but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster. Taking the 1D regression task as an example, the traditional NP model needs to train 200 epochs using 256 batches per epoch of batch size 16 to get the final result. When using our stable strategy, in order to train and obtain the pre-trained model, only 100 epochs are needed to select the easily predicted subset, and relying on the pre-trained model parameters as the initialization of the stable version, only 100 epochs are also needed to converge to a satisfactory result. Therefore, it can be said that the computational complexity of NPs with our stable solution is similar to the original NPs.\n>\n> We have made some explanations in Section 5.1 of the original version, and we will make a more detailed description to explain the computational loss of the stable model compared to the traditional model.\n>\n> \n\n**Q4 Performance comparison in downstream tasks such as Bayesian optimization.**\n\n> Following the setting in BANP (Bootstraping attentive neural process), we conducted the Bayesian optimization experiment. Taking GP data with RBF and Matern priors as examples, we gave the results of ANP, BANP, and our SANP. Please see Figure 1 in the uploaded pdf file for details. It can be seen that SANP has better overall performance.\n\nFinally, thanks again for your valuable comments."}, {"review_id": "4UKyLjTcN0", "reviewer": "Reviewer_jySn", "summary": "In this paper, the stability of neural processes was studied. The scenario is when the context data points are noisy and this paper introduces the concept of stability, proposes adaptive strategies and draw some conclusions from empirical observations.", "questions": "No other questions.", "limitations": "See the weakness section.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "This paper is written in a neat way and the consideration of noisy context points can be necessary in some risk sensitive applications. The literature work is well overviewed.", "weaknesses": "Even though this work is easy to follow, there exists unfixable issues concerning the research motivation, definitions of stability and derived theorems.\n\nTo begin with, let us rethink types of uncertainty in neural processes. *As we know, there primarily exist two types of uncertainty, namely aleatoric uncertainty and epistemic uncertainty.* The considered noisy context points belong to the aleatoric type, which is irreducible. The number of context points or observation extent corresponds to the epistemic, which means the prediction is more accurate with increasing the number of context points.\n\n**1. Concept of algorithmic stability in Definition4.1.** In Line 141, the concept of stability is presented, and it suggests that the approximation results remain stable if the change of the datasets given a stable NP (see Line178-179). These definition and conclusion violate the property of predictive functions in stochastic processes, where increase of context points decreases the epistemic uncertainty.\nParticularly, **in Line133, the stable property defined in NPs is “A stable learning algorithm has the property  that replacing one element in the training set does not result in a significant change to the algorithm’s output”.** **I strongly disagree with this trait since NPs are Bayesian  non-parametric, which means the change of context points definitely changes the predictive distribution like that in Gaussian processes. This property enables the Bayesian optimization using NPs.** If the proposed stability is achieved, the crucial Bayesian non-parametric property no longer holds.\n\n\n**2. Connection between subset selection and noisy context data points.** It is unclear in a theoretical sense that how the subset selection alleviate the negative effect of noisy context points. This work considers the scenario of noise context points but focuses on the algorithmic stability solved by the bootstrapping trick. Hence, the logics are not well organized.\n\n**3. Overstated or imprecise claims.** In line 29-30, to motivate this work, it says “As demonstrated in previous work [7, 8, 12, 9], existing NPs cannot provide stable predictions under noisy conditions. However, as far as I know, not all of these work mentioned the importance of noisy conditions and did investigations in an empirical and theoretical way. In Line 101, it states that the ELBO is maximized in NPs, however, the real optimization objective is the approximate ELBO rather than real ELBO since the Jessen inequality no longer holds.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n\n\n**Q1 Concept of algorithmic stability in Definition 4.1.**\n\n> Thank you for your suggestions. In fact, the definition of algorithm stability we gave in Definition 4.1 is based on the difference between the empirical risk and the expected risk of NP, rather than the accuracy of a certain type of prediction. Therefore, we hope that when the model is disturbed, especially by noise disturbance, it can still be close to the expected risk. And what we said when an element in the dataset changes, especially when it is disturbed or irrelevantly changed, the approximate result of NP is still stable, which does not contradict what you said about uncertainty. The example of Gaussian processes you gave “the change of context points definitely changes the predictive distribution like that in Gaussian processes” refers to the change of prediction results, while we define the change of NP model from the perspective of stability (the difference between expected risk and empirical risk), that is, when changing a sample or an element in the dataset, the model result will not have a large disturbance.\n\n\n\n**Q2 Connection between subset selection and noisy context data points.** \n\n> This paper discusses algorithmic stability, and algorithmic stability is strongly related to the noise data. For this reason, we gave the algorithm stability definition applicable to NP in the problem description in Section 4. Based on this, we explored the relationship between stability and generalization error by relying on the experiment of replacing noise data and thus introduced the use of stability definition to prove the impact of subset selection on NP stability. The experimental results also proved that the subset selection strategy has significant improvements compared to the baselines under different noise conditions. \n\n\n\n**Q3 Overstated or imprecise claims.** \n\n> What we are expounding here is that the results of existing work can show that classical models are easily affected by noise, especially under fewer context samples, the results of different models are very different, and they are far from the correct results. For this reason, we listed a few classic references to illustrate, such as CNP, NP, ANP, and ConvCNP. Here, we will make a more detailed description and reference adjustment. In addition, you mentioned the ELBO of NP in line 101. Usually, we can express the objective function in the form of ELBO (such as Eq. 3 in the original ANP paper), which is also a common expression for the objective function of VAE type.\n\n\n\nFinally, thanks again for your valuable comments."}, {"review_id": "UeXYhci7aN", "reviewer": "Reviewer_i5nU", "summary": "This paper focuses on the scenario where noisy data is included in the context points during the training of Neural Processes. In order to address algorithmic instability, the authors provide theoretical guidelines to obtain stable solutions that exhibit high generalization capabilities. These guidelines use hard predictable subsets as additional different training datasets.", "questions": "Experiment\n- In Algorithm 1, it appears that the NP model requires early training to classify hard and easy datasets. I'm curious to know the number of early training epochs needed for the NP model. Additionally, it would be helpful to understand if the training epochs for other baselines are the same as those for the proposed method. This would help ascertain whether the performance gain stems from additional training or the effectiveness of the proposed method itself.\n- Could you provide separate log likelihood results for the context dataset, target dataset, and the entire dataset (referred to as \"target\" in this paper) by splitting them? In reference [1] and [2], log likelihoods of the context and target are reported separately, and increasing the context log likelihood can sometimes lead to an increase in the whole log likelihood without any actual performance improvement (or even a decrease in performance) for the true target dataset.\n- It would be valuable to conduct experiments using other baselines and additional tasks as mentioned earlier. The current set of experiments, including 1D regression with RBF kernel, 2D regression, and the physics engine experiment, may not be sufficient to demonstrate the superiority of the proposed model across various tasks.\n- How are the values of 'w' in the loss function balanced? Could you perform an ablation study on these hyperparameters to determine the robustness of the training procedure with respect to these hyperparameters?\n\nReferences\n\n[1] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, and V. Oriol. Attentive neural processes. In International Conference on Learning Representations (ICLR), 2018.\n\n[2] J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.", "limitations": "Limitation\n- The proposed model incurs K times additional computational cost during training compared to other models, as it requires calculating a new predictive distribution with K-1 augmented dataset.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Originality\n\n- This paper introduces numerous theoretical findings regarding stable solutions.\n- The idea is overall interesting and novel.\n\nClarity\n- The paper is effectively written and presents its content in a clear and comprehensible manner.", "weaknesses": "Experiment\n- The choice of baseline models appears to be somewhat outdated. Considering the existence of newer NP models after Attentive Neural Processes [1] and Bootstrapping Neural Processes [2], it would be beneficial to include more recent approaches such as Convolutional Neural Processes [3,4] as baselines.\n- The tasks employed in the experiments seem to be too simplistic. The results obtained from simple 1D regression and 2D regression datasets may not be sufficient to demonstrate the superiority of the proposed model. I recommend conducting tasks such as Bayesian Optimization and the Predator-Prey task from [2] to showcase the general applicability of the method. Additionally, it would be valuable to include results using a periodic kernel to address model-data mismatch scenarios, as the Matern kernel's similarity to the RBF kernel may not adequately demonstrate the robustness of the proposed model.\n\n\nReferences\n\n[1] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, and V. Oriol. Attentive neural processes. In International Conference on Learning Representations (ICLR), 2018.\n\n[2] J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.\n\n[3] J. Gordon, W. P. Bruinsma, A. Y. K. Foong, J. Requeima, Y. Dubois, and R. E. Turner. Convolutional conditional neural processes. In International Conference on Learning Representations (ICLR), 2020.\n\n[4] A. Y. K. Foong, W. P. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. E. Turner. Metalearning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n**Q1 Additional baselines including ConvCNP and ConvNP**\n\n> Thank you for your suggestions. We conducted experiments on ConvCNP and ConvNP, and built our stable versions, SConvCNP and SConvNP, based on them. The table below lists the results on RBF for 1D regression (due to word limit, we only list the results on original data and Noise(+15%)). By comparing the results in the table below with those in the main text, we can see that ConvCNP and ConvNP results are better than other traditional NPs (including CNP, NP, and ANP). In addition, the stable versions SConvCNP and SConvNP results are better than the original ConvCNP and ConvNP, which also confirms the effectiveness of our weighting strategy on ConvCNP and ConvNP.\n>\n> We will add the results of ConvCNP, ConvNP, SConvCNP, and SConvNP on various tasks in the new version.\n>\n>\n> | RBF Kernel | Original-Context | Original-Target | Noise(+15%)-Context | Noise(+15%)-Target |\n> | :--------: | :--------------: | :-------------: | :-----------------: | :----------------: |\n> |  ConvCNP   |      1.2631      |     0.6421      |       1.0241        |       0.4262       |\n> |   ConvNP   |      1.2874      |     0.6503      |       1.0523        |       0.4311       |\n> |  SConvCNP  |      1.3991      |     0.6793      |       1.1641        |       0.4396       |\n> |  SConvNP   |      1.4036      |     0.6831      |       1.1825        |       0.4425       |\n\n**Q2 Additional tasks including  Bayesian Optimization and the Predator-Prey task** \n\n> Following the setting in BANP (Bootstraping attentive neural process), we conducted the Bayesian optimization experiment. Taking GP data with RBF and Matern priors as examples, we gave the results of ANP, BANP, and our SANP. See Figure 1 in the uploaded pdf file for details. It can be seen that SANP has better overall performance.\n>\n> Following the setting in ConvCNP and BANP, we conducted experiments on the Predator-prey task. We first trained the models using simulated data generated from a Lotka-Volterra model and tested on real-world data (Hudson’s Bay hare-lynx data).  Due to the response limitation, we list the results in Table 1 of the uploaded pdf file.  Similar to the previous observation, our stable version still outperforms the original version. Among stable versions, SConvNP achieves the best performance. \n\n**Q3 Additional kernel (periodic) in the 1D regression task**\n\n> Thank you for your suggestions. We conducted experiments on 1D regression tasks with Periodic kernel. Following the setting of BANP and similar noise settings of our previous kernels, we list the results on both original data and noise(+15) data in Table 2 of the uploaded pdf file.  In this model-data mismatch data, stable versions still significantly outperform their corresponding original versions.\n>\n> We will add those results in the new version.\n\n**Q4 The number of training epochs**  \n\n> Thank you for your good suggestions. As shown in Algorithm 1, we need to pre-train the base model to select samples. In fact, the pre-trained model can not only be used for sample selection, but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster. Taking the 1D regression task as an example, the traditional NP model needs to train 200 epochs using 256 batches per epoch of batch size 16 to get the final result. When using our stable strategy, in order to train and obtain the pre-trained model, only 100 epochs are needed to select the easily predicted subset, and relying on the pre-trained model parameters as the initialization of the stable version, only 100 epochs are also needed to converge to a satisfactory result. \n>\n> We have made some explanations in Section 5.1 of the original version, and we will make a more detailed description to explain the computational loss of the stable model compared to the traditional model.\n\n**Q5 Separate log likelihood results for the context dataset, target dataset, and the entire dataset** \n\n> Thank you for your suggestion. This point you mentioned is very important, so here we assume that the target set $\\mathcal{T} = \\{1,2,\\cdots, N\\} - \\mathcal{C}$, and calculate the log-likelihood on the context dataset, target dataset, and the entire dataset. Table 3 of the uploaded pdf file lists the results for 1D regression with RBF kernel.  It can be seen that the log-likelihood of the true target is significantly improved under the action of the stable strategy. In fact, in other experimental settings, the stable version has a significant improvement over the base version on the target dataset. This also proves that the proposed stable solution can improve the predictive performance on the target dataset. We will distinguish the log-likelihood under different target settings in the new version.\n\n **Q6 The values of weights $w$**.\n\n> In the experimental setting, we only set $w_k = \\frac{1}{K+1} \\forall k=0,1,2,\\cdots,K$ to satisfy the requirement of $w_0+w_1+\\cdots+w_K=1$ in the theorem. In fact, we performed an ablation study on $K$ in the main text and the appendix to investigate how our stable solution performs by varying the number of  subsets $K$.  It should be noted that, when $K = 0$, the stable model is degraded to the original base NP model. Thus, the fact that the stable solution can achieve better performance than the original model with additional terms $\\sum_{k=1}^K w_k R_{D_{1:M}^k}(\\theta)$ . In addition, the log-likelihood in all cases increases as $K$ increases. This further confirms Theorem 5.4: probing easily predictable entries to form harder predictable entry sets can better increase the model performance.\n\nFinally, thanks again for your valuable comments."}, {"review_id": "kOcrhnx1YV", "reviewer": "Reviewer_Gzsw", "summary": "This paper proposes a new way of training neural processes that increases their algorithmic stability, a property known to be helpful for generalisation in a frequentist sense. They work out theory to motivate their procedure, which is then compared against other existing ways of training neural processes. They show their method improves performance on a variety of tasks.", "questions": "- The method shows consistent improvement in quantitative metrics, but when looking at Fig 2 it is difficult to see why this is the case. It's not clear to me why the SNP predictions are such a significant improvement over the NP. Could the authors give a qualitative explanation why? If not, I'm concerned that the improvement in numbers could be an artefact of something not genuinely significant.", "limitations": "The authors give a very small amount of the paper to address limitations. I would expect a more extensive discussion here. In particular, how does the runtime of their algorithm compare with regular NPs? Is there some kind of trade-off here?", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and well-motivated\n- The method seems to show consistent improvements in a wide variety of tasks\n", "weaknesses": "- I think the authors could spend more time arguing for the importance of stability. I agree that it would be helpful, but it's not clear to me that it is the single crucial factor in making neural processes generalise.\n- The authors try of the NP and ANP, but could consider more SOTA methods such as the convolutional neural process.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable feedback. We made efforts to address your every concern and question. If we have any misunderstanding or further questions, please feel free to let us know and we will reply quickly.\n\n**Q1 The importance of stability**\n\n> Thank you for your suggestion. This suggestion is very important for us to improve the motivation of the paper. Based on the empirical risk and expected risk of neural processes defined in Section 3 (note that due to the unique meta-learning mechanism of NP, this type of empirical risk and expected risk is different from that of general machine learning models), we define the algorithmic stability of neural processes in detail in Section 4 and explain what kind of model is more stable. Based on this, we give the experiments of NP on stability and generalization error and use examples to prove the relationship between model generalization error and stability. We did not say that stability is the only key factor for NP to generalize, we just described their relationship, and thus gave the theoretical analysis based on stability.\n>\n> For your question, we will also add the following expressions about the importance of stability in the problem introduction part. The sparsity of the data, incomplete and noisy introduces challenges to the algorithm stability. NP models are biased to the quality of context data and target data, so small changes in the training data (noisy) may significantly change the models. In this case, unstable solutions will introduce high training error variance, and minimizing the training error may not guarantee consistent error reduction on the testing dataset, i.e., low generalization performance. In other words, the algorithm stability has a direct impact on generalization performance, and an unstable NP solution has low generalization performance.\n\n**Q2 Could consider more SOTA methods such as the convolutional neural process**\n\n> Thank you for your suggestions. We conducted experiments on ConvCNP and ConvNP, and built our stable versions, SConvCNP and SConvNP, based on them. The table below lists the results on RBF kernel on 1D regression (due to word limit, we only list the results on original data and Noise(+15%)). By comparing the results in the table below with those in the main text, we can see that ConvCNP and ConvNP results are better than other traditional NPs (including CNP, NP, and ANP). In addition, the stable versions SConvCNP and SConvNP results are better than the original ConvCNP and ConvNP, which also confirms the effectiveness of our weighting strategy on ConvCNP and ConvNP.\n>\n> We will add the results of ConvCNP, ConvNP, SConvCNP, and SConvNP on various tasks in the new version.\n>\n> | RBF Kernel | Original-Context | Original-Target | Noise(+15%)-Context | Noise(+15%)-Target |\n> | :--------: | :--------------: | :-------------: | :-----------------: | :----------------: |\n> |  ConvCNP   |      1.2631      |     0.6421      |       1.0241        |       0.4262       |\n> |   ConvNP   |      1.2874      |     0.6503      |       1.0523        |       0.4311       |\n> |  SConvCNP  |      1.3991      |     0.6793      |       1.1641        |       0.4396       |\n> |  SConvNP   |      1.4036      |     0.6831      |       1.1825        |       0.4425       |\n\n\n**Q3 About the question of Fig 2**\n\n> We apologize for the confusion caused by the content shown in Fig 2. In fact, Fig 2 (a) and (c) only show the visualization of the mean and variance of 50 samples randomly selected from one task in NP and SNP under different kernels (RBF and Matern). It can be seen that in this example, both NP and SNP fit the context data well, while for the target data, some parts of SNP have better fitting characteristics than NP. One thing worth noting is that SNP and NP have similar variances when fitting the target without context information. SNP may produce slightly larger variances to indicate the uncertainty of this part of information. For some deterministic regions, SNP will produce smaller variances to indicate the certainty of this part of information. Although it may be difficult to see the significant advantages of SNP from a single visualization result, in fact, the overall fitting result of SNP is better than NP.\n\n**Q4 Limitations**\n\n> We pointed out the limitations of the proposed theoretical proof in the submission version. That is, it only applies to regression tasks, and its validity for classification tasks remains to be verified. In fact, as you mentioned, computational complexity is also a point worth paying attention to. As shown in Algorithm 1, we need to pre-train the base model to select samples. In fact, the pre-trained model can not only be used for sample selection, but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster. Taking the 1D regression task as an example, the traditional NP model needs to train 200 epochs using 256 batches per epoch of batch size 16 to get the final result. When using our stable strategy, in order to train and obtain the pre-trained model, only 100 epochs are needed to select the easily predicted subset, and relying on the pre-trained model parameters as the initialization of the stable version, only 100 epochs are also needed to converge to a satisfactory result. Therefore, it can be said that the computational complexity of NPs with our stable solution is similar to the original NPs.\n>\n> Although the number of training epochs of our model is similar to that of existing methods, it incurs $K$ times additional computational cost during training compared to other models. Thus, in addition to the applicability problem of the theoretical proof that has been pointed out, we will also have a more detailed discussion on the limitation of computational complexity.\n\n\nFinally, thanks again for your valuable comments."}]

Author rebuttal:
Dear Reviewers, AC and SAC,

We would like to thank the reviewers for their valuable feedback and insightful comments and AC and SAC for their efforts in the review work. In this rebuttal, we address the main concerns and answered the questions raised by each reviewer individually. 

Meta-review: This paper introduces an approach to enhance the predictive performance and robustness of neural process families by incorporating algorithmic stability considerations. The method is designed to be seamlessly integrated with various types of neural processes, demonstrating significant improvements in accuracy and robustness across a variety of out-of-distribution scenarios.

While the average review score falls below the threshold for acceptance, I would like to highlight that the current score might not accurately represent the overall sentiment of the reviews. Notably, reviewer nXGu (borderline reject) initially raised concerns regarding formulation errors and inference time slowdown, which the authors appear to have effectively addressed during the rebuttal phase. However, these improvements do not seem to have been fully reflected in the final review score. Reviewer jYSn (reject) expressed reservations about the stability concept, leading to their recommendation for rejection. However, I align more closely with the viewpoint of Reviewer Gzsw (advocating acceptance) on this particular issue.

In summary, I find the proposed method to be versatile, applicable across a wide spectrum of neural process models. The negative assessments from the current reviews do not appear to definitively warrant rejection of the paper. As such, my recommendation is to accept it.
Decision: Accept (poster)