Title: Neural Processes with Stability
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural processes

stability
Abstract: Unlike traditional statistical models depending on hand-specified priors, neural processes (NPs) have recently emerged as a class of powerful neural statistical models that combine the strengths of neural networks and stochastic processes. NPs can define a flexible class of stochastic processes well suited for highly non-trivial functions by encoding contextual knowledge into the function space. However, noisy context points introduce challenges to the algorithmic stability that small changes in training data may significantly change the models and yield lower generalization performance. In this paper, we provide theoretical guidelines for deriving stable solutions with high generalization by introducing the notion of algorithmic stability into NPs, which can be flexible to work with various NPs and achieves less biased approximation with theoretical guarantees. To illustrate the superiority of the proposed model, we perform experiments on both synthetic and real-world data, and the results demonstrate that our approach not only helps to achieve more accurate performance but also improves model robustness.

Neural Processes with Stability

Abstract

Unlike traditional statistical models depending on hand-specified priors, neural processes (NPs) have recently emerged as a class of powerful neural statistical models that combine the strengths of neural networks and stochastic processes. NPs can define a flexible class of stochastic processes well suited for highly non-trivial functions by encoding contextual knowledge into the function space. However, noisy context points introduce challenges to the algorithmic stability that small changes in training data may significantly change the models and yield lower generalization performance. In this paper, we provide theoretical guidelines for deriving stable solutions with high generalization by introducing the notion of algorithmic stability into NPs, which can be flexible to work with various NPs and achieves less biased approximation with theoretical guarantees. To illustrate the superiority of the proposed model, we perform experiments on both synthetic and real-world data, and the results demonstrate that our approach not only helps to achieve more accurate performance but also improves model robustness.

1 Introduction

Neural processes (NPs) [9, 10] constitute a family of variational approximation models for stochastic processes with promising properties in computational efficiency and uncertainty quantification. Different from traditional statistical modeling for which a user typically hand-specifies a prior (e.g., smoothness of functions quantified by a Gaussian distribution in Gaussian process [25]), NPs implicitly define a broad class of stochastic processes with neural networks in a data-driven manner. When appropriately trained, NPs can define a flexible class of stochastic processes well suited for highly non-trivial functions that are not easily represented by existing stochastic processes.

NPs meta-learn a distribution over predictors and provide a way to select an inductive bias from data to adapt quickly to a new task. Incorporating the data prior into the model as an inductive bias, NPs can reduce the model complexity and improve model generalization. Usually, an NP predictor is described as predicting a set of data (target set) given a set of labeled data (context set). However, the number of noise in data introduces challenges to the algorithmic stability. In NPs, models are biased to the meta-datasets (a dataset of datasets), so small changes in the dataset (noisy or missing) may significantly change the models. As demonstrated in previous work [9, 10, 14, 11], existing NPs cannot provide stable predictions under noisy conditions, which may introduce high training error variance, and minimizing the training error may not guarantee consistent error reduction on the test set, i.e. low generalization performance [2]. In this case, algorithmic stability and generalization performance have strong connections and an unstable NP model has low generalization performance.

A stable model is one for which the learned solution does not change much with small changes in training set [2]. In general, heuristic techniques, such as cross-validation and ensemble learning, can be adopted to improve the generalization performance. Cross-validation needs to sacrifice the limited training data, while ensemble learning is computationally expensive on training submodels. Recently, there are several improved NPs focused on considering model stability and

improving generalization performance empirically, such as hierarchical prior [27], stochastic attention mechanism [15], bootstrap [13], and Mixture of Expert [28]. However, most of them are unable to investigate the theoretical bound of the generalization performance of NPs. It is desirable to develop robust algorithms with low generalization error and high efficiency.

In this paper, we investigate NP-related models and explore more expressive stability toward general stochastic processes by proposing a stable solution. Specifically, by introducing the notion of stability into NPs, we focus on developing theoretical guidelines for deriving a stable NPs solution. We propose a method to find out subsets that are harder to predict than average, which is a key step for constructing this optimization problem. Based on it, a new extension of NPs with stable guarantees is formulated, which can be flexible to work with various NPs and achieves less biased approximation with theoretical guarantees. Considering the model adaptivity, an adaptive weighting strategy is proposed. To illustrate the superiority of the proposed stable solution, we perform experiments on synthetic 1D regression, system identification of physics engines, and real-world image completion tasks, and the results demonstrate that NPs with our stable solution are much more robust than original NPs.

2 Related Work

In this section, we briefly review two different areas which are highly relevant to the proposed method, neural processes, and algorithmic stability.

Neural Processes Neural processes are a well-known member of the stochastic process family by directly capturing uncertainties with deep neural networks, which are not only computationally efficient but also retain a probabilistic interpretation of the model [9, 10, 14, 13]. Starting with conditional neural processes (CNP) [9], there have been several follow-up works to improve NPs in various aspects [6]. Vanilla CNP combines neural networks with the Gaussian process to extract prior knowledge from training data. NP [10] introduces a global latent variable to model uncertainty in a variational manner. Considering the problem of underfitting in the vanilla NP, Attentive NP [14] introduces the attention mechanism to improve the model's reconstruction quality. [11] introduced convolutional conditional neural process (CONVCNP) models translation equivariance in the data. Wang and Van Hoof [27] presented a doubly stochastic variational process (DSVNP), which combines both global and local latent variables. Lee et al. [18] extended NP using Bootstrap and proposed the bootstrapping neural processes (BNP). Kawano et al. [13] presented a group equivariant conditional neural process by incorporating group equivariant into CNPs in a meta-learning manner. Wang and van Hoof [28] proposed to combine the Mixture of Expert models with NPs to develop more expressive exchangeable stochastic processes. Kim et al. [15] proposed a stochastic attention mechanism for NPs to capture appropriate context information. Although there are many NP variants to improve the model performance, those do not consider stability to yield high generalization performance.

Algorithmic Stability Stability, as known as algorithmic stability, is a computational learning theory of how a machine learning algorithm is perturbed by small changes to its inputs [2]. Many efforts have been made to analyze various notions of algorithmic stability and prove that a broad spectrum of learning algorithms are stable in some sense [2, 3, 29, 12]. [3] proved that l 2 regularized learning algorithms are uniformly stable and able to obtain new bounds on generalization performance. [29] generalized [3]'s results and proved that regularized learning algorithms with strongly convex penalty functions on bounded domains. Hardt et al. [12] showed that parametric models trained by stochastic gradient descent algorithms are uniformly stable. Li et al. [19] introduced the stability notation to low-rank matrix approximation. Liu et al. [22] proved that tasks in multi-task learning can act as regularizers and that multi-task learning in a very general setting will therefore be uniformly stable under mild assumptions. This is the first work to investigate the stability of NPs from theoretical guidelines and derive NPs solutions with high stability.

3 Preliminary

Let calligraphic letters (e.g., A ) indicate sets, capital letters (e.g., A ) indicate scalars, lower-case bold letters (e.g., a ) denote vectors, and capital bold letters (e.g., A ) indicate matrices. Suppose there is a dataset D = ( X , y ) = { ( x i , y i ) } N i =1 with N data points X = [ x 1 , x 2 , · · · , x N ] glyph[latticetop] ∈ R N × D , and corresponding labels y = [ y 1 , y 2 , · · · , y N ] ∈ R N . Considering an arbitrary number of data points D C = ( X C , y C ) = { ( x i , y i ) } i ∈C , where C ⊆ { 1 , 2 , · · · , N } is an index set defining context information, neural processes model the conditional predictive distribution of the target values y T = { y i } i ∈T

at some target data points X T = { x i } i ∈T based on the context D C , i.e. P ( y T | X T , D C ) . Usually, target set is defined as T = { 1 , 2 , · · · , N } . Only in CNP [9], T ⊆ { 1 , 2 , · · · , N } and T ∩ C = ∅ . In this paper, we define T = { 1 , 2 , · · · , N } for all NPs, i.e. conditional predictive distribution is P ( y | X , D C ) = ∏ N i =1 P ( y i | x i , D C ) .

Fundamentally, there are two NP variants: deterministic and probabilistic. Deterministic NP [9], i.e. CNP, models the conditional distribution as P ( y | X , D C ) = P ( y | X , r C ) , where r C ∈ R d is an aggregated feature vector processed by a function that maps D C into a finite-dimensional vector space in a permutation-invariant way. In probabilistic NPs [10], a latent variable z ∈ R d is introduced to capture model uncertainty and the NPs infer P θ ( z |D C ) given context set using the reparameterization trick [16] and models such a conditional distribution as P θ ( y i | x i , D C ) = ∫ P θ ( y i | x i , D C , z ) P θ ( z |D C ) d z and it is trained by maximizing an ELBO: E z ∼ P θ ( z | X , y ) [log P θ ( y | X )] -KL [ P θ ( z | X , y ) ‖ P θ ( z |D C )] .

Meta Training NP Prediction To achieve fast prediction on a new context set at test time, NPs meta-learn a distribution over predictors. To perform meta-learning, we require a meta-dataset (dataset of datasets). We consider an unknown distribution µ on an instance space X × Y , and a set of independent sample D = { ( x i , y i ) } N i =1 drawn from µ : ( x i , y i ) ∼ µ and D ∼ µ N . Suppose meta-dataset contains M datasets D 1: M = {D m } M m =1 with D m = {D C m , D T m } , we assume that all M datasets drawn from a common environment τ , which is a probability measure on the set of probability measures on X × Y . The draw of µ ∼ τ indicates the encounter of a specific learning task µ in the environment τ . For simplicity, we assume that each dataset has the same sample size N . Following the previous work related to multi-task learning [24] and meta learning [5], The environment τ induces a measure µ N,τ on ( X × Y ) N such that µ N,τ ( A ) = E µ ∼ τ [ µ N ( A )] , ∀ A ⊆ ( X × Y ) N . Thus a dataset D m is independently sampled from a task µ encountered in τ , which is denoted as D m ∼ µ N,τ for m ∈ [ M ] .

Suppose there exists a meta parameter θ indicating the shared knowledge among different tasks. In this case, a meta learning algorithm A meta for NPs takes meta-datasets D 1: M as input, and then outputs a meta parameter θ = A meta ( D 1: M ) ∼ P θ |D 1: M . When given a new test dataset D , we can evaluate the quality of the meta parameter θ by the following true risk:

where R µ ( θ ) = -E ( x i ,y i ) ∼ µ log P θ ( y i | x i , D C ) . Usually, τ and µ are unknown, we can only estimate the meta parameter θ from the observed data D 1: M . In this case, the empirical risk w.r.t θ is:

NPs have various strengths: 1) Efficiency : meta-learning allows NPs to incorporate information from a new context set and make predictions with a single forward pass. The complexity is linear or quadratic in the context size instead of cubic as with Gaussian process regression; 2) Flexibility : NPs can define a conditional distribution of an arbitrary number of target points, conditioning an arbitrary number of observations; 3) Permutation invariance : the encoders of NPs use set property [32] to make the target prediction permutation invariant. Thanks to these properties, NPs are widely-used in lots of tasks, e.g., Bayesian optimization [8], recommendation [20, 21], physics engines controlling [27] etc. While there are many NP variants to improve the performance of NPs [9, 10, 14, 13, 15, 28], those do not take model's stability into consider account yet, which is the key to the robustness of the model.

4 Problem Formulation

Stability of NP Astable learning algorithm has the property that replacing one element in the training set does not result in a significant change to the algorithm's output [2]. Therefore, if we take the training error as a random variable, the training error of a stable learning algorithm should have a small variance. This implies that stable algorithms have the property that the training errors are close to the testing error [2]. Based on the defined risks, the algorithmic stability of approximate { y i } i ∈T in NPs is defined as follows.

Definition 4.1. (Algorithmic Stability of Neural Processes) For any measure µ N,τ on ( X ×Y ) N such that µ N,τ ( A ) = E µ ∼ τ [ µ N ( A )] , ∀ A ⊆ ( X × Y ) N , sample M datasets D 1: M from µ N,τ randomly. For a given glyph[epsilon1] > 0 , we say that R D 1: M ( θ ) is δ -stable if the following holds:

The above stability for NPs has the property that the generalization error is bounded, which indicates that minimizing the training error will have a high probability of minimizing the testing error. This new stability notion makes it possible to measure the generalization performance between different NP approximations. For instance, for any two meta-datasets D 1 1: M and D 2 1: M from µ N,τ , train NPs on D 1 1: M and D 2 1: M are δ 1 -stable and δ 2 -stable, respectively. Then R D 1 1: M ( θ ) is more stable than R D 2 1: M ( θ ) if δ 1 < δ 2 . This implies that R D 1 1: M ( θ ) is close to R τ ( θ ) with higher probability than R D 2 1: M ( θ ) , i.e. minimizing R D 1 1: M ( θ ) will lead to solutions that are of high probabilities with better generalization performance than minimizing R D 2 1: M ( θ ) .

Based on the above analysis, we can see that the reliability of data points is crucial to the success of NPs and frail NPs are susceptible to noise.

Stability vs. Generalization Error The sparsity of the data, incomplete and noisy introduces challenges to the algorithm stability. NP models are biased to the quality of context data and target data, so small changes in the training data (noisy) may significantly change the models. In this case, unstable solutions will introduce high training error variance, and minimizing the training error may not guarantee consistent error reduction on the testing dataset, i.e., low generalization performance. In other words, the algorithm stability has a direct impact on generalization performance, and an unstable NP solution has low generalization performance. We take NPs with 1D regression task as an example [9] to inves-

Generalization

Stability

Stability

Generalization

1

10

20

50

80

100

The

number

of

replaced

datasets

0%

20%

40%

60%

80%

100%

0

0.05

0.1

0.15

0.2

0.25

tigate the relationship between generalization performance and stability of NPs. The total number of training and testing datasets is 200 and 100 . We trained the NPs model with curves generated from the Gaussian process with RBF kernels by replacing the normal data dataset with a noisy dataset, i.e. the number of replaced datasets is turned in { 1 , 10 , 20 , 50 , 80 , 100 } . We quantify stability changes of NPs with the generalization error when the number of replaced datasets increases from 1 to 100 . We compute the difference between training error and test error to measure generalization error. We define the difference between test error and training error as R τ ( θ ) -R D 1: M ( θ ) , and compute P ( | R τ ( θ ) -R D 1: M ( θ ) | ≤ glyph[epsilon1] ) with 100 different runs to measure stability. Here we choose glyph[epsilon1] in Definition 4.1 as 0.0015 to cover all error differences when the number of replaced datasets is 1 . As shown in Figure 1, the generalization error increases when the number of replaced points increases since the testing error becomes lower. On the contrary, the stability of NP decreases with the number of replaced points increases. This indicates that stability decreases with generalization error increases. This study demonstrates that existing NPs suffer from lower generalization performance due to low algorithmic stability. Therefore, it is important to develop a stable solution for NPs that offers good generalization performance.

5 Method

In this section, inspired by the previous work [19], we present a stable solution for NPs with stability and high generalization. Algorithmic stability provides an intuitive way to measure the changes in the outputs of a learning algorithm when the input is changed. Various ways have been introduced to measure algorithmic stability. Following the definition of uniform stability [2], given a stable NP, the approximation results remain stable if the change of the datasets. For instance, we can remove a subset of easily predictable data points from D 1: M to obtain D ′ 1: M . It is desirable that the solution of minimizing both D 1: M and D ′ 1: M together will be more stable than the solution of minimizing D 1: M only. The following Theorem formally proves the statement.

Theorem 5.1. Let D 1: M ( M ≥ 2 ) be a sampled meta-dataset of measure µ N,τ . Let D s ∈ D 1: M be a subset of the meta-dataset, which satisfy that ∀ ( x i , y i ) ∈ D s , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) . Let D ′ 1: M = D 1: M - D s , then for any glyph[epsilon1] > 0 and 1 > w 0 > 0 , 1 > w 1 > 0 ( w 0 + w 1 = 1 ), w 0 R D 1: M ( θ ) + w 1 R D ′ 1: M ( θ ) and R D 1: M ( θ ) are δ 1 -stable and δ 2 -stable, respectively, then δ 1 ≤ δ 2 .

Proof. Let's assume that R τ ( θ ) -R D 1: M ( θ ) ∈ [ -a 1 , a 1 ] and R τ ( θ ) -( w 0 R D 1: M ( θ ) + w 1 R D ′ 1: M ( θ )) ∈ [ -a 2 , a 2 ] are two random variables with zero mean, where a 1 = sup { R τ ( θ ) -R D 1: M ( θ ) } and a 2 =

sup { R τ ( θ ) -( w 0 R D 1: M ( θ ) + w 1 R D ′ 1: M ( θ )) } . Based on Markov's inequality 1 , for any t > 0 , we have

Based on Hoeffding's lemma 2 , we have E [ e t ( R τ ( θ ) -R D 1: M ( θ )) ] ≤ e 1 2 t 2 a 2 1 , i.e. P ( R τ ( θ ) -R D 1: M ( θ ) ≥ glyph[epsilon1] ) ≤ e 1 2 t 2 a 2 1 e tglyph[epsilon1] . Similarly, we have P ( R τ ( θ ) -R D 1: M ( θ ) ≤ -glyph[epsilon1] ) ≤ e 1 2 t 2 a 2 1 e tglyph[epsilon1] . Combining those two inequalities, we have P ( | R τ ( θ ) -R D 1: M ( θ ) | ≥ glyph[epsilon1] ) ≤ 2 e 1 2 t 2 a 2 1 e tglyph[epsilon1] , i.e.

Similarly, we have

In this case, the relationship between a 1 and a 2 is

Since ∀ ( x i , y i ) ∈ D s , -log P θ ( y i | x i , D C s ) ≤ R D 1: M ( θ ) , we have -(1 /N ) ∑ N i =1 log P θ ( y i | x i , D C s ) ≤ R D 1: M ( θ ) . Then, since D 1: M = D s ∪ D ′ 1: M . This means that sup { R D 1: M ( θ ) -R D ′ 1: M ( θ ) } ≤ 0 . Thus, we have a 2 ≤ a 1 . This turns out that 2 e 1 2 t 2 a 2 2 e tglyph[epsilon1] ≤ 2 e 1 2 t 2 a 2 1 e tglyph[epsilon1] , i.e. δ 1 ≤ δ 2 .

Theorem 5.1 indicates that, if we remove a subset that is easier to predict than average from D 1: M to form D ′ 1: M , then w 0 R D 1: M ( θ ) + w 1 R D ′ 1: M ( θ ) has higher probability of being close to R τ ( θ ) than R D 1: M ( θ ) . Therefore, minimizing w 0 R D 1: M ( θ ) + w 1 R D ′ 1: M ( θ ) will lead to solutions that have better generalization performance than minimizing R D 1: M ( θ ) .

However, Theorem 5.1 only proves that it is beneficial to remove an easily predictable dataset from D 1: M to obtain D ′ 1: M , but does not show how many datasets we should remove from D 1: M . Actually, removing more datasets that satisfy -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) can obtain better D ′ 1: M , as shown in following Theorem 5.2.

Theorem 5.2. Let D 1: M ( M ≥ 2 ) be a sampled meta-dataset of measure µ N,τ . Let D s 1 and D s 2 be two subsets of D 1: M , which satisfy D s 2 ⊂ D s 1 ⊂ D 1: M . D s 2 and D s 1 satisfy that ∀ ( x i , y i ) ∈ D s 1 , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) . Let D 1 1: M = D 1: M - D s 1 and D 2 1: M = D 1: M - D s 2 , then for any glyph[epsilon1] > 0 and 1 > w 0 > 0 , 1 > w 1 > 0 ( w 0 + w 1 = 1 ), w 0 R D 1: M ( θ ) + w 1 R D 1 1: M ( θ ) and w 0 R D 1: M ( θ ) + w 1 R D 2 1: M ( θ ) are δ 1 -stable and δ 2 -stable, respectively, then δ 1 ≤ δ 2 .

Proof. Let's assume that R τ ( θ ) -( w 0 R D 1: M ( θ )+ w 1 R D 1 1: M ( θ )) ∈ [ -a 1 , a 1 ] and R τ ( θ ) -( w 0 R D 1: M ( θ )+ w 1 R D 2 1: M ( θ )) ∈ [ -a 2 , a 2 ] are two random variables with 0 mean, where a 1 = sup { R τ ( θ ) -( w 0 R D 1: M ( θ ) + w 1 R D 1 1: M ( θ )) } and a 2 = sup { R τ ( θ ) -( w 0 R D 1: M ( θ ) + w 1 R D 2 1: M ( θ )) } .

Then, based on Markov's inequality and Hoeffding's lemma, we have

Since ∀ ( x i , y i ) ∈ D s 1 , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) and D s 2 ⊂ D s 1 ⊂ D 1: M , we have R D 1 1: M ( θ ) ≤ R D 2 1: M ( θ ) . Thus, we have sup { R D 1 1: M ( θ ) -R D 2 1: M ( θ ) } ≤ 0 since a 1 = a 2 + w 1 sup { R D 1 1: M ( θ ) -R D 2 1: M ( θ ) } and a 1 ≤ a 2 . Then, we can conclude that δ 1 ≤ δ 2 .

Theorem 5.2 indicates that removing more data points that are easy to predict will obtain more stable NPs. Therefore, it is desirable to choose D ′ 1: M (i.e. D 1: M -D s ) as the whole set which is harder to predict than average, i.e. the whole set satisfying ∀ ( x i , y i ) ∈ D ′ 1: M , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) . Without loss of generality, we can extend Theorem 5.2 by considering several harder predicted sets to obtain a more stable solution by minimizing them all together. In this case, we need to prove that the stability of a model with K subsets is better than the model with K -1 subsets, as shown in Theorem 5.3.

Theorem 5.3. Let D 1: M ( M ≥ 2 ) be a sampled meta-dataset of measure µ N,τ . Let D s 1 , D s 2 , · · · , D sK ⊂ D 1: M be K subsets and satisfy ∀ ( x i , y i ) ∈ D sk ( k ∈ [ K ]) , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) . Let D k 1: M = D 1: M - D sk for all k ∈ [ K ] , then for any glyph[epsilon1] > 0 and 1 > w k > 0 for all k ∈ [ K ] , ( w 0 + w 1 + · · · + w K = 1 ), w 0 R D 1: M ( θ ) + ∑ K k =1 w k R D k 1: M ( θ ) and ( w 0 + w K ) R D 1: M ( θ ) + ∑ K -1 k =1 w k R D k 1: M ( θ ) are δ 1 -stable and δ 2 -stable, respectively, then δ 1 ≤ δ 2 .

Proof. For simplicity, we denote w 0 R D 1: M ( θ ) + ∑ K k =1 w k R D k 1: M ( θ ) as R 1 and ( w 0 + w K ) R D 1: M ( θ ) + ∑ K -1 k =1 w k R D k 1: M ( θ ) as R 2 . Let's assume that R τ ( θ ) -R 1 ∈ [ -a 1 , a 1 ] and R τ ( θ ) -R 2 ∈ [ -a 2 , a 2 ] are two random variables with 0 mean, where a 1 = sup { R τ ( θ ) -R 1 } and a 2 = sup { R τ ( θ ) -R 2 } . Then, based on Markov's inequality and Hoeffding's lemma, we have

Similar to the proof of Theorem 5.3, we have R D K 1: M ( θ ) ≥ R D 1: M ( θ ) , which indicates that sup { R D 1: M ( θ ) -R D K 1: M ( θ ) } ≤ 0 . Since a 2 = a 1 + w K sup { R D 1: M ( θ ) -R D K 1: M ( θ ) } , we know that a 2 ≤ a 1 . Thus we can conclude that 2 e 1 2 t 2 a 2 2 e tglyph[epsilon1] ≤ 2 e 1 2 t 2 a 1 2 e tglyph[epsilon1] , i.e. δ 1 ≤ δ 2 .

Based on Theorem 5.3, we know that optimization on D 1: M and more than one hard predictable subsets of D 1: M can achieve more stable prediction. However, how to select data points that are difficult to predict from D 1: M is still a challenging problem, especially, since we need to select K subsets.

5.1 The Proposed Solution

According to the analysis of stability in NPs, we propose a stable solution for NPs to achieve model stability with the aid of hard predictable subsets selection. Specifically, we introduce a solution to obtain those hard predictable subsets based on only one set of easily predicted data points which can be broken into four steps:

Selecting a existing NP model (e.g., CNP [9], NP[10]) and training it with meta-dataset D 1: M ;

Selecting an easily predicted subset D s ⊂ D 1: M , which satisfies that ∀ ( x i , y i ) ∈ D s , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) ;

Dividing D s into K non-overlapping subsets D s 1 , D s 2 , · · · , D sK and satisfying ∪ K k =1 D sk = D s ;

Defining K subsets that are difficult to predict, i.e. D k 1: M = D 1: M -D sk for all k ∈ [ K ] ;

Thus, a new extension of NPs is given as

where w 0 , w 1 , · · · , w K indicate the contributions of each component and satisfy ∑ K k =0 w k = 1 .

The whole learning algorithm is given in Algorithm 1. From steps 1 to 12, we obtain K different hard predictable subsets, and the complexity of lines 1 to 12 is O ( MN ) . The complexity of line 11 is related to the applied NP models (such as CNP, NP, ANP, etc). Thus, the computational complexity of NPs with our stable solution is similar to the original NPs. As shown in Algorithm 1, we need to pre-train the base model to select samples. In fact, the pre-trained model can not only be used for sample selection but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster.

Stability Guarantee Here we give a theoretical guarantee of the proposed stable solution.

Theorem 5.4. Let D 1: M ( M ≥ 2 ) be a sampled meta-dataset of measure µ N,τ . Let D s ⊂ D 1: M which satisfies that ∀ ( x i , y i ) ∈ D s , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) . By dividing D s into K subsets

Algorithm 1 Learning algorithm for stable NPs

Input: Meta-dataset D , β > 0 . 5 is a predefined probability for data selection. D = ∅ .

D s 1 , D s 2 , · · · , D sK which satisfy that ∪ K k =1 D sk = D 1: M . Let D 0 1: M = D 1: M -D s and D k 1: M = D 1: M -D sk for all k ∈ [ K ] , then for any glyph[epsilon1] > 0 and 1 > w k > 0 for all k ∈ [ K ] , ( w 0 + w 1 + · · · + w K = 1 ), w 0 R D 1: M ( θ ) + ∑ K k =1 w k R D k 1: M ( θ ) and w 0 R D 1: M ( θ ) + (1 -w 0 ) R D 0 1: M ( θ ) are δ 1 -stable and δ 2 -stable, respectively, then δ 1 ≤ δ 2 .

Proof. For simplicity, we denote w 0 R D 1: M ( θ ) + ∑ K k =1 w k R D k 1: M ( θ ) as R 1 and w 0 R D 1: M ( θ ) + (1 -w 0 ) R D 0 1: M ( θ ) as R 2 . Let's assume that R τ ( θ ) -R 1 ∈ [ -a 1 , a 1 ] and R τ ( θ ) -R 2 ∈ [ -a 2 , a 2 ] are two random variables with 0 mean, where a 1 = sup { R τ ( θ ) -R 1 } and a 2 = sup { R τ ( θ ) -R 2 } . Then, based on Markov's inequality and Hoeffding's lemma, we have

∀ k ∈ [ K ] , D sk ⊂ D s and ∀ ( x i , y i ) ∈ D s , -log P θ ( y i | x i , D C ) ≤ R D 1: M ( θ ) , we have R D k 1: M ( θ ) ≤ R D 0 1: M ( θ ) . By combining the above inequalities over all k ∈ [ K ] , we have

According to the above theorem, we can achieve model stability by selecting only one easily predicted subset.

6 Experiments

We started with learning predictive functions on synthetic datasets, and then high-dimensional tasks, e.g., system identification on physics engines, image completion, and Bayesian optimization, were performed to evaluate the properties of the NP-related models.

6.1 1D Regression

To verify the proposed stable solution, we combined the stable solution with different baseline NP classes (CNP [9], NP [10], ANP [14],ConvCNP [11], ConvNP [6], and their bootstrapping versions [18]) and compared them on 1D regression task. Among them, BCNP, BNP, BANP, BConvCNP, and NConvNP are recently proposed stable strategies for NPs with Bootstrap. Specifically, the stochastic process (SP) initializing with a 0 mean Gaussian Process (GP) y (0) ∼ GP (0 , k ( · , · )) indexed in the interval x ∈ [ -2 . 0 , 2 . 0] were used to generate data, where the radial basis function kernel and Matern Kernel were adopted for model-data mismatch scenario. More detailed information can be obtained in the Appendix. We investigated the model performance in terms of different noise settings. We introduced Gaussian noise N (0 , 1) and added noise to different proportions of the data, such as { 0% , 5% , 10% , 15% } . Table 1 lists the average log-likelihoods comparison in terms of different noise proportions. The best result is marked in bold. First, we can see that if we adopt the robust solution in baselines, the model achieves the best results on all the datasets, showing the effectiveness of the

stable solution. Besides, performances on all methods become less accurate in more complicated settings, while our solution has fewer effects. One interesting observation is that the improvements against the base model on CNP are less significant than NP and ANP. The possible reason is that CNP only predicts points out of the context set.

To investigate the model's ability to address model-data mismatch scenarios, we conducted experiments on 1D regression tasks with Periodic kernel. Following the setting of BANP and similar noise settings of our previous kernels, we list the results on both original data and noise( +15 ) data in Table 2. In this model-data mismatch data, stable versions still significantly outperform their corresponding original versions.

6.2 Image Completion

Image completion can be regarded as a 2D function regression task and be interpreted as being generated from a stochastic process (since there are dependencies between pixel values). Following the setting in previous work [14], we trained the NPs on EMNIST [4] and 32 × 32 CELEBA [23] using the standard train/test split with up to 200 context/target points at training. Detailed experiment settings are given in the Appendix. We evaluated average log-likelihoods over all points on realizations from image completion. Table 3 lists the comparisons between NPs with and without our stable solution in terms of original setting and noise setting, and the performance demonstrates the superiority of our stable solution.

6.3 System Identification on Physics Engines

The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which is detailed in [7, 27]. The Cart-Pole swing-up task is a standard benchmark for nonlinear control due to the non-linearity in the dynamics, and the requirement for nonlinear

Dataset

Method

Original

Noise

(+5%)

Noise

(+10%)

Noise

(+15%)

context

target

context

target

context

target

context

target

EMNIST

CNP

0.9522

±

0

.

023

0.7515

±

0

.

0015

0.8977

±

0

.

0016

0.6336

±

0

.

017

0.8242

±

0

.

0018

0.5784

±

0

.

009

0.6566

±

0

.

0017

0.5341

±

0

.

016

BCNP

0.9678

±

0

.

010

0.8058

±

0

.

008

0.9015

±

0

.

008

0.6711

±

0

.

009

0.8415

±

0

.

007

0.6089

±

0

.

009

0.6788

±

0

.

006

0.5715

±

0

.

006

SCNP

0.9716

±

0

.

008

0.8343

±

0

.

006

0.9251

±

0

.

008

0.6971

±

0

.

007

0.8674

±

0

.

006

0.6343

±

0

.

007

0.6986

±

0

.

005

0.5877

±

0

.

005

NP

0.9678

±

0

.

004

0.7756

±

0

.

005

0.9011

±

0

.

009

0.6941

±

0

.

006

0.8544

±

0

.

009

0.6455

±

0

.

007

0.7034

±

0

.

009

0.5865

±

0

.

006

BNP

0.9757

±

0

.

005

0.8358

±

0

.

005

0.8116

±

0

.

007

0.7625

±

0

.

006

0.8759

±

0

.

007

0.6773

±

0

.

007

0.7451

±

0

.

005

0.6237

±

0

.

005

SNP

0.9847

±

0

.

005

0.8562

±

0

.

006

0.8368

±

0

.

006

0.7844

±

0

.

005

0.8984

±

0

.

005

0.6984

±

0

.

005

0.7653

±

0

.

005

0.6456

±

0

.

004

ANP

1.1125

±

0

.

002

1.0321

±

0

.

004

0.9815

±

0

.

002

0.6366

±

0

.

006

0.9021

±

0

.

004

0.7053

±

0

.

008

0.8454

±

0

.

002

0.7034

±

0

.

005

BANP

1.1355

±

0

.

003

1.0615

±

0

.

005

1.0236

±

0

.

002

0.6549

±

0

.

005

0.9155

±

0

.

004

0.7521

±

0

.

006

0.8612

±

0

.

003

0.7515

±

0

.

005

SANP

1.1531

±

0

.

000

1.0877

±

0

.

004

1.0421

±

0

.

002

0.6776

±

0

.

005

0.9321

±

0

.

002

0.7843

±

0

.

006

0.8732

±

0

.

003

0.7657

±

0

.

005

ConvCNP

1.1363

±

0

.

002

1.0461

±

0

.

004

1.0252

±

0

.

006

0.6448

±

0

.

005

0.9116

±

0

.

005

0.7115

±

0

.

006

0.8621

±

0

.

005

0.7246

±

0

.

004

BConvCNP

1.1425

±

0

.

004

1.0787

±

0

.

006

1.0311

±

0

.

005

0.6626

±

0

.

005

0.9252

±

0

.

006

0.7617

±

0

.

006

0.8717

±

0

.

006

0.7627

±

0

.

005

SConvCNP

1.1631

±

0

.

003

1.0894

±

0

.

004

1.0563

±

0

.

004

0.6778

±

0

.

004

0.9356

±

0

.

004

0.7885

±

0

.

005

0.8813

±

0

.

005

0.7692

±

0

.

006

ConvNP

1.1415

±

0

.

002

1.0563

±

0

.

004

1.0286

±

0

.

006

0.6536

±

0

.

006

0.9168

±

0

.

005

0.7171

±

0

.

007

0.8675

±

0

.

005

0.7368

±

0

.

005

BConvNP

1.1526

±

0

.

004

1.0837

±

0

.

005

1.0415

±

0

.

005

0.6684

±

0

.

005

0.9285

±

0

.

006

0.7762

±

0

.

006

0.8742

±

0

.

006

0.7727

±

0

.

005

SConvNP

1.1753

±

0

.

003

1.0934

±

0

.

004

1.0641

±

0

.

004

0.6837

±

0

.

005

0.9402

±

0

.

005

0.7925

±

0

.

006

0.8923

±

0

.

005

0.7853

±

0

.

005

CELEBA

CNP

1.0323

±

0

.

016

0.7845

±

0

.

013

1.0177

±

0

.

016

0.7438

±

0

.

017

0.8956

±

0

.

009

0.7344

±

0

.

011

0.7677

±

0

.

012

0.6096

±

0

.

009

BCNP

1.0452

±

0

.

009

0.8015

±

0

.

008

1.0275

±

0

.

009

0.7726

±

0

.

008

0.9351

±

0

.

006

0.8376

±

0

.

009

0.8015

±

0

.

010

0.6816

±

0

.

008

SCNP

1.0525

±

0

.

008

0.8243

±

0

.

006

1.0348

±

0

.

008

0.7868

±

0

.

006

0.9562

±

0

.

004

0.8545

±

0

.

008

0.8344

±

0

.

006

0.7045

±

0

.

005

NP

1.1333

±

0

.

004

0.8766

±

0

.

005

1.1043

±

0

.

015

0.8355

±

0

.

015

1.0034

±

0

.

008

0.8456

±

0

.

006

0.8935

±

0

.

009

0.6893

±

0

.

006

BNP

1.1732

±

0

.

005

0.8901

±

0

.

006

1.1378

±

0

.

007

0.8678

±

0

.

006

1.0411

±

0

.

008

0.8711

±

0

.

005

0.9256

±

0

.

008

0.7671

±

0

.

006

SNP

1.1952

±

0

.

005

0.9062

±

0

.

006

1.1565

±

0

.

005

0.8846

±

0

.

005

1.0542

±

0

.

006

0.8956

±

0

.

004

0.9425

±

0

.

006

0.7985

±

0

.

005

ANP

1.1633

±

0

.

002

1.0163

±

0

.

004

1.1377

±

0

.

004

0.9866

±

0

.

006

1.0418

±

0

.

004

0.8845

±

0

.

006

0.9363

±

0

.

004

0.7346

±

0

.

008

BANP

1.1751

±

0

.

002

1.0389

±

0

.

005

1.1488

±

0

.

004

1.0155

±

0

.

005

1.0602

±

0

.

005

0.9255

±

0

.

006

0.9489

±

0

.

004

0.8415

±

0

.

007

SANP

1.1854

±

0

.

000

1.0594

±

0

.

004

1.1685

±

0

.

002

1.0353

±

0

.

004

1.0772

±

0

.

002

0.9455

±

0

.

004

0.9655

±

0

.

003

0.8673

±

0

.

005

ConvCNP

1.1697

±

0

.

004

1.0366

±

0

.

004

1.1445

±

0

.

006

0.9947

±

0

.

006

1.0542

±

0

.

005

0.8971

±

0

.

007

0.9521

±

0

.

005

0.7563

±

0

.

006

BConvCNP

1.1822

±

0

.

004

1.0467

±

0

.

004

1.1511

±

0

.

005

1.0271

±

0

.

005

1.0686

±

0

.

006

0.9317

±

0

.

005

0.9542

±

0

.

005

0.8527

±

0

.

005

SConvCNP

1.1889

±

0

.

003

1.0615

±

0

.

005

1.1753

±

0

.

004

1.0478

±

0

.

004

1.0752

±

0

.

004

0.9485

±

0

.

005

0.9693

±

0

.

007

0.8714

±

0

.

006

ConvNP

1.1767

±

0

.

004

1.0451

±

0

.

004

1.1485

±

0

.

005

1.0156

±

0

.

006

1.0594

±

0

.

005

0.9066

±

0

.

007

0.9573

±

0

.

004

0.7779

±

0

.

005

BConvNP

1.1822

±

0

.

004

1.0555

±

0

.

004

1.1573

±

0

.

004

1.0351

±

0

.

005

1.0762

±

0

.

005

0.9461

±

0

.

003

0.9661

±

0

.

006

0.8636

±

0

.

003

SConvNP

1.1867

±

0

.

003

1.0668

±

0

.

004

1.1846

±

0

.

004

1.0487

±

0

.

005

1.0863

±

0

.

004

0.9511

±

0

.

005

0.9743

±

0

.

006

0.8836

±

0

.

003

controllers to successfully swing up and balance the pendulum. More detailed experimental settings are given in the Appendix.

For each configuration of the simulator including training and testing environments, we sampled 400 trajectories of the horizon as 10 steps using a random controller. During the testing process, 100 state transition pairs were randomly selected for each configuration of the environment, working as the maximum context points to identify the configuration of dynamics. Figure 2 shows the predictive Log-Likelihoods (LL) and Mean Average Error (MAE) on Cart-Pole State

0

1

2

3

4

CNP

NP

ANP

LL

Original

Bootstrap

Ours

(a) LL

0

0.02

0.04

0.06

0.08

0.1

CNP

NP

ANP

MAE

(b) MAE

Transition Testing Dataset. We can see that our stable NPs achieve better likelihood and lower prediction error than the original ones. The variances on the stable one are consistently smaller than all original baselines.

6.4 Bayesian Optimization

Following the setting in BANP [18], we conducted the Bayesian optimization experiment. Taking GP data with RBF, Matern, and Periodic prior functions as examples, we gave the results of ANP, ConvCNP, ConvNP, Boostraping versions, and our stable versions. To maintain consistent comparison, we standardized the initializations and normalized the results. We reported the best simple regret, which represents the difference between the current best observation and the global optimum. As shown in Table 4, we can see that our stable solutions consistently achieve lower regret than other NPs.

6.5 Predator-prey Models

Following [18] and [11], we consider the Lotka-Volterra model [30], which is used to describe the evolution of predator-prey populations. We first trained the models using simulated data generated from a Lotka-Volterra model and tested them on real-world data (Hudson's Bay hare-lynx data), which is quite different from the simulated data and can be considered as a mismatch scenario. Table 5 lists the results on both simulated and real data. Similar to the previous observation, our stable version still outperforms the original version. Among stable versions, SConvNP achieves the best performance.

6.6 Ablation Study

The key parameter in our stable solution is the number of hard predictable subsets K . Taking SANP as an example, we investigated the average log-likelihood in terms of different K on the 1D regression task, as shown in Figure 3. We can see that SANP performs better as K increases, reaches the best value at around K = 4 , and then becomes stable in performance as K grows larger. As proved in Theorem 5.3, optimization on D 1: M and more than one hard predictable subset of D 1: M can achieve more stable prediction. We also conducted different experiments to explore the impact of different w k . Taking the 1D regression task with RBF-GP data as an example, we set K = 3 and different w k for experiments, as shown in Table 6. It can be seen from the table below that when different weights are set, the model using a stable strategy is better than the original model, and when the weight is set to be equal, its performance is optimal. In addition, when the value of w k ( k ≥ 1 ) is significantly different from w 0 , such as (0 . 625 , 0 . 125 , 0 . 125 , 0 . 125) and (0 . 0625 , 0 . 3125 , 0 . 3125 , 0 . 3125) , its performance is more significantly reduced compared to (0 . 25 , 0 . 25 , 0 . 25 , 0 . 25) , but it still has a significant improvement compared to the original non-stable model.

1.24

1.25

1.26

1.27

1.28

1.29

0

1

2

4

8

10

LL

K

(a) RBF/context

0.56

0.57

0.58

0.59

0.6

0.61

0

1

2

4

8

10

LL

K

(b) RBF/Target

1.15

1.2

1.25

1.3

1.35

1.4

0

1

2

4

8

10

LL

K

(c) Matern/context

0.62

0.63

0.64

0.65

0.66

0.67

0

1

2

4

8

10

LL

K

(d) Matern/target

Figure 3: Log-likelihood (SANP) comparisons with different K on 1D regression task.

7 Conclusion and Future Work

In this paper, we provided theoretical guidelines for deriving stable solutions for NPs, which can obtain good generalization performance. Experiments demonstrated the proposed stable solution can help NPs to achieve more accurate and stable predictions. Although the theoretical analysis we give is based on regression models, it is still open to question whether this conclusion is appropriate for classification models. Therefore, we are interested in extending our theory, expecting it to apply to more different types of tasks.

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Olivier Bousquet and André Elisseeff. Algorithmic stability and generalization performance. Advances in Neural Information Processing Systems , pages 196-202, 2001.

Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research , 2:499-526, 2002.

Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN) , pages 2921-2926. IEEE, 2017.

Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via pac-bayes and uniform stability. Advances in Neural Information Processing Systems , 34:2173-2186, 2021.

Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems , 33:8284-8295, 2020.

Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, ICML , volume 4, page 25, 2016.

Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint arXiv:1903.11907 , 2019.

Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning , pages 1704-1713. PMLR, 2018.

Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622 , 2018.

Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.13556 , 2019.

Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning , pages 12251234. PMLR, 2016.

Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. arXiv preprint arXiv:2102.08759 , 2021.

Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05761 , 2019.

Mingyu Kim, Kyeongryeol Go, and Se-Young Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In International Conference on Learning Representations , 2022.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Volodymyr Kuleshov, Nathan Fenner, and Stefano Ermon. Accurate uncertainties for deep learning using calibrated regression. In International Conference on Machine Learning , pages 2796-2804. PMLR, 2018.

Juho Lee, Yoonho Lee, Jungtaek Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6606-6615. Curran Associates, Inc., 2020.

Dongsheng Li, Chao Chen, Qin Lv, Junchi Yan, Li Shang, and Stephen Chu. Low-rank matrix approximation with stability. In International Conference on Machine Learning , pages 295-303. PMLR, 2016.

Xixun Lin, Jia Wu, Chuan Zhou, Shirui Pan, Yanan Cao, and Bin Wang. Task-adaptive neural process for user cold-start recommendation. In Proceedings of the Web Conference , pages 1306-1316, 2021.

Huafeng Liu, Liping Jing, Dahai Yu, Mingjie Zhou, and Michael Ng. Learning intrinsic and extrinsic intentions for cold-start recommendation with neural stochastic processes. In Proceedings of the 30th ACM International Conference on Multimedia , pages 491-500, 2022.

Tongliang Liu, Dacheng Tao, Mingli Song, and Stephen J Maybank. Algorithm-dependent generalization bounds for multi-task learning. IEEE transactions on pattern analysis and machine intelligence , 39(2):227-241, 2016.

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV) , December 2015.

Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. The benefit of multitask representation learning. Journal of Machine Learning Research , 17(81):1-32, 2016.

Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning , pages 63-71. Springer, 2003.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems , pages 5998-6008, 2017.

Qi Wang and Herke Van Hoof. Doubly stochastic variational inference for neural processes with hierarchical latent variables. In International Conference on Machine Learning , pages 10018-10028. PMLR, 2020.

Qi Wang and Herke van Hoof. Learning expressive meta-representations with mixture of expert neural processes. In Advances in neural information processing systems , 2022.

Andre Wibisono, Lorenzo Rosasco, and Tomaso Poggio. Sufficient conditions for uniform stability of regularization algorithms. 2009.

Darren J Wilkinson. Stochastic modelling for systems biology . CRC press, 2018.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan Salakhutdinov, and Alexander Smola. Deep sets. arXiv preprint arXiv:1703.06114 , 2017.

Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

A Inductive biases

Here, we revisit some properties, which would help us understand NPs. First, we give a concept of Permutation Invariant Function which is the basic property of stochastic process, e.g., NPs.

Definition A.1. (Permutation Invariant Function) A function f ( · ) : × N i R D → R d mapping a set of data points { x i } N i =1 is Permutation Invariant Function if

where x i ∈ R D and the function output is a d dimensional vector. Operation π : [1 , 2 , · · · , N ] → [ π 1 , π 2 , · · · , π N ] is a permutation set over the order of elements in the set.

Definition A.2. (Permutation Equivariant Function) A function f ( · ) : × N i R D → R N mapping a set of data points { x i } N i =1 is Permutation Invariant Function if

where the function output contains N elements keeping the order of inputs.

Permutation Equivariant Function keeps the order of elements in the output consistent with that in the input under any permutation operation π . Permutation invariant functions are candidate functions for learning embeddings of a set or other order uncorrelated data structure { x i } N i =1 , and the invariant property is easy to be verified. Here, we give a mean operation structure over the output

B Model Architecture

We show the architectural details of the CNP, NP, and ANP models used for the 1D and 2D function regression experiments. The neural process aims to learn a stochastic process (random function) mapping target features x i to prediction y i given the context set D C as training data (a realization from the stochastic process), i.e., learning

Conditional neural process (CNP) [9] describes P ( y i | x i , D C ) with a deterministic neural network taking D C to output the parameters of P ( y i | x i , D C ) . CNP consists of an encoder f enc ( · ) , an aggregator f agg ( · ) and a decoder f dec ( · ) ; the encoder summarizes D C and x i into latent representations [ r 1 , · · · , r |C| ] ∈ R | C |× d via permutation-invariant neural network [31], where d is the number of latent dimensions, and aggregator summarizes the encoded context features to a single representation r C , and decoder takes as input the aggregated representations r C and x i and output the single output-specific mean µ i and variance σ 2 i for the corresponding value of y i .

where f enc ( · ) and f dec ( · ) are feed-forward neural networks. The decoder output µ i and variance σ 2 i are predicted mean and variance. We use Gaussian distribution N ( y i ; µ i , σ 2 i ) as predictive distribution. CNP is trained to maximize the expected likelihood E P ( T ) [ P ( y i | x i , D C ) ] .

Neural process [10] further models functional uncertainty using a global latent variable. Unlike CNP, which maps a context into a deterministic representation ˜ r i , NP encoders a context into a Gaussian latent variable z , giving additional stochasticity in function construction. Following [14], we consider

an NP with both a deterministic path and latent path, where the deterministic path models the overall skeleton of the function ˜ r i , and the latent path models the functional uncertainty:

with f (1) enc ( · ) and f (2) enc ( · ) having the same structure as f enc ( · ) in Eq.(17). In this scenario, the conditional distribution is lower bounded as:

We further approximate q ( z |D C ) ≈ P ( z |D C ) and train the model by maximizing this expected lower bound over tasks. Furthermore, ANP introduces attention mechanisms into NP to resolve the issue of under-fitting.

The architectural details of the CNP, NP, and ANP are the same as in [14]. Here we give the detailed architectures of the encoder and decoder of NPs.

B.1 Encoder without attention

Encoder focuses on learning embeddings for each data point in the context set, and the basic component is multi-layer perceptron, which is defined by

where l is the number of layers, d in , d h and d out are dimensinalities of inputs, hidden unites and outputs. Here RELU ( · ) is adapted as activation function.

The encoder in Vanilla CNP uses a deterministic encoder which focuses on learning embeddings for each data point in context set.

where d x and d y are the dimensionalities of x i and y i .

To follow the encoder structure in NP, we introduce another encoder aligned with original deterministic encoder to permit the same number parameters, i.e.,

The encoder in NP contains a deterministic path and a latent path, i.e.,

In this case, the encoder outputs deterministic representation φ and latent representation z .

B.2 Encoder with attention

The attention mechanism is widely used in NPs, Specifically, multi-head attention [26] is adapted, which is defined by

where d q , d v , d k are the dimensionalities of query Q , key K , and value V , respectively. n head is the number of head. Here Layer normalization [1] LAYERNORM ( · ) is adapted. It is easy to derive self-attention by setting Q = K = V , i.e.,

For CNP, the encoder with attention still contains two deterministic paths,

Similarly, encoder with attention in NP contains a deterministic path and a latent path, i.e.,

and

B.3 Decoder

The decoder focuses on predicting output for target points based on the encoder's outputs φ . For target point { x i } i ∈T , the decoder of CNP is defined by

Decoder of NP is defined by

C Implementation Details and Experiments

For CNP [9], BCNP [18], and our SCNP, we apply the encoder with attention described in Eq (26) and decoder described in Eq (29). For NP [10], ANP [14], BNP [18], BANP [18] and our SNP and SANP models, we apply encoder with attention described in Eq (27) and (28), and decoder described in Eq (30).

C.1 1D Regression

For synthetic 1D regression experiments, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP/SNP/SANP refer to Appendix B. The number of hidden units is d h = 128 and latent representation d z = 128 . The number of layers are l e = l de = l la = l qk = l v = 2 .

We generate datasets for synthetic 1D regression. Specifically, the stochastic process (SP) initializes with a 0 mean Gaussian Process (GP) y (0) ∼ GP (0 , k ( · , · )) indexed in the interval x ∈ [ -2 . 0 , 2 . 0] , where the radial basis function kernel k ( x, x ′ ) = σ 2 exp( -‖ x -x ′ ‖ 2 / 2 l 2 ) with s ∼ U (0 . 1 , 1 . 0) and σ ∼ U (0 . 1 , 0 . 6) . Furthermore, GP with Matern Kernel is adopted for model-data mismatch scenario, which is defined as k ( x, x ′ ) = σ 2 (1 + √ 5 d/l +5 d 2 / (3 l 2 )) exp( - √ 5 d/l ) and d = ‖ x -x ′ ‖ with s ∼ U (0 . 1 , 1 . 0) and σ ∼ U (0 . 1 , 0 . 6) . For a fair comparison, we set the same data generation, training, and testing for all models.

We trained all models for 100 , 000 steps with each step computing updates with a batch containing 100 tasks. We used the Adam optimizer with an initial learning rate 5 · 10 -4 and decayed the learning rate using Cosine annealing scheme for baselines. For SCNP/SNP/SANP, we set K = 3 . The size of the context C was drawn as |C| ∼ U (3 , 200) . Testings were done for 3 , 000 batches with each batch containing 16 tasks ( 48 , 000 tasks in total).

We investigate the model stability from the size of the context set and the percent of added noise. First, we conduct experiments on different size of context set, i.e., |C| ∈ { 20 , 50 , 100 , 200 } . Table 7 shows the Average Log-likelihoods performance comparison between different methods in terms of different context size. We can see that the performance becomes better with the increasing of |C| and NPs with stable solution still achieve better performance. Second, we investigate the model performance in terms of different noise setting. Here we introduce Gaussian noise N (0 , 1) and add noise to different proportions of the data, such as { 0% , 5% , 10% , 15% } . Table 8 lists the Average Log-likelihoods performance comparison in terms of different noise proportions.

Kernel

Method

20

50

100

200

context

target

context

target

context

target

context

target

RBF

CNP

0.8724

±

0

.

008

0.4334

±

0

.

007

0.9533

±

0

.

005

0.4854

±

0

.

010

1.1224

±

0

.

005

0.5322

±

0

.

007

0.1563

±

0

.

004

0.5783

±

0

.

004

BCNP

0.9015

±

0

.

009

0.4579

±

0

.

007

0.9787

±

0

.

007

0.5215

±

0

.

008

1.1687

±

0

.

007

0.5716

±

0

.

009

0.1985

±

0

.

005

0.6086

±

0

.

005

SCNP

0.9255

±

0

.

008

0.4733

±

0

.

004

0.9944

±

0

.

005

0.5433

±

0

.

006

1.1833

±

0

.

006

0.5918

±

0

.

005

0.2111

±

0

.

004

0.6333

±

0

.

004

NP

0.8215

±

0

.

004

0.3853

±

0

.

005

0.9124

±

0

.

006

0.4234

±

0

.

003

1.0855

±

0

.

003

0.4767

±

0

.

005

1.1225

±

0

.

002

0.5233

±

0

.

004

BNP

0.8714

±

0

.

004

0.4122

±

0

.

004

0.9712

±

0

.

004

0.4718

±

0

.

004

1.1426

±

0

.

005

0.5269

±

0

.

006

1.1716

±

0

.

004

0.5698

±

0

.

004

SNP

0.8955

±

0

.

003

0.4356

±

0

.

004

0.9866

±

0

.

004

0.4934

±

0

.

005

1.1637

±

0

.

004

0.5434

±

0

.

004

1.1958

±

0

.

003

0.5933

±

0

.

003

ANP

1.2563

±

0

.

002

0.5763

±

0

.

004

1.3233

±

0

.

002

0.6322

±

0

.

003

1.4633

±

0

.

003

0.6866

±

0

.

006

1.4982

±

0

.

006

0.7322

±

0

.

004

BANP

1.2715

±

0

.

003

0.5878

±

0

.

005

1.3325

±

0

.

004

0.6465

±

0

.

004

1.4778

±

0

.

003

0.6915

±

0

.

004

1.5115

±

0

.

005

0.7436

±

0

.

005

SANP

1.2831

±

0

.

000

0.5994

±

0

.

004

1.3452

±

0

.

002

0.6577

±

0

.

003

1.4898

±

0

.

001

0.7043

±

0

.

002

1.5285

±

0

.

002

0.7534

±

0

.

005

Matern

CNP

0.8531

±

0

.

005

0.2431

±

0

.

010

0.9123

±

0

.

005

0.2984

±

0

.

003

1.0522

±

0

.

004

0.3542

±

0

.

002

1.0984

±

0

.

008

0.4022

±

0

.

005

BCNP

0.8765

±

0

.

006

0.2788

±

0

.

009

0.9411

±

0

.

006

0.3266

±

0

.

005

1.0752

±

0

.

004

0.3762

±

0

.

004

1.1245

±

0

.

007

0.4326

±

0

.

006

SCNP

0.8963

±

0

.

003

0.2953

±

0

.

006

0.9555

±

0

.

004

0.3467

±

0

.

003

1.0967

±

0

.

003

0.3967

±

0

.

003

1.1467

±

0

.

008

0.4556

±

0

.

005

NP

0.7643

±

0

.

015

0.2041

±

0

.

015

0.8221

±

0

.

002

0.2547

±

0

.

003

0.9322

±

0

.

003

0.3155

±

0

.

004

1.0452

±

0

.

005

0.3563

±

0

.

005

BNP

0.8052

±

0

.

008

0.2651

±

0

.

007

0.8678

±

0

.

003

0.3163

±

0

.

004

0.9672

±

0

.

003

0.3656

±

0

.

005

1.1052

±

0

.

005

0.4015

±

0

.

005

SNP

0.8368

±

0

.

006

0.2844

±

0

.

005

0.8956

±

0

.

002

0.3326

±

0

.

004

0.9959

±

0

.

003

0.3849

±

0

.

004

1.1215

±

0

.

003

0.4313

±

0

.

004

ANP

1.2421

±

0

.

002

0.6366

±

0

.

004

1.3022

±

0

.

001

0.6881

±

0

.

004

1.4211

±

0

.

004

0.7331

±

0

.

003

1.4631

±

0

.

002

0.7753

±

0

.

008

BANP

1.3452

±

0

.

007

0.6513

±

0

.

004

1.4056

±

0

.

003

0.7015

±

0

.

004

1.4505

±

0

.

004

0.7531

±

0

.

005

1.4986

±

0

.

004

0.7996

±

0

.

006

SANP

1.3721

±

0

.

002

0.6653

±

0

.

004

1.4322

±

0

.

003

0.7126

±

0

.

004

1.4633

±

0

.

004

0.7644

±

0

.

003

1.5153

±

0

.

005

0.8125

±

0

.

004

C.2 System Identification on Physics Engines

The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which is detailed in [7, 27]. The Cart-Pole swing-up task is a standard benchmark for nonlinear control due to the non-linearity in the dynamics, and the requirement for nonlinear controllers to successfully swing up and balance the pendulum. A pendulum of length l is attached to a cart by a frictionless pivot. The system begins with the cart at position x c = 0 and the pendulum hanging down: θ . The goal is to accelerate the cart by applying horizontal force u t at each time-step t to invert and then stabilize the pendulum's endpoint at the goal. There are some parameters that need to be known, such as cart mass m c , pendulum mass m p , acceleration of gravity g = 9 . 82 m/s 2 , time horizon T , time discretization glyph[triangle] t and ground friction coefficient f c . In this case, the Cart-Pole swing-up task aims to forecast the transited state [ x c , θ, x ′ c , θ ′ ] in time step t +1 based on the input as a state action pair [ x c , θ, x ′ c , θ ′ , a ] in time step t .

For system identification task on physics engines, the neural architectures for CNP, NP, ANP, BANP and our RNP refer to Appendix B. The number of hidden unites is d h = 32 and latent representation d z = 32 . The number of layers are l e = l de = l la = l qk = l v = 2 .

To generate a variety of trajectories under a random policy for this experiment, the mass mc and the ground friction coefficient f c are varied in the discrete choices m c ∈ { 0 . 3 , 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 } and f c ∈ { 0 . 06 , 0 . 08 , 0 . 1 , 0 . 12 } . Each pair of [ m c , f c ] values specifies a dynamics environment, and we formulate all pairs of m c ∈ { 0 . 3 , 0 . 5 , 0 . 7 } and f c ∈ { 0 . 08 , 0 . 12 } as training environments with the rest 16 pairs of configurations as the testing environments. For each configuration of the simulator including training and testing environments, we sample 400 trajectories of horizon as 10 steps using a random controller, and more details refer to Supplementary material. During the testing process,

Goal

𝑚

!

𝜃

𝑙

𝑚

!

𝑓

!

𝑥

!

100 state transition pairs are randomly selected for each configuration of the environment, working as the maximum context points to identify the configuration of dynamics.

C.3 Image Completion

Analogous to the 1D experiments, we take random pixels of a given image at training as targets, and select a subset of this as contexts, again choosing the number of contexts and targets randomly ( n ∼ U [3 , 200] , m ∼ n + U [0 , 200 -n ] ). The x i are rescaled to [ -1 , 1] and the y i are rescaled to [ -0 . 5 , 0 . 5] . We use a batch size of 16 for both EMNIST and CelebA, i.e. use 16 randomly selected images for each batch.

For image completion experiments on EMNIST and CelebA dataset, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP, SNP, and SANP refer to Appendix B. The number of hidden unites is d h = 128 and latent representation d z = 128 . The number of layers are l e = l de = 4 , l la = l qk = l v = 5 . h head = 8

C.4 Uncertainty Measuring

Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. We further analyze the learned models using the framework introduced in [17] to quantify uncertainty by investigate the calibration error and sharpness of the models. By assuming predictive distribution P θ ( y i | x i , D C ) as Gaussian distribution N ( y i ; µ i , σ 2 i ) , we can get the probabilistic forecast F i ( x i ) . More formally, m confidence levels 0 ≤ p 1 < p 2 < · · · p m ≤ 1 are choose. For each threshold p j , we compute the empirical frequency

In this case, the calibration error is defined as a numerical score describing the quality of forecast calibration:

here w j is weight and we set w j = 1 .

Sharpness is measured by using the variance var ( F i ) = σ 2 i of the random variable whose CDF is F i . Low-variance predictions are tightly centered around one value. A sharpness score can be defined by

We evaluated the CE and sharpness of CNP, NP, ANP, BCNP, NBP, BANP, and corresponding stable versions SCNP, SNP, and SANP trained in the experiments. Table 10, 11, 12 list the calibration error and sharpness score on 1D regression, system identification, and image completion tasks. In several settings, models with our stable solution can achieve better calibration and sharpness but work worse in some settings, such as the calibration error being worse than NP and ANP in terms of 1D regression tasks with Matern. The possible reason is that our method tends to produce conservative credible intervals, so become under-confident or less over-confident in some settings.

1.24

1.25

1.26

1.27

1.28

1.29

0

1

2

4

8

10

LL

K

(a) RBF/context

0.56

0.57

0.58

0.59

0.6

0.61

0

1

2

4

8

10

LL

K

(b) RBF/Target

1.15

1.2

1.25

1.3

1.35

1.4

0

1

2

4

8

10

LL

K

(c) Matern/context

0.62

0.63

0.64

0.65

0.66

0.67

0

1

2

4

8

10

LL

K

(d) Matern/target

0

1

2

3

4

0

1

2

4

8

10

LL

K

(a) LL

0

0.02

0.04

0.06

0.08

0

1

2

4

8

10

MAE

K

(b) MAE

1.08

1.1

1.12

1.14

1.16

0

1

2

3

4

5

6

7

8

9

10

LL

K

(a) EMNIST

1.25

1.26

1.27

1.28

1.29

0

1

2

3

4

5

6

7

8

9

10

LL

K

(b) CELEBA

C.5 Ablation Study

The key parameter in our stable solution is the number of hard predictive subsets K . Taking SANP as an example, we investigated the average log-likelihood in terms of different K on 1D regression task, as shown in Figure 5. We can see that SANP performs better as K increases, reaches the best value at around K = 4 , and then becomes stable in performance as K grows larger. As proved in Theorem 5.4 in the main manuscript, optimization on D C and more than one hard predicted subsets of D C can achieve more stable prediction. Similarly, we also conducted experiments to investigate the effect of K on System identification and image completion task and similar observations can be seen in Figure 6 and 7.

Peer reviews:
comments: SUMMARY:
This paper proposes a new way of training neural processes that increases their algorithmic stability, a property known to be helpful for generalisation in a frequentist sense. They work out theory to motivate their procedure, which is then compared against other existing ways of training neural processes. They show their method improves performance on a variety of tasks.

STRENGTHS:
- The paper is well-written and well-motivated
- The method seems to show consistent improvements in a wide variety of tasks

WEAKNESSES:
- I think the authors could spend more time arguing for the importance of stability. I agree that it would be helpful, but it's not clear to me that it is the single crucial factor in making neural processes generalise.
- The authors try of the NP and ANP, but could consider more SOTA methods such as the convolutional neural process.

QUESTIONS:
- The method shows consistent improvement in quantitative metrics, but when looking at Fig 2 it is difficult to see why this is the case. It's not clear to me why the SNP predictions are such a significant improvement over the NP. Could the authors give a qualitative explanation why? If not, I'm concerned that the improvement in numbers could be an artefact of something not genuinely significant.

LIMITATIONS:
The authors give a very small amount of the paper to address limitations. I would expect a more extensive discussion here. In particular, how does the runtime of their algorithm compare with regular NPs? Is there some kind of trade-off here?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the scenario where noisy data is included in the context points during the training of Neural Processes. In order to address algorithmic instability, the authors provide theoretical guidelines to obtain stable solutions that exhibit high generalization capabilities. These guidelines use hard predictable subsets as additional different training datasets.\n\nSTRENGTHS:\nOriginality\n\n- This paper introduces numerous theoretical findings regarding stable solutions.\n- The idea is overall interesting and novel.\n\nClarity\n- The paper is effectively written and presents its content in a clear and comprehensible manner.\n\nWEAKNESSES:\nExperiment\n- The choice of baseline models appears to be somewhat outdated. Considering the existence of newer NP models after Attentive Neural Processes [1] and Bootstrapping Neural Processes [2], it would be beneficial to include more recent approaches such as Convolutional Neural Processes [3,4] as baselines.\n- The tasks employed in the experiments seem to be too simplistic. The results obtained from simple 1D regression and 2D regression datasets may not be sufficient to demonstrate the superiority of the proposed model. I recommend conducting tasks such as Bayesian Optimization and the Predator-Prey task from [2] to showcase the general applicability of the method. Additionally, it would be valuable to include results using a periodic kernel to address model-data mismatch scenarios, as the Matern kernel's similarity to the RBF kernel may not adequately demonstrate the robustness of the proposed model.\n\n\nReferences\n\n[1] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, and V. Oriol. Attentive neural processes. In International Conference on Learning Representations (ICLR), 2018.\n\n[2] J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.\n\n[3] J. Gordon, W. P. Bruinsma, A. Y. K. Foong, J. Requeima, Y. Dubois, and R. E. Turner. Convolutional conditional neural processes. In International Conference on Learning Representations (ICLR), 2020.\n\n[4] A. Y. K. Foong, W. P. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. E. Turner. Metalearning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.\n\nQUESTIONS:\nExperiment\n- In Algorithm 1, it appears that the NP model requires early training to classify hard and easy datasets. I'm curious to know the number of early training epochs needed for the NP model. Additionally, it would be helpful to understand if the training epochs for other baselines are the same as those for the proposed method. This would help ascertain whether the performance gain stems from additional training or the effectiveness of the proposed method itself.\n- Could you provide separate log likelihood results for the context dataset, target dataset, and the entire dataset (referred to as \"target\" in this paper) by splitting them? In reference [1] and [2], log likelihoods of the context and target are reported separately, and increasing the context log likelihood can sometimes lead to an increase in the whole log likelihood without any actual performance improvement (or even a decrease in performance) for the true target dataset.\n- It would be valuable to conduct experiments using other baselines and additional tasks as mentioned earlier. The current set of experiments, including 1D regression with RBF kernel, 2D regression, and the physics engine experiment, may not be sufficient to demonstrate the superiority of the proposed model across various tasks.\n- How are the values of 'w' in the loss function balanced? Could you perform an ablation study on these hyperparameters to determine the robustness of the training procedure with respect to these hyperparameters?\n\nReferences\n\n[1] H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, and V. Oriol. Attentive neural processes. In International Conference on Learning Representations (ICLR), 2018.\n\n[2] J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.\n\nLIMITATIONS:\nLimitation\n- The proposed model incurs K times additional computational cost during training compared to other models, as it requires calculating a new predictive distribution with K-1 augmented dataset.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the stability of neural processes was studied. The scenario is when the context data points are noisy and this paper introduces the concept of stability, proposes adaptive strategies and draw some conclusions from empirical observations.\n\nSTRENGTHS:\nThis paper is written in a neat way and the consideration of noisy context points can be necessary in some risk sensitive applications. The literature work is well overviewed.\n\nWEAKNESSES:\nEven though this work is easy to follow, there exists unfixable issues concerning the research motivation, definitions of stability and derived theorems.\n\nTo begin with, let us rethink types of uncertainty in neural processes. *As we know, there primarily exist two types of uncertainty, namely aleatoric uncertainty and epistemic uncertainty.* The considered noisy context points belong to the aleatoric type, which is irreducible. The number of context points or observation extent corresponds to the epistemic, which means the prediction is more accurate with increasing the number of context points.\n\n**1. Concept of algorithmic stability in Definition4.1.** In Line 141, the concept of stability is presented, and it suggests that the approximation results remain stable if the change of the datasets given a stable NP (see Line178-179). These definition and conclusion violate the property of predictive functions in stochastic processes, where increase of context points decreases the epistemic uncertainty.\nParticularly, **in Line133, the stable property defined in NPs is “A stable learning algorithm has the property  that replacing one element in the training set does not result in a significant change to the algorithm’s output”.** **I strongly disagree with this trait since NPs are Bayesian  non-parametric, which means the change of context points definitely changes the predictive distribution like that in Gaussian processes. This property enables the Bayesian optimization using NPs.** If the proposed stability is achieved, the crucial Bayesian non-parametric property no longer holds.\n\n\n**2. Connection between subset selection and noisy context data points.** It is unclear in a theoretical sense that how the subset selection alleviate the negative effect of noisy context points. This work considers the scenario of noise context points but focuses on the algorithmic stability solved by the bootstrapping trick. Hence, the logics are not well organized.\n\n**3. Overstated or imprecise claims.** In line 29-30, to motivate this work, it says “As demonstrated in previous work [7, 8, 12, 9], existing NPs cannot provide stable predictions under noisy conditions. However, as far as I know, not all of these work mentioned the importance of noisy conditions and did investigations in an empirical and theoretical way. In Line 101, it states that the ELBO is maximized in NPs, however, the real optimization objective is the approximate ELBO rather than real ELBO since the Jessen inequality no longer holds.\n\nQUESTIONS:\nNo other questions.\n\nLIMITATIONS:\nSee the weakness section.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the algorithmic stability and generalization error of neural processes (NPs). Motivated by the stability analysis of NPs, a new method is proposed to train NPs that results in improved stability and better generalization performance with theoretical guarantees. The proposed method is based on the idea of removing easy datasets from the meta-dataset, which is essentially a weighted-training algorithm that down weights easy samples. The authors also empirically verify the usefulness of the proposed method on both synthetic and real-world datasets.\n\nSTRENGTHS:\n1. The idea of studying the algorithmic stability and generalization error of NPs is interesting and well-motivated.\n2. The proposed method for improving the robustness of NPs has theoretical guarantees and appears to be useful in practice.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. In the beginning of the proof for every theorem, the random variables $R_{\\tau}(\\theta)-(...)$ are assume to have zero mean with no justifications. It is unclear why they have zero means. The authors should justify these assumptions since all proofs are based on them.\n2. The proposed method is essentially a weighted-training algorithm that down weights easy datasets. According to the definition in the paper, a dataset is easy if the NLL of *every data point* in it is smaller than $R_{\\mathcal{D}_{1:M}}(\\theta)$ under a regularly trained NP. I can imagine that in practice each dataset will have some easy data points and some hard data points, but it is less likely that all data points in a data set are easy. The authors should discuss the applicability of the proposed method in this regard.\n3. The proposed method is computationally more expensive, since it requires 1) training a regular NP; 2) filtering out easy datasets according to some criteria; 3) reweighting each dataset; and 4) training the final NP. The extra computational costs will especially be enlarged when the NP is used for downstream tasks such as Bayesian optimization where one needs to constantly update the model as new data points/datasets arrive. The authors should discuss and compare the computational costs between the proposed method and regular NPs.\n4. It would be more convincing if the authors can show that the NPs trained by the proposed method can lead to improved performance in downstream tasks such as Bayesian optimization.\n\nQUESTIONS:\nSee the weaknesses section. I will increase my score accordingly if some or all of the weakness points are addressed.\n\nLIMITATIONS:\nLimitations are not discussed in the paper.\n\nThis work does not have obvious potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}