SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Theorem 5.1 indicates that, if we remove a subset that is easier to predict than average from  $\mathcal{D}_{1:M}$  to form  $\mathcal{D}_{1:M}^{\prime}$ , then  $w_{0}R_{\mathcal{D}_{1:M}}(\theta) + w_{1}R_{\mathcal{D}_{1:M}^{\prime}}(\theta)$  has higher probability of being close to  $R_{\tau}(\theta)$  than  $R_{\mathcal{D}_{1:M}}(\theta)$ . Therefore, minimizing  $w_{0}R_{\mathcal{D}_{1:M}}(\theta) + w_{1}R_{\mathcal{D}_{1:M}^{\prime}}(\theta)$  will lead to solutions that have better generalization performance than minimizing  $R_{\mathcal{D}_{1:M}}(\theta)$ .
However, Theorem 5.1 only proves that it is beneficial to remove an easily predictable dataset from  $\mathcal{D}_{1:M}$  to obtain  $\mathcal{D}_{1:M}^{\prime}$ , but does not show how many datasets we should remove from  $\mathcal{D}_{1:M}$ . Actually, removing more datasets that satisfy  $-\log P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^{\mathcal{C}})\leq R_{\mathcal{D}_{1:M}}(\theta)$  can obtain better  $\mathcal{D}_{1:M}^{\prime}$ , as shown in following Theorem 5.2.
Theorem 5.2. Let  $\mathcal{D}_{1:M}$  ( $M \geq 2$ ) be a sampled meta-dataset of measure  $\mu_{N,\tau}$ . Let  $\mathcal{D}_{s1}$  and  $\mathcal{D}_{s2}$  be two subsets of  $\mathcal{D}_{1:M}$ , which satisfy  $\mathcal{D}_{s2} \subset \mathcal{D}_{s1} \subset \mathcal{D}_{1:M}$ .  $\mathcal{D}_{s2}$  and  $\mathcal{D}_{s1}$  satisfy that  $\forall (\mathbf{x}_i, y_i) \in \mathcal{D}_{s1}$ ,  $-\log P_{\theta}(y_i | \mathbf{x}_i, \mathcal{D}^{\mathcal{C}}) \leq R_{\mathcal{D}_{1:M}}(\theta)$ . Let  $\mathcal{D}_{1:M}^{1} = \mathcal{D}_{1:M} - \mathcal{D}_{s1}$  and  $\mathcal{D}_{1:M}^{2} = \mathcal{D}_{1:M} - \mathcal{D}_{s2}$ , then for any  $\epsilon > 0$  and  $1 > w_0 > 0$ ,  $1 > w_1 > 0$  ( $w_0 + w_1 = 1$ ),  $w_0 R_{\mathcal{D}_{1:M}}(\theta) + w_1 R_{\mathcal{D}_{1:M}^1}(\theta)$  and  $w_0 R_{\mathcal{D}_{1:M}}(\theta) + w_1 R_{\mathcal{D}_{1:M}^2}(\theta)$  are  $\delta_1$ -stable and  $\delta_2$ -stable, respectively, then  $\delta_1 \leq \delta_2$ .
Proof. Let's assume that  $R_{\tau}(\theta) - (w_0R_{\mathcal{D}_{1:M}}(\theta) + w_1R_{\mathcal{D}_{1:M}}^1 (\theta)) \in [-a_1,a_1]$  and  $R_{\tau}(\theta) - (w_0R_{\mathcal{D}_{1:M}}(\theta) + w_1R_{\mathcal{D}_{1:M}^2}(\theta)) \in [-a_2,a_2]$  are two random variables with 0 mean, where  $a_1 = \sup \{R_{\tau}(\theta) - (w_0R_{\mathcal{D}_{1:M}}(\theta) + w_1R_{\mathcal{D}_{1:M}^1}(\theta))\}$  and  $a_2 = \sup \{R_{\tau}(\theta) - (w_0R_{\mathcal{D}_{1:M}}(\theta) + w_1R_{\mathcal{D}_{1:M}^2}(\theta))\}$ .
Then, based on Markov's inequality and Hoeffding's lemma, we have
$$
P \left(\left| R _ {\tau} (\theta) - \left(w _ {0} R _ {\mathcal {D} _ {1: M}} (\theta) + w _ {1} R _ {\mathcal {D} _ {1: M} ^ {1}} (\theta)\right) \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {1} ^ {2}}}{e ^ {t \epsilon}}, \tag {8}
$$
$$
P \left(\left| R _ {\tau} (\theta) - \left(w _ {0} R _ {\mathcal {D} _ {1: M}} (\theta) + w _ {1} R _ {\mathcal {D} _ {1: M} ^ {2}} (\theta)\right) \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {2} ^ {2}}}{e ^ {t \epsilon}}.
$$
Since  $\forall (\mathbf{x}_i, y_i) \in \mathcal{D}_{s1}, -\log P_\theta(y_i | \mathbf{x}_i, \mathcal{D}^{\mathcal{C}}) \leq R_{\mathcal{D}_{1:M}}(\theta)$  and  $\mathcal{D}_{s2} \subset \mathcal{D}_{s1} \subset \mathcal{D}_{1:M}$ , we have  $R_{\mathcal{D}_{1:M}^1}(\theta) \leq R_{\mathcal{D}_{1:M}^2}(\theta)$ . Thus, we have  $\sup \{R_{\mathcal{D}_{1:M}^1}(\theta) - R_{\mathcal{D}_{1:M}^2}(\theta)\} \leq 0$  since  $a_1 = a_2 + w_1 \sup \{R_{\mathcal{D}_{1:M}^1}(\theta) - R_{\mathcal{D}_{1:M}^2}(\theta)\}$  and  $a_1 \leq a_2$ . Then, we can conclude that  $\delta_1 \leq \delta_2$ .
$^{1}$  (Extended version of Markov's Inequality) Let  $x$  be a real-valued non-negative random variable and  $\varphi(\cdot)$  be a nondecreasing nonnegative function with  $\varphi(a) > 0$ . Then, for any  $\epsilon > 0$ ,  $P(x \geq \epsilon) \leq \frac{\mathbb{E}[\varphi(x)]}{\varphi(\epsilon)}$ .
$^2$  (Hoeffding's Lemma) Let  $x$  be a real-valued random variable with zero mean and  $p(x \in [a, b]) = 1$ . Then, for any  $z \in \mathbb{R}$ ,  $\mathbb{E}[e^{zx}] \leq \exp \left(\frac{1}{8} z^2 (b - a)^2\right)$ .
Theorem 5.2 indicates that removing more data points that are easy to predict will obtain more stable NPs. Therefore, it is desirable to choose  $\mathcal{D}_{1:M}^{\prime}$  (i.e.  $\mathcal{D}_{1:M} - \mathcal{D}_s$ ) as the whole set which is harder to predict than average, i.e. the whole set satisfying  $\forall (\mathbf{x}_i,y_i)\in \mathcal{D}_{1:M}^{\prime}, - \log P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^c)\leq R_{\mathcal{D}_{1:M}}(\theta)$ . Without loss of generality, we can extend Theorem 5.2 by considering several harder predicted sets to obtain a more stable solution by minimizing them all together. In this case, we need to prove that the stability of a model with  $K$  subsets is better than the model with  $K - 1$  subsets, as shown in Theorem 5.3.
Theorem 5.3. Let  $\mathcal{D}_{1:M}$  ( $M \geq 2$ ) be a sampled meta-dataset of measure  $\mu_{N,\tau}$ . Let  $\mathcal{D}_{s1}, \mathcal{D}_{s2}, \dots, \mathcal{D}_{sk} \subset \mathcal{D}_{1:M}$  be  $K$  subsets and satisfy  $\forall (\mathbf{x}_i, y_i) \in \mathcal{D}_{sk} (k \in [K])$ ,  $-\log P_{\theta}(y_i | \mathbf{x}_i, \mathcal{D}^{\mathcal{C}}) \leq R_{\mathcal{D}_{1:M}}(\theta)$ . Let  $\mathcal{D}_{1:M}^k = \mathcal{D}_{1:M} - \mathcal{D}_{sk}$  for all  $k \in [K]$ , then for any  $\epsilon > 0$  and  $1 > w_k > 0$  for all  $k \in [K]$ ,  $(w_0 + w_1 + \dots + w_K = 1)$ ,  $w_0 R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K} w_k R_{\mathcal{D}_{1:M}^k}(\theta)$  and  $(w_0 + w_K) R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K-1} w_k R_{\mathcal{D}_{1:M}^k}(\theta)$  are  $\delta_1$ -stable and  $\delta_2$ -stable, respectively, then  $\delta_1 \leq \delta_2$ .
Proof. For simplicity, we denote  $w_0 R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K} w_k R_{\mathcal{D}_{1:M}}^k(\theta)$  as  $R_1$  and  $(w_0 + w_K) R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K-1} w_k R_{\mathcal{D}_{1:M}^k}(\theta)$  as  $R_2$ . Let's assume that  $R_\tau(\theta) - R_1 \in [-a_1, a_1]$  and  $R_\tau(\theta) - R_2 \in [-a_2, a_2]$  are two random variables with 0 mean, where  $a_1 = \sup \{R_\tau(\theta) - R_1\}$  and  $a_2 = \sup \{R_\tau(\theta) - R_2\}$ . Then, based on Markov's inequality and Hoeffding's lemma, we have
$$
P \left(\left| R _ {\tau} (\theta) - R _ {1} \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {1} ^ {2}}}{e ^ {t \epsilon}}, \quad \left(\left| R _ {\tau} (\theta) - R _ {2} \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {2} ^ {2}}}{e ^ {t \epsilon}}. \tag {9}
$$
Similar to the proof of Theorem 5.3, we have  $R_{\mathcal{D}_{1:M}^K}(\theta) \geq R_{\mathcal{D}_{1:M}}(\theta)$ , which indicates that  $\sup \{R_{\mathcal{D}_{1:M}}(\theta) - R_{\mathcal{D}_{1:M}^K}(\theta)\} \leq 0$ . Since  $a_2 = a_1 + w_K \sup \{R_{\mathcal{D}_{1:M}}(\theta) - R_{\mathcal{D}_{1:M}^K}(\theta)\}$ , we know that  $a_2 \leq a_1$ . Thus we can conclude that  $\frac{2e^{\frac{1}{2}t^2a_2^2}}{e^{t\epsilon}} \leq \frac{2e^{\frac{1}{2}t^2a_1^2}}{e^{t\epsilon}}$ , i.e.  $\delta_1 \leq \delta_2$ .
Based on Theorem 5.3, we know that optimization on  $\mathcal{D}_{1:M}$  and more than one hard predictable subsets of  $\mathcal{D}_{1:M}$  can achieve more stable prediction. However, how to select data points that are difficult to predict from  $\mathcal{D}_{1:M}$  is still a challenging problem, especially, since we need to select  $K$  subsets.
5.1 The Proposed Solution
According to the analysis of stability in NPs, we propose a stable solution for NPs to achieve model stability with the aid of hard predictable subsets selection. Specifically, we introduce a solution to obtain those hard predictable subsets based on only one set of easily predicted data points which can be broken into four steps:
(1) Selecting a existing NP model (e.g., CNP [9], NP[10]) and training it with meta-dataset  $\mathcal{D}_{1:M}$ ;
(2) Selecting an easily predicted subset  $\mathcal{D}_s\subset \mathcal{D}_{1:M}$ , which satisfies that  $\forall (\mathbf{x}_i,y_i)\in \mathcal{D}_s$ $-\log P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^{\mathcal{C}})\leq R_{\mathcal{D}_{1:M}}(\theta);$
(3) Dividing  $\mathcal{D}_s$  into  $K$  non-overlapping subsets  $\mathcal{D}_{s1},\mathcal{D}_{s2},\dots ,\mathcal{D}_{sK}$  and satisfying  $\cup_{k = 1}^{K}\mathcal{D}_{sk} = \mathcal{D}_s$
(4) Defining  $K$  subsets that are difficult to predict, i.e.  $\mathcal{D}_{1:M}^{k} = \mathcal{D}_{1:M} - \mathcal{D}_{sk}$  for all  $k\in [K]$
Thus, a new extension of NPs is given as
$$
\mathcal {L} = \arg \min  _ {\theta} w _ {0} R _ {\mathcal {D} _ {1: M}} (\theta) + \sum_ {k = 1} ^ {K} w _ {k} R _ {\mathcal {D} _ {1: M} ^ {k}} (\theta), \tag {10}
$$
where  $w_0, w_1, \dots, w_K$  indicate the contributions of each component and satisfy  $\sum_{k=0}^{K} w_k = 1$ .
The whole learning algorithm is given in Algorithm 1. From steps 1 to 12, we obtain  $K$  different hard predictable subsets, and the complexity of lines 1 to 12 is  $O(MN)$ . The complexity of line 11 is related to the applied NP models (such as CNP, NP, ANP, etc). Thus, the computational complexity of NPs with our stable solution is similar to the original NPs. As shown in Algorithm 1, we need to pre-train the base model to select samples. In fact, the pre-trained model can not only be used for sample selection but its model parameters can be used as the initialization of the stable version model. At this time, the training of the stable version can converge faster.
Stability Guarantee Here we give a theoretical guarantee of the proposed stable solution.
Theorem 5.4. Let  $\mathcal{D}_{1:M}$  ( $M \geq 2$ ) be a sampled meta-dataset of measure  $\mu_{N,\tau}$ . Let  $\mathcal{D}_s \subset \mathcal{D}_{1:M}$  which satisfies that  $\forall (\mathbf{x}_i, y_i) \in \mathcal{D}_s, -\log P_\theta(y_i | \mathbf{x}_i, \mathcal{D}^{\mathcal{C}}) \leq R_{\mathcal{D}_{1:M}}(\theta)$ . By dividing  $\mathcal{D}_s$  into  $K$  subsets
Algorithm 1 Learning algorithm for stable NPs
Input: Meta-dataset  $\mathcal{D}_{1:M}$ $\beta >0.5$  is a predefined probability for data selection.  $\mathcal{D}_s = \emptyset$  1: Train a NP model with parameter  $\theta$  based on  $R_{\mathcal{D}_{1:M}}(\theta)$  .   
2: for  $(\mathbf{x}_i,y_i)\in \mathcal{D}_{1:M}$  do   
3: randomly generate split parameter  $\rho_{i}\in [0,1]$  .   
4: if  $((-\log P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^c)\leq R_{\mathcal{D}_{1:M}}(\theta))\& \rho_i\leq \beta)$    
5: or  $((-\log P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^c) > R_{\mathcal{D}_{1:M}}(\theta))\& \rho_i\leq 1 - \beta)$  then   
6:  $\mathcal{D}_s\gets \mathcal{D}_s\cup (\mathbf{x}_i,y_i)$  .   
7: end if   
8: end for   
9: divide  $\mathcal{D}_s$  into  $\mathcal{D}_{s1},\mathcal{D}_{s2},\dots ,\mathcal{D}_{sk}$  with  $\cup_{k = 1}^{K}\mathcal{D}_{sk} = \mathcal{D}_s$  .   
10: for  $k = 1,2,\dots ,K$  do   
11:  $\mathcal{D}_{1:M}^{k} = \mathcal{D}_{1:M} - \mathcal{D}_{sk};$    
12: end for   
13: update parameters by optimizing  $\theta^{*} = \arg \min_{\theta}w_{0}R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k = 1}^{K}w_{k}R_{\mathcal{D}_{1:M}^{k}}(\theta);$  Output: The learned optimal parameters  $\theta^{*}$
$\mathcal{D}_{s1}, \mathcal{D}_{s2}, \dots, \mathcal{D}_{sK}$  which satisfy that  $\cup_{k=1}^{K} \mathcal{D}_{sk} = \mathcal{D}_{1:M}$ . Let  $\mathcal{D}_{1:M}^{0} = \mathcal{D}_{1:M} - \mathcal{D}_{s}$  and  $\mathcal{D}_{1:M}^{k} = \mathcal{D}_{1:M} - \mathcal{D}_{sk}$  for all  $k \in [K]$ , then for any  $\epsilon > 0$  and  $1 > w_{k} > 0$  for all  $k \in [K]$ ,  $(w_{0} + w_{1} + \dots + w_{K} = 1)$ ,  $w_{0} R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K} w_{k} R_{\mathcal{D}_{1:M}^{k}}(\theta)$  and  $w_{0} R_{\mathcal{D}_{1:M}}(\theta) + (1 - w_{0}) R_{\mathcal{D}_{1:M}^{0}}(\theta)$  are  $\delta_{1}$ -stable and  $\delta_{2}$ -stable, respectively, then  $\delta_{1} \leq \delta_{2}$ .
Proof. For simplicity, we denote  $w_0R_{\mathcal{D}_{1:M}}(\theta) + \sum_{k=1}^{K}w_kR_{\mathcal{D}_{1:M}^k}(\theta)$  as  $R_1$  and  $w_0R_{\mathcal{D}_{1:M}}(\theta) + (1 - w_0)R_{\mathcal{D}_{1:M}^0}(\theta)$  as  $R_2$ . Let's assume that  $R_{\tau}(\theta) - R_1 \in [-a_1, a_1]$  and  $R_{\tau}(\theta) - R_2 \in [-a_2, a_2]$  are two random variables with 0 mean, where  $a_1 = \sup \{R_{\tau}(\theta) - R_1\}$  and  $a_2 = \sup \{R_{\tau}(\theta) - R_2\}$ . Then, based on Markov's inequality and Hoeffding's lemma, we have
$$
P \left(\left| R _ {\tau} (\theta) - R _ {1} \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {1} ^ {2}}}{e ^ {t \epsilon}}, \quad P \left(\left| R _ {\tau} (\theta) - R _ {2} \right| \leq \epsilon\right) \geq 1 - \frac {2 e ^ {\frac {1}{2} t ^ {2} a _ {2} ^ {2}}}{e ^ {t \epsilon}}. \tag {11}
$$
$\forall k \in [K], \mathcal{D}_{sk} \subset \mathcal{D}_s$  and  $\forall (\mathbf{x}_i, y_i) \in \mathcal{D}_s, -\log P_\theta(y_i | \mathbf{x}_i, \mathcal{D}^\mathcal{C}) \leq R_{\mathcal{D}_{1:M}}(\theta)$ , we have  $R_{\mathcal{D}_{1:M}^k}(\theta) \leq R_{\mathcal{D}_{1:M}^0}(\theta)$ . By combining the above inequalities over all  $k \in [K]$ , we have
$$
\sum_ {k = 1} ^ {K} w _ {k} R _ {\mathcal {D} _ {1: M} ^ {k}} (\theta) \leq \sum_ {k = 1} ^ {K} w _ {k} R _ {\mathcal {D} _ {1: M} ^ {0}} (\theta) = (1 - w _ {0}) R _ {\mathcal {D} _ {1: M} ^ {0}} (\mathcal {M}). \tag {12}
$$
Thus,  $\sup \{R_{\tau}(\theta) - R_1\} \leq \sup \{R_{1:M}(\theta) - R_2\}$ , i.e.  $a_1 \leq a_2$ . Thus we have  $\delta_1 \leq \delta_2$ .
According to the above theorem, we can achieve model stability by selecting only one easily predicted subset.
We started with learning predictive functions on synthetic datasets, and then high-dimensional tasks, e.g., system identification on physics engines, image completion, and Bayesian optimization, were performed to evaluate the properties of the NP-related models.
6.1 1D Regression
To verify the proposed stable solution, we combined the stable solution with different baseline NP classes (CNP [9], NP [10], ANP [14], ConvCNP [11], ConvNP [6], and their bootstrapping versions [18]) and compared them on 1D regression task. Among them, BCNP, BNP, BANP, BConvCNP, and NConvNP are recently proposed stable strategies for NPs with Bootstrap. Specifically, the stochastic process (SP) initializing with a 0 mean Gaussian Process (GP)  $y^{(0)} \sim GP(0, k(\cdot, \cdot))$  indexed in the interval  $x \in [-2.0, 2.0]$  were used to generate data, where the radial basis function kernel and Matern Kernel were adopted for model-data mismatch scenario. More detailed information can be obtained in the Appendix. We investigated the model performance in terms of different noise settings. We introduced Gaussian noise  $\mathcal{N}(0, 1)$  and added noise to different proportions of the data, such as  $\{0\%, 5\%, 10\%, 15\% \}$ . Table 1 lists the average log-likelihoods comparison in terms of different noise proportions. The best result is marked in bold. First, we can see that if we adopt the robust solution in baselines, the model achieves the best results on all the datasets, showing the effectiveness of the
stable solution. Besides, performances on all methods become less accurate in more complicated settings, while our solution has fewer effects. One interesting observation is that the improvements against the base model on CNP are less significant than NP and ANP. The possible reason is that CNP only predicts points out of the context set.
To investigate the model's ability to address model-data mismatch scenarios, we conducted experiments on 1D regression tasks with Periodic kernel. Following the setting of BANP and similar noise settings of our previous kernels, we list the results on both original data and noise  $(+15)$  data in Table 2. In this model-data mismatch data, stable versions still significantly outperform their corresponding original versions.
Table 2: Experiments on 1D regression data with Periodic kernel.
Periodic	Original	Noise(+15%)
context	target	context	target
ANP	0.5730±0.006	-4.2345±0.005	0.3521±0.005	-5.3211±0.007
ConvCNP	0.5983±0.006	-4.0215±0.005	0.3658±0.005	-4.5233±0.008
ConvNP	0.6125±0.005	-3.8952±0.006	0.3756±0.006	-4.3413±0.008
BANP	0.6253±0.003	-3.5413±0.005	0.3651±0.006	-4.2511±0.015
BConvCNP	0.6342±0.005	-3.4142±0.004	0.3712±0.004	-4.1750±0.008
BConvNP	0.6355±0.004	-3.3627±0.005	0.3768±0.008	-4.0116±0.007
SANP	0.6315±0.002	-3.3317±0.004	0.3748±0.004	-4.0515±0.003
SConvCNP	0.6433±0.002	-3.1515±0.004	0.3866±0.005	-3.9851±0.004
SConvNP	0.6551±0.000	-3.1062±0.004	0.3981±0.002	-3.8895±0.004
6.2 Image Completion
Image completion can be regarded as a 2D function regression task and be interpreted as being generated from a stochastic process (since there are dependencies between pixel values). Following the setting in previous work [14], we trained the NPs on EMNIST [4] and  $32 \times 32$  CELEBA [23] using the standard train/test split with up to 200 context/target points at training. Detailed experiment settings are given in the Appendix. We evaluated average log-likelihoods over all points on realizations from image completion. Table 3 lists the comparisons between NPs with and without our stable solution in terms of original setting and noise setting, and the performance demonstrates the superiority of our stable solution.
6.3 System Identification on Physics Engines
The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which is detailed in [7, 27]. The Cart-Pole swing-up task is a standard benchmark for nonlinear control due to the non-linearity in the dynamics, and the requirement for nonlinear
Table 4: Bayesian optimization experiments on data generated by different GP kernels.
Method	ANP	BANP	SANP	ConvCNP	BConvCNP	SConvCNP	ConvNP	BConvNP	SConvNP
RBF	0.1245±0.003	0.1341±0.003	0.1142±0.002	0.1215±0.002	0.1168±0.003	0.1037±0.002	0.1197±0.002	0.1156±0.003	0.1015±0.003
Matern	0.1518±0.003	0.1316±0.004	0.1201±0.002	0.1489±0.003	0.1301±0.002	0.1216±0.002	0.1446±0.002	0.1242±0.003	0.1204±0.004
Periodic	0.1892±0.002	0.1788±0.005	0.1672±0.001	0.1652±0.002	0.1526±0.004	0.1487±0.003	0.1611±0.002	0.1498±0.004	0.1446±0.002
controllers to successfully swing up and balance the pendulum. More detailed experimental settings are given in the Appendix.
For each configuration of the simulator including training and testing environments, we sampled 400 trajectories of the horizon as 10 steps using a random controller. During the testing process, 100 state transition pairs were randomly selected for each configuration of the environment, working as the maximum context points to identify the configuration of dynamics. Figure 2 shows the predictive Log-Likelihoods (LL) and Mean Average Error (MAE) on Cart-Pole State
Transition Testing Dataset. We can see that our stable NPs achieve better likelihood and lower prediction error than the original ones. The variances on the stable one are consistently smaller than all original baselines.
(a) LL
(b) MAE
Figure 2: The predictive Log Likelihood (LL) and Mean Average Error (MAE) on Cart-Pole state transition testing dataset.
6.4 Bayesian Optimization
Following the setting in BANP [18], we conducted the Bayesian optimization experiment. Taking GP data with RBF, Matern, and Periodic prior functions as examples, we gave the results of ANP, ConvCNP, ConvNP, Boostraping versions, and our stable versions. To maintain consistent comparison, we standardized the initializations and normalized the results. We reported the best simple regret, which represents the difference between the current best observation and the global optimum. As shown in Table 4, we can see that our stable solutions consistently achieve lower regret than other NPs.
Table 5: Predator-prey model results.
Method	ANP	BANP	SANP	ConvCNP	BConvCNP	SConvCNP	ConvNP	BConvNP	SConvNP
Simulated-context	2.5801±0.003	2.5912±0.002	2.6127±0.004	2.5912±0.002	2.6068±0.003	2.6164±0.002	2.5925±0.002	2.6125±0.005	2.6231±0.003
Simulated-target	1.8265±0.003	1.8635±0.004	1.8844±0.004	1.8352±0.004	1.8524±0.003	1.9516±0.002	1.9228±0.005	1.9442±0.003	1.9662±0.004
Real-context	1.7234±0.002	1.8496±0.005	1.8412±0.001	1.7950±0.002	1.8026±0.004	1.8744±0.003	1.8342±0.002	1.8476±0.005	1.8796±0.002
Real-target	-7.8042±0.002	-5.4836±0.004	-5.2527±0.001	-5.3414±0.005	-5.1526±0.004	-5.3513±0.004	-5.3155±0.002	-5.2615±0.004	-5.2145±0.003
6.5 Predator-prey Models
Following [18] and [11], we consider the Lotka-Volterra model [30], which is used to describe the evolution of predator-prey populations. We first trained the models using simulated data generated from a Lotka-Volterra model and tested them on real-world data (Hudson's Bay hare-lynx data), which is quite different from the simulated data and can be considered as a mismatch scenario. Table 5 lists the results on both simulated and real data. Similar to the previous observation, our stable version still outperforms the original version. Among stable versions, SConvNP achieves the best performance.
6.6 Ablation Study
The key parameter in our stable solution is the number of hard predictable subsets  $K$ . Taking SANP as an example, we investigated the average log-likelihood in terms of different  $K$  on the 1D regression task, as shown in Figure 3. We can see that SANP performs better as  $K$  increases, reaches the best value at around  $K = 4$ , and then becomes stable in performance as  $K$  grows larger. As proved in Theorem 5.3, optimization on  $\mathcal{D}_{1:M}$  and more than one hard predictable subset of  $\mathcal{D}_{1:M}$  can achieve more stable prediction. We also conducted different experiments to explore the impact of different  $w_{k}$ . Taking the 1D regression task with RBF-GP data as an example, we set  $K = 3$  and different  $w_{k}$  for experiments, as shown in Table 6. It can be seen from the table below that when different weights are set, the model using a stable strategy is better than the original model, and when the weight is set to be equal, its performance is optimal. In addition, when the value of  $w_{k}(k\geq 1)$  is significantly different from  $w_{0}$ , such as (0.625, 0.125, 0.125, 0.125) and (0.0625, 0.3125, 0.3125, 0.3125), its performance is more significantly reduced compared to (0.25, 0.25, 0.25, 0.25), but it still has a significant improvement compared to the original non-stable model.
(a) RBF/context
(b) RBF/Target
Figure 3: Log-likelihood (SANP) comparisons with different  $K$  on 1D regression task.
(c) Matern/context
(d) Matern/target
Table 6: Log-likelihood comparisons with different  $w_{k}$  on 1D regression task.
Weights	(0.25,0.25,0.25,0.25)	(0.4,0.2,0.2,0.2)	(0.625,0.125,0.125,0.125)	(0.1,0.3,0.3,0.3)	(0.0625,0.3125,0.3125,0.3125)
SCNP	(0.9255,0.4125)	(0.9127,0.4019)	(0.9035,0.3998)	(0.9149,0.4086)	(0.8927,0.3991)
SNP	(0.8955,0.3925)	(0.8737,0.3817)	(0.8716,0.3809)	(0.8831,0.3859)	(0.8657,0.3775)
SANP	(1.2831,0.5215)	(1.2776,0.5187)	(1.2738,0.5196)	(1.2791,0.5203)	(1.2712,0.5193)
SConvCNP	(1.3991,0.5996)	(1.3916,0.5933)	(1.3841,0.5915)	(1.3934,0.5946)	(1.3854,0.5919)
SConvNP	(1.4036,0.6015)	(1.3991,0.6004)	(1.3931,0.5992)	(1.4012,0.6015)	(1.3957,0.5995)
In this paper, we provided theoretical guidelines for deriving stable solutions for NPs, which can obtain good generalization performance. Experiments demonstrated the proposed stable solution can help NPs to achieve more accurate and stable predictions. Although the theoretical analysis we give is based on regression models, it is still open to question whether this conclusion is appropriate for classification models. Therefore, we are interested in extending our theory, expecting it to apply to more different types of tasks.
This work was partly supported by the National Natural Science Foundation of China under Grant 62176020; the National Key Research and Development Program (2020AAA0106800); the Joint Foundation of the Ministry of Education (8091B042235); the Beijing Natural Science Foundation under Grant L211016; the Fundamental Research Funds for the Central Universities (2019JBZ110); and Chinese Academy of Sciences (OEIP-O-202004).