SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 8: Average Log-likelihoods over all context and target points on realizations from Synthetic Stochastic Process on different percent of added noise. Here we set the context size to 20. (Mean ± Std). Note that adding 'S' before the original model name is a model with our stable solution.
Kernel	Method	Original	Noise(+5%)	Noise(+10%)	Noise(+15%)
context	target	context	target	context	target	context	target
RBF	CNP	0.8724±0.008	0.4334±0.007	0.8522±0.005	0.4001±0.010	0.8014±0.006	0.3552±0.004	0.7152±0.006	0.2853±0.005
BCNP	0.9042±0.009	0.4589±0.006	0.8774±0.006	0.4278±0.008	0.8316±0.006	0.3767±0.005	0.7487±0.007	0.3017±0.006
SCNP	0.9255±0.008	0.4733±0.004	0.8935±0.005	0.4478±0.006	0.8517±0.004	0.3986±0.005	0.7621±0.005	0.3279±0.006
NP	0.8215±0.004	0.3853±0.005	0.8011±0.004	0.3511±0.006	0.7611±0.005	0.3042±0.008	0.6722±0.005	0.2435±0.007
BNP	8722±0.004	0.4211±0.004	0.8321±0.003	0.3876±0.004	0.7922±0.004	0.3389±0.005	0.7189±0.004	0.2776±0.007
SNP	0.8955±0.003	0.4356±0.004	0.8567±0.003	0.4046±0.005	0.8165±0.004	0.3568±0.006	0.7356±0.005	0.2955±0.006
ANP	1.2563±0.002	0.5763±0.004	1.2245±0.007	0.5347±0.006	0.1742±0.005	0.4871±0.007	0.9821±0.005	0.4151±0.004
BANP	1.2722±0.004	0.5887±0.006	1.2411±0.005	0.5471±0.005	0.1886±0.006	0.4917±0.006	1.0642±0.006	0.4327±0.005
SANP	1.2831±0.000	0.5994±0.004	1.2564±0.004	0.5578±0.004	1.2052±0.004	0.5025±0.006	1.1243±0.005	0.4356±0.006
Matern	CNP	0.8531±0.005	0.2431±0.010	0.8231±0.005	0.2144±0.010	0.7761±0.008	0.1784±0.007	0.7052±0.005	0.1452±0.006
BCNP	0.8778±0.005	0.2762±0.009	0.8487±0.006	0.2477±0.009	0.8015±0.007	0.2051±0.007	0.7378±0.005	0.1766±0.006
SCNP	0.8963±0.003	0.2953±0.006	0.8689±0.005	0.2658±0.007	0.8268±0.006	0.2258±0.006	0.7567±0.004	0.1936±0.005
NP	0.7643±0.015	0.2041±0.015	0.7342±0.002	0.1725±0.008	0.6892±0.004	0.1542±0.006	0.6235±0.008	0.1342±0.007
BNP	0.8156±0.005	0.2689±0.007	0.7789±0.004	0.2215±0.005	0.7421±0.005	0.2117±0.007	0.6715±0.006	0.1828±0.006
SNP	0.8368±0.006	0.2844±0.005	0.8036±0.003	0.2483±0.003	0.7635±0.004	0.2325±0.006	0.6973±0.006	0.2016±0.005
ANP	1.2421±0.002	0.6366±0.004	1.2115±0.001	0.6001±0.008	1.1784±0.004	0.5622±0.006	1.125±0.007	0.5274±0.008
BANP	1.3456±0.003	0.6514±0.005	1.3125±0.005	0.6115±0.002	1.2672±0.004	0.5711±0.005	1.2236±0.006	0.5306±0.006
SANP	1.3721±0.002	0.6653±0.004	1.3461±0.003	0.6256±0.004	1.3011±0.003	0.5782±0.004	1.2457±0.005	0.5356±0.002
C.2 System Identification on Physics Engines
The second synthetic experiment focuses on evaluating model dynamics on a classical simulator, Cart-Pole systems, which is detailed in [7, 27]. The Cart-Pole swing-up task is a standard benchmark for nonlinear control due to the non-linearity in the dynamics, and the requirement for nonlinear controllers to successfully swing up and balance the pendulum. A pendulum of length  $l$  is attached to a cart by a frictionless pivot. The system begins with the cart at position  $x_{c} = 0$  and the pendulum hanging down:  $\theta$ . The goal is to accelerate the cart by applying horizontal force  $u_{t}$  at each time-step  $t$  to invert and then stabilize the pendulum's endpoint at the goal. There are some parameters that need to be known, such as cart mass  $m_{c}$ , pendulum mass  $m_{p}$ , acceleration of gravity  $g = 9.82m / s^{2}$ , time horizon  $T$ , time discretization  $\triangle t$  and ground friction coefficient  $f_{c}$ . In this case, the Cart-Pole swing-up task aims to forecast the transited state  $[x_{c},\theta ,x_{c}^{\prime},\theta^{\prime}]$  in time step  $t + 1$  based on the input as a state action pair  $[x_c,\theta ,x_c',\theta ',a]$  in time step  $t$ .
For system identification task on physics engines, the neural architectures for CNP, NP, ANP, BANP and our RNP refer to Appendix B. The number of hidden unites is  $d_h = 32$  and latent representation  $d_z = 32$ . The number of layers are  $l_e = l_{de} = l_{la} = l_{qk} = l_v = 2$ .
To generate a variety of trajectories under a random policy for this experiment, the mass mc and the ground friction coefficient  $f_{c}$  are varied in the discrete choices  $m_c \in \{0.3, 0.4, 0.5, 0.6, 0.7\}$  and  $f_{c} \in \{0.06, 0.08, 0.1, 0.12\}$ . Each pair of  $[m_c, f_c]$  values specifies a dynamics environment, and we formulate all pairs of  $m_c \in \{0.3, 0.5, 0.7\}$  and  $f_{c} \in \{0.08, 0.12\}$  as training environments with the rest 16 pairs of configurations as the testing environments. For each configuration of the simulator including training and testing environments, we sample 400 trajectories of horizon as 10 steps using a random controller, and more details refer to Supplementary material. During the testing process,
Figure 4: Cart-Pole Dynamical Systems. The cart and the pole are with masses  $m_{c}$  and  $m_{p}$ , and the length of the pole is  $l$ . And the configuration of the simulator is up to parameters of the cart-pole mass and the ground friction coefficient here with other hyper-parameters fixed in this experiment.
Table 9: Average Log-likelihoods over all context and target points on EMNIST and CELEBA.
Dataset	Method	Original	Noise(+5%)	Noise(+10%)	Noise(+15%)
context	target	context	target	context	target	context	target
EMNIST	CNP	0.9522±0.023	0.7515±0.0015	0.8977±0.0016	0.6336±0.017	0.8242±0.0018	0.5784±0.009	0.6566±0.0017	0.5341±0.016
BCNP	0.9678±0.010	0.8058±0.008	0.9015±0.008	0.6711±0.009	0.8415±0.007	0.6089±0.009	0.6788±0.006	0.5715±0.006
SCNP	0.9716±0.008	0.8343±0.006	0.9251±0.008	0.6971±0.007	0.8674±0.006	0.6343±0.007	0.6986±0.005	0.5877±0.005
NP	0.9678±0.004	0.7756±0.005	0.9011±0.009	0.6941±0.006	0.8544±0.009	0.6455±0.007	0.7034±0.009	0.5865±0.006
BNP	0.9757±0.005	0.8358±0.005	0.8116±0.007	0.7625±0.006	0.8759±0.007	0.6773±0.007	0.7451±0.005	0.6237±0.005
SNP	0.9847±0.005	0.8562±0.006	0.8368±0.006	0.7844±0.005	0.8984±0.005	0.6984±0.005	0.7653±0.005	0.6456±0.004
ANP	1.1125±0.002	1.0321±0.004	0.9815±0.002	0.6366±0.006	0.9021±0.004	0.7053±0.008	0.8454±0.002	0.7034±0.005
BANP	1.1355±0.003	1.0615±0.005	1.0236±0.002	0.6549±0.005	0.9155±0.004	0.7521±0.006	0.8612±0.003	0.7515±0.005
SANP	1.1531±0.000	1.0877±0.004	1.0421±0.002	0.6776±0.005	0.9321±0.002	0.7843±0.006	0.8732±0.003	0.7657±0.005
CELEBA	CNP	1.0323±0.016	0.7845±0.013	1.0177±0.016	0.7438±0.017	0.8956±0.009	0.7344±0.011	0.7677±0.012	0.6096±0.009
BCNP	1.0452±0.009	0.8015±0.008	1.0275±0.009	0.7726±0.008	0.9351±0.006	0.8376±0.009	0.8015±0.010	0.6816±0.008
SCNP	1.0525±0.008	0.8243±0.006	1.0348±0.008	0.7868±0.006	0.9562±0.004	0.8545±0.008	0.8344±0.006	0.7045±0.005
NP	1.1333±0.004	0.8766±0.005	1.1043±0.015	0.8355±0.015	1.0034±0.008	0.8456±0.006	0.8935±0.009	0.6893±0.006
BNP	1.1732±0.005	0.8901±0.006	1.1378±0.007	0.8678±0.006	1.0411±0.008	0.8711±0.005	0.9256±0.008	0.7671±0.006
SNP	1.1952±0.005	0.9062±0.006	1.1565±0.005	0.8846±0.005	1.0542±0.006	0.8956±0.004	0.9425±0.006	0.7985±0.005
ANP	1.2633±0.002	1.0163±0.004	1.1377±0.004	0.9866±0.006	1.0418±0.004	0.8845±0.006	0.9363±0.004	0.7346±0.008
BANP	1.2751±0.002	1.0389±0.005	1.1488±0.004	1.0155±0.005	1.0602±0.005	0.9255±0.006	0.9489±0.004	0.8415±0.007
SANP	1.2854±0.000	1.0594±0.004	1.1685±0.002	1.0353±0.004	1.0772±0.002	0.9455±0.004	0.9655±0.003	0.8673±0.005
100 state transition pairs are randomly selected for each configuration of the environment, working as the maximum context points to identify the configuration of dynamics.
C.3 Image Completion
Analogous to the 1D experiments, we take random pixels of a given image at training as targets, and select a subset of this as contexts, again choosing the number of contexts and targets randomly  $(n\sim U[3,200],m\sim n + U[0,200 - n])$ . The  $\mathbf{x}_i$  are rescaled to  $[-1,1]$  and the  $y_{i}$  are rescaled to  $[-0.5,0.5]$ . We use a batch size of 16 for both EMNIST and CelebA, i.e. use 16 randomly selected images for each batch.
For image completion experiments on EMNIST and CelebA dataset, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP, SNP, and SANP refer to Appendix B. The number of hidden unites is  $d_h = 128$  and latent representation  $d_z = 128$ . The number of layers are  $l_e = l_{de} = 4$ ,  $l_{la} = l_{qk} = l_v = 5$ .  $h_{head} = 8$
C.4 Uncertainty Measuring
Methods for reasoning under uncertainty are a key building block of accurate and reliable machine learning systems. We further analyze the learned models using the framework introduced in [17] to quantify uncertainty by investigate the calibration error and sharpness of the models. By assuming predictive distribution  $P_{\theta}(y_i|\mathbf{x}_i,\mathcal{D}^{\mathcal{C}})$  as Gaussian distribution  $\mathcal{N}(y_i;\mu_i,\sigma_i^2)$ , we can get the probabilistic forecast  $F_{i}(x_{i})$ . More formally,  $m$  confidence levels  $0\leq p_1 < p_2 < \dots p_m\leq 1$  are choose. For each threshold  $p_j$ , we compute the empirical frequency
$$
\hat {p} _ {j} = \frac {\left| y _ {i} \right| F _ {i} \left(y _ {i}\right) \leq p _ {j} , i \in \mathcal {T} |}{\left| \mathcal {T} \right|} \tag {31}
$$
Table 10: Calibration error and sharpness of the models for 1D regression experiments (Mean ± Std).
Method	RBF	Matern
CAL	SHA	CAL	SHA
CNP	0.0724±0.008	0.0887±0.007	0.0514±0.005	0.0831±0.010
BCNP	0.1015±0.007	0.1151±0.005	0.0724±0.005	0.1152±0.009
SCNP	0.1225±0.005	0.1357±0.006	0.0425±0.005	0.1325±0.008
NP	0.0615±0.004	0.0715±0.005	0.0343±0.015	0.0717±0.015
BNP	0.0871±0.005	0.1052±0.006	0.0325±0.009	0.0715±0.008
SNP	0.1155±0.004	0.1257±0.005	0.0347±0.008	0.0718±0.007
ANP	0.1532±0.002	0.0616±0.004	0.0921±0.004	0.0871±0.006
BANP	0.2353±0.002	0.0689±0.004	0.0752±0.005	0.0741±0.006
SANP	0.2633±0.000	0.0741±0.004	0.0415±0.002	0.0667±0.004
Table 11: Calibration error and sharpness of the models for system identification experiments (Mean ± Std).
Method	CAL	SHA
CNP	0.0872±0.008	0.0415±0.007
BCNP	0.1051±0.005	0.0765±0.006
SCNP	0.1124±0.005	0.0833±0.005
NP	0.0821±0.003	0.0581±0.005
BNP	0.0952±0.003	0.0616±0.005
SNP	0.1001±0.001	0.0668±0.005
ANP	0.0863±0.001	0.0673±0.004
BANP	0.1235±0.001	0.0815±0.005
SANP	0.1431±0.000	0.1094±0.004
In this case, the calibration error is defined as a numerical score describing the quality of forecast calibration:
$$
\operatorname {C A L} \left(F _ {1}, y _ {1}, \dots , F _ {| \mathcal {T} |}, y _ {| \mathcal {T} |}\right) = \sum_ {j = 1} ^ {m} w _ {j} \cdot \left(p _ {j} - \hat {p} _ {j}\right) ^ {2} \tag {32}
$$
here  $w_{j}$  is weight and we set  $w_{j} = 1$
Sharpness is measured by using the variance  $var(F_i) = \sigma_i^2$  of the random variable whose CDF is  $F_i$ . Low-variance predictions are tightly centered around one value. A sharpness score can be defined by
$$
\mathrm {S H A} \left(F _ {1}, \dots , F _ {| \mathcal {T} |}\right) = \frac {1}{| \mathcal {T} |} \sum_ {i \in \mathcal {T}} \sigma_ {i} ^ {2} \tag {33}
$$
We evaluated the CE and sharpness of CNP, NP, ANP, BCNP, NBP, BANP, and corresponding stable versions SCNP, SNP, and SANP trained in the experiments. Table 10, 11, 12 list the calibration error and sharpness score on 1D regression, system identification, and image completion tasks. In several settings, models with our stable solution can achieve better calibration and sharpness but work worse in some settings, such as the calibration error being worse than NP and ANP in terms of 1D regression tasks with Matern. The possible reason is that our method tends to produce conservative credible intervals, so become under-confident or less over-confident in some settings.
Table 12: Calibration error and sharpness of the models for image completion experiments (Mean ± Std).
Method	EMNIST	CELEBA
CAL	SHA	CAL	SHA
CNP	0.0182±0.002	0.0574±0.002	0.0253±0.002	0.0743±0.001
BCNP	0.0415±0.003	0.0716±0.002	0.0412±0.002	0.0981±0.002
SCNP	0.0543±0.001	0.0846±0.002	0.0457±0.002	0.1136±0.002
NP	0.0163±0.001	0.0671±0.002	0.0261±0.001	0.0711±0.001
BNP	0.0352±0.002	0.0815±0.002	0.0463±0.001	0.0981±0.002
SNP	0.0446±0.002	0.0918±0.000	0.0532±0.001	0.1046±0.001
ANP	0.0156±0.002	0.0656±0.002	0.0261±0.004	0.0815±0.006
BANP	0.0412±0.002	0.0871±0.002	0.0513±0.003	0.1125±0.003
SANP	0.0558±0.000	0.0954±0.001	0.0632±0.002	0.1265±0.004
(b) RBF/Target
Figure 5: The Log-likelihood comparisons with different  $K$  on 1D regression task.
(b) MAE
Figure 6: The LL and MAE comparisons with different  $K$  on system identification task.
(a) EMNIST
Figure 7: The LL comparisons with different  $K$  on image completion task.
(b) CELEBA
The key parameter in our stable solution is the number of hard predictive subsets  $K$ . Taking SANP as an example, we investigated the average log-likelihood in terms of different  $K$  on 1D regression task, as shown in Figure 5. We can see that SANP performs better as  $K$  increases, reaches the best value at around  $K = 4$ , and then becomes stable in performance as  $K$  grows larger. As proved in Theorem 5.4 in the main manuscript, optimization on  $\mathcal{D}^{\mathcal{C}}$  and more than one hard predicted subsets of  $\mathcal{D}^{\mathcal{C}}$  can achieve more stable prediction. Similarly, we also conducted experiments to investigate the effect of  $K$  on System identification and image completion task and similar observations can be seen in Figure 6 and 7.