SlowGuess/NeurIPS_2023_OCR huggingface dataset:
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
[2] Olivier Bousquet and André Elisseeff. Algorithmic stability and generalization performance. Advances in Neural Information Processing Systems, pages 196-202, 2001.
[3] Olivier Bousquet and André Elisseeff. Stability and generalization. The Journal of Machine Learning Research, 2:499-526, 2002.
[4] Gregory Cohen, Saeed Afshar, Jonathan Tapson, and Andre Van Schaik. Emmist: Extending mnist to handwritten letters. In 2017 International Joint Conference on Neural Networks (IJCNN), pages 2921-2926. IEEE, 2017.
[5] Alec Farid and Anirudha Majumdar. Generalization bounds for meta-learning via pac-bayes and uniform stability. Advances in Neural Information Processing Systems, 34:2173-2186, 2021.
[6] Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. Advances in Neural Information Processing Systems, 33:8284-8295, 2020.
[7] Yarin Gal, Rowan McAllister, and Carl Edward Rasmussen. Improving pilco with bayesian neural network dynamics models. In Data-Efficient Machine Learning workshop, ICML, volume 4, page 25, 2016.
[8] Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Garnelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint arXiv:1903.11907, 2019.
[9] Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pages 1704-1713. PMLR, 2018.
[10] Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXiv:1807.01622, 2018.
[11] Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. arXiv preprint arXiv:1910.13556, 2019.
[12] Moritz Hardt, Ben Recht, and Yoram Singer. Train faster, generalize better: Stability of stochastic gradient descent. In International Conference on Machine Learning, pages 1225-1234. PMLR, 2016.
[13] Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. arXiv preprint arXiv:2102.08759, 2021.
[14] Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.05761, 2019.
[15] Mingyu Kim, Kyeongryeol Go, and Se-Young Yun. Neural processes with stochastic attention: Paying more attention to the context dataset. In International Conference on Learning Representations, 2022.
A Inductive biases
Here, we revisit some properties, which would help us understand NPs. First, we give a concept of Permutation Invariant Function which is the basic property of stochastic process, e.g., NPs.
Definition A.1. (Permutation Invariant Function) A function  $f(\cdot): \times_{i}^{N} \mathbb{R}^{D} \to \mathbb{R}^{d}$  mapping a set of data points  $\{\mathbf{x}_i\}_{i=1}^N$  is Permutation Invariant Function if
$$
\mathbf {x} = \left[ \mathbf {x} _ {1}, \mathbf {x} _ {2}, \dots , \mathbf {x} _ {N} \right]\rightarrow f = \left[ f _ {1} \left(\mathbf {x} _ {\pi (1: N)}\right), f _ {2} \left(\mathbf {x} _ {\pi (1: N)}\right), \dots , f _ {d} \left(\mathbf {x} _ {\pi (1: N)}\right)\right], \tag {13}
$$
where  $x_{i} \in \mathbb{R}^{D}$  and the function output is a  $d$  dimensional vector. Operation  $\pi : [1,2,\dots ,N] \to [\pi_1,\pi_2,\dots ,\pi_N]$  is a permutation set over the order of elements in the set.
Definition A.2. (Permutation Equivariant Function) A function  $f(\cdot): \times_{i}^{N} \mathbb{R}^{D} \to \mathbb{R}^{N}$  mapping a set of data points  $\{\mathbf{x}_i\}_{i=1}^N$  is Permutation Invariant Function if
$$
\mathbf {X} _ {\pi} = \left[ \mathbf {x} _ {\pi_ {1}}, \mathbf {x} _ {\pi_ {2}}, \dots , \mathbf {x} _ {\pi_ {N}} \right]\rightarrow f _ {\pi} = \pi \circ f \left(\mathbf {x} _ {1: N}\right), \tag {14}
$$
where the function output contains  $N$  elements keeping the order of inputs.
Permutation Equivariant Function keeps the order of elements in the output consistent with that in the input under any permutation operation  $\pi$ . Permutation invariant functions are candidate functions for learning embeddings of a set or other order uncorrelated data structure  $\{\mathbf{x}_i\}_{i=1}^N$ , and the invariant property is easy to be verified. Here, we give a mean operation structure over the output
$$
F \left(\mathbf {X} _ {\pi (1: N)}\right) = \left(\frac {1}{N} \sum_ {i = 1} ^ {N} \phi_ {1} \left(\mathbf {x} _ {i}\right), \frac {1}{N} \sum_ {i = 1} ^ {N} \phi_ {2} \left(\mathbf {x} _ {i}\right), \dots , \frac {1}{N} \sum_ {i = 1} ^ {N} \phi_ {D} \left(\mathbf {x} _ {i}\right)\right) \tag {15}
$$
B Model Architecture
We show the architectural details of the CNP, NP, and ANP models used for the 1D and 2D function regression experiments. The neural process aims to learn a stochastic process (random function) mapping target features  $\mathbf{x}_i$  to prediction  $y_i$  given the context set  $\mathcal{D}^{\mathcal{C}}$  as training data (a realization from the stochastic process), i.e., learning
$$
\log P \left(\mathbf {y} ^ {\mathcal {T}} | \mathbf {X} ^ {\mathcal {T}}, \mathcal {D} ^ {\mathcal {C}}\right) = \log P \left(\mathbf {y} | \mathbf {X}, \mathcal {D} ^ {\mathcal {C}}\right) = \sum_ {i = 1} ^ {N} P \left(y _ {i} | \mathbf {x} _ {i}, \mathcal {D} ^ {\mathcal {C}}\right). \tag {16}
$$
Conditional neural process (CNP) [9] describes  $P(y_{i}|\mathbf{x}_{i},\mathcal{D}^{c})$  with a deterministic neural network taking  $\mathcal{D}^{\mathcal{C}}$  to output the parameters of  $P(y_{i}|\mathbf{x}_{i},\mathcal{D}^{c})$ . CNP consists of an encoder  $f_{\mathrm{enc}}(\cdot)$ , an aggregator  $f_{\mathrm{agg}}(\cdot)$  and a decoder  $f_{\mathrm{dec}}(\cdot)$ ; the encoder summarizes  $\mathcal{D}^{\mathcal{C}}$  and  $\mathbf{x}_i$  into latent representations  $[\mathbf{r}_1,\dots ,\mathbf{r}_{|\mathcal{C}|}]\in \mathbb{R}^{|c| \times d}$  via permutation-invariant neural network [31], where  $d$  is the number of latent dimensions, and aggregator summarizes the encoded context features to a single representation  $\mathbf{r}^{\mathcal{C}}$ , and decoder takes as input the aggregated representations  $\mathbf{r}^{\mathcal{C}}$  and  $\mathbf{x}_i$  and output the single output-specific mean  $\mu_i$  and variance  $\sigma_i^2$  for the corresponding value of  $y_i$ .
$$
\mathbf {r} _ {i} = f _ {\mathrm {e n c}} \left(\mathbf {x} _ {i}, y _ {i}\right), \quad i \in \mathcal {C}
$$
$$
\mathbf {r} ^ {\mathcal {C}} = \frac {1}{| \mathcal {C} |} \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} \tag {17}
$$
$$
\boldsymbol {\phi} = f _ {\mathrm {a g g}} (\mathbf {r} ^ {\mathcal {C}})
$$
$$
\left(\mu_ {i}, \sigma_ {i}\right) = f _ {\mathrm {d e c}} \left(\phi , \mathbf {x} _ {i}\right), \quad p \left(y _ {i} | \mathbf {x} _ {i}, \mathcal {D} ^ {\mathcal {C}}\right) = \mathcal {N} \left(y _ {i}; \mu_ {i}, \sigma_ {i} ^ {2}\right) \quad i \in \mathcal {T}
$$
where  $f_{\mathrm{enc}}(\cdot)$  and  $f_{\mathrm{dec}}(\cdot)$  are feed-forward neural networks. The decoder output  $\mu_i$  and variance  $\sigma_i^2$  are predicted mean and variance. We use Gaussian distribution  $\mathcal{N}(y_i; \mu_i, \sigma_i^2)$  as predictive distribution. CNP is trained to maximize the expected likelihood  $\mathbb{E}_{P(\mathcal{T})}[P(y_i | \mathbf{x}_i, \mathcal{D}^\mathcal{C})]$ .
Neural process [10] further models functional uncertainty using a global latent variable. Unlike CNP, which maps a context into a deterministic representation  $\widetilde{\mathbf{r}}_i$ , NP encoders a context into a Gaussian latent variable  $\mathbf{z}$ , giving additional stochasticity in function construction. Following [14], we consider
an NP with both a deterministic path and latent path, where the deterministic path models the overall skeleton of the function  $\widetilde{\mathbf{r}}_i$ , and the latent path models the functional uncertainty:
$$
\mathbf {r} _ {i} = f _ {\mathrm {e n c}} ^ {(1)} \left(\mathbf {x} _ {i}, y _ {i}\right), \quad i \in \mathcal {C}
$$
$$
\begin{array}{l} \mathbf {r} ^ {\mathcal {C}} = \frac {1}{| \mathcal {C} |} \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} \\ \phi = f _ {\text {a g g}} (\mathbf {r}) \end{array} \tag {18}
$$
$$
\left(\mu_ {z}, \sigma_ {z}\right) = f _ {\mathrm {e n c}} ^ {(2)} \left(\mathcal {D} ^ {\mathcal {C}}\right), \quad q (\mathbf {z} | \mathcal {D} ^ {\mathcal {C}}) = \mathcal {N} \left(\mathbf {z}; \mu_ {z}, \sigma_ {z} ^ {2}\right)
$$
$$
\left(\mu_ {i}, \sigma_ {i}\right) = f _ {\mathrm {d e c}} \left(\phi , \mathbf {z}, \mathbf {x} _ {i}\right), \quad p \left(y _ {i} \mid \mathbf {x} _ {i}, \mathbf {z}, \mathcal {D} ^ {\mathcal {C}}\right) = \mathcal {N} \left(y _ {i}; \mu_ {i}, \sigma_ {i} ^ {2}\right) \quad i \in \mathcal {T}
$$
with  $f_{\mathrm{enc}}^{(1)}(\cdot)$  and  $f_{\mathrm{enc}}^{(2)}(\cdot)$  having the same structure as  $f_{\mathrm{enc}}(\cdot)$  in Eq.(17). In this scenario, the conditional distribution is lower bounded as:
$$
\log P (\mathbf {y} | \mathbf {X}, \mathcal {D} ^ {\mathcal {C}}) \geq \sum_ {i = 1} ^ {N} \mathbb {E} _ {q (\mathbf {z} | \mathcal {D} ^ {\mathcal {C}})} \left[ \log \frac {P (y _ {i} | \mathbf {x} _ {i} , \mathbf {z} , \mathcal {D} ^ {\mathcal {C}}) P (\mathbf {z} | \mathcal {D} ^ {\mathcal {C}})}{q (\mathbf {z} | \mathbf {X} , \mathbf {y})} \right]. \tag {19}
$$
We further approximate  $q(\mathbf{z}|\mathcal{D}^{\mathcal{C}}) \approx P(\mathbf{z}|\mathcal{D}^{\mathcal{C}})$  and train the model by maximizing this expected lower bound over tasks. Furthermore, ANP introduces attention mechanisms into NP to resolve the issue of under-fitting.
The architectural details of the CNP, NP, and ANP are the same as in [14]. Here we give the detailed architectures of the encoder and decoder of NPs.
B.1 Encoder without attention
Encoder focuses on learning embeddings for each data point in the context set, and the basic component is multi-layer perceptron, which is defined by
$$
\operatorname {M L P} \left(l, d _ {i n}, d _ {h}, d _ {o u t}\right) = \operatorname {L I N E A R} \left(d _ {h}, d _ {o u t}\right) \circ \underbrace {\left(\operatorname {R E L U} \circ \operatorname {L I N E A R} \left(d _ {h} , d _ {h}\right) \circ \cdots\right)} _ {\times (l - 2)} \circ \operatorname {L I N E A R} \left(d _ {h}, d _ {i n}\right) \tag {20}
$$
where  $l$  is the number of layers,  $d_{in}$ ,  $d_h$  and  $d_{out}$  are dimensionalities of inputs, hidden unites and outputs. Here  $\mathrm{RELU}(\cdot)$  is adapted as activation function.
The encoder in Vanilla CNP uses a deterministic encoder which focuses on learning embeddings for each data point in context set.
$$
\mathbf {r} _ {i} = \operatorname {M L P} \left(l _ {e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right),
$$
$$
\mathbf {r} ^ {\mathcal {C}} = \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i}, \quad \phi = \operatorname {M L P} \left(l _ {e 2}, d _ {h}, d _ {h}\right) \left(\mathbf {r} ^ {\mathcal {C}}\right) \tag {21}
$$
where  $d_{x}$  and  $d_{y}$  are the dimensionalities of  $\mathbf{x}_i$  and  $y_{i}$ .
To follow the encoder structure in NP, we introduce another encoder aligned with original deterministic encoder to permit the same number parameters, i.e.,
$$
\mathbf {r} _ {i} ^ {(1)} = \mathrm {M L P} (l _ {e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}) ([ \mathbf {x} _ {i}, y _ {i} ])
$$
$$
\mathbf {r} _ {\mathcal {C}} ^ {(1)} = \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} ^ {(1)}, \quad \phi_ {1} = \operatorname {M L P} \left(l _ {e 2}, d _ {h}, d _ {h}\right) \left(\mathbf {r} _ {\mathcal {C}} ^ {(1)}\right)
$$
$$
\mathbf {r} _ {i} ^ {(2)} = \operatorname {M L P} \left(l _ {e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right) \tag {22}
$$
$$
\mathbf {r} _ {\mathcal {C}} ^ {(2)} = \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} ^ {(2)}, \quad \phi_ {2} = \mathrm {M L P} (l _ {e 2}, d _ {h}, d _ {h}) (\mathbf {r} _ {\mathcal {C}} ^ {(2)})
$$
$$
\phi = [ \phi_ {1}, \phi_ {2} ]
$$
The encoder in NP contains a deterministic path and a latent path, i.e.,
$$
\mathbf {r} _ {i} ^ {(1)} = \operatorname {M L P} \left(l _ {d e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right)
$$
$$
\mathbf {r} _ {\mathcal {C}} ^ {(1)} = \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} ^ {(1)}, \quad \phi = \operatorname {M L P} \left(l _ {d e 2}, d _ {h}, d _ {h}\right) \left(\mathbf {r} _ {\mathcal {C}} ^ {(1)}\right)
$$
$$
\mathbf {r} _ {i} ^ {(2)} = \operatorname {M L P} \left(l _ {l a 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right) \tag {23}
$$
$$
\mathbf {r} _ {\mathcal {C}} ^ {(2)} = \sum_ {i \in \mathcal {C}} \mathbf {r} _ {i} ^ {(2)}, \qquad [ \mu_ {z}, \sigma_ {z} ^ {\prime} ] = \mathrm {M L P} (l _ {l a 2}, d _ {h}, d _ {h}) (\mathbf {r} _ {\mathcal {C}} ^ {(2)})
$$
$$
\sigma_ {z} = 0. 1 + 0. 9 \cdot \operatorname {S I G M O I D} \left(\sigma_ {z} ^ {\prime}\right), \quad \mathbf {z} \sim \mathcal {N} \left(\mu_ {z}, \operatorname {d i a g} \left(\sigma_ {z} ^ {2}\right)\right).
$$
In this case, the encoder outputs deterministic representation  $\phi$  and latent representation  $\mathbf{z}$ .
B.2 Encoder with attention
The attention mechanism is widely used in NPs, Specifically, multi-head attention [26] is adapted, which is defined by
$$
\mathbf {Q} ^ {\prime} = \left\{\text {L I N E A R} \left(d _ {q}, d _ {\text {o u t}}\right) (\mathbf {q}) \right\} _ {\mathbf {q} \in \mathbf {Q}}, \quad \left\{\mathbf {Q} _ {i} ^ {\prime} \right\} _ {i = 1} ^ {n _ {\text {h e a d}}} = \text {S P L I T} \left(\mathbf {Q} ^ {\prime}, n _ {\text {h e a d}}\right),
$$
$$
\mathbf {K} ^ {\prime} = \left\{\text {L I N E A R} (d _ {k}, d _ {o u t}) (\mathbf {k}) \right\} _ {\mathbf {k} \in \mathbf {K}}, \quad \left\{\mathbf {K} _ {i} ^ {\prime} \right\} _ {i = 1} ^ {n _ {h e a d}} = \text {S P L I T} (\mathbf {K} ^ {\prime}, n _ {h e a d}),
$$
$$
\mathbf {V} ^ {\prime} = \left\{\text {L I N E A R} (d _ {v}, d _ {o u t}) (\mathbf {v}) \right\} _ {\mathbf {v} \in \mathbf {V}}, \quad \left\{\mathbf {V} _ {i j} ^ {\prime} \right\} _ {i = 1} ^ {n _ {h e a d}} = \text {S P L I T} (\mathbf {V} ^ {\prime}, n _ {h e a d}),
$$
$$
\mathbf {H} _ {i} = \operatorname {S O F T M A X} \left(\mathbf {Q} _ {i} ^ {\prime} \left(\mathbf {K} _ {i} ^ {\prime}\right) ^ {\top} / \sqrt {d _ {o u t}}\right) \mathbf {V} _ {i} ^ {\prime}, \quad \mathbf {H} = \operatorname {C O N C A T} \left(\left\{\mathbf {H} _ {i} \right\} _ {i = 1} ^ {n _ {h e a d}}\right)
$$
$$
\mathbf {H} ^ {\prime} = \text {L A Y E R N O R M} (\mathbf {Q} ^ {\prime} + \mathbf {H})
$$
$$
\mathrm {M H A} \left(d _ {\text {o u t}}\right) (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \text {L A Y E R N O R M} \left(\mathbf {H} ^ {\prime} + \text {R E L U} \left(\text {L I N E A R} \left(d _ {\text {o u t}}, d _ {\text {o u t}}\right)\right)\right) \tag {24}
$$
where  $d_q, d_v, d_k$  are the dimensionalities of query  $\mathbf{Q}$ , key  $\mathbf{K}$ , and value  $\mathbf{V}$ , respectively.  $n_{head}$  is the number of head. Here Layer normalization [1] LAYERNORM( $\cdot$ ) is adapted. It is easy to derive self-attention by setting  $\mathbf{Q} = \mathbf{K} = \mathbf{V}$ , i.e.,
$$
\mathrm {S A} \left(d _ {\text {o u t}}\right) (\mathbf {X}) = \mathrm {M H A} \left(d _ {\text {o u t}}\right) (\mathbf {X}, \mathbf {X}, \mathbf {X}) \tag {25}
$$
For CNP, the encoder with attention still contains two deterministic paths,
$$
f _ {q k} = \operatorname {M L P} \left(l _ {q k}, d _ {x}, d _ {h}, d _ {h}\right)
$$
$$
\mathbf {Q} = f _ {q k} (\mathbf {x} _ {i}), \quad i \in \mathcal {T}
$$
$$
\mathbf {K} = \left\{f _ {q k} (\mathbf {x} _ {i}) \right\}, \quad i \in \mathcal {C}
$$
$$
\mathbf {V} = \operatorname {S A} \left(d _ {h}\right). \left(\left\{\operatorname {M L P} \left(l _ {v}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right) \right\} _ {i \in \mathcal {C}}\right)
$$
$$
\phi_ {1} = \mathrm {M H A} \left(d _ {h}\right) (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {26}
$$
$$
\mathbf {H} = \mathrm {S A} (d _ {h}) \left(\{\mathrm {R E L U} \circ \mathrm {M L P} \left(l _ {e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) ([ \mathbf {x} _ {i}, y _ {i} ]) \} _ {i \in \mathcal {C}}\right)
$$
$$
\phi_ {2} = \operatorname {M L P} \left(l _ {e}, d _ {h}, d _ {h}\right) \left(\frac {1}{| \mathcal {C} |} \sum_ {i \in \mathcal {C}} \mathbf {h} _ {i}\right)
$$
Similarly, encoder with attention in NP contains a deterministic path and a latent path, i.e.,
$$
f _ {q k} = \mathrm {M L P} \left(l _ {q k}, d _ {x}, d _ {h}, d _ {h}\right)
$$
$$
\mathbf {K} = \left\{f _ {q k} \left(\mathbf {x} _ {i}\right) \right\}, \quad i \in \mathcal {C} \tag {27}
$$
$$
\phi = \mathrm {M H A} (d _ {h}) (\mathbf {Q}, \mathbf {K}, \mathbf {V})
$$
$$
\mathbf {H} = \operatorname {S A} \left(d _ {h}\right) \left(\left\{\operatorname {R E L U} \circ \operatorname {M L P} \left(l _ {e 1}, d _ {x} + d _ {y}, d _ {h}, d _ {h}\right) \left(\left[ \mathbf {x} _ {i}, y _ {i} \right]\right) \right\} _ {i \in \mathcal {C}}\right)
$$
$$
\left[ \mu_ {z}, \sigma_ {z} ^ {\prime} \right] = \operatorname {M L P} \left(l _ {l a}, d _ {h}, d _ {h}\right) \left(\frac {1}{| \mathcal {C} |} \sum_ {i \in \mathcal {C}} \mathbf {h} _ {i}\right) \tag {28}
$$
$$
\sigma_ {z} = 0. 1 + 0. 9 \cdot \operatorname {S I G M O I D} \left(\sigma_ {z} ^ {\prime}\right),
$$
$$
\mathbf {z} \sim \mathcal {N} \left(\mu_ {z}, \operatorname {d i a g} \left(\sigma_ {z} ^ {2}\right)\right).
$$
B.3 Decoder
The decoder focuses on predicting output for target points based on the encoder's outputs  $\phi$ . For target point  $\{\mathbf{x}_i\}_{i\in \mathcal{T}}$ , the decoder of CNP is defined by
$$
[ \mu_ {i}, \sigma_ {i} ^ {\prime} ] = \operatorname {M L P} \left(d _ {d e c}, 2 d _ {h} + d _ {x}, d _ {h}, 2 d _ {y}\right) [ \phi , \mathbf {x} _ {i} ], \quad i \in \mathcal {T}
$$
$$
\sigma_ {i} = 0. 1 + 0. 9 \cdot \text {S O F T P L U S} \left(\sigma_ {i} ^ {\prime}\right) \tag {29}
$$
$$
y _ {i} \sim \mathcal {N} (\mu_ {i}, \sigma_ {i})
$$
Decoder of NP is defined by
$$
[ \mu_ {i}, \sigma_ {i} ^ {\prime} ] = \operatorname {M L P} (d _ {d e c}, d _ {h} + d _ {z} + d _ {x}, d _ {h}, 2 d _ {y}) [ \phi , \mathbf {x} _ {i}, \mathbf {z} ], \quad i \in \mathcal {T}
$$
$$
\sigma_ {i} = 0. 1 + 0. 9 \cdot \text {S O F T P L U S} \left(\sigma_ {i} ^ {\prime}\right) \tag {30}
$$
C Implementation Details and Experiments
For CNP [9], BCNP [18], and our SCNP, we apply the encoder with attention described in Eq (26) and decoder described in Eq (29). For NP [10], ANP [14], BNP [18], BANP [18] and our SNP and SANP models, we apply encoder with attention described in Eq (27) and (28), and decoder described in Eq (30).
C.1 1D Regression
For synthetic 1D regression experiments, the neural architectures for CNP, NP, ANP, BCNP, BNP, BANP, and our SCNP/SNP/SANP refer to Appendix B. The number of hidden units is  $d_h = 128$  and latent representation  $d_z = 128$ . The number of layers are  $l_e = l_{de} = l_{la} = l_{qk} = l_v = 2$ .
We generate datasets for synthetic 1D regression. Specifically, the stochastic process (SP) initializes with a 0 mean Gaussian Process  $(\mathrm{GP})y^{(0)}\sim GP(0,k(\cdot ,\cdot))$  indexed in the interval  $x\in [-2.0,2.0]$ , where the radial basis function kernel  $k(x,x^{\prime}) = \sigma^{2}\exp (-\| x - x^{\prime}\|^{2} / 2l^{2})$  with  $s\sim U(0.1,1.0)$  and  $\sigma \sim U(0.1,0.6)$ . Furthermore, GP with Matern Kernel is adopted for model-data mismatch scenario, which is defined as  $k(x,x^{\prime}) = \sigma^{2}(1 + \sqrt{5} d / l + 5d^{2} / (3l^{2}))\exp (-\sqrt{5} d / l)$  and  $d = \| x - x^{\prime}\|$  with  $s\sim U(0.1,1.0)$  and  $\sigma \sim U(0.1,0.6)$ . For a fair comparison, we set the same data generation, training, and testing for all models.
We trained all models for 100,000 steps with each step computing updates with a batch containing 100 tasks. We used the Adam optimizer with an initial learning rate  $5 \cdot 10^{-4}$  and decayed the learning rate using Cosine annealing scheme for baselines. For SCNP/SNP/SANP, we set  $K = 3$ . The size of the context  $\mathcal{C}$  was drawn as  $|\mathcal{C}| \sim U(3,200)$ . Testings were done for 3,000 batches with each batch containing 16 tasks (48,000 tasks in total).
We investigate the model stability from the size of the context set and the percent of added noise. First, we conduct experiments on different size of context set, i.e.,  $|\mathcal{C}| \in \{20, 50, 100, 200\}$ . Table 7 shows the Average Log-likelihoods performance comparison between different methods in terms of different context size. We can see that the performance becomes better with the increasing of  $|\mathcal{C}|$  and NPs with stable solution still achieve better performance. Second, we investigate the model performance in terms of different noise setting. Here we introduce Gaussian noise  $\bar{\mathcal{N}}(0,1)$  and add noise to different proportions of the data, such as  $\{0\%, 5\%, 10\%, 15\% \}$ . Table 8 lists the Average Log-likelihoods performance comparison in terms of different noise proportions.
Table 7: Average Log-likelihoods over all context and target points on realizations from Synthetic Stochastic Process on different size of context set. (Mean ± Std).
Kernel	Method	20	50	100	200
context	target	context	target	context	target	context	target
RBF	CNP	0.8724±0.008	0.4334±0.007	0.9533±0.005	0.4854±0.010	1.1224±0.005	0.5322±0.007	0.1563±0.004	0.5783±0.004
BCNP	0.9015±0.009	0.4579±0.007	0.9787±0.007	0.5215±0.008	1.1687±0.007	0.5716±0.009	0.1985±0.005	0.6086±0.005
SCNP	0.9255±0.008	0.4733±0.004	0.9944±0.005	0.5433±0.006	1.1833±0.006	0.5918±0.005	0.2111±0.004	0.6333±0.004
NP	0.8215±0.004	0.3853±0.005	0.9124±0.006	0.4234±0.003	1.0855±0.003	0.4767±0.005	1.1225±0.002	0.5233±0.004
BNP	0.8714±0.004	0.4122±0.004	0.9712±0.004	0.4718±0.004	1.1426±0.005	0.5269±0.006	1.1716±0.004	0.5698±0.004
SNP	0.8955±0.003	0.4356±0.004	0.9866±0.004	0.4934±0.005	1.1637±0.004	0.5434±0.004	1.1958±0.003	0.5933±0.003
ANP	1.2563±0.002	0.5763±0.004	1.3233±0.002	0.6322±0.003	1.4633±0.003	0.6866±0.006	1.4982±0.006	0.7322±0.004
BANP	1.2715±0.003	0.5878±0.005	1.3325±0.004	0.6465±0.004	1.4778±0.003	0.6915±0.004	1.5115±0.005	0.7436±0.005
SANP	1.2831±0.000	0.5994±0.004	1.3452±0.002	0.6577±0.003	1.4898±0.001	0.7043±0.002	1.5285±0.002	0.7534±0.005
Matern	CNP	0.8531±0.005	0.2431±0.010	0.9123±0.005	0.2984±0.003	1.0522±0.004	0.3542±0.002	1.0984±0.008	0.4022±0.005
BCNP	0.8765±0.006	0.2788±0.009	0.9411±0.006	0.3266±0.005	1.0752±0.004	0.3762±0.004	1.1245±0.007	0.4326±0.006
SCNP	0.8963±0.003	0.2953±0.006	0.9555±0.004	0.3467±0.003	1.0967±0.003	0.3967±0.003	1.1467±0.008	0.4556±0.005
NP	0.7643±0.015	0.2041±0.015	0.8221±0.002	0.2547±0.003	0.9322±0.003	0.3155±0.004	1.0452±0.005	0.3563±0.005
BNP	0.8052±0.008	0.2651±0.007	0.8678±0.003	0.3163±0.004	0.9672±0.003	0.3656±0.005	1.1052±0.005	0.4015±0.005
SNP	0.8368±0.006	0.2844±0.005	0.8956±0.002	0.3326±0.004	0.9959±0.003	0.3849±0.004	1.1215±0.003	0.4313±0.004
ANP	1.2421±0.002	0.6366±0.004	1.3022±0.001	0.6881±0.004	1.4211±0.004	0.7331±0.003	1.4631±0.002	0.7753±0.008
BANP	1.3452±0.007	0.6513±0.004	1.4056±0.003	0.7015±0.004	1.4505±0.004	0.7531±0.005	1.4986±0.004	0.7996±0.006
SANP	1.3721±0.002	0.6653±0.004	1.4322±0.003	0.7126±0.004	1.4633±0.004	0.7644±0.003	1.5153±0.005	0.8125±0.004