Neural Processes with Stochastic Attention: Paying more attention to the context dataset
Authors: Mingyu Kim
Kyeong Ryeol Go
Se-Young Yun
Venue: ICLR 2022 Poster (2022)
Keywords: neural processes
stochastic attention
variational inference
information theory

Abstract: Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. NPs essentially leverage a given dataset as a context representation to derive a suitable identifier for a novel task. To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant. In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information. From the perspective of information theory, we demonstrate that the proposed method encourages context embedding to be differentiated from a target dataset, allowing NPs to consider features in a target dataset and context embedding independently. We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings. We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regression, predator-prey model, and image completion. Moreover, the proposed method is also validated by MovieLens-10k dataset, a real-world problem.

Published as a conference paper at ICLR 2022
NEURAL PROCESSES WITH STOCHASTIC ATTENTION: PAYING MORE ATTENTION TO THE CONTEXT DATASET
Mingyu Kim, Kyeongryeol Go, Se-Young Yun
KAIST AI
(cal (call lingu, kyeongryeol go, yunseyoung @kai Stkaist.aa kr.
ABSTRACT
Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. NPs essentially leverage given dataset context representation to derive suitable identifier for novel task. To improve the prediction accuracy, many variants of NPs have investigated context embedding approaches that generally design novel network architectures and aggregation functions satisfying permutation invariant. In this work, we propose a stochastic attention mechanism for NPs to capture appropriate context information. From the perspective of information theory. we demonstrate that the proposed method encourages context embedding to be differentiated from target dataset, allowing NPs to consider features in a target dataset and context embedding independently. We observe that the proposed method can appropriately capture context embedding even under noisy data sets and restricted task distributions, where typical NPs suffer from a lack of context embeddings. We empirically show that our approach substantially outperforms conventional NPs in various domains through 1D regres- sion, predator-pre prey model, and image completion. Moreover, the proposed method is also validated by MovieLens- 10k dataset, a real-world problem.
INTRODUCTION
Neural processes (NPs) have been in the spotlight as they stochastically complete unseen target points considering given context dataset without huge inference computation (Garnelo et al. 2018a b Kim al. 2019). NPs leverage neural networks to derive an identifier suitable for a novel task using context representation, which contains information about given context data points. These methods enable us handle considerable amounts of data points, such as in image-based applications, that the Gaussian process cannot naively deal with. Many studies have revealed that the prediction performance relies on the way of context representation (Kim et al. 2019 Volpp et al., 202T). The variants of NPs have mainly investigated novel network urchitectures and aggregation functions that are permutation invariant. For example, Garnelo al. (2018a b) developed NPs using MIPP layers embed context via mean aggregation. Bayesian aggregation satisfying permutation invariance enhanced the prediction accuracy by varying weights for individual context data points (Volpp
context embedding approaches that generally design
2021). From the prospective of network architectures to increase model capacity, Kim et al. (2019) suggested the way of local representation to consider relations between context dataset and target dataset using deterministic attention mechanism (Vaswani et al.) 2017). For robustness under noisy situations, bootstrapping method was proposed to orthogonally apply to variants of NPs (Lee al. 2020).
Despite many appealing approaches. one significant drawback of previous NPs is that they still underfit when confronted with noisy situations like real-world problems. This manifests as inaccurate predictions at the locations of the context set as seen in Figure Ta Additionally, in FFigure 1b) the attentive neural process (Kim et al. 2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset. In the case of the Bootstrapping ANP (Lee et al., 2020) and ANP with information dropout, the quality of heat-map is slightly improved, but still falls short of ours. This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training.

---

Published as a conference paper at ICLR 2022
ANP
AMP wis
Doars
Ous
ANP 4:8 9VG, Dopert
Recanage ANP
(a) ID regression predcitions
(b) Heatmaps of asscociated attention weights
Figure 1: Comparison of ID regression predictions and asscociated attention weights as specified by ANP (Kim et al., 2019), ANP with Information dropout, Bootstrapping ANP (Lee et al. 2020) and ours. The training data for ID regression is fairly noisy. (a) Ours more accurately captures the context datasets and significantly better predicts than baselines. (b) The horizontal axis indicates the value of features in the context dataset, while the vertical axis indicates the value of features in the target dataset these heatmaps. The best pattern for this heat-map diagonal because all feature values arranged in ascending order. Among all models, ours comes closet to the ideal. The detailed analysis is shown in Appendix
To address this issue, we propose newly designed neural process to fundamentally improve perfor- mance by paying more attention to the context dataset. The proposed method expedites a stochastic attention to adequately capture the dependency of the context and target datasets by adjusting stochas- ticity. It results in improving prediction performance due to maintained context information. We observe that our proposed algorithm works well by utilizing contextual information in an intended manner as seen in Figure This method outperforms current NPs and their regularization methods in all experiment settings. Thus, this paper clarifies the proposed method as regularized NPs in terms of information theory, explaining that the stochastic attention encourages the context embedding to be differentiated from the target dataset. This differentiated information induces NPs to become appropriate identifiers for target dataset by paying more attention to context representation. To summarize, we make the following contributions:
We propose the novel neural process that pay more attention to the context dataset. We claims for the first time, using the information theory framework, that critical conditions for contextual embeddings in NPs are independent of target features and close contextual datasets.
Through comprehensive analyses, we illustrate how stochastic local embeddings are crucial for NPs to focus on capturing the dependencies of context and target datasets. Even when context dataset contains noise or is somewhat different from target dataset, as shown in Lec) et (2020), the proposed method is capable of adapting novel task while preserving pre- dictive performance. Particularly, this method significantly enhances performance without additional architectures and data augmentation compared to the attentive neural process (Kim et al. 2019).
The experimental results show that the proposed model substantially outperforms conven- tional NPs in typical meta- regression problems. For instance, the proposed method achieves to obtain the state of the art score in the image completion task with the CelebA dataset. Es- pecially, the proposed model maintains performance the limited task distribution regimes such as the MovieLenz-100 dataset with small number of users.
This paper is organized as follows. We introduce background knowledge in section 2 presents a neural process with stochastic attention and related works illustrated in secsection 5 Experi- mental results are shown in section 4, concluding with section 6
section 3
BACKGROUND
2.1 NEURAL PROCESSES
Suppose that we have an observation set X {2.}]=1 and label set Y {u H NPs (Garneoo jet al. 20188 B Kim et al. 2019) are designed to obtain the probabilistic mapping from the observation set to the label set p( X Ya) given small subset (Xe: Y.) = (æt, My )= Basically, it is built
2

---

Published as a conference paper at ICLR 2022
upon the neural network with an encoder-decode architecture where the encoder fo outputs task representation by feed dorwardnwg (Xe. Y2) through
set encoding (Zaheer et al. 2017; Edwards & Storkey 2017) and the decoder fo models the distribution of Y (e.g Gaussian case estimating μ4, a) using along with the encoder outputs. Its objective is to maximize the log likelihood over the (unknown) task distribution p(7). All tasks provided by the data generating process are considered as Monte Carlo samples, respectively:
Σ Temp(T)
1og = Σ
Σ log (usluus TT>P(T)
where (ui, so(os,r); r= [=(((,)))))E)
The set encoding architecture differentiates the type of NPs. In Conditional neural process(CNP)) (Gar nelo et al. 2018a), the representation r is a deterministic variable such that, by using mean aggregation, the encoder maps the context set into single deterministic representation r = 1/m Σド= 1 S (z yi Neural process (NP) (Garnelo al.) 2018b) introduces probabilistic latent variable 2 to model functional uncertainty as a stochastic process such that the parameters of output distribution may change according the sampled value of Due to the intractable log-likelihood, a training objective is derived based on variational inference which can be decomposed into two terms, reconstruction term and regularization term:
log Po(Y| (11XX,,=)p00 s)po(//X, Y.)
logp(Y|XX, Ye)= 2 (x
= -=,
[log pa rr,,), (L(go(z|X, pp (x)
For simplicity, the prior distribution pa [= Yo) is approximated by qo Xe, Y.). However, as pointed out in Kim et al. 2019), the mean aggregation over the context set is too restrictive to describe the dependencies of the set elements. To enhance the expressiveness of the task representation, Attentive neural process(ANP) accommodates an attention mechanism (Vaswani et al.j (017)) into the encoder, which generates the local deterministic representation ri corresponding to target data point T; and addresses the underfitting issue in NP. Although additional set encoding methods such as the kernel method and the bayesian aggregation considering task information have been suggested (Xu et al. 22200; Volpp et al. [222I) the attentive neural process is mainly considered as the baseline in terms of model versatility.
2.2 BAYESIAN ATTENTION MODULE
Consider m key-value pairs, packed into a key matrix K E RRmxds and a value matrix V E RRmxd and л queries packed into Q E gnxdd, where the dimensions of the queries and keys are the same. Attention mechanisms aim to create the appropriate values O E Rnxd, corresponding to Q based on the similarity metric K, which are typically computed via an alignment score function g such that +- s(2, K). Then, softmax function is applied to allow the attention weight W E Raxm to satisfy the simplex constraint so that the output features can be obtained by O,,, W Vy:
W7,
Σ M ンラ= exp(
Note that there are many options for the alignment score function g where a scaled dot-product or a neural network widely used.
Bayesian attention module (Fan et al. 2020) considers a stochastic attention weight W. Compared
other stochastic attention methods Shankaar & Sarawagi 2018 Lawson al. 2018 Bahuleyan
et al. 2018 Deng et al. 2018), it requires minimal modification to the deterministic attention mechanism described above so that compatible with the existing frameworks, which can be adopted in a straightforward manner. Specifically, un-normalized attention weights W such that
Wi, = W/ W., are sampled via the variational distribution qo (WIQ, K), which can trained via amortized variational inference:
(Ogg( K, V) 2 E (W. logpo(O(1 w>] KL(gg (WIQ, K)llpa())))
(4)
3

---

Published as a conference paper at ICLR 2022
Considering the random variable W to be non-negative such that it can satisfy the simplex constraint by normalization, the variational distribution do (WIQ, K) set to Weibull A) and the prior distri- bution Pa (W) set to Gamma(o, B). This can be obtained by the standard attention mechanism, on the other hands, can be either a learnable parameter or hyper-parameter depending on that the model follows key-based contextual prior described the following paragraph. The remaining variables & and 3 are regarded user-defined
peper--armetterss By introducing constant y (Zhang et al. 2018 Bauckhage 2014). the KL divergence in [Equation 4 4can be computed in analytical expression. The detailed derivation is shown in AAppenixx A
7/ log (AT 1
KL (t) .))
a log 3 (oggg (5)
k
Samples from Weibull distribution can be obtained using a reparameterization trick exploiting an inverse CDF method: X log(1 =))+ (ei X) are computed as XT + 1/4) and X? [T(1+ 2/(c) (T + 1/s)]]. It can be observed that the variance of obtained samples decreases as increases.
Uniform(0, 1). Note that mean and variance of
Key-based contextual prior To model the prior distribution of attention weights, key-based con- textual prior was proposed. This method allows the neural network to calculate the shape parameter of the gamma distribution a prior distribution. This leads to the stabilization of KL divergence between standard attention weights and sampled attention weights, and it prevents overfitting of the attention weights (Fan al. 2020). In this paper, we explain the reason why the key-based context prior is important for capturing an appropriate representation focusing on a context dataset from the prosepective of information theory.
NEURAL PROCESSES WITH STOCHASTIC ATTENTION
We expect NPs to appropriately represent the context dataset to pre- dict target data points in a novel task. However, if the context dataset has limited task distribution and noise like real-world problems, conventional NPs tend to sensitively react to this noise and maximize the objective function including irreducible noise. These irreducible noises do not completely correlate with context information, so that this phenomenon derives meaningless set encoding of the context dataset in training phase and hinders adaptation to new tasks. In other words, the output of NPs po Po z, r:) does not depend on (z,rs). To preserve the quality of context representation, we pro- pose a method that better utilizes the context dataset by exploiting stochastic attention with the key-based contextual prior to NPs. We show that the proposed method enables to create more precise con- text encoding by adjusting stochasticity, even in very restricted task distribution regimes.
jEC
X;
iET
Figure 2: Probabilistic graph- ical models for the proposed method
3.1 GENERATIVE PROCESS
As with the Attentive neural process{King Kim et al.) 2019), the pro- posed method consists of the two types of encoders and a single
decoder architecture. The first encoder embeds (Xe, Y.) to a global representation z and the sec- ond encoder makes local representation by (æi, Xa, Y.). The global representation serves to represent entire context data points (Xe, Yc), whereas the local representation ry is in charge of fine- grained information between target data 2;, and context (Xe, Y.) for predicting the output distribution My (2)r.).. Unlike the ANP, the proposed model considers all intermediate representations (z, rs) stochastic variables. We exploit the Bayesian attention module to create local representa- tion r; by the stochastic attention weights w; with the key-based contextual prior (Fan et al. (020). To use the reparameterization trick for sampling stochastic variables, draw random noise samples el ~ Uni 1),e2 ~ A 1). First, as mentioned in section 2, the stochastic attention weights {w,) N are obtained via the inverse CDF of the Weibull distribution with random noise c1. For amortized variational inference, the prior distribution go (.V ( is also derived with context
4

---

Published as a conference paper at ICLR 2022
dataset X. for implementing key- contextual prior. Meanwhile, e2 is used to obtain the global representation sampled from normal distribution. The entire scheme is shown in Fngure D.2c As shown in sessctiion 2] we can calculate all the KL divergences of 2 and {w) The decoder follows the standard neural processes p(ya [2,, 2, rz) = N (y (& (Ts, 2, nx), o oz r;)). See Appendix for implementation details.
closed-form solutions.
3.2 LEARNING AND INFERENCE
Unlike the Attentive neural process Kim al. 2019). the proposed method regards all representations and W = {we ]=1 as stochastic variables, so that we clearly drive the objective function according to amortized variational inference. Based on the objective function of Neural process et al.. 2018b), the proposed method adds KL divergence of stochastic attention weight W {w;)
Assuming the independence between and { such that individual u, is only dependent on x; the objective function for each task Ta presented as follows:
and (Xes Ya) as seen in Figure 2
LT. (ò, Σl po (1s (1:(la Xx)lgo(w;|(x)))
(6)
Note that (X, Y) follows task To and each task Ty is drawn from task distribution p(7). The final objective function is L( 0) = Eyg [Cr, (ò. 8)]. From the perspective of amortized variational inference, the prior distributions of p(z) and p((,;)( follow the standard neural processes wherein p(z) is defined as (=(=1(Xe,) Y.} (Garnelo et al. 201 8b).
should be defined. With regard to
In the case of {w:)..r we introduce the strategy of Bayesian attention modules (Fan et al. 2020
The prior distribution of [w:) defined as a key-based contextual prior, go(w;] Xe), not only stabilizes the KL divergence but also activates the representation w; to pay more attention to the context dataset. In the next section, this objective function can be described in terms of information theory as role of regularization to pursue the original goal of NPs, which is deriving appropriate identifiers for target datasets in novel tasks.
3.3 DISCUSSION OF OBJECTIVE FUNCTION FROM VIEWS OF INFORMATION THEORY
A novel part of the proposed method is the use of the stochastic attention mechanism to more leverage context information, thereby stably capturing the dependency context and larget datasets noisy and very limited task distributions. In this subsection, based on information theory, we elaborate the goal of NPs and our novel stochastic attention mechanism explained as the regularization of latent variables to pay more attention to context datasets. This enables us to understand latent variables and their requirements semantically. According to sobsbecctio 3.2, the objective function is categorized into two terms: reconstruction log po (yrl [44,, 2, r;) and regularization as two KL Divergences. Suppose that {x,, yi a data point in a target dataset, D is a given context dataset, latent variables z and r; are considered Z, which is defined as the information bottleneck for D, suggest that maximizing a reconstruction term corresponds to increase Tyy D p);)) and minimizing two KL. Divergence means to decrease T(Z, x; D).
First, the mutual information of the target value y) and the context dataset given the input feature Ji, I (y,, D|a,), is the metric identify that NPs is adapted to novel task. Suppose p(ys (z.,, D) output value of NPs and p(y, (z.) ED [( (r, D)] is output value not conditioned context dataset D. NPs completely fail to adapt new tasks using the context dataset D. the suggested metric Tyy, D( goes to O.
Po(uila,, D) (o (xx)
Dla:) Ey,DD log
Assumed that target data point (r) is sampled from Th. which uncontrollable data generating
process, the reconstruction term Σ ((ra))) log p(v (xs, Z) can be regarded to [yy:: Z|2;) based on information bottleneck theorem; T(y;; D(are) T(y:: 7 7/x))) holds. The detailed explanation is given
5

---

Published as a conference paper at ICLR 2022
in Appendix C The objective function should be designed to increase [(; D/z;) by appropriate identifier for novel task considering a context dataset D. However, as mentioned in the previous sections, NPs suffer from irreducible noises and limited task distribution in meta training. This phenomena means that NPs increase the objective function by learning the way to directly map from 3; yi including noises and some features only in meta-training, instead of taking into consideration of the context D. To make latent variables Z capture proper dependencies, we define the regularization accounting for latent variables pay more attention to the context dataset.
This regularization is the mutual information of latent variable Z and input 2; given context dataset D to indicates how similar information latent variable and target z; contain. We regard p(Z [z D) as the distribution of latent variables that considers both context dataset D and target 2;, meanwhile p(Z] D) = F [p(Z|ara m. D)] as the distribution of latent variables only depending on the context dataset 'D. If latent variables Z have totally different information against the target z; given the
context dataset D, can be I x. (D) =
p(Z|rr D)
I(Z, I = Ez,z; log p(Z|D)
We bring (b; P|z.) 1(2,,; 2, D) to be maximized instead of (1,, D|z;) so as to model adequately dependency between context and the target dataset. By decreasing I(Z, (D) and increasing [y,, D[æ;) in meta-training, we expect that the model learns the way to differentiate and x; and construct identifier which can further consider Z and together. Note that ø is the parameter of the function to transform D to the information bottleneck Z and 0 is the parameter of the identifier for target data point y/, we show that [y,;; D|a;) I(Z, xr (D) Z Is,0 Z(x,)) (2 x (D( if 6, 0 is perfectly able to leverage all information about Z and IG. We assume that neural network architectures can perform intended manners.
Theorem 1.
Let Z be the representation of the context datasel 'D and it follows an information bottleneck. The following equation holds when the latent variable Z can be split into {z, w,} and z is only dependent on D, but wi is dependent D and T's
LT 0) S [y:: D(v,) - I(Z,x:|D)
(9)
where w; is obtained by the Bayesian attention modules (Fan et al. 2020, and L.T, is defined in Equation Gand its probabilistic graphical model follool Figure 2
As mentioned, if the neural network architectures d, are able to completely represent to Z and
Y: given D and Fg. Lr. (ó. 8) T (ys: Z| I& xZ; (D) ideally holds. In this subsection, we reveal that stochastic attention mechanism satisfy on this regularization as mentioned. p( (Z 24, D) is regarded as 4o (w:la, (Xe, Ye)) and p(Z[D) is the key-based contextual prior 4o (w: K ((.)).
Therefore, the stochastic attention mechanism with key-prior that requires D and a; as well as being able to this regularization for latent variables to pay more attention to the context dataset in Equation &
is considered. By Theorem I, the network parameters á and 0 are updated by the gradient of the objective function in Equation 6) semantically improves [1; ((zz)) present the experimental results in the next section. They indicate that we can implement a model that can complete unseen data points in novel tasks by fully considering the context representations. The detailed analysis is presented in Appendix G and Appendix H
1(Z, x; (D). Therefore,
4 EXPERIMENT
In this section, we describe our experiments to answer three key questions: 1) Are existing regulariza- tion methods such as weight decay, importance- ELBO as well as recent methods(Informan dropout, Bootstrapping, and Bayesian aggregation) effective to properly create context represen- tations ?: 2) Can the proposed method reliably capture dependencies between target and context datasets even under noisy situations and limited task distributions ?; 3) Is it possible improve performance via the appropriate representation of the dataset? ?
To demonstrate the need for neural process with stochastic attention, we choose CNP, NP (Garnelo let al.] 2018a]b). and ANP (Kim et al. 2019) as baselines, which are commonly used. For all baselines, we employ weight decay and importance ELBO that requires several samples against one
6

---

Published as a conference paper at ICLR 2022
Table Results of likelihood values on the synthetic ID regression experiment. The test samples are drawn from GPs with various kernel functions; All methods are trained on the samples from the RBF GP function. To consider noisy setting, we artificially generate a periodic noise in the training step.
Bold entries indicates the best results.
RBF kernel GP(noties)
REF keruel GP
Matern kermel GP
Periodic kernel GP
contect
larget
contccal
larget
coutect
target
conteca
target
CNP
0.233=00036
4447800054
0:440=00013
0,026+00014
0.246+00211
-0.544-00124
07664000022
-0.97=00:03
NP
-0015
-0 0069020059
00
-0.177
0.2355_0.52
-0.918=00129
0.400=00046
-1.321
ANP
0,228=0:021
-0.00500036
0,000000010
-0.097000224
0,254+0.006
-0.488-0/014
0.111+0034
-0.51+00000
(Weight doeay;
0.001)
CNP
0.24010.041
-0471100077
0.460100.15
0.04500:0000
0.27810.035
004472_00041
081510012
4.970000007
NP
-0.15320003
-0.79900070
0.122
-0.178:00006
-0.288-00054
-0.926-01126
-0.401=0:047
-1.315=00.77
ANP
0.957
-0,M210000
1.050100011
0.05310.004
1,01110,099
-0,209_.0.036
0.926.10007
-0,644L.0,027
(luportacce Weighard ELRO:
CNP
0.271
-0.44200011
0.478=0012
0.05940:029
0.321#00019
-0.460=00066
0.21
4.957=00:04
NP
-0.155+-0014
-0.63+00026
0.067+0015
-0.13-00.00
-0:28-00.015
-0.779-00188
-0.30000008
-1.09440.088
ANP
0.771=0.012
-0.470=0054
0.89540.016
-0.031=0.026
0.800+0.031
00.324-0.027
0 0.704+0.266
4.687=0.099
(Bayesion Aggregation)
CNP
0.351200049
-0 00.50810084
0.57510016
0.113:1:005
0.34910.018
-0.569-0.007
0.27110018
-1.02610:051
NP
00406130055
-0723320007
0,201110:04
-0.359-0.024
-0.150_00001
-0.837-000036
-0.537=00011
-0.877-0.021
(Functional representation)
ComeCNP
1.314=0,00
-0.48=a00ss
1.32640000
0.084+0.000
1.31+00.077
.0.119-0.023
1.35++0000
4.502-n1.22
ConvN
0.87320.154
-0.49=00041
0.729
0,052a0017
0.832=00005
-0,163=00015
0.953:40077
40.52=00:25
(Regolariztion for local representatio)))
ANP (dropaut)
0.158=0.019
00599+0005
0.372=00026
-0.163-0.025
0.136++0.000
-0:575-
0.01400017
4.877--0mm6
Boststapping ANP
0.754_0.018
-0.40710005
0.872.10.009
0.04310:003
0.788.10010
-0,303 0.032
0.711100225
-0.717.0.003
Ours
1.374:0.084
-0.337::00027
1.363:0.10
0,244::0026
1.365:00,09
-0,175:00031
1.372:00005
40.612:00009
data point and aggregated by log Σ, exp Lt. Recent regularization methods can be also considered. Information dropout(Kingma et al. 2013 Achille & Soatto| 2018), Bayesian aggregation (Volpp) itt al. 2021 I and Bootstrapping (Lee et al. 2020) are chosen. Bayesian aggregation is used for CNP and NP according the original paper, meanwhile Information dropout and Bootstrapping are used for the ANP to verify regularization of local representations. When training, all methods follow their own learninig policy; however, for fair comparison, all methods perform one Monte- Carlo sampling across a test data point during testing. We respectively conduct 5 trials for each experiment and report average scores. As a metric to evaluate the model's 's predictive performance, we use the likelihood of the target dataset, while the likelihood of the context dataset is considered as metric to measure the extent to which models can represent a context dataset. Since these metrics are proportional to model performance, a higher value indicates better performance. In these experiments, we show that the proposed method substantially outperforms various NPs and their regularization methods.
4.1 1D REGRESSION UNDER NOISY SITUATIONS
We conduct synthetic ID regression experiments test whether our model adequately captures the dependency of dataset under noisy situation. We trained all models with samples from the RBF GP functions. We tested them with samples from various kernel functions (Matern, Periodic inclduing RBF) identify that models is capable of adaptatation to new tasks. To consider noisy environments, we artificially generate noises into the training dataset. Our model maintains outstanding performance compared to other baselines. In particular, when the test data comes from periodic kernel GP, which is the most different from the RBF GP, the proposed model utilizes context datasets appropriately regardless of the test data sources, whereas other models do not properly use the context data. When comparing the likelihood values of context datasets, the proposed method preserves equivalent scores in all cases, but the others do not. As shown in Figuree Ia] the proposed method captures context points better than other methods, as only it method correctly captures the relationship even under noisy situations. These results support our claim, the proposed method utilizes context datasets in prediction. We report graphical explanation and the additional experimental result of 1D regression without noises in Apppendix E
4.2 PREDATOR-PREY MODEL AND IMAGE COMPLETION
We apply NPs to domain shift problem called the predator-pres -prttorr-peyyyoo model, proposed in the Con- vCNP(Gooroon et al., 2019). In this experiment, we train models with the simulated data and applied models predict the real-world data set, Hudson's bay the hare-lynx. Note that the hare-lynx dataset
7

---

Published as a conference paper at ICLR 2022
Table 2: Results of likelihood values on the Sim2Real experiment using predator-prey simulation and image completion with the CelebA dataset. In the image completion, there are several experimental cases; The number of context points in {50, 100, 300, 500}. We report that likelihood of context points is averaged over all experiment cases and elaborate individual likelihood values of target points. Bold entries indicates the best results.
Sim2Real Predator-Prey
Image Completion CelebA
Simulation
Real
Target
context
target
context
farget
50
100
300
500
CNP
0.395=0.000
0.274+0.000
2.6455=0.085
3.120::0.245
3.452++0.002
2.662
3.077
3.359
3,414
NP
-0.259+0.004
00.369=00002
-25566+0.090
-28816+00.08
3.072++0.05
2,483
2.786
2.999
3.042
ANP
1.211=0.001
0.961=40:00
-1.756:10.187
3.742-10:448
3.100::0.012
2.492
2.806
3.02
3.06
(Weight decuy;
0.001)
CNP
0.393-0.000
0.265+00000
-.666
3.049+00:06
2.969+0.0014
2.21
2.6
2,866
2,918
-0.0692000033
-01185=0.001
-2.571
-2.914+0.108
2.429::0.008
1.878
2.17
2.368
2.406
ANP
1.520=0.001
1.180:0.002
0.21240.129
-2.807=0.665
2.832:10.025
2.339
2.611
2.795
2,829
(mportancce Weighted ELBOO s
CNP
0.476-0.000
0.337+0000
-2.596+0.04K
13.038+0.185
3.474+0.022
2.67
3.091
3.379
3.436
NP
0.055=00002
0.066==0.002
-2.51420.048
-2.760 107
3.179:10.025
2.49
2.856
3.112
3.163
ANP
1.004=0.001
0.775110001
-1.800=10.203
-3.728:10446
3.13111:002
2.477
2.826
3.057
3,101
(Bavesion Aggregation)
CNP
0.551=0000
0.429:000000
12.654=00116
-2.952+0.112
3.583++0.013
2.733
3.18
3.474
3.53
NP
0,332=0.003
0. 0.192:0.005
-2.489:00.065
-3.00910:115
3.483:10.032
2.529
3,072
3.4
3.46
(Functional representation)
ConyCNP
2.509_0.000
2.13911:000
L.75810.089
+0,43110.457
ConyNP
2.467=00.000
2.124:+0.000
1.65240.0099
-1.180::0:439
(Regularization for local representation)
ANP (dropous)
1.492=0.003
1.144 0.004
1.068:100099
-6.215:1.577
3.078:10.06
2.501
2.801
3.005
3.043
Bootstrapping ANP
2.537=0.001
2.166:: 0.001
2.45110.0019
13.38241.288
3.172:40.009
2.453
2.837
3.095
3.145
Ours
2.711=0.000
2.297:+0.00
2.429:00.011
-1,766:00885
4.119::0.000
2.653
3.21
3.787
3.948
is very noisy, unlike the simulated data. The detailed explanation of datasets is written in Appendix H Recent regularization methods such as Bayesian aggregation, Information dropout, and Bootstrapping are effective to improve performance. In particular, Bootstrapping method significantly influences predictive performance compared of other ANPs as mentioned in the original paper(Lee et al. 2020). However, we observe that the proposed method is superior to the other models in both simulated test set and real-world dataset. In particular, all baselines drastically decrease the likelihood values due to noises during the test, while the proposed model preserves the performance because it is robust to noises. See the left side of Table 2) We report the additional experimental result with periodic noises and the graphical explanation is presented in Appendix F
Second, we conduct the image completion task which the models generate images under some given pixels. The experiment setting follows the previous experiments (Garnelo et al. 2018a b Kim et al. 2019 Lee et al.) 2020). To conduct fair comparison, all models employ only MLP encoders and decoders and the multi-head cross attention is used for local representations; the variant of ANPs. We indicate that the proposed method records the hest score as seen in the right side of Table 2 The best baseline records 3.53 with 500 context pixels in Table 2] whereas
method attains 2.653 of likelihood even with 50 context pixels and grows to 3.948 with 500 context pixels. Referred to Lec al. (2020)'s s paper, the previous highest score is 450 of likelihood for context pixels and 1.9 of likelihood for target pixels. As Kim et al. (2019) mentioned that the more complex architecture like the self-attention mechanism enhances the completion quality than the MLP encoders, these scores were obtained by ANP with self-attention encoders and employing bootstrapping. However, we achieves to obtain comparable results for context points and exceed the previous highest score for target points by significant margins without exhausted computations and complicated architectures. The completed images by ours and basclines are shown in Appendix J
4.3 MOVIBLENS- 100K DATA
We demonstrate the robustness and effectiveness of our method using a real-world dataset, the Movie- Lenz dataset, which is commonly used in recommendation systems. This setting has expensive data collection process that restricts the number of users. In this section, we report how well the proposed method can be generalized novel tasks using limited tasks during the meta-training. To
8

---

Published as a conference paper at ICLR 2022
train NPs, we decide to split this dataset according to user ID and regard the rating samples made by one user as a task. The purpose of this experiment is that NP provides an identifier to serve for new visitor using very few rating samples. We follow the setting used in the existing work et al. 2019). The proposed model performs better than the other methods. As mentioned in subseccttoo 4.2 captures the information from the context dataset, while other methods suffer from noise in the data and lack of users in meta-training. The experiment result of comaprison with baselines is reported in [Appendix I To validate use of real applications, we compare with existing studies. According to Movielens- -00k benchmarks, the state-of-the-arg RMSE score is about 0.890, which can be obtained using graph structures about users and films(Rahed et al. 2019). For fair comparison, we train and test the proposed method on the same setting, named as UI splits. When evaluate baselines on the ul test dataset, we randomly draw samples of corresponding users from the ul. base used in the meta-training and regard as context data points. Although we do not use graph structures, as seen in Table 3 The ANP points the comparable result of 0.909, meanwhile the proposed model attains promis-
Table 3: Comparison of RMSE scores on u.test in Movel 100k
RMSE
GLocal-K (Han et al. 2021
0.890
GraphRec + Feat (Rashed et al.
2019
0.897
GraphRec Rashed el al. 2019
0,904
GC-MC + Vreat[Berg cl al. 201
0,905
GC-MC (Berg ci al. 2017
0,910
ANP (contexts: 10m et al
2019
0.909
Ours (contexts: 10)
0.895
ing result, 0.895 of the RMSE value. This experiment indicates that the proposed method can reliably adapt to new tasks even provides small histories, and we identify again that our model can properly work on noisy situations.
5 RELATED WORKS
The stochastic attention mechanism enables the capturing of complicated dependencies and reg- ularizing weights based on the user's prior knowledge. However, such methods cannot utilize back-propagee because they do not consider the reparameterization trick to draw samples (Shankar & Sarawagil 2018 Lawson et al. 2018 Bahuleyan et al., 2018: Deng et al., 2018). Even if these posterior bustiobution of latent variables, satisfying the
methods employ normal distribution simplex constraints, which sum to one (Bahuleyan et al. 2018), is impossible. Recently, the Bayesian attention module suggests that the attention weights are samples from the Weibull distribution whose parameters can be reparameterized, so this method can be stable for training and maintaining good scalability (Fan et al. 2020).
Since the Conditional neural process have been proposed et al., 2018a b). several studies have been conducted to improve the neural processes in various aspects. The Attentive neural process modify the set encoder as an cross-attention mechanism to increase the performance of the predictability and interpretability (Kim al., 2019). Some studies investigate NPs for sequential data (Yonn et al. 2020: Singh al. 2019). Trials have been combined with optimization-basee meta- learning to increase the performance NPs for the image classification task with a pre-trained image encoder (Rusu et al. 2018 Xu al. 2020). The convolutional network can be used for set encoding to obtain translation ialaaatiiooninnnanttt predictions with the assistance of supplementary data points (Gordon) let al. 2019 Foong al. 2020). Similar to this study, the Bayesian context aggregation suggests the importance of context information over tasks et al. (021) and the Bootstrapping NPs orthogonally improves the predictive performance under the model-data mismatch. (Lee et al. 2020). Unfortunately, there is no clear explanation of how stochasticity has helped improve performance and does not show a fundamental approach to enhancing dependency between the context and the target dataset in terms of set encoding.
6 CONCLUSION
In this work, we propose an algorithm, neural processes with stochastic attention that effectively leverages context datasets and adequately completes unseen data points. We utilize a stochastic attention mechanism to capture the relationship between the context and target dataset and adjust stochasticity during the training phase for making the model insensitive to noises. We demonstrate that the proposed method can be explained based information theory. The proposedregullarizatoon

---

Published as a conference paper at ICLR 2022
leads representations paying attention to the context dataset. We conducted various experiments to validate consistent enhancement of the proposed model. We identify that the proposed method substantially outperforms the conventional NPs and their recent regularization methods substantial margins. This evidence from this study suggests that the proposed method can provides a better identifier to the novel tasks that the model has not experienced.
ACKNOWLEDGMENTS
This work conducted by Center for Applied Research in Artificial Intelligence (CARAI) grant funded by DAPA and ADD [UDI9003IRD].
REPRODUCIBILITY STATEMENT
Our code is available at https: wttp:h/ww.cmmm./roooooonn/////// reproducibility, both training and evaluation codes are included.
For convenience
REFERENCES
Alessandro Achille and Stefano Soatto. Information dropout: Learning optimal representations through noisy computation. IEEE transactions on pattern analysis and machine intelligence, 40 (12):2897-2905, 2018.
Hareesh Bahuleyan, Lili Mou, Olga Vechtomova, and Pascal Poupart. Variational attention for sequence-to-sequeecee models. In Proceedings of the 27th International Conference on Compula- fional Linguistics, pp. 1672-1682, 2018.
Christian Bauckhage. Computing the kullback-leibler divergence between two generalized gamma distributions. arXiv preprint arXiv: 1Xiv::400.6653. 2014.
Rianne van den Berg, Thomas N Kipf, and Max Welling. Graph convolutional matrix completion. arXiv preprint Ttl:::700..2000 2017.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. In ICLR (Poster), 2016.
Yuntian Deng, Yoon Kim, Justin Chiu, Demi Guo, and Alexander Rush. Latent alignment and variational attention. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran As- sociates. Inc., 2018. URL httptsssss//pprcccdiggg wewpeiiiinnnn.euugunee neurips httppppss.c.cpeee//////LLL 06513340010000000144444777788883
pdf
Harrison Edwards and Amos Storkey. Towards neural statistician. In International Conference on Learning Representations, 2017.
Xinjie Fan, Shujian Zhang. Bo Chen, and Mingyuan Zhou. Bayesian attention modules. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 16362-16376. Curran Asso- ciates, Inc., 2020. URL TRLLtttps://proceednne neurips. cc pctppappr/222//E...ee Paper.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural pro- cesses. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 2284-8299 Curran Asso- ciates, Inc., 2020. URL htttpss .wwrips cC/ maaper/20200 file/ 5dE0385cba256al. htw.ae255aaaabbbeeeeeeeeeeeeeee
pdf
Alexandre Galashov, Jonathan Schwarz, Hyunjik Kim, Marta Gamelo, David Saxton, Pushmeet Kohli, SM Eslami, and Yee Whye Teh. Meta-learning surrogate models for sequential decision making. arXiv preprint arXiv: 1903. /907, 2019.
10

---

Published as a conference paper at ICLR 2022
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International Conference Machine Learning, pp. 1 1704-1713 PMLR, 2018a
Marta Gamell,, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint 1rXivv1877.01663 2018b.
Jonathan Gordon, Wessel P Bruinsma, Andrew YK Foong, James Requeima, Yann Dubois, and Richard E Turner. Convolutional conditional neural processes. In International Conference on Learning Representations, 2019.
Soyeon Caren Han, Taejun Lim, Siqu Long, Bernd Burgstaller, and Josiah Poon. Glocal-k: Global and local kernels for recommender systems. arXiv preprint irXiv:2108.12188 2021.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Tch. Attentive neural processes. In International Conference on Learning Representations, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
Durk P Kingma, Tim Salimans, and Max Welling. Variational dropout and the local reparameterization trick. In Advances neural information processing systems, pp. 2575-2583, 2015.
Dieterich Lawson, Chung-Cheng Chiu, George Tucker, Colin Raffel, Kevin Swersky, and Navdeep Jaitly. Learning hard alignments with variational inference. In 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5799-5803. IEEE, 2018.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang, Sung Ju Hwang, and Yee Whye Tch. Bootstrap- ping neural processes. In Advances in neural information processing systems, 2020.
Janarthanan Rajendran, Alexander Irpan, and Eric Jang. Meta-learning requires meta-augmennation. In Advances in Neural Information Processing Systems, volume 33, 2020.
Ahmed Rashed, Josif Grabocka, and Lars Schmidt-Thieme Attribute-wware non-linear co- embeddings of graph features. In Proceedings of the 13th ACM Conference on Recommender Systems, pp. 314-321, 2019.
Andrei Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osindero, and Raia Hadsell. Meta-learning with latent embedding optimization. In International Conference on Learning Representations, 2018.
Shiv Shankar and Sunita Sarawagi. Posterior attention models for sequence to sequence learning. In International Conference on Learning Representations, 2018.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural pro- cesses. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019, URL htttps://proceeedingg htwp://eeeeeein....nne.n..e neurips /ttppp/pp/1//1lllll 741774441 755.9988667777777010070377777713
pdf
Edney W Stacy et al. A generalization of the gamma distribution. The Annals of mathematical statistics, 33(3):
1962.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Michael Volpp. Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. In International Conference Learning Representations, 2021.
Darren J Wilkinson. Stochastic modelling for systems biology. Chapman and Hal//CRC, 2018
11

---

Published as a conference paper at ICLR 2022
Jin Xu, Cran-Francois Ton, Hyunjik Kim, Adam Kosiorek, and Yee Whye Teh. Metafun: Meta- learning with iterative functional updates. In International Conference on Machine Learning, pp- 10617-10627. PMLR, 2020.
Jaesik Yoon, Gautam Singh, and Sungjin Ahn. Robustifying sequential neural processes. In International Conference on Machine Learning, pp. 10861-10870. PMLR, 2020.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. Advances in Neural Information Processing Systems, 2017.
Hao Zhang, Bo Chen, Dandan Guo, and Mingyuan Zhou. Whai: Weibull bybrid autoencoding inference for deep topic modeling. In International Conference Learning Representations, 2018.
12

---

Published as a conference paper at ICLR 2022
A CLOSED FORM SOLUTION FOR THE KL DIVERGENCE BETWEEN WEIBULL AND GAMMA DISTRIBUTION
The generalized gamma distribution contains both Weibull and gamma distribution special cases. We are concerned with the three-parameter version of generalized gamma distribution introduced in Stacy (Stacy et al. 1962). Its parameters can be categorized into one scale parameter and two shape parameters d and Its probability density function is defined for I E [0, oo) and given by
p xd-1 a (ddpp)
[(x d. p)
exp (1)
(A.1)
where r()) is the gamma function, and parameters a, d, > For d = p. it corresponds to the Weibull distribution, and if p 1, it becomes the gamma distribution. Bauckhage (Bauckhage (2014) derives a closed form solution for the kullback-leibler divergence between two generalized gamma distribution follow.
kl((1152) = 1 11(x/11,11,11 d1 log
0O
f d1, d1.P1) dx f2 (22 d2) P2)
P1ah-I
(d11/p))
-log
P2af" T(d1/p))
+
+ log
(di -d)
(A.2)
p1
r(d P1
t2)
01
P2
d1
+
d1
02
P1
where f1, /2 are the generalized gamma distributions, and 1p(-) is digamma function. The Weiull distribution with scale parameter À and shape parameter K coincides with the generalized gamma distribution À and d, p K. For gamma distribution with scale parameter 3 and shape parameter a, it becomes the generalized gamma distribution = 1/2, d = a and p = The KL divergence between the Weibull distribution (A, k) and the gamma distribution (a,, 3) amounts to
(L(filfe) = loog a log 3 log (a)) a
(A.3)
+ +1)
(1) a log A + (81(1+1/44
ks
We introduce (11) = -y 20 0.5772 as Euler constant. Finally, we obtain Equation 5
KL(Weibull(k; / Gamma B)) = 7a
a log A + log k + 3AT
1 1
k
To
(A.4)
- - - 1 11-alog log 3 + ((a)
13

---

Published as a conference paper at ICLR 2022
B ELBO DERIVATIONS
In this section, we derive the objective function in the manuscript of this paper. Without loss of generality, target dataset {X, Y = {z,, M/))== and context dataset such that N M. Let the log-likelihood of target data points in task Ta be log p( X, (,, We begin to maximizing the oogliikcihood of target data points based on context dataset and representations. We suppose that follows the graphical model in Figure D.2c
(Xer Yo) far Vis
logp(Y (X, - log / p(Y, DOY,2IX,)JC
(B.5)
2X D) (Z)X, d2
log
(B.6)
()
p(Y, 9(2 X. D. ((((X,D)
log
(B.7)
p(y1 IT, Zi)
log
II i=1
2(zz D) (2, D)dz
(B.8)
(( 24, D)
P(U Ze)
log gx)
(Zz D) p(zz D)]
(B.9)
Applying Jensen's inequality to
p(ya [r,, Z:)p(Z:l (7.| D) <(26 Ir,, D)
log logp(Y 2 Eg(a))
log
(B.10)
1=1
q(Z, [r,, D)
2 By(a)
log po (2 Z,) llog
(B.11)
p( (Z 2, D)
From Z, = { in Figure D.2c
w/
we define = dependent on context dataset and w; is dependent
target v; and context dataset D
Σ log (o Z,) log (wu: (z;, D) p(w;
log 7(= D) p(2 (D)
log (og((Y)),)) 2 Eg(e)
(B.12)
We assume that p(w; [r; D) Toh, ((x)) by regularization of latent variables to pay attention on context dataset. It also follows the strategy the standard neural processes: posterior distribution of z is qoa (wilk (X,, (.),Y))) and prior distribution of is Jos [X,, Thereby, we show that ELBO become the objective function the proposed method Equation 6
4dh, (wy D)
Los (= (x,, Yil)
log pY X D) Bay())
log po [r, 74) llog
log yos (=
L..
(wi), K(X.))
(B.13)
14

---

Published as a conference paper at ICLR 2022
C PROOF OF THEOREM
We present that Ty;: xra) has relation to the objective function of the proposed method. Suppose that Z is information bottleneck for D and all variables follow the graphical models Figure 2 As described in [quation 7, the mutual information [1,, D(æ;) is to measure dependency of y and context dataset D. We should maximize this information generalize the novel task. For simplicity, we set all latent variables as information bottleneck Z for the context dataset D. Note that Z = ([w,)]=11 2) is information bottleneck corresponding to context representations in NPs, where W = (w),..., Wn } is an attention weights and X, Y {(21,,)) vi)...., (In, yn)] is respectively data points in the target data set. & is regarded as the parameter of the function transform D to the information bottleneck Z and / is the parameter of the identifier for target data point y,. By information bottleneck theorem. [Equation C.14 holds.
I (y:: D (x)) 2 (u: Z(x,) = Ho. (y) 2) H (y)xx))
(C.14)
The target data point (xe, } is drawn by the data generating process To ~ p(T), so that Hy) can be regarded as a constant value due to uncontrollable factors. As a result, -H H Z) is written as
- -Ho,0(yi 2) = E(z.o) [ log ( 2)]
(C.15)
We employ E(z.ge log log (yi xi Z which is unbiased estimate for - Ho.ooollee 2) based on Monte Carlo sampling. Given that the concrete information bottleneck Z is given, we expect that the Z allows to generalize a novel task {zra; yi by enhancing E(z.,.)) log (y: [z; Zo). However, in practice, Z induced by neural networks is more likely to memorize the training dataset including irreducible noises when the number of tasks is insufficient. The conventional methods accomplished maximizing E(z.gg) log po po(y; [ Zz) by simply finding the relation between y/8 and x; only in the training dataset. It means that the information in representation becomes increasingly identical to the information in in the meta training dataset. It does not satisfy the condition of information bottleneck for the context dataset D.
To avoid this issue, we introduce I(Z, zz (D) as a regularization. By reducing dependencies of Z and Hy, we can make the information bottleneck focus on D. means that as p( [x,, D) and P(Z) D) get closer, the target input v; less influences Z, but D has high correlation with Z. In other words, minimizing I(Z, xz (D) is to make Z and target z; more independent given context dataset D. encourages latent variables Z (wi, 2) to pay more attention to context dataset D
I(Z,x;1D) Ezz [EEz [KL 2 |. (9.,
(C.16)
=Ez, E... [LL u. (w D) (l [D)] + E, (L((()) Dilla()D))
where, the g(w, I2, D) follows the Weibull distribution and g(w,|D) follows the gamma distribution. In this study, q(u; (2, D) can be modeled as the stochastic attention weights and q (w; can be modeled as the key-based contextual prior qa (). We can factorize all latent variables because the attention weight w; does not have a dependency on data points except for the specific data point {æ,, J. We denote that wi has conditional independence over (w, Jizy given 24. For the global rep- resentation; 2, to follow objective function of neural processes, we assume q q D) = [ Y) mentioned in ssection 3 we suppose that the global representation follows normal distribution. Therefore, we derive the regularization term 1(2, xze (D) have relation to KL Divergence terms in our loss function. From this fact, we recognize KL Divergence terms in our objective function helps latent variables give attention on context dataset.
and (x) {>=[D).
To summarize the mutual information between y; and D, and the newly designed regularization, we found that
[y D(r.))
( 2(z;)) - I(Z,,,,1))
(C.17)
2 - 1o 1; Z) (2, x, I + Constant
15

---

Published as a conference paper at ICLR 2022
Based on the graphical models in Figure 2 and assumptions of representations 2 and wa we identify that this equation has relation to our objective function LT.
(yo Z|zs) - ((Z =
Σ
log Zω) E. XXa Y.))
{
Ew [ X, (
Constant
=
CT. + Constant
(C.18)
Finally, we can derive Theorem
as below
Cr. (o. 0) [y:: I(Z,,;
(C.19)
From Equation C.19 the gradients of Φ and with respect to LT. can be regarded as the direction of increasing [gs;; D( I(Z, x; (D), where, Lr, defined as the target likelihood and KL divergence of z and H= in single task.
N
(o,00) =
Σ (e (L(ss(w,/z; ( (w:l(())
(C.20)
where Xp is input features in the context dataset and (s XD) follows key-based contextual prior as described in (eccion 3
16

---

Published as a conference paper at ICLR 2022
D IMPLEMENT DETAILS
We referred to most of the architectures from the paper(Kimi et al. 2019) and their released source code"] The information dropout and importance weighted ELBO were respectively borrowed from
these (xperrsKingma et al. 2015 Burda et al. 2016 38 The stochastic attention can be implemented based on Bayesian attention modules(Fan et al. 2020 We migrated and revised all codes to meet our purpose. In this chapter, we follow the notation of this paper (Lee et al. 2020).
D.1 (ATTENTIVE) NEURAL PROCESS
MLP Encoder We suppose that multi-layer modules(MLP) have the structure as Equation D.21 where is the number of layers, di is the dimension of input features, dhidden is the dimension of hidden units and dout is the dimension of outputs.
MLP(1, din dhidden; dous) Linear (dnidten; clour)
(ReLU Linear( Linear(dniddeen (haddeen) ...)
(D.21)
1-2
(ReLU o Linear(din, dhhitten))
The variants of neural processes have two types of MLP encoders: The deterministic path is used in the conditional neural process Garnelo et al. 8a), and the stochastic path is employed in the neural process Garnelo et al. 2018b). attentive neural process(Kim et al. 2019) and ours. The deterministic path is to aggregate all hidden units by the MLP encoder.
h
Σ MLP((pre d₂ + d,y dn d, (,),,,))1)))) jEC
(D.22)
r= MLP((ppst: dn, dh)
Instead, the stochastic path is to aggregate all hidden units and then feed-forward a single network to generate Mor and de.. We obtain the stochastic path r via the reparameterization trick.
1 Σ MLP((ppc: de + dy. dr, dr) (1z,, vyl)
h
C
JEC
(D.23)
Pb Goi MI. dr, dp (H)
r Por + e2 0
Attention encoder We introduce cross-altention describe the dependency of context and target dataset. Let MHA be a multi-head attention (Vaswani et al. 2017) computed as follows:
Q' = [Linear dg. d, (a)
K' = [Linear (d,, (d)))))())))EE
V' = (Linear (d,, dr. (x))
(D.24)
H = (ottmax('KK///aa,
MHA(Q, K, V) LayerNorm(H)
Where, {dy. d,) are respectively the dimensions of query, key and value components. The LayerNorm is the layer normalization in terms of heads.
MLP Decoder The architecture of MLP decoder is similar to the stochastic encoder. In case of CNP(Gameloo et al., (0188) and NP(Gamnll al. 2018b), this decoder transforms all representation and the input feature (x JeT to target distribution, the normal distribution, {14, σ, Her.
17

---

Published as a conference paper at ICLR 2022
(uy σy) MLP(s dg + dn dn, 2dz) ri, rz,
(D.25)
Meanwhile, in case of attentive neural process (Kim et al. 2019) and ours, the inputs of this decoder are the global representation z, local representation (ri heT and the input feature {or Jer.
(py oy) = MMPe( d2 + dr + dn. dn, (2,) x;l)
(D.26)
The detailed information of each architecture described Table D.1
Table D.I: Architecture details and hyperparameters for the neural processes. Attention indicates variants of attentive neural processes used. Encoder and decoder indicate the MLP network sizes used.
Eacaler
Exceder Dpooder
Woggi
isspples
Kep, fuaciino
Informatoo deopoit
Stockanic dlanso
UNP
3x:125
3-1155
(1001 ISe MQ2)
U/WAA)
NP
3x:125
3- 122
0.101
(/BAA)
(U,MET)
Leyrs
Li:lb
RBP knnce KBD konee
(U,NET)
12 leyes
Ufer SimmHell 27" U/WAE)
Melb-bcab
CJM2 Aer__N__
Bonzzagnn ANP
3x:128
Malb-bzab
Oue
Mali-tcab 8
Tille Coenl (Rooeddd 011 Supple rarrery
Table D.2: The number of network parameters required for all experimental cases.
Models
regression
Sim2Real
MovieLenz-100K
Image Completion
CNP
99,842
100,228
110,850
1,583,110
NP
116,354
116.740
127,362
1,845,766
CNP BA
116,354
116,740
127,362
1,845,766
NP BA
116,354
116,740
127,362
1,845,766
ConvCNP
50,612
50,655
N/A
N/A
ConvNP
51.156
51,199
N/A
N/A
ANP
595,714
596.228
617,730
9,466.886
ANP (dropout)
628,610
629,124
650,626
9,991,686
Bootstrapping ANP
628,610
629,124
650,626
9,991,686
Ours
597,015
597,529
619,031
9,472,027
Table D.3: Time required for inference in all experiment cases. (1 epoch).
Models
ID regression
Sim2Real
MovieLenzz-10k
Image Completion CelebA
50
100
300
500
CNP
0.860
0.896
0.829
1.274
1.321
1.390
1,390
NP
1.411
1.422
1,356
1.975
1.990
1.959
2.081
CNP BA
1.143
1.215
2.149
1.629
1.644
2.091
2.221
NP BA
1,930
2,003
3.198
3.074
3.106
3.709
4.017
ConvCNP
2,788
4.128
N/A
N/A
N/A
N/A
N/A
ConvNP
2,907
4.289
N/A
N/A
N/A
N/A
N/A
ANP
2,442
2,500
2.222
5.294
5.710
9.010
11.740
ANP (dropout)
3.283
3.328
2.967
5.961
6.376
9.926
12.710
Bootstrapping ANP
6.474
6.968
6,250
26.186
30.012
60.619
88.332
Ours
3.152
5.848
3.121
11.350
19.217
57.898
95.031
Unit second
In this paper, we set the dimension of all latent variables as 128, namely d, = 128. The number of heads in multi-head attention is 8, which is the same as the original paper of attentive neural processes (Kim et al. 2019). For all models and all experiments, we use the Adam Bhimizer(Kingman & Ba 2015) with the learning rate 0.001, and set the number of update steps 200000.
18

---

Published as a conference paper at ICLR 2022
For training and evalation, used AMD Ryzen 2950X(16-cores), RAM 64GB and RTX2080Ti. The ID regressions has a batch size of and total of 160 batches. The Sim2Real configures the batch size to be 10 and the total number of batches to be 16. The MovieLenz- 100k has a batch size of 1 and the number of batches to be 459., The image completion task consists of 227 batches and its size is 4.
D.2 MODEL ARCHITECTURE
We graphically show the architecture of the proposed method in Figure D.I This model has two types of encoder as attentive neural process Kimm al.) 2019). The encoder parameters e consists of Φ1 d2 The Ø1 is responsible for the local representation and the Φ1 is responsible for global representation 2. The encoder of global representation is same as neural process Garnelo et al., (018b), however, the encoder of local representation is different from the standard cross attention Vaswani et al., 2017). After obtaining standard attention weight Wstandard: we introduce reparameterization trick for wy, which follows the Weiubll distribution(Fan et al. 2020)). The important thing is that the key conceptual prior can be made by MLP&s, $1 ((z, (Z))).) The decoder is the same as the attentive neural process(Kim) et al. 2019).
globs
HLP
ML3
MILFF
W
TA
MUL's
Figure D.I: Model Architectures of the proposed method
D.3 ALGORITIM
The proposed method requires hyper-paramen parameters and 3 for reparameterization of the Weibull distribution and KL divergence between the Weibull and the gamma distribution. We conduct the grid searches for to find the best value. We identify that the proposed method with k: = 300 adequately captures the dependency and generates noises avoid memorization for all experiments. In the case of 3, we follow the setting of the Bayesian attention module(Fan et al. 2020). We suggest the entire procedure of our algorithm as follows:
D.4 COMPARISON OF PROBABILISTIC GRAPHICAL MODELS FOR VARIANTS OF NPs
We describe NPs with probabilistic graphical models to differentiate the variants of NPs. The Neural process employs mean aggregate function and reparameterization trick to obtain a global context representation 2Garnel et al., 2018b). This model follows Figure D.2a The Attentive neural process expedites the multi-head cross attention to obtain local representation r; with global context representation We show that the graphical model for ANP is the middle of Figure D.2 all variables in the attention mechanism are regarded as the determinstic variable.
In this work, we design that all latent variables contain stochasticity and achieves by the repa- rameterization trick. By Bayesian attention module, we present our graphical model as shown in Figure
19

---

Published as a conference paper at ICLR 2022
Algorithm 1: Neural Process with stochastic attention
Input: Task distribution p(T), the shape parameters of weibull distribution and gamma distribution
300ee completion task 100), ,3 1, Stepsize y
output: encoder parameters : Φ = {Ø1, Ø2), decoder parameters : Initialization:: and randomly; while not converged do Sample tasks [ from p(7) for all Ts: 3 {T] do
Sample context dataset (Xe, Ye) = {Iy, My 1 and target dataset f
(Xe, Yi) (aa, ys ) from the task Tk
Sample random noises 61 ~ Unif(O, 1) and 62 ~ (O),
Local representation {ri {r:)e=]
Q K. f (X,, (X.: Y.))
Watandard &- Softmax (alimax(Q""" /vax)
(Reprameterizatoon sampling of weibull distribution for W)
À +- Wstandard [(1 + 1/k)
W {w] Wn - log(1 - =1
W [wl. Wn +- { W,,, 55 W
(Parameters of the prior distribution for W) a + fo, (K)
(ri)==1 (W.VV... W., V}
N
Global representation =:
pbi T21 & Aggregator (1 Vi )=))=1))
(Reprameterization sampling of normal distribution for 2)
2 = Hdz + 62 Péo
Decode X,, (r:)
Py. Jy + (fo(Xe, (ri)em))
Evaluation loss
L0.0 To Em log (osllo Jg.)) - KL KL (2l (X,, 1i)) (X,,
(1. (w. Xx) (wy (x))
end
Update + + +V à + ΣT. Tt.
Update 0 + 0 + 7V O ETg Ch.
end
4
JEC
lEc
Y iET
Deteminisiic variebles
y
Stochastcc variates
Geoerttion
Inference
ieT
iET
Figure D.2: Comparison of graphical models; (a) Neural process(Garnelo 2018b), (b) Attentive neural process(Kin Kim et al. 2019) and (c) Proposed method
20

---

Published as a conference paper at ICLR 2022
E ID REGRESSIONS
For the synthetic ID regression experiment, we set de = 1, dy = 1. The number of layers in encoder and decoder in all basclines is respectively lene Upre 3, ppost = 1} 4 and ldec 3. We set the dimension of latent variable as d,- = 128.
When it comes to data generation process, the training data is generated from the Gaussian process with RBF kernel. For each task, we randomly generate 2 ~ f and then y is the function value by the Gaussian process with RBF kernel, k(r;, xx) 8² exp( (xxp--1e=<'1//22" To validate our model, we establish two types of datasets in the this ID regression experiment.
First, we the parameters of the RBF kernel all datasets are drawn from continuous and smooth functions. Second, we consider the noisy situations in the first scheme. We intend that the suggested scheme represents the actual weal-world
3 and 1 3. Hence, shown in Figure E.3a
situation. Unlike the existing studies (Garnelo et al. 2018a] Kim et al.) 2019 Lee et al. 2020). we modify the RBF kernel function adding high frequency periodic kernel function kyy. y
s² exppp2 s (~||Ix noises sampled by t-distribution; however, these noises often have exaggerated values so that the generated function does not have any tendency and seems to be entire noises. On the other hand, the function generated by Gaussian processes with a high frequent periodic kernel is smooth but is satisfied with random function every trial. Thus, this function does not interfere with the smoothness of the RBF GP function and maintains the smoothness of all support ranges. Therefore, we decide to use the dataset generated by the RBF GP function with periodic noises to synthetically test all baselines and the proposed method for the robustness of noises. The sampled dataset is graphically presented in Figure E.3b.
z' 1 /pp)²)) Lee al. (2020) proposed noisy situation by using random
-2-
-4
(b) The RBF kernel GP with peri- odic noises
(a) The RBF kernel GP
Figure E.3: Comaprison of RBF GP functions in 1D regression problems
To generate function values of the Gaussian process, the GPy library provides various func- tions compatible for Py TorcH In the GPy, ||-- I'12 can be regarded as the pre-defined vector {1. ................................ ( freq). We set the parameters of the periodic kernel as freq 30, p 2m, 8 = 1. The generated functions for training datasets are shown in Figure E.3
To test generalization to novel task, we introduce other functions such as the Matern32 kernel GP and the periodic kernel GP as shown in Lee et al. (2000). In this experiment, we expect the trained models capture context points despite different functions. Naturally, all models perform well on the RBF GP function because the meta-training set is generated from RBF GP functions; instead, we test all baselines on other types of functions such the Matern32 kernel GP and the Periodic kernel GP as well as the RBF kernel GP. For the Matern32 kernel k(r, x²) s² (1+v3lxx-)) [l) exp(-v3) 3 and the periodic kernel is same as mentioned earlier. The shared parameters of these three GPs are same as s 3, 1 3. The periodic kernel requires another parameter freq 10, which is the default value in the GPy.
To elaborate the meta-training framework, the minimum number of context points is 3, and the minimum number of target points is 6, including the context data points. The maximum number of context points and target points respectively 97 and 100. In the following subsection, we report experimental results, including clean and noisy situations.
httt:://gthub.co//hheffidddd "hupss//pyorrhh.org
21

---

Published as a conference paper at ICLR 2022
Table E.4: Results of likelihood values on the synthetic ID regression without noises. Bold entries indicates the best results.
RBF kernel GP
Matern kernel GP
Periodic kernel GP
larget
context
target
context
target
1.136:10.044
0.762:10.000
-0.465-10.106
-3.684:0.222
-2.354
-9.612+0.0287
NP
0.687++0.030
0,312#0.127
-1.222:40.055
-3.546=0.268
-2.252:=0.115
-6.038+0.10 195
ANP
1.306:+00.04
0.819++0003
0.963++0.039
-1.538+00.083
-0.193-+0.219
-7,447+4.2278
(Weight decay; À 1.0c 3)
CNP
1..67:40.02
0.827#0.051
+0.310+0..66
13.135+0.178
-2.
-8.840:10.259
NP
0.695110.031
0,297=0.051
-1.297::.201
3.58330.536
-2.577
-6,489:10246
ANP
1.355:+0.002
0.924::0035
1.077:+0.023
-1.753=0.089
0.379=0.080
-7.663-+0.325
(mportance Weighted ELBO;
5)
CNP
1.170:+0.023
0.803+00083
10,443+-..100
0.724
-2.422++0.151
-96639+0..300
NP
0.728:10.028
0.255=10.156
-1,476:10.054
1,208
-3,04220..99
-7,463-10.624
ANP
1.344:+0.004
0,889=0.037
0.854+0.050
-1.854=0.114
-0,483+0.199
-7, -7,900++0200
(Bayesion Aggregation)
CNP
1,282:10.014
0,97811.019
-0.141+10.100
-3.443=0.294
-2.016:10.168
-9.703-10.221
NP
0.379:+0.023
0.069
-0.051
1.00006
1.466::0.092
-3.308 10.177
(Functional representation)
ConyCNP
1.351=10.02
0.891=0.024
1.336=10.008
-1 00.06
1.020=0.062
-9663--0.224
ConyNP
1.318:10002
0.905:10.029
1.287
-1 1,806=0.169
0.953=00.057
-9.580
[Regularization for local representation)
ANP (dropout)
1.348:1:0.001
0.866:2.0029
0.902:10.051
-1.753:10.044
00.520:0.211
-8.052:1 0.324
Bootstrapping ANP
1.347:10.005
0.895=10.026
0.790+0.044
-1.83420.004
-0.983=10.309
-8, 8.4633+ 0.343
Proposed Method
1.343-1-0.006
0.937-10.000
1.170+00013
-1.708
0.681+0.052
-7.807-40.39
E EXPERIMENT RESULT ON RBF GP FUNCTIONS
We train all baselines on the RBF GP functions without noises. We demonstrate experimental results in terms of predictability and context set encoding.
As shown in Table H.4 all models capable of fitting the RBF GP function; meanwhile, all models are degraded in cases of Matern kernel GP and Periodic GP. Unlike all baseline models, of which performances drop substantially, the proposed method has relatively small degradation. Particularly, the proposed method records that the likelihood value of context points at Matern kernel GP and Periodic GP is respectively 1.170 and 0.680. Meanwhile, the best performance among baselines is 1.077 and 0.379 by ANP with weight decay. This result shows that the proposed method performs better than baselines. The graphical results are described Figure E.4
E.2 EXPERIMENT RESULT ON RBF GP FUNCTIONS WITH NOISES
As mentioned early in the current section, train all models the RBF GP functions adding the random periodic noise. Looking at the column of RBF kernel GP in Table I most of the baselines are degraded due noisy situations. However, the performances of all baselines GP and Periodic GP. We guess that this phenomenon can be explained as mitigating memorization issues by injecting random noise to the target value (Rajendran effectively random noises improve generalization performances for all baselines and our method in this experiment. However, we can recognize that injecting random noises cannot fundamental solution. As seen in Apppndixx G, we emphasize that capturing and appropriately utilizing context information when predicting target values is a fundamental solution to avoid memorization in a given situation. The experimental result is described in Table on manuscript and the detailed graphical explanation is suggested in Figure E.5
improved in Matern
al. 2020). We present how
22

---

Published as a conference paper at ICLR 2022
RBF GP
Matern GP
Periodic GP
CNP
(BA)
NP
(BA)
ConvCNP
ANP
ANP with Info. Dropout
Bootstrap.
ANP
Ours
context points predictive mean predictive std ground truth
Figure E.4: 1D regression results by NPs trained on the RBF kernel function without periodic noises. All baselines and ours fit the RBF GP function, but degrade their performances cases of Matern and Periodic kernel GP.
23

---

Published as a conference paper at ICLR 2022
RBF GP
Matern GP
Periodic GP
CNP
(BA)
NP
(BA)
ConvCNP
ANP
ANP with Info. Dropout
Bootstrap ANP
Ours
context points predictive mean predictive std ground truth
Figure E.5: 1D regression results by NPs trained on the RBF kernel function with periodic noises. All models improve performance in Matern and Periodic GP than Figure E.4, The proposed method outperforms baselines and regularization methods. Even this model achieves better performance than the result on RBF GP without noises.
24

---

Published as a conference paper at ICLR 2022
F PREDATOR PREY MODEL
We follow the experimental detail in this paperq[eee et al. 2020) Gordon et al. 2019). This experiment designed evaluate all baselines capable of adapting new tasks, which have slightly different task distributions. It is called "Sim22eel".. We assume that, in this situation, we easily obtain simulation data, but it requires a high expense collect real-world data points. We try train all baselines on the dataset generated by simulations and test them on real data sets that are relatively small compared to the simulation data.
The way of simulated data is generated from the Lotka- Volterra model(LV model) Wilkinson 201 8). Note that X is the number of predators, and Y is the number of prey any time step in our simulation. According to the explanation, one of the following four events should occur. The following events require the parameter 01 = 0.01, 02 = 0.5, 03 = 1, 04 = 0.01
1. A single predator is born according to rate 01 X Y, increasing by one.
2. A single predator dies according to rate 0₂ X, decreasing by one.
3. A single prey is born according rate 03- Y. decreasing by one.
4. A single prey dies(or is eaten) according to rate 04 X Y, decreasing Y by one.
The initial X and Y are randomly chosen. On the other hand, Hudson's 's bay lynx-hare dataset follows a similar tendency to the LV model; however, it is oscillating time series because it has been collected in real-world records. There were unexpected outliers, unexplained events, and noise. For the detailed explanation, refer Gordon et al. 2019)'s work. All simulation codes are available on this URI The generated simulation data can be graphically shown as the left side Fiiuure F.6)
In this experiment, we set d2 1, dy = 2, and all remaining settings are the same as the ID regression experiment except that the number of context points is least 15 and the extra target points is also at least 15. When training models, we use the dataset generated by the LV model and test on the lynx-hare dataset. The experimental result is shown in Table 2] and the prediction performance can be graphically shown in Figure F.6
We conduct additional experiments to validate whether the random periodic noise positively influences the performance. As conducted in the 1D regression experiment, we utilize the periodic kernel GP function, which has very high frequency freq = 100 to add noises to the training dataset. As shown in Table F.5 compared to Table 2] We recognize that adding random periodic noise empirically improves all baselines and our method. Among models, our models perform best due to the full representation of context and target datasets. The prediction results can be graphically shown in Figure F66b
Table F.5: Predator-prey model results. All models trained on data with periodic noises.
Simulation
Real data
context
larget
context
target
CNP
-0.181
-0.308
-2.420
-2.789
NP
-0.641
-0.739
-2.464
-2.710
ANP
0.645
0.290
-0.634
-1.962
(Importance weighted ELBOO (s = 51)
CNP
-0.284
-0.390
-2.484
-28644
NP
-0.527
-0.628
-2.335
-2.641
(Bayesian Aggregatoo))
CNP
-0.060
-0.171
-2.393
-2.776
NP
-0.357
-0.458
-2.260
-2.618
(Functional representaaionn)
ConvCNP
2.395
1.679
1.879
-0.205
ConvNP
2.368
1.687
1,691
-0.521
(Regularization for local representation)
ANP (information dropout)
-0.749
-0,990
-1.766
-2,458
Bootstrapping ANP
0.326
-0.008
-1.183
-2.008
Ours
2.745
1.819
2.699
-0.076
25

---

Published as a conference paper at ICLR 2022
Prodake
Thiiiing
Traning
CoNCCNP
COMMENP
ANP
AhD with Strppooe
Bockamppiig
Bootstapriing ANP
(a) On the clean data
(b) On the noisy dataset.
Figure F.6: Prediction results on the hare-lynx data. (left) All models are trained on the clean dataset. (right) All models are trained on the noisy dataset.
26

---

Published as a conference paper at ICLR 2022
G ANLAYSIS OF CONTEXT EMBEDDINGS VIA ATTENTION WEIGHTS IN ID REGRESSIONS AND THE HARE DATASET
The attention mechanism can explicitly display the distance between context and target data points via attention weights. Especially, when the dimension of features is one, I E IR' the distance between features is simply calculated and sorted, so allowing for straightforward comprehension when the attention weights are shown in heat-maps. As result, we present several heatmaps to compare ours with baselines in both clean and noisy ID regression datasets. The horizontal axis in these graphs represents the value of features in the context dataset, while the vertical axis represents the value of features the target dataset. The best pattern for this heat-map is diagonal because all feature values are arranged in ascending order. Additionally, we will provide the simplified heat-map that take into account only target points with same value in the context dataset. Due to the fact that the original version has one-hundred target points, labels and ticks are necessarily small. We guess that readers may be unable to decipher the detailed information such as labels and attention scores. This simplified version allows the readers instantly grasp what we attempt convey by exhibiting more distinct diagonal pattern. From Figure G.7 to Figure G.9. on the left side, we show a simplified version of the attention score; on the right side, we show the result for the entire target dataset. Plus, we provide Sim2Real's 's attention heat-maps in both clean and noisy environments. As a result, our discovery appears be consistent.
First, the heatmaps of the ID regression problem without perioide noises are shown in Figure G.7 In this graph, all models including ours can accurately depict the similarity between the context and targete dataset. Although the attention scores for each model vary, it is clear that the majority pattern of all models diagonal. Therefore, all models are trained using the clean dataset in the intended manner. However, other phenomena are found in the noisy data. We identify that the attentive neural process Kiim et al. (2019) fails to capture contextual embeddings because the attention weights of all target points highlight on the lowest value or the maximum value in the context dataset. In the case of the Bootstrapping [ee et al. (020)) and ANP with information dropout, the quality of heat-map is slightly improved, but it still falls short of ours. This indicates that the present NPs are unable to properly exploit context embeddings because the noisy situations impair the learning of the context embeddings during meta-training. On the other hands, even in the noisy situation, the attention score in ours still appear clearly. Second, we include heatmaps for the Sim2Real problem. As seen in Figure F.6. the context and target datapoints are much too many for the label information and ticks to be recogmized. Hence, we recommend that readers verify the presence of diagonal patterns rather than examining detailed numerical values. As seen in Figure G.9a| all models are capable of capturing properly similarity between the context and target datasets. However, as with ID regression, the diagonal pattern of the baselines disrupted as seen in the left graph of Figure G.9b, nevertheless, ours retains the ideal pattern in both clean and noisy situations.
When comparing our model's s heat-map pattern clean and noisy environments, there a noteworthy point. Our model is capable of learning adaptively how to focus on certain context datapoints depending on the extent of dirty data in the meta-train dataset. The model is trained on the clean dataset to take into consideration nearby points well the corresponding point in the context dataset, hence, the heat-map gradually changes. This is because the clean dataset has smooth values, Yi.j. near certain feature z... Meanwhile, in the noisy dataset, the model is trained to focus exclusively on corresponding points to the context dataset. Hence, the heat-map in Fig E.5 (b) indicates that attention score of target datapoints that includes the context dataset has high value, whilst the remainder points treat all context datapoints uniformly. This phenomena occurs because there is less correlation between adjacent features zi and its labels yog in the noisy situation during the meta- teta-training.
27

---

Published as a conference paper at ICLR 2022
RBF GP
Periodic GP
12
1.7
34
07
ANP
03
60
1.9
20
ANP with
Info. Dropout
4.49
17
10
Bootstrap.
ANP
43
10
1.49
1.75
135
Ours
4379
675
1.5
48
6.
62
335
63
Figure G.7: Heatmaps of asscociated attention weights in 1D regression problems when test datasets are sampled from RBF and Periodic kemel functions. All models are trained on the clean dataset. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints.
28

---

Published as a conference paper at ICLR 2022
RBF GP
Periodic GP
Cooee
251
140
38
ANP
450
40
41
86
64
389
04
10
444
48
50
05
05
ANP with
Info. Dropout
60
04
63
199
3.4
144
655
Cooko
Altegg Aergic
177
06
31
Bootstrap.
430
ANP
64
34
10
02
1.2
1.7
00
1.0
34
Ours
230
440
31
440
1.0
0.4
82
2.2
A1
33
Figure G.8: Heatmaps of asscociated attention weights in ID regression problems when test datasets are sampled from RBF and Periodic kernel functions. All models are trained on the data with periodic noises. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints.
29

---

Published as a conference paper at ICLR 2022
54
ANP
ANP
53
ANP with Info. Dropout
ANP with Info. Dropout
9M
Bootstrap. ANP
Bootstrap. ANP
04
Ours
Ours
01
(b)
Figure G.9: Heatmaps of asscociated attention weights in the hare-lynx dataset. (Left) A simplified heat-map that takes into account only target datapoints with same value in the context dataset. (Right) A heat-map that takes into account entire target datapoints. (a) All models are trained on the clean dataset by the lotka-volterra simulation (b) All models trained on the lotka-voltces simulation with periodic noises.
30

---

Published as a conference paper at ICLR 2022
H ABLATION STUDY: RELATIONSHIP BETWEEN REGULARIZATION OF PAYING MORE ATTENTION TO THE CONTEXT DATASET AND CONTEXT EMBEDDINGS
This section discusses the effect of suggested regularization, paying more attention to the context dataset attention scores. We demonstrated how the proposed regularization term results in an increase in learning stability and embedding quality. Prior to conducting a thorough comparison of stochastic attention with and without regularization, we analyze learning curves between base- lines, such as CNP, ConvCNP and ANPs in ID regression under a noisy situation. Because the proposed regularization naturally incorporates the attention mechanism, we begin by comparing the proposed model to NPs that do not employ the attention mechanism. We choose CNP with Bayesian Aggregation, NP with Bayesian Aggregation( et al.] 2021) and ConoCoP(Gordon et al.] 2019) as baselines, taking into account prediction performance as demonstraed in Table As shown in Figure H. the negative loglikelihood of CNP_ _B and NP BA have converged to about 10.0 and 0.0 However, the Conv and ours record values around -30.0. In the case on ConvCNP, this method optimally computes similarity between context and augmented data points (called grid) that include target points with high probability using the thereotically well-defined functional representation by RBF kernel function. On the other hand, the proposed regularization approaches the ideal loglikelihood value without the assistance of extra data points. The proposed can train how to establish similarity between context and target points and predict labels of target points during meta mnta--rraining Second, we compare training curves between ANP, ANP ANP(dropour) (Kim (Kim ct al., 2019). Bootstrapping ANe(Lee et al. 2020) and ours. This comparison indicates that while all methods make use of the attention mechanism, the chosen baselines exclude stochastic attention and regularization; T(Z, / DD. When we look at Figure H.1 we observe the same results as shown the quality of context embeddings in Appenddix GJ While the Bootstrapping ANP had superior training curves to ANP and ANP(dropout), but it was unable to be match ours.
Nepatho Wasthod
20
BA
AlPidhopeatt Boonstoo ANP
CORSENNP
Clrro
Seqalire
-30
Ouar
300
-40
2000
5000
103001 20 17003 1090020200 Stepe (1/10 wO
-40
2500 2000
7998 10003 1S500 100000 754020003
50ep (11T araae)
(a)
(b)
Figure 10: Comparison of training curves with their respective baselines. (a) The baselines that do not employ the attention mechanism. (b) The baselines that employ the attention mechanism.
We examine the general effect of the proposed regularization to the stochastic attention by al- tering hyper-parameter K. We conduct experiments in which all models use same generative process for stochastic attentions, as desribed in Figure 2] but with varied hyper parameter, K E [1, 20, 30, 40, 50, 60, 70, 80, 100, 300, 500} and the presence of KL(qs KL[gs(w,/2;; (ux]lz [X)]) To begin, when we analyze Table H.6 we observe that variation of the hyper- parameter K has little influence on the prediction performance of ID regressions. If the models properly converge, the prediction outcomes should be statistically indistinguishable. However, the proposed regularization has an effect on model's learnability. While experiments cases that employ the proposed regu- larization converge, a few models with the hyper-parameter K E {10, 50, 70] that do not use the proposed regularization, do not. Additionally, we demonstrate this fact via negative log likelihood values in the meta-training, the proposed regularization have converged, whereas some models that do not use the proposed regularization and use the hyper-parameter K (10, 50, 70). have significantly different training
seen in Figure H.II As Table H.6 indicates, all models that use
31

---

Published as a conference paper at ICLR 2022
curves when compared of converged models. Therefore, anticipate that the proposed method will ensure learnability and convergence regardless of hyper- parameter value.
Table H.6: Prediction performance of ours varying the hyper- pypee-parameeer K and regularization. All models employ the generative process stocahstic attentions. The hyphen means a model converges.
No Regularization
Regularization
Periodic
Periodic
Models
Trainable
cocsest
target
CODAEXt
turget
Trainable
context
Iarget
comtext
target
K=/
1.303:000000
0.421=0,005
1.382=00.00
-0.32210.023
1.376:100001
0.405510.007
1.37620,001
-0.34720,000
K-10
0.543=00007
0.199-0.309
0.301-0.015
-0.645+0..00
1.383+00000
0.417:00008
1.369-0.022
40 -0,402=0:0225
K-20
1.380=0000
0.431=0.005
1.300=0.000
0.355:00.029
1.377=0005
0.408=100011
1.381=0.001
40 40366220017
X=30
1.381=0.00)
0.356=0.011
1.380=0.000
0.370:::027
1.382:00000
0.452:40007
1.381=0.000
X-40
1.380=0000
0.505-0.009
1.30000000
-0313+0001
1.379:+0/022
0.446:100008
1.380=0.001
-1330000021
K-50
X
0.730=00002
0.267=00022
0.520-0.012
0.6422+0043
1.379=00002
0.355++0.004
1.380==0.001
J0 .0.01:=0000
K=60
1.378:00003
0.433=0.207
1.811=0.001
0.460020.029
1.300:00000
0.349:20.044
1.300=0.001
-0. 40.338:0.022
K=70
0.64220.005
0.235_0,010
0.001
0.64110.029
1.373:00001
0.02910:008
1.377
00.37020.024
K-90
1.579=00002
0.425-00011
1.381=0.001
-0.374200054
1.375+00002
0,445+10009
1.379=0.002
-0.353-0.000
0.375=0.002
0.573=0.007
1.578=0.001
0.436:0.000
1.377=0000
0.457=00009
1.378=0.001
0.331100.554
K=300
1.38020,002
0.393=00006
1.382-00.00
0,369:10.026
1.379
0,417:10:066
1.382=0.001
40 0.34620,024
K-500
1.383-0.000
0.559-0.00
1.383-0.000
.03316+0000
1,381
0,402:+0:009
1.380-00.001
J0. -0,352-0003
20
10
Nogeche hathbaa
10
14
20
80
320
2003
520
203
-10
-30
-60
-00
2500
5000
7503
20000
12908
19300 L7200 20003
2500 5000
7500 10000 12500 25000 17500 20000
Stage 10110 swale)
Stopo (1210 soale)
(a) No regularization.
(b) Regularization
Figure H.11: Comparison of training curves varying the hyper parameter K. All models employ the generative process of stocahstic attentions.
We study the learned attention scores in detail to verify the proposed regularization's 's efficiency. To conduct fair comparison, we fix the number of context points and choose target points that have same value as the context dataset. The ideal scenario is that all scores in the diagonal components have identical values and little variation regardless of the placement of target points as the attention mechanism should focus exclusively on context and target dataset.
As seen in the case of no regularization in Figure H.12 despite somewhat varied scores, all converged models exhibit the same diagonal pattern. As result, these models present similar prediction performance. We argue that 1D) regressions do not need precise context embeddings. However, when we compare the attention scores with and without regularization, we discover findings regarding the quality of context embeddings. In the absence of regularization, the values of attention weights changes according the tlpper-paaameter K. Certain models place less emphasis on the diagonal components than others, while others excessively place a greater emphasis on them. Without the regularization, the attention scores of diagonal components range from 0.3 to 0.64. We instantly notice that without regularization, the attention heat-map of stochastic attention has an inconsistent degree of brightness in relation the hyper parameter K. partieular, the heatmaps of K = 60, 100 is too bright in comparison other heat-maps, whilst the heatmaps of K 40 is too dark. While the diagoanl pattern is presented, there is no consistency in the brightness of any heat-maps.
On the other hand, the attention scores of regularized models exhibit little fluctuation. Regardless of the hyper parameter K, these scores are closed to 0.5. Table H.7 contains the quantitative result. In the regularized model, the variance of attention scores for diagonal components is 0.003 and the mean value of attention scores is 0.469. Nonetheless, the mean of the model without the regularization is 0.367 and the variance is 0.026. This demonstrate that regularized models consistently
32

---

Published as a conference paper at ICLR 2022
Diagonal
Off-Diagonal
Mean
Variance
Mcan
Variance
No Regularization
0.367
0.026
0.059
0.001
Regularization 0.469
0.003
0.070
0.002
Table H.7: Descriptive statistics of attention scores between diagonal and off-diagonal compoenents cases of regularization, [(Z, 24 D) and no-regularization
properly emphasize the context dataset according to property of context and target dataset. The regularization cases in Figuur H.12 illustrate this fact graphically. All heatmaps of stochastic attention that employ the proposed method regularization have very comparable brightness. We identify that the proposed regularization ensures learning stability by ensuring the quality of context embeddings remains consistent indepedent of the hyper-parameter K. As a consequence, we declare that the proposed method increases model's insensitivity to hyper-parameters K by preserving the context embeddings consistently.
No regularization
Regularization
No regularization
Regularization
K 80
=
K
277 6.
K= 30
K 300
2777
737
K= 500
Figure H.12: Comparison of attention varying the dypee-paraamtte k and the presense of regularization, in 1D regressions drawn from the RBF kernel GP.
The intriguing fact is that the difference attention scores for the diagonal component is rather sig- nificant within the same models. Since the values of target points are same the context dataset, the attention scores should be constant. For instance, when K 40 and no regularization used, the at- tention scores for the same model with regularization are (0.51, 0.52, 0.44, 0.43, 0.45, 0.43, 0.45, 0.46, 0.5, 0.52). The variance in absence of regularization is 0.003, whereas the variance in the presence of regulariza-
f0.64, 0.64. 0.54, 0.47, 0.64, 0.54, 0.57, 0.62, 0.64, 0.64). The attention scores
33

---

Published as a conference paper at ICLR 2022
Niterta S0rns
Attertaa 500066
0.4
0.3
0.3
0.2
02
0.1
0.0
0.0
welbou distribation
disiilaation
(a) No regularization.
(b) Regularization
Figure H.13: Mean and variance for attention scores of diagonal components varying the hypere- parameter K and regularization.
0.200
0.200
0.125
0.125
0.075
0.075
0.225
0.025
20
40
50
60
700
80
100
300
500
40
100
300
50C
Tae shape parameter wettoll statribration (S)
sbape parameter wetbal clarmtration
(a) No regularization.
(b) Regularization
Figure H.14: Mean and variance for attention scores of off diagonal components varying the hypere- parameter K and regularization.
tion 0.001. As result of the consistent scores across all target points ranges, we realize that the regularized model improves context embeddings.
The detailed analysis is shown in Figure H.14 While the mean values of attention scores vary between individuals, the proposed regularization assures that attention scores are constantly near 0.5 value. From the perspective of NP's aim, the proposed regularization enbales models achieve those objectives. The attention scores of all target points must not be influenced by feature magnitudes and their regions. Consequently, this method has to concentrate only on the similarity between features. The regularization aims to derive appropriate attention mechanism that evenly weights across important regions and eliminates biases during the metat-raining. It leads to achieving task adaptation, as it equally prioritizes context data points regardless of features magnitude. Whenever the novel arises, the regularization ensures that the models retain the quality of their context embeddings.
We investigate
research into the cases of converged models. As seen in Figure H.15 there are completely distinct phenomena from what we have observed so far. When models are not converged, all attention scores remains constently uniform across all region. As result, the prediction performance degrades. However, when the models use the proposed regularization, the attention scores are comparable
the others. This means that the regularization crucial for preventing models from trapped in the local optima. Additionally, as seen in Figure H.1l] this regularization enables models can approach to maximize log likelihood values. By examining these phenomenon, is clear that the approach proposed in this study performed as a regularization role.
34

---

Published as a conference paper at ICLR 2022
No regularization
Regularization
K 10
K 50
K 70
Figure H.15: Comparison of attention scores when the models without regularization not properly trained. K E {10, 50, 70}. in ID regressions drawn from the RBF kernel GP. (left) All methods do utilize the suggested regularization. They employ the reparameterization trick exclusively for attention scores. (right) All methods employ the suggested regularization.
35

---

Published as a conference paper at ICLR 2022
I NOVIELENZ-100K DATA
We follow the experimental detail in the paper (Galashov et al. 2019). The way to split into train, test, validation dataset is described in this work Briefly, all ratings are first divided into two datasets, which respectively have user IDs and are not duplicated each other. We define that one is the meta-training dataset and the other the meta-test dataset. this experiment, we evaluate baselines that generalize to new customer with small history data points. To explain a training regime, the number of context points. All users do not have the same amount of ratings so that the number of data points is varying during meta-training. When testing baselines, we sample few ratings from unobserved users and then evaluate the likelihood values for the remaining points. To exclude users' unique information, we drop the User ID from the set of features and only use general features such as age, occupation. Lastly, in this experiment, we set d- 44, dy 1, and all remaining settings the same the 1D regression experiment. The detailed explanation is Galashov et al. (2019)'s s work. The likelihood values on the MovieLens-10K reported in Tabll 1.8
least three and the objective function computed with the remaining data
Table 1.8: Results of likelihood values the Moviel 10k. Bold entries indicates the best results.
context
Target
CNP
-24.718
2239008
NP
-23.096
-22.135
ANP
-1.456
-2.042
(Weight decay; X 0.001)
CNP
-23.524
23.0988
-15.876
-16.109
ANP
-0.891
-1.665
(Importacce Weighted ELBO:
CNP
-17.276
-17.799
NP
-25.322
-25.233
ANP
-17.205
-16.427
(Bayesian Aggregation)
CNP
-12.472
-12.141
NP
-1.911
-11.859
(Regularization for local representation)
ANP (dropout)
-1.742
-2.434
Bootstrapping ANP -16.267
-16.191
Ours
-0.349
-1.617
To compare RMSE scores with existing methods for Movielens- 100k data, suggest evaluating NPs using ul. base and ul.test. As mentioned, the prediction by NPs is based on given context data points. First, we randomly choose customer in the ul.test. Second, we randomly sample small number of ratings from ul. base, which is used as the meta-training data. In this experiment, we decided to draw ten samples from bl.e and employ these points as the context data set. The ten ratings is relatively a short history compared to the size of the ul aase.. Third, we perform predictions for the data points the ul.test based on the context data points. The important thing that context and target data points are sampled from the same user but the different data set. Plus, we do not touch in information in the test set, at least. Based on the relationship between sampled context points and report five runs and average these target points, the performance of NPs is somewhat hat variable, results. The experimental result is in Table 3
hhaps://tnnyrr..om//ff//222
36

---

Published as a conference paper at ICLR 2022
J IMAGE COMPLETION
In the image completion task, we follow the training regime suggested by NPr(Gar et al. 2018a b). In this regime, dz means the pixel location and dy means the color RGB data. Namely, we set dz = 2 and dy 3. For high capacity representation, the dimension of all hidden units determined as d,, 512. All remaining settings, including layer sizes, are the same as the ID regression experiments.
When it comes to image completion tasks, according to these papers, NPs are known complete facial images given partial pixels. Although a few pixels are provided, NPs are capable of understanding the common appearance of a human face. These models generate eye, mouth, and nose even if only a small number of local information is given, which is generally hard to recognize. The Conditional neural process properly performs; however, often loses its context representations. The Attentive neural process tackles this issue to employ an attention mechanism. Instead of the encoders consisting of MLP layers, the self-ateention can improve the quality of pixels generated at the context data points. However, as mentioned on subsussctt 4.2, scores of generated pixels in unseen regions are still problems. Even though many pixels are provided, the quality of completed images is not improved. In this work, the proposed method concentrates given contextual pixels and complete images based on that information. Hence, the proposed method completes an image with high uncertainty when few pixels are given. Plus, can be seen that the higher the pixels information, the higher the the graphical evidences are quality of generated images. The numerical result is presented in Table 2] shown in from Figure J.16 to Figure J.19
The proposed method allows for completion of images using only partially provided pixels, so it enables complete larget images than the image size in the training dataset. We produce images of various sizes using the proposed method, which is trained on the CelebA dataset with size of 32 32. The completed images are presented in Figure J.20, When we qualitatively evaluate the completion results, the proposed method properly perform for generating large images without sacrificing performance. All completed images are quite close to the ground truth. As a result, we conclude that the proposed method can be used for task adaptation since it can complete images that the model have never encountered during meta- meeaa-raining and whose sizes differ from the training dataset. In particular, while the percentage of contributed pixels to the total number of pixels remains constant, the quality of completed images are enhanced according to the amount of provided pixels. We understand that the number of pixels is more significant than the ratio of context points. This is because the model has sufficient pixel counts to work with when images are large. For instance, employing only 300 pixels as a context dataset is insufficient complete the images. Nonetheless, the completed images with 300 pixels are comparable to those with higher pixel counts and is of higher quality than the baselines in Figure J.18
There is an interesting discovery of uncertainty values in the image completion task. Basically, we expect that many pixels reduce the uncertainty generated images. However, the existing methods stably generate facial images, but uncertainty values do not decrease even if available pixels increase. The proposed method makes high variance for a completed image under the small number of context points. As shown in Figure J.2I it can decrease the uncertainty of completed images when the number context points is large. From this fact, we identify that the proposed method performs in an intended manner.
37

---

Published as a conference paper at ICLR 2022
Ground Truth
Contexts 50
CNP (BA)
NP (BA)
Boostrapping
ANP
ANP
Ours
Figure J.16: Completed images using 50 pixels
Ground Truth
Contexts 100
CNP
NP (BA)
Bontstrapping
ANP
ANP
Ours
(BA)
Figure J.17: Completed images using 100 pixels
38

---

Published as a conference paper at ICLR 2022
Ground Contexts Truth
CNP
NP (BA)
Bookstrupping
ANP
ANP
Ours
300
(BA)
Figure J.18: Completed images using 300 pixels
Ground Truth
Contexts 500
CNP
NP
Bontstrapping
ANP
ANP
Ours
(BA)
(BA)
Figure J.19: Completed images using 500 pixels
39

---

Published as a conference paper at ICLR 2022
x 32 48 x 48
64 X 64
80 X 80
Ground Truth
(a) Example
32 32 48 48
64 64
80 X
Ground Truth
(b) Example 2
32 x 32 48 x 48
64 x 64
80 X 80
Ground Truth
(c) Example
Figure J.20: Our completed images in variety of sizes. The training dataset consists of the CelebA images with size of 32 x 32. The completed images range in size from 32 32 to 80 x 80. Each image is completed with 0.3 of the overall pixels. As the image size increases, the following values used the number of context points: (300, 690, 1230, 1920). Both the context dataset and its completed image are displayed. Additionally the most right column has the ground truth image. Each image is shown in proportion to its image size.
40

---

Published as a conference paper at ICLR 2022
Context Pixels
CNP (BA)
Ours
Ground Truth
50
100
300
500
Figure J.21: Comparison of quality of uncertainty by CNP with Bayesian aggregation and Ours.
41


Reviews:
[{"review_id": "kSCdjngMNLg", "reviewer": "Reviewer_DEA4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n- Even if, to me, the technical contribution is not major - integrating bayesian attention in the attention neural process - I enjoy the effort made to explained the advantages brought by the method in an information theory setup. This improves the understanding of the field and open up the path towards further directions.\n- The experimental part show consistent improvement over multiple setups and several datasets\n\n**Cons:**\n\n- The paper mention that the model does not require exhausted computations. Can you also report the comparison between the proposed methods and the other baselines presented in the paper in terms of time efficiency and the number of parameters?\n- For the experiments in Movielens-100K dataset, isn't it possible to integrate the graph structure in the NP approach for a fair and higher performant model?\n- In Figure 1 a) it is not clear what it is depicted in the second row. I guess it represents the attention weights similar to the one in the supp material, but it should be clearly specified in the caption.", "summary_of_the_paper": "This paper proposes a neural process enhanced with stochastic attention to focus more on the context dataset. The method replace the classical attention used in ANP with the Bayesian attention module, showing that this design choice improves the performance also in noisy scenarios or when target datasets mismatched (by changing the kernel used to generate the target dataset). Moreover, the paper offers an interpretation of the method from an information theory perspective, proving that the NP with stochastic attention can be seen as a regularization of the latent space such that it pays more attention to the context dataset. The method is tested on both synthetic and real-world datasets and improves the scores especially in noisy or more complicated scenarios.", "summary_of_the_review": "To me, the paper seems interesting not necessarily because of the technical novelty, but because of the theoretical side: the information theory based explanation of the final loss. However, I want to mention that I do not have much experience on this particular field so it might be the case that I missed some similar prior work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "UL1CfZWrfYQ", "reviewer": "Reviewer_TBTA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength: 1. The paper clearly indicates the idea of incorporating stochastic attention with key-based contextual prior into the neural processing. \n\nWeakness: 1. My biggest concern for this paper is whether there is a novelty of the proposed method. Although I see the benefit of stochastic attention as it is proposed by Fan et al, this paper basically combines neural processing with stochastic attention. Then what is the main novelty here? It is simply A (neural processing) + B (stochastic attention) is better than A. Also the contextual prior, how to optimize and inference, etc are similar as the Fan et al 2020. \n\n2. The information theory as well as the derivation of the equations in the main paper and appendix such as reparameterization, kl, and elbo are all covered in previous or early literature. It is not new.\n\n3. As the key contextual prior is one of the important parts of this paper, it is also less ablation study to show the comparison between including the contextual prior and not including the contextual prior.\n\n4. In the experiment section, the Movielens 10k dataset is a very real-world dataset as claimed by the author. However, the improvements on MovieLens-10k dataset are very marginal and even raise my concern about the practical benefits of this proposed method. \n", "summary_of_the_paper": "Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. This paper incorporates the stochastic attention in NPs to capture the context information. We empirically show that our approach on 1D regression, predator-prey model, image completion, and MovieLens-10k dataset. \n", "summary_of_the_review": "Please see the details in my main review. In summary, I think this paper clearly indicates their ideas of incorporating stochastic attention into neural processing. However, the novelty as well the comprehensiveness of the study and experiments need more elaborations and works. Thus, I recommended marginally below the acceptance threshold.\n\n\n----------------------------------Post rebuttal-----------------------------------------------------------------------------------------------------------------\nAfter carefully reading the authors' response, I am still holding the question regarding the novelty of this paper. The theoretical proof in this paper is very limited and basic. And most of them including the contextual prior, stochastic attention, elbo, etc have been fully covered by the bayesian attention modules [Fan et al 2020] paper. Thus I don't agree with the author about the theoretical proof. Thus, this paper can only be seen as a simple implementation of the bayesian attention module in the NP field.  \nIn addition, although I see new results in the rebuttal, my concern is that the accepted version of the paper might not be a substantially improved version of the paper. The rebuttal stage is not similar to a journal revision where you can submit an improved version of your paper. Rather, it should be used to clarify misunderstandings.\nOverall, I would like to hold my original score for this paper and still think this paper is not ready for the ICLR conference.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "8-12aqoqDbQ", "reviewer": "Reviewer_uH8j", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The application of Bayesian attention to neural processes seems novel.\n- The authors provide an information-theoretic perspective of the proposed method.\n\n### Weaknesses\n- The authors claim that conventional NPs are sensitive to noise, but this claim is not well-supported.\n- Being sensitive to noise is caused by multiple factors, such as the model's flexibility, the amount of data, the number of training steps, etc. With enough data, therefore, larger sensitivity can result in better performance. \n- I think claiming \"superior\" performance based on the reconstruction error is like claiming VAE is better than autoencoders in terms of the reconstruction error. ", "summary_of_the_paper": "This paper proposes to improve attentive neural processes (ANP; Kim et al.) by replacing deterministic attention with a Bayesian attention module.\nSince the inference requires variational approximation, I think the proposed method is a variational counterpart of ANP, similar to the relation between VAEs and autoencoders.", "summary_of_the_review": "I think the main idea is novel and worth publishing.\nHowever, some of the claims are not grounded properly and should be toned down.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2q4447ve6ke", "reviewer": "Reviewer_fqvy", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty:\n\nIn my opinion, the novelty is somewhat limited, as it is a plug-in method (the Bayesian Attention Module), to the well-established Attentive Neural Processes framework. Both methods and the regularization term are already introduced in the literature. While the paper has the merit to propose it on the very specific domain of Neural Processes, its effectiveness against newer methods is not proven enough so as to make the impact outweigh the lack of technical novelty. The development of the Bayesian Attention Module directly replicates that of Fan’s paper, including the mathematical derivation and formalisms. \n\n\nPresentation:\n\nThe writing and presentation are quite inadequate and make the paper hard to be followed and understood. It requires several reads to get a glimpse of the contributions and the figures and results are quite poorly presented (the quality of the images is especially poor). In the same line, the text is more based on suggestions than on actual statements that are linked to the proven concepts, which makes the narrative rather weak. While this is not on itself a heavy point towards my personal evaluation of this paper, I believe that in either case authors should devote a good amount of time to improving the paper readability. \n\n\nExperiments:\n\nThe method only compares against the simple baselines of Attentive Neural Processes and Neural Processes, and does not consider the recent advances in the topic such as the Convolutional Conditional Neural Processes and the ConvNP works of Gordon et al. and Foong et al. These works tackle the problem of domain shift with functional representations and seem to improve over the baselines in a big extend. The authors are encouraged to test their method against the newer NP families to prove the effectiveness of the proposed approach. \n\nI would like to see an analysis of the learned distributions w.r.t. the “distance” between the context and the target points, with a comparison w.r.t. the weights that would be computed by a deterministic attention module, which would act as a baseline. \n\nOther comments:\n\n- It is my understanding that at inference time the weights for the attention module are drawn from the Weibull distribution, meaning that for each query (x_i in the target) there will be N keys (x_i in the context) and hence N weights. In Eqn. (6), q_phi represents that distribution, which is conditioned to x_i on the target, and X_c and Y_c on the context. This seems to indicate that the parameter lambda is determined by an MLP whose inputs are x_i, X_c and Y_c. However, I do not see the role of Y_c on computing such weights, as the text seems to refer to the fact that the weights for x_i, X_c will be computed probabilistically according only to these values, which represent the weights to compute the value r_i for x_i as a linear combination of the r_i in the context, which are now represented by X_c AND Y_c. However, why is Y_c necessary to obtain the KL loss in Eqn. 6? Are the weights computed from x_i on the target and the r_i on the context? Please do clarify as I might have misunderstood something here.\n- One of the main problems of NPs is the domain shift, which occurs e.g. when the inputs are farther from the context set. NPs compute the latent variable independently of the target points i.e. they do not account for the distance between the context and the target points. ANPs partially solve this problem, as well as the ConvCNP and ConvNP family. In this paper, this problem is not analysed, and indeed the motivation seems to lie on the idea of making the attention weights be more focused on the context set rather than on the distance between the context and target sets. Again I might be misunderstanding something here, but I would like to ask the authors to elaborate on the similarities and differences w.r.t. the recent works aforementioned. How would the proposed method work in the image completion experiment for images of larger size than those used to train (e.g. 32x32 to 64x64). \n\n\n- The NP loss is formulated in a similar fashion to Variational Autoencoder, where z is computed during training from the target points. However, as pointed out in Foong et al. this regularization term might not be needed and indeed one can train the Neural Process without it by conditioning z only on the context points. Such approach improves the expressiveness of the NP family. I wonder how would the proposed method perform in such case, i.e. without the regularizations and by computing the latent distributions for the global and local variables directly from the context points only. This should be a necessary experiment towards understanding whether the mutual information gap needs to be maximised, or not. The paper does not provide any experimental analysis on the two terms of the mutual information mentioned in Section 3.3. Proving that this gap is an upper bound might not necessarily entail that indeed I(Z,x_i|D) is being minimized. \n\nIn summary, I believe the paper holds an interesting research direction worth exploring in the context of Neural Processes. However, in my opinion, the current manuscript does not meet the standards for acceptance at ICLR 2022. The authors are strongly encouraged to address the limitations and improve readability, as I do believe this is a promising idea. \n", "summary_of_the_paper": "The paper proposes a novel method for the Attentive Neural Processes paradigm by adding stochasticity to the weights of the cross-attention module between the context and target representations. These weights are drawn from a proposal distribution which is a Weibull distribution with parameters determined by the context and target input points. A new KL regularization term is added to the total loss to enforce the proposal distribution be close to the gamma distribution determined only by the context points, in a similar fashion to the regularization term for the latent representation between the whole set of target and context points and that of the context points only. The authors include, from Fan’s paper, the closed-form KL divergence between the Weibull and Gamma distributions, making the whole loss be differentiable w.r.t. the network parameters thanks to the reparametrization trick, as in the original NP paper. The new loss is linked to the gain of the target distribution by means of information theory, by making the assumption that the new regularization implies not only that the mutual information between the target distribution and the target points is maximized, but also the mutual information between the representation for the context and target points and the latent representation of the context variables only is minimized, enforcing the latent variable to also consider the context points, and not otherwise as suggested by the original NP formulation. ", "summary_of_the_review": "The paper develops on a promising idea towards incorporating stochastic attention weights to the Attentive Neural Processes framework. However, the current manuscript is far from being ready for its publication.\n\n\n--- Post rebuttal\n\nAfter interacting with the authors and considering the comments and revisions, I am happy to raise my score. The paper can make a good contribution to ICLR 2022. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work receives mostly positive rates. Most reviewers agree that the use of Bayesian attention to neural processes is novel, and its interpretation is interesting. Since the reviewer TBTA requests a substantial revision of the submission and fortunately authors’ feedback is thoroughly satisfactory, we highly recommend the authors to prepare for a significantly improved camera-ready version that clarifies most of reviewers’ concerns.
Decision: Accept (Poster)