Attentive Sequential Neural Processes
Authors: Jaesik Yoon
Gautam Singh
Sungjin Ahn
Venue:  (2020)
Keywords: meta-learning
neural processes
attention
sequential modeling

Abstract: Sequential Neural Processes (SNP) is a new class of models that can meta-learn a temporal stochastic process of stochastic processes by modeling temporal transition between Neural Processes. As Neural Processes (NP) suffers from underfitting, SNP is also prone to the same problem, even more severely due to its temporal context compression. Applying attention which resolves the problem of NP, however, is a challenge in SNP, because it cannot store the past contexts over which it is supposed to apply attention. In this paper, we propose the Attentive Sequential Neural Processes (ASNP) that resolve the underfitting in SNP by introducing a novel imaginary context as a latent variable and by applying attention over the imaginary context. We evaluate our model on 1D Gaussian Process regression and 2D moving MNIST/CelebA regression. We apply ASNP to implement Attentive Temporal GQN and evaluate on the moving-CelebA task.

Under review as a conference paper at ICLR 2020
ATTENTIVE SEQUENTIAL NEURAL PROCESSES
Anonymous authors
Paper under double-blind review
ABSTRACT
Sequential Neural Processes (SNP) is new class of models that temporally ex- tends Neural Processes (NP) and can meta-learn a sequence of stochastic pro- cesses. This learned function however under-fits the provided contexts as also the case in NP. Applying attention to the contexts resolves this in NP but sim- ply extending this SNP and applying attention on a buffer of context history is sub-optimal as our findings show. In this paper, we propose Attentive Sequential Neural Processes (ASNP) which resolves the under-fitting in SNP by introducing a novel imaginary context, modeled as latent variable, over which the attention can then be applicd. We evaluate our model on ID Gaussian Processes regres- sion and 2D moving MNIST/CelebA regression. We apply ASNP to implement Attentive Temporal-GQN and further evaluate on the moving CelebA task.
INTRODUCTION
Neural Processes (NP) (Garnelo et al. 2018b) combine the strengths of neural networks and Gaus- sian processes (GP) such that, like Gaussian processes, it is flexible in learning new stochastic process test time and also provides fast O(1) prediction speed like neural networks. Learning from small datasets of multiple tasks (i.e., multiple stochastic processes), NP can be seen as a prob- abilistic latent variable framework for meta-learning. Sequential Neural Processes (SNP) (Singh et al. 2019) extend the power of NP to sequence of stochastic processes thus introducing new class of sequential latent generative models. It targets a large class of problems where the sequence of stochastic processes is governed by underlying transition dynamics and where learning to transfer information between stochastic processes is useful.
Despite these strengths, the remained question is whether NP and SNP can scale more complex stochastic processes when the individual observations about the true process are highly partial, which prevails in many situations. For example, an agent in a large complex landscape can only obtain very limited information from its immediate surroundings and it is desirable that it still maintains an accurate global scene representation that could also be changing with time. Studying NP from this perspective. Kim et al. (2119) demonstrates that the under-fitting, problem significantly deteriorates the performance NP. To address this, ANP (Kim al. 2019) and the method in /Rooenbaum et al.) 2018) augment attention to NP and GQN (Eslami et al. 2018), respectively.
Given the severe performance degradation of NP due to the under-fitting, an important key question is whether SNP would also suffer from the under-fitting problem not, and its follow-up questions such as if so, how severely and in which settings would it happen? and how can we resolve the problem-woud
the same attention mechanism that worked for NP work for SNP well? In this paper, our goal is to answer these questions. We achieve this by not only showing that our proposed attention for SNP significantly improves the standard SNP under the under-fitting settings (see Fig. I}. but also by making a stronger claim i.c., in a sequential setting like ours, it is not optimal to per- form attention on a memory buffer that simply stores all the observed contexts, Instead, through our experiments, we claim that attention should be performed on a memory that is also updated (learned) at every time-step. Consequently, a) this memory would learn to store an optimized representation of the past geared towards prediction and b) it would require fewer storage locations as it does not need to naively store every observed context points.
To implement such memory, we introduce the concept of imaginary context that augments the set of real context points observed at any time-step. The imaginary context contains a fixed number of context points and is modeled
random variable sampled at every time-step. Given query, we then apply attention on the union of the real and the imaginary contexts to generate predictions. We call the proposed model Attentive Sequential Neural Processes (ASNP). In experiments on

---

Under review as a conference paper at ICLR 2020
SNP
ASNP
Contoxt
Tarpet
SNP
ASMP
Figure 1: Demonstration of under- fitting in SNP by comparing against ASNP for ID Gaussian Process and 2D moving CelebA regression tasks. Samples in the first and second rows from t = 0 and 4. In the context column for CelehA, white pixels for the background drawn blue for visualization.
ID regression tasks, we validate that SNP which simply attends on the memory buffer of all the past context points under-performs the proposed ASNP endowed with imaginary contexts. Then in a comprehensive set of experiments on 1D and 2D regression and rendering tasks, we demonstrate ASNP's performance gains over NP, ANP, SNP, GQN, and TGQN in different context regimes.
Our main contributions are as follows. a) We show that SNP suffers from significant under-fitting by testing in settings when the true stochastic process complex (e.g., complex CelebA faces moving on white canvas) or when each observation is highly partial (e.g., each context point is just one pixel). b) We introduce new model, Attentive Sequential Neural Processes (ASNP), that resolves the problem of under-fitting by augmenting SNP with an attention mechanism. To implement this, we introduce the novel concept of imaginary context. We also empirically demonstrate that SNP which simply performs attention on a lossless memory of all the past context points still under- performs the proposed model and therefore the use of imaginary context a better design choice. c) Using the attention paradigm introduced in ASNP, extend TGQN to Attentive TGQN (ATGQN) for scene rendering. d) We show that ASNP outperforms the baselines in various ID and 2D tasks performing future prediction and meta-transte transfer learning i.e., making simultaneous use of the past knowledge and the current contexts.
2 BACKGROUND
Neural Processes (NP) Garnelo et al. 2018b) learns to learn a stochastic process P that maps an input dd to random variable y E Rdd It does using set of context observations === (Xo, Ye) = {I,, y1 (Ez(C)) Here, I(C') is the set of indices for the elements in set C. Specifically, to learn stochastic process from the context, NP uses an inference distribution P(=[C) and then uses the representation prediction given query x. This is modeled by p(y|æ, 2). The full generative process of NP conditioned on the context can be written as:
~ P(=[C) of the inferred stochastic process to generate the
P(Y X( P(YIX.2)P(EIC)))
(1)
where P(Y|X,2) = IIEEID) P( Fz, 2) and D contains the targer observations D = (),Y)= {x,, y4 (e))))) The training data for NP is obtained by drawing a stochastic process and then sampling (C, D) from the sampled stochastic process. Treating each stochastic process task, NP can be seen as a probabilistic meta-learning framework.
Generative Query Networks (GQN) [Eslami et al. 20188) uses modeling similar NP for the problem of learning representation and rendering of 3D scenes from a set of partial 2D observations. In GQN, a query z becomes camera viewpoint
3D environment and an output y corresponds to an image taken from the viewpoint The latent variable in the original GQN are query guery-dependent and may thus produce uncorrelated generations across queries. Consistent GQN (CGQN) (Kumar) et al. 2018) resolves this problem by having single stochastic scene representation. For simplicity, we use the term GQN to denote CGQN in the remainder of the paper.
Attentive Neural Processes (ANP) (Kim et al. 2019) resolves the problem of under-fitting in NP. To this end, ANP uses two context- -enoxxt-nncoiing paths. The first path is the same as the scene encoder in NP that summarizes all the context points agnostic to the target query and produces single vector
2

---

Under review as a conference paper at ICLR 2020
of the global representation by average pooling. The other path computes representations for each target query using query-dependent attention. The set of representations returned from the set of target queries X may be written = fanend (X, vc), where vc: are the encoded representa- tions of C' via MIP or self-attention. The attention function sattend is a key-value based attention which computes a response for each query Oj E X by first computing its similarity with the key set XC as W = far (k,, qj) and then performing a weighted-sum on the value set i.c., 8/ = >; WyyVi.. Here, the keys and values are the context queries ø and the representations vo. respectively. The generative process is as follows: P(Y C* f P(Y|X, fatend (X, Xc. vC) P( P( Kim al.) (2019) propose ANP as a general model class which may be realized using different atten- tion mechanism choices such as Dot-Product Laplace or Multi-Head (Vaswani et al.) (017). ANP
needs more computation than NP but Kim et al. 2019) mitigates this by showing faster training convergence than NP.
Sequential Neural Processes (SNP) (Singh et al. 2019) extend the capabilities of NP to model stochastic processes which may be decomposed sequence of stochastic processes P1, P2. Pr such that P. at time t depends on the past processes P&t through some underlying temporal dynam- ics. At any 1, 'e is modeled 2₁ which depends the current context C, and the past processes Eer using the transition model p(zz| G.). This enables us to use the general past trend as well as the current context to meta-transfer learn the process Pt. Given & and query Tt, the prediction Yt is generated using the observation model p(ya (4x zt). In the SNP framework, at time-step t, we cannot directly access the past contexts, but we can still directly access the current context Ct. Thus, to model Pe, we can use the stochastic encoding together with the direct encoding of C.. Alter- natively, this allows us to write the observation model as p(ye (r, 2t, Ct). Combining the transition and the observation models, the complete generative process of SNP may be written as follows.
Z|X, II pluc ze, C,)p(Eslees,, C+).
(2)
3 ATTENTIVE SEQUENTIAL NEURAL PROCESSES
In this section, we describe the proposed Attentive Sequential Neural Processes (ASNP) that aug- ments SNP with attention mechanism.
3.1 GENERATIVE PROCESS
We recall that ANP uses the observation model p(y|2,, 2, C') which, based on the query x, performs at- tention on the context points in C. In SNP, which uses the observation model p(ye [2,, 2t, Ct), an analogous ap- proach would use I2- de-deeendent attention on Ct. However with this approach, attention is not effective because the current context is likely to be limited or even empty. In- stead, we want to perform attention not only on the cur- rent but also on the past contexts. But to do that, we found that simply attending on a memory buffer of context his- tory does not produce enough gains and leaves room for further improvement (see Appendix D.3). Instead, our hypothesis is that attention should be performed sequentially-updatee memory so that the representations optimized for prediction and are more size-efficient. To realize this, we propose augmenting the set of real context points CT with a set of imaginary context points Cj resulting in the full context set C. =Cr U Cj so that the observation model can now perform attention on this augmented C,- Here, C. = (X;, U?) consists of imaginary queries (or keys) X; = { 1 and its corresponding
Uncortaiity
CUC
Imagnnay
Figure 2: Illustration of imaginary context. Conditioned on real context, the imaginary context generated which reduces the un- certainty at important points in the query space.
{ where K is a hyperparameter similar to the number of slots in memory. To generate Cj we introduce novel and attentive transition model P(C) CC2,, CT) treating Cj as a random variable. In this way, Cj is a noisy reconstruction of the past sampled using an atten- tion mechanism on the past contexts (real and imaginary) and on the real contexts of the currett
values U/- = {ui 1K
3

---

Under review as a conference paper at ICLR 2020
time-step. Augmenting SNP with these, the generative model for ASNP may be written as follows.
T
PlY, II P(yl 2e, C)) C.
(3)
Implementing the transition model P(z Ct). With this model, we obtain the global represen- tation 2, which is a query-agnostic encoding of the current stochastic process. This module can be considered similar to its counterpart in SNP without attention. Note that is design choice whether to choose C, or C7 for the conditioning context of this module. If we use CT. 2t becomes global context-encoding of real contexts. Otherwise, it becomes an encoding of the combined contexts.
Implementing the context imagination P(C) Cet, Cf). To implement this, we keep track of the K' imaginary context points at all time-steps and their generative model is as follows:
K
K
P(C Cet, CT) = II k=1
P(ai,es a Cet, C.)
II k=1
Pluh,alai, Cet, P() CF). (4)
For each imagination slot k, we first sample the key Fi,u- For this, we use an RNN that takes the past imag- inary inputs X;_ and the encoding of the real contexts of = innter (CT) to produce hidden state hyy from which we then sample the imaginary key set X3. Given these keys, the corresponding imaginary values 4i are generated next. For each imagination slot k, we maintain
Algorithm 1 Attention with Imagination
A = RNNNRNNXXXX 2-1 ((x; of), (
X -NS (ne ) (h²
h[, RNNN((E 11-1,k) 1 h2-1k)
of = fanena (T), St)
"i,k ~ NNI (al fo (at,))
an imagination-tracker RNN that takes (xf,a u/ 1f-1,x) ) as input and produces a hidden state hý, These hidden
Txa Samena (21, Ct)
states act as intermediate imaginary values and we use them to construct a key-value set S = {( "",,) U self-attention using only the imaginary st_, attention queries and obtain the corresponding atten- tion encodings d Such encodings therefore also model the interactions among the context points. From these, we sample the final imagination values ui,e ~N ft
v()) On this set, we apply
Implementing the observation model P(yelz,, 24, Ct). Taking Iy as the attention query, we attend on C., the union of the real CT and imagined C; context, as the key-value set and obtain the attention encoding Tx, Jatena C.). This is done after each real context E X; and y E Y? is encoded using fy-u- The attention encoding Tz, is then concatenated with the global encoding = to perform the prediction yr. Algorithm D shows the described process of context imagination and encoding.
Discussion. An interesting interpretation of the proposed modeling can be drawn in comparison to the memory retrieval process in the human brain (Eichenbaum, (017). As is well-known in neuro- science, "human memory is not a literal reproduction of the past, but instead relies on constructive processes that are sometimes prone error and distortion" (Schacter) 2012). Our model resem- bles this process in the sense that it (i) compresses the past observation and importantly what it has imagined in the past through an RNN (similarly memory consolidation), and then after observ- ing current context, (ii) recalls from the compressed memory not a simple copy of the past but a constructive recreation of representations that is optimized help prediction as illustrated in Fig.
3.2 LEARNING AND INFERENCE
Due to the intractability of the true posterior, ASNP is trained via variational approximation with the following posterior approximation:
T
P(Z,C' D) OZZZEE C, C",
(5)
where Q(C: C", D) QUU C", D)Q((I) C", D) and D (X, For training, the following evidence lower bound (ELBO) is maximized w.r.t. 0 and Φ:
LASNP(0, Φ) EQstnn.yoorr.) [og Polyd 21, Cj cT)
-EQs(zen [KL (QooE, C² D) Po(ai, Cilzer, C2) (6)
4

---

Under review as a conference paper at ICLR 2020
10 GP Segrecin Sectato
1D CPP Regoosioa Sorara
OF Hagremssn Scematio Fd)
0.7
NP
0.2
10.
AND
-0.4
-0.7
4.77
NP ANP
ENP
ASKP
Figure 3: Top: Negative oigliike (NLL) for target points at each time- for ID regression. Bottom: Samples for ID regression task (c) at 33. Dotted line: True function. Blue line: Predicted function. Shaded light-blue region: Uncertainty. Black dot: Current context points at 33. Blue dots: Past contexts where color intensity reflects the recentness.
For backpropagation, we use reparameterization trick (Kingma & Welling 2013) for continuous variables. Derivation of eq, is provided in Appendix A
4 RELATED WORKS
Learning to learn stochastic processes from few data points at test time has become attractive as meta-learning recent times. In this line work, Conditional Neural Processes (CNP) (Garnelo) al. (018a) models regression function but without global latent which causes inconsistency be- tween values for different queries. NP (Garnelo et al., 2018b) resolves this by proposing an explicit global latent path. While CNP and NP frameworks demonstrated simpler tasks, GQN (Es-) lami et al. 2018) develops it to render 3D scenes. Like CNP, GQN also produces inconsistency between queries due the absence of global latent. CGQN (Kumar et al. 2018) resolves it by introducing global scene latent. To address the challenge of scalability to larger and more complex stochastic processes, ANP (Kim al. 2019) identifies and resolves the problem of under-fitting using query-dependent attention. Similarly, Rosenbaum et al. (2018) introduces attention into the GQN model to render complex 3D scenes large peocedurally-genrrate
maps as Minecraft. To extend the model to stochastic processes decomposable as sequence of stochastic processes, SNP (Singh et al. 2019) was proposed. It was then used to extend GQN to TGQN for rendering dynamic 3D scenes. Functional Neural Processes (FNP) (Louizos et al.) (2199) was proposed to learn a graph dependencies between pre-selected set of points and the training points for modeling distributions over functions.
The concept of imaginary context i.c. pseudo-context, which act as trainable representative points related to the inducing points in Set Transformer (Lee et al., 2019) and sparse GPs (Snelson & Ghahramani] 2006) Titsias 2009). VampPrior (Tomezak & Welling 2017) also uses few trainable pseudo-inputs resolve over-fitting, over-regularization and high computational complexity. The analogous reference set used in FNP to model a graph of small number of points also motivated from the inducing points of sparse GPs. With regard the use of trainable memory. Differentiable Neural Dictionary (Pritzel et al.) 2017) stores and updates the previous trajectories as key-value pairs to quickly learn a variety of environments.
5 EXPERIMENTS
We evaluate ASNP the following selection of tasks: a) ID Gaussian Process (GP) regression b) 2D moving MNIST (LeCun et al., 1998) and 2D moving CelebA (Liu et al. 2015). On these tasks, evaluate and compare the proposed model against NP, ANP, SNP the baselines. Each task is evaluated with three scenarios, which is used in (Singh et al. 2019) for ID GP regression: (a) To evaluate the model's ability extrapolate the future, provide high-amount of context
5

---

Under review as a conference paper at ICLR 2020
20 Moria MNIST Segresten Scouatb 6o)
ID Sintng Mhill Regesam Screaro (8)
2D MonERE MISIST Regresson Scesate
NP ANP
-00
(L-100)
-12
1ime
233 Moving Draba Regnecios Sconarts (s)
3D Mosing CabRA Regmodoe Somerts th)
3D ColebA Segreson Scanatto
NP
ANP
-20
Timee
Figure 4: Target NLL each time-step for 2D moving MNIST (Top) and CelebA (Bottom) regression.
in the first 10 time-steps out of the 20. (b) To test the model on its ability to track the dynamics using intermittently arriving context, we provide high-amount of context in 10 randomly chosen time-steps out of the 20. (c) To test the model's ability to gather and make use of low amount of context received over long segments of time, we provide low amount of context in 45 randomly chosen time-steps out of the 50. In each setting, the remaining time-steps are used for prediction. The details about model architectures can be found in Appendix B.I
To test the benefits of attention for rendering 2D images, we used moving CelebA because it has higher uncertainty when given partial knowledge in comparison to an alternative data set such as moving simple shapes (e.g. circle). We test on the scenario (c) as described above, but with shorter sequence lengths and smaller context sizes. Details are explained in the section on 2d rendering.
We first evaluated is an extension of the GP data set through the addition of linear dynamics on the kernel hyper- parameters as used in (Singh et al.) 2019). In each transition, introduce stochasticity. For scenarios (a) and (b), the number of context observations n is randomly selected in [5, 50] or empty set. The number of target observations
ID Regression:
method on sequential ID Gaussian Process data set. This
add a small Gaussian noise to
is randomly selected in [1,51 n). For scenario (c), n is 1 or O, and mn is chosen from [1, 11 m)). The target set subsumes the context. The more details about this task is described in Appendix C
In the quantitative results of Fig 3 SNP outperforms NP as reported in (Singh et al., 2019). On the other hand, in scenario (a), the performance of ANP degrades steeply after the context are removed since it does not model the dynamics explicitly. Even though it appears to outperform NP and SNP, this can be credited to the attention in ANP that prevents underfitting (Kim et al., 2019). ASNP shows better performance for all the scenarios, outperforming ANP with a greater gap in scenario (c). Further, it saturates the fastest among the tested methods for the entire scenarios as described in Appendix D.I. We hypothesize that when the dense context is given each time-step in (a) and (b), ANP can fit well as does not need to rely on the dynamics much. But in scenario (c), the context is low in each time-step and the model must capture dynamics to make the best use of the context seen thus far.
In the qualitative results of Fig 3 we illustrate a sample of validation set in scenario (c) with pre- dictions from ASNP and the baselines. As shown, even for the points not shown recently, ASNP predicts better than the baselines with lower uncertainty. ANP and SNP fail to predict those points well. This shows that the update of the imaginary queries and the representations is necessary over and above the sequential update of the global representations or the expensive modeling the entire past using the time information. More pieces of evidence about a useful role of imaginary context are shown in Appendix D.2 and [D. where comparisons with larger size baselines and ASNP without imaginary context are described and ASNP outperforms them. It also shows having more param- eters in baselines is not sufficient and imaginary context plays a significant role. More qualitative results are in Appendix E.

---

Under review as a conference paper at ICLR 2020
20
92
larget
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9
kp
APP
She
7
7
4
Ashi
T
9
9
9
9
4
4
?
12
15
18
24
20
36
20
42
Contect
Lanpe.
8
Alp
Sbp
Figure 5: 2D moving MNIST (Top) and CelebA (Bottom) regression samples from scenario (c). In each Context row, white pixels at the background are drawn blue visualization.
2D Regression: We take a step further to evaluate our model on a more challenging 2D setting where the queries are the pixel locations and the regression is over the pixel values of images. The image a 42 42-sized black canvas with a moving MNIST digit (sized x 28) or a cropped CelebA face (resized to 32 32). The digits or the faces start moving from random location heading towards a direction randomly selected at t = 0 with a speed bouncing perfectly upon encountering wall. Like in the regression task, small Gaussian noise is added to the location obtained after each transition. Same as the regression task, 3 context regimes designed. In regime (a) and (b), is either 0 or n E [5, 500) while is drawn from [1, 501 - n]. In (c), n is either 0 or 30 and m is drawn from [1,51 51 - r²]. The target set subsumes the context.
pixels per time-step and
We present the quantitative results for 2D regression tasks in Fig. 4 From these, we infer the following points. We observe that SNP performs better than ANP except in the early time-steps of scenario (a) when large number of contexts points are shown without interruptions. We reason that in the presence of such ample context points, NP and SNP still suffer from under-fitting while ANP can make an effective use of those points without having to model the dynamics. But when the task setting demands that the model capture the dynamics, in task (c), both NP and ANP perform poorer than SNP and ASNP. This is more pronounced in the 2D setting than in the ID because tracking dynamics of images without explicit temporal modeling is harder. We further note that MNIST digits are simpler than the CelebA faces. Therefore in the latter case, SNP struggles with underfitting and ANP struggles with modeling the dynamics, causing them both to furnish similar performances in task (b). ASNP does not suffer from either of these problems and it outperforms all baselines in all the tasks. In scenario (a), we also note a sudden performance drop in NP, ANP and ASNP. While unsurprising in NP and ANP, it also occurs in ASNP due to sudden switch-over to a purely imaginary context. This drop does not appear in the ID tasks since the number of context points provided are fewer and therefore ASNP is less impacted by their absence.
We also evaluate the effect of varying the imaginary context sizes on the moving MNIST data set for the scenarios (a) and (b). We observe that a larger à yields better performance. This gain, however, saturates quickly as the improvement between & = 9 and h: = 100 is not large. This shows that the imaginary context can represent the stochastic process with only few points.
7

---

Under review as a conference paper at ICLR 2020
2D Moving CelebA rendering secnario
20
24
GQN TOQN
ATGON
18-
Torget NLL
Context
14
Target
10
GON
TGON
10
16
19
ATGON
Figure Left: Target NLL moving CelebA 2D scene rendering task for scenario (c) and Right: examples.
In Fig. 5 we present the qualitative results on the two data sets for scenario (c). Here, we note that NP and ANP can produce the digit location but not the appearance. SNP captures the appearance of the digit but not clearly. Although ASNP also produces fuzzy appearance in the early time-steps, the digit becomes identifiable by 10 and the details keep getting clearer over 20. We also see a similar trend for the moving CelebA task. More illustrations of scenarios (a), (b) and (c) are included in Appendix F.2
2D Scene Rendering. We also introduce the attention paradigm of ASNP into Temporal GQN (Singh al., 2019) and test it on the moving CelebA rendering task. We call Attentive TGQN (ATGQN). To design this, we use the Temporal Temporal-ConvDRAN ConvDRAW as in TGQN (Singh et al.) 2019 for encoding a global latent. For computing query-dependent representation, we apply the ASNP's attention encoder with imagination to get a scene-wise matrix. We use the same decoder as TGQN. While it is limited solution working on the task where Ye can be divided as points with queries, it resolves the underfitting for overlapping pixels between the context and the target. Unlike NPs, partial overlap exists, which is difficult to predict by scene-wise attention. More details about the model architecture is described in Appendix B.2
The canvas size of the data set is 80 x 80 and cropped face image size is resized in 64 x 64. The randomly selected at
direction of motion and initial position speed. The size of output is 32 x 32. The sequence length is 25. One context view is provided to the model each of the randomly chosen 20 time-steps out of 25 time-steps. and an empty is given on the remaining 5 time-steps. The target size m is drawn in the range [1, 11 n].
with 13 pixels per time-step
In Fig. 6 we show quantitative and qualitative results for scenario (c). We can see that ATGQN outperforms TGQN because for the overlapping areas, ATGQN resolves the underfitting. The per- formance gap between TGQN and ATGQN, however, is smaller compared to the previously seen and 2D regression tasks. A similar trend is observed in scenario (a) whose results are in Appendix D.4 The reason for this is low uncertainty in the presence of the context that provides a wide view of the canvas. This and the small number of context causes lesser underfitting than what would have arisen had the context observations been highly partial as in the regression tasks. More examples included in Appendix E.3
CONCLUSION
In this paper, we addressed the problem of under-fitting that affects Neural Processes and Sequential Neural Processes. Although this is resolved in the former by ANP, it is not possible to resolve it sufficiently in SNP with direct application of attention on context history. We introduced Attentive Sequential Neural Processes which comprises novel memory mechanism of imaginary context to resolve the under-fitting geared towards making good predictions with low uncertainty. In the experiments, the proposed model shows superior performance on various tasks for a number of sequential scenarios. Since ASNP is a model based on attention, its limitations include those of any attention-base based model. One such case is the scene-wir attention. Scene-wise attention cannot fit properly for partial overlapped region between context and target. Although partially solve it with pixel-wise attention, it would still be interesting to encode query dependent representation for scene observation.
not only compresses the past knowledge but does in a way that
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, al. Tensorflow: system for large- scale machine learning. In 12th {USENIX) Symposium on Operating Systems Design and Imple- mentation ((OSDI) 16), pp. 265-283, 2016.
Howard Eichenbaum. Memory: organization and control. Annual review of psychology. 68:19-45, 2017.
SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Gar- nelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, al. Neural scene representation and rendering. Science, 36066394):1204-2221 2018.
Marta Garnelo, Dan Rosenbaum, Chris J Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo Rezende, and SM Eslami. Conditional neural processes. arXiv preprint arXiv:1807.01613. 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint arXivv/807.01662, 20.
Karol Gregor, Ivo Danihelka, Alex Graves, Danilo Jimenez Rezende, and Daan Wierstra. Draw: A recurrent neural network for image generation. arXiv preprint arXiv: arXiv:1502.0463: 04623, 2015.
Karol Gregor, Frederic Besse, Danilo Jimenez Rezende, Ivo Danihelka, and Daan Wierstra. Towards conceptual compression. Advances In Neural Information Processing Systems, pp. 3549-3557, 2016.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo,, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. arXiv preprint arXiv:1901.0576) 2019.
Diederik Kingma and Max Welling.
Auto-encoding variational bayes.
arXiv preprint
1312v:1312.61 14, 2013.
Ananya Kumar, SM Eslami, Danilo J Rezende, Marta Garnelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistent generative query networks. arXiv preprint 1rXXiv:8077.025 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio. Patrick Haffner, et al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324, 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-invariiat neural networks. In Interna- tional Conference on Machine Learning, pp. 3744-3753, 2019.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference computer vision, pp. 3730-3738, 2015.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. arXiv preprint a0Xiv:1906.00000 08324, 2019.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference machine learning (ICML-10), pp. 807-814, 2010.
Alexander Pritzel, Benigno Uria, Sriram Srinivasan, Adria Puigdomenech Badia, Oriol Vinyals, Demis Hassabis, Daan Wierstra, and Charles Blundell. Neural episodic control. In Proceedings of the 34th International Conference on Machine Learning-Von Volume 70, pp. 2827-2836. JMLR. org, 2017.
Dan Rosenbaum, Frederic Besse, Fabio Viola, Danilo J Rezende, and SM Eslami. Learning models for visual 3d localization with implicit mapping. arXiv preprint arXiv:1807.00 1807.03/49, 2018.
Daniel L Schacter. Constructive memory: past and future. Dialogues in clinical neuroscience, 14 (1):7, 2012.

---

Under review as a conference paper at ICLR 2020
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. arXiv preprint arXiv: 10264, 2019.
Edward Snelson and Zoubin Ghahramani. Sparse gaussian processes using pseudo-inputs. In Ad- vances in neural information processing systems, pp. 1257-1264, 2006.
Michalis Titsias. Variational learning of inducing variables in sparse gaussian processes. In Artificial Intelligence and Statistics, pp. 567-574, 2009.
Jakub M Tomezak and Max Welling. Vac with a vampprior. arXiv preprint arXiv: 1rXX::1705.07120.22 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008 2017.
SHI Xingjian, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In Ad- vances neural information processing systems, pp. 802-810, 2015.
10

---

Under review as a conference paper at ICLR 2020
APPENDIX A
ELBO DERIVATIONS
Jog
P(Y, C(
= log (0
QZz,"" T
P(yilze, 21, (a)lea C) C.
Qualieei Ci Cr, D)Q(U) C", DI Cr,)
C)P C CT. P(yil 21, 2,,
log EQ(Z,090r,))
1=1 Q C2 Cr, D)Q(U) C22, Cr, D. CT
P(ui 24, (2)l(e C(), Can Cd, C log
=Bqiz,y0or
Q(E1kze1, C,, Cr, X)Q(U Ca Cr, 2)Q D)
[log P( (e: Re, Ci) - llog Q(zekzet, Cj. "","",D Q(Ui) C'ess CV, Pizelzzee Ct)
=Eq1z,09100 D)
log P(Ui|Xp Cer C%)
log C(X
Bii Quaulzze C. [log P(yclut, 2e, Ct)]
1=1
log Qlal2er,C% Cj, Cr,D). P(allzetCC. Ct) log Q(U:IX), C2 P(UN| Caa, CT)
Tiy CC
E QUX,1 O CC.D)
log Q(X:CC2 C)
-II (rx
P(Xx Ct)
T
-E CTT htt.ee/eeeeeenc......000CCCCC Cr,)) D) [log P(yel 2. Ct)
1=1
P.rciccc.e...1112333:ttte....... wttp://ww......ooooooo.ooo..oooa (ie.el/l.....ll...cc.c....CCCCC P(zlz<c)))
POAN://AAA0EEEE00/0///CCCCCCC.C KIC C CT, ) P(U) Ccn, CF))
C" - I 000 t KL(Q C². D) C()
T
Σ BITT- Cf. 1=1
[log P(yl 21, Ci)
aer Ca.. C... (IQ C,,"", P(alzces C.)
-ETI @C 1
KU KLCC C", P(C)C),, ( C,))
KL
T
- BTT (f
[log P(yel 2t, Ct)
1=1
- -T Ql2, Claee a. (L (Q1 Ce Cen Cen, C", D) P(zs Cilese, C))
T
Σ Bestn [log Po(un 2,, Cú CT)
1=1
- -Equleen
[KLL (Qo(2, (Qo(,, C/CC", D) Pa(a, Cilzen Z2 GD)]
11

---

Under review as a conference paper at ICLR 2020
where Q0(21, CilC",Di Qolzen 22 = II Qzar, Zelee Oze C", D) for simplicity.
IT= Qtrrkeer C, ("D)) C", and
APPENDIX B MODEL DETAILS
Imaginary Context Update
Encoder
Decoder
Key
Value
Query
Decoder
Context
encoder
Crose- Attention
Cross-
Y
Query
Global
Value
Serapling
Figure 7: Model architecture of Attentive Sequential Neural Processes (ASNP) imaginary context update (left) and encoder/deccder (right)
B. NPS
Every models share basic architecture (e-g. context encoder and decoder). They have two encoders. One of them is for global latent. It consists of 3 layers MIP with ReLU (Nair & Hinton 201O) used as encoder and layers MLP is used to encode and sample latent. Another encoder is consisted of 6 layers MLP with ReLU activation function and mean function for NP and SNP and layers MLP with ReLU activation
attention module for ANP and ASNP. Decoder is consisted function and layer MLP is used sample with uncertainty.
For non sequential models (NP and ANP), time is encoded as normalized float scalar, e, = 0.25 + 0.5(t/T), where T is the length of sequence of data. After encoding, it is appended in query as x' = (xx
For attention models (ANP and ASNP), Multihead attention (Vaswani et it showed the best performance in variety of attention methods (Kim et al. attention modules, which shares parameters.
used because 2019 ASNP has two
2017
For sequential models (SNP and ASNP), they uses same temporal architecture for global latent. LSTM with a default setting of TensorFlow (Abadi et al. 2016) used and the dimension hidden unit of LSTM is d, where d, is the dimension of representations. For ASNP, two RNN modules for the imaginary queries and representations is LSTM with default setting of Tensorflow. The dimension of hidden unit of LSTMs is d,.. Here, the number of state of LSTM for representations is batch size.
The dimension of representation d, is 128 and the dimension of query d, is 1 for ID regression and 2 for 2D regression. Note that for non-sequential model, d, is 2 and 3 by attaching encoded time ex. The number of imaginary context 25. The initial imaginary context Cj is trainable parameters. Learning rate 0.0001 and batch size is 16 for 1D regression and 8 for 2D regression. The diagram for ASNP architecture is in Fig. 7
B.2 GQNS
For an encoder, tower representation (Eslami et al. 2018) is used. A decoder similar between models (GQN, TGQN and ATGQN) based on the decoder of CGQN (Kumar et al.) (2088) that is deterministic generative model working on auto-regressive manner. The different is inputs that are described in bellows.
For GQN, time encoded and appended in query same to NP and ANP. To sample 2,, we apply convolutional DRAW(ConvDRAN) (Gregor et al. 2015 2016) like GQN (Kumar et al. 2018)
12

---

Under review as a conference paper at ICLR 2020
To render g. the decoder imputs 2,, X, and by that is output of the decoder. oa is averaged representation that is an input for every target queries.
al. 2019) is applied sample 2t with ConvL- STM (Xingjian et al. 2015) state, dig We take RSSM to implement as (Singh et al. 2019). The decoder imput and h₂ are independent to target queries.
For TGQN, Temporal- ConYDRAW (Singh
of the global latent zz. the target queries X, and ConvLSTM state h4, in which,
For ATGQN, global latent is sampled same to TGQN and query dependent representation of ASNP is used. To make pixel-wise path, we split scene view as pixel view with queries per each pixel. When many context are given, overlapped between context exists. We leave to optimize batch operation. We encode the pixel-wise representation with 2 convolutional layers to make same dimension with scene-wise,
dimension of which (scene-height/ /4, seneehhight/4,33 The input of decoder are X, and T, {ri heiD)) that is query dependent representations for each target query 2!
The dimension representation d, 128. The number of steps for (Temporal) ConyDRAW to sample 24 is 3 and the dimension of is 4. Note that is the output of the last roll-out of (Tem- poral) Conv DRAW. For TGQN and ATGQN, the number of hidden unit for ConvLSTM is 40. The dimension of hidden unit on the decoder is 32 and the number of steps for auto-regressive decoding is and output is cultivated. The query size (for GQN, it is 3 with e,). The number of pseudo context is 100. Learning rate and batch size 0,0001 and
APPENDIX C GAUSSIAN PROCESS DATA SET
The kernel hyper- ppper-pprameters, length-scale I and kernel-scale are selected randomly [0.7, 1.2) and [1.0, 1.6] for scenarios (a) and (b). For scenario (c), and a are chosen in [1.2, 1.9) and [1.6, 3.1] at
0 in
0. The true underlying dynamics of the kernel hyper-parameter: Al and Ao are in [-0.0,, 0.03] and |-0.5,, 0.05] chosen at = 0. Gaussian noise on the dynamics is ~ N NN00, 0.1).
APPENDIX D ADDITIONAL EXPERIMENTAL RESULTS
D.1 LEARNING CURVE ON GP TASK
ID regesccin
GP regreddon
GP regoeggdnoon (C)
NV
3.5
ANP
SNP ASNP
0.2
Tage:: NLL
0.1
-0.1
1
1.8
-303
Bhommaddd) .10F
ID
1D CP task
0.2
3.2
40.1
ASNP
Tage ML.
-0.3
-2.1
0.5
-0.7
-37
0.5
-13
0.2
0.4
0.5
Socondica trandedd thonando
Seconditio Seododiioluudeeedee wwwdeedddddsdddddee.e thessado) .10>
Secondinn cillioos)
Figure 8: Top: Target NLL for iteration. Bottom: Target NLL for wall clock time.
In this section, we show target NLL for iterations and wall clock time (Figs. 8) for ID GP data set. ASNP and ANP computation complexity bigger than NP O(1) due to attention. ANP needs Om En n,) where n, is the context size at time-step t. Because it needs to attention to entire context current. ASNP computation complexity is ((n++))m) for generating ry and ((n+k)k)
13

---

Under review as a conference paper at ICLR 2020
for updating the pseudo context. When the sequence of data T is not long, ANP computation complexity is smaller than the complexity of ASNP. However, the learning curve for wall clock time shows ASNP quickly saturates lower loss in same time.
D.2
EVALUATION ABOUT BASELINES WITH VARIOUS PARAMETER SIZES FOR GP TASK
gurannter aizz teaing UP (g)
paranneer soingg for GP (b)
Varisca panamater ane teating for
0.5
05
NIN230
022
02
1.5
ANPU220
ANP5512)
TegeNIL
SNP1128)
0.5
SNPG512)
ASNPU12
Figure Target NLL comparison with baselines having various sizes of d, (the number in bracket) at each time-step for ID regression.
In this section, we evaluate our model with baselines implemented with larger size of neural networks. The representation dimension of baselines (i.e., NP, ANP, and SNP) are selected as d, = 512. This is chosen the performance improvement saturates beyond this value in (Kim et al.
2019. .. The remaining settings are the same as described in the previous sections.
Fig. shows that NP and SNP implemented with larger network perform better than their smaller sized counterparts. On the other hand, larger sized ANP shows similar performance to its smaller counterpart. We thus hypothesize that the smaller model capacity is enough to train ANP on this task. ASNP, although implemented with smaller network, still clearly outperforms the rest in scenario (c) even against baselines of larger sizes and this gap gets bigger as the time goes on. It shows that the baselines having bigger network is not sufficient represent sequence of stochastic processes and temporal encoding of ASNP and the imaginary context plays a significant role.
D.3 ASNP WITHOUT IMAGINARY CONTEXT
Sootin ASNP wabeut Imogiraaionn fhe (P
sithet Theiig ASN viheot Ingiiaaiin for (Pi
Traiig ASN waboos brugination for (PP
0.5g
0.11
00
Terge
0.4
0.4
-0.7
0.7
-0.5
-18
76
10
Time
Tims
Thme
Figure 10: Comparison of target NLL of the proposed ASNP versus ASNP without the imaginary context (ASNPK) at each time-step for ID regression. The number shown in the bracket against ASNPK is K, the number of most recent timestepss whose contexts we store for the attention at any given time-step. K inf denotes that the entire set of the past contexts are attended on.
In this section, evaluate the effect of imaginary context on the ASNP performance. To validate it, we design ASNP that does not produce imaginary contexts but simply stores all the real contexts from K most recent time-steps. We call this model as ASNPK in this section. To validate not just the effect of the imaginary context but also the past knowledge on prediction, we test with K = 1, 3, 5 and infinity. Here, when K infinity, ASNPK stores the entire set of the past contexts like ANP and the number of context to attend of ASNPK is usually bigger than the number of imaginary context per each time-step (i.e.
3, could be up to 150 contexts for scenario (a) and (b) while ASNP attends up to 75.). ASNPK has two RNN modules for the global latent and the deterministic variable, but it attends contexts from multiple time-steps. To optimize the attention performance of ASNPK, we also encode the time information into the queries like NP and ANP.
In Fig. I0] quantitative results for the GP tasks are plotted. As K increases from 1 to infinity, AS- NPK shows performance enhancement which quickly saturates. However, it still under-performs
14

---

Under review as a conference paper at ICLR 2020
with respect to ANP and ASNP. For ANP, attending all the past knowledge (ANP) makes bet- ter performance than what using some past context (ASNPKs except K=infinity.. ASNPK with K =infinity shows similar performance ANP because ASNPK(inf) tightly depends more in- formative attention than temporal encoding. For ASNP, the simple copy of the past cannot be better than the optimized learned representation of ASNP imaginary contexts. It shows that sequential memory update mechanism like our brain is more effective than attention on the lossless copies of real past contexts.
D.4 2D RENDERING TASK FOR SCENARIO (A)
We also test model the moving CelebA rendering task for scenario (a). The sequence length sets as 15 and the context is given to f = 7. The context size n randomly selected in [1, 10) or O.
The target size is randomly chosen in [1, 11 n]. Other environment setting is same scenario (c).
Moving scene rendering
GQN
30
TGON ATGQN
40
207
10
10
13
Time
10
12
Image
Contest
Tarpet
GON
TGON
Figure 11: Top: Target NLL on moving CelebA 2D scene rendering task for scenario (a) and Bottom: exam- ples.
Fig. I1 shows results that are both quantitative and qualitative. Target NLL value of GQN is very high. On the other hand, the generation performance is not the same as the target NLL. The ex- planation for this is inability to predict the complex face image's location. When context is given, ATGQN outperforms TGQN because ATGQN solves overlapping region underfitting. ATGQN however shows similar outcomes to TGQN on non-context prediction. The explanation for this is less uncertainty due to a lot of contextual information.
15

---

Under review as a conference paper at ICLR 2020
APPENDIX E QUALITATIVE RESULTS
E. ID REGRESSION
OF
Figure 12: ID GP regression examples for scenario (c). Columns are NP, ANP, SNP and ASNP. Each row is examples each time-step. Due to space limitations, every 5th time-step is shown here instead of every time-step.
16

---

Under review as a conference paper at ICLR 2020
E.2 2D REGRESSION
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
5
0
5
0
5
0
5
0
5
G
0
5
6
5
0
5
5
0
Figure 13: 2D moving MNIST regression examples for scenario (a). Columns Context, Target, NP, ANP, SNP ASNP, Each row examples each time-step. Due to space limitations, every 2 time-step shown here instead every time-step.
17

---

Under review as a conference paper at ICLR 2020
Z
Z
2
2
Z
Z
Z
2
Z
2
Z
Z
Z
Z
2
Z
7
5
7
7
7
5
7
7
7
7
5
7
7
7
5
7
>
7
7
5
7
7
7
5
7
7
7
7
5
6.
7
7
7
7
5
S
7
7
7
7
5
2D moving MNIST regression examples for scenario (b). Columas are Context, Target, NP, ANP. SNP and ASNP. Each is examples each time-step. Due space limitations, every 2 time-step is shown here instead of every time-step. It is shown less than time-steps have context because it doesn't 't show every time-steps.
Figure 14:
18

---

Under review as a conference paper at ICLR 2020
O
6
0
6
0
0
6
/
/
2D moving MNIST regression examples for scenario (c). Columns are Context, Target, NP, ANP. SNP and ASNP. Each is examples each time-step. Due to space limitations, every 5thh time-step is shown here instead of every time-step.
Figure 15:
19

---

Under review as a conference paper at ICLR 2020
2D moving CelebA regression examples for scenario (a). Columns are Context, Target, NP, ANP. SNP and ASNP. Each is examples each time-step. Due space limitations, every 2 time-step is shown here instead of every time-step.
Figure 16:
20

---

Under review as a conference paper at ICLR 2020
2D moving CelebA regression examples for scenario (b). Columns are Context, Target, NP, ANP. SNP and ASNP. Each is examples each time-step. Due space limitations, every 2 time-step is shown here instead of every time-step. It is shown less than time-steps have context because it doesn't 't show every time-steps.
Figure 17:
21

---

Under review as a conference paper at ICLR 2020
2D moving CelebA regression examples for scenario (c). Columns are Context, Target, NP, ANP. SNP and ASNP. Each is examples each time-step. Due space limitations, every 5thh time-step is shown here instead of every time-step.
Figure 18:
22

---

Under review as a conference paper at ICLR 2020
E.3 2D SCENE RENDERING
C
Target
GQN
TGQN
ATGQN
Target
GQN
TGQN
ATGQN
6
8
10
12
14
14
16
18
18
20
20
22
22
24
Target
GON
TGQN
ATGON
Target
GQN
TGON
ATGON
2
4
6
8
10
12
14
14
16
18
18
20
20
22
22
24
Figure 19: moving CelebA 2D rendering task examples for scenario (c). Columns Image (I), Con-
(C), Target, GQN, TGQN, ATGQN, Each examples at time-step.
23


Reviews:
[{"review_id": "ryeKOFZW5S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper deals with the underfitting problem happening in neural process and sequential neural process (SNP). The idea is to incorporate the attention scheme in SNP and carry out the so-called attentive sequential neural process (ASNP) for sequence learning.\n\nStrength:\n1. A combination of attention into SNP.\n2. Some formulations were provided.\n3. Different tasks were evaluated to investigate the merit of this method.\n\nWeakness:\n1. The comparison for time complexity and parameter size was missing.\n2. The labels in figures were inconsistent.\n3. An incremental research.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "rkgjX7waFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper combines ideas from attentive and sequential neural processes to incorporate an attention mechanism to the existing sequential neural process, which results in an attentive sequential neural processes framework.\n\nWhile the idea is somewhat interesting, I think this paper is technically vague and not well-motivated, which makes it hard for me to feel convinced that the problem exists and is non-trivial, and that the proposed solution is significant. Let me elaborate on my thoughts below:\n\nFirst, the authors stated that SNP is subject to the underfitting problem that plagues NP but it is not clear to me why, in the temporal context of SNP, do we need to focus our attention on past contexts, which are no longer relevant. Could the authors please motivate this with a concrete application scenario? Without a concrete scenario, I do not feel very convinced that the problem exists.\n\nSecond, the argument that augmenting SNP with an attention mechanism is not trivial is somewhat contrived. In particular, the reason for this non-triviality is that (in the authors' own words) SNP assumes that it cannot store the past context as is -- so what if we simply store the past context & condition the representation on the entire history of past context instead? \n\nApparently, this can come across trivially by replacing C_t with both C_<t and C_t in Eq. (2). This is in fact very similar to what the authors did in Eq. (4) which summarizes the generative process of ASNP -- the only difference is the generation of imaginary contexts, whose necessity is again questionable, as I elaborate next.\n\nThird, the motivation for imaginary context is pulled from a very distant literature on how a human brain memorizes past experiences in a lossy memory consolidation, which only retains the most important sketches. In the context of ASNP, it is not, however, clear to me why this mechanism is necessary given that entire lossless memory can be stored except that without a lot of contexts, there is not a need for an attention component (as implied in first paragraph of Section 3) which is a contrived motivation.\n\nFourth, the technical exposition of this paper is too vague. Given that the key contribution here is about an attention component, the background review on ANP is surprisingly informal with no technical detail at all. For the other parts, the technical part is also mostly abstracted away -- what is presented is therefore not that much different from a typical generative model with latent variables, which makes it unclear whether there is a technical challenge here. \n\nIn fact, from what I see, going from Eq. (2) to Eq. (4) is not much of a conceptual challenge and the execution of Eq. (4) (particularly the attention component described in Section 3.2) seems like a bunch of arbitrary engineering ideas which were put together to substantiate Eq. (4). \n\nIs there a technical challenge in the entire pipeline that should have been highlighted?\n\nFor the experiment, could the author compare the performance between ASNP and ASNP without the imaginery component (but with the attention mechanism)? It would be a good experiment to see if the imaginery component is necessary.\n\nTo summarize, I believe the paper in its current state is not well-motivated and appears very incremental given the prior works of SNP and ANP. Even its imaginery component, which is the key contribution here,  is, if I understand Eq. (3) correctly, not much different from context sampling of a NP.\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1lXjVv2KH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors present a method to address the problem of underfitting found in sequential neural processes. They cover the literature appropriately in regards to neural processes and developments pertaining to tackling the underfitting problem by applying an attention mechanism. Although, this has successfully been achieved with Neural Processes, the case is different with sequential neural processes, as they cannot store the past context.\nAuthors addressed this problem by introducing an attention mechanism and model, i.e. Attentive sequential neural processes, which incorporates a memory mechanism of imaginary context. This imaginary context is generated through an RNN network and are treated as latent variables.\nThe results presented show some promising improvements over other methods used and more results have been included in the appendix. It would be nice to demonstrate the performance in more challenging tasks as well, however the results presented and the new context-imagination introduced are quite promising indeed.\nI have read the rebuttal carefully. I appreciate the extra effort put by the authors to address the issues raised from the other reviewers. I think, albeit not ground-breaking research, it could be a good addition to the programme nonetheless.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This manuscript outlines a method to improve the described under-fitting issues of sequential neural processes. The primary contribution is an attention mechanism depending on a context generated through an RNN network. Empirical evaluation indicates empirical results on some benchmark tasks.

In reviews and discussion, the reviewers and AC agreed that the results look promising, albeit on somewhat simplified tasks. It was also brought up in reviews and discussions that the technical contributions seem to be incremental. This combined with limited empirical evaluation suggests that this work might be preliminary for conference publication. Overall, the manuscript in its current state is borderline and would be significantly improved wither by additional conceptual contributions, or by a more thorough empirical evaluation.
Decision: Reject