Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper deals with the underfitting problem happening in neural process and sequential neural process (SNP). The idea is to incorporate the attention scheme in SNP and carry out the so-called attentive sequential neural process (ASNP) for sequence learning.

Strength:
1. A combination of attention into SNP.
2. Some formulations were provided.
3. Different tasks were evaluated to investigate the merit of this method.

Weakness:
1. The comparison for time complexity and parameter size was missing.
2. The labels in figures were inconsistent.
3. An incremental research.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper combines ideas from attentive and sequential neural processes to incorporate an attention mechanism to the existing sequential neural process, which results in an attentive sequential neural processes framework.

While the idea is somewhat interesting, I think this paper is technically vague and not well-motivated, which makes it hard for me to feel convinced that the problem exists and is non-trivial, and that the proposed solution is significant. Let me elaborate on my thoughts below:

First, the authors stated that SNP is subject to the underfitting problem that plagues NP but it is not clear to me why, in the temporal context of SNP, do we need to focus our attention on past contexts, which are no longer relevant. Could the authors please motivate this with a concrete application scenario? Without a concrete scenario, I do not feel very convinced that the problem exists.

Second, the argument that augmenting SNP with an attention mechanism is not trivial is somewhat contrived. In particular, the reason for this non-triviality is that (in the authors' own words) SNP assumes that it cannot store the past context as is -- so what if we simply store the past context & condition the representation on the entire history of past context instead? 

Apparently, this can come across trivially by replacing C_t with both C_<t and C_t in Eq. (2). This is in fact very similar to what the authors did in Eq. (4) which summarizes the generative process of ASNP -- the only difference is the generation of imaginary contexts, whose necessity is again questionable, as I elaborate next.

Third, the motivation for imaginary context is pulled from a very distant literature on how a human brain memorizes past experiences in a lossy memory consolidation, which only retains the most important sketches. In the context of ASNP, it is not, however, clear to me why this mechanism is necessary given that entire lossless memory can be stored except that without a lot of contexts, there is not a need for an attention component (as implied in first paragraph of Section 3) which is a contrived motivation.

Fourth, the technical exposition of this paper is too vague. Given that the key contribution here is about an attention component, the background review on ANP is surprisingly informal with no technical detail at all. For the other parts, the technical part is also mostly abstracted away -- what is presented is therefore not that much different from a typical generative model with latent variables, which makes it unclear whether there is a technical challenge here. 

In fact, from what I see, going from Eq. (2) to Eq. (4) is not much of a conceptual challenge and the execution of Eq. (4) (particularly the attention component described in Section 3.2) seems like a bunch of arbitrary engineering ideas which were put together to substantiate Eq. (4). 

Is there a technical challenge in the entire pipeline that should have been highlighted?

For the experiment, could the author compare the performance between ASNP and ASNP without the imaginery component (but with the attention mechanism)? It would be a good experiment to see if the imaginery component is necessary.

To summarize, I believe the paper in its current state is not well-motivated and appears very incremental given the prior works of SNP and ANP. Even its imaginery component, which is the key contribution here,  is, if I understand Eq. (3) correctly, not much different from context sampling of a NP.

review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Authors present a method to address the problem of underfitting found in sequential neural processes. They cover the literature appropriately in regards to neural processes and developments pertaining to tackling the underfitting problem by applying an attention mechanism. Although, this has successfully been achieved with Neural Processes, the case is different with sequential neural processes, as they cannot store the past context.
Authors addressed this problem by introducing an attention mechanism and model, i.e. Attentive sequential neural processes, which incorporates a memory mechanism of imaginary context. This imaginary context is generated through an RNN network and are treated as latent variables.
The results presented show some promising improvements over other methods used and more results have been included in the appendix. It would be nice to demonstrate the performance in more challenging tasks as well, however the results presented and the new context-imagination introduced are quite promising indeed.
I have read the rebuttal carefully. I appreciate the extra effort put by the authors to address the issues raised from the other reviewers. I think, albeit not ground-breaking research, it could be a good addition to the programme nonetheless.
review_title: Official Blind Review #2

Meta-review: This manuscript outlines a method to improve the described under-fitting issues of sequential neural processes. The primary contribution is an attention mechanism depending on a context generated through an RNN network. Empirical evaluation indicates empirical results on some benchmark tasks.

In reviews and discussion, the reviewers and AC agreed that the results look promising, albeit on somewhat simplified tasks. It was also brought up in reviews and discussions that the technical contributions seem to be incremental. This combined with limited empirical evaluation suggests that this work might be preliminary for conference publication. Overall, the manuscript in its current state is borderline and would be significantly improved wither by additional conceptual contributions, or by a more thorough empirical evaluation.
Decision: Reject