Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to improve attentive neural processes (ANP; Kim et al.) by replacing deterministic attention with a Bayesian attention module.
Since the inference requires variational approximation, I think the proposed method is a variational counterpart of ANP, similar to the relation between VAEs and autoencoders.
main_review: ### Strengths
- The application of Bayesian attention to neural processes seems novel.
- The authors provide an information-theoretic perspective of the proposed method.

### Weaknesses
- The authors claim that conventional NPs are sensitive to noise, but this claim is not well-supported.
- Being sensitive to noise is caused by multiple factors, such as the model's flexibility, the amount of data, the number of training steps, etc. With enough data, therefore, larger sensitivity can result in better performance. 
- I think claiming "superior" performance based on the reconstruction error is like claiming VAE is better than autoencoders in terms of the reconstruction error. 
summary_of_the_review: I think the main idea is novel and worth publishing.
However, some of the claims are not grounded properly and should be toned down.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a neural process enhanced with stochastic attention to focus more on the context dataset. The method replace the classical attention used in ANP with the Bayesian attention module, showing that this design choice improves the performance also in noisy scenarios or when target datasets mismatched (by changing the kernel used to generate the target dataset). Moreover, the paper offers an interpretation of the method from an information theory perspective, proving that the NP with stochastic attention can be seen as a regularization of the latent space such that it pays more attention to the context dataset. The method is tested on both synthetic and real-world datasets and improves the scores especially in noisy or more complicated scenarios.
main_review: **Pros:**

- Even if, to me, the technical contribution is not major - integrating bayesian attention in the attention neural process - I enjoy the effort made to explained the advantages brought by the method in an information theory setup. This improves the understanding of the field and open up the path towards further directions.
- The experimental part show consistent improvement over multiple setups and several datasets

**Cons:**

- The paper mention that the model does not require exhausted computations. Can you also report the comparison between the proposed methods and the other baselines presented in the paper in terms of time efficiency and the number of parameters?
- For the experiments in Movielens-100K dataset, isn't it possible to integrate the graph structure in the NP approach for a fair and higher performant model?
- In Figure 1 a) it is not clear what it is depicted in the second row. I guess it represents the attention weights similar to the one in the supp material, but it should be clearly specified in the caption.
summary_of_the_review: To me, the paper seems interesting not necessarily because of the technical novelty, but because of the theoretical side: the information theory based explanation of the final loss. However, I want to mention that I do not have much experience on this particular field so it might be the case that I missed some similar prior work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Neural processes (NPs) aim to stochastically complete unseen data points based on a given context dataset. This paper incorporates the stochastic attention in NPs to capture the context information. We empirically show that our approach on 1D regression, predator-prey model, image completion, and MovieLens-10k dataset. 

main_review: Strength: 1. The paper clearly indicates the idea of incorporating stochastic attention with key-based contextual prior into the neural processing. 

Weakness: 1. My biggest concern for this paper is whether there is a novelty of the proposed method. Although I see the benefit of stochastic attention as it is proposed by Fan et al, this paper basically combines neural processing with stochastic attention. Then what is the main novelty here? It is simply A (neural processing) + B (stochastic attention) is better than A. Also the contextual prior, how to optimize and inference, etc are similar as the Fan et al 2020. 

2. The information theory as well as the derivation of the equations in the main paper and appendix such as reparameterization, kl, and elbo are all covered in previous or early literature. It is not new.

3. As the key contextual prior is one of the important parts of this paper, it is also less ablation study to show the comparison between including the contextual prior and not including the contextual prior.

4. In the experiment section, the Movielens 10k dataset is a very real-world dataset as claimed by the author. However, the improvements on MovieLens-10k dataset are very marginal and even raise my concern about the practical benefits of this proposed method. 

summary_of_the_review: Please see the details in my main review. In summary, I think this paper clearly indicates their ideas of incorporating stochastic attention into neural processing. However, the novelty as well the comprehensiveness of the study and experiments need more elaborations and works. Thus, I recommended marginally below the acceptance threshold.


----------------------------------Post rebuttal-----------------------------------------------------------------------------------------------------------------
After carefully reading the authors' response, I am still holding the question regarding the novelty of this paper. The theoretical proof in this paper is very limited and basic. And most of them including the contextual prior, stochastic attention, elbo, etc have been fully covered by the bayesian attention modules [Fan et al 2020] paper. Thus I don't agree with the author about the theoretical proof. Thus, this paper can only be seen as a simple implementation of the bayesian attention module in the NP field.  
In addition, although I see new results in the rebuttal, my concern is that the accepted version of the paper might not be a substantially improved version of the paper. The rebuttal stage is not similar to a journal revision where you can submit an improved version of your paper. Rather, it should be used to clarify misunderstandings.
Overall, I would like to hold my original score for this paper and still think this paper is not ready for the ICLR conference.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a novel method for the Attentive Neural Processes paradigm by adding stochasticity to the weights of the cross-attention module between the context and target representations. These weights are drawn from a proposal distribution which is a Weibull distribution with parameters determined by the context and target input points. A new KL regularization term is added to the total loss to enforce the proposal distribution be close to the gamma distribution determined only by the context points, in a similar fashion to the regularization term for the latent representation between the whole set of target and context points and that of the context points only. The authors include, from Fan’s paper, the closed-form KL divergence between the Weibull and Gamma distributions, making the whole loss be differentiable w.r.t. the network parameters thanks to the reparametrization trick, as in the original NP paper. The new loss is linked to the gain of the target distribution by means of information theory, by making the assumption that the new regularization implies not only that the mutual information between the target distribution and the target points is maximized, but also the mutual information between the representation for the context and target points and the latent representation of the context variables only is minimized, enforcing the latent variable to also consider the context points, and not otherwise as suggested by the original NP formulation. 
main_review: Novelty:

In my opinion, the novelty is somewhat limited, as it is a plug-in method (the Bayesian Attention Module), to the well-established Attentive Neural Processes framework. Both methods and the regularization term are already introduced in the literature. While the paper has the merit to propose it on the very specific domain of Neural Processes, its effectiveness against newer methods is not proven enough so as to make the impact outweigh the lack of technical novelty. The development of the Bayesian Attention Module directly replicates that of Fan’s paper, including the mathematical derivation and formalisms. 


Presentation:

The writing and presentation are quite inadequate and make the paper hard to be followed and understood. It requires several reads to get a glimpse of the contributions and the figures and results are quite poorly presented (the quality of the images is especially poor). In the same line, the text is more based on suggestions than on actual statements that are linked to the proven concepts, which makes the narrative rather weak. While this is not on itself a heavy point towards my personal evaluation of this paper, I believe that in either case authors should devote a good amount of time to improving the paper readability. 


Experiments:

The method only compares against the simple baselines of Attentive Neural Processes and Neural Processes, and does not consider the recent advances in the topic such as the Convolutional Conditional Neural Processes and the ConvNP works of Gordon et al. and Foong et al. These works tackle the problem of domain shift with functional representations and seem to improve over the baselines in a big extend. The authors are encouraged to test their method against the newer NP families to prove the effectiveness of the proposed approach. 

I would like to see an analysis of the learned distributions w.r.t. the “distance” between the context and the target points, with a comparison w.r.t. the weights that would be computed by a deterministic attention module, which would act as a baseline. 

Other comments:

- It is my understanding that at inference time the weights for the attention module are drawn from the Weibull distribution, meaning that for each query (x_i in the target) there will be N keys (x_i in the context) and hence N weights. In Eqn. (6), q_phi represents that distribution, which is conditioned to x_i on the target, and X_c and Y_c on the context. This seems to indicate that the parameter lambda is determined by an MLP whose inputs are x_i, X_c and Y_c. However, I do not see the role of Y_c on computing such weights, as the text seems to refer to the fact that the weights for x_i, X_c will be computed probabilistically according only to these values, which represent the weights to compute the value r_i for x_i as a linear combination of the r_i in the context, which are now represented by X_c AND Y_c. However, why is Y_c necessary to obtain the KL loss in Eqn. 6? Are the weights computed from x_i on the target and the r_i on the context? Please do clarify as I might have misunderstood something here.
- One of the main problems of NPs is the domain shift, which occurs e.g. when the inputs are farther from the context set. NPs compute the latent variable independently of the target points i.e. they do not account for the distance between the context and the target points. ANPs partially solve this problem, as well as the ConvCNP and ConvNP family. In this paper, this problem is not analysed, and indeed the motivation seems to lie on the idea of making the attention weights be more focused on the context set rather than on the distance between the context and target sets. Again I might be misunderstanding something here, but I would like to ask the authors to elaborate on the similarities and differences w.r.t. the recent works aforementioned. How would the proposed method work in the image completion experiment for images of larger size than those used to train (e.g. 32x32 to 64x64). 


- The NP loss is formulated in a similar fashion to Variational Autoencoder, where z is computed during training from the target points. However, as pointed out in Foong et al. this regularization term might not be needed and indeed one can train the Neural Process without it by conditioning z only on the context points. Such approach improves the expressiveness of the NP family. I wonder how would the proposed method perform in such case, i.e. without the regularizations and by computing the latent distributions for the global and local variables directly from the context points only. This should be a necessary experiment towards understanding whether the mutual information gap needs to be maximised, or not. The paper does not provide any experimental analysis on the two terms of the mutual information mentioned in Section 3.3. Proving that this gap is an upper bound might not necessarily entail that indeed I(Z,x_i|D) is being minimized. 

In summary, I believe the paper holds an interesting research direction worth exploring in the context of Neural Processes. However, in my opinion, the current manuscript does not meet the standards for acceptance at ICLR 2022. The authors are strongly encouraged to address the limitations and improve readability, as I do believe this is a promising idea. 

summary_of_the_review: The paper develops on a promising idea towards incorporating stochastic attention weights to the Attentive Neural Processes framework. However, the current manuscript is far from being ready for its publication.


--- Post rebuttal

After interacting with the authors and considering the comments and revisions, I am happy to raise my score. The paper can make a good contribution to ICLR 2022.
review_title: null

Meta-review: This work receives mostly positive rates. Most reviewers agree that the use of Bayesian attention to neural processes is novel, and its interpretation is interesting. Since the reviewer TBTA requests a substantial revision of the submission and fortunately authors’ feedback is thoroughly satisfactory, we highly recommend the authors to prepare for a significantly improved camera-ready version that clarifies most of reviewers’ concerns.
Decision: Accept (Poster)