Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce NeuBANP, an extension of B(A)NP that replaces the iterative prediction method of B(A)NP with a single prediction step following the idea behind the neural bootstrapper. This new model is computationally more efficient and able to produce more accurate uncertainty estimates as well as correctly model heteroscedastic uncertainties. The authors provide experimental results on nonparametric regression tasks, Bayesian optimisation, contextual multi-armed bandit tasks and image inpainting (in the appendix).
main_review: Positive aspects: 
* Overall the idea of extending the bootstrapping element of B(A)NPs using neural bootstrapping makes sense and seems to work well in practice. 
* The authors do provide a nice set of experiments with the appropriate baselines and the results seem promising.


Criticisms:
* My main concern is that it is not clear to me why one would expect that applying this neural version of bootstrapping would improve the uncertainty prediction of the model and allow for the heteroscedastic uncertainty prediction. From the results it seems that this is indeed the case, but it would be useful to have some intuition behind this explained in the paper. The way I understand the paper this is the main benefit of this model upgrade so this should be elaborated on. 
* The second benefit of this model seems to be that it is computationally more efficient (given that it does not require the iterative bootstrapping step, this makes sense). It might be worth analysing how much this buys you in terms of computation/time. Given that this is one of the two selling points it would be nice to have a more quantitative evaluation of this.
* Finally, in terms of novelty, while this is the first time I hear of this particular combination of models, it is a somewhat straightforward combination of previously existing ideas (this in itself should not render the work unworthy of being published, of course).

Suggestions:
* I personally find the notation used to state that you are sampling the context points  $C$ from some set of indices $[n]$ a bit confusing. In equation 1 the inner expectation is wrt C but then the conditional probability is on $D_C$. I would personally write it as: $E_{C \sim D_C}-\log p_{\theta}(Y|X, C)$.
* In the paragraph on BANP I would define the variable $b$ much earlier (right when it is first used in the fourth line) rather than waiting to the end, as it is not clear that these are the different ensemble samples. 
* In figure 2 I would make clear that both the bootstrapping path in B(A)NP and the encoding part in NeuBANP is repeated b times (if I understood correctly this is only shown for one instance in the figure).
* Also in figure 2 I would add the $\hat{r}$ variable in the diagram just before the decoder, since you mention it in the text.
* In figure 3 I would explicitly mention that the minimum variance of ANP and BANP is as large as it appears because that is the fixed minimum value. Not to say that there is no benefit from NeuBANP not having such a lower bound, but for those that might not be familiar with that fixed value in the other models the results might seem odd.

summary_of_the_review: Overall I think the paper is nice, the explanations are not hard to follow and it is overall well written. While the novelty of this model is not outstanding, the results section covers an interesting range of experiments with the right baselines. I would personally recommend adding more intuition as to why this extension improves the performance the way it does, as it is the main selling point.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes Bootstrapping Attentive Neural Processes (NeuBANP), which utilize random sum-to-one weight in the encoder following from Neural Bootstrapper (Shin et al., 2021). The authors argue that this utilization resolves the overfitting problem in attentive NPs, and the proposed NeuBANP is more efficient in computation and memory perspective. Various experiments, such as synthetic examples and contextual multi-armed bandit, are conducted to compare against previous works of NPs and GP.
main_review: Strength:
- The paper is written in clear manner.
- The proposed method is easy-to-implement in NP schemes.
- The code is also provided together for the reproducibility.

Weakness:
- The novelty is somewhat limited, that this is a direct utilization of Neural Bootstrapper.
- I'm confused on the message of the paper. In Fig 1, the authors argue that the baselines (1) overfits the data from the linear regression samples, and (2) are incapable of capturing uncertainty. However, in Fig 3, the proposed NeuBANP seems overfitted and the baselines seems that they can capture the uncertainty (at least) in interpolation. If I missed something, please let me know.



summary_of_the_review: My opinion is that the paper is on borderline. The work seems neat and probable while the novelty is limited. Considering the quality of the venue, I'm bit negative on the direct utilization of existing work. Also, there is a disalignment (probably my misunderstanding) of the motivation and the experiment result, which I discussed in the weakness, want to be addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a new class of neural process algorithm called neural bootstrapping attention for neural processes (NeuBANP). This method utilizes efficient Neural Bootstrapping (NeuBoots) to improve Bootstrapped Attentive Neural Processes (BANP) in capturing functional uncertainty. Authors show that NeuBANP achieves state-of-the-art performance in benchmark experiments including Bayesian optimization and contextual multi-armed bandits. 

main_review: Strength:

- Problem of modeling functional uncertainty is clearly motivated. 
- Simplifies and improves BANP using NeuBoots.
- Evaluates on standard benchmarks and shows superior results on Bayesian optimization and contextual multi-armed bandit tasks. 
- Provides clean structured code for reproducibility. 

Weakness:

- The empirical evaluation on 1D regression remains mixed. It’s hard to tell whether the failure of modeling GP target samples from the periodic kernel (Table 2) is intrinsic to the proposed method or if the experiment is not executed properly without having further analysis. Also it seems like context LL is significantly better for regression tasks but that has not been explained enough. 

- One of slight worry from the reviewer is that the paper’s method is hinging on two of very recently proposed approaches (Bootstrapping Attentive Neural Process and Neural Bootstrapper) where each of the merits are not quite validated thoroughly in the literature yet. 


Suggestions:

- Main interesting idea this paper brings into NP space is utilizing Neural Bootstrapper. However, current analysis heavily relies on building on BANP. It would be great ablation analysis to apply neural bootstrapper on non-attentive models and compare to vanilla BNP and NPs.  While the idea of using attentive neural processes may fade-away,  the idea of efficient bootstrapping of neural processes could remain important. 

- Figure 1 / Figure 6 Training setting is unclear and makes it hard to interpret what message is being delivered.

- In section 4.1. 1D regression seems cherry-picked since it only shows the RBF kernel and does not discuss failure in the Periodic (in Appendix) while standard 1D evaluation runs on various kernels (RBF, Matern, Periodic).

- Section 4.3 results on CMAB are quite promising and demonstrate competitiveness. It would be great to include a comparison to BANP and ANP as well to see that Neural Bootstrapping is the main reason for superior performance. 

- In Figure 5, it would be interesting to have a sense of order of evaluated points to see how each model zooms into particular (often suboptimal) minima. 

Questions:
- As mentioned earlier, do the authors have a good idea why improvements on the context point are huge? According to the motivation NeuBANP was introduced to solve ANPs “overfitting” issue. Having high context LLs doesn't quite align with the motivation?

- In Figure 3, NeuBANP samples as well as posterior distribution are discontinuous; why is this the case? (e.g around -0.5, and 0.5) With the discontinuity, it does not seem like a great model but still able to obtain good context/target LLs?

- In Table 5, ANP’s context LL for both seen classes and unseen classes is exactly the same, did authors check that if there’s mistake on reporting or the actual numbers turned out to have the same mean and std. 

- While authors cite Lakshminarayanan et al. 2017 for bootstrapping to be a reliable approach to estimate predictive uncertainty in neural networks, the reviewers take from that paper is that bootstrapping was detrimental compared to deep ensemble. In Nixon et al., 2020 (https://openreview.net/forum?id=dTCir0ceyv0), they further observe bootstrapping doesn't improve uncertainty beyond the vanilla ensemble method. Could authors elucidate in what situation we expect using bootstrapping to be beneficial vs not in neural networks?  

- Caption in Table2: is reporting on the 48,000 evaluation standard procedure? seems quite arbitrary and high


------------------------------------------------------------------------------

post rebuttal : I thank the author for the response and clarification. My assessment still is that this is somewhat borderline paper for ICLR and slightly leaning supporting acceptance thanks to solid result on sequential decision making tasks. 
summary_of_the_review: This paper proposes an efficient way of modeling functional uncertainty building on recent work of NeuBoots and BANP. On the benchmark of sequential decision making, authors demonstrated state-of-the art performance which shows great promise. Method is presented in a clear fashion and evaluation is done on standard set up. There are some desired analyzes pointed out in the main review that could improve scientific understanding of the proposed method. Overall, I believe the proposed idea is sound and interesting to be shared among the ICLR audience.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new bootstrapping method in NP family called NeuBANP. The authors acknowledge the limitations of BANP and used the neural bootstrapping method to make development from BANP. While BANP does better functional uncertainty estimation than ANP, it carries a higher computational burden compared to ANP since it requires multiple computations of the encoder network, the adaptation layer, and additional heuristics. By incorporating neural bootstrapping to the bootstrapping procedure, NeuBANP is a more computationally efficient model, estimates functional uncertainty better than BANP, and also alleviates the overfitting problem that ANP and BANP carries. Experimental results show that it achieves state-of-the-art performance on stochastic optimization problems, including multidimensional Bayesian optimization and contextual multi-armed bandit.
main_review: The strength of the paper is that the authors report multiple experiments in different settings that show the superior performance of NeuBANP. They are clear about the comparative advantage from ANP, BANP, and NeuBANP. The figure comparing BANP and NeuBANP makes it easy to understand the difference and highlights the efficiency of NeuBANP. The figures for experimental results also depict the superior performance of NeuBANP well.

My main concern to hold this paper to the high standard of ICLR is the significance and originality of the paper. As it has been discussed well in the paper, Bootstrapping Attentive Neural Processes have been proposed in previous work. Also, Neural Bootstrapper, which replaces the sampling problem of bootstrapping with the augmentation of the loss function, has also been proposed in Shin et al. 2021. This paper looks to me is an application of the ideas in (Shin et al 2021) to the published Bootstrapping Attentive Neural Processes. The paper is well written and the ideas are discussed clearly. However, I am not sure if a combination of existing ideas can win out in a highly competitive venue such as ICLR.

There are some suggestions for the paper as well. First, it would be helpful to mention the significance of NP models compared to other models a little more. Second, one of the contributions the authors mention is computational efficiency. Although this is well depicted in the figure, some more experimental results regarding this point would be better for highlighting this point. Third, if there exist other NP methods apart from ANP and BANP, it might be good to introduce them briefly. Fourth, it would be better if the others made it clearer if the state-of-the-art performance is regarding all other machine learning models or other NP methods. Lastly, including some mathematical details or reasoning of why NeuBANP does better in estimating functional uncertainty would be good. 

summary_of_the_review: The authors propose a new bootstrapping method in NP family called NeuBANP. The authors acknowledge the limitations of BANP and used the neural bootstrapping method to make development from BANP. The experimental results show great improvement over other NP methods. While the paper is written well and the figures are clear, there is some room for improvement. In particular, the combination of existing ideas may not hold up to the high standard of ICLR.
review_title: null

Meta-review: Overall, the work is borderline with no reviewer feeling strongly for or against the paper.

The paper is well-written and proposes a simple approach, along with code for reproducibility. Criticism stems primarily in the work's technical novelty, being an incremental improvement of ideas from ANP and BANP, and related work like Neural Bootstrapper. In addition, the experimental validation involves regression on 1-to-2D functions, Bayesopt on synthetic functions, and contextual bandits on the synthetic wheel bandit problem. This is fairly toy, and multiple reviewers raise unaddressed concerns on the regression experiments. Ignoring orginality in and of itself (which is overvalued in conferences), the work does not yet provide a sufficiently convincing demonstration of its practical importance.

I recommend the authors use the reviewers' feedback to enhance their preprint should they aim to submit to a later venue.
Decision: Reject