Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a Sparse Gaussian Process Attention (SGPA) for the Transformer architecture. The SGPA is used instead of the classical scaled dot-product attention and directly performs an approximation of the Bayesian inference in the attention blocks in Transformers. In addition, the authors provide decoupled SGPA to overcome the time and memory consumption limitations of the standard SGPA. This approach is evaluated on image classification, graph property regression, and NLP classification tasks. The authors show that this method is able to produce better calibrated predictions. The SGPA also helps to produce better uncertainty estimates and shows improvements in the OOD detection task in CV.
strength_and_weaknesses: Strength:
* This paper is the first to consider a Gaussian process in attention block and extends SGPA in Transformer for uncertainty estimation.
* The authors carefully provide limitations of the proposed method and propose decoupled SGPA which overcomes the time and memory consumption issues.
* Derivations of the evidence lower-bound objective (ELBO) for optimizing the variational parameters in SGPA and of the decoupled SGPA for the Transformer's attention are provided.
* This method shows significant improvement in in-distribution calibration quality and OOD detection performance while achieving competitive accuracy against the standard Transformers.
* The results are well-presented and clearly discussed.

Weaknesses:
* Not mentioned other approaches that consider stochastic evaluation in the attention block for UE with Transformers. For example, [1] uses stochastic self-attention with the Gumbel-Softmax trick.
* It would be useful to see the results of OOD detection in NLP tasks, for example, for the following tasks: [2], [3].  
* It is not clear how exactly the variational parameters are calculated in the decoupled SGPA in equation (11).
* It is not clear what method refers to MLE in the Experiments section. Is it a method that leverages maximum probability? If no, please, add a reference or exact formula.
* It would be interesting to see the actual computational time of the inference stage of the proposed method with 10 Monte Carlo samples in comparison with the classical MC dropout.
* The improvement of the proposed method over the MLE baseline in the OOD detection task is not stable taking into account the cost of computation.

Typos:
* On page 5 after equation (12): “viewd” -> “viewed” 
* On page 8 in section 4.4: “kernel based” -> “kernel-based”

References
[1] Jiahuan Pei, Cheng Wang, and György Szarvas. 2022. Transformer uncertainty estimation with hierarchical stochastic attention. In AAAI 2022.
[2] Larson, S. et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. In EMNLP 2019.
[3] Gangal, V. et al. Likelihood Ratios and Generative Classifiers for Unsupervised Out-of-Domain Detection in Task Oriented Dialog. In AAAI 2020.
clarity,_quality,_novelty_and_reproducibility: The quality and novelty of this work are good (see Strengths). However, the clarity of this paper may be improved. (See Weaknesses (2)-(4))

summary_of_the_review: This paper provides a theoretically derived method that helps to calibrate prediction and capture better uncertainty estimates for Transformers models. The authors make a significant contribution with minor weaknesses that can be solved. 

[After the review] The response by authors resolved part of my concerns and I decided to increase the score.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes to formulate the mult-head attention operation used in transformer models as a sparse gaussian process. The work shows how the computation of such a GP attention mechanism can be sped up by relying on inducing points and standard approaches for increasing Gaussian process scalability. Multiple such attention operations can be stacked by sampling between layers. In the experiments of the paper, the proposed model shows comparable performance in terms of accuracy, but scores better in terms of NNL and calibration metrics on small vision and graph datasets. The baselines used in the paper where Maximum likelihood, mean field VI in weight space and Monte Carlo Dropout.
strength_and_weaknesses: **Strengths:**

 * The paper is mostly clearly written and the motivation of the method is easy to follow and explained well.
 * The approach is to some degree new and the experiments look promising. 

**Weaknesses:**

 * In the introduction of the method, it would be beneficial to differentiate the application of Gaussian processes for interpolation compared to classification. Both approaches are common in machine learning, yet in one case the goal is to condition on the observations of the instance in order to predict values at unknown query locations whereas a classification GP predicts the output of an instance on the test data, while conditioning on the training data. The former is closer to what the authors are proposing in their work.
 * The work does not compare against approaches relying on deterministic uncertainty estimates while citing papers from that field (for instance Liu et al., NeurIPS 2020). This is particularly relevant in the OOD experiment.
 * The performance of all models is relatively low on the datasets presented. For instance, 77% accuracy is far from SOTA on CIFAR10. This questions the usefulness of the experiments on these data. While the trends compared to baseline approaches are generally ok, it is difficult to rely on these results when models are not applied on dataset/model combinations where there is existing data. I would suggest to at least run one comparison on imagenet, where sufficient performance measurements of vision transformers are available in other papers.
 * The approach is very similar to Set Transformers, which use a similar idea to speed up the computations. A reference would be appropriate.
 * Utilization of `[]` in the equations is slightly confusing. I would recommend to discard these or indicate that they solely are used to keep the subscripts separated in the text.
clarity,_quality,_novelty_and_reproducibility: The paper is generally clear and well written. The experiments are sounds, but rather limited in terms of models to which the approach is compared and the model is not evaluated on a task where state of the are performance of vision transformers is know. In terms of novilty, the approach is to some degree novel, but related to the work on kernel attention and set transformers.
summary_of_the_review: The approach is interesting and to some degree novel. While the results look promising, further experiments are necessary to completely convince me. The model is never trained on a data set where there are external reports on the current maximal performance achievable. The rather low performance on CIFAR10 and 100 is in this context slightly alarming. If the authors, can show me references where a transformer model reached similarly low performance or if they can demonstrate the utility of their method on a data set where the performance of transformers is already well quantified I will be happy to increase my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents a new methodology to introduce uncertainty quantification in Transformer models. This probabilistic approach is based on Gaussian processes. The driving idea is to exploit the similarity between the scaled dot-product (Vaswani, 2017) and the Kernel-attention (Tsai, 2019) with the posterior mean of sparse Gaussian processes (SVGP), to obtain posterior approximations in the attention heads of transformers. Despite the computational cost, which is reduced using global and amortized "keys/inducing-points", the methodology shows a significant performance which is compared with other baselines based on probability in several datasets for different tasks.
strength_and_weaknesses: **Strengths:** The main strength of the paper is on the identifiable equivalences. Find the simplification of transformers models as certain parallelisation of multi-head self-attention modules, where some scaled dot-product operations are similar to the core linear-operators of GPs when considering kernels is an important point. Additionally, exploiting this equivalence and building a new uncertainty estimation framework on top of the transformer's architecture is also nice. The paper provides a lot of details to make sure that is clear where the linking points between SVGPs and Transformers join. Experimental results are somehow clear and provide a fair comparison which the chosen baselines.

**Weaknesses:** In my opinion, there are perhaps several equivalences and connections (i.e. with multi-output GPs and Deep GPs) that are introduced but not really developed or consistently analysed. For example in the paragraph before Section 3.2, the following is added:

> (...) multi-output SVGP (...) and the output of a kernel attention block

I see and appreciate the similarity between K-attention and the posterior mean of SVGP, but the MO-SVGP connection with multi-head attention is not entirely clear to me if I think in papers like (Alvarez, 2008). What I want to remark is that MOGP with SVGPs seems a bit more complicated to me, and does not seem that easy to say that both are equivalent... At least I would like to see this justified.

Similarly, for Deep GPs, the connection is done, but no experiments or further derivation is added on this line. That for me is a flaw of the paper, which I think is not that difficult to solve...

**Questions:**

*[Q1].* How expensive is to train wrt Eq. (14). The use of Monte-Carlo in expectations for SVGPs is in general not a good idea (long training, not very scalable wrt dimensionality, noisy evaluation on the likelihood model), so I am trying to figure out what would be the effect here? Could the authors clarify a bit on this point?

*[Q2].* Thinking in terms of how difficult is to bring posterior computations to standard NNs, and how the "Gaussian nature" is broken in Deep GPs when one layer after the other is connected passing GPs through, I am somehow surprised that SVGPs fit that well in multiple layers of the Transformer without any of these problems. Are non-linearities or multiple layers affecting the approximation considered? 

**References:**

(Alvarez, 2008) https://proceedings.neurips.cc/paper/2008/file/149e9677a5989fd342ae44213df68868-Paper.pdf

clarity,_quality,_novelty_and_reproducibility: **Clarity & Quality:** The paper is clear in general, providing a great review/perspective of the Transformers architectures. On the quality side, all decision are properly justified as well as the experimental results seems rigorous and sufficient to me. 

**Novelty & Reproducibility:** The paper seems novel in the way that it aims to improve performance of Transformers with uncertainty estimation via GPs. Perhaps, the use of SVGPs is not particularly novel, but its use in this context for the attention products is justified and significant. (This may open a door to other ideas or probabilistic approaches in the future).

The reproducibility is perhaps another weak point in this submission, or at least, I perceive that are quite a lot of missing details on the initialization, setup of key/inducing-points, need or no-need of pretraining, optimization, limitations on dimensionality, convergence and time metrics. I say this because SVGPs usually struggle on these points and extra efforts/tricks are required. (I.e. one can easily find details, explanations and discussion on these aspects in GP submissions). Here, no particular details are added, and I think they are somehow necessary.
summary_of_the_review: The paper is valuable and relevant for the community, as it builds a new door to introduce probabilistic methods into Transformers via the equivalences in kernel-attention heads. The methods brought are perhaps revisited and there are not of extreme novelty, but the ideas presented for the solution, the equivalences and their connection are still important. The experiments are thorough and I believe the results presented. However, there are some weaknesses and missing details that make me decrease my score for strong acceptance to weak acceptance. If these ones are clarified, I would be glad to raise my score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors suggest an efficient implementation of GP for the attention layers of transformers. Through a few experiments they argue it helps in improving the accuracy, the covariate-shift robustness, the calibration and the out-of-distribution detection performance when fine-tuning transformers on smaller scale datasets.
strength_and_weaknesses: Strenghts:
- The paper is clearly written, easy to follow
- The inclusion of preliminaries is nicely done and can help people not familiar with the topic 
- The method seems to positively affect not only the uncertainty properties but also the accuracies  
- The authors perform an adequate selection of the type of experiments to perform
- The idea of developing a computationally efficient GP implementation for attention is interesting 

Weaknesses:
- Q1: Some very important baselines seem to be missing, and the used baselines are generally acknowledged to be extremely weak (MCD and MFVI are notorious for being outperformed by much better SOTA methods). I would suggest integrating your experiments (1) for accuracies on in-distribution and covariate-shift datasets, it's important to evaluate against methods that have been observed to produce improved accuracies (and also improved calibration and out-of-distribution performance) like PixMix [1], RegMixup [2], Deep Ensembles [3] (and optionally any of its more computationally efficient variants like Snapshot ensembles [4] or BatchEnsemble), (2) for calibration experiments, as a bare minimum Temperature Scaling [5] should be considered too, and it would be good to compare with more sophisticated calibration techniques too, (3) for out-of-distribution detection performance techniques like G-ODIN [6] and VOS [7]. Also, since your methodology falls in the Bayesian inference with GPs, it would be good to at least compare with KFAC-LLLA [8], but I understand it might be tricky to apply SNGP, DUE and other GP-based techniques to transformers. 
- Q2: The motivation of the methodology is not clear. This is a general problem of Bayesian Deep Learning, but it's not clear why having GP on the self-attention should meaningful. While works like SNGP and KFAC-LLLA motivate their usage of (approximations) of GP with a distance-aware uncertainty estimation narrative (though it is questionable that euclidean distances are meaningful at all), I don't feel like the paper provides a clear motivation of why introducing GPs in a few parts of the network should be theoretically meaningful. The paper does not even provide aneddotical evidence that the uncertainty estimates the GP provide in the self-attention can actually detect ambiguous features of the input.
- Important comment on Q1 and Q2: if Q2 cannot be addressed due to the limitations of Bayesian DL, working on Q1 and showing the effectiveness of your method at least empirically would convince me to rise the score. 
- Q3: The paper is not really about calibrating for pre-training but for fine-tuning. Modern approaches to fine-tuning are quite sophisticated and might affect uncertainty estimates or the usefulness of the proposed method. Did you consider advanced fine-tuning techniques?  

Improvements (Marginal):
- Probably the paper should emphasize more it's about fine-tuning and not really calibrating large-scale pre-training
- The NLL is known to be a questionable metric for calibration, but a lot of literature still reports it 



[1] https://arxiv.org/abs/2112.05135
[2] https://arxiv.org/abs/2206.14502
[3] https://arxiv.org/abs/1612.01474
[4] https://arxiv.org/abs/1704.00109
[5] https://arxiv.org/abs/1706.04599
[6] https://arxiv.org/abs/2002.11297
[7] https://arxiv.org/abs/2202.01197
[8] https://arxiv.org/abs/2002.10118
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written, but the experiment setup is lacking and the method is not well motivated. The work is somehow novel and the derivation seems alright. As for the reproducibility, it would be nice if the authors could provide an efficient implementation.
summary_of_the_review: The paper is interesting, but it either needs a better motivation or much stronger experimental evidence. If either (or both) these issues are improved, I am happy to rise the score.

--------------------
POST-REBUTTAL AND DISCUSSION WITH OTHER REVIEWERS
I increase my score to 5. 

- Although it is difficult to interpret the meaning of GPs over self-attention, the additional comparisons with strong baselines (both deterministic and non-deterministic) like TS, KFAC-LLLA, SNGP and Deep Ensembles shows the method is at least empirically effective
- The justification provided for the usage of GPs on self-attention is intuitive, and although still not completely supported and questionable given the lack of an understanding of how uncertainties compose through the layers of a network in the presence of additional components (normalization layers, non-linearities, skip-connections etc.), I think it would be unfair considering this to be a limitation of this specific work, as it is a known problem of Bayesian Deep Learning.
- The usefulness of the method might depend on the fact Transformers don't converge to a good minimum without enough pre-training data on such small datasets. The technique can be considered effective probably also because it somehow regularizes the training procedure. Therefore, the improve the impact of the paper, it would be important to show the proposed method can be effective also when fine-tuning pre-trained models.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The reviewers and meta reviewer all carefully checked and discussed the rebuttal. They thank the authors for their response and their efforts during the rebuttal phase.

The reviewers and meta reviewer all acknowledge the original and sound contribution of considering a (sparse) GP modeling in the attention layers of transformers. Similarly, they all call out the overall good quality of the manuscript (writing, structure). The nice unveiled connection between the structure of the attention and the form of the posterior is likely to inform future research (as acknowledged by the reviewers). This was regarded as the most important contribution of the submission.

The rebuttal has strengthened some aspects of the work (e.g., measure of computational overhead, more explanation about the from-scratch training, additional comparisons). However, there are still some remaining important concerns. _As a result, the reviewers and the meta reviewer are weakly inclined to accept the paper._

In particular, the authors are urged to carefully update their final manuscript with the following points:

(i)  Practical significance: The results would be more convincing in the most standard and popular settings where ViT shines, i.e., with a pre-training. Why not load an existing pre-trained checkpoint (to minimize the cost of the experiments, acknowledging the concerns of the authors) and introduce the new GP-related parameters only at fine-tuning time? (e.g., similar checkpoint surgery successfully done in [A, B]). Moreover, having experiments based on a pre-trained model would help further understand whether the gains observed are also due to an implicit regularizing effect (see Reviewer GcWL), especially important in the training-from-scratch setup.

(ii) Scalability and comparison with other approaches to capture uncertainty for ViT: In [A], it is shown that simple and scalable components (efficient ensemble and/or last-layer input-dependent noise modeling) greatly help for tasks similar to those tackled in the submission. Adding a comparison with such an approach, _at ImageNet scale_, would be important to strengthen the practical significance of the paper. This is all the more important as the proposed approach is arguably more complex (and substantially more expensive, as already measured at cifar-scale) than the components of [A]. Furthermore, hyperparameters such as the number of inducing points are difficult to set in a reliable way at a larger scale.

If the paper was submitted to a journal, it would be accepted conditioned on those key changes, the meta-reviewer thus expects all those changes to be carefully implemented.

**References**

[A] Tran et al., 2022, Plex: Towards Reliability using Pretrained Large Model Extensions.

[B] Allingham et al., 2022, Sparse MoEs meet Efficient Ensembles.

justification_for_why_not_higher_score: * Concerns about practical significance (scalability, robustness, demonstration on the most popular settings) 
justification_for_why_not_lower_score: * Original and interesting connection between attention and posterior
* Manuscript of good quality (writing, structure)
Decision: Accept: poster