Kati8/ICLR_2023_2025 huggingface dataset:
authors: James Harrison
authors: John Willes
authors: Jasper Snoek
reviews: 
Comments: This paper introduces Variational Bayesian Last layers, a technique to perform uncertainty estimation in standard neural network architectures. 
The method performs bayesian learning only for the last layer in the neural networks using Variational Inference.
This results in a scalable and simple technique that shows strong performances in standard benchmarks for regression and classification.
Strengths: 1. I found the paper very interesting and easy to read. Uncertainty estimation is an important and active research area in deep learning.

1. To the best of my knowledge, the idea is novel (although not groundbreaking)

1. The method is scalable, simple to implement in standard architectures, and achieves very competitive performances (especially considering its simplicity)

1. While being Bayesian only on the last neural network layer the method is in principle not as powerful as other techniques, as the authors rightly claim simpler methods that are easy to implement are what is being more commonly used in practice (e.g. Bayesian dropout, stochastic weight averaging)

1. The appendix is extensive and addresses all the details I felt were missing in the main paper
Weaknesses: I did not identify any major weaknesses, only some points for improvement

1. To increase the impact of the paper you need to make sure that people that are not too familiar with VI are able to easily implement the paper. This means:
    1. Make the code publicly available, especially to show how to best implement the "mixed parameterization" discussed in appendix D 
    1. set good default hyperparameters

1. It would be useful to draw the graphical models of the models presented in Section 2, to help the reader visualize the random variables in play and their (hyper)priors/parameters

1. In case you need more space in the paper, I would move to the appendix some of the details on the generative classification model, especially considering the poorer performances.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent
reviews: 
Comments: This paper derives efficient optimization objectives of the posterior distribution over the parameters of the last layer of neural networks for common machine learning applications such as classification, regression, and generative classification. Theoretically, their variational inference objective functions are derived in closed form and therefore enjoy the property of not requiring sampling - meaning the cost of enabling uncertainty quantification for a broad class of deep learning architectures is marginal (parameters other than the last layer are learned by maximum a posteriori). Experimentally, they validate these novel variational inference algorithms using standard benchmarks from UCI, a large language model used for sentiment analysis, and an image classification problem.
Strengths: This is an excellent paper and a significant contribution - well done! The authors make clear how they build on the existing literature in Bayesian deep learning to create a novel advance that is practical and easy to implement. This is significant and should enable more work to push the frontier of the "best of both worlds", with neural networks serving as function approximators and Bayesian methods enabling sample-efficiency and quantification of uncertainty that is required for practical deployment of deep learning.
Weaknesses: Visualizations of how tight or loose the bounds in the main text could help build more intuition; comparisons in terms of speed or efficiency to variational inference algorithms that do require sampling (such as Monte-Carlo objectives like VIMCO) could also help guide practitioners in making the correct trade-off depending on FLOPs of compute available versus the required accuracy of posterior approximation/uncertainty quantification.
Recommendation: 10: strong accept, should be highlighted at the conference
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 4 excellent
Clarity: 4 excellent