reviews: 
Comments: The authors introduce variational Bayesian last layers as a novel approach for approximate inference in Bayesian deep learning models. The main contribution is three-fold: (i) following the current trend in Bayesian deep learning the authors propose to use a variational approximation to the last-layer posterior, (ii) the authors introduce closed-form bounds on the ELBO for different likelihood functions, (iii) the authors show that the simple approach can result in improved performance for regression and some classification tasks.

--

I have adjusted my score based on the author's response.
Strengths: 1. The paper is well-written and easy to follow in most parts. Moreover, the work is well-motivated and I enjoyed that the authors brought back old ideas to the BDL community, e.g., using the discriminant analysis as a likelihood model.
2. I believe the exposition of the method is well done in most places, though slightly dense here and there, and helped in understanding the general idea of the proposed method. Moreover, I believe that the method is correct and an interesting contribution to the field. I think it is important to see more work on deterministic approaches to uncertainty quantification in deep learning. 
3. The experimental section shows promising results, especially in the case of regression.
Weaknesses: Overall: My main concern with the paper is the weak empirical evaluation and limited novelty of the work, that is, it seems it is essentially an application of known techniques to the special case of last-layer posteriors.

Comments:
1. Section 2.4 lists various related works, which I believe the author claims to optimize the log marginal via gradient descent. I have not checked every citation, but it appears to me that this statement is false for at least a subset of the cited papers. It might be good to revise the exposition.
2. Eq 12 is some weighted ELBO, weighted with T for the purpose of generality, according to the authors. However, T never seems to be used later and makes the connection to the common ELBO less transparent. I believe the paper would improve in clarity if T is dropped.
3. Section 3.4 is very dense and it could help the reader if this section is improved in its presentation. 
4. For the experiments, I would have expected assessments under distributional shift, a comparison to recent deterministic approaches (e.g., Zeng et al 2023 or Dhawan et al 2023), and a large-scale application of the approach as it acts on the last-layer only and should be applicable in more realistic scenarios (e.g., ImageNet).


Minor:
- Page 3, Eq 11 cites "Harrison et al 2018", which I looked up but didn't find any relevant content that would discuss the use of the marginal in Bayesian deep learning as a standard objective. What is the reason for the citation?

Citations:
- [Zeng et al 2023] Zeng et al, "Collapsed Inference for Bayesian Deep Learning", NeurIPS 2023.
- [Dhawan et al 2023] Dhawan et al, "Efficient Parametric Approximations of Neural Network Function Space Distance", ICML 2023.
Recommendation: 8: accept, good paper
Reviewer confidence: 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
Soundness/correctness: 3 good
Clarity: 3 good