Are you using test log-likelihood correctly?
Authors: Sameer Deshpande
Soumya Ghosh
Tin D. Nguyen
Tamara Broderick
Venue: Accepted by TMLR ()

Abstract: Test log-likelihood is commonly used to compare different models of the same data or different approximate inference algorithms for fitting the same probabilistic model. We present simple examples demonstrating how comparisons based on test log-likelihood can contradict comparisons according to other objectives. Specifically, our examples show that (i) approximate Bayesian inference algorithms that attain higher test log-likelihoods need not also yield more accurate posterior approximations and (ii) conclusions about forecast accuracy based on test log-likelihood comparisons may not agree with conclusions based on root mean squared error.

Published in Transactions on Machine Learning Research (01/2024)
Are you using test log- correctly?
Sameer K. Deshpande
University of Wisconsin-Madiion
Soumya Ghosh
ghoshso
MIT-IBM Watson Al Lab
IBM Research
Tin Nguyen"
tdn.@mit.. edu
MIT-IBM Watson AI Lab
Massachusetts Institute of Technology
Tamara Broderick
tbroderick@mii.. edu
MIT-IBM Watson AI Lab
Massachusetts Institute of Technology
Reviewed on OpenReview: https: // openeeview. net/ mett forum? id- id=n2Y6fD4Da
Abstract
Test log-likelihood is commonly used to compare different models of the same data or different approximate inference algorithms for fitting the same probabilistic model. We present simple examples demonstrating how comparisons based on test log-likelihood can contradict comparisons according to other objectives. Specifically, our examples show that (i) approximate Bayesian inference algorithms that attain higher test log- log-likelihoods need not also yield more accurate posterior approximations and (ii) conclusions about forecast accuracy based on test. lig-likelihood comparisons may not agree with conclusions based on root mean squared error.
1 Introduction
Test logllikl, also known as predictive log-likelihood or test log-predietive, is computed the log- predictive density averaged same data or to compare different algorithms used to fit the same probabilistic model. Although there are compelling reasons for this practice (Section 2.1), we provide examples that falsify the following, usually implicit, claims:
set of held-out data. It is often used to compare ciffferent models of the
The higher the test ligelioel,oood, the more accurately an approximate inference algorithm recovers the Bayesian posterior distribution of latent model parameters (Section 3).
Claim:
Claim: The higher the test log-likelihood, the better the predictive performance on held-out data
according to other measurements, like root mean squared error (Section 4).
Our examples demonstrate that test log-likelihood is not always good proxy for posterior approximation error. They further demonstrate that forecast evaluations based on test log-likelihood may not agree with forecast evaluations based on root mean squared error.
We are not the first to highlight discrepancies between test log-likelihood and other analysis objectives. For instance, Settioner--Candele et al. (2005) and Kohonen & Suomela (2005) showed that when predicting
These authors contributed equally to this work.

---

Published in Transactions on Machine Learning Research (01/2024)
discrete data with contimuuss distributions, test log- log-likelihood can be made arbitrarily large by concentrating probability into vanishingly small intervals. Chang et al. (2009) observed that topic models with larger test log-predietvve densities can be less interpretable. Yao et al. (2019) highlighted the disconnect between test log-likelihood and posterior approximation error in the context of Bayesian neural networks. Our examples, however, reveal more fundamental discrepancies between lest. log-likelihood and other evaluation metrics. particular, we show how comparisons based on test log- log-likelihood can contradict comparisons based on other objectives even in simple models like linear regression.
After introducing our notation, precisely define test log-likelihood and review arguments for its use in Section 2. In Sections 3.1-3.3, we present several examples showing that across a range of posterior approximations, those with higher test log- log-likelhhoods may nevertheless provide worse approximation quality. Then, in Section 3.4, we provide some intuition about why this phenomenon can occur when the model is severely misspecified (Section 3.1); when using sophisticated posterior approximation methods (Soction 3.2); and even when there little or no model misspccificttinn (Section 3.3). In Section 4, we show examples both complex and simple models where test. log-likelihodd higher but root mean squared error on held-out. data is worse. Our examples in Section 4 do depend on model misspecification, but we note that model misspccification is unavoidable in practice. We conclude in Scction 5 with reflection on when we should use test likeliool in practice.
2 Background
We assume we have access to training and lesting data such that all data points are independently and identically distributed (i.i.d)) from an unknown probability distribution P. Let D == {yn) 1 denote the training data. In many standard analyses, practitioners will have access to a predictive density of future data point y* given the observed D: n ( (D). For instance, consider the following three cases.
Case A: Practitioners often model the observed data by introducing a parameter 0 and specifying that the data are i.i.d. from a conditional distribution [0 with density T TV/0). In a non-Bayesian analysis, one usually computes a point estimate Ô of the unknown parameter (e.g. by maximum likelihood). Given a point estimate 0, the predictive density Tty" (D) is just TV= (0).
Case B: A Bayesian analysis elaborates the conditional model from Case A by specifying a prior distribution II(0) and formally computes the density (0(2)) of the posterior distribution [I(0)) from the assumed joint distribution II(D, 0). The Bayesian posterior predictive density is given by
<ly" (D) = / T (()')))
(1)
Case C: An approximate Bayesian analysis proceeds as in Case B but uses an approximation in place of the exact posterior. If we let (()) represent an approximation to the exact posterior, Equation (1) yields the approximate Bayesian posterior predictive density m(y"[D), Sometimes, due the difficulty of the integral in Equation (1), further approximation may be used to yield a predictive density (D)
In all of these cases, we will refer to the practitioner as having access to a model II that determines the predictive distribution [D); in particular, we allow "model" henceforth to encompass fitted models and posterior approximations. One can ask how well the resulting [Iy"" predicts new data generated from P. Practitioners commonly assess how well their model predicts out-of-sample using a held-out set of testing data D* {ur W which was not used to train the model. To compute test log-likel,hood, they average evaluations of the log- ore-predietive density function over the testing set:
TLL(D";11) = 1 Σ log ग (v,lD))
(2)
where our notation makes explicit the dependence of the test ligelikelihood (TLL) on testing data D and the chosen model IT. In particular, researchers commonly test logllikelihood to select between two models of
2

---

Published in Transactions on Machine Learning Research (01/2024)
the data, say II and II; that is, they select model II over II whenever TLL(D";II] :I] is higher than TLL(D";II) Note that the abbreviation NLPD (negative log predictive density) is also commonly used in the literature for the negative TLL (Quinonero-Caii discuss some alternative metrics for model comparison.
et al., 2005; Kohonen & Suomela, 2005). In Appendix C, we briefly
2.1 The case for test log-likelihoo likelihood
In what follows, we first observe that, if we wanted to choose a model whose predictive distribution is closer to the true data distribution in a certain KL sense, then equivalent to choose a model with higher expected log- sog-predictive density (elpd). Second, we observe that TLL a natural estimator of elpd when we have access to finite dataset.
The unrealistic case where the true data-generating distribution is known. The expected log- predictive density is defined
clpd(II) := f log x (()")D)dP()))
f
Our use the abbreviation elpd follows the example of Gelman et al. (2014, Equation 1). If we ignore an additive constant not depending on II, elpd(IT) is equal to the negative Kullback-Leibler divergence from the predictive distribution (D) to the true data distribution P(y"). Specifically, if we assume P has density p(y"), we have
KL (P(y") (1y = = plu") log plyppyyyy"y elpd(II).
Thus, eIId(II)= elpd((i) if and only if the predictive distribution (Py is closer, in a specific KL sense, the true data distribution than the predictive distribution Ii(y" (D) is.
Test log-likclihood as an estimator. Since we generally do not know the true generating distribution P. computing eIId exactly is not possible. By assumption, though, the test data are i.i.d. draws from P. So TLI.(D"; II) is a computable Monte Carlo estimate of eIId(II). If we assume elpd(IT) is finite, it follows that a Strong Law of Large Numbers applies: as N + 20, TLL(D";1) converges almost surely to elpd(1I). Therefore, with a sufficiently high amount of testing data, we might compare the estimates TLL(D"; II) and TLI:(D*;;) in place of the desired comparison of elpd and elpd(TT). Note that the Strong Law follows from the assumption that the y values are iidd. under P; it does not require any assumption on the model II and holds even when the model Il is misspecified.
2.2 Practical concerns
Since TLL(D*;II) is an estimate of eIId(II), it is subject to sampling variability, and careful comparison would ideally take this sampling variability into account. We first elaborate on the problem and then describe one option for estimating and using the sampling variability in practice; we take this approach in our experiments below.
To start, suppose we had another set of N testing data points, Dr. Then generally TLI. II) * TDL IT) So it possible, in principle, draw different conclusions using the TLL based on different testing datasets. We can more reasonably express confidence that elpd(II) is larger than elpd if the lower bound of confidence interval for (IId exceeds the upper bound of confidence interval for elpd(II).
We next describe one way to estimate useful confidence intervals. To do so, we make the additional (mild) assumption that
TiLL (II) : f [log (D) elId dP(y") xo.
Then, since the ve are i.dd draws from P, Central Limit Theorem applies: as N oc,
(TLL II) - eIId(II)) $ (0 (II,
Although we cannot generally compute TTLL we can estimate it with the sample standard deviation @TT.((I) of the evaluations (D) N' The resulting approximate 95% confidence interval for elpd(IT)
3

---

Published in Transactions on Machine Learning Research (01/2024)
is TLL(D"; 20T
/0TLL/VN /N=.. In what follows, then, we will conclude elpd(II) > elpd(II) if
TLI(D";;1) 20T (I////- TLL(D"; IT) + (0)
(3)
For the sake of brevity, we will still write TLL(DD"; IT) TLI,(D"; fT) in place of Equation (3) below.
To summarize: for a sufficiently large test dataset D", we expect predictions made from a model with larger TLL to be closer (in the KL sense above) to realizations from the true mata-generating process. In our experiments below, we choose large test datasets so that we expect TLL comparisons to reflect elpd comparisons. Our experiments instead illustrate that closeness between [T (D) and P (in the KL, sense above) often does not align with different stated objective.
3 Claim: higher test log-likelihood corresponds to better posterior approximation
In this section, we give examples where lest. log-likelihodd higher though the (approximation) quality of approximate posterior mean, variance, other common summary is lower. We start with examples in misspecified models and then give correctly specified example. We conclude with discussion of the source of the discrepancy: even in the well-specified case, the Bayesian posterior predictive need not be close to the true data-generating distribution.
Practitioners often use posterior expectations to summarize the relationship between a covariate and a response. For instance, the posterior mean serves as a point estimate, and the posterior standard deviation quantifies uncertainty. However, the posterior density <(((() is analytically intractable, practitioners must. instead rely on approximate posterior computations. There are myriad approximate inference algorithms e.g. Laplace approximation, Hamiltonian Monte Carlo (HMC), mean-field variational inference, to name just a few. All these algorithms aim to approximate the same posterior I[(0)D). Test log- rigliiklihhod is often used to compare the quality of different approximations, with higher TLL values assumed to reflect more accurate approximations, e.g. the context of variational inference (see, e.g., Hoffman et al., 2013; Ranganath el al., 2014; Hernández-Lobato et al., 2016; Liu & Wang, 2016; Shi al., 2018) or Bayesian deep learning (see, e.g., Hernández-Lobate & Adams, 2015; Gan et al., 2016; Li et al., 2016; Louizos & Welling, 2016; Sun et al., 2017; Ghosh et al., 2018; Mishkin et al., 2018; Wu et al., 2019; Izmailov et al., 2020; 2021; Ober & Aitchison, 2021).
Formally, suppose that our exact posterior is II(0)D) and that we have two approximate inference algorithms that produce two approximate posteriors, respectively (0 and 11 DD). The exact posterior and its approximations respectively induce predictive distributions ID()))D), (v and (12 For instance, iT (0)D) could be the empirical distribution of samples drawn using HMC and (T( could be a mean-field variational approximation. Our first example demonstrates that it is possible that (i) TLL(D"; fI,) TLL(D"; II) but (ii) using Ii, could lead to different inference about model parameters than using the exact posterior II. Our second example demonstrates that it is possible that (i) TLL(D";1) mI,) > TLL(D"; Il,) but (ii) f1, (G) is worse approxination to the exact posterior IT(0[D) than (T)(D)
3.1 TLL and downstream posterior inference
Relying TLL for model selection can lead to different inferences than we would find by using the exact posterior. To illustrate, suppose we observe D100 = {( )00 drawn from the following heteroscedastic model:
In ~ A((0 1), Yn In ~ Nar,, 1 + log(1 + exp(x.))).
(4)
Further suppose we model these data with misspecified homoscedastic model:
0 O[ 00
yn 0, ~NO0 On, 1),
(5)
where d. = [Tn, IJ_ and 0 = [0,, 02). Figure/ shows the posterior mean and the 95% predictive interval of the misspecified regression line 8 o from (A) the exact Bayesian posterior; (B) the mean field variational approximation restricted isotropic Gaussians; and (C)-(F) variational approximations with ro-scaled marginal variances. Each panel includes a scatter plot of the observed data, D100- We also report the 2- Wasserstein distance between the exact posterior and each approximation and the TLL averaged over
4

---

Published in Transactions on Machine Learning Research (01/2024)
0.000,
-1.828
0,000, -1.829
0.033, -1.798
(A)
(B)
(c)
10
0-
(E)
(F)
test log- likelihoodd
-1.750
(D)
10
-1.775
Approximations
0.101. -1.772
0.178. -1.755
0.433,
-1.732
(C)
Exact posterior
(D)
(E)
(F)
10
-1.80
0-
-1.825
-10-
0.0
0.2
0.4
-25 0.0 2.5 -25
0.0 2.5 x -2.
2.5
2-sssssstein distance to the posterior
x
Figure 1: (Left). Predictive distributions under the Bayesian posterior and mean field variational approx- imations. The two numbers in the title of cach plot are the 2-Wasscrstcin distance to the exact postcrior and test log- lieliikl computed on 10* test set observations. Two standard errors in the test bog-iikeihoodd estimate are (A) 0.03, (B) 0.03, (C) 0.02, (D) 0.02, (F) 0.02, (F) 0.02. (Right). The relationship between 2- Wasserstein distance to the posterior and test log- lieeliol..o.dd
N 10 test data points drawn from Equation (4); note that the 2- Wassersttii distance can be used to bound differences in means and variances (Huggins et al., 2020). The variational approximation (panel (B) of Figure 1) quite accurate: the Wasserstein distance between the approximation and the exact posterior is ~10-4 See also Figure 2, which shows the contours of the exact and approximate posterior distributions. As scale up the variance of this approximation, but the posterior predictive distribution covers more data, yielding higher TLL. The left panel of Figure 11 in Appendix B.3 shows the same pattern using the KI. divergence instead of the 2- Wasserstein distance.
move away from the exact posterior over the parameters
TLL and a discropancy inferences. Researchers are often interested in understanding whether there is a relationship between a covariate and response; a Bayesian analysis will often conclude that there is no relationship if the posterior on the corresponding effect-size parameter places substantial probability on an interval not containing zero. In our example, we wish check whether 0, 0. Notice that the exact positive 01 values. The 95% credible
posterior distribution (panel (A) in Figures 1 and 2) is concentrated interval of the exact posterior' is [0.63, 1.07]. Since the interval does not contain zero, would infer that 01 * O. On the other hand, as the approximations become more diffuse (pancls (B)-(F)), TLL increases, and the approximations begin to place non-negligible probability mass on negative 0, values. In fact, the approximation with highest TLL (panel (F) in Figures 1 and 2) yields an approximate 95% credible interval of (-0.29,1.90) which covers zero. Had we used this approximate interval, we would have failed to conclude 0 # That is, in this case, we would reach a different substantive conclusion about the effect 6 if we (i) the exact posterior (ii) the approximation selected by highest TLL.
3.2 TLL in the wild
Next, we examine more realistic scenario in which the difference between the quality of the posterior approximation and the exact posterior distribution TLL arises naturally, without the need to artificially increase the marginal variance of the variational approxinations. To explore this situation, we will first. introduce another example of misspecification and repeat the type of analysis described in Section 3.1.
Throughout used symmetric credible intervals formed by computing quantiles: the 95% interval is equal the 2.5%-97.5% interquantile range.
5

---

Published in Transactions on Machine Learning Research (01/2024)
(A)
(C)
1
0-
2
(D)
(E)
(F)
8:
81
Figure 2: Contours of (A) the exact posterior, (B) the mean field variational approximation restricted to
isotropic Gaussians, and (C)-(F) re-scaled mean field approximations. The line 01 = is highlighted in red.
20
0.000, -3.118
0.004, 22.998
0.006, -2.979
(A)
(B)
(c)
(A)
(B)
(C)
0-
0
-1
20
0.009, -2.907
0.202, -2.010
2.135, -1.686
(D)
(D)
(E)
(F)
1
S
0
0+
-2.5 0.0 X
2.5
2.5
0.0
2.5
-255 0.0
2.5 X
2
X
0
0
0
Figure 3: (Left). Predictive distributions under the Bayesian posterior (A) and the SWAG posterior with SWAG learning rate of (B) 10-3, (C) 10-2, (D) 10-1, (E) 1, and (F) 10. The two numbers in the title of each plot are the 2- Wasserstein distance to the exact posterior and test log-likelihood computed on 10a test set observations. Two standard errors in the test log-likelihood estimates are (A) 0.16, (B) 0.15, (C) 0.14, (D) 0.13, (E) 0.05, (F) 0.01. (Right). Contours of the (A) exact posterior, and (B)-(F) SWAG approximations with different. learning rates. The line 01 0 is highlighted in red.
Consider the following case: we observe 500 observations D500 = {(In, 300 drawn from non-lineat model:
0. 0- -1]T, En (0,,1), Vn 0., NOO. or 6,,t40 +
(6)
where on = [[n,1]] T Further suppose we modeled these data with a misspecified linear model
0 ~ 0~
on NOOTT Φn 0.5).
(7)
While the misspecification here might appear egregions, lincar models are widely used in practice for modeling non-linear phenomena when one is primarily interested in inferring whether the covariates are posilively correlated, negatively correlated, or are uncorrelated with the responses (Berk et al., 2014; 2018; Blanca et al., 2018; Vowels, 2023). Next, we use SWAG (Maddox et al., 2019), an off-tho-shelf approximate inference algorithm, approximate the posterior IIO)) We also repeat the re-scaled variational inference experiment from Section 3.1 with this set of data and models (Equations (6) and (7)); Appendix B.2.

---

Published in Transactions on Machine Learning Research (01/2024)
SWAG uses gradient-basee optimizer with learning rate schedule that encourages the optimizer to oscillate around the optimal solution instead of converging to it. Then, a Gaussian distribution is fit. to the set of solutions explored by the optimizer around the optimum using moment matching. In general, one must select the learning rate schedule in a hcuristic fashion. One might be tempted to use TLL to tune the learning rate schedule. We use this heuristic and run SWAG for a thousand epochs, annealing the learning rate down to a different constant value after 750 epochs. Although used pedagogically here, similar heuristics have been used in practice (di Langosco al., 2022), where the learning rate tuned based on the accuracy achieved held-out data. We vary this constant value over the set {10-3, 10 10 10). In Figure 3, show the resulting posterior mean and the 95% predictive interval of the misspecified regression line 0 from (A) the Bayesian posterior; (B) (F) the SWAG posteriors using different learning rate schedules. In each plot, we overlay the observed data D500 (black dots) with the true data generating function in dashed black. We also report the 2- Wasserstein distance between the exact posterior and each approximation and the TLL averaged over N 10 test data points drawn from Equation (6). In all cases, SWAG overestimates the postcrior variance, with predictive distributions that better cover the data and consequently lead to higher TLL. However, these SWAG posterior approximations learning rate of 10 (Figure 3, Left, panel (F)) maximized TLL but led to the worst approximation of the exact posterior.
forther from the exact posterior. In fact, found that
As in the previous section, next suppose we fit this misspecified linear model to understand whether there is a relationship between the covariates and the responses, i.e., whether =. Notice that the exact posterior distribution (Figure 3, Right, panel (A)) is concentrated on negative 81 values, with the 95% posterior credible interval being [- [-96 -1.79). Since the interval is to the left of zero, would infer that 0, O and that the covariate and the response are negatively correlated. In contrast, if sclect the SWAG approximation with the highest TLL,
select the posterior approxination in panel (F) the right side of Figure 3. The corresponding 95% posterior credible interval is [-4.46, 0.74], which places non non-negligible probability mass on 0, > 0. this case, we would not conclude that the response and the covariate are negatively correlated by contrast to the conclusion using the exact posterior.
0.000, -0.416 0.002, -0.436 (B)
0.012, -0.382 (C)
(A)
-1.0
-0.38
-1.3
-1.6
True param
test log-likelihood
-0.40
0.019, -0.384
0.035, -0.396
0.052,
-0.416
(D)
(E)
(F)
-1.0
-0.42
S -1.3
Variational approximations
-1.6
Exact posterior
-0.44
-2.4
-2.0
-1.6
-2.4
2.0
-1.6
-2.4
-2.0
-1.6
0.05 2-Wasserstein distance to the posterior
0.01
01
01
01
Figure 4: (Left). Contours of (A) the exact posterior, (B) the mean field variational approximation restricted to isotropic Gaussians, and (C)-(F) re-scaled mean field approximations. The two numbers in the title of each plot are the 2- Wasserstein distance to the exact posterior and test log- liglliiss computed on 10 test set observations. Two standard errors in the test log-likelihood estimates are (A) 0.019, (B) 0.020, (C) 0.014, (D) 0.013, (E) 0.011, (F) 0.009. (Right). The non-monotonic relationship between distance to posterior and test log- likelihood. Observe that the exact posterior does not achieve highest test log- leeelikel
3.3 TLL and well-specified models
The examples above demonstrated that TLL is not reliable praxy to posterior approximation quality when the model is misspecified. Though misspecified models the norm in practice, we demonstrate that

---

Published in Transactions on Machine Learning Research (01/2024)
distribution with higher TLL may not provide more accurate posterior approximation even when the model is correctly specified.
To this end, consider the following Bayesian linear model:
O~ 0.9, I)), yn 0. NO on, 0.252).
(8)
where on = [2 1]" Now, suppose we observe ten data points D10 { Un Un) m sampled as
0. = (-2 In ~ N), Yn 0n NOO on, on, 0.25*).
(9)
The left panel of Figure 4 plots the contours of (A) the exact posterior distribution (o (B) the mean field variational approximation constrained to the isotropic Gaussian family; and (C)-(F) variational approximations with re-scaled marginal variances. In each panel, we report the 2- Wasserstein distance between the approximate and exact posterior and the test log-predictive averaged over N 10 test data points drawn from Equation (9).
Although we have correctly specified the conditional model of yl(0,)) Φ), the exact posterior has lower TLL than some of the approximate posteriors; in particular, the 95% confidence intervals for (C) and (D) are disjoint from the 95% confidence interval for the exact posterior, shown in (A). The left panel of Figure 4 suggests that the more probability mass an approximate posterior places around the truc data-generating parameter, the higher the TLL. Eventually, as the approximation becomes more diffuse, TLL begins to decrease (Figure 4 (right)). The non-monotonicity demonstrates that an approximate posterior with larger implied TLL can in fact be further away from the exact posterior in 2- Wassersstnin sense than an approximate posterior with smaller implied TLL. The right panel of Figure 11 in Appendix B.3 demonstrates the same pallern using the KT, divergence instead of the 2- 2-Wasserseein distance. And Figure 9 in Appendix B.3 shows that, in the well-specified case, distribution with larger TLL can provide worse approximation of the posterior standard deviation than a distribution with smaller TLL.
3.4 What is going on?
We next discuss why we should not expect TLL to closcly track posterior approximation quality, or posterior- predictive approximation quality. Essentially the issue is that, even in the well-specified case, the Bayesian posterior predictive distribution need not be close to the true data-generating distribution.
We illustrate these distinctions Figure The lower surface represents the space of distributions over a latent parameter 0. The upper surface represents the space of distributions over an observable data point y². Each dot in the figure represents distribution. The two dots in the lower surface are the exact posterior I[([D) (left, green dot) and an approximate posterior [i(0[D) (right, red dot). The three dots in the upper surface are the posterior predictive distribution II(y (D) (left, green dot), the approximate posterior predictive (D() (lower right, red dot), and the true data- tetttaggnerrting distribution P(y") (upper right, black dot). The gray lines on the left and right indicate that the distribution in the upper surface can be obtained from the corresponding (connected) distribution in the lower surface via Equation (1).
The remaining three (non-gray) lines represent three different discrepancies. Recall from Section 2 that TLL(D"; captures how close the approximate posterior predictive Πi(y" (D) is to the true data-generating process P(y") in a particular KL sense:
TLL(D", m) KL (D))))) constant.
To illustrate this notion of closeness, or equivalently discrepancy, in Figure 5, we draw a pink line between P(y*) and Πo (D). We observe that the TLL importantly does not approximate (even up to a constant) the analogous discrepancy from the approximate posterior predictive iD(y*[D) to the exact posterior predictive [D)*[D) (blue line in the upper surface); that is, it does not capture how close the posterior prodictive approximation is to the exact posterior predictive. The TLL likewise does not approximate (even up constant) the corresponding discrepancy from the approximate posterior [T(ODD) to the exact. posterior
8

---

Published in Transactions on Machine Learning Research (01/2024)
P(y))
Data space
Hgg D)
TLL
Iiyy1D) (D)
Ii(e)D)
[1())))
Parameter space
Figure 5: Cartoon illustration highlighting the difference between three different discrepancies explored Section 3.4. The surfacos are spaces of distributions over latent paramcter (lower surface) or an observable data point y" (upper surface). The pink line indicates that TLL(D*; [f) estimates discrepancy between the approximate posterior predictive ii(y" (D) (upper surface, lower right, red dot) and the true data-generating distribution Ply") (upper surface, upper right, black dot). The blue line represents a different discrepancy between the exact posterior predictive (upper surface, left, green dot) and the approximate posterior predictive (upper surface, lower right, red dot). The yellow line represents another different discrepancy between the exact posterior (lower surface, left, green dot) and the approximate posterior (lower surface, right, red dot). Gray lines connect distributions over parameters with their corresponding predictive distributions.
[[(DD) (yellow line in the lower surface); that is, it does not capture how close the posterior approximation is to the exact posterior.
The pink and blue lines would (nearly) align if the posterior predictive were very close the true data- generating distribution. For misspecified model, the posterior predictive need not be close the true data-generating distribution. For well-specified model, the posterior predictive and true data-generatingg distribution may still be far for finite dataset. On that view, suggested by anonymous referee, we might expect the observed phenomenon to disappear asymptotically in the well-specified sctting if sufficient regularity conditionn hold. The argument, essentially, is that (i) the actual posterior [I(0)D) converges to a point- point-mass mass at the true data generating parameter; this convergence implies (ii) that the actual posterior predictive ID) converges to the true data distribution P(y*), from which it follows that for large enough training datasets (iii) KL (I()*)D)) I| ((r"D))
KL (P(x*) || (((r"))) However, we emphasize first that essentially every real data analysis is misspecified. And second, if a practitioner is in setting where they are confident there is no uncertainty in the unknown parameter value, there may be little reason to take a Bayesian approach or go to the conetimes-consideraae computational burden of approximating the Bayesian posterior.
4 Claim: higher test log-likelihooe corresponds to lower predictive error
As noted in Sections 2.1 and 3.4, TLL estimates how close predictive distribution from the true data- generating process in a specific KL sense. On that view and analogous to Section 3, we would not expect conclusions made by TLL to match conclusions made by comparing other predictive losses. Rather than

---

Published in Transactions on Machine Learning Research (01/2024)
focus more esoteric losses in experiments, note that TLL and RMSE often reported as default measures of model fit. quality in papers. If conclusions made between TLL. and RMSE do not always agree (as we expect and reinforce experimentally next), we should not expect TLL to always reflect performance according to other predictive losses beyond RMSE. If the TLL is of fundamental interest, this observation is of little consequence; TL.L is a convenient stand-in for a potential future loss of interest, this observation may be meaningful.
Misspecified Gaussian process regression. We next construct two models II and II such that III(D";II) < TLI(P";II) but II yields larger predictive RMSE. Suppose observe D100 {(In. Vn me) from the following data generating process:
In U(-5,+5) yn ynl ~ (((ssi((22))))0)))
(10)
Further suppose model this data using zero-mean Gaussian process (GP) with Gaussian noise,
GP k(x,z)),, Yn In ~ ((),,
(11)
where Jn is shorthand for f(x,). First consider the case where employ periodic kernel, constrain the noise nugget o² to 1.6, and fit. all other hyper-parameters by maximizing the marginal likelihood. The resulting fit is shown in Figure (A). Next, consider an alternate model where we use squared-exponential kernel and fit all ayper-parametert including the noise mugget via maximum marginal likelihood. The resulting Figure (B). The squared exponential model fails to recover the predictive mean and reverts back the prior mean (RMSE 0.737, 95% confidence interval [0.729, 0.745)), while the periodic model recovers the predictive mean accurately, measured by RMSE 0.355 (95% confidence interval 0.360).. Despite the poor mean estimate provided by the squared exponential model, it scores substantially higher TLL.
displayed
TLL = 11.9 - 0.00
(A)
2.5
0.0
-2.5
TLL = 11.08 + 0.01
(B)
2.5
0.0
-2.5
-7.5
-5.0
-2.5
0.0
2.5
5.0
7.5
10.0
Figure 6: The plots display two Gaussian processes trained on the same set of data (represented by black plus symbols). The dashed red line shows the mean of the posterior Gaussian process, while the red highlighted region represents the 95% predictive interval. The subplot titles display the TLL (+22 standard error) allained by each Gaussian process. Although the Gaussian process in panel (A) achieves better mean fit compared panel (B), it has a worse TLL when evaluated on 1 test instances (represented by black dots).
*PRRO in htwp:/hwww..mm//eefeeeeee//////
10

---

Published in Transactions on Machine Learning Research (01/2024)
In this example, we see that, even with accurate point estimate, TLL can be reduced by, for instance, inflating the predictive uncertainly. And this discrepancy between TLL and RMSE is not necessarily removed by optimizing the parameters of a model.
Misspecified linear regression. Our next example illustrates that even when all parameters in a model are fit. with maximum likelihood, comparison based on TI.I. may still disagree with a comparison based on RMSE also illustrates that the discrepancy between TLL and RMSE can arise even in very simple and low-dimensional models and even when the training dataset is very large.
100,000 Specifically, suppose that we observe D = ((In, n= generated according to
In ((0025)) yn Laplace(zn, 1/v2),
(12)
which we model using one of the following misspecified conditional linear models:
11 yn In N(xx o²)
or
(13)
î Un Fn Taplace(0.45, Ozn, A).
Both II and If depend on two unknown parameters. II depends on slope 9 and a residual variance o² and II depends on a slope 0 and residual scale A. The kind of misspecifictiin is different across models; while IT has the correct mean specification but incorrect noise specification, has incorrect mean specification but correct noise specification.
We computed the maximum likelihood estimates (MLEs) (0n, δn) and (0 for both models. The two fitted models induce the following predictive distributions of y'''"* l=
[yy"<=",2 D) y² lx* NOn o
and
(14)
ii(y"I=", D) |-* ~ 10..ace(0.45 + "n==">pp>
The means of these predictive distributions are natural point estimates of the output y² at input xt.
Using a test set of size N = 395,000, we observed TLL(D";;I) 1.420 < -1.389 = TLL(D"; Ii). The standard error of either TLL estimate is only 0.002. Hence, based on sample mean and standard error, we conclude that IT has better elpd than IT. These values suggest that on average is closer to Ply*l=*) than [Ty"|>*, D) in a KL sense. However, using the same test set, we found that II yielded more accurate point forecasts, measured by root mean square error (RMSE):
inputs x", [y*""==, D)
1/2
1/2
Σ Ene (0=))xx))
1.000 < 1.025
1
Σ (vt 0.45 0722)²
(15)
In addition, the 95% confidence intervals for the RMSE do not overlap: the interval for II's RMSE is [0.997, 1.005] and that for IT's RMSE is [1.022, 1.029]. The comparison of RMSEs suggests that average over inputs =², the predictive mean of [Ty xx", D) is closer to the mean of P(y" (x*) than the predictive mean of [2" D). In other words, the model with larger TLL whose predictive distribution is ostensibly closer to P makes worse point predictions than the model with smaller TLL.
5 Discussion
Our paper is neither a blanket indictment nor recommendation of test log-likelihood. Rather, we hope to encourage researchers to explicitly state and commit to a particular data-analysis goal and recognize that different methods may perform better under different goals. For instance, when the stated goal is approximate (summary statistics of) Bayesian posterior, we argue that it inappropriate rely on test log-likelihood to compare different approximation methods. We have produced examples where model
11

---

Published in Transactions on Machine Learning Research (01/2024)
can provide a better test log-likelihood but yield (much) poorer approximation to the Bayesian posterior in particular, leading to fundamentally different inferences and decisions. We have described why this phenomenon occurs: test log-likelihood tracks closeness of approximate posterior predictive distributions to the data-generating process and not to the posterior (or posterior predictive) distribution. At the same time, we recognize that evaluating posterior approximation quality is a fundamentally difficult problem and will generally necessitate the use of a proxy. It may be useful to consider multiple of the available options; a full accounting is beyond the scope of this paper, but they include using conjugate models where exact posterior summary statistics are available; comparing to established MCMC methods on models where sufficiently large compute budget might be expected to yield reliable approximation; simulation-based calibration (Talts et al., 2018); sample-quality diagnostics (Gorham & Mackey, 2015; Chwialkowski et al., 2016; Liu et al., 2016); and a host of visual diagnostics (Gabry et al., 2019). A careful investigation to understand how a particular method struggles or succeeds may be especially illuminating.
On the other hand, in many data analyses, the goal is to make accurate predictions about future observables identify whether treatment will help people who receive it. In these cases and many others, using a Bayesian approach is just one possible means to an end. And many of the arguments for using the exact Bayesian posterior in decision making assume correct model specification, which we cannot rely upon in practice. In predictive settings in particular, test log- logliikelhoodd may provide a compelling way to assess performance. In addition to being essentially the only strictly proper local scoring rule (Bernardo & Smith, 2000, Proposition 3.13), TLL sometimes advertised as a "non-informative" choice of loss function (Robert, 1996). Importantly, however, non-informative does not mean all.encompassing: as our examples in Section 4 show, test log-likelihood does not necessarily track with other notions of predictive loss. As we discuss in Section 2.1, test log-likclihood quantifics predictive discrepancy only in a particular Kullback-Leible sense. It is important to note, however, that just because two distributions are close in KL, their means and variances need not be close; in fact, Propositions 3.1 & 3.2 of Huggins et al. (2020) show that the means and variances of distributions that are close in KI. can be arbitrarily far apart. So even in settings where prediction is of interest, we recommend users clearly specify their analytic goals and use evaluation metrics tailored to those goals. Tf there a quantity of particular interest the data-generating process, such as moment or quantile, a good choice of evaluation metric may be an appropriate scoring rule. Namely, one might choose a scoring rule whose associated divergence function is known to quantify the distance between the forecast's 's quantity of interest and that of the data-generating process. For instance, when comparing the quality of mean estimates, one option is using the squared-error scoring rule, whose divergence function is the integrated squared difference between the forecast's mean estimate and the mean of the generainnn process. Another option is the -ewad-Sebaatianii score (Dawid & Sebastiani, 1999), which prioritizes accurately estimating predictive means and variances. See Gneiting & Raftery (2007) for a list of commonly used scoring rules and their associated divergences.
Acknowledgments
We are grateful to Will Stephenson for helping us find examples of discrepancies between posterior approxi- mation quality and TLL.
This work was supported in part by the MIT-IBM Watson AI Lab, an NSF Career Award, an ONR Early Career Grant, the DARPA 120 LwLL program, an ARPA-E project with program director David Tew, and the Wisconsin Alumni Research Foundation.
References
Richard Berk, Lawrence Brown, Andreas Buja, Edward George, Emil Pitkin, Kai Zhang, and Linda Zhao. Misspecified mean function regression: Making good use of regression models that are wrong. Sociological Methods & Research, 43(3):422-45) 451, 2014.
Richard Berk, Lawrence Brown, Andreas Buja, Edward George, and Linda Zhao. Working with misspecified regression models. Journal of Quantätative Oriminology, 34:633 655, 2018.
12

---

Published in Transactions on Machine Learning Research (01/2024)
Robert H. Berk. Limiting behavior of posterior distributions when the model is incorrect. Annols of Mathematical Statistics, 37(1):51-555 -58,,8, 1966.
José M Bernardo and Adrian F.M. Smith. Bayesian Theory. Wiley, 2000.
María J Blanca, Rafael Alarcón, and Roser Bono. Current practices in data analysis procedures in psychology: What has changed? Frontiers in Psychology, 9:2558, 2018.
Jonathan Chang, Sean Gerrish, Chong Wang, Jordan Boyd-Graben, and David Blei. Reading tea leaves: How
humans interpret topic models. Advances in Neural Information Processing Systems, 22, 2009.
Kacper Chwialkowski, Heiko Strathmann, and Arthur Gretton. A kernel test of goodness of fit. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
A. Philip Dawid and Paola Sebastiani. Coherent dispersion criteria for optimal experimental design. Annals of Statisties, (761) 81, 1999.
Lauro Langosco di Langosco, Vincent Fortuin, and Heiko Strathmann. Neural variational gradient descent. In Fourth Symposium on Aduances in Approzimate Bayesian Inference, 2022.
Jonah Gabry, Daniel Simpson, Aki Vehtari, Michael Betancourt, and Andrew Gelman. Visualization in Bayesian workflow. Journal of the Royal Statisticnl Society Series A, 182( :899 402, 2019.
Zhe Gan, Chumyuan Li, Changyou Chen, Yunchen Pu, Qinliang Su, and Lawrence Carin. Scalable Bayesian learning of recurrent neural networks for language modeling. arXiv pre-print arXiv:1611.05034 2016.
Andrew Gelman, Jessica Hwang, and Aki Vehtari. Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24:997-1016, 2014.
Soumya Ghosh, Jiayu Yao, and Finale Doshi-Velez. Structured variational learning of Bayesian neural networks with horseshoe priors. In Proceedings of the 35th International Conference on Machine Learning. 2018.
Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association, 102:359 378, 3 2007.
Jackson Gorham and Lester Mackey, Measuring sample quality with Stein's 's method. Advances in Neural Information Processing Systems, 28, 2015.
José Miguel Hernández-Lobato and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the 32nd International Conference on Machine Learning. 2015.
José Miguel Hernández-Lobato, Yingzhen Li, Mark Rowland, Daniel Hernández-Lobato, and Richard Turner. Black-box a-divergence minimization. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
Matthew D. Hoffman, David M Blei, Chong Wang, and John Paisley, Stochastic variational inference. Journal of Machine Learning Research, 14:1303-1347, 2013.
Jonathan H. Huggins and Jeffrey W. Miller. Reproducible model selection using bagged postcriors. Boyesian Analysis, 18(1):79-10)) 100,,9999,41111111 2023.
Jonathan H. Huggins, Mikolaz Kasprzak, Trevor Campbell, and Tamara Broderick. Validated variational inference via practical posterior error bounds. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics, 2020.
Pavel Izmailov, Wesley J Maddox, Polina Kirichenko, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Subspace inference for Bayesian deep learning. In Uncerlainty in Artificial Intelligence. 2020.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Pavel Izmailov, Sharad Vikram, Matthew D. Hoffman, and Andrew Gordon Wilson. What are Bayesian neural work posteriors really like? In Proceedings of the 38th International Conference on Machine Learning. 2021.
Jukka Kohonen and Jukka. Suomela. Lessons learned in the challenge: making predictions and scoring them. In Machine Learning Challenges Workshop, pp. 95-116. Springer, 2005.
Chunyuan Li, Changyou Chen, Kai Fan, and Lawrence Carin. High-order stochastic gradient thermostates for Bayesian learning of deep models. In Proceedings of the Thirtieth AAAI Conference on Artifical Intelligence. 2016.
Qian Liu and Dilin Wang. Stein variaitonal gradient descent: A general purpose Bayesian inference algorithm. In Advances in Neural Informational Processing Systems. 2016.
Qiang Liu, Jason Lee, and Michael Jordan. A kernelized Stein discrepancy for goodness-of-fit tests. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
Sanae Lofti, Pavel Izmailov, Gregory Benton, Micah Goldblum, and Andrew Gordon Wilson. Bayesian model selection, the marginal likelihood, and generalization. In Proceedings of the 39th International Conference Machine Learning. 2022.
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In Proceedings of the 33rd International Conference on Machine Learning. 2016.
David J.C. MacKay. Information Theory, Inference, and Learning Algorithms. Cambridge University Press, 2003.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for Bayesian uncertainty in deep learning. Advances in Neural Information Processing Systems, 32, 2019.
Aaron Mishkin, Frederik Kunstner, Didrik Nielsen, Mark Schmidt,, and Mohammad Emtiyaz Khan. SLANG: Fast structured covariance approximations for Bayesian deep learning with natural gradient. In Advances in Newral Informational Processing Systems. 2018.
Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes. In Proceedings of the 38th International Conference on Machine Learning. 2021.
Joaquin Cuiiionco-Candee Carl Edward Rasmussen, Fabian Sinz, Olivier Bousquet, and Bernhard Schölkopf. Evaluating predictive uncertainty challenge. In Machine Learning Challenges Workshop, pp. 27. Springer, 2005.
Rajesh Ranganath, Sean Gerrish, and David M Blei. Black box variational inference. In Proceedings of the 17 International Conference on Artificial Intelligence and Statistics, 2014.
Christian P Robert. Intrinsic losses. Theory and Decision, 40:191-211 1244, 1996.
Jiaxin Shi, Shengyang Sun, and Jun Zhu. Kernel implicit variational inference. In International Conference on Learning Representations. 2018.
Shengyang Sun, Changyou Chen, and Lawrence Carin. Learning structured weight uncertaitny in Bayesian neural networks. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. 2017.
Sean Talts, Michael Betancourt, Daniel Simpson, Aki Vehtari, and Andrew Gelman. Validating Bayesian inference algorithms with simulation- basslt calibration. arXiv preprint arXiv:1804.00 067888 2018.
Matthew J Vowels. Misspeeification and unreliable interpretations in psychology and social science. Psycho- logical Methods, 28(3):507, 2023.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Anqi Wu, Sebatian Nowozin, Edward Meeds, Richard E Turner, José Miguel Mernández-Lobatte and Alexander I. Gaunt. Deterministic variational inference for robust Bayesian neural networks. In International Conference on Learning Representations. 2019.
Jiayu Yao, Weiwei Pan, Soumya Ghosh, and Finale Vesli-Velez. Quality of uncertainty quantification for Bayesian neural network inference. ar pre-print Xix::1966096688 2019.
15

---

Published in Transactions on Machine Learning Research (01/2024)
A Variational Approximations
In Section 3 we formed isotropic Gaussian approximations to the exact posterior. In our illustrative examples, the exact posterior itself is a Gaussian distribution, (w, Σ). In Sections 3.1 and 3.3 we use variational approximations that share the same mean as the exact posterior and are isotropic, (H, where is a two-dimennional identity matrix and > is a scalar. In this family distributions, the optimal variational approximation is N(p, (p, p"I), where,
p= argmin KL (W(H, (H,E)), PER
(16)
2
tr(x-1)
The result follows from setting the gradient v,KL (N pl) (M,, E))) to zero and rearranging terms,
tr
V KT. (Ni(,, pl) (1, E))) = 0, = V
Ip
(17)
tr(E-1) 2
2
p tr(2-1))
Note that p" is guaranteed to be positive since E-1 is positive definite and thus tr(2-1) > O. This optimal variational approximation, (M, p'I) is used in Panel (B) of Figure 1, Figure 2, and Figure 4. The other panels use ((,, Ap*I), I), with À E [1, 5, 10, 15, 30) for Figure and Figure 2. For Figure 4 (Left), À takes values in 7, and in [1, 2, 4, 4, 6, 8, 9, 10, 11] for Figure 4 (Right).
B Experimental details and additional experiments
B.1 Confidence Intervals
An additional note on confidence intervals for TLL. Suppose we are comparing two models II and II. Although TLL(D"; II) (respectively, (II) will generally be correlated with TLL(D"; îī) (respectively, (TT), we do not expect more careful treatment of that correlation to change our substantive conclusions.
Confidence intervals for RMSE. To compute the RMSE confidence interval, we first compute the mean of the squared errors (MSE, m) and its associated standard error of the mean (s). Since have large number of data points and the MSE takes the form of mean, assume the sampling distribution of the MSE is well-approximated by normal distribution. We use [m - 2s, m + 2s] as the 95% confidence interval for the MSE. We use [vm - 28, vm + 28] as the 95% confidence interval for the RMSE. Note that the resulting RMSE confidence interval will generally not symmetric.
B.2 Additional TLL in the wild experiments
SWAG with higher learning rates. In Figure 7 we continue the experiment described in Section 3.2 but using higher lcarning rates of 12, 15, and 20. Despite moving further from the exact posterior the test log-likelihood remains higher than those achieved by SWAG approximations with lower learning rates (panels (B) through (E) of Figure 3).
Mean field variational inference. Next, we reproduce the experimental setup described in Section 3.2, mean field variational inference and examine
but instead of using SWAG to approximate the posterior, the relationship between TLL and posterior approximation quality under different re-scalings of the marginal variance of the optimal variational approximation. Figure 8 shows the posterior mean and the 95% predictive interval the misspecified regression line 8 & from (A) the Bayesian posterior; (B) the mean field variational approximation restricted isotropic Gaussians; and (C)-(F) several ro-scaled variational approximations. In each plot, we overlaid the observed data Ds00, the true data generating function dashed black, and also report the 2- Wasserstein distance between the true posterior and each approximation and the TLL
16

---

Published in Transactions on Machine Learning Research (01/2024)
20
-1.722
3.770,
-1.780
5.564 -1.879
[G)
H))
(G)
(H)
(I)
0
0-
-2.5
0.0
2.5
2.5
0.0
X -2.5
2.5
0.0
2.5
2.5
0.0
-25
0.0
225
0.0
X
0
0
0
Figure 7: (Left). Predictive distributions under the SWAG posterior with SWAG learning rate of (G) 12, (H) 15, (I) 20. The two numbers in the title of each plot are the 2-Wasserstein distance to the exact posterior and test log- oi-liikl computed on 10* test set observations. Two standard errors in the test log- bgg-ikkeihoood estimates are (G) 0.01, (H) 0.009, (T) 0.08. (Right). Contours of the SWAG approximations with different. learning rates. The line 0, = 0 is highlighted in red.
0.000, -3.118 (A)
0,000, -3.118
0.545. -1.765
20
(B)
(C)
-1.8
test log-likelihood
-2.2
Variational approximations
0.738, -1.712
1.323. 1.622
93.792,
-3.128
Exact posterior
20
(D)
(E)
(F)
-2.6
0
-3.0
-25 0.0 2.5 X
2.5
0.0 2.5 -2.5
0.0 2.5
15
-10
-5
0
log 2-Wasserstein distance to the posterior
X
X
Figure 8: (Left). Predictive distributions under the Bayesian posterior and mcan field variational approxima- tions. The two numbers the title of each plot are the 2-Sassersnein distance to the true posterior and test. log-likelthoods computed on 10 test set obscrvations. Two standard errors in the test log-likelihood estimates are (A) 0.16, (B) 0.16, (C) 0.03, (D) 0.02, (E) 0.02, (F) 0.01. (Right). The relationship between distance to posterior and test log- predictive density. Observe the log scale of the horizontal axis and the non-monotonic relationship between test log-predictive density and 2- Wassersstei distance to the Bayesian posterior.
averaged over N 10 test data points drawn from Equation (4) Like in our previous example, the mean field approximation (panel (B) of Figure 8) is very close to the exact posterior. Further, the marginal variance of the approximate posteriors, the posterior predictive distributions cover more data, yielding higher TLL, while simultaneonsly moving away from the exact posterior over the model parameters in a. 2- 2-Waaserstein sense. Interestingly, when the approximation is diffuse enough, TLL decreases, again highlighting its non-monotonic relationship with posterior approximation quality. this example of a misspecified model, the non-monotonic relationship between TLL and Wasserstein distance means that. TLL is, best, poor proxy of posterior approximation quality.
scale up
B.3 The highest TLL does not match the best estimate of a posterior summary statistic or the lowest KL
We first reproduce the experimental setup that produced Figure 4, but now in Figure 9, we plot TLL against the error in estimating the posterior standard deviation. In particular, the horizonial axis shows the absolute value the difference between (a) the marginal standard deviation the parameters interest under the approximation and (b) the marginal standard deviation under the exact posterior. As in the right panel of Figure 4, we observe that the highest (best) TLL does not correspond to the lowest (best) error in estimating the posterior standard deviation.
17

---

Published in Transactions on Machine Learning Research (01/2024)
-0.38
test log liklliilhooodd
-0.40
-0.42
Approximations Exact posterior
Approximations Exact posterior
-0.44
0.05
0.10
0.15
0.00
0.05
0.10
0.15
0.20
Difference of 0 's standard deviation
Difference of 0z's standard deviation
Figure 9: The non-monotonic relationship between difference in marginal standard deviations and TLL in a well-specified case. (Teft) The horizontal axis reports the absolute difference in the standard deviation of the weight 8 between an approximation and the posterior. (Right) The horizontal axis reports the absolute difference in the standard deviation of the bias 02 between an approximation and the posterior.
To create Figure 10, we reproduce the experimental setup from Figure 8. Relative to the right panel of Figure 8, we change only what is plotted on the horizontal axis; for Figure 10, we plot the log of the absolute value of the difference between marginal standard deviations. Again, that the highest (best) TLL does not correspond to the lowest (best) error in estimating the posterior standard deviation.
Approximations
Approximations
test log lillkiiiioood
-2.0
Exact posterior
Exact posterior
-2.5
-3.0
8
-6
-2
-8
-6
-4
-2
0
Log difference of 0,'s standard deviation
Log difference of 0z's standard deviation
Figure 10: The non-monotonic relationship between difference marginal standard deviations and TLL in a misspecified case. The meaning of horizontal axis similar to that of Figure 9.
Finally, Figure 11 reproduces analyses from the main text but uses KL divergence instead of 2- 2-Wasserstein distance to measure posterior approximation quality, In particular, the left panel Figure recreates the right panel of Figure 1; as Figure 1, we see that the highest (best) TLL does not correspond the lowest (best) divergence value. Likewise, the right panel of Figure recreates the right panel of Figure 4; Figure 4, the highest. TLI. again does not correspond to the lowest divergence value.
18

---

Published in Transactions on Machine Learning Research (01/2024)
-0.38
-1.74
(E)
(F)
test log lggliiklihood
-1.76
(D)
test log- likelihooddo
- 0..40
-1.78
(C)
-1.80
-0.42
Approximations
-1.82 (B
Exact posterior
Variational approximations
Exact posterior
-0.44
5
10
15
20
25
0 1 2
3
4
5
6 7 8
KLlapproximation posterior)
KLlapproximation posterior)
Figure 11: The smallest KL divergence does not correspond the largest TLL, misspecified case (left) and well-specified case (right). The left panel reproduces the experimental results presented in the right panel of Figure 1, but uses the reverse KL divergence measure discrepancy with the exact posterior instead of the 2- Wasserstein distance. The right panel reproduces the results in the right panel of Figure 4.
C Alternative model comparison metrics
In this work, we focused on one model comparison metric, test log-likelihodd. However, there are many other model comparison metrics. In what follows, we consider marginal likelihood and the joint log predictive density potential model comparison metrics.
C.1 Marginal likelihood
Instead of choosing the model with higher test log-likelihood, one might instcad choose the model with higher marginal likelihood, where the marginal likelihood for model is TD) = (0)) That is, given models IT and IT, one might choose whenever x(D) R(D);
As anonymous referee pointed out, the marginal likelihood is well-established Bayesian model selection criterion; see, e.g., pg. 348 of MacKay (2003). However, the marginal likelihood criterion has several well- known limitations. First, marginal likelihood comparisons can be unreliable when the models being compared are misspecified; see $2 and references within Huggins & Miller (2023), and see also Berk (1966). Beyond this concern, marginal likelihood quantifies only how well a model fits the available training data. It is hereewise silent or "peripherally related" (Tofti et al., 2022) to the predictive quality of the model. And so, just because model yiclds a higher marginal likelihood than II, does not necossarily follow that II produces better predictions of future data than II even when the training and testing data are i.i.d. from the same distribution.
These limitations notwithstanding, one might still try to use marginal likelihood to assess posterior approxi- mation. Though we have not explicitly constructed examples, we anticipate similar phenomena as reported above; namely, we expect it is possible for one posterior approximation to achieve higher marginal likelihood than another while providing a worse approximation phenomena occur for reasons analogous to the behavior we saw for TLL; what marginal likelihood measures is not directly related to posterior approximation quality.
the exact posterior. And we would expect these
19

---

Published in Transactions on Machine Learning Research (01/2024)
C.2 The logarithm of the joint predictive density
Lofti et al. (2022) consider the logarithm of the joint predictive densily, log i (v],, DD,, as alternative metric for assessment of predictive quality, Analogous to Equation (1),
(Vi..... VN- VN- - [ = 1o.......vv-10)000 (0)
(18)
If the training data are strictly independent from the test data under the specified model II (not just conditionally independent given latent parameter), then the log joint predictive density will equal the TLL but such would generally be uninteresting in Bayesian modeling. In general, the log joint predictive density need not equal the TLL.
We have already seen that the TLL is a Monte Carlo estimate of the elpd; that is, the TEL is an estimate of "how well is a model expected to predict single held-out observation, on average?" The number of samples in that estimate (i.c., the number of test data points) will dictate the Monte Carlo noise of that estimate. perspectives
There are analogously (at least) predictive measures "how well does a model predict specific collection of held-out data?" A second is that the log joint predictive is estimate (from just single noisy Monte Carlo draw and therefore very high in variance) of "how well is a model expected to predict the next N held-out observations, on average?" The TLL and log joint predictive can be seen as two extremes of a spectrum, as we describe next. Suppose one could divide a test set of size N* evenly into mini-batches of test data. Then one could make a Monte Carlo estimate with M samples of "how well is a model expected to predict the next N /M/M held-out observations, on average?" As M increases, the Monte Carlo noise decreases and the size of the held-out set of interest decreases.
the log joint predictive density. One is that the log joint
Depending the application, any of these questions may be of interest to a practitioner. While the object of study in this paper has been the TLL, the arguments in Section 3.4 suggest that the log joint predictive density (or any other point on the spectrum above) would exhibit a similar issue to the one described this paper. After all, just like TLL, log joint predictive densily does not directly track discrepancies between distributions of latent model parameters.
20


Reviews:
[{"review_id": "T0fUJUL337", "reviewer": "Reviewer_MA33", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\nThe paper is overall very clear and organized. I vote for the acceptance. \n\n## Weakness\n\nExplanation of some parts would be lacking.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "By giving examples with experiments, they refuted the following statement: \n\n* In Section 3, “The higher the test log-likelihood, the more accurately an approximate inference algorithm recovers the Bayesian posterior distribution of latent model parameters.” \n* In Section 4, “The higher the test log-likelihood, the better the predictive performance on held-out data according to other measurements, like root mean squared error.” \n\nIn the last section, the author suggested that it is important understand that different analysis requires different goals with several suggestions:  \n* If the goal is approximating posterior, don’t use TTL. \n* Change scoring rule depending on goals.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "### Sections 1, and 2  \n \nClearly written \n\n#### Section 3 \n\nThis section is still clear. However, readers would appreciate more high-level explanations for this phenomenon, not necessarily specific to each instance. For example, in Section 3.3, I have the following natural questions.\n\n* \"I agree that the phenomenon (High TTL does not necessarily indicate a better posterior approximation) is observed in Section 4. However, it is better to explain more why it occurs from more broader viewpoints. According to the Bernstein–von Mises theorem, as the sample size approaches infinity and the model is correctly specified, the posterior should converge to the true parameter. Therefore, when the sample size is sufficiently large, I guess this phenomenon should not occur? So, is it due to an insufficient sample size?\n\n* I mentioned the Bernstein–von Mises theorem. However, the assumptions of this theorem are unmet in complex models like in the case of Bayesian deep neural networks, although I believe they are satisfied in Section 3. This could also contribute to the observed phenomenon.\n\n* In this context, the author evaluates the difference between the exact posterior and the approximated posterior using the Wasserstein distance. However, what if we employ alternative metrics, such as the KL distance? Perhaps this is also linked to the underlying cause of the observed phenomenon? \n\nSummary: Primarily, I aim to convey that this phenomenon would arise from several intertwined, overarching factors. Readers would benefit from a clearer explanation of the specific causes behind this phenomenon. Could you succinctly incorporate these reasons?\n\n### In Section 5\n\nI concur with the primary assertion that TTL may not be appropriate depending on the objectives of our analysis. However, the subsequent recommendations in the paper are rather ambiguous. It remains unclear how these suggestions can be implemented in practical terms. Readers would appreciate more explicit and actionable recommendations. Here are some specific comments.\n\n* The author argues that if our analysis aims to approximate the posterior, TTL might not be suitable. In that case, what types of experiments should we report? Should we calculate the Wasserstein distance, as the author did in toy experiments with correctly-specified models?\n\n* The author suggests that depending on their goals, it might be better to use a different scoring rule than KL. However, I believe analysts often lack a clear intuition about which divergence measure to use, which is why people typically resort to KL divergence. In such cases, what should analysts do to compare methods?\n\n   *  I believe another practical and natural suggestion we can offer is to include results with various divergence measures as a robust sanity check.\n\n* As the author pointed out, in cases of model misspecification (which is common in the real world), TTL becomes impractical. Similarly, when dealing with complex data, calculating the ground truth posterior is impossible. What approach should analysts take to compare methods in such situations?\n\n   * In general, there may be limited options. However, in specific domains like images, if our objective is to obtain samples of \"natural images,\" we can employ scoring functions that quantify the naturalness of images. Many variations of such functions are commonly utilized as evaluation metrics for natural images. It might be worthwhile to highlight such cases where objectives are well-defined.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mfGKuWLOox", "reviewer": "Reviewer_m596", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "It is worth starting out to say that Bayesian orthodoxy would focus on the marginal likelihood $p(D) = \\int p(D|\\theta) p(\\theta) d\\theta$ as a way of comparing models. There are reasons why one might consider the ELPD instead; see e.g. secs 3.8.4-3.8.6 in Murphy PML2 (2023) for discussion on this and related topics.\n\nCertainly TLL makes sense wrt the standard supervised learning setup.  However, in the Bayesian view, we should actually consider $p(\\mathbf{y}^*|{\\cal D}) = \\int p(\\theta|{\\cal D}) [ \\prod_{i=1}^{N^*} p(y^*_i|\\theta)] d \\theta$, where $\\mathbf{y}^*$ is the vector of $N^*$ test cases. I.e. in the Bayesian view the test cases are *conditionally* independent given $\\theta$, rather than being fully independent.\n\nThe paper asks \"Are you using the test log-likelihood correctly?\".  Given the above, one answer is no, because one should be looking at $1/N^* \\log p( \\mathbf{y}^*| {\\cal D})$, and not $\\sum_{i=1}^{N^*} \\log p (y^*_i|{\\cal D})$, which is considered here.\n\nBeyond this point, I do not find that the examples argue against using the test log likelihood, especially the correct version $L(\\mathbf{y}^*) = 1/N^* \\log p(\\mathbf{y}^*|{\\cal D})$.  To me, what the paper is really about is the initially surprising result that in the examples of sec 3.1 and 3.3, it is the case that using an approximate posterior can give better TLL than using the exact posterior. This can be made sense of more thorough investigation -- see my comments below.\n\n### Example 3.1. \n\nThe data (N=100) is generated from a heteroscedastic model, as in eq 2. The mis-specified model is linear regression with unknown offset and slope (theta vector), and a fixed variance of 1.  For this simple setup, it is possible to compute exact Bayesian inference for the theta vector, and also a \"mean field\" variational approximation. Fig 1 panels C-F show \"variational approximations with re-scaled marginal variances\", but absolutely no details are given of this rescaling. All these experimental details and derivations (including exact and VI) should be provided in an appendix, and code also provided. Details should also be given (in the appendix) for the computation of the 2-Wasserstein distance (it is a closed form for Gaussians, I believe).\n\nThe key observation from this experiment is that the test log likelihood (on a sample of size $N^*=10^4$) is higher for panels C-F, where presumably the posterior variance has been inflated.  So for this mis-specified model, one can obtain higher TLL for worse posterior predictions.\n\nFig 1 (left) shows why this result is not unexpected -- inflating the variance of the posterior \"wobbles\" the theta_1 parameter (coeff of x) more, and this translates to error bars that increase with |x|, as in panel F. This mimics (to some extent) the effect of the heteroscedastic variance. But it would be very interesting to see how this works for $L(\\mathbf{y}^*)$, as the plots show the marginal erorr bars for one test point, while this quantity requires all $N^*$ datapoints *to be generated by the same $\\theta$.*\n\nI would also like to see the marginal likelihood (or ELBO for VI) for each model A-F, and the true test log likelihood $L(\\mathbf{y}^*)$.  Also it is readily possible to compute the true log-likelihood under the model of eq 2, this should be given when comparing to the marginal likelihood/ELBOs of the mis-specified models. Please also add a panel to Figure 1 (left) showing the true data generator and error bars -- this will be useful to compare with panel A-F.\n\n### Sec 3.2. \n\nThis is a very similar example to sec 3.1, with a mis-specified linear model compared to an exact data generator, here incorporating a quadratic term (eq 4). Similar patterns are observed as in the previous example, with higher TLL for the inexact posteriors.\n\nThis example is presented as being \"in the wild\". However it is again on a toy example (where one can easily see how posterior variance inflation will help explain the unmodelled quadratic component).  The \"in the wild\" claim presumably relates to the use of SWAG (Maddox et al, 2019) which is claimed to be an off-the-shelf approximate inference algorithm.  However, SWAG essentially fits a (low-rank) Gaussian to the trajectory of a SGD optimization method (with learning rates set in a particular way). The main issue is that there are no guarantees that the method produces a correct estimate of the posterior distribution. Indeed the variance inflation in Fig 3 (right) is not at all unexpected given the higher learning rates used (p 6 para 1).\n\nI believe one could cover the use of SWAG by commenting at the end of sec 3.1 that SWAG with increasing learning rates mimics the effect of variance inflation.\n\nIn my view, this example 3.2 thus does not add significantly to that of sec 3.1.\n\n### Sec 3.3. Well-specified models.\n\nIn this example, exact posterior analysis is carried out for a small dataset size for a parameter value $\\theta_{*} = [-2,-1]$ which is unlikely under the heavily correlated prior of eq 6, as illustrated in Fig 4A. Again here variance inflation can lead to a higher TLL.\n\nThis is initially surprising, in that doing exact Bayesian inference in a well-specified model is \"doing the right thing\". So how is it possible to do better, and that $\\Delta(p,q|D) = elpd(p|D) - elpd(q|D)$ is not $\\ge 0$?\n\nMy analysis explaining this is given below.\n\n\nFor a given datset ${\\cal D}$ and testpoint $x_*$ we have\n\n\\$p(y_*|x_*, {\\cal D}) = \\int p(y_*|x_*,\\theta) p(\\theta|{\\cal D}) d\\theta,\\ $\n\n\\$q(y_*|x_*, {\\cal D}) = \\int p(y_*|x_*,\\theta) q(\\theta|{\\cal D}) d\\theta.\\ $\n\nHence the elpd at location $x_*$ for the exact and approximate\nmethods is given as\n\n$ \\mathrm{elpd}(p,x_*|{\\cal D}) =  \\int \\log p(y_*|x_*, {\\cal D}) p(y_*|x_*,\\theta_*) dy_*$ , \n\n$ \\mathrm{elpd}(q,x_*|{\\cal D}) =  \\int \\log q(y_*|x_*, {\\cal D}) p(y_*|x_*,\\theta_*) dy_*$ .\n\nNotice that the integration here is with respect to\n$p(y_*|x_*,\\theta_*)$ and not $p(y_*|x_*,D)$. This is critical for\nunderstanding what is going on. We have the graphical model\n$y_* \\leftarrow \\theta_* \\rightarrow {\\cal D}$, so that ${\\cal D}$ depends on\n$\\theta_*$, but is not a function of it.\n\nWe now integrate the expressions for $\\mathrm{elpd}(p,x_*|{\\cal D})$ and $\\mathrm{elpd}(q,x_*|{\\cal D})$\nover $p(x_*)$  to obtain\n\n$\\mathrm{elpd}(p|{\\cal D}) = \\int \\mathrm{elpd}(p,x_*|{\\cal D}) p(x_*)\ndx_* $ ,\n\n$\\mathrm{elpd}(q|{\\cal D}) = \\int \\mathrm{elpd}(q,x_*|{\\cal D}) p(x_*) dx_* $. \n\n\nFinally we consider the difference of these two quantities:\n\n$\\Delta(p,q|{\\cal D}) =   \\mathrm{elpd}(p|{\\cal D}) - \\mathrm{elpd}(q|{\\cal D}) = \\int \\int \\log \\frac{p(y_*|x_*,\n    {\\cal D})}{q(y_*|x_*, {\\cal D})} p(y_*|x_*,\\theta_*) p(x_*) dx_* dy_*$ .\n\nIf $p(y_*|x_*,\\theta_*)$ and $p(y_*|x_*,{\\cal D})$ were the same thing, this\nwould give rise to a KL divergence term averaged over $p(x_*)$, and we would\nobtain non-negativity for the elpd difference. In general this does\nnot hold. However, if $p(\\theta|{\\cal D})$ converges to $\\delta(\\theta -\n\\theta_*)$ for large training sample sizes (as would normally be\nexpected), then the KL result will hold.\n\nIf you use the above analysis in this or a subsequent paper, I request that you make sure to acknowledge the anonymous referee who provided it, and not represent it as your own work.\n\nGiven this analysis, it is of interest to know how often we will find $\\Delta(p,q|D)$ to be negative, For a fixed $\\theta_*$ this can be addressed by repeatedly sampling datasets. It is also important to know how this depends on $\\theta_*$; my intuition is that the more probable $\\theta_*$'s under the prior will be less likely to give rise to negative differences.\n\nThe analysis in section 3.3 is very incomplete, and needs to be expanded to address the issue of *why* the TLL differences occur. It may be good to keep the same test set for all cases, and look at differences in $elpd(p,x_*|D) - elpd(q,x_*|D)$ on a case-by-case basis.\n\n\n### Sec 4, Claim 2.\n\nClaim 2 seems to me to be much less interesting than Claim 1. Already on page 1 the authors mention some examples highlighting discrepancies between test log-likelihood and other analysis objectives.\n\nThe GP example is annoying, in that to make things come out well for the incorrect model (panel B in Figure 5), the authors do two things: (i) sample test data in the interval [5,10], which is disjoint from the training data, and (ii) handicap model A by setting its noise variance too high. But in general in machine learning, if the training and test distributions are different (issue (i)), all bets are off.\n\nThe other point here is that if we wish to optimize for (say) RMSE, it is natural to create an objective (e.g.  based on leave-one-out CV) that evaluates/estimates RMSE, and optimize that. For example with GPs it is efficient to compute LOO predictions (see e.g.  Rasmussen and Williams, 2006, sec 5.4.2) and one could optimize this wrt kernel parameters.\n\n### Other points\n\nThe title is opaque and inadequate -- it is essential that it at least mentions Bayesian somewhere. The real issues, as discussed above, is that the examples show that approximate Bayesian methods can sometimes give rise to better predictive performance than exact Bayesian methods.\n\nThe paper lacks experimental details and derivations; these should be provided in appendices, and code also provided.\n\nIn Figs 1 and 4 I find no use for the two numbers above each plot -- these can be obtained from the plots on the RHS. The same could hold for Fig 3 if another plot were made like Fig 1 (right).\n\nFig 1 caption. The explanation of labels A-F should occur in the caption. A figure+caption must always be self-contained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses two claims about Bayesian modelling:\n\nClaim 1: The higher the test log-likelihood, the more accurately an approximate inference algorithm recovers the Bayesian posterior distribution of latent model parameters\n\nClaim 2: The higher the test log-likelihood, the better the predictive performance on held-out data according to other measurements, like root mean squared error.\n\nThe way that these claims are addressed is primarily via examples.\n\nI find claim 1 most interesting, as will be discussed below. Secs 3.1 and 3.2 consider mis-specified models, while sec 3.3 considers a well-specified model.  For claim 2, in the era of end-to-end learning, it seems clear that if one wants to optimize RMSE, one should create an objective (e.g.  based on leave-one-out CV) that evaluates/estimates RMSE, and optimize that. There is no particular reason why TLL should be well-correlated with the alternative objective.\n\nThe paper lacks experimental details and derivations; these should be provided in appendices, and code also provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Fundamentally I believe the paper is asking the wrong questions.  It is using an incorrect version of the TLL for the Bayesian case, where it should be using $L(\\mathbf{y}^*)$.  The paper needs to be redone to consider $L(\\mathbf{y}^*)$ and not the TLL as defined in eq (1).\n\nThe examples in secs 3.1 and 3.3 are quite interesting, and could potentially make an interesting paper about some seemingly counter-intutive aspects of Bayesian analysis, where an approximate posterior gives rise to better TLL than the exact posterior. But this will require an entire re-write of the paper. I do not believe that this can be properly completed within the 2 weeks timeframe of TMLR. Thus I believe the correct decision on this paper will be reject, with leave to resubmit.\n\nFor detailed comments on sec 3.1, please see above.\n\nThe point made in sec 3.3 where an incorrect posterior can give rise to higher TLL is of also interest, but it needs much more exploration to really explain what is going on here. I believe my analysis is very germane to this.\n\nFor the reasons given above, I do not find sec 3.2 and sec 4 to be of sufficient interest and recommend that they be cut.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "dAfQVFI639", "reviewer": "Reviewer_bq8h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The message is very important. Test log-likelihood is indeed often used as primary or even sole measure of approximation quality, and it is not sufficient.\n2. Even though the basic observations are largely known within the community, we do not have clear references that could be used to justify the choices of the metrics. This paper could serve as one.\n3. The examples where approximation width is artificially changed are innovative and clear.\n\n**Weaknesses**\n1. The paper fails to provide very concrete recommendations or suggestions as replacement (or for complementing) TLL as the evaluation metric, limiting the impact.\n2. The second claim feels a bit like straw man; I do not believe people misuse test log-likelihood in that sense.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors raise awareness of issues in using test log-likelihood as metric for evaluating inference techniques, especially from the perspective of using it for selecting which posterior approximations to use. They point out that a worse approximation can result in better test log-likelihood, and that a solution with better test log-likelihood can still have worse predictive accuracy in terms of another metric.\n\nThe main message is to warn practitioners about these aspects, in form of concrete examples that make it clear how test log-likelihood for even extremely poor posterior approximations can be the best one, especially for misspecified models.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "There is clear need for a paper that discusses the topics, since it indeed has become common to access various posterior approximation techniques in terms of TLL in large-scale problems where direct evaluation of the approximation quality is not possible. In other words, I agree that various authors implicitly rely on the first claim as basis for their empirical evaluation. Even though the observations made here are not strictly new and most researchers working with the methods are likely aware that TTL is merely a proxy and not a direct measure of the approximation quality, it is highly useful to have a paper that can be conveniently cited e.g. to justify why measuring directly the approximation quality instead of \"the standard TLL that others use\".\n\nHowever, I am not so sure whether the second claim is really being used. I have always thought that people report TLL and RMSE separately as the most common metrics exactly because they are well aware you cannot make conclusions across the two metrics. The experimental results in Section 4 are valuable in explicating the discrepancy, but I do not think there is convincing case of calling that particular misuse a common scenario. In light of this, I would like to see the statements in Introduction to be reformatted -- I do not think the paper would be any weaker even if the second claim was characterised from the perspective of \"we additionally show how ... as a reminder that different test accuracy metrics do not necessarily align\", instead of claiming that particular misunderstanding to be common. As a further evidence of Claim 1 being more believable: In Section 3 (2nd paragraph) you give a long list of references to papers that rely on this implicit claim, whereas in Section 4 you do not have any examples like this. Either you need to find those references, or should fine-tune the narrative.\n\nI like the synthetic examples as they explain very clearly how the effects are seen throughout very broad range of approximation widths; e.g. Figures 3 and 4 shows that we can indeed have approximation widths that are off by an order of magnitude and still get nice TLL. This is, in my opinion, better approach than simply showing differences in ordering for some real approximations computed with different methods (that would have fairly similar approximations and TLLs).\n\nEven though I see value in the paper, I was a bit disappointed on the depth of the discussion and lack of concrete recommendations. I understand that you are not willing to directly say \"don't use TLL to compare approximations\", but the paper would still be stronger if there was an actionable recommendation for the authors who are working on approximate inference methods in context of large problems where explicit comparison against true posterior is not feasible. For instance, you could recommend some sort of template sentences to use when explaining limitations of TLL, as well as alternative metrics people should routinely report. What would be your best recommendation for a metric we could use to compare e.g. Bayesian NN posteriors, given that TLL is not a valid measure? Or would you still say that in absence of better measures we should keep on reporting TLL but explain more transparently why it could lead to (potentially severe) mis-ranking of alternative approximations. It would be very useful to have a clear statement on this.\n\n\n**Other remarks:**\n- A bit more discussion on epld and Dawid-Sebastiani scores would be useful (e.g. definition of the latter). Section 2.1 now kind of suggests epld would be a more useful quantity to estimate directly and it would be better to discuss why people are not doing this but are using TTL instead. If we were (able to) estimate epld directly, you would get rid of the implicit assumption of TTL being a close approximation to epld at the end of Section 2.1, so it would be useful to explain for the readers the challenges in that.\n\n- You occasionally talk about \"model comparison\" (e.g. end of Section 2.1, beginning of Section 3.1), in places where I believe \"approximation comparison\" would be more appropriate. I think the misuse happens more commonly in cases where people compare different inference methods, not different models. This is what most of the examples in the beginning of Section 3 are doing.\n\n- Formatting: You heavily use footnotes to explain rather complex aspects. At least footnotes 1 and 7 are quite detailed and important parts, so I would integrate them into the main body.\n\n- Would be good to discuss the choice of 2-Wasserstein as the sole measure of approximation quality. At least give a definition and explain why you used it, since it is not really (yet?) a standard metric in the field. Would your results look the same if you used e.g. (symmetric) KL-divergence to measure the distance to the true posterior? Is 2-Wasserstein in some sense ideal measure here, or does it have limitations/weaknesses as well?", "broader_impact_concerns": "None. The paper makes important remarks that should help the community be more transparent in evaluating scientific contributions and hence has only positive ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper studies the use of test log-likelihood to assess Bayesian inference algorithms or to compare forecast accuracy of different models. It provides examples and theory that well demonstrate what test log-likelihood does and does not measure.

While not proposing new methods, the reviewers believe that the paper will be useful for parts of the community either as a learning tool or as useful reference.

The reviewers have raised a number of concerns during the review cycle. The majority of them are satisfied that the revision has addressed the concerns. Moreover, the concerns raised by reviewer m596 are discussed, while not as fully as one could potentially do, at least sufficiently in Appendix C.
Decision: Accept (poster)