The Implicit Delta Method
Authors: Nathan Kallus
James McInerney
Venue: NeurIPS 2022 Accept (2022)
Keywords: Uncertainty quantification
Wald confidence intervals
maximum likelihood inference

Abstract: Epistemic uncertainty quantification is a crucial part of drawing credible conclusions from predictive models, whether concerned about the prediction at a given point or any downstream evaluation that uses the model as input. When the predictive model is simple and its evaluation differentiable, this task is solved by the delta method, where we propagate the asymptotically-normal uncertainty in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies include the bootstrap, but it can be computationally infeasible when training the model even once is costly. In this paper, we propose an alternative, the implicit delta method, which works by infinitesimally regularizing the training loss of the predictive model to automatically assess downstream uncertainty. We show that the change in the evaluation due to regularization is consistent for the asymptotic variance of the evaluation estimator, even when the infinitesimal change is approximated by a finite difference. This provides both a reliable quantification of uncertainty in terms of standard errors as well as permits the construction of calibrated confidence intervals. We discuss connections to other approaches to uncertainty quantification, both Bayesian and frequentist, and demonstrate our approach empirically.

The Implicit Delta Method
Nathan Kallus" Cornell University Netflix Research kallus@cornell.en
James McInerney' Netflix Research
jmcinerneeydnetliix.comm
Abstract
Epistemic uncertainty quantification is crucial part of drawing credible con- clusions from predictive models, whether concerned about the prediction at a given point any downstream evaluation that uses the model input. When the predictive model is simple and its evaluation differentiable, this task solved by the delta method, where propagate the asymptotically-none in the predictive model through the evaluation to compute standard errors and Wald confidence intervals. However, this becomes difficult when the model and/or evaluation becomes more complex. Remedies include the bootstrap. but it can be computationally infeasible when training the model even once costly. In this paper, we propose an alternative, the implicit delta method, which works by in- finitesimally regularizing the training loss of the predictive model automatically assess downstream uncertainty. We show that the change in the evaluation due to regularization consistent for the asymptotic variance of the evaluation estimator, even when the infinitesimal change is approximated by finite difference. This provides both reliable quantification of uncertainty in terms of standard errors as well as permits the construction of calibrated confidence intervals. We discuss connections other approaches to uncertainty quantification, both Bayesian and frequentist, and demonstrate our approach empirically.
uncertainty
Introduction
In this paper, we consider quantifying uncertainty in evaluations of predictive models trained on data. Consider the following examples. We fit complex model (such as a neural net) to predict mean service time for an incoming call to call center given some features, and we use it to prioritize calls in queuing system. We may be interested in confidence intervals the average wait time of incoming calls in the queue. Such confidence intervals would be crucial for drawing credible conclusions about such evaluations, since know cannot take the point prediction face value given the sampling uncertainty in the data. We may, alternatively, be fitting ranking algorithm by predicting user interaction from user-item features and then applying some fixed business rules on top, and we want to assess how often certain item categories would end up at the top. Of course, we would want to understand how certain we are this assessment. Or. we fit complex model to predict mean demand given price and user features from a price experiment we ran, and we use it target discounts by optimizing demand confidence intervals on the average profit over given distribution of features.
price times unit profit. We may be interested in
All of these examples have three important features: they involve (1) computationally burdensome step of fitting a large-scale model, (2) evaluating the result using a complicated function that need not even be known explicitly, and (3) requiring the epistemic uncertainty of the evaluation given a model and finite data set, in contrast to the total uncertainty comprising both epistemic and irreducible aleatoric uncertainty M Were the first thereof), we could just use the classic delta method I0O (see next section for detail). However, when these aspects are complex and the model involves many parameters, it may be prohibitive to
of these simple (simple model and simple function
"qqual contribution, alphabetical order.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Implicit Delta Method
(((,(1))) v().))
x1.96
log [(2+0)+ Am10)
Σ log (Z::0)
Figure I: Illustration of how the implicit delta method (IDM) estimates 95% confidence intervals for a statistic of interest ()))). Both the original MLE objective and the vé-regularized objective are optimized, the vi-evaluations of the two fitted models are compared, and 1.96 multiples of the square root of the difference in evaluations is added and subtracted from the nominal evaluation to create a 95% confidence interval.
either analytically derive the whole inverse Fisher information matrix in the many model parameters or compute and invert the Hessian of the training loss empirically as well as compute the gradient of the final evaluation as a function of all parameters [8 Even one aspect being complex may pose serious challenge (e.g.. uncertainty quantification for the prediction of a complex model at point). remedy may be to bootstrap the whole process from data to final evaluation, but that can prove very computationally burdensome [12]. Usually just fitting the model once is already an expensive task; fitting it hundreds of times can be operationally infeasible. Other remedies, in the case of neural nets, may be the use of Langevin dynamics [3 or random dropout [S]. But these assess uncertainty in network weights and/or network predictions, which must then be translated to uncertainty in the final evaluation.
In this paper, we propose direct yet inexpensive way to generically assess uncertainty in such settings. Specifically, we consider conducting inference when the estimator is some specified function of (conditional) maximum likelihood estimator (MLE), such as regression or classification model. Our proposal. the implicit delta method, works by simply adding an infinitesimal regularization the MLE objective (e.g., the sum of squared errors). We prove that the infinitesimal change in the final estimator due to this regularization consistent for asymptotic variance, the same variance that would have been predicted by the delta method in theory. Hence, the name of our method: we are conducting a delta-method quantification of uncertainty implicitly without explicitly propagating the uncertainty through the derivative of the evaluation function, analytically deriving the possibly-huge Fisher information matrix, or approximating it empirically. We prove that even when we approximate the infinitesimal change with finite difference with constant width, the change measure is still consistent for the asymptotic variance. This not only gives an assessment of uncertainty in terms of standard errors, it also permits us construct calibrated confidence intervals. We demonstrate this in experiments, comparing to other popular approaches for uncertainty quantification, both Bayesian and frequentist.
2 Problem Set Up and the Delta Method
We consider an estimate constructed in two steps: first we fit a model using maximum likelihood estimation (MLE) and then apply some function to it. Namely, we consider data given by n independent and identically distributed (iid) observations Z, E Z, = 1,..., drawn from a population with density [=; 0g) with respect to some base measure μ on Z. For example, the data may consist of observations of features X and responses Y, with Z = (X, Y).
In the first step, fit model to the data by MLE. Given model {f( 0) A Q] of densities (with respect to μ1) parametrized by 0 E O C CRd we set
Model fitting: O, argmmax
log f(Z::0).
BEB
prominent example is generalized regression, where we observe features and responses Z (X, Y). have parametrized predictor he(x) E RR, and parametric model g 1)) with 1 e R., We then set

---

f(z, v); ho Examples include least-squares regression, binary classification with cross-entropy loss, and Poisson regression, all with possibly complex and nonlinear predictors (e.g..
neural nets).
In the second step, process the trained model in some way come up with our estimate. Given some vi ( + E, we compute
Evaluation using fitted model: &a = (b())))
One example in the case of generalized regression is evaluation of the predictor at a specified point, (0() ge(zo). Another example is, when ge (2x p) corresponds to predicted mean de- mand at price p given features z, we may be interested the average optimal profit, x0 = go(z;, PD - c)). for an evaluation dataset {ry : I,.... m). More generally, & could be more opaque: it could involve, for example, simulating a queuing system with a controller parametrized by 0, such as priority policy with priority score go (z))
m 1
We are interested in conducting uncertainty quantification for ei. and in particular in inference on its population limit, v = (0n). One way to this inference is to propagate through v the uncertainty within on about 8g provided we understand the latter uncertainty. This is the so-called delta method.
To apply it, we must first understand the uncertainty On- Provided some regularity holds, this uncertainty can be characterized by the curvature of the objective function P((0, -) at 0 = 0o: if the curvature is sharp (resp.. flat) then perturbing the objective and minimizing (,,,((,)) instead does not (resp., does) move the minimizer far away. This curvature is exactly the Fisher information matrix:
(0) = - (V2 log f(x;0)) (x;
Specifically, under appropriate regularity conditions.
vn((0, - 0)) NO (8)))
(2)
In the above, (Mr, Σ) refers to the multivariate normal distribution and ~ refers to convergence in distribution. There variety of specific technical conditions that establish this result. For an abstract presentation see theorems 9.27 and 9.28 in Wasserman [B For more rigorous treatments see theorem 13.2 of Wooldridge [. theorem 3.3 of Newey and McFadden 227. theorem 5.1 of Lehmann and Casella A or theorem 8.3 of Davidson and MacKinnon slightly different technical regularity conditions.
7 each of which uses
Given Eq. 2 holds with I(f))= > 0, the delta method would then guarantee that (see theorem 5.15 in 33
Vn v/) (O0 Vo), Vo (.b (0)) provided (o)( exists and Vo 0.
(0)
An immediate and very important consequence of this that we can construct calibrated confidence intervals for thu under Eq.
if nVV, +pp Vo. then P(1bo E [ + -- 0- (1)2))/)1 - 3 vB E (0,
(4)
where O refers to the cumulative distribution function of the standard normal distribution and 7pp refers to convergence in probability. For example, as long I(6) and (i)(O) are continuous at dg. we can use
y DeltaMethod
(v( 1- 7
(5)
As discussed in the introduction, however, this approach may prove intractable in practice, especially when 0 has many dimensions. Since in θ,. it may seem unnecessary and overly cumbersome to first compute the uncertainty in the latter and then propagate it. We next present our method, which does this all implicitly. never working directly with the vector 0 except perturbation thereof.
only truly concerned with the uncertainty in ehn and not
optimization variable in maximizing the MLE objective and
i NNte since we are not interested in the distribution we here use only conditional density of X. Setting f((x, y); 0) = g(y: he (x)){(x) using the true unknown density fxx of X does not change the MLE nor any of the results compared to omitting f(z) altogether as we do here, which is referred to as the conditional MLE by Wooldridge 4I]
3

---

3 The Implicit Delta Method
We would like to construct calibrated confidence intervals in Eq. +. but computing the estimated standard error as in Eq. S can be prohibitive. The IDM is a way to compute the estimated standard error while neither explicitly computing the uncertainty in 6n nor propagating this uncertainty through v Instead, will simply slightly perturb the original MLE in Eq. ① using a little bit of regularization, which will implicitly do both of these difficult tasks for us.
To define the IDM, we first define regularized version of the MLE. Given any À 2 O, we consider adding the regularizer (0) to Eq. ① as well as the corresponding final estimator after passing through 10:
(A: E arginax
log +2,;;)+++vp -n())= - v(n((A;
IEe
We refer to this as virregularized MLE.
We then define the infinitesimal IDM (IIDM) as the infinitesimal change (i.e., derivative) in our final estimate using d-rregularized MLE as we infinitesimaaly increase from 0:
I-IDM
i i (2) ax
lim 2-0
1 (da(A)) -..)
A==
Our first result shows that the IIDM estimate consistent for the true asymptotic variance in Eq. (
Theorem 1. Suppose that 0r ->p fo E Interior(O)), T(0g) > O, and that, in a neighborhood of Oo. (0) is continuously differentiable and [ 0) is almost surely twice continuously differentiable in O with a Hessian that is bounded in operator norm by an integrable function of Z. Then
n IVIDDM
7p Vo.
The significance of Theorem I is that, per Eq. (4. it implies that (.. = +b (11+ VIDM
a calibrated A-confidence interval for 4/0-
Note that, aside from conditions on vé (which are the same as needed for Eqs. 3 and 5 to work), the regularity conditions required in Theorem [ are implied by the regularity conditions required for establishing Eq. (2) by, for example, any of Davidson and MacKinnon 77. Lehmann and Casella [24, Newey and McFadden [27]], Wooldridge 41. In that sense, these conditions are not strong as they are already needed for D'DeltaMethod to be a good estimate of uncertainty to begin with, and they fit into the existing framework for the asymptotic analysis of MLE.
The implication of Theorem Tis is that we may be able implicitly complete the steps of the delta method (compute the uncertainty in Ôn, then propagate it through v/) by simply assessing the impact of regularizing the MLE. However, this requires regularization coefficient. While this requires computing just one first derivative (rather than many first and second derívatives in Eq. (5)) it is still not clear how to do this in practice.
actually differentiate with respect the
In practice, we might approximate this derivative using finite differences, i.e., replace the limit in Eq. 7 with very small X. This gives rise to what we call the diffe-difference IDM (FDIDM), defined as follows for a given An > 0:
1 (Wi()n) En))
PDIDM
(8)
Ay
Our next result shows that in fact suffices to choose An constant. In fact any choice of X, growing strictly slower than n. yields that n / FDDIMM is also consistent for just like VIIDM provided just slightly more regularity holds.
Theorem 2. Fix any An o(n). Suppose that addition the assumptions of Theorem neighborhood of lu sp(0) is thrice continuously differentiable and f(Z;0) 0) is almost surely thrice continuously differentiable integrable function of Z. Then
with a third-order derivative that is bounded in operator norm by an
nV D.DDDDM
-p Vo.

---

It may seem surprising that a constant X suffices or that X is even allowed to grow, but that can be seen as an artifact of the fact we did not normalize the sum over the data in Eq. (6) by 1/n. If we did normalize, it would he equivalent to rescaling À by n, so that o(n) becomes o (T), i.e., requiring a vanishing increment for the finite differencing. Nonetheless, writing Eq. (6) as we did very convenient, as it matches how one usually applies optimization algorithms such as stochastic gradient descent to training objectives, and it makes the choice of X for Eq. S very easy: just fix some constant and do not worry about the scaling with n. For example, setting An 1 suggests very
simple-looking 95%--coffidenc interval: din + 1.96 (n.(1) - 6p Note that it is not necessarily
better choose smaller X: the smaller À the closer V V.EDIDM is to V.LDMM but that need not mean it is a better estimate (see numerical illustration in Fig. B). Finally, note that Eq. (8) is but one way to make mifte-difference approximation of derivative, and other finite-difference formulae for derivatives (see ch. 4 of such as central differences could possibly be used.
Remark (Regression Using Squared Error Loss). When training regression models usually minimize over model parameters (e.g.. neural net weights) the sum over the data of squared error loss, [1r, 0) = (y- - go (z:))². This differs from the corresponding Gaussian log likelihood by a factor of 20z (and some constants that do not matter), where x² is the residual variance of Y given X. Therefore, apply IDM, all should do is simply regularize the uum-of-squared-erron minimization problem by- -2a² 22)2 ,), as that would be equivalent to dividing the log likelihood part by -20 Of course, we do not know o², but we can estimate it by o
(0).
(y 9a. (z))², that is, the minimum average sum of squared errors. Since O -tp o², as it is in fact the MLE estimate for o², the asymptotic guarantees of Theorems 1 and 2 will continue to hold after this rescaling. Note that the standard errors given correspond to the MLE formulation of caast-suares (usual standard errors) rather than the M-estimation formulation thereof (so-called robust or sandwich standard errors).
Σm
(Using IDM to Compute the Fisher Information). A by-product of the proof of Theorem is that, if we looked at the (vector-valued) derivative Wn, Ox 0n (A; ) lim x (A: - 8....
Remark 2
x== then nWW, +p I(0b) ( Vv) Therefore, if we set (0) 0,, i.e., the hth component of 0, then Wr converges to the 7th column of I(6))- Thus, by regularizing each component of 9 in turn, we obtain the whole matrix.
Nonetheless, the whole raison d'être of IDM is to avoid working directly with the parameter vector altogether, and simply propagate its uncertainty automatically via the MLE optimization problem. For example, if we consider neural net regression, IDM would never make explicit reference to the vector of weights itself, only to the trained prediction model and its prediction performance data. The above, wherein we compute the uncertainty in 0 directly, stands in contradiction to this. Nonetheless, it can be a useful observation when inference on 0 itself is for some reason of interest.
3.1 Extension to Multivariate Evaluations
We have so far focused on scalar evaluations for ease of presentation and as it covers the most important cases. We now show how our method easily extends to the multivariate case, where (0) = (O)
(0)) E K The reason may not suffice to run IDM separately for each component is that we may be interested in the covariance of the evaluations. Under the appropriate conditions, the extension of the delta method for MLE (Fq.
3h to multivariate evaluations is
Vnlu -w) ~ NOO Vo). Vo = (--
8 v (0) is the K x dd Jacobian of (()). where (0)
20,
Our extensions of IDM and FDIDM to multivariate evaluations are as follows:
Ay(() = ( (O, - w0 (B))). (V ).y = lim (A). (V.FDIDM) (nn(
X
Theorem 3. IVVIDMM Vo under the conditions of Theorem and n V FDIDM Ap Vo under the conditions of Theorem both K x K matrices.
Surprisingly, this shows one need only solve K + 1 (possibly) regularized MI.Es to get the full K x K covariance. (See Alg. 3 in supplement.)
5

---

3.2 Handling Non-differentiable Evaluations and Evaluation Uncertainty
So far we have assumed that the evaluation function is known and differentiable function. Both statements may be false when we interested in evaluating average performance on a population but we only have a finite evaluation data set and unit performance is not differentiable.
Specifically, let Wi...., Wm W denote the evaluation data set (which may be the same as the training set or otherwise dependent or may be an independent data set) and let h(w; 0) the unit evaluation function. Consider the empirical evaluation map
(0) 1 ΣØ= OW
If () is almost surely not differentiable in 8, then v is also almost surely not differentiable, which poses a challenge. We will show, however, that even though is not differentiable (which would break the usual delta method), FDIDM actually remains valid, without any changes to the method, provided certain avverage-diffrrentablll holds.
To motivate the challenge of nondifferentiability and the plausibility of pn--averggedifferentaaabl consider an example where go (x) represents an order quantity to stock in context and w (xr, d) represents features and demand. If (x( d); = max gu (x). O} then (0) quantifies average unmet demand, but h is not differentiable. Other non-differentiable examples include evaluating regression and classification models' performance using non con-diferreniablle utility functions. While may not be differentiable and hence neither v. it may still be plausible that its expectation E[h(W;;)) (0) is differentiable. For example, if the distribution of demand conditioned on features continuous, then the derivative of (W; 0)] in the example of average unmet demand will be the average of the conditional cumulative distribution function at (1) times- - -oogo (z)), and the second derivative will involve the conditional density.
We next show FDIDM still works with non-differentiable v; given some on-averagg--diferenniaabiil Theorem 4. Consider
(()). Fix A, À O constant. Suppose the assumptions of Theorem hold, that Eq. 2 holds, that h (W; 0) is almost surely L-Lipschitz in 0, and that for some M > O.
lim P(On {0: ||0 Boll S <). h(W; 8) is twice differentiable in with [[))())) s M) = -
Then
V 1FDIDMM-1/2(0 -/) ~ NO, 1).
In the above example of average unmet demand, and M would be bounds of the gradient and Hessian of ga(z) in 0, and sufficient condition for the assumption to hold would be that go (x) boundedly differentiable in x for in a neighborhood of 0o and W = D) has a continuous distribution.
Although the asymptotic variance of Sh. 2o is now different (in particular DeettaMmthod in Eq. S
may be ill-defined), Theorem shows that PFDIDM actually remains consistent for this new asymp- n totic variance. Thus, it provides consistent estimate of standard errors and it still gives calibrated confidence intervals (note tho is now random but we can still have a confidence interval for it).
In some cases we may want to directly conduct inference on the population version of the evaluation, d (E[(WW; Bu)]. To do this, all we have to do is simply also add the uncertainty due finite evaluation data set. Under standard regularity conditions, we have
(i (1/ vD) NO 1), where V m- Σイ
(b())))²
Therefore, provided the training and evaluation data independent,
(DFIIDM
-
(12 v6) ~ (0,
If not independent, then /FDIDM Vi provides consistent upper bound on the standard error.
3.3 Implementation
FDIDM is given in pseudocode in Alg. 1 Given objective function L - Σx log f(z; f), evaluation function vh, and scalar width X, FDIDM returns the estimated variance of (0,)) The first step is maximize the original objective w.r.t. 0. Usually, this task has already been solved as this is

---

Algorithm 1: Finite-difference implicit delta method (FDIDM)
Input:
Learning objective L, evaluation t scalar À
Function FDIDM (C. 1/, A)
On c arg max; [())
// optimize learning objective // optimize v-regularized objective // estimated variance of (0..)
(A) t arg max [()) (p)(0)
return (1b(0, - -bb,))
the trained predictive model. Then, maximize the ei-regularized objective w.r.t. 0. Finally, return the estimated variance using the finite-difference method evaluated at À = O. See Appendix D for the corresponding algorithm when vb is multivariate.
In practice, one can further reduce the computational cost of FDIDM due to the fact that the th- regularized objective can be made arbitrarily close to original objective by choosing À small enough, subject to numerical instability at extremely small values. Specifically, when using stochastic gradient ascent in FDIDM, once the optimum On has been found, only a small number of gradient updates may be required to also find (A).
FDIDM also admits non-gradient- graddient-based approaches. Consider the case that v is simulator that takes a fitted model and returns set of evaluations and no gradient. Then the ei-regularized objective may optimized by gradient-free methods such Nelder-Mead 266 and Bayesian optimization (14)
Alternatives for Uncertainty Quantification and Related Work
Uncertainty quantification in machine learning is a topic of major interest due to the need to make downstream inferences and decisions based on the predictions of large-scale networks trained on massive datasets in either frequentist [16
28 51
Our focus here is or Bayesian fashion B 15
on methods that can flexibly isolate epistemic uncertainty in an evaluation, representing data sampling uncertainty of that evaluation under given model. In cases where the total uncertainty for predictions desired, a broader set methods may be brought to bear, such as conformal prediction [I] 37], Platt scaling [1 [3), indeed, any of the aforementioned statistical methods used in conjunction with a term or terms for alcatoric uncertainty. An exhaustive account of the literature is outside the scope of this paper. We highlight the principal ideas and points of contact with work.
The Bootstrap The bootstrap simulates sampling from the true data generating distribution by resampling from the observed dataset (see, e.g., [12 for an introduction and [22 for theory on when it works). The key advantage is that it enables quantification for estimators. comes may comprise a model-fitting algorithm and prediction, needs to be executed many times. In the context of deep learning, many useful adaptations of the bootstrap and the related jackknife have been proposed to increase its computational efficiency [I 30 B1 Maintaining an ensemble of models as a representation of the variability of the evaluation an appealing intuition that does not restrict one to local approximations, and may be combined with local approximations where necessary.
purpose and casy-to- to--tom-mmmmeett uncertainty high computational burden because the estimator, which
The Functional Delta Method The delta method 10 is a classic approach that is widely used with small models with analytic Fisher information matrix (e.g., linear regression) and, more recently, ditteefifferntiiaiion unlocks the delta method for a larger class of models [X] The bottleneck is the need to calculate then invert the Fisher information matrix, for which there various approximations 225 32) The delta method applies to a wide range of (differentiable) estimators subject regularity conditions that ensure asymptotic normality of the parameter estimates and this constraint carries over to the implicit delta method. The functional delta method extends the delta method to evaluations of infinite-dimensional parameters (see Ch. 12 ( but usually restricted to analytically deriving influence functions in theory by differentiating the population estimand with respect to distributions and then approximating the influence function by plugging in estimates of unknown nuisances
[5] [8].
Bayesian Uncertainty Quantification Tractable methods for approximate Bayesian inference
in neural networks, such as variational inference in feed-forward nets R autoencoders 22
35
normalizing flows 3d dropout uncertainty (1),, stochastic gradient Langevin dynamics 39 and
related approaches, present impressive range of options for uncertainty quantification. In cases
7

---

data
0625
90%
12-
0320
15
03-
20
62
0.000
-10
0,0s
69
0.
-15
-13-
79
0.00
pridction
predittion
IDM (this paper)
(b) Delta Method
2.0
23-
15-
0.00
gredicion
0,02
62-
84
01
1.-
0.00
15
14
2.2
prediciion
proditioon
(c) GP Materm22
(d) True sampling uncertainty by simulation
Figure 2: Fits along with uncertainty bounds and estimated preiition-covarianne matrix for data 10
generated from (in
3
10 where ~ ~N(0 1)
where it sufficient only consider single mode the posterior, local methods can prove useful. In particular, an alternative interpretation of the delta method is as special case of the Laplace approximation to Bayesian inference, where Eq. ( arises the posterior predictive distribution for local multivariate Gaussian approximation around the maximum posteriori estimate. Several recent works have investigated the potential of the Laplace approximation as a way to avoid having to characterizing the full posterior in deep networks 8 19 20). IDM can provide another way to perform a Laplace approximation and may be orthogonally combined with the above methods.
Experiments
In this section, we evaluate finite-difference implicit delta method (FDIDM) range of tasks that require confidence intervals Our goal is to quantify the extent to which FDIDM applies in practice and how it compares to alternative methods. We start with ID synthetic data in Sec. [. where apply a neural net to recover known functions from small datasets. Then, in Sec. 5.2 we consider the task of inferring average utility under neural net trained In Sec. 5.3 we apply FDIDM to variational autoencoders and use the implicit delta perspective to understand the effect of KL deighweigh We find that the motivation and convergence properties of FDIDM are empirically observed and this may be useful to practitioners seeking to quantify the epistemic uncertainty of complex models variety of regression and classification tasks.
set of real-world benchmark datasets.
5.1 1D Synthetic Examples
We consider known quadratic and sinusoidal functions from which draw random dataset. Fig. gives the data generating stochastic function for sin wave and the resulting fits for FDIDM, the classic delta method, Gaussian process (GP) with Matern-52 kernel, as well as simulation from the true data generating function. (Appendices B and C provide the results on the quadratic function and further experimental details, respectively.) The quadratic example has evenly dispersed input data and there is close alignment between the methods. The sin wave is more challenging because it requires extrapolation also known "in-between" uncertainty in of given inputs. Results for IDM, DM, and simulation are all based on estimates using a neural net with hidden layer of 50 tanh units fl It should be noted that the GP not trying to estimate frequentist sampling variance (shown in the simulation results) but rather the Bayesian posterior uncertainty (although they can coincide asymptotically:; 38 theorem 11.5); we include largely for qualitative comparison to a popular epistemic-uncertainn quantification method. In particular, unlike
from outside the ranges
P The source code available at
wtcchuu.coom/.../ "Tii architecture is in line with [3] which also provided the basis for sin example.

---

6:75
6.175
0.19-
ambda 012020
8:100
0.255
4m 0250000
1219000
PMC SMISE vxlanee
6.255
256
125
18 05:200
0.300
4m arrode1 024020
8.300
smbda=2 049026
ambdem__020000
=mb00=8.122000
0.080
88000
0.000
5,000
50
102
130
200
250
Eog ambds)
(a) Number of data points as independent variable.
(b) A as independent variable.
Figure 3: Convergence of IDM in the quadratic task for different values of n and X. There is wide
dynamic range of acceptable
Table 1: Run time (seconds)
Vehicle
Waveform
Satellite
MNIST
IDM
39
129
111
303
Bootstrap
806
2,334
3,192
7,164
the IDM, DM, and simulation results, the GP does not yield an interval around the neural-net based mean estimate and instead has a different mean function
As expected, IDM agrees most with the delta method while the GP overestimates uncertainty, particularly for extrapolation at the outer edges. The corresponding full covariance matrix of the predictions also given in Fig. 2 All the methods recover the high-level structure of covariance for both examples, though the scale lactors differ considerably.
Fig. shows the convergence of the root mean squared error of IDM w.r.t. the true variance as determined by 50 resamples from the data generating distribution in the quadratic task. The squared errors are rescaled by n² account for decreasing scale ( of the true the variance as n grows. Shaded error bars indicate one standard error. Convergence for the standard delta method is also shown for reference. We find there is wide dynamic range of acceptable values of X. Small values of X < 0.01 perform poorly, likely due to numerical instability, but performance improves for larger A. The setting À 0.512 even outperforms the delta method. These findings support the implication of Theorem indicating that convergence holds as long as À grows sublinearly to n.
5.2 Confidence in Predicted Cost Downstream of Classification
classification tasks are fitted with a neural net with one hidden layer and 50 tanh hidden units. In this setting, we wish to calculate confidence intervals over total cost in a downstream task under predictions from the network. An arbitrary cost function set up, in this case, the average cross entropy of the observations on held-out validation dataset, though in practice we could have wide variety of cost functions relating to the task downstream of the classifier. It is challenging to form confidence interval for even this simple cost function because it is a function of predictions from the network. Under this scenario, is typical to make a bootstrapping estimate, requiring B times the cost of training the network (here, we use B 50). FDIDM is also applicable in this setting. We show both methods on MNIST image classification
X and set of UCI benchmark datascts [I]] in Fig. 4 We find that FDIDM has good coverage for fraction of the computational cost of the bootstrap estimate. Specifically, a time complexity comparison is provided in Table
5.3 Dein-tegg KL in Variational Autoencoders
The variational autoencoder (VAE) is prominent example of approximate inference in deep gen- crative models 21 (35). In practice, it has been observed that woightiig the KL term in the results in significantly better accuracy held-out variational objective by factor 7, where T >
$ 'ee also applied the arecos kernel GP which imitates neural network 6 found that the Matern-3 kernel inferred a mean that was closer in practice to the mean inferred by neural net.
Runn time measured MacBook Pro 2.3 GHz Quad-Cor Intel Core i7 with 32 GB RAM.

---

IDM Bookscrap
501 4900
zolse fraction 0.00 moder frantion 0.50
windfomm
vtoline
Figure 4: Predicted distribution of utility in clas- sification on benchmark datasets. Datasets are shown in order of number of examples.
Figure 5: Improvement in reconstruction quality over the unweighted likelihood as a function of
test data in approximate inference such as data augmentation (2 and the cold posterior effect [40]. Various explanations for the benefit of setting T > have been posited, e.g., model misspecification. Here, we briefly explore an IDM interpretation of this phenomenon.
This is closely related to other ways of reweighting the prior and likelihood terms
Observing that any objective does change its critical points under (non-zero) rescaling, it holds that down-weighting the KI. term yields the same optimization problem as up-weighting the reconstruction error by T. We consider this setting and define v to be the reconstruction quality (i.e., negative reconstruction error) while optimizing the standard (unweighted) evidence lower bound defined by the VAE. Rearranging the terms Eq. B8. letting A, T
have (a (T 1)V: FDDDM + (0)
(10)
The immediate implications of Eq. 10 for fixed T 1 of o(n) are that,
the reconstruction quality for the objective implied by variational inference upper bounded by that of the objective with up-weighted likelihood; and,
2. the higher the variance in the reconstruction quality, determined by the dataset and model, the greater the benefit of up-weighting the likelihood term variational inference.
Since (1) is already supported by existing empirical work, we focus on evaluating whether (2) also holds in practice. To do this, we artificially increase the variance of the reconstruction quality by perturbing a proportion of the dataset and compare vi for various T. Fig. shows the results for Uniform (0, 26 ) to each pixel in
applying VAE to the MNIST dataset after adding i.i.d. noise randomly chosen fraction of the images. The figure indicates that, as predicted by Eq. TC,, the gap in the reconstruction quality for 7 > relative to T = 11 increases more variance is introduced. Higher values of T do indeed result in better reconstruction quality and this advantage grows with the amount of variance. In sum, these findings from KL down- weighting may be explained as the residual between the vi-regularzedd variational objective and the objective implied by the evidence lower bound, though further studies required.
consistent with the hypothesis that the advantage
Discussion, Limitation, and Conclusions
In this paper we develop the implicit delta method for forming calibrated confidence intervals via careful regularization of the model objective. Like the delta method, the method requires certain regularity conditions (Theorem [ and for the MLE to be at a stable optimum, where perturbations around the optimum reliably capture sampling uncertainty. If this is not the case, e.g., the parameter has failed to converge or the objective itself is changing, it yields unreliable results. For this reason, IDM like the delta method and the bootstrap may be misleading for small data, and indeed uncertainty quantification with small data is fundamentally difficult. The most appealing feature of IDM is that does not require the variance of the parameters to made explicit, which also suggests future research in exploring the compatibility of nonparametric models with IDM. There is also the potential to explore IDM constrained MLE and general -essimaation.
Acknowledgments
We are grateful for the insightful comments of the anonymous reviewers and our colleagues Netflix.
10

---

References
[1] Anastasios Nikolas Angelopoulos, Stephen Bates, Michael I. Jordan, and Jitendra Malik. Un- certainty sets for image classifiers using conformal prediction. In International conference on learning representations, 2021.
[2] Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in- domain uncertainty estimation and ensembling in deep learning. In International conference on learning representations, 2020.
[3] Charles Blundell, Julien Comebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. In International conference machine learning, pages 1613-1622. PMLR, 2015.
[4] Richard L Burden, J Douglas Faires, and Annette M Burden. Numerical analysis. 2015.
[5] Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. Oxford University Press, 2018.
[6] Youngmin Cho and Lawrence Saul. Kernel methods for deep learning. Advances neural information processing systems, 22, 2009.
[7] Russell Davidson and James G MacKinnon. Estimation and inference in econometrics, volume 63. Oxford New York, 1993.
[8] Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace redux-effortless Bayesian deep learning. Advances in neural information processing systems, 34, 2021.
[9] Armen Der Kiureghian and Ove Ditlevsen. Aleatory epistemic? Does it matter? Structural safety, (2): 112, 2009.
[10] L. Doob. The limiting distributions of certain statistics. The annals of mathematical statistics, 6(3):160 169, 1935. doi: 10.12 4/20mm/1
[11] Dheeru Dua and Casey Graff. UCI machine learning repository [http://archive, ics. uci. edu/ml]. Irvine, CA: University of California. School of Information and Computer Science, 25:27, 2019.
[12] Bradley Efron and Trevor Hastie. Computer age statistical inference, volume 6. Cambridge University Press, 2021.
[13] Andrew YK Foong, Yingzhen Li, José Miguel Hernández-Lobato and Richard E Turner. "In-between' uncertainty in Bayesian neural networks. arXiv preprint arXiv: 11337, 2019,
[14] Peter Frazier. Bayesian optimization. In Recent advances in optimization and modeling of contemporary problems, pages 255 278.. Informs, 2018.
[15] Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In International conference machine learning, pages 1050-1059. PMLR, 2016.
[16] Ryan Giordano, William Stephenson, Runjing Liu, Michael I. Jordan, and Tamara Broderick. A Swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1139-1144 1147, PMLR, 2019.
[17] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On calibration of modern neural networks. In International conference on machine learning, volume 70, pages 1321-1330. PMLR, 2017.
[18] Hidehiko Ichimura and Whitney K Newey. The influence function of semiparametric estimators. Quantitative Economics, 13(1):29-61, 2022.

---

[19] Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions of Bayesian neural nets via local linearization. In International conference on artificial intelligence and statisticy, volume 130, pages 703-711. PMLR, 2021.
[20] Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate inference turns deep networks into Gaussian processes. In Advances in neural information processing systems, volume 32. Curran Associates, Inc., 2019.
[21] Diederik P Kingma and Max Welling. Auto-encoding variational Bayes. arXiv preprint 1rXiv:131..611,, 2013.
[22] Michael Kosorok. Introduction to empirical processes and semiparametric inference. Springer, 2008.
[23] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-233 1998.
[24] Erich L. Lehmann and George Casella. Theory of point estimation. Springer Science & Business Media, 2006.
[25] Jonathan Lorraine, Paul Vicol, and David Duvenaud. Optimizing millions of hyperparameters by implicit differentiation. In International conference on artificial intelligence and statistics, pages 1540-1552. PMLR, 2020.
[26] John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal, ((4):308-313, 1965.
[27] Whitney K Newey and Daniel McFadden. Large sample estimation and hypothesis testing. Handbook of econometrics, 4:2111-2244 1994.
[28] Geir K Nilsen, Antonella Z Munthe-Kann Hans J Skaug, and Morten Brun. Epistemic uncer- tainty quantification in deep learning classification by the delta method. Neural networks, 145: 164-176, 2022.
[29] Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz Khan, Anirudh Jain, Runa Eschen- hagen, Richard E Turner, and Rio Yokota. Practical deep learning with Bayesian principles. Advances in neural information processing systems, 32, 2019.
[30] Ian Osband and Benjamin Van Roy. Bootstrapped Thompson sampling and deep exploration. arXiv preprint 0rXiv:/507.003000 2015.
[31] Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped DQN. Advances in neural information processing systems, 29:4026-4034, 2016.
[32] Barak A Pearlmutter. Fast exact multiplication by the Hessian. Neural computation, 6(1): 147-160, 1994.
[33] John Platt al. Probabilistic outputs for support vector machines and comparisons regularized likelihood methods. Advances in large margin classifiers, 10(3):1177,, 1999.
[34] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[35] Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278-1286 PMLR, 2014.
[36] Yaniv Romano, Matteo Sesia, and Emmanuel Candes. Classification with valid and adap- tive coverage. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, edi- tors, Advances in Neural Information Processing Systems, volume 33, pages 3581-3591. Cur- ran Associates, Inc.. 2020. URL https:t: wwwpediinnnnnnniinneeeeeee ctt.rpppp..pp/pppee///////////// 244
pdf
[37] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world. Springer Science & Business Media, 2005.
12

---

[38] Larry Wasserman. All of statistics. Springer, 2004.
[39] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient Langevin dynamics. In Proceedings of the 28th international conference on machine learning, pages -688. Citeseer, 2011.
[40] Florian Wenzel, Kevin Roth, Bastiaan S Veeling, Jakub Swiqtkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin, How good is the Bayes posterior in deep neural networks really? In International conference machine learning, volume 119, pages 10248-10259, PMLR, 2020.
[41] Jeffrey M Wooldridge. Econometric analysis of cross section and panel data. MIT press, 2010.
[42] Guodong Zhang, Shengyang Sun, David Duvenaud, and Roger Grosse. Noisy natural gradient as variational inference. In International conference on machine learning, pages 5852-5861. PMLR, 2018.
Checklist
1. (a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] Proof of consistency in Sec. connections discussed
in Sec. f evaluation in Sec.
(b) Did you describe the limitations of your work? [Yes] See Sections 2 and
(c) Did you discuss any potential negative societal impacts of your work? [N/]] To the best of the authors' knowledge, our work does not have negative social impacts
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. (a) Did you state the full set of assumptions of all theoretical results? [Yes] See Sec. and Appendix A
(b) Did you include complete proofs of all theoretical results? [Yes] See Sec. Appendix A
and
3. (a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] See online repository https::/ htt:://ggttu..bb homt:/u..ooomm/// hat.m//aaaaeeccceeeeeeee/iii.pea
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 5 Appendix B and C, and source code
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix
4. (a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes] Source code
(d) Did you discuss whether and how consent obtained from people whose data you 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
13


Reviews:
[{"review_id": "wRMrK4ddzTD", "reviewer": "Reviewer_41Kr", "summary": "This paper focuses on estimating sampling uncertainty for typical machine learning models (M-estimators) with some evaluation function $\\psi$ (e.g. the test loss). They do so by using the delta method to estimate the asymptotic variance of $\\psi$. While this is not a new idea, the authors note that there are two downsides to the traditional use of the delta method: (1) if the model has parameters of dimension $D$, it requires the inversion of a $D \\times D$ matrix, and (2) if $\\psi$ is non-differentiable, then the delta method does not apply. The authors show that solving a regularized version of the M-estimator can get around both (1) and (2).", "questions": "- The discussion in Remark 1 seems to imply that the loss needs to be the log likelihood of some probability distribution (otherwise I'm not following why we care about $\\sigma$). Why is this? It seems like a lot of the arguments in the paper go through as long as $\\hat\\theta_n$ is any M-estimator.\n\n- In the statement of Theorem 4, $m$ has to satisfy $m \\asymp n$ or $m \\gg n$. Could this be more concisely stated as $m$ is $\\Omega(n)$?\n\n- Is Theorem 1 not just proving conditions under which $\\hat V_n^{IIDM} = \\hat V_n^{DeltaMethod}$? If true, this seems like a more precise statement of the theorem. \n\n- This is a much broader question. How do the authors think such local approximations should be used when the model objective is multimodal? Should we just be approximating around the discovered mode? Should we discover multiple local optima and then run this kind of approximation for each optimum? I don't think it's necessarily the current authors' job to solve such questions, but it could be worth commenting on in the paper if they have insights.\n\n", "limitations": "Yes, I think the authors have addressed the potential negative impact of their work.", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. Issues (1) and (2) I mentioned in the summary are really big problems with previous work in estimating uncertainty in large / complex machine learning models, and I think the authors make a pretty convincing case that they have solved these issues. I actually think the authors are underselling the \"you don't have to invert a big Hessian matrix\" thing. A number of people have tried to deal with this issue for related approximations (i.e. infinitesimal approximations involving the inverse Hessian) and been unsuccessful. Koh and Liang 2017 use a stochastic approximation to the inverse Hessian. But Stephenson et al. 2020 find it poor for their more quantitative purposes (I suspect the current authors would find it unusable for their implicit delta method as well) and propose a different approximation. But Frangella et al. 2021 then show that the approximation of Stephenson et al. 2020 is flawed and propose yet another approximation. Their analysis might be the final word in this chain, but who knows! So, it's really great to not have to deal with solving systems with large Hessians.\n2. The writing quality in the paper is great (significantly above average of the typical NeurIPS submission).\n\n**Weaknesses**\n\nI think the current empirical evaluation needs to be significantly strengthened. I have a few points:\n1. To me, the minimal thing the experimental section should show is whether or not the proposed method improves over baselines in some way. If someone asked me to point them to parts of this paper for concrete evidence why they shouldn't just always run the bootstrap or the standard delta method to estimate sampling uncertainty, I wouldn't know what parts to point them to. I would be surprised if the authors couldn't demonstrate that their IDM is faster than the bootstrap and, in high dimensions, the delta method. Or show a model on which IDM runs but the delta method does not due to lack of differentiability. But this needs to be actually shown (or, if this isn't true, then it should be investigated why a good-sounding idea doesn't work in practice).\n2. I'm not sure what to take away from Fig 3; I can't tell from this figure whether IDM is doing something good or bad. First, why should we trust the bootstrap here (how do we know both the bootstrap and IDM aren't just terrible here -- could one argue that the dimension of the model is too high compared to the amount of training data for any asymptotic estimate of variance to have kicked in)? Second, assuming we do trust the bootstrap, why does this figure say that we should trust IDM? There are differences between IDM and the bootstrap. In what sense are these large or small? It's a little hard to tell, since $\\psi$ doesn't relate to any particular downstream task. It might be more helpful if $\\psi$ were something more meaningful. Also, how fast is IDM versus the bootstrap here? If IDM takes 500 years to run, I sure would rather run the bootstrap!\n3. On the synthetic example: again, it's hard to tell if IDM is actually doing something sensible here -- is it closer (or reasonably close compared to other methods) to the true sampling uncertainty? Since you have access to ground truth here, this could be a great opportunity to give a much more quantitative assessment. \n4. Why are Gaussian processes included in the experiments? Bayesian methods don't measure sampling uncertainty, so what is being gained by comparing them to a method for approximating sampling uncertainty?\n\nOverall, I think this could be a really great paper if the evaluation section were stronger. I would definitely raise my score and push for an accept if the authors can provide more compelling experiments that carefully investigate whether or not there are practically meaningful regimes in which their method is an improvement (e.g. is much faster without much loss in accuracy).\n\n**Smaller comments**\n\n- On line 242, [13] is cited as an example of approximating the bootstrap with similar infinitestimal approximations. I wouldn't cite this paper as an example of this, as it doesn't really work for the bootstrap. You might cite Giordano et al. 2019 instead (see their  Proposition 4 about the bootstrap).\n- Line 225: I don't think you only need a small number of stochastic gradient steps; this should be true for pretty much any optimizer.\n- Eq (5): I would think of the version of Eq (5) using the empirical estimate of the delta method estimator as more standard in practice. This also fits better with the following sentence which says Eq (5) won't work well if $\\theta$ has many dimensions (if we're actually using the population version, then it won't work well in way more cases because $I$ is hard to compute!)\n- Line 135 \"it is still not clear how to do [the differentiation needed to compute the delta method] in practice\". I think this is clear -- we can use the implicit function theorem, which is exactly what the proof of Theorem 1 is doing.\n- I think the experiments in Fig 2 should be changed (see comments above), but at the very least the scale of the color bars should be the same so we can compare the different methods.\n\n\nZ. Frangella, J. A. Tropp, and M. Udell. Randomized Nystrom Preconditioning. arXiv. 2021.\n\nR. Giordano, M. I. Jordan and T. Broderick. A Higher-Order Swiss Army Infinitesimal Jackknife. arXiv. 2019.\n\nP. W. Koh and P. Liang. Understanding Black-box Predictions via Influence Functions. ICML. 2017.\n\nW. T. Stephenson, M. Udell, and T. Broderick. Approximate cross-validation with low-rank data in high dimensions. NeurIPS. 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ozXsD_IbLs", "reviewer": "Reviewer_ZZii", "summary": "Delta method allows one to construct confidence intervals. However the method involves substantial computational difficulties, such as inverting the Fisher information matrix, which is quite large if the number of parameters is large.\n\nThe paper proposes an alternative method. it involves regularisation and implies asymptotical convergence. The method is easier computationally and converges asymptotically. A number of convergence results are proven. The method returns good performance in practice.", "questions": "Please correct me if I am wrong. I might have missed important points.", "limitations": "Yes.", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I am worried this paper belongs to the domain of information theory rather than generic machine learning and some key points may be lost on the NeurIPS audience as they no doubts were lost on me.\n\nOtherwise the paper appears to make a substantial contribution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cT2rc_VYt9c", "reviewer": "Reviewer_PMCG", "summary": "The paper proposes the _implicit delta method_ for estimating the asymptotic variance of an estimator of the form $\\hat\\psi_n = \\psi(\\hat\\theta_n)$, where $\\hat\\theta_n$ is the MLE and $\\psi: \\Theta \\to \\mathbb{R}$ is a functional. The implicit delta method is computationally efficient in situations where fitting a model is computationally burdensome and/or the target functional is not known explicitly. The proposed method works by solving an additional estimation problem regularized with $\\lambda \\psi(\\theta)$. The paper shows that subject to regularity conditions, $\\hat V^{\\mathrm{FDIDM}} = \\lambda^{-1} (\\psi(\\hat\\theta_n(\\lambda)) - \\hat\\psi_n)$, where $\\hat\\theta_n(\\lambda)$ is the regularized estimate resulting from the additional estimation problem, consistently estimates the asymptotic variance of $\\hat\\theta_n$.", "questions": "1. For the finite-difference IDM, one can be so free in the choice of $\\lambda$: Isn't there a benefit in making $\\lambda$ as small as possible without causing numerical instability? Judging by the proof of Theorem 2, it looks like the choice of $\\lambda$ should depend on the curvature of $\\hat\\psi_n$ with respect to $\\lambda$. Would it be possible to see experimental results using a grid of values of $\\lambda$ for different problems?\n\n2. Section 3.2 was more difficult to follow compared to other parts. I think it may help to start the section with concrete examples rather than with a general description. Also, it is said $\\psi_0$ is random, but what is $\\psi_0$ in this case? $\\psi_0 = m^{-1} \\sum_{j=1}^{m} h(W_j; \\theta_0)$?\n\n3. Figure 2 is not ideal for comparing the performances of different methods; the reader has to imagine putting one panel on top of another. Could you create a different visualization that highlights the differences more?\n\n4. How do the methods compare in terms of computational cost?", "limitations": "The authors discuss some of the limitations of their work as a method but not the societal impact. Since this is a work in statistical methodology, I do not believe this is a defect.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes an intriguing alternative approach to computing the asymptotic variance that does not require inverting the Fisher information matrix (or an estimate thereof). The approach is shown to lead to a consistent estimate of the asymptotic variance, albeit under some regularity conditions.\n\nOne weakness of the paper is that the applicability of the method appears to be very much limited to the settings for which the delta method also applies. Thus, although the proposed approach works as a computationally lighter alternative to the delta method, it cannot be used with estimators that fall outside of the framework (e.g., lasso-type estimators or high-dimensional regimes). Perhaps it was not the intention of the authors to imply that their method can be applied to any estimation procedures, but this was the impression formed by reading earlier parts of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HK5mFsSMxW", "reviewer": "Reviewer_JWsG", "summary": "Authors propose a method for uncertainty estimation for predictive models. This method aims to directly compute uncertainty estimates for the cost function used to evaluate the predictive model. Main advantage of this method is that it avoids uncertainty quantification of the model parameters. This work provides consistency results in the large sample limit and it is empirically evaluated on different predictive tasks. \n", "questions": "Could you please provide empirical evidence comparing your approach to a (non-bootstrapped) ensemble of neural networks? \n\nWhy don’t you also employ the Fashion-MNIST data set and a simple neural network like Lenet-5? If you want to focus on UCI-data sets, the set of selected data sets is very reduced. \n\nWhich are the benefits of your approach against other approaches such as conformal predictions? This discussion should be added to the paper. \n\n", "limitations": "I ask authors to discuss the problem of not capturing the multimodality of the log-likelihood function. \n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The approach provides an alternative approach for uncertainty quantification which circumvents the problem of computing uncertainty estimates over the model parameters. This could potentially be a very interesting research direction, because modelling parameter uncertainty has proven to be extremely challenging in many models, especially neural networks. \n\n - The approach is theoretically motivated by some consistency guarantees in the large sample limit.\n\n- Usually, uncertainty quantification is  computationally very expensive. The proposed method provides a much cheaper way to compute confidence intervals. \n\n- Although the approach builds on previously widely known results, it’s adaptation to more general and larger models is novel and interesting. \n\n\nWeaknesses:\n\n- The only uncertainty that this approach is able to capture is the uncertainty around the exact location of the MLE estimate. In the case of a model with a highly multimodal loss landscape, the proposed method will be able to compute the uncertainty which belongs to a single mode. In models like NNs, it is widely known that capturing this kind of “local” uncertainty is much less powerful than capturing uncertainty involving multiple modes of the loss landscape, as it is done by ensemble of NNs or by powerful stochastic-gradient-Monte-Carlo methods: \n\nB. Lakshminarayanan, A. Pritzel, and C. Blundell, “Simple and scalable predictive uncertainty estimation using deep ensembles,” in Advances in neural information processing systems, 2017, pp. 6402–6413.\n\nWenzel, Florian, et al. \"How good is the Bayes posterior in deep neural networks really?.\" arXiv preprint arXiv:2002.02405 (2020\n\n\n*** Updated after discussion with authors ***\nI have updated this part of the review after the discussion with other authors and reviewers. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept