Detecting Extrapolation with Local Ensembles
Authors: David Madras
James Atwood
Alexander D'Amour
Venue:  (2020)
Keywords: extrapolation
reliability
influence functions
laplace approximation
ensembles
Rashomon set

Abstract: We present local ensembles, a method for detecting extrapolation at test time in a pre-trained model. We focus on underdetermination as a key component of extrapolation: we aim to detect when many possible predictions are consistent with the training data and model class. Our method uses local second-order information to approximate the variance of predictions across an ensemble of models from the same class. We compute this approximation by estimating the norm of the component of a test point's gradient that aligns with the low-curvature directions of the Hessian, and provide a tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pre-trained model is extrapolating on test data, with applications to out-of-distribution detection, detecting spurious correlates, and active learning.

Published as a conference paper at ICLR 2020
DETECTING EXTRAPOLATION WITH LOCAL ENSEMBLES
David Madras"
James Atwood
Alex D'Amour Google Brain al a_exdamour@googge..oo
University of Toronto
Google Brain
Vector Institute
atwoodj@googlee.com com
madrasêcs saaras@cs..trontt.coo toronto..od.....
ABSTRACT
We present local ensembles. method for detecting extrapolation at test time in a pre-trained model. We focus on underdetermination as a key component of extrapolation: aim to detect when many possible predictions are consistent with the training data and model class. Our method uses local second-order information to approximate the variance of predictions across ensemble of models from the same class, We compute this approximation by estimating the norm of the component of test point's gradient that aligns with the low-curvature directions of the Hessian, and provide tractable method for estimating this quantity. Experimentally, we show that our method is capable of detecting when a pre- trained model is extrapolating on test data, with applications to out-of-distributin detection, detecting spurious correlates, and active learning.
INTRODUCTION
As machine learning is deployed in increasingly vital areas, there is increasing demand for metrics that draw attention to potentially unreliable predictions. One important source of unreliability is extrapolation. Extrapolation can be formalized in number of ways: it can refer to making predictions on inputs outside the support of the training data, making predictions with high Bayesian or Frequentist uncertainty, or making predictions that depend strongly on arbitrary choices outside of the learning problem specification (e.g.. a random seed). In this paper, we develop a method for detecting this last form of extrapolation. Specifically, we say that trained model is extrapolating on test input if the prediction at this input is underdetermined - meaning that many different predictions all equally consistent with the constraints posed by the training data and the learning problem specification (i.e., the model architecture and the loss function).
Underdeterminationn is just one form of extrapolation, but it is particularly relevant in the context of overparameterized model classes (c.g. deep neural networks). Recently, simple (but computationally expensive) ensembling methods (Lakshminaray yananannnnaannnnnnnaaaaaa yanan al. (017). which train many models on the same data from different random seeds, have proven highly effective at uncertainty quantification tasks (Oadia et al. 2019). This suggests that underdetermination a key threat to reliability in deep learning, and motivates flexible methods that can detect underdetermined predictions cheaply.
With this motivation, we present local ensembles, a post-hoc method for measuring the extent to which a pre-trained model's prediction is underdetermined for a particular test input. Given a trained model, our method returns an extrapolation score that measures the variability of test predictions across local ensemble, i.e. set of local perturbations of the trained model parameters that fit the training data equally well. Local ensembles are a computationally cheap, post-hoc alternative to fully trained ensembles. and do not require special training procedures of approximate ensembling methods that measure related, but distinct, notions of uncertainty (Gal & Ghahramani 2015] Blundell et al.) 2015). Local ensembles also address gap in approximate methods for estimating prediction uncertainty. Specifically, whereas exact Bayesian or Frequentist uncertainty includes underdetermination as one component, approximate methods such Laplace approximations (MacKay 1992) or influence function- methods (Schulam & Saria 2019) break down when underdetermination is present.
"ome of this work was done while this author was an intern Google Brain.

---

Published as a conference paper at ICLR 2020
In contrast, method leverages the pathology that makes these methods struggle (an ill-conditioned Hessian).
Our contributions in this paper follows:
We present local ensembles, test-time method for detecting underdetermination--uee extrapolation in overparameterized models.
We demonstrate theoretically that our method approximates the variance of a trained ensem- ble with local second-order information.
We give a practical method for tractably approximating this quantity, which is simpler and cheaper than alternative second-order reliability methods.
Through experiments aimed at testing underdetermination, we show our method approxi- mates the behavior of trained ensembles, and can detect extrapolation in range of scenarios.
EXTRAPOLATION SCORE AND LOCAL ENSEMBLES
2.1 SETUP
Let a (x, y) be example onput-output pair, where is vector of features and y is label. We define a model in terms loss function L with parameters as a sum over training examples (44) i.e., L())= = Σ, {(z;, 0), where l is an example-wis loss (e.g., mean-squared error cross entropy). Let 0* be the parameters of the trained model, obtained by, e.g., minimizing the loss over an
this dataset, i.e., 0* arg ming [()). We write the prediction function given by parameters input a as f(x, 0). We consider the problem of auditing a trained model, where unlabeled test points x' arrive one at time in stream, and we wish assess extrapolation on point-by conttby-poiit basis.
In this section, we introduce our local ensemble extrapolation score en (z") for an unlabeled test point x' (significance of m explained below). The score is designed to measure the variability that would be induced by randomly choosing predictions from an ensemble of models with similar training loss. Our score has simple form: it the norm the prediction gradient go- (x") == Vağ(z', 0*) mul-
tiplied by matrix of Hessian eigenvectors spanning subspace of low curvature Um (defined in more detail below).
En (x') = ||U TT (22)
(1)
Here, we show that this score is proportional the standard de- viation of predictions across local ensemble of models with near-identical training loss, and demonstrate that this approxima- tion holds in practice.
In this quadratic bowl, arrows denote the small cigendi- rection, where predictions are slow to change. We argue this direction key to extrapolation.
Figure I:
2.2 DERIVATION
Our derivation proceeds in two steps. First, we define a local ensemble of models with similar training loss, then we state the relationship between our extrapolation score and the variability of predictions within this ensemble.
The spectral decomposition of the Hessian Ha- plays key role our derivation. Let
Hg. = T.UT
(2)
where U is a square matrix whose columns are the orthonormal eigenvectors of Hør, writ- ten (E()):" ... E(p)), and A is square, diagonal matrix with the eigenvalues of Hor, written (A(1): ) along its diagonal. As convention, we index the eigenvectors and eigenvalues in decreasing order of the eigenvalue magnitude.
To construct a local ensemble of loss-preserving models, we exploit the fact that eigenvectors with large corresponding eigenvalues represent directions of high curvature, whereas eigenvectors with small corresponding eigenvalues represent directions of low curvature. In particular, under the
2

---

Published as a conference paper at ICLR 2020
assumption that the model has been trained to local minimum saddle point, parameter pertur- bations in flat directions (those corresponding to small eigenvalues A(z) do not change the training loss substantially (see Fig, D.. We characterize this subspace by the span of eigenvectors with corresponding small eigenvalues. Formally, let m be the eigenvalue index such that the eigenvalues (AG) : > m), are sufficiently small to be considered "flat". We call the subspace spanned by (Su) j E ) the ensemble subspace. Parameter perturbations in the ensemble subspace generate an ensemble of models with near-identical training loss.
Our score mm (z') characterizes the variance of test predictions with respect to random parameter perturbations Ag that occur in the ensemble subspace. We now show that our extrapolation score Em (x') ) is, to the first order, proportional the standard deviation of predictions a point x'.
Let D be a random parameter perturbation with mean zero and covariance pro- portional to the identity e I. projected into the ensemble subspace spanned by {&G) : > m}. Let (2') be the linearized change in a test prediction at x' induced by perturbing the learned parameters 8* by Ag:
Proposition
Pa(x') = =900 (0- A i('''0 + Ao) 00(x'00000
Then Em((') 1 (D(PA))))
Proof. First, we characterize the distribution the loss-preserving parameter perturbation Ag. Let Um be the matrix whose columns {E()) : > m) span the ensemble subspace. Then U, is projection matrix that projects vectors into the ensemble subspace, so the projected perturbation As has covariance Um From this, derive the variance of the linearized change in prediction
Var(Pa) = e-go-(x') Um Un-on (x') (c')==+)))) = e II (a')112 (z))²
We test this hypothesized relationship on several tabular datasets. We train an ensemble of twenty neural networks using the same architecture and training set, varying only the random seed. Then, for each model in the ensemble, we cal- culate our local ensemble extrapolation score Em (z) for each input in the test set (see Sec. 3 for details). For each a', compare, across the ensemble, the mean value of Em to the standard deviation j(x'). In Fig. [ we plot these two quantities against each other for of the datasets, finding nearly linear relationship. On each dataset, we found a similar, significantly linear relationship (see Table I and Appendix AA) We note the relationship is weaker with the Diabetes dataset; the standard deviations of these ensembles are an order of mag- nitude higher than the other datasets, indicating much noisier data. Finally, we note that we can obtain similar results for the standard deviation of the loss at test points if we redefine 9 as the loss gradient rather than the prediction gradient.
Extapalation Score
1.0
0.5
0.5
-1.0
R-0.87
-15
-2.5
-1.0
0.0
Ensemble Standard Deviation
Figure 2: Mean extrapolation score vs. ensemble prediction standard deviation WineQuality dataset. Shows line of best fit and Pearson coefficient R. Both axes log-scaled.
COMPUTING EXTRAPOLATION SCORES
'To see this, consider second-order Taylor approximation to the training loss when the parameters are perturbed by vector Ao: L(O" + Ao) -(0*) Rs AT Hg- As Σ Ago (ST) Ao)² Note that the first-order term drops out because we assume 8* lies at a local minimum or saddle point. Perturbations As that lie in the subspace spanned by cigenvectors Egg whose corresponding cigenvalues Ag) are close to zero contribute negligibly this sum.
2 For now, we take m to he given, and discuss tradeoffs choosing m practice Section We note that not all loss-preserving ensembles will necessarily be local; however, the existence of some such local ensemble sufficient enable underdetermination.
3

---

Published as a conference paper at ICLR 2020
We now discuss practical method for computing our extrapolation scores. The key operation is constructing the set of eigenvectors that span a suitably pre-sereegg ensemble subspace (i.e. have sufficiently small corresponding eigenvalues). Because eigenvectors corresponding large eigenvalues are easier to estimate, we con- struct this subspace by finding the top cigenvectors and defining the ensemble subspace as their orthogonal complement. Our method can be implemented with any algorithm that returns the top m eigen- vectors. See below for discussion on some tradeoffs in the choice of m, well as the algorithm we choose (the Lanczos iteration Lanczos 1950)).
Dataset
Pearson
Boston
0.76
Diabetes
0.50
Abalone
0.76
Wine 0.87
Table 1: Correlation of extrap- olation scores and ensemble std. deviations on 4 datasets.
Our method proceeds as follows. For a given choice of m, we calculate the eigenvectors of H with the largest eigenvalues. These eigenvectors define an m-dimensional subspace which is the orthogonal complement to the ensemble subspace. We use these eigenvectors matrix that projects gradients into the ensemble subspace. Specifically, let Un be the matrix whose columns these large-eigenvalue eigenvectors (5() 1 S m). Then Um-UT is the projection matrix that projects vectors into the "large eigenvalue" subspace, and I Um U m= projects into its complement: the ensemble subspace. Now, for any test input x', we take the norm of this projected gradient to compute our score
construct
Em (x') = ||(1-- Um UmL+ -mm+UTT) go- (z')l22
The success of this approach depends on the choice of m. Specifically, the extrapolation score &m (x') is the most sensitive underdetermination in the region of the trained parameters if we set to be the smallest index for which the training loss is relatively flat in the implied ensemble subspace. If i is set too low, the ensemble subspace will include well-constrained directions, and Em(('' will over-estimate the prediction's sensitivity to loss- preserving perturbations. If m is set too high, the ensemble subspace will omit some under-constrained directions. and Em (m()' will be less sensitive. For models where all parameters are well-constrained by the training data, a suitable m may not exist. This will usually not be the case for deep neural network models, which are known to have very ill-conditioned Hessians (see, e.g., [Sagun et al. (017). Whether a direction is "well-constrained" is ultimately judgment call for the user. One potential heuristic is to consider a small parameter perturbation of a fixed norm in the ensemble subspace, and to set a tolerance for how much that perturbation can change the training loss; for small perturbations, the change in loss is a linear function of the curvature in the direction of the perturbation, which is upper bounded by A(m)-
We use the Lanczos iteration to estimate the top m eigenvectors, which presents a number practical advantages for usage in our scenario. Firstly, performs well under early stopping, returning good estimates of the top m eigenvectors after m iterations. Secondly, we can cache intermediate steps, meaning that computing the m + 1-th eigenvector fast once we have computed the first m. Thirdly, it requires only implicit access to the Hessian through a function which applies matrix multiplication, meaning we can take advantage of efficient Hessian-vector product methods (Pearlmutter) 1994).
Finally, the Lanczos iteration is simple - it can be implemented in less than 20 lines of Python code (see Appendix[B.. It contains only one hyperparameter, the stopping value m. Fortunately, tuning this parameter is efficient - given a maximum value M, we can try many values m < M at once, by estimating M eigenvectors and then calculating &,, by using the first m eigenvectors. The main constraint of our method is space rather than time- while estimating the first m eigenvectors enables easy caching for later use, it may be difficult to work with these eigenvectors in memory as m and model size p increase. This tradeoff informed our choice of m in this paper; note in some cases that increasing m further could have improved performance (see Appendix E This suggests that further work on techniques for mitigating this tradeoff, e.g. online learning of sparse representations (Wang & Lu 2016 Wang et al., (2012), could improve the performance of our method. See Appendix for more details on the Lanczos iteration.
4

---

Published as a conference paper at ICLR 2020
4 RELATED WORK
4.1 RELATION TO BAYESIAN AND FREQUENTIST SECOND-ORDER METHODS
It is instructive compare our extrapolation score two other approximate reliability quantification methods that are aimed at Bayesian and Frequentist notions of extrapolation, respectively. Like our extrapolation score, both of these methods make use of local information in the Hessian to make an inference about the variance of a prediction. First, consider the Laplace approximation of the posterior predictive variance. This metric is derived by interpreting the loss function as being equivalent to Bayesian log-posterior distribution over the model parameters 0, and approximating it with Gaussian. Specifically, (see, e.g., MacKay 1992)
Var(y x') ~ go (1') H 9o- (x') = Σ )=1
Second, consider scores such as RUE (Resampling Under Uncertainty) designed to approximate the variability of predictions by resampling the training data (Schulam & Saria 2019). These methods approximate the change in trained parameter values induced by perturbing the training data via influence functions (Koh & Liang Cook
Cook & Weisberg 1982). Specifically, the gradient of the parameters with respect to the weight of a given training example 2; is given by
I(2)) = H Vo((2,00) = = Σxel (50) Vollen))))))
(4)
J=1
Schulam & Saria (2019) combine this influence function with specific random distribution of weights to approximate the variance of predictions under bootstrap resampling, other similar formulations are possible, sometimes with theoretical guarantees (Giordano et al. 2019).
Importantly, both of these methods work well when model parameters are well-constrained by the training data, but they struggle when predictions are (close to) underdetermined. This is because, in the presence of underdeterminnttion, the Hessian becomes ill-conditioned. Practical advice for dealing with this ill-conditioning is available (Koh & Liang 2017), but we note that this not merely a numerical pathology; by our argument above, a poorly conditioned Hessian is a clear signal of extrapolation. In contrast to these methods, our method focuses specifically on prediction variability induced by underconstrained parameters. Our extrapolation score incorporates only those terms with small eigenvalues, and removes the inverse eigenvalue weights that make inverse Hessian methods 2
break down. This is clear from summation representation: En (x') Σ;>
(S6)90 (x"))
Our method also has computational advantages over approaches that rely on inverting the Hessian. Firstly, implicitly inverting the Hessian is a complex and costly process - by finding only the impor- tant components of the projection explicitly, our method is simpler and more efficient. Furthermore, we only need to find these components once; we can reuse them for future test inputs. This type of caching is not possible with methods which require to calculate the inverse Hessian-vector product for each new test input (e.g. conjugate gradient descent).
4.2 RELATED WORK IN DETECTING EXTRAPOLATION
Some recent works explore the relationship between test points, the learned model, and the training set. Several papers examine re reliability criteria that are based on distance in some space: within/between group distances (Jiang et al., 2018), pre-specified kernel in a learned embedding space (Card et al.) 2019), or the activation space of a neural network (Papernot & McDaniel 2018). We implement some nearest-neighbor baselines inspired by this work in Sec. [5] Additionally, range of methods exist for related tasks, such as OOD detection (Choi & Jangl 2018 Liang al. 2017 7 Gal & Ghahramani 2015 Schölkop et al. 2001) and calibration (Nacini et al. 2015 Guo et al. (017)). Some work using generative models for OOD detection uses related second-orde analysis (Nalisnick et al. 2018))
A line of work explores the benefits of training ensemble methods explicitly, discussed in detail in Dietterich 2000). These methods have been discussed for usage in some of the applications
present in Sec. 5 including uncertainty detection (Lakshminarayanan et al. (017). active learning
(Melville & Mooney 2004) and OOD detection (Choi & Jang 2018). A number of approximate
5

---

Published as a conference paper at ICLR 2020
ensembling methods have also been proposed. For example, MC-Dropout (Gal & Ghahramani 2015)
and Bayes by Backprop (Blundell et al. 2015) are approximate Bayesian model averaging methods that leverage special training procedures to represent ensemble of prediction functions. These target a distinct notion of uncertainty from our loss-preserving ensembles (see Appendix F Finally, line of work "Rashomon sets" explores loss- pesserregg ensembles more formally simpler Fisher et al. 2018).
model classes (Semenova & Rudin 2019
5 EXPERIMENTS
In this section, we give evidence that local ensembles can detect extrapolation due to underdetermina- tion in trained models. In order explicitly evaluate underdetermination, we present range of exper- iments where a pre-trate pre-trained model has a "blind spot", and evaluate its ability to detect when an input that blind spot. We probe our method's ability to detect a range extrapolation, exploring cases where the blind spot is: I. easily visualized, 2. well-defined by the feature distribution, 3. well-defined by a latent distribution, and 4. unknown, but where we can evaluate our model's s detection perfor- mance through auxiliary task. See Appendix D for experimental details. Code for running the local ensembles method can be found atttts:: /ttp//gtttttttt/tt wtt.c/////c//...c haamm/dmddraaaaaaooo /local ensembles
5.1
VISUALIZING EXTRAPOLATION DETECTION
0.5
1.0
2
Standard Deviation
0.9
1.2
0.4
0.8
0.8
Cosine Sinilaaity
0.7
0.6
0.3
0.6
-1
Train
0.4
Test
0.4
-1
0.3
20
(a) Data and trained model
(b) Extrapolation scores
(c) Eigenvector estimates & AUC
Figure 3: We train neural network ensemble (Fig. 3ap. We compute extrapolation scores (solid line), which correlate with the standard deviation of the ensemble (dotted line) (Fig. (3b). Our OOD performance achieves high AUC (solid line) even though some of our eigenvector estimates have low cosine similarity to ground truth (dotted line) (Fig. Bcb.
We begin with an easily visualized toy experiment. In Fig. 33 we show
our data (y/ sin 4x: + +N(0, 4)). We generate in-distribution training
data from x E [-1, 0] (11, 2). but at test time, we consider all x E [-3, 4.. We train 20 neural networks with the same architecture (two hidden layers 25
Lanczos
True
of three units each). As shown in Fig. 3al the ensemble disagrees most This means that should most mistrust predictions from this model class on these extreme values, since there are many models within the class that perform equally well on the training data, but differ greatly on those inputs. We should also mistrust predictions from [0, 1], although detecting this extrapolation may be harder since the ensemble agrees more strongly on these points.
20
on 1, x: >
5
1.0
15
Figure True and es- timated eigenspectrums for toy model Hessian. We note that the first few eigenvalues account for most of the variation, and that our estimates are ac- curate.
For each model in the ensemble, we test our method on an OOD task: can we flag test points which fall outside the training distribution? Since OOD examples may be uncommon in practice, we use AUC to measure our performance. We show that the extrapolation score is empirically related to the standard deviation of the ensemble's predictions at the input, which in turn is related to whether the input is OOD (Fig. 3bb. Examining one model from this ensemble, we observe that by estimating only m = eigenvectors, achieve 90% AUC (Fig. Bep. It turns out that task/model. As we complete more iterations (m > 10) we start finding smaller cigenvalues, which more important the ensemble subspace and whose eigenvector we do not wish to project out.
10 performs best on this

---

Published as a conference paper at ICLR 2020
1.0
0.85
0.9
0.9
0.25
0.80
0.8
MN iFinal
Mi (indut))
0..00
2.75
0.7
MN
0.7
0.65
0.6
Locall Ensembles
Feotures Simulated
Features Simuatted
ofFeztures Simulated
Number Fastuure Simulated
Figure 5: AUC achieved on WineQuality simulated features task (y-axis) compared to the number of extra features simulated (x-axis). Noise parameter a increased from left to right a E (0, 0.1, 0.2, 0.5). Solid line is our method. Results averaged across 5 random seeds, standard deviations shown. Results for the other three datasets are qualitatively similar and shown in Appendix C
We note our AUC improves even with some eigenvector estimates having low cosine similarity the ground truth cigenvectors (the Lanczos iteration has some stochasticity due to minibatch estimation see Appendix [or details). We hypothesize this robustness is because the ensemble subspace of this model class is relatively low-dimensional. Even if an estimated vector is noisy, the non-overlapping parts of the projection will likely be mostly perpendicular to the ensemble subspace, due to the properties of high- high-dimensional space.
5.2 SIMULATED FEATURES
In this experiment, we create blind spot in given dataset by extending and manipulating the data's feature distribution. We induce collinearity in feature space by generating new features which are linear combination of two other randomly selected features in the training data. This means there is potential for underdetermination: multiple, equally good learnable relationships exist between those features and the target. However, at test time, we will sometimes sample these simulated features from their marginal distribution instead. This breaks the linear dependence, requiring extrapolation (the model is by definition underconstrained), without making the new data trivially out-of-distribution. We can make this extrapolation detection task easier by generating several features this way, or make it harder by adding some noise ~ (O, o²) to these features.
We run this experiment on four tabular datasets, training a two hidden-layer neural network for each task. We compare three neirest-n baselines, where the metric is the distance (in some space) of the test point to its nearest neighbour by Euclidean distance in an in-distribution validation set. NN (Inputs) uses input space: NN (Reprs) uses hidden representation space, which is formed by concatenating all the activations of the network together (inspired by Papernot & McDaniel (2018). who propose similar method for adversarial robustness); and NN (Final Layer), uses just the final hidden layer of representations. We note that since our method is post-hoc and can be applied to any wvice-differentiable pre- pre-trained model, we do not compare to training-basse methods e.g. those producing Bayesian predictive distributions (Gal & Ghahramani, 2016 Blundell et al., 2015). Our metric is AUC: we aim assign higher scores to inputs which break the collinearity (where the feature is drawn from the marginal), than those which do not. In Figure 5 we show that local ensembles (LE) outperform the baselines for each number of extra simulated features, and that this performance is fairly robust to added noise.
5.3 CORRELATED LATENT FACTORS
by inducing blind spot in latent space. The rationale similar: if two latent factors are strongly correlated training time, the model may grow reliant on that correlation, and at test-time may be underdetermined if that correlation is broken.
Here, we extend the experiment from Section 5.2
We use the CelebA dataset (Liu et al. 2015) of celebrity faces, which annotates every image with 40 latent binary attributes describing the image (e.g. "brown hair"). To induce the blind spot, we choose two attributes: label L and a spurious correlate C. We then create a training set where L and C' are perfectly correlated: point is only included in the training set if L = C. We train a convolutional neural network (CNN), with two convolutional layers and a fully- cally-connected layer at the end, as binary classifier predict L. Then, create test set of held-out data where P(L = C) = P(L + C). The test data where L + C in our model's blind spot; these are the inputs for which want to output high extrapolation scores. We show in Appendix that the models
7

---

Published as a conference paper at ICLR 2020
dramatically fail classify these inputs (L + C). We compare to four baseline extrapolation scores: the three nearest-neige neighhour methods described in Sec. 5.2) as well as MaxProb, where we use 1- the maximum outputted probability of the softmax. We test two values of L (Male and Attractivee and two values of C (Eyeglasses and WearingHat). We chose these specific values of I. because they are difficult to predict and holistic i.e.and not localized to particular areas of image space. In Table 2 we present results for each of the four L, C' settings, showing both the loss gradient and the prediction gradient variant of local ensembles. Note that the loss gradient cannot be calculated at test time since we do not have labels available instead, we calculate a separate extrapolation score using the gradient for the loss with respect to each possible label, and take the minimum. Our method achieves the best performance on each. However, the variation between the tasks is quite noteworthy. We note two patterns in particular. Firstly, we note the performance of MaxProb and the loss gradient variant of our method are quite correlated, which we hypothesize relates to V. Additionally, the effeet of increasing m inconsistent between experiments: we discuss possible relationships to the cigenspectrum of the trained models. See Appendix for a discussion on these patterns.
most settings, and is competitive with the best baseline
5.4 ACTIVE LEARNING
Random
0.60
Random
Classfiiction Error
0.4
Local Ensembles
Classification Error
0.55
Local Ensembles
0.3
0.50
0.45
0.2
0.40
0.1
0.35
0.30
0.0
20
60
100
20
100
Rounds of Active Learning
Rounds of Active Learning
(a) MNIST
(b) FashionMNIST
Figure 6: Active learning results. Average of random seeds shown with std. deviation error bars.
Finally, consider the where we know our model has blind spots, but do not know where they are. We use active learning to probe this sit- uation. If underdetermination is an important factor in extrapolation, then local ensembles should be useful for selecting the most useful points to add to our training set. We use MNIST (LeCun et al., (010)) and FashionM- NIST (Xiao et al. (017) for our ac- tive learning experiments. We begin the first round with a training set of twenty: two labelled data points from each of the ten classes. In each round, we train our model (a small CNN with two convolutional layers and fully connected layer at the end) mum validation loss using the current training set. After each round, we select ten new points from a randomly selected pool of 500 unlabelled points, and add those ten points and their labels to our training set. We compare local ensembles (selecting the points with the highest extrapolation scores using the loss- boss-gradient variant) to a random baseline selection mechanism. In Fig. 6 we show that our method outperforms the baseline on both datasets, and this improvement increases in later rounds of active learning. We only used 10 eigenvectors in our Lanczos approximation. which we found to be a surprisingly effective approximation; did not observe improvement with more eigenvectors. This
Method
M/E
M/H
A/E
A/H
MaxProb
0.738
0.677
0.461
0.433
NN (Pixels)
0.561
0.550
0.521
0.547
NN (Reprs)
0.584
0.578
0.503
0.533
NN (Final Layer)
0.589
0.517
0.480
0.497
LE (Loss)
0.770
0.684
0.454
0.456
LE (Predictions) 0.364
0.544
0.519
0.582
Table 2: AUC for Latent Factors OOD detection task. Col- umn heading denotes in-distribution definitions: labels are M (Male) and A (Attractive); spurious correlates are F (Eye- glasses) and И (Wearing Hat). Image is in-distribution iff label = spurious correlate. LE stands for local ensembles. Each Lanczos iteration uses 3000 eigenvectors. 500 exam- ples from each test are used. 95% CI is bolded.
mini-
8

---

Published as a conference paper at ICLR 2020
experiment serves emphasize the flexibility of our method: by detecting underlying property of the model, we can use the method for a range of tasks (active learning as well as OOD detection).
CONCLUSION
We present local ensembles, post-hoc method for detecting extrapolation due to underdetermination in trained model. Our method uses local second-order information to approximate the variance of an ensemble. We give a tractable implementation using the Lanczos iteration to estimate the largest eigenvectors of the Hessian, and demonstrate its practical flexibility and utility, Although this method is not a full replacement for ensemble methods, which can characterize more complexity (e.g. multiple modes), we believe fills an important role in isolating one component of prediction unreliability. In future work, we hope to scale these methods to larger models and to further explore the properties of different stopping points m. also hope to explore applications fairness and interpretability, where understanding model and training bias is of paramount importance.
7 ACKNOWLEDGEMENTS
Thank you to Jamie Smith, Yaniv Ovadia, Yoni Halpern, Pang Wei Koh, Jackson Wang, James Lucas, Saren-Eteenne Brunct, Kamyar Ghasemipour and Elliot Creager for helpful comments and discussions.
REFERENCES
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural networks. arXiv preprint arXiv:1505.05422 2015.
Dallas Card, Michael Zhang, and Noah A Smith. Deep weighted averaging classifiers. In Proceedings of the Conference on Fairness, Accountability, and Transparency, pp. 369-378. ACM, 2019.
Hyunsun Choi and Eric Jang. Generative ensembles for robust anomaly detection. arXiv preprint 19.. 2018
Dennis Cook and Sanford Weisberg. Residuals and influence in regression. New York: Chapman and Hall, 1982.
Paulo Cortez, António Cerdeira, Fernando Almeida, Telmo Matos. and José Reis. Modeling wine preferences by data mining from physicochemical properties. Decision Support Systems, 47(4): 547-553, 2009.
Jan JM Cuppen. A divide and conquer method for the symmetric tridiagonal eigenproblem. Nu- merische Mathematik, 36(2):177-195, 1980.
Inderjit S Dhillon. new o (n2) algorithm for the symmetric tridiagonal eigenvalue/eigenvvetor
problem. Technical report, CALIFORNIA UNIV BERKELEY GRADUATE DIV, 1997.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pp. 1-15. Springer, 2000.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL
ics. www..du/mii.........
Bradley Efron, Trevor Hastie, lain Johnstone, Robert Tibshirani, et al. Least angle regression. The
Annals of statistics, 32(2):407-499 2004.
Aaron Fisher, Cynthia Rudin, and Francesca Dominici. All models are wrong but many are useful: Variable importance for black-box, proprietary, misspecified prediction models, using model class reliance. arXiv preprint /r0iv: 2018.
Gal and Ghahramani. Dropout as bayesian approximation: Representing model uncertainty in deep learning. arxiy, 2015.

---

Published as a conference paper at ICLR 2020
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian upproximation: Representing model uncertainty in deep learning. In international conference on machine learning, pp. 1050-1059, 2016.
Ryan Giordano, William Stephenson, Runjing Liu, Michael Jordan, and Tamara Broderick. swiss army infinitesimal jackknife. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1139-1144 2019.
Luc Giraud, Julien Langou, Miroslav Rozložník, and Jasper van den Eshof. Rounding error analysis of the classical gram-schmidt orthogonalization process. Numerische Mathematik, 101(1):87-100 2005.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1321-1330 JMLR. org, 2017.
David Harrison Jr and Daniel L Rubinfeld. Hedonic housing prices and the demand for clean air. Journal of environmental economics and management, 5(1):81-102, 1978.
Heinrich Jiang, Been Kim, Melody Guan, and Maya Gupta. To trust or not to trust a classifier. In Advances in Neural Information Processing Systems, pp. 5541-5552, 2018.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pp. 1885-1894. JMLR. org, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Comelius Lanczos. An iteration method for the solution the eigenvalue problem of linear differential and integral operators. United States Governm. Press Office Los Angeles, CA, 1950.
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. AT&T Labs [Online]. Available: htlp://yann, lecun. com/exdb/mnist, 2:18, 2010.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. arXiv preprint / 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015, 2015.
David JC MacKay. Information- objective functions for active data selection. Neural computa- tion, 4(4):590-604, 1992.
Prem Melville and Raymond Mooney. Diverse ensembles for active learning. In Proceedings of the /wenty-first international conference on Machine learning, pp. 74. ACM, 2004.
Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht. Obtaining well calibrated prob- abilities using bayesian binning. In Twenty-Ninth AAAI Conference on Artificial Intelligence, 2015.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? arXiv preprint arxiv:1810.091136, 2018.
Warwick J Nash, Tracy L Sellers, Simon R Talbot, Andrew J Cawthorn, and Wes B Ford. The population biology of abalone (haliotis species) in tasmania. blacklip abalone rubra) from the north coast and islands of bass strait. Sea Fisheries Division, Technical Report, 48, 1994.
Travis E Oliphant. A guide to NumPy, volume 1. Trelgol Publishing USA, 2006.
Yaniv Ovadia, Emily Fertig, Jie Ren, Zachary Nado, D Sculley, Sebastian Nowozin, Joshua V Dillon, Balaji Lakshminarayanann and Jasper Snock. Can you trust your model's uncertainty? evaluating predictive uncertainty under dataset shift. arXiv preprint /906.0990000000 2019.
10

---

Published as a conference paper at ICLR 2020
Nicolas Papernot and Patrick McDaniel. Deep k-nearest neighbors: Towards confident, interpretable and robust deep learning. arXiv preprint arXiv:1803.04765 2018.
Barak A Pearlmutter. Fast exact multiplication by the hessian. Neural computation, 6(1):147-160, 1994.
Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825-2830, 2011.
Levent Sagun, Utku Evei, V Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis of the hessian of over-parametrized neural networks. arXiv preprint arXiv:1706.044444 2017.
Bernhard Schölkopf, John C. Platt, John Shawe-Taylor, Alex J Smola, and Robert C Williamson. Estimating the support of high-dimensional distribution. Neural computation, 1347311443-1471
2001.
Peter Schulam and Suchi Saria. Auditing pointwise reliability subsequent training. In Proceedings of the 22nd International Conference on Ar-tificial Intelligence and Statistics (AISTATS), 2019.
Lesia Semenova and Cynthia Rudin. study in rashomon curves and volumes: A new perspective on generalization and model simplicity in machine learning. arXiv preprint 3xXiv:9908.007555 2019.
Chuang Wang and Yue M Lu. Online learning for sparse pea high dimensions: Exact dynamics and phase transitions. In 2016 IEEE Information Theory Workshop (ITW), pp. 186-190, IEEE, 2016.
Dong Wang, Huchuan Lu, and Ming- Hsuan Yang. Online object tracking with sparse prototypes. IEEE transactions on image processing, 22(1):314-325, 2012.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnin a novel image dataset for benchmarking machine learning algorithms, 2017.

---

Published as a conference paper at ICLR 2020
Brioppo Score
20-
1.2
3.0
usten
2.6
5.0
0.50 50 0.75 125 1.50 2.0.5.2..02.2 Ensomble Standard Deviation
4 Ensomblc Sondard Dov
10
14
0.2
0.4
0.6
08
Ersemblc Standord Devittion
Figure 7: Datasets are (left right) Boston, Diabetes, Abalone. Dotted line represents linear fit of data. R is Pearson correlation coefficient.
2.0
Excopaaaioon Score
2.5
2.0
R-0.76
R=0.50
R=0.76
Ensembble Standardd Deviation
Ensemmale Standard Deviation
Ensemble Standard Deviation
Figure 8: Datasets are (left to right) Boston, Diabetes, Abalone. Each axis log-scaled. Dotted line represents linear fit of data. R is Pearson correlation coefficient.
A ENSEMBLES: OTHER DATASETS
We show Figures 7 and g the same plots as Fig. 2 for other tabular datasets, demonstrating the strongly linear relationship between extrapolation score and ensemble standard deviation.
B THE LANCZOS ITERATION: FUTHER DETAILS
The Lanczos iteration (aanzoos 1950) is a method for tridiagonalizing a Hermitian matrix. It can be thought of as variant of power iteration, iteratively building larger basis through repeatedly multi- plying an initial vector by M, ensuring orthogonality at each step. Once M has been tridiagonalized, computing the final eigendecomposition relatively simple number of specialized algorithms exist which are relatively fast (0(p²) (Dhillon 1997 Cuppen 1980).
The Lanezos iteration simple implement, but presents some challenges. The first challenge is numerical instability - when computed in floating point arithmetic, the algorithm is no longer quarnteed to return good approximation to the true eigenbasis, or even an orthogonal one. As such, the standard implementation of the Lanczos iteration is unstable, and can be inaccurate on simple problems. Fortunately, solutions exist: a procedure known as two-step classical orthogonalization
which involves ensuring twice that each vector is linearly independent of the previous ones - guaranteed to produce an orthogonal basis, with errors on the order of machine roundoff (Giraud al. 2005). A second potential challenge presented by the stochasticity of minibatch computation. Since we access our Hessian only through Hessian-vector products, we use only minibatch computation each stage. This means that each iteration will be stochastic, which will decrease the accuracy (but not the orthogonality) of the eigenvector estimates provided. However, in practice, we found that even fairly noisy estimates were nonetheless useful Sec. 5.1 for more discussion.
B.1 LANCZOS ALGORITHM CODE SNIPPET
The Lanczos algorithm is quite simple to implement. Figure shows a a short implementation using Python/Numpy (Oliphant 2006).
12

---

Published as a conference paper at ICLR 2020
import numpy np
def lanczos (matmul_ff, En, dim, num_iters, eps-le-8):
Givenn implicit access to a matrix of size (dim x dim) through mamtul_fn, tridiagonalize M into diagonal elements Alpha and off-diagonal elements Beta. Also return the associated orthonormal basis Q."!!
Initialize orthonormal basis.
Q = zeposeros (dim, 1
Initialize off-diagonal elements.
Beta = []
Initialize diagonal elements. Alpha = []
# Begin with random initial vector.
q = np, ra.random. wnw.ornnffomooooo (size- (dim, 1
apapee (q np. linallllgggll (orrmmggg.))g()))))) norm(g)) 0_k_ragge [[0]
for k in range (1, num_iters + 1):
# Compute next step power iteration.
2 [attul__n([[K])
Alpha. Alpha.append (ap. matkml .T, 2))
0 k_k_r = np. concateenne ([o_k_range, @[k]], axis=1)
# Reorthogonalize using two-step classical Gram-Schmidt.
for in range (2):
(2)
z_orth np. (np. (z.T, (_k_range)
0_k_range, axis=l, keepdims=True) 2_orth
Beta Beta. append (np. www.lllnlg.ooo... norm (z)
Q. append (2 / Beta[k]]
# Check for convergence. if np.linalg.norm
eps:
break
return Q Beta, Alpha
Figure 9: Example Python implementation of Lanczos algorithm for tridiagonalizing an implicit matrix M.
1.0
0.9
N0% Oirpiis)
NW (Final Layur)
0.9
0.20
0.85
0.75
0.80
0.8
Cheprs!
9.975
0.7
0.60
0.8
0.6
0.55
0.50
Namber Fedt Simdiated
Namber and. Simdlated
Number of Fealures Simdlated
Number of Fualures Simdluied
Figure 10: Boston dataset
C SIMULATED FEATURES OTHER DATASETS
In Section 5.2
present results for experiment where aim to detect broken collinearities in
the feature space. In Figures I0 II and [2 we show results on three more tabular datasets. See the main text for more experimental details.
13

---

Published as a conference paper at ICLR 2020
1.0
0.9
0.5
MN
(Reprs
0.7
0.9
0.9
0.7
0.8
0.60
0.65
0.75
0.70
0.5
Locall Ensemblee
0.6
0.6
0.55
0.50
0.45
Feotures
Simdated
of
Figure 11: Diabetes dataset
0.950
0.55
0.90
2 0.500
0.925
MI linouts)
y
0.80
0.875
MN
0.550
0.825
(Reprsi
0.70
Number Features Sim.lated
Number Features Sim.lated
Number ofFeutures Sim.lated
Number Festures Simulated
Figure 12: Abalone dataset
D EXPERIMENTAL DETAILS
Here we give more experimental details on the datasets and models used.
D.1 DATASETS
D.1.1 TABULAR DATASETS
We subtracted the mean and divided by standard deviation for each feature (as calculated from the training set).
Boston (Harrison Jr & Rubinfeld 1978) and Diabetes (Efron et al. 2004). These datasets were loaded from Seikit-Leam (Pedregosa et al. 201T).
Abalone Nash et al. 1994). This dataset was downloaded from the UCI repository (Dua & Grafi
2017) at http:://t//// /tatt:://rrchiii /archiieeee.. ics uci http:////aaatttttt/aaaaaaaaaaaaa We converted sex to three-dimensional one-hot vector (M, F, I).
WineQuality (Cortez et al. 2009). This dataset was downloaded from the UCI repository (Dua Graff 2017) at htthtt::// /aarhhiiiee. ics. http://.....aae///aaaaaaaaaataaa used both red and white wines.
We
D.2 MNIST, FASHIONMNIST AND CELEBA.
These datasets were loaded using Tensorflow Datasets http:::/tttttttiiie datasets We divided the pixel values by 255 and, for MNIST and FashionMNIST, binarized by thresholding at 0.7.
D.3 EXPERIMENTAL DETAILS
D.3.1 TOY DATA EXPERIMENTS
For the first experiment, train two-layer neural network with 3 hidden units in each layer and tanh units. We train for 400 optimization steps using minibatch size 32. Our data is generated from y sin(4r) (O, 1). We generate 200 training points, 100 test points, and 200 OOD points. We aggregate over grid of 10 points from to 1, with aggregation function min. We run the Lanczos algorithm until convergence.
For the second experiment, we train a two-layer neural network with 5 hidden units in each layer and ReLU units. Our data is generated from y 3xx2 + (N(0, 1). Our training set consists of drawn
14

---

Published as a conference paper at ICLR 2020
Test Set
M/E
M/H
A/E
A/H
In-Distribution
0.03
0.06
0.01
0.02
Dut-of-Distribuun 0.98
0.96
0.90
0.93
Table Error rate for in and out of distribution test set with correlated latent factors setup. Column heading denotes in-distribution definitions: labels are M (Male) and A (Attractive); spurious correlate are (Eyeglasses) and (Wearing Hat). Image is in-distribution label = spurious correlate.
uniformly from |-0.5, 0.5] and [2.5. 3.5). However, at test time, we will consider x: E [-3, 6]. We generate 200 training points, 100 test points, and 200 OOD points. We aggregate Y over grid of points from to 9, with aggregation function min.
D.3.2 SIMULATED FEATURES
For each dataset we use the same setup. We use a two-layer MLP with ReLU activations and hidden layer sizes of 20 and 100. We trained all models with mean squared error loss. We use batch size 64, patience 100 and 100-step running average window for estimating current performance. For the Lanczos iteration, we run up to 2000 iterations. We always report numbers from the final iteration run. For estimating the Hessian in the HVPs minibatches.
Lanczos iteration, we use batch size 32 and sample
To pre-process the data, we first split randomly out 30% the dataset as OOD, We choose random features i, and number 3 ~ 2/(0, 1). and generate the new feature ă Br [i] (1 (1-8)x[j] We also normalize this feature by its mean and standard deviation as calculated on the training set. We add random noise the features after splitting into in-distribution and OOD) meaning we redrawing from the same marginal noise distribution. We use 1000 examples from in-distribution and OOD for testing.
D.3.3 CORRELATED LATENT FACTORS
We CNN with ReLU hidden activations. We two convolutional layers with 50 and 20 filters each and stride size 2 for total of 1.37 million parameters. We trained all models with cross entropy loss. We a 100-step running average window for estimating current performance. We sample the validation set randomly as 20% of the training report numbers from the final iteration run. We use 500 examples from in-distribution and OOD for testing. For estimating the Hessian in the HVPs in the Lanezos iteration, we use batch size 16 and sample 5 minibatches.
extra dense layer on top with 30 units. We batch size 32, patience 100 steps, and
For the Lanczos iteration, we run 3000 iterations. We always
D.3.4 ACTIVE LEARNING
We use a CNN with ReLU hidden activations. We use two convolutional layers with 16 and 32 layers, stride size and dense layer on top with units. We trained models with mean squared error loss. We use batch size 32, patient 100 steps, and 100-step running average window for estimating current performance.
E CORRELATED LATENT FACTORS
E. PERFORMANCE OF BINARY CLASSIFIERS
In Section 5.3 we discuss an experiment where we correlated a latent label and confounder C attribute. Table 3 shows the in-distribution and dut-of-distribution test error. These drastically different, meaning that learning to detect this type of extrapolation is critical to maintain model performance.
15

---

Published as a conference paper at ICLR 2020
0.455
0.705
0.6855
0.445
0.70
0.555
0.6845
0.435
0.550
0.6840
0.425
1002
3002
300 2000000000000000
1000
2300 3020
502 1000
3020
Figure 13: Latent Factors OOD detection task, using loss gradient. Y-axis shows AUC, X-xxis shows the number of eigenvectors estimated by the Lanczos algorithm, data sampled every 500 eigenvectors. Tasks from left to right are M/E, M/H, A/E, A/H.
0.38
0.58
0.53
0.57
0.36
0.56
0.51
0.34
0.50
0.54
0.32
0.48
0.52
1900 1500 2000
2500
3300
1000 1509 2000 2500 3800 550 1000
2500 2500 3000
1002
1500
2000 2503 3850
Figure 14: Latent Factors OOD detection task, using loss gradient. Y-axis shows AUC, X-axis shows the number of cigenvectors estimated by the Lanczos algorithm, data sampled every 500 cigenvectors. Tasks from left to right are M/E, M/H, A/E, A/H.
E.2 BEHAVIOUR OF AUC WITH MORE ESTIMATED EIGENVECTORS
13 and 14 we show that the tasks present differing behaviours more eigenvectors are estimated. We observe that for the MalelEyeglasses and AttractiveWWarrigglaa tasks, we get improved performance with more eigenvectors, but for the others do not necessarily the same improvements. Interestingly, this upward trend occurs both times that our method achieves a statistically significant improvement of the task and not others, but we hypothesize that this a sign that the method is working more correctly in these settings.
In Fig.
baselines. It is unclear why this occurs for some settings
E.3 RELATIONSHIP BETWEEN Loss GRADIENT AND MaxProb METHOD
As discussed in Sec. 5.3 we have the relationship between the loss f, prediction Y, and parameters Val = Vye- VoÝ. Using min as an aggregation function, we find that miny ( V Y)
has an inverted V-shape (Fig. 15 This similar shape max(Ý, Y). which is the metric implicitly measured by MaxProb.
E.4 ESTIMATED EIGENSPECTRUN OF DIFFERENT CORRELATED LATENT FACTOR TASKS
In Fig. we examine the estimated eigenspectrums of the four tasks we present in the correlated latent factors experiment, to see if we can detect a reason why performance might differ on these tasks.
In Fig. 6a we show that the two tasks with the label Attractive, the eigenvalues are larger. These are also the tasks where the loss- gradient-based variant of local ensembles failed, indicating that that variant the method may worse handling larger eigenvalues.
3.6
2.0
3.0
0.2
0.6
08
15
In Fig. 16b we show that the two tasks where the local ensembles A/ H, achieving statistically signifi-
Figure 15: X-axis is P, Y-axis is minye/0.1 V),(
methods performed best (M/E: E, cant improvements over the baselines also showing improvement as more eigenvectors were estimated), the most prominent negative eigenvalue is relatively smaller mag- nitude compared to the most prominent positive eigenvalue. This could mean that the local ensembles method was less successful in the other tasks (M/H,
95% confidence level, and
16

---

Published as a conference paper at ICLR 2020
1.00
2.5
Male Eyeelasssss
Normalized Eigenvalue
0.75
Male Wearing HHt
0.0
(og((Eigenvalue)
0.50
Attractive, Eyeglasses
-2.5
0.25
Attractive, aetarig Hat
-5.0
0.00
-7.5
Male_ Eelelasssssssss
-0.25
-10.0
Male_ Seee_eering__he Hat
-0.50
Attractive Eyeglasses seraclive_Eyeglasses
-12.5
-0.75
Attractive Wearing_ Hat
-15.0
-1.00
500
1000 1500 2000 2500 3000
0
500
1000 1500 2000 2500 3000
(a) Log absolute eigenvalues
(b) Normalized eigenvalues
Figure 16: We show the estimated eigenspeetrum of the four CNNs we train on the correlated latent factors task. In Fig.
16a show the absolute estimated eigenvalues sorted by absolute value, a log scale. In Fig. eigenvalues. We only show 3000 estimated eigenvalues because 3000 iterations, meaning we did not estimate the rest of the eigenspectrum.
16b we show the estimated eigenvalues divided by the maximum estimated
the Lanczos iteration for only
simply because those models were not trained close enough to convex minimum and still had fairly significant eigenvalues.
F EMPIRICAL COMPARISON TO MC- -C-DRRPOUT
One of the strengths of our method is that it can be applied post-hoc, making it usable a wide range of situations. Therefore, choose to compare the local ensembles method to only baselines which can also be applied post-hoc. However, one might still be interested in how some of these methods compare against ours. One such method the MC-Dropout method (Gal & Ghahramani 2016), which constructs an ensemble at test time from model trained with dropout, by averaging over several random dropout inferences. This method is not considered post-hoc, since only applicable to methods which have been trained with dropout (and works best for models with dropout on each layer, rather than just a single layer).
Both MC-dropout and local ensembles estimate the vari- ance of an ensemble achieved through perturbations trianed model. In MC-dropout, those perturbations take form of stochastically setting some weights the model to O; local ensembles, they Gaussian pertur- bation projected into the ensemble subspace. However, despite these similarities, we provide empirical evidence here that our method is categorically different than MC- Dropout, by showing that our perturbations find "flatter" (more loss-presedd preserving) ensemble. This means that our method is more directly measuring underdetermination. since we are comparing nearby models which are equally justified by the training data.
0.50
0.50
Boston
dabetes
Mtaione
0.35
Figure 17: Performance of MC-dropout on the simulated features task from Sec.
We use the four tabular datasets discussed in Sec 5.2 For each dataset, we train neural network with dropout on each weight (as in MC-dropout), to minimize a mean squared error objective. We use a dropout parameter p 0.1, suggested in Gal & Ghahramani (2016). We then
5.2 Y-axis is AUC, X-axis is number of features simulated. This the noise setting.
create an MC-dropout ensemble of size 50: that is, we randomly dropout each weight with probability 0.1, and repeat 50 times. We calculate the training loss for each model in the MC-dropout ensemble. To estimate how "flat" this ensemble (i.c. how loss loss-prese on the training set), we simply subtract the training loss of the full model (no weights dropped out) from each model in the dropout-
17

---

Published as a conference paper at ICLR 2020
Increase in Training Lose
2.5
MC- MC-Dropout
Increase in Training Loss
80
MC MC-Dropout
Local Ensembles
Local Ensembles
2.0
60
1.5
40
1.0
20
0.5
0.0
500
1000
1500
2000 m
2500 3000 3500
0
500
1000
1500 2000 2500 3000 3500 m
(a) Boston
(b) Diabetes
0.8
0.30
Increase in Training Loss
MC- MC-Dropout
Increase Training Loss
MC-Dropout
Local Ensembles
0.25
Local Ensembles
0.6
0.5
0.20
0.15
0.3
0.10
0.2
0.05
0.1
0.00
500
1000 1500 2000 2500 3000 3500
500 1000 1500 2000 2500 3000 3500
(c) Abalone
(d) WineQuality
Figure 18: Comparison of loss-preservingness of ensembles found by MC-Dropout vs Local En- sembles methods. X-axis shows different values for m (number of dimensions projected out), with m dimensions in the ensemble subspace. Y-axis shows expected increase in training loss across the ensemble, as compared to the non-perturbed model (no weights dropped out/perturbation 0). Gaussian perturbation is scaled to have the same expected size as dropout perturbations for that task and m-value. Standard deviations shown across 20 random initializations. For each initialization, 50 perturbations were performed. Dotted red horizontal line shows average training loss increase for an MC-dropout ensemble (20 models, 50 perturbations each).
ensemble, and average these differences. We repeat this process 20 times, changing only the random seeds.
We can now estimate a similar quantity (average increase training loss across the ensemble) for our method. For each dataset, we train neural network with the same architecture in the dropout experiment. We then choose standard deviation equal to the average magnitude of the perturbations created through MC-dropout on the corresponding task. This is to ensure fair comparison - for each method, we are using perturbations on the same scale. We run the Lanezos iteration to convergence. We then compute the ensemble subspace for each Lanczos iteration (recall that - m is the dimensionality of the ensemble subspace, and m is the number of dimensions projected out). For each ensemble subspace of dimensionality p m, sample Gaussian noise from N(O, - project it into the ensemble subspace, and add it to the model's 's trained parameters.
This ensures that when projected into the ensemble subspace, the Gaussian perturbation has the same expected magnitude as the dropout perturbations for the corresponding task. Using these projected Gaussian perturbations,
create an ensemble of size 50 for each m, and calculate the training loss for each model in the local ensemble. We subtract the training loss of the original model (perturbation 0) from each model in the local ensemble, and average across the ensemble. We repeat this process 20 times, changing only the random seeds.
show the results of this comparison for each of the four datasets. The decreasing curve shows that as we project out more cigenvectors (increasing m)) the resulting ensemble subspace becomes flatter, i.e. perturbations in this subspace increase training loss by smaller amount. The
In Figure 18
18

---

Published as a conference paper at ICLR 2020
horizontal dotted line is the average training loss increase for model in an MC-dropout ensemble. We see that this horizontal line well above the descending curve representing local ensembles. Recall that we scaled the perturbations to be the same size for both MC-Dropout and all value of m for local ensembles. This means that, for perturbations m, the ensembles found by method are much flatter than those found by MC-dropout. While MC-dropout ensembles may be useful for other tasks, this shows that our method is more directly measuring underdetermination in the model. Additionally, we run the MC dropout method on the simulated features task from Sec. 5.2] We find (see Fig. [7) that it is unable to match the performance of our method, reinforcing the point that it is concerned mostly with other types of uncertainty.
the same magnitude, for most values of
19


Reviews:
[{"review_id": "S1gq0LRRKH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focusses on underdetermination as being key to extrapolation. In the case of pretrained models, the model extrapolates on a test input if the prediction at this input is underdetermined or multiple predictions are equally consistent with models characterized by similar architecture and loss functions.\nThe underdetermination in the case of over-parameterized model classes as in deep neural networks is used here as a way of detecting extrapolation.\n\nThe authors define an extrapolation score for trained models on unlabelled test point by measuring variance of predictions across an ensemble selected from local perturbations on trained model parameters that fit the training data well or having similar training loss.\n\nThe score approximates the variance of predictions by estimating the norm of the component of the test point’s gradient that aligns well with the low curvature directions of the Hessian, thus providing a tractable quantity in quantifying uncertainty in predictions. The motivation is that if the models have been trained to a local minimum or saddle point, then parameter perturbations in flat directions (small eigenvalues of the Hessian) do not change the training loss substantially.  These models with small perturbations on the flat regions then form the local ensemble for measuring the extrapolation and predicting on out of distribution samples, spurious correlated samples and for  active learning on uncertain data.\n\nThe authors prove that the extrapolation score is proportional to the standard deviations of predictions across a model ensemble with similar training loss. The math in the derivation checks out.\n\n \n\nOne of the novel contributions of the paper is in using computationally cheap post-hoc local ensembles over fully trained ensembles in the baselines that require complicated training procedure. The other key differentiation over baselines is their method leverages the ill conditioned Hessian where  the baselines struggle in requiring an inverse of that ill conditioned Hessian.\n\nThe limitations of their method is in the determination of sufficiently small eigenvalues from the ensemble subspace. Further, the sensitivity of the small set of eigenvalues towards overestimating the prediction’s sensitivity to loss preserving perturbations and being less sensitive to some other under-constrained directions.\nBelow are the potential places where more clarity will help:\nIt would have been good to see a way of measuring the sensitivity in the set of small eigenvalues determination. I urge the authors to think of a way  of quantifying this sensitivity if possible especially since the model class is low dimensional.\n\nIn the experimental section with label, class prediction task, how correlated are the confounders Eyeglasses and Hat? What happens if the models are allowed to train for a longer; does the inconsistency in the behavior of AUC over more eigenvalues change?\n\nIn section 5.4, I think a comparison with the Resampling Under Uncertainty baselines is imperative. \nThere is a typo in E.4 label Attribute->Attractive.\n\nThere is lack of clarity in how similar the models are during training. Although, the ensemble is used post-hoc, its unclear if the models during training differ in initialization only? \n\nWhat are the implications of the method in the case of finetuning models especially is the training data available for fine-tuning is low. \n\nFurther, is there any notion of how the method scales with increasing depth in the neural network models? A comparison with larger test set and models trained on deeper architecture such as ResNet and the like will be interesting to see.\n\nWith noisier data and the inconsistencies in the expected behavior of the method, is there a way of quantifying the amount of noise and the extrapolation? I do see the empirical experiments demonstrating this but some more insight into this is perhaps important.\n\nOverall, its an extremely well written paper with great clarity. The method described by the authors is well differentiated from the baselines in making the clever use of the projection of the ill conditioned Hessian on the low curvature directions of the test point’s gradient. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "ByeyHIaqtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary of contribution\n- The paper provides a novel fast and simple approximation of second-order local parameter sensitivity of neural networks, to estimate a form of uncertainty wrt to a test sample, which is further used and tested as a novelty detector. \n- The method analyzes the most significant eigenvector/eigenvalues of the Hessian (of training loss), and use the compliment of their span to get directions of local perturbations to network parameters that affect training loss little (\"ensemble subspace\"). The novelty score is then based on how much the prediction is influenced by these perturbations.\n- The idea of estimating \"ensemble subspace\" is interesting and computationally effective. Compared to other recent methods that also use second-order gradients for uncertainty, this paper is more generally applicable, and can be faster at test time. The paper demonstrates good performance on both simulated data and real data (CelebA faces with CNN, etc.).\n\n# Decision TL;DR\nI am giving a weak reject. The paper is strong in its idea, formulation, and theory, but is too similar to recent related works which this paper is reluctant to compare to (either in theory, efficiency, or performance). Since the contribution of the paper lies in the efficient approximation of local ensemble methods, readers cannot gauge how beneficial the contribution is compared to other approximations of ensembles.\n\n\n# Pros\n- Novel way to estimate a local neighborhood that affects training loss little (Note: not an expert in this line of research, not sure if it is completely novel) by estimating significant eigenvectors and using their compliment space.\n- The paper is well-written, and relatively easy to understand, despite a few hard-to-follow spots\n- Widely applicable post-hoc to any trained neural networks, and potentially faster training than ensembles / Bayesian approximated ensembles\n- (Theoretical) stability compared to full Hessian inversion\n\n# Cons\nMotivation wrt other papers unclear, and a lack of comparison.\n- Two of the cited papers (Gal & Ghahramani, 2016; Blundell et al., 2015) both work on local ensembles. The former uses MC dropout, the latter estimates a diagonal covariance of a Gaussian distribution of network parameters. These methods are not mentioned in the motivation or related work, which makes it hard to say this paper is well-placed in the literature.\n- The reason that these methods are not compared to is insufficient. The paper only argues that they are not \"post-hoc\" methods. It is very unclear why in any circumstance (or use case) a post-hoc estimation of local ensemble must be (or is preferred to be) used, rather than having network parameters and local neighborhood jointly estimated. If it is for efficiency reasons, the paper does not provide any experimental comparison of the efficiency. Also, it is hard to argue that the \"post-hoc\" nature of this paper makes it so different from the two prior work. For the first prior work, the only time it is not post-hoc is when the original network does not have any dropout layer, and that circumstance is not very common. For the second prior work, one can easily make it post-hoc by training the network first, and estimate the diagonal covariance post-hoc using their loss. \n- The advantage of this paper is that it is more efficient and stable than alternatives, but only  the full hessian inversion is discussed. In particular, it may be necessary to discuss this paper's efficiency against MC-dropout (Gal & Ghahramani, 2016). This method can be done in mini-batches, while the proposed method has to run forward and back-propagation separately for each sample to get gθ*(x'), and it is unclear how well that scales.\n\nEfficiency analysis lacking\n- As discussed above, the proposed method seems to need to back-prop for each test sample separately without using a batch. How much this affects test efficiency is unclear.\n\nExperimental comparison with similar methods missing.\n- The paper would benefit from comparing to the two cited papers (among which MC-dropout is so easy to implement) as well as a full hessian estimation (for toy datasets at least).\n- The paper poses itself as an efficient alternative, so it would be essential to gauge experimentally how fast each method is.\n\nOthers. (not crucial issues) \n- The performance of the paper's main method (LE w/ predictions) underperforms in Table 2, and a variant had to be proposed to make up for the performance drop. This suggests instability of the proposed method wrt new datasets.\n- The claim in contribution \"We identify underdetermination as a key factor in the unreliability of predictions\" is not verified.\n- Inability to scale up to large networks with larger m needed, compared to real ensemble methods or Bayesian networks with Gaussian distributions.\n\n\n# Room for improvement (decreasing order of importance)\n- Improve placement in the literature by discussing when this paper is more useful than prior work (Gal & Ghahramani, 2016; Blundell et al., 2015).\n- Detailed theoretical or experimental analysis of efficiency against alternative approximations of ensembles.\n- Performance comparison to ensemble and local Bayesian methods.\n\n# Editorial issues\n- Figure 3(c) x axis meaning unclear\n- Figure 4 not mentioned in text, and unclear which experiment this refers to \n- Table 2 experiment's loss gradient version is not explained.\n\n\n#############################################################\nPOST REBUTTAL\n#############################################################\n\nTL;DR: The rebuttal addresses some but not all of my concerns. The MC-dropout comparison especially shows the difference between some approximate ensemble methods and ensembles that specifically changes the loss little (this paper). In the end, it is a good paper in terms of theory, although the experiments is lacking in crucial places (no comparison with many existing papers that do attempt to invert hessian) and lack of analysis of claimed efficiency, which I can only hope don’t turn out to be a big deal. I am increasing the score, to marginally above borderline, but please consider the following feedback for the camera-ready version.\n\n> MC Dropout and Bayes by Backprop effectively measure different types of uncertainty from what our method targets, and so they should not be considered competing methods.\nI disagree; by the same logic we can never compare SVM with random forest because they are so different.\n\n> However, as we discussed in 4.1, small eigenvalues make turning this representation into a proper posterior distribution difficult, because these eigenvalues need to be inverted to obtain a covariance matrix, but these inverted eigenvalues can be numerically infinite.\nI agree; but the main difference is the full hessian is hard to implement while there are papers that have implemented the diagonal hessian. It would benefit the paper to compare to that and prove the issue of instability on top of arguing theoretically.\n\n> First, we clarify that our method can in fact be performed using mini-batches of test points (both the forward and backward passes).\nAs far as I know, with mini-batches, the gradients averaged over all samples in the mini-batch are computed for each network parameter. But this method needs the gradient wrt each sample separately.\nIf the authors have some implementation trick that allows computing network parameters’ gradients wrt each sample in the mini-batch separately, please include in the implementation details. Otherwise, please mention this drawback in any “fast” claim.\n\nI will change the score upwards due to the informative rebuttal. Please include much of the discussion in the paper or appendix.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "BJxhlmtn9H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents local ensembles, a method for detecting underdetermination when extrapolating to test points. The authors define an extrapolation score which is used to estimate the standard deviation of predictions at test points. The extrapolation score is chosen to represent the variability in predictions that would be generated by models with similar training loss. By considering the eigenvectors of the Hessian that are associated with minimum eigenvalues the directions of the loss surface with minimal curvature are found, and perturbations of the parameters in the subspace of minimal curvature correspond to models with similar training loss. \n\nThe authors show that their extrapolation score is proportional to the first order approximation to the change in prediction under a perturbation of the parameters with minimal change in loss. In practice the minimum eigenvalue/eigenvector pairs are computationally challenging to compute for large matrices. For this reason the subspaces with minimal change in loss are computed by finding sets of vectors that are mutually orthogonal to the eigenvectors associated with dominant eigenvalues of the Hessian. \n\nThe method is validated experimentally first through out-of-distribution detection on synthetic data. The authors then test performance by constructing a \"blind spot\" by generating features that are a linear combination of existing features. Data can be generated as either within or out of distribution and the AUC metric can be applied to test model performance. In the final experiment the authors demonstrate the use of local ensembles in active learning. By determining which of the training samples are in the model's blind spots at each iteration and training based on these examples rather than randomly selecting training examples rates of convergence can be increased. \n\nI vote to accept this paper as the proposed local ensemble method builds on a growing body of literature regarding loss-surface inference, providing a new way to connect the shape of the loss surface to extrapolation detection. The theoretical result showing the first order relationship between the standard deviation of extrapolation predictions and perturbations in solutions is a useful insight. \n\nThere are some points that should be addressed for clarity however. Firstly the proof of proposition 1 should be made clearer. This is central to the work of the paper and a more full treatment of the proof here could help illuminate some intuition about the connection to perturbations and variance of predictions. \n\nThe other main point that is not addressed is that in principal we aim to find the subspace associated with minimal eigenvalues, but in practice this is computationally prohibitive. Therefore a space that has a basis that is mutually orthogonal to the dominant eigenvectors is sought (the found subspace), and this could have minimal relation to the subspace that is actually sought (the optimal subspace). Some experimentation showing how the found subspace relates to the optimal subspace would be informative, as well as how sensitive the results are to how much the found and optimal subspaces differ.\n\nSome minor points:\n- Many of the plots lack axis labels, although many are explained in the captions the figure labeling needs to be improved\n- Some explanation about the choice of AUC as a metric would be informative and could help connect to the initial motivation of the method\n- Experiment details should be given in the main body of the paper rather than the appendix; i.e. in section 5.2 it is only explained that a \"neural network\" is trained, the architecture should be specifically given alongside the discussion of the experiment\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an ensembling approach to detect underdetermination for extrapolating to test points. The problem domain is interesting and the approach is simple and useful. While reviewers were positive about the work, they raised several points for improvement. The authors are strongly encouraged to include the discussion here in the final version.
Decision: Accept (Poster)