Title: The Memory-Perturbation Equation: Understanding Model's Sensitivity to Data
Venue: NeurIPS 2023
Accepted: yes
Keywords: model interpretability

model understanding

bayesian learning

robustness

adaptive learning
Abstract: Understanding model’s sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.

The Memory Perturbation Equation: Understanding Model's Sensitivity to Data

Abstract

Understanding model's sensitivity to its training data is crucial but can also be challenging and costly, especially during training. To simplify such issues, we present the Memory-Perturbation Equation (MPE) which relates model's sensitivity to perturbation in its training data. Derived using Bayesian principles, the MPE unifies existing sensitivity measures, generalizes them to a wide-variety of models and algorithms, and unravels useful properties regarding sensitivities. Our empirical results show that sensitivity estimates obtained during training can be used to faithfully predict generalization on unseen test data. The proposed equation is expected to be useful for future research on robust and adaptive learning.

1 Introduction

Understanding model's sensitivity to training data is important to handle issues related to quality, privacy, and security. For example, we can use it to understand (i) the effect of errors and biases in the data; (ii) model's dependence on private information to avoid data leakage; (iii) model's weakness to malicious manipulations. Despite their importance, sensitivity properties of machine learning (ML) models are not well understood in general. Sensitivity is often studied through empirical investigations, but conclusions drawn this way do not always generalize across models or algorithms. Such studies are also costly, sometimes requiring thousands of GPUs [38], which can quickly become infeasible if we need to repeat them every time the model is updated.

A cheaper solution is to use local perturbation methods [21], for instance, influence measures that study sensitivity of trained model to data removal (Fig. 1(a)) [8, 7]. Such methods too fall short of providing a clear understanding of sensitivity properties for generic cases. For instance, influence measures are useful to study trained models but are not suited to analyze training trajectories [14, 54]. Another challenge is in handling non-differentiable loss functions or discrete parameter spaces where a natural choice of perturbation mechanisms may not always be clear [32]. The measures also do not directly reveal the causes of sensitivities for generic ML models and algorithms.

In this paper, we simplify these issues by proposing a new method to unify, generalize, and understand perturbation methods for sensitivity analysis. We present the Memory-Perturbation Equation (MPE) as a unifying equation to understand sensitivity properties of generic ML algorithms. The equation builds upon the Bayesian learning rule (BLR) [28] which unifies many popular algorithms

Figure 1b)

Training without

the i'th example

Start

Iterations

Training on

full dataset

Estimate

Truth

Current

(a) Estimating the effect of an example removal

LOO-CV

Test NLL

(b) Predicting test NLL during training, CIFAR10

from various fields as specific instances of a natural-gradient descent to solve a Bayesian learning problem. The MPE uses natural-gradients to understand sensitivity of all such algorithms. We use the MPE to show several new results regarding sensitivity of generic ML algorithms:

We show that sensitivity to a group of examples can be estimated by simply adding their natural-gradients; see Eq. 6. Larger natural-gradients imply higher sensitivity and just a few such examples can often account for most of the sensitivity. Such examples can be used to characterize the model's memory and memory-perturbation refers to the fact that the model can forget its essential knowledge when those examples are perturbed heavily.

We derive Influence Function [8, 31] as a special case of the MPE when natural-gradients with respect to Gaussian posterior are used. More importantly, we derive new measures that, unlike influence functions, can be applied during training for all algorithms covered under the BLR (such as those used in deep learning and optimization). See Table 1.

Measures derived using Gaussian posteriors share a common property: sensitivity to an example depends on the product of its prediction error and variance (Eq. 12). That is, most sensitive data lies where the model makes the most mistakes and is also least confident. In many cases, such estimates are extremely cheap to compute.

We show that sensitivity of the training data can be used to accurately predict model generalization, even during training (Fig. 1(b)). This agrees with similar studies which also show effectiveness of sensitivity in predicting generalization [22, 12, 19, 4].

2 Understanding a Model's Sensitivity to Its Training Data

Understanding a model's sensitivity to its training data is important but is often done by a costly process of retraining the model multiple times. For example, consider a model with a parameter vector θ ∈ R P trained on data D = {D 1 , D 2 , . . . , D N } by using an algorithm A t that generates a sequence { θ t } for iteration t that converges to a minimizer θ ∗ . Formally, we write

and we use the loss ℓ i ( θ ) for D i and a regularizer R ( θ ) . Because θ t are all functions of D or its subsets, we can analyze their sensitivity by simply 'perturbing' the data. For example, we can remove a subset M⊂D to get a perturbed dataset, denoted by D \M , and retrain the model to get new iterates θ \M t , converging to a minimizer θ \M ∗ . If the deviation θ \M t -θ t is large for most t , we may deem the model to be highly sensitive to the examples in M . This is a simple method for sensitive analysis but requires a costly brute-force retraining [38] which is often infeasible for long training trajectories, big models, and large datasets. More importantly, conclusions drawn from retraining are often empirical and may not hold across models or algorithms.

Acheaper alternative is to use local perturbation methods [21], for instance, influence measures that estimate the sensitivity without retraining (illustrated in Fig. 1(a) by the dashed red arrow). The simplest result of this kind is for linear regression which dates back to the 70s [7]. The method makes use of the stationarity condition to derive deviations in θ ∗ due to small perturbations to data. For linear regression, the deviations can be obtained in closed-form. Consider input-output pairs ( x i , y i ) and the loss ℓ i ( θ ) = 1 2 ( y i -f i ( θ )) 2 for f i ( θ ) = x ⊤ i θ and a regularizer R ( θ ) = δ ∥ θ ∥ 2 / 2 . We can obtain closed-form expressions of the deviation due to the removal of the i 'th example as shown below (a proof is included in App. A),

where we denote H \ i ∗ = H ∗ -x i x ⊤ i defined using the Hessian H ∗ = ∇ 2 L ( θ ∗ ) . We also denote the prediction error of θ ∗ by e i = x ⊤ i θ ∗ -y i , and prediction variance of θ \ i ∗ by v \ i i = x ⊤ i ( H \ i ∗ ) -1 x i .

The expression shows that the influence is bi-linearly related to both prediction error and variance, that is, when examples with high error and variance are removed, the model is expected to change a lot. These ideas are generalized using infinitesimal perturbation [21]. For example, influence functions [8, 32, 31] use a perturbation model θ ϵ i ∗ = arg min θ L ( θ ) -ϵ i ℓ i ( θ ) with a scalar perturbation ϵ i ∈ R . By using a quadratic approximation, we get the following influence function,

This works for a generic differentiable loss function and is closely related to Eq. 2. We can choose other perturbation models, but they often exhibit bi-linear relationships; see App. A for details.

Despite their generality, there remain many open challenges with the local perturbation methods:

Influence functions are valid only at a stationary point θ ∗ where the gradient is assumed to be 0, and extending them to iterates θ t generated by generic algorithmic-steps A t is nontrivial [14]. This is even more important for deep learning where we may never reach such a stationary point, for example, due to stochastic training or early stopping [33, 53].

Applying influence functions to a non-differentiable loss or discrete parameter spaces is difficult. This is because the choice of perturbation model is not always obvious [32].

Finally, despite their generality, these measures do not directly reveal the causes of high influence. Does the bi-linear relationship in Eq. 2 hold more generally? If yes, under what conditions? Answers to such questions are currently unknown.

Studies to fix these issues are rare in ML, rather it is more common to simply use heuristics measures. Many such measures have been proposed in the recent years, for example, those using derivatives with respect to inputs [23, 2, 38], variations of Cook's distance [17], prediction error and/or gradients [3, 51, 42, 40], backtracking training trajectories [16], or simply by retraining [13]. These works, although useful, do not directly address the issues. Many of these measures are derived without any direct connections to perturbation methods. They also appear to be unaware of bi-linear relationships such as those in Eq. 2. Our goal here is to address the issues by unifying and generalizing perturbation methods of sensitivity analysis.

3 The Memory-Perturbation Equation (MPE)

We propose the memory-perturbation equation (MPE) to unify, generalize, and understand sensitivity methods in machine learning. We derive the equation by using a property of conjugate Bayesian models which enables us to derive a closed-form expression for the sensitivity. In a Bayesian setting, data examples can be removed by simply dividing their likelihoods from the posterior [52]. For example, consider a model with prior p 0 = p ( θ ) and likelihood ˜ p j = p ( D j | θ ) , giving rise to a posterior q ∗ = p ( θ |D ) ∝ p 0 ˜ p 1 ˜ p 2 . . . ˜ p N . To remove ˜ p j , say for all j ∈ M ⊂ D , we simply divide q ∗ by those ˜ p j . This is further simplified if we assume conjugate exponential-family form for p 0 and ˜ p j . Then, the division between two distributions is equivalent to a subtraction between their natural parameters. This property yields a closed-form expression for the exact deviation, as stated below.

Theorem 1 Assuming a conjugate exponential-family model, the posterior q \M ∗ (with natural parameter λ \M ∗ ) can be written in terms of q ∗ (with natural parameter λ ∗ ), as shown below:

where all exponential families are defined by using inner-product ⟨ λ , T ( θ ) ⟩ with natural parameters λ and sufficient statistics T ( θ ) . The natural parameter of ˜ p j is denoted by ˜ λ j .

The deviation λ \M ∗ -λ ∗ is obtained by simply adding ˜ λ j for all j ∈ M . Further explanations and examples are given in App. B, along with some elementary facts about exponential families. We use this result to derive an equation that enables us to estimate the sensitivity of generic algorithms.

Our derivation builds on the Bayesian learning rule (BLR) [28] which unifies many algorithms by expressing their iterations as inference in conjugate Bayesian models [26]. This is done by reformulating Eq. 1 in a Bayesian setting to find an exponential-family approximation q ∗ ≈ p ( θ |D ) ∝ e -L ( θ ) . At every iteration t , the BLR updates the natural parameter λ t of an exponential-family q t which can equivalently be expressed as the posterior of a conjugate model (shown on the right),

where ˜ g j ( λ ) = F ( λ ) -1 ∇ λ E q [ ℓ j ( θ )] is the natural gradient with respect to λ defined using the Fisher Information Matrix F ( λ t ) of q t , and ρ > 0 is the learning rate. For simplicity, we denote ℓ 0 ( θ ) = R ( θ ) = -log p 0 , and assume p 0 to be conjugate. The conjugate model on the right uses a prior and likelihood both of which, by construction, belong to the same exponential-family as q t . By choosing an appropriate form for q t and making necessary approximations to ˜ g j , the BLR can recover many popular algorithms as special cases. For instance, using a Gaussian q t , we can recover stochastic gradient descent (SGD), Newton's method, RMSprop, Adam, etc. For such cases, the conjugate model at the right is often a linear model [25]. These details, along with a summary of the BLR, are included in App. C. Our main idea is to study the sensitivity of all the algorithms covered under the BLR by using the conjugate model in Eq. 5.

Let q \M t be the posterior obtained with the BLR but without the data in M . We can estimate its natural parameter λ \M t in a similar fashion as Eq. 4, that is, by dividing q t by the likelihood approximation at the current λ t . This gives us the following estimate of the deviation obtained by simply adding the natural-gradients for all examples in M ,

where ˆ λ \M t is an estimate of the true λ \M t . We call this the memory-perturbation equation (MPE) due to a unique property of the equation: the deviation is estimated by a simple addition and characterized solely by the examples in M . Due to the additive nature of the estimate, examples with larger natural-gradients contribute more to it and so we expect most of the sensitivity to be explained by just a few examples with largest natural gradients. This is similar to the representer theorem where just a few support vectors are sufficient to characterize the decision boundary [29, 47, 10]. Here, such examples can be seen as characterizing the model's memory because perturbing them can make the model forget its essential knowledge. The phrase memory-perturbation signifies this.

The equation can be easily adopted to handle an arbitrary perturbation. For instance, consider perturbation L ( θ ) -∑ j ∈M ϵ j ℓ j ( θ ) . To estimate its effect, we divide q t by the likelihood approximations raised to ϵ i , giving us the following variant,

where we denote all ϵ j in M by ϵ M . Setting ϵ j = 1 in the left reduces to Eq. 6 which corresponds to removal. The example demonstrates how to adopt the MPE to handle arbitrary perturbations.

3.1 Unifying the existing sensitivity measures as special cases of the MPE

The MPE is a unifying equation from which many existing sensitivity measures can be derived as special cases. We will show three such results. The first result shows that, for conjugate models, the MPE recovers the exact deviations given in Thm. 1. Such models include textbook examples [6], such as, mixture models, linear state-space models, and PCA. Below is a formal statement.

Theorem 2 For conjugate exponential-family models, Eq. 4 is obtained as a special case of the MPE in Eq. 6 evaluated at λ ∗ of the exact posterior q ∗ when we set ℓ j ( θ ) = -log ˜ p j and ρ = 1 .

The result holds because, for conjugate models, one-step of the BLR is equivalent to Bayes' rule and therefore ˜ g j ( λ ∗ ) = -˜ λ j (see [24, Sec. 5.1]). A proof is given in App. D along with an illustrative example on the Beta-Bernoulli model. We note that a recent work in [49] also takes inspiration from Bayesian models, but their sensitivity measures lack the property discussed above. See also [15] for a different approach to sensitivity analysis of variational Bayes with a focus on the posterior mean. The result above also justifies setting ρ to 1, a choice we will often resort to.

Our second result is to show that the MPE recovers the influence function by Cook [7].

Theorem 3 For linear regression, Eq. 2 is obtained as a special case of the MPE in Eq. 6 evaluated at λ ∗ of the exact posterior q ∗ = N ( θ | θ ∗ , H -1 ∗ ) .

The proof in App. E relies on two facts: first, the natural parameter is λ ∗ = ( H ∗ θ ∗ , -1 2 H ∗ ) , and second, the natural gradients for a Gaussian q with mean m can be written as follows,

where ˆ g i = E q [ ∇ ℓ i ( θ )] and ˆ H i = E q [ ∇ 2 ℓ i ( θ )] . This is due to [28, Eqs. 10-11], but a proof is given in Eq. 27 of App. C. The theorem then directly follows by plugging ˜ g i ( λ ∗ ) in Eq. 6. This derivation is much shorter than the classical techniques which often require inversion lemmas (see App. A.1). The estimated deviations are exact, which is not a surprise because linear regression is a conjugate Gaussian model. However, it is interesting (and satisfying) that the deviation in θ ∗ naturally emerges from the deviation in λ ∗ .

Our final result is to recover influence functions for deep learning, specifically Eq. 3. To do so, we use a Gaussian posterior approximation q ∗ = N ( θ | θ ∗ , H -1 ∗ ) obtained by using the so-called Laplace's method [34, 50, 37]. The Laplace posterior can be seen a special case of the BLR solution when the natural gradient is approximated with the delta method [28, Table 1]. Remarkably, using the same approximation in the MPE, we recover Eq. 3.

Theorem 4 The influence function in Eq. 3 is obtained as a special case of the MPE in Eq. 7 evaluated at λ ∗ of the posterior q ∗ = N ( θ | θ ∗ , H -1 ∗ ) when we approximate ˜ g i ( λ ) of Eq. 8 with the delta method by substituting E q ∗ [ ∇ ℓ i ( θ )] ≈ ∇ ℓ i ( θ ∗ ) and E q ∗ [ ∇ 2 ℓ i ( θ )] ≈ ∇ 2 ℓ i ( θ ∗ ) .

Aproof is in App. F. We note that Eq. 3 can be justified as a Newton-step over the perturbed data but in the opposite direction [32, 31]. In a similar fashion, Eqs. 6 and 7 can be seen as natural-gradient steps in the opposite direction. Using the natural-gradient descent, as we have shown, can recover a variety of existing perturbation methods as special cases.

3.2 Generalizing the perturbation method to estimate sensitivity during training

Influence measures discussed so far assume that the model is already trained and that the loss is differentiable. We will now present generalizations to obtain new measures that can be applied during training and do not require differentiability of the loss . We will focus on Gaussian q but the derivation can be extended to other posterior forms. The main idea is to specialize Eqs. 6 and 7 to the algorithms covered under the BLR, giving rise to new measures that estimate sensitivity by simply taking a step over the perturbed data but in the opposite direction.

We first discuss sensitivity of an iteration t of the BLR yielding a Gaussian q t = N ( θ | m t , S -1 t ) . The natural parameter is the pair λ t = ( S t m t , -1 2 S t ) . Using Eq. 8 in Eq. 6, we get

Plugging S t from the second equation into the first one, we can recover the following expressions,

For the second equation, we omit the proof but it is similar to App. F, resulting in preconditioning with S t . For computational ease, we will approximate ˆ S \M t ≈ S t even in the first equation. We will also approximate the expectation at a sample θ t ∼ q t or simply at the mean θ t = m t . Ultimately, the suggestion is to use S -1 t ∇ ℓ i ( θ t ) as the sensitivity measure, or variations of it, for example, by using a Monte-Carlo average over multiple samples.

Based on this, a list of algorithms and their corresponding measures is given in Table 1. All of the algorithms can be derived as special instances of the BLR by making specific approximations (see App. C.3). The measures are obtained by applying the exact same approximations to Eq. 10. For example, Newton's method is obtained when m t = θ t , S t = H t -1 , and expectations are approximated by using the delta method at θ t (similarly to Thm. 4). With these, we get

which is the measure shown in the first row of the table. In a similar fashion, we can derive measures for other algorithms that use a slightly different approximations leading to a different preconditioner. The exact strategy to update the preconditioners is given in Eqs. 31 to 34 of App. C.3. For all, the sensitivity measure is simply an update step for the i 'th example but in the opposite direction.

Table 1 shows an interplay between the training algorithm and sensitivity measures. For instance, it suggests that the measure H -1 t -1 ∇ ℓ i ( θ t ) is justifiable for Newton's method but might be inappropriate otherwise. In general, it is more appropriate to use the algorithm's own preconditioner (if they use one). The quality of preconditioner (and therefore the measure) is tied to the quality of the posterior approximation. For example, RMSprop's preconditioner is not a good estimator of the posterior covariance when minibatch size is large [27, Thm. 1], therefore we should not expect it to work well for large minibatches. In contrast, the ON method [28] explicitly builds a good estimate of S t during training and we expect it to give better (and more faithful) sensitivity estimates.

For SGD, our approach suggests using the gradient. This goes well with many existing approaches [40, 42, 51, 3] but also gives a straightforward way to modify them when the training algorithm is changed. For instance, the TracIn approach [42] builds sensitivity estimates during SGD training by tracing ∇ ℓ j ( θ t ) ⊤ ∇ ℓ i ( θ t ) for many examples i and j . When the algorithm is switched, say to the ON method, we simply need to trace ∇ ℓ j ( θ t ) ⊤ S -1 t ∇ ℓ i ( θ t ) . Such a modification is speculated in [42, Sec 3.2] and the MPE provides a way to accomplish exactly that. It is also possible to mix and match algorithms with different measures but caution is required. For example, to use the measure in Eq. 11, say within a first-order method, the algorithm must be modified to build a well-conditioned estimate of the Hessian. This can be tricky and can make the sensitivity measure fragile [5].

Extensions to non-differentiable loss functions and discontinuous parameter spaces is straightforward. For example, when using a Gaussian posterior, the measures in Eq. 10 can be modified to

handle non-differentiable loss function by simply replacing E q t [ ∇ ℓ i ( θ )] with ∇ m E q t [ ℓ i ( θ )] , which is a simple application of the Bonnet theorem [44] (see App. G). The resulting approach is more principled than [32] which uses an ad-hoc smoothing of the non-differentiable loss: the smoothing in our approach is automatically done by using the posterior distribution. Handling of discontinuous parameter spaces follows in a similar fashion. For example, binary variables can be handled by measuring the sensitivity through the parameter of the Bernoulli distribution (see App. D).

3.3 Understanding the causes of high sensitivity estimates for the Gaussian case

The MPE can be used to understand the causes of high sensitivity-estimates. We will demonstrate this for Gaussian q but similar analysis can be done for other distributions. We find that sensitivity measures derived using Gaussian posteriors generally have two causes of high sensitivity.

To see this, consider a loss ℓ i ( θ ) = -log p ( y i | σ ( f i ( θ ))) where p ( y i | µ ) is an exponential-family distribution with expectation parameter µ , f i ( θ ) is the model output for the i 'th example, and σ ( · ) is an activation function, for example, the softmax function. For such loss functions, the gradient takes a simple form: ∇ ℓ i ( θ ) = ∇ f i ( θ )[ σ ( f i ( θ )) -y i ] [6, Eq. 4.124]. Using this, we can approximate the deviations in model outputs by using a first-order Taylor approximation,

where we used θ \ i t -θ t ≈ H -1 t -1 ∇ ℓ i ( θ t ) which is based on the measure in the first row of Table 1. Similarly to Eq. 2, the deviation in the model output is equal to the product of the prediction error and (linearized) prediction variance of f i ( θ t ) [25, 20]. The change in the model output is expected to be high, whenever examples with high prediction error and variance are removed.

We can write many such variants with a similar bi-linear relationship. For example, Eq. 12 can be extended to get deviations in predictions as follows:

Eq. 12 estimates the deviation at one example and at a location θ t , but we could also write them for a group of examples and evaluate them at the mean m t or at any sample θ ∼ q t . For example, to remove a group M of size M , we can write the deviation of the model-output vector f ( θ ) ∈ R M ,

where y is the vector of labels and we used the sensitivity measure in Eq. 10. An example for sparse Gaussian process is in App. H. The measure for SGD in Table 1 can also be used which gives f i ( θ \ i t ) -f i ( θ t ) ≈ ∥∇ f i ( θ ) ∥ 2 e it which is similar to the scores used in [40]. The list in Table 1 suggests that such scores can be improved by using H t or S t , essentially, replacing the gradient norm by an estimate of the prediction variance. Additional benefit can be obtained by further employing samples from q t instead of using a point estimate θ t or m t ; see an example in App. H.

It is also clear that all of the deviations above can be obtained cheaply during training by using already computed quantities. The estimation does not add significant computational overhead and can be used to efficiently predict the generalization performance during training. For example, using Eq. 12, we can approximate the leave-one-out (LOO) cross-validation (CV) error as follows,

The approximation eliminates the need to train N models to perform CV, rather just uses e it and v it which are extremely cheap to compute within algorithms such as ON, RMSprop, and SGD. Leave-group-out (LGO) estimates can also be built, for example, by using Eq. 14, which enables us to understand the effect of leaving out a big chunk of training data, for example, an entire class for classification. The LOO and LGO estimates are closely related to marginal likelihood and sharpness, both of which are useful to predict generalization performance [22, 12, 19]. Estimates similar to Eq. 15 have been proposed previously [43, 4] but none of them do so during training.

Estimated Deviation

Estimated Deviation

Estimated Deviation

Estimated Deviation

True Deviation

Estimated Deviation

Low Sensitivity High Sensitivity

'3'

Estimated Deviation

0.4

0.1

0.3

True Deviation

True Deviation

0.4%

0.1%

0.3%

0.4%

0.2%

0.3%

True Deviation

Low Sensitivity High Sensitivity

'3'

'3'

'7'

'7'

True Deviation

'Bag'

'Bag'

'Shoe'

'Shoe'

Low Sensitivity

High Sensitivity

'Dog'

'Bird'

Low Sensitivity

(a) MLP on MNIST

0.4

0.1

0.3

True Deviation

True Deviation

True Deviation

0.4%

0.1%

0.3%

0.4%

0.2%

0.3%

True Deviation

Low Sensitivity High Sensitivity

'3'

'3'

'7'

'7'

True Deviation

'Bag'

'Bag'

'Shoe'

'Shoe'

Low Sensitivity

High Sensitivity

True Deviation

'Dog'

'Dog'

'Bird'

Low Sensitivity

(b) LeNet on FMNIST

0.4

0.1

0.3

True Deviation

True Deviation

0.4%

0.1%

0.3%

0.4%

0.2%

0.3%

True Deviation

'3'

'7'

True Deviation

'Bag'

'Bag'

'Shoe'

'Shoe'

Low Sensitivity

High Sensitivity

True Deviation

'Dog'

'Dog'

'Bird'

'Bird'

Low Sensitivity

High Sensitivity

(c) CNN on CIFAR-10

1.0

1.0

0.5

0.5

0.5

0

82%

3

3

3

7.

True Deviation

True Deviation

High Sensitivity

High Sensitivity

'Bird'

'Dog'

'Bird'

'7'

4 Experiments

We show experimental results to demonstrate the usefulness of the MPE to understand the sensitivity of deep-learning models. We show the following: (1) we verify that the estimated deviations (sensitivities) for data removal correlate with the truth; (2) we predict the effect of class removal on generalization error; (3) we estimate the cross-validation curve for hyperparameter tuning; (4) we predict generalization during training; and (5) we study evolution of sensitivities during training. All details of the experimental setup are included in App. I and the code is available at https://github.com/team-approx-bayes/memory-perturbation .

Estimated deviations correlate with the truth: Fig. 2 shows a good correlation between the true deviations σ ( f i ( θ \ i ∗ )) -σ ( f i ( θ ∗ )) and their estimates σ ′ ( f i ( θ ∗ )) v i ∗ e i ∗ , as shown in Eq. 13. We show results for three datasets, each using a different architecture but all trained using SGD. To estimate the Hessian H ∗ and compute v i ∗ = ∇ f i ( θ ∗ ) ⊤ H -1 ∗ ∇ f i ( θ ∗ ) , we use a Kronecker-factored (K-FAC) approximation implemented in the laplace [11] and ASDL [39] packages. Each marker represents a data example. The estimate roughly maintains the ranking of examples according to their sensitivity. Below each panel, a histogram of true deviations is included to show that the majority of examples have extremely low sensitivity and most of the large sensitivities are attributed to a small fraction of data. The high-sensitivity examples often include interesting cases (possibly mislabeled or simply ambiguous), some of which are visualized in each panel along with some low-sensitivity examples to show the contrast. High-sensitivity examples characterize the model's memory because perturbing them leads to a large change in the model. Similar trends are observed for removal of groups of examples in Fig. 6 of App. I.2.

Predicting the effect of class removal on generalization: Fig. 3(a) shows that the leave-group-out estimates can be used to faithfully predict the test performance even when a whole class is removed. The x-axis shows the test negative log-likelihood (NLL) on a held-out test set, while the y-axis shows the following leave-one-class-out (LOCO) loss on the set C of a left-out class,

Leave-one-class-out Estimation

(a) Predicting the effect of class removal

Examples

Estimated Deviation

50'th

25'th

10'th

High Sensitivity

Low Sensitivity

High Sensitivity

Low Sensitivity

Class

'Bag'

1'st epoch

(b) Evolution of sensitivities during training

0.1 --

0-

LOO-CV

Test NLL

(a) MLP on MNIST

10 1

10 2

10 3

0.3

0.5

(b) LeNet5 on FMNIST

10 1

10 2

10 3

1.2

1.8

(c) CNN on CIFAR-10

0.2

0.1

一

100

101

102

103

8

The estimate uses an approximation: f i ( θ \C ∗ ) -f i ( θ ∗ ) ≈ ∇ f i ( θ ∗ ) ⊤ H -1 ∗ ∑ j ∈C ∇ ℓ j ( θ ∗ ) ≈ v i ∗ e i ∗ , which is similar to Eq. 12, but uses an additional approximation ∑ j ∈C ∇ ℓ j ( θ ∗ ) ≈ ∇ ℓ i ( θ ∗ ) to reduce the computation due to matrix-vector multiplications (we rely on the same K-FAC approximation used in the previous experiment). Results might improve when this approximation is relaxed. We show results for two models: MLP and LeNet. Each marker corresponds to a specific class whose names are indicated with the text. The dashed lines indicate the general trends, showing a good correlation between the truth and estimate. The classes Shirt , Pullover are the most sensitive, while the classes Bag , Trousers are least sensitive. A similar result for MNIST is in Fig. 11(d) of App. I.3.

Predicting generalization for hyperparameter tuning: We consider the tuning of the parameter δ for the L 2 -regularizer of form δ ∥ θ ∥ 2 / 2 . Fig. 4 shows an almost perfect match between the test NLL and the estimated LOO-CV error of Eq. 15. Additional figures with the test errors visualized on top are included in Fig. 7 of App. I.4 where we again see a close match to the LOO-CV curves.

Predicting generalization during training: As discussed earlier, existing influence measures are not designed to analyze sensitivity during training and care needs to be taken when using ad-hoc strategies. We first show results for our proposed measure in Eq. 10 which gives reliable sensitivity estimates during training. We use the improved-BLR method [35] which estimates the mean m t and a vector preconditioner s t during training. We can derive an estimate for the LOO error at the mean m t following a derivation similar to Eqs. 14 and 15,

0

25

50

75

100

Epochs

0.4

0.8

(a) iBLR & LOO of Eq. 16

(b) SGD & diagonal-GGN-LOO

0

25

50

75

100

Epochs

0.4

0.8

(c) SGD & K-FAC-LOO

The first panel in Fig. 5 shows a good match between the above LOO estimate and test NLL. For comparison, in the next two panels, we show results for SGD training by using two ad-hoc measures obtained by plugging different Hessian approximations in Eq. 11. The first panel approximates H t with a diagonal Generalized Gauss-Newton (GGN) matrix, while the second panel uses a K-FAC approximation. We see that diagonal-GGN-LOO does not work well at all and, while K-FACLOO improves this, it is still not as good as the iBLR result despite using a non-diagonal Hessian approximation. Not to mention, the two measures require an additional pass through the data to compute the Hessian approximation, and also need a careful setting of a damping parameter.

A similar result for iBLR is shown in Fig. 1(b) where we use the larger ResNet-20 on CIFAR10, and more such results are included in Fig. 8 of App. I.5. We also find that both diagonal-GGNLOO or K-FAC-LOO further deteriorate when the model overfits; see Fig. 9. Results for the Adam optimizer are included in Fig. 10, where we again see that using ad hoc measures may not always work. Overall, these results show the difficulty of estimating sensitivity during training and suggest to take caution when using measures that are not naturally suited to analyze the training algorithm.

Evolution of sensitivities during training: Fig. 3(b) shows the evolution of sensitivities of examples as the training progresses. We use the iBLR algorithm and approximate the deviation as σ ( f i ( m \ i t )) -σ ( f i ( m t )) ≈ σ ′ ( f i ( m t )) v it e it where v it and e it are obtained similarly to Eq. 16. The x-axis corresponds to examples sorted from least sensitive to most sensitive examples at convergence. The y-axis shows the histogram of sensitivity estimates. We observe that, as the training progresses, the distribution concentrates around a small fraction of the data. At the top, we visualize a few examples with high and low sensitivity estimates, where the high-sensitivity examples included interesting cases (similarly to Fig. 2). The result suggests that the model concentrates more and more on a small fraction of high-sensitivity examples, and therefore such examples can be used to characterize the model's memory. Additional experiments of this kind are included in Fig. 11 of App. I.6, along with other experiment details.

5 Discussion

We present the memory-perturbation equation by building upon the BLR framework. The equation suggests to take a step in the direction of the natural gradient of the perturbed examples. Using the MPE framework, we unify existing influence measures, generalize them to a wide variety of problems, and unravel useful properties regarding sensitivity. We also show that sensitivity estimation can be done cheaply and use this to predict generalization performance. An interesting avenue for future research is to apply the method to larger models and real-world problems. We also need to understand how our generalization measure compares to other methods, such as those considered in [22]. We would also like to understand the effect of various posterior approximations. Another interesting direction is to apply the method to non-Gaussian cases, for example, to study ensemble methods in deep learning with mixture models.

References

Vincent Adam, Paul Chang, Mohammad Emtiyaz Khan, and Arno Solin. Dual Parameterization of Sparse Variational Gaussian Processes. Advances in Neural Information Processing Systems , 2021. 20

Chirag Agarwal, Daniel D'Souza, and Sara Hooker. Estimating Example Difficulty using Variance of Gradients. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2022. 3

Devansh Arpit, Stanisław Jastrz˛ ebski, Nicolas Ballas, David Krueger, Emmanuel Bengio, Maxinder S Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Bengio, and Simon Lacoste-Julien. A Closer Look at Memorization in Deep Networks. In International Conference on Machine Learning , 2017. 3, 6

Gregor Bachmann, Thomas Hofmann, and Aurélien Lucchi. Generalization Through The Lens of Leave-One-Out Error. In International Conference on Learning Representations , 2022. 2, 7

Samyadeep Basu, Phil Pope, and Soheil Feizi. Influence Functions in Deep Learning Are Fragile. In International Conference on Learning Representations , 2021. 6

Christopher M. Bishop. Pattern Recognition and Machine Learning . Springer, 2006. 5, 7

R Dennis Cook. Detection of Influential Observation in Linear Regression. Technometrics , 1977. 1, 3, 5

R Dennis Cook and Sanford Weisberg. Characterizations of an Empirical Influence Function for Detecting Influential Cases in Regression. Technometrics , 1980. 1, 2, 3

R Dennis Cook and Sanford Weisberg. Residuals and Influence in Regression . Chapman and Hall, 1982. 14

Corinna Cortes and Vladimir Vapnik. Support-Vector Networks. Machine learning , 1995. 4

Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace Redux-Effortless Bayesian Deep Learning. Advances in Neural Information Processing Systems , 2021. 8

Gintare Karolina Dziugaite and Daniel M Roy. Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters Than Training Data. In Proceedings of the Conference on Uncertainty in Artificial Intelligence , 2017. 2, 7

Vitaly Feldman and Chiyuan Zhang. What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation. In Advances in Neural Information Processing Systems , 2020. 3

Wing K Fung and CW Kwan. A Note on Local Influence Based on Normal Curvature. Journal of the Royal Statistical Society: Series B (Statistical Methodology) , 1997. 1, 3

Ryan Giordano, Tamara Broderick, and Michael I Jordan. Covariances, robustness and variational Bayes. Journal of Machine Learning Research , 19(51), 2018. 5

Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data Cleansing for Models Trained with SGD. In Advances in Neural Information Processing Systems , 2019. 3

Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating Informativeness of Samples with Smooth Unique Information. In International Conference on Learning Representations , 2021. 3

James Hensman, Nicolo Fusi, and Neil D Lawrence. Gaussian Processes for Big Data. In Proceedings of the Conference on Uncertainty in Artificial Intelligence , 2013. 20

Alexander Immer, Matthias Bauer, Vincent Fortuin, Gunnar Rätsch, and Mohammad Emtiyaz Khan. Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning. In International Conference on Machine Learning , 2021. 2, 7

Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving Predictions of Bayesian Neural Nets via Local Linearization. International Conference on Artificial Intelligence and Statistics , 2021. 7

Louis A Jaeckel. The Infinitesimal Jackknife. Technical report, Bell Lab., 1972. 1, 3

Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic Generalization Measures and Where To Find Them. In International Conference on Learning Representations , 2020. 2, 7, 10

Angelos Katharopoulos and Francois Fleuret. Not All Samples Are Created Equal: Deep Learning with Importance Sampling. In International Conference on Machine Learning , 2018. 3

Mohammad Emtiyaz Khan. Variational Bayes Made Easy. Fifth Symposium on Advances in Approximate Bayesian Inference , 2023. 5

Mohammad Emtiyaz Khan, Alexander Immer, Ehsan Abedi, and Maciej Korzepa. Approximate Inference Turns Deep Networks into Gaussian Processes. Advances in Neural Information Processing Systems , 2019. 4, 7, 17

Mohammad Emtiyaz Khan and Wu Lin. Conjugate-Computation Variational Inference: Converting Variational Inference in Non-Conjugate Models to Inferences in Conjugate Models. In International Conference on Artificial Intelligence and Statistics , 2017. 4, 15, 20

Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam. In International Conference on Machine Learning , 2018. 6, 17, 18, 25

Mohammad Emtiyaz Khan and Håvard Rue. The Bayesian Learning Rule. Journal of Machine Learning Research , 2023. 1, 4, 5, 6, 16, 17, 18

George S Kimeldorf and Grace Wahba. A Correspondence Between Bayesian Estimation on Stochastic Processes and Smoothing by Splines. The Annals of Mathematical Statistics , 1970. 4

Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In International Conference on Learning Representations , 2015. 6

Pang Wei Koh, Kai-Siang Ang, Hubert Teo, and Percy S Liang. On the Accuracy of Influence Functions for Measuring Group Effects. In Advances in Neural Information Processing Systems , 2019. 2, 3, 5

Pang Wei Koh and Percy Liang. Understanding Black-Box Predictions via Influence Functions. In International Conference on Machine Learning , 2017. 1, 3, 5, 7

Aran Komatsuzaki. One Epoch is All You Need. ArXiv e-Prints , 2019. 3

Pierre-Simon Laplace. Mémoires de Mathématique et de Physique. Tome Sixieme , 1774. 5

Wu Lin, Mark Schmidt, and Mohammad Emtiyaz Khan. Handling the Positive-Definite Constraint in the Bayesian Learning Rule. In International Conference on Machine Learning , 2020. 6, 9, 17, 18

Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. International Conference on Learning Representations , 2019. 23

David JC MacKay. Information Theory, Inference and Learning Algorithms . Cambridge University Press, 2003. 5

Roman Novak, Yasaman Bahri, Daniel A Abolafia, Jeffrey Pennington, and Jascha SohlDickstein. Sensitivity and Generalization in Neural Networks: An Empirical Study. In International Conference on Learning Representations , 2018. 1, 2, 3

Kazuki Osawa, Satoki Ishikawa, Rio Yokota, Shigang Li, and Torsten Hoefler. ASDL: A Unified Interface for Gradient Preconditioning in PyTorch. In NeurIPS Workshop Order up! The Benefits of Higher-Order Optimization in Machine Learning , 2023. 8

Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep Learning on a Data Diet: Finding Important Examples Early in Training. In Advances in Neural Information Processing Systems , 2021. 3, 6, 7

Daryl Pregibon. Logistic Regression Diagnostics. The Annals of Statistics , 1981. 14

Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating Training Data Influence by Tracing Gradient Descent. In Advances in Neural Information Processing Systems , 2020. 3, 6

Kamiar Rahnama Rad and Arian Maleki. A Scalable Estimate of the Out-of-Sample Prediction Error via Approximate Leave-One-Out Cross-Validation. Journal of the Royal Statistical Society Series B: Statistical Methodology , 2020. 7

Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In International Conference on Machine Learning , 2014. 7, 17

Hugh Salimbeni, Stefanos Eleftheriadis, and James Hensman. Natural Gradients in Practice: Non-Conjugate Variational Inference in Gaussian Process Models. In International Conference on Artificial Intelligence and Statistics , 2018. 20

Frank Schneider, Lukas Balles, and Philipp Hennig. DeepOBS: A Deep Learning Optimizer Benchmark Suite. In International Conference on Learning Representations , 2019. 21

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A Generalized Representer Theorem. In International Conference on Computational Learning Theory , 2001. 4

Saurabh Singh and Shankar Krishnan. Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2020. 21

Ryutaro Tanno, Melanie F Pradier, Aditya Nori, and Yingzhen Li. Repairing Neural Networks by Leaving the Right Past Behind. Advances in Neural Information Processing Systems , 2022. 5

Luke Tierney and Joseph B Kadane. Accurate Approximations for Posterior Moments and Marginal Densities. Journal of the American Statistical Association , 1986. 5

Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J. Gordon. An Empirical Study of Example Forgetting during Deep Neural Network Learning. In International Conference on Learning Representations , 2019. 3, 6

Robert Weiss. An Approach to Bayesian Sensitivity Analysis. Journal of the Royal Statistical Society Series B: Statistical Methodology , 1996. 3

Fuzhao Xue, Yao Fu, Wangchunshu Zhou, Zangwei Zheng, and Yang You. To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis. ArXiv e-Prints , 2023. 3

Hongtu Zhu, Joseph G. Ibrahim, Sikyum Lee, and Heping Zhang. Perturbation Selection and Influence Measures in Local Influence Analysis. The Annals of Statistics , 2007. 1

A Influence Function for Linear Regression

We consider N input-output pairs ( x i , y i ) . The feature matrix containing x ⊤ i as rows is denoted by X and the output vector of length N is denoted by y . The loss is ℓ i ( θ ) = 1 2 ( y i -f i ( θ )) 2 for f i ( θ ) = x ⊤ i θ . The regularizer is assumed to be R ( θ ) = δ ∥ θ ∥ 2 / 2 . The minimizer is given by

We define a perturbation model as follows with ϵ i ∈ R :

For ϵ i = 1 , it corresponds to example removal. An arbitrary ϵ i simply weights the example accordingly. The solution has a closed-form expression,

where H ∗ = ∑ N i =1 x i x ⊤ i + δ I P is the Hessian of L ( θ ) . We first derive a closed-form expressions for θ ϵ i ∗ -θ ∗ , and then specialize them for different ϵ i .

A.1 Derivation of the leave-one-out (LOO) deviation

We denote Σ ∗ = H -1 ∗ and use the Sherman-Morrison formula to write

In line 3 we substitute v i = x ⊤ i Σ ∗ x i and θ ∗ = Σ ∗ X ⊤ y and in the last step we use e i = x ⊤ i θ ∗ -y i .

We define e \ i i = e i / (1 -v i ) which is the prediction error of θ \ i ∗ ,

Therefore, we get the following expressions for the deviation,

These expressions can be written in the form of Eq. 2 by left-multiplying with Σ -1 ∗ = H \ i ∗ + x i x ⊤ i ,

A.2 Derivation of the infinitesimal perturbation approach

We differentiate θ ϵ i ∗ in Eq. 19 to get yielding the following expressions:

The second equation in Eq. 22 follows from the chain rule. We get a bi-linear relationship of the influence measure with respect to v i and prediction error e i . It is also possible to evaluate Eq. 21 at ϵ i = 1 representing an infinitesimal perturbation about the LOO estimate, ∂ θ ϵ i ∗ /∂ϵ i | ϵ i =1 = Σ \ i ∗ x i e \ i i From this and Eq. 22, we can interpret Eq. 2 as the average derivative over the interval ϵ i ∈ [0 , 1] [9] or the derivative evaluated at some 0 < ϵ i < 1 (via an application of the mean value theorem) [41].

B Conjugate Exponential-Family Models

Exponential-family distributions take the following form:

where λ ∈ Ω are the natural (or canonical) parameter for which the cumulant (or log partition) function A ( λ ) is finite, strictly convex and differentiable over Ω . The quantity T ( θ ) is the sufficient statistics, ⟨· , ·⟩ is an inner product and h ( θ ) is some function. A popular example is the Gaussian distribution, which can be rearranged to take an exponential-family form written in terms of the precision matrix S = Σ -1 ,

From this, we can read-off the quantities needed to define an exponential-form,

Both the natural parameter and sufficient statistics consist of two elements. The inner-product for the first elements is simply a transpose to get the θ ⊤ Sm term, while for the second element it is a trace which gives -Tr ( θθ ⊤ S / 2) = -1 2 θ ⊤ S θ .

Conjugate Exponential-Family Models are those where both the likelihoods and prior can be expressed in terms of the same form of exponential-family distribution with respect to θ . For instance, in linear regression, both the likelihood and prior take a Gaussian form with respect to θ ,

Note that ˜ p i is a distribution over y i but it can also be expressed in an (unnormalized) Gaussian form with respect to θ . The sufficient statistics of both ˜ p i and p 0 correspond to those of a Gaussian distribution. Therefore, the posterior is also a Gaussian,

The third line follows because θ ∗ = H -1 ∗ X ⊤ y , as shown in Eq. 17.

These computations can be written as conjugate-computations [26] where we simply add the natural parameters,

In the same fashion, to remove the contributions of certain likelihoods, we can simply subtract their natural parameters from λ ∗ . These are the calculations which give rise to the following equation:

C The Bayesian Learning Rule

The Bayesian learning rule (BLR) aims to find a posterior approximation q ( θ ) ≈ p ( θ |D ) ∝ e -L ( θ ) . Often, one considers a regular, minimal exponential-family q ∈ Q , for example, the class of Gaussian distributions. The approximation is found by optimizing a generalized Bayesian objective,

where H ( q ) = E q [ -log q ( θ )] is the entropy of q and Q is the class of exponential family approximation. The objective is equivalent to the Evidence Lower Bound (ELBO) when L ( θ ) corresponds to the negative log-joint probability of a Bayesian model; see [28, Sec 1.2].

The BLR uses natural-gradient descent to find q ∗ , where each iteration t takes the following form,

where ρ > 0 is the learning rate. The gradient is computed with respect to λ (through q ), and we scale the gradient by the Fisher Information Matrix (FIM) defined as follows,

The second equality shows that, for exponential-family distribution, the above FIM is also the second derivative of the log-partition function A ( λ ) .

C.1 The BLR of Eq. 5

The BLR in Eq. 5 is obtained by simplifying the natural-gradient using the following identity,

where µ is the expectation parameter. The identity works because of the minimality of the exponential-family which ensures that there is a one-to-one mapping between λ and µ , and also that the FIM is invertible. Using this, we can show that the natural gradient of H ( q ) is simply equal to -λ ; see [28, App. B]. Defining ℓ 0 ( θ ) = R ( θ ) , we get the version of the BLR shown in Eq. 5,

C.2 The conjugate-model form of the BLR given in Eq. 5

To express the update in terms of the posterior of a conjugate model, we simply take the inner product with T ( θ ) and take the exponential to write the update as

The simplification of the second term on the left to p 0 happens when p 0 is a conjugate prior, that is, p 0 ∝ exp( ⟨ λ 0 , T ( θ ) ⟩ ) for some λ 0 (see an example in App. B where we show that L 2 regularizer leads to such a choice). In such cases, we can simplify,

Using this in Eq. 26, we recover the conjugate model given in Eq. 5.

C.3 BLR for a Gaussian q and the Variational Online Newton (VON) algorithm

By choosing an appropriate form for q t and making necessary approximations to ˜ g j , the BLR can recover many popular algorithms as special cases. We will now give a few examples for the case of a Gaussian q t = N ( θ | m t , Σ t ) which enables derivation of various first and second-order optimization algorithms, such as, Newton's method, RMSprop, Adam, and SGD.

As shown in Eq. 23, for a Gausian N ( θ | m , Σ ) , the natural parameter and sufficient statistics are shown below, along with the expectation parameters µ = E q [ T ( θ )] .

Using these, we can write the natural gradients as gradients with respect to µ , , and then using chain-rule to express them as gradients with respect to m and Σ ,

where in the last equation we define two quantities written in terms of ∇ ℓ j ( θ ) and ∇ 2 ℓ j ( θ ) by using Price's and Bonnet's theorem [44],

Plugging these into the BLR update gives us the following update,

where ˆ g j,t -1 and ˆ H j,t -1 are quantities similar to before but now evaluated at the q t -1 . The conjugate model can be written as follows,

The prior above is Gaussian and defined using q t -1 and p 0 . The model uses likelihoods that are Gaussian distribution with information vector ˆ i j,t -1 = ρ ( ˆ H j,t -1 m t -1 -ˆ g j,t -1 ) and information matrix ˆ I j,t -1 = ρ ˆ H j,t -1 . The likelihood is allowed to be an improper distribution, meaning that its integral is not one. This is not a problem as long as S t remains positive definite. A valid S t can be ensured by either using a Generalized Gauss-Newton approximation to the Hessian [27] or by using the improved BLR of [35]. The former strategy is used in [25] to express BLR iterations as linear models and Gaussian processes. Ultimately, we want to ensure that perturbation in the approximate likelihoods in q t yields a valid posterior and, as long as this is the case, the conjugate model can be used safely. For instance, in Thm. 4, this issue poses no problem at all.

The BLR update can be rearranged and written in a Newton-like form show below,

This is called the Variational Online Newton (VON) algorithm. A full derivation is in [27] with details on many of its variants in [28]. The simplest variant is the Online Newton (ON) algorithm, where we use the delta method,

Then denoting m t = θ t , we get the following ON update,

To reduce the cost, we can use a diagonal approximation S t = diag ( s t ) where s t is a scale vector. Additionally, we can use minibatching to estimate the gradient and hessian (denoted by ˆ ∇ and ˆ ∇ 2 ),

where · indicates element-wise product two vectors and diag ( · ) extracts the diagonal of a matrix.

Several optimization algorithms can be obtained as special cases from the above variants. For example, to get Newton's method, we set ρ = 1 in ON to get

RMSprop and Adam can be derived in a similar fashion [28].

In our experiments, we use the improved BLR or iBLR optimizer [35]. We use it to implement an improved version of VON [27, Eqs. 7-8] which ensures that the covariance is always positive-definite, even when the Hessian estimates are not. We use diagonal approximation S t = diag ( σ 2 ) -1 , momentum and minibatching as proposed in [27, 35]. For learning rate α t > 0 , momentum β 1 , β 2 ∈ [0 , 1) the iterations are written as follows:

Here, δ > 0 is the L 2 -regularization parameter and ̂ g t -1 = 1 | B | ∑ i ∈ B E q t -1 ( θ ) [ ∇ ℓ i ( θ )] , ̂ h t -1 = 1 | B | ∑ i ∈ B E q t -1 ( θ ) [ ∇ ℓ i ( θ )( θ -m t -1 ) / σ 2 t -1 ] denote Monte-Carlo approximations of the expected stochastic gradient and diagonal Hessian under q t -1 ( θ ) = N ( θ | m t -1 , diag ( σ 2 t -1 )) and minibatch B . As suggested in [27, 35], we used the reparametrization trick to estimate the diagonal Hessian via gradients only. In practice, we approximate the expectations using a single random sample. We expect multiple samples to further improve the results.

D Proof of Thm. 2 and the Beta-Bernoulli Model

From Eq. 27, it directly follows that

Using this in Eq. 6, we get the deviation given in Eq. 4.

Wewill now show an example on Beta-Bernoulli model, which is a conjugate model. We assume the model to be p ( D , θ ) ∝ p ( θ ) ∏ i p ( y i | θ ) where the prior is p ( θ ) = Beta ( θ | α 0 , β 0 ) and likelihoods are p ( y i | θ ) = Ber ( y i | θ ) with D i = y i . This is a conjugate model and the posterior is Beta distribution, that is, it takes the same form as the prior. An expression is given below,

The posterior for the perturbed dataset D \ i is also available in closed-form:

̸

̸

Therefore the deviations in the posterior parameters can be simply obtained as follows:

This result can also be straightforwardly obtained using the MPE. For the Beta distribution q λ ( θ ) = Beta ( θ | α, β ) , we have λ = ( α -1 , β -1) , therefore λ \ i ∗ -λ ∗ = ( α \ i ∗ -α ∗ , β \ i ∗ -β ∗ ) . For Beta distribution, T ( θ ) = (log θ, log(1 -θ )) and writing the likelihood in an exponential form, we get

therefore ˜ λ i = ( y i , y i -1) . Setting λ \ i ∗ -λ ∗ = -˜ λ i , we recover the result given in Eq. 35.

E Proof of Thm. 3

For linear regression, we have

Using these in Eq. 8, we get,

The natural parameter is λ ∗ = ( H ∗ θ ∗ , -1 2 H ∗ ) . In a similar way, we can define q \ i ∗ and its natural parameter. Using these, we can write Eq. 6 as

Substituting the second equation into the first one, we get the first equation below,

The last equality is exactly Eq. 2. Since linear regression is a conjugate model, an alternate derivation would be to directly use the parameterization ˜ λ j of ˜ p i (derived in App. B) and plug it in Thm. 2.

F Proof of Thm. 4

For simplicity, we denote

with ˆ λ ϵ i ∗ as defined in Eq. 7 in the main text. For Gaussian distributions, the natural parameter comes in a pair ˆ λ ϵ i ∗ = ( H ϵ i ∗ θ ϵ i ∗ , -1 2 H ϵ i ∗ ) . Its derivative with respect to ϵ i at ϵ i = 0 can be written as the following by using the chain rule:

Here, we use the fact that, as ϵ i → 0 , we have ( θ ϵ i ∗ , H ϵ i ∗ ) → ( θ ∗ , H ∗ ) and also assumed that the limit of the product is equal to the product of the individual limits. Next, we need the expression for the natural gradient, for which we will use Eq. 8 but approximate the expectation by using the delta approximation E q ∗ [ g ( θ )] ≈ g ( θ ∗ ) for any function g , as shown below to define:

The claim is that if we set the perturbed ∂ ˆ λ ϵ i =0 ∗ = ˆ g i ( λ ∗ ) we recover Eq. 3, that is, we set

Plugging the second equation into the first, the second term cancels and we recover Eq. 3.

G Extension to Non-Differentiable Loss function

For non-differentiable cases, we can use Eq. 28 to rewrite the BLR of Eq. 29 as

where Σ = S -1 . Essentially, we take derivative outside the expectation instead of inside which is valid because the expectation of a non-differentiable function is still differentiable (under some regularity conditions). The same technique can be applied to Eq. 8 to get

and proceeding in the same fashion we can write: ˆ m \ i t -m t = ( ˆ S \ i t ) -1 ∇ m E q t [ ℓ i ( θ )] . This is the extension of Eq. 10 to non-differentiable loss functions.

H Sensitivity Measures for Sparse Variational Gaussian Processes

Sparse variational GP (SVGP) methods optimize the following variational objective to find a Gaussian posterior approximation q ( u ) over function values u := ( f ( z 1 ) , f ( z 2 ) , . . . , f ( z M )) where Z := ( z 1 , z 2 , . . . , z M ) is the set of inducing inputs with M ≪ N :

where p ( u ) := N ( u | 0 , K uu ) is the prior with K uu as the covariance function κ ( · , · ′ ) evaluated at Z , q ( f i ) = N ( f i | a ⊤ i m , a ⊤ i Σa i + σ 2 i ) is the posterior marginal of f i = f ( x i ) with a i := K -1 uu k u i and σ 2 i := κ ii -a ⊤ i K uu a i as the noise variance of f i conditioned on u . The objective is also used to optimize hyperparameters ϕ and inducing input set Z .

We can optimize the objective using the BLR for which the resulting update is identical to the variational online-newton (VON) algorithm. We first write the natural gradients,

where we define

We define A to be a matrix with a ⊤ i as rows, and e t , β t to be vectors of e it , β it . Using these in the VON update, we simplify as follows:

For Gaussian likelihood, the updates in Eqs. 39 and 40 coincide with the method of [18], and for non-Gaussian likelihood they are similar to the natural-gradient method by [45], but we use the specific parameterization of [26]. An alternate update rule in terms of site parameters is given by [1] (see Eqs. 22-24).

We are now ready to write the sensitivity measure essentially substituting the gradient in Eq. 10),

We can also see the bi-linear relationship by considering the deviation in the mean of the posterior marginal f i ( m ) := a ⊤ i m ,

where v it = a ⊤ i Σ t a i is the marginal variance of f i .

I Experimental Details

I.1 Neural network architectures

Below, we describe different neural networks used in our experiments,

MLP (500, 300): This is a multilayer perceptron (MLP) with two hidden layers of 500 and 300 neurons and a parameter count of around 546 000 (using hyperbolic-tangent activations).

MLP (32, 16): This is also an MLP with two hidden layers of 32 and 16 neurons, which accounts for around 26 000 parameters (also using hyperbolic tangent activations).

LeNet5: This is a standard convolutional neural network (CNN) architecture with three convolution layers followed by two fully-connected layers, corresponding to around 62 000 parameters.

CNN: This network, taken from the DeepOBS suite [46], consists of three convolution layers followed by three fully-connected layers with a parameter count of 895 000 .

ResNet-20: This network has around 274 000 parameters. We use filter response normalization (FRN) [48] as an alternative to batch normalization.

MLP for USPS: For the experiment on binary USPS in Fig. 6(a), we use an MLP with three hidden layers of 30 neurons each and a total of around 10 000 parameters.

I.2 Details of 'Do estimated deviations correlate with the truth?'

In Fig. 2, we train neural network classifiers with a cross-entropy loss to obtain θ ∗ . Due to the computational demand of per-example retraining, the removed examples are randomly subsampled from the training set. We show results over 1000 examples for MNIST and FMNIST and 100 examples for CIFAR10. In the multiclass setting, the expression yields a per-class sensitivity value. We obtain a scalar value for each example by summing over the absolute values of the per-class sensitivities. For training both the original model θ ∗ and the perturbed models θ \ i ∗ , we use SGD with a momentum parameter of 0 . 9 and a cosine learning-rate scheduler. To obtain θ \ i ∗ , we retrain a model that is warmstarted at θ ∗ . Other details regarding the training setup are given in Table 2. For all models, we do not use data augmentation during training. The resulting θ ∗ for MNIST, FMNIST, and CIFAR10 have training accuracies of 99 . 9% , 95 . 0% , and 99 . 9% , respectively. The test accuracies for these models are 98 . 4% , 91 . 2% and 76 . 7% .

Additional group removal experiments: We also study how the deviation for removing a group of examples in a set M can be estimated using a variation of Eq. 14 for the deviation in predictions at convergence. Denoting the vector of f i ( θ ) for i ∈ M by f M ( θ ) , we get

where Λ ( θ ∗ ) is a diagonal matrix containing all σ ′ ( f i ∗ ) , V M ( θ ∗ ) = ∇ f M ( θ ∗ ) S -1 ∗ ∇ f M ( θ ∗ ) ⊤ is the prediction covariance of size M × M where M is the number of examples in M , and e M ( θ ∗ ) is the vector of prediction errors. The last approximation above is done to avoid building the covariance, where we ignore the off-diagonal entries of V M ( θ ∗ ) .

In Fig. 6(a) we consider a binary USPS dataset consisting of the classes for the digits 3 and 5. Using |M| = 16, we show the first and second approximations in Eq. 43 both correlate well with the truth obtained by removing a group and retraining the model. In Fig. 6(b) we do the same on MNIST with |M| = 64, where we see similar trends. For the experiment on binary USPS in Fig. 6(a), we train a MLP with three hidden layers with 30 neurons each. The original model θ ∗ is trained for 500 epochs with a learning-rate of 10 -3 , a batch size of 32 and a L 2 -regularization parameter δ = 5 . It has 100% training accuracy and 94 . 8% test accuracy. For the leave-group-out retraining to obtain θ \M ∗ , we initialize the model at θ ∗ , use a learning-rate of 10 -3 and train for 1000 epochs. For the MNIST result in Fig. 6(b) we use the MLP (500, 300) model with the same hyperparameters as for

0.0

0.3

0.6

0.9

1.2

True Deviation

0.0

0.3

0.6

0.9

1.2

Estimated Deviation

group

individual

(a) MLP on USPS-3vs5,

|M|

= 16

0.0

0.4

0.8

1.2

1.6

True Deviation

0.0

0.3

0.6

0.9

individual

(b) MLP on MNIST,

|M|

= 64

I.3 Details of 'Predicting the effect of class removal on generalization'

For the FMNIST experiment in Fig. 3(a), we use the MLP (32, 16) and LeNet5 models. For the MNIST experiment in Fig. 11(d), we use the MLP (500, 300) and LeNet5 models. The hyperparameters are given in Table 3. The MLP on MNIST has a training accuracy of 99 . 9% and a test accuracy of 98 . 4% . When using LeNet5, the training and test accuracies are 99 . 2% and 99 . 1% . On FMNIST, the LeNet5 has an accuracy of 95 . 0% on the training set, and an accuracy of 91 . 2% on the test set. On the same dataset, the MLP has a training accuracy of 89 . 9% and a test accuracy of 86 . 2% . For all models, we use a regularization parameter of 100 and a batch size of 256. The leave-one-class-out training is run for 1000 epochs and the rest of the training setup is same as the previous experiment.

I.4 Details of 'Estimating the leave-one-out cross-validation curves for hyperparameter tuning'

The details of the training setup are in Table 2. Fig. 7 is the same as Fig. 4 but additionally shows the test errors. For visualization purposes, each plot uses a moving average of the plotted lines with a smoothing window. Other training details are similar to previous experiments. All models are

10 0

10 1

10 2

10 3

0.1

0.2

2%

3%

4%

Test NLL

LOO-CV

Test error

(a) MNIST, MLP

10 1

10 2

10 3

0.3

0.5

10%

12%

15%

(b) FMNIST, LeNet5

10 1

10 2

10 3

1.2

1.8

30%

40%

50%

60%

Test error

(c) CIFAR10, CNN

trained from scratch where we use Adam for FMNIST, AdamW [36] for CIFAR10, and SGD with a momentum parameter of 0.9 for MNIST. We use a cosine learning-rate scheduler to anneal the learning-rate. The other hyperparameters are similar to the settings of the models trained on all data from the leave-one-out experiments in Table 2, except for the number of epochs for CIFAR10 where we train for 150 epochs. Similarly to App. I.2, we use a Kronecker-factored Laplace approximation for variance computation and do not employ data augmentation during training.

I.5 Details of 'Predicting generalization during the training'

Details of the training setup: The experimental details, including test accuracies at the end of training, are listed in Table 5. We use a grid search to determine the regularization parameter δ . The learning-rate is decayed according to a cosine schedule. For diagonal-GGN-LOO and K-FAC-LOO, we use the SGD optimizer with an exception on the FMNIST dataset where we use the AdamW optimizer [36]. In that experiment, we use a weight decay factor of δ/N replacing the explicit L 2 -regularization term in the loss in Eq. 1. The regularizer R ( θ ) is set to zero. We do not use training data augmentation. For all plots, the LOO-estimate is evaluated periodically during the training, which is indicated with markers.

Additional details on hyperparameters of iBLR are as follows, where h 0 is the initialization of the Hessian:

MNIST, MLP (32, 16): h 0 = 0 . 1

MNIST, LeNet5: h 0 = 0 . 1

FMNIST, LeNet5: h 0 = 0 . 1

CIFAR10, CNN: h 0 = 0 . 05

CIFAR10, ResNet20: h 0 = 0 . 01

We set β 1 = 0 . 9 and β 2 = 0 . 99999 in all of those experiments. The magnitude of the prediction variance can depend on h 0 , which therefore can influence the magnitude of the sensitivities that are perturbing the function outputs in the LOO estimate of Eq. 16. We choose h 0 on a grid of four values [0.01, 0.05, 0.1, 0.5] to obtain sensitivities that result in a good prediction of generalization performance.

Additional Results: In Fig. 8, we show additional results for MNIST and CIFAR10 that are not included in the main text. For MNIST, we evaluate both on a the MLP (32, 16) model and a LeNet5 architecture. For the additional CIFAR10 results, we use the CNN. In Fig. 9 we include an additional

0

25

50

75

100

0.2

0.8

1.4

MLP, MNIST

SGD & Diagonal-GGN-LOO

Test NLL

LOO-CV

0

25

50

75

100

0.2

0.8

1.4

SGD & K-FAC-LOO

0

25

50

75

100

0.2

0.8

1.4

iBLR & LOO

0

25

50

75

100

0.0

0.8

1.6

LeNet5, MNIST

0

25

50

75

100

0.0

0.8

1.6

0

25

50

75

100

0.0

0.8

1.6

0

75

150

225

300

Epochs

1.2

1.8

2.4

CNN, CIFAR10

0

75

150

225

300

Epochs

0.8

1.6

2.4

0

100

200

Epochs

0.8

1.6

2.4

experiment where the model overfits. The K-FAC-LOO estimate deteriorates in this case, but we can still use the LOO as a diagnostic for detecting overfitting and as a stopping criterion. We train

0

25

50

75

100

Epochs

0.4

0.8

(a) Diagonal-GGN-LOO

0

25

50

75

100

Epochs

0.4

0.8

Test NLL

LOO-CV

(b) K-FAC-LOO

0

25

50

75

100

0.2

0.3

0.4

0.5

Batch size 8

LeNet5, FMNIST

Test NLL

LOO-CV

0

25

50

75

100

0.1

0.2

0.2

0.2

0.3

MLP (32, 16), MNIST

0

25

50

75

100

0.0

0.1

0.1

0.2

0.2

LeNet5, MNIST

0

25

50

75

100

0.8

1.2

1.6

2.0

2.4

CNN, CIFAR10

0

25

50

75

100

Epochs

0.2

0.3

0.4

0.5

Batch size 32

0

25

50

75

100

Epochs

0.1

0.2

0.2

0.2

0.3

0

25

50

75

100

Epochs

0.0

0.1

0.1

0.2

0.2

0

25

50

75

100

Epochs

0.8

1.2

1.6

2.0

2.4

a LeNet5 on FMNIST with AdamW and predict generalization. The trend of the estimated NLL matches the trend of the test NLL in the course of training.

In Fig. 10, we include further results for sensitivity estimation with the Adam optimizer. We use the following update

where g t is the minibatch gradient, β 1 and β 2 are coefficients for the running averages, ρ is a learning-rate, and ϵ a small damping to stabilize. We construct a diagonal matrix S t = diag ( N √ s t ) to estimate sensitivity with MPE as suggested in Table 1 ( N is the number of training examples). Better results are expected by building better estimates of S t as discussed in [27]. As described in section 3.4 of [27], a smaller batch size should improve the estimate, which we also observe in the experiment.

I.6 Details of 'evolution of sensitivities during training'

We use the MPE with iBLR for neural network classification on MNIST, FMNIST and CIFAR10, as well as MPE for logistic regression on MNIST. Experiment details are in Table 8.

For the experiment in Fig. 11(a), we consider Bayesian logistic regression. We set δ = 0 . 1 . The Hessian is always positive-definite due to the convex loss function therefore we use the VON algo-

Estimated Deviation

Examples

Low Sensitivity

High Sensitivity

Low Sensitivity

High Sensitivity

5th epoch

10th

25th

125th

(a) MNIST, Bayesian logistic regr.

High Sensitivity

Low Sensitivity

Examples

Low Sensitivity

High Sensitivity

1st Epoch

2nd

20th

50th

(b) MNIST, iBLR with MLP

High Sensitivity

Low Sensitivity

Estimated Deviation

Examples

Low Sensitivity

High Sensitivity

1st Epoch

20th

100th

250th

(c) CIFAR-10, iBLR & ResNet-20

0

1

2

3

4

Test NLL

0.02

0.05

0.08

0.11

LOCO estimation

6

2

0

4

3

5

7

9

8

1

MLP

LeNet

(d) Class removal result for MNIST

0.014

10

4

0

2

1

3

2

3

3

3

5

5

4

0.007-

0.000-

0-

↑

0.12

0.06

0.00-

rithm given in Eq. 29. We use 125 updates with batch-size 200 , reaching a test accuracy of around 91% using the mean m t . We use linear learning-rate decay from 0 . 005 to 0 . 001 for the mean m and a learning-rate of 10 -5 for the precision S . The expectations are approximated using 3 samples drawn from the posterior. We plot sensitivities at iteration t = 5 , 10 , 25 , 125 . For this example, we use samples from q t to compute the prediction variance and error ( 150 samples are used). We sort examples according to their sensitivity at iteration t = 125 and then plot their average sensitivities in 60 groups with 100 examples in each group.

For the experiments in Fig. 3(b), Fig. 11(b) and Fig. 11(c), we consider neural network models f ( θ t ) on FMNIST, MNIST and CIFAR10. We do not use training data augmentation. For CIFAR10 we use a ResNet-20. The expectations in the iBLR are approximated using a single sample drawn from the posterior. For prediction, we use the mean m t . The test accuracies are 91 . 3% for FMNIST, 98 . 5% for MNIST and 80 . 9% for CIFAR10. We use a cosine learning-rate scheduler with an initial learning-rate of 0.1 and anneal to zero over the course of training. Other experimental details are stated in Table 8. Similar to before, we use sampling to evaluate sensitivity ( 150 samples are used).

J Author Contributions Statement

Authors list: Peter Nickl (PN), Lu Xu (LX), Dharmesh Tailor (DT), Thomas Moellenhoff (TM), Mohammad Emtiyaz Khan (MEK)

All co-authors contributed to developing the main idea. MEK and DT first discussed the idea deriving sensitivity measure based on the BLR. MEK derived the MPE and the results in Sec 3 and DT helped in connecting them to influence functions. PN derived the results in 3.3 and came up with the idea to predict generalization error with LOO-CV. LX adapted it to class-removal. PN wrote the code with help from LX. PN and LX did most of the experiments with some help from TM and regular feedback from everybody. TM did the experiment on the sensitivity evolution during training with some help from PN. All authors were involved in writing and proof-reading of the paper.

K Differences Between Camera-Ready Version and Submitted Version

We made several changes to take the feedback of reviewers into account and improve the paper.

The writing and organization of the paper were modified to emphasize the generalization to a wide variety of models and algorithms and the applicability of MPE during training.

The presentation was changed in Section 3 to emphasize the focus on the conjugate model. Detailed derivations were pushed to the appendices and more focus was put on big picture ideas. Arbitrary perturbations parts were made explicit. Table 1 was added and more focus was put on training algorithms.

We added experiments using leave-one-out estimation to predict generalization on unseen test data during traininig. We also added results to study the evolution of sensitivities during training using MPE with iBLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes measuring sensitivity to data in some problems in machine learning through what they refer to as the “memory perturbation equation” (MPE), derived using the Bayesian learning rule. They argue that sensitivity properties of machine learning models are not as well understood as they could be in the literature, and therefore propose a remedy to this issue that generalizes some prior approaches. Much of the paper is about justifying the equation and showing that some prior works are special cases. There is also an experimental section that provides some empirical validation of some equations.\n\nSTRENGTHS:\nAlthough I am not familiar with this general area of research, which seems to be related to influence functions in machine learning, I found the paper interesting and novel. I also found the topic to be highly relevant for NeurIPS and an important one for machine learning in general. The fact that the approach generalizes some prior approaches in a non-trivial way is powerful.\n\nWEAKNESSES:\nWhile I’m generally supportive of this paper, I had a hard time following many of the details in this paper and cannot verify aspects of the approach. This is partly because of my lack of familiarity with the subject, but also because I think the authors have not provided enough background and explanation for a reader with little know-how around related topics. Also, I think the presentation can be improved significantly – please see my detailed questions/comments for some suggestions. I will rely heavily on more knowledgeable reviewers to provide a more thorough assessment.\n\nQUESTIONS:\nSome questions/comments follow:\n\n- For the title, I don’t think “model’s” is the most suitable choice of term. I suggest using either “model” or “models’”.\n\n- “MPE” also stands for most probable explanation in Bayesian modeling, so this could be a point of confusion for some, without context.\n\n- I’m not sure how the word “memory” is suitable for what the authors intend in this work. This is what is written in Section 1: “These highly sensitive examples can be seen as those characterizing the model’s memory; the model is highly sensitive to them and perturbing them can make the model forget its essential knowledge”. Perhaps the authors could explain and justify their choice of term? It may help to connect any other work that they deem to be relevant or inspirational.\n\n- For Fig 1, what is the dataset on which a) and b) panels are based on? Are these all for FMNIST?\n\n- What is the relation between output function f_i(.) and the terms in equation 1? I don’t think this is ever specified. Is f_i the same as l_i?\n\n- Several of the equations (such as those in Section 3) are stated with pointers to prior work but without clear derivation. I understand that there are space limitations, but I had trouble following the correctness of the equations.\n\n- There is a claim in line 207 about how the proposed approach is more principled than some prior work – as someone not familiar with the field, I did not observe much justification for this claim, or perhaps just did not understand it clearly.\n\n- “Sharpness” is mentioned in line 227 but never defined. There are other such terms that don't get enough explanation.\n\n- There are typos, grammatical errors or other such issues in at least the following lines: 39, 41, 59, 61, 63, 68, 110, 113, 124, 161, 221, 236, 244, 249, 268, etc. Note that many of the early typos are due to the incorrect use of verbs for single vs. plural nouns. I suggest the authors carefully review the paper and fix these errors.\n\nLIMITATIONS:\nLimitations are not mentioned in enough detail.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of the model’s sensitivity to its training data (e.g., the counterfactual of how the model's performance will change if trained without certain data), commonly referred to as the \"data influence\" problem. This paper presents the \"memory perturbation equation (MPE)\" to study the model’s sensitivity to such perturbations and uses Bayesian learning rule (BLR) approximated using an exponential family distribution. Then, the BLR can be used to estimate the deviation caused by the removal of a group of data.\n\nThe paper shows that MPE with Gaussian posterior recovers the celebrated Influence Functions (INF) in both linear and nonlinear cases, and INF can be considered as a special case of the MPE. The work also provides a variety of empirical results on MNIST to CIFAR-10 that shows the estimated sensitivity highly correlates with the actual deviation that would be caused by removing the data and thus can be used as its predictions.\n\nSTRENGTHS:\nThis paper is an interesting addition to the research line on \"data influence\" problems. The paper is well-written. The conceptual development of this paper is clear and smooth. The methodology development and its derivations are solid and the paper also provides insightful remarks supporting the narrative. The method presented in the paper is general and can be potentially applied to a variety of work. Experiment results are nicely presented, clear and interesting.\n\nWEAKNESSES:\nI am listing some minor issues that may be improved. \n\nIn this work, despite strong correlations to the Influence Function have been shown, the comparison to data influence methods in general isn't comprehensive. There are often works talking about recovering linear influence function or using Taylor approximation to extend to nonlinear cases. It would be nice to see how it relates to or compares to broader works, such as [1] or TracIn [2], just to name a few.\n[1] Repairing Neural Networks by Leaving the Right Past Behind. NeurIPS 2022.\n[2] Estimating training data influence by tracing gradient descent. NeurIPS 2020.\n\nThe empirical results of the proposed approach seem isn't especially strong. As we can see, the correlation between prediction and the actual counterfactual deteriorates quickly as the model becomes bigger. It seems quite well for MNIST/MLP, but less satisfactory for LeNet/FMNIST and CNN/CIFAR-10. I appreciate the honesty of authors that show these results in a straightforward manner such that the reader can directly see the real capability of these methods. Because of that, I would think the relatively weak empirical results do not pose a major compromise to the contribution of this work.\n\nQUESTIONS:\nI was wondering about the approximation error for using exponential family distributions or Gaussian priors. It would be nice if the paper can provide some discussions on this to help better understanding.\n\nWhat is the computational overhead of the proposed method, in terms of time complexity and memory demand?\n\nLIMITATIONS:\nThe empirical results are relatively weak. The computational demand and its scalability are unclear.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce the Memory Perturbation Equation (MPE), a generalization of sensitivity measures of models that uses the recently proposed Bayesian Learning Rule (BLR) as a foundation. Thanks to its Bayesian foundation, the MPE can be used for used for non-converged models and for non-differentiable loss functions.\n\nSTRENGTHS:\nThe biggest strength of the paper is the writing. This is an *extremely* well written paper that conveys the idea beautifully. It flows extremely well and is very easy to read and follow along. The generality of the MPE is also mind boggling. This is an extremely general method that I think will be pivotal for deep learning. Lastly, the experiments section did a great job verifying the theory of the method and giving a taste of all of the potential uses for it.\n\nWEAKNESSES:\nWhile I think the writing is fantastic, I think a small exposition on the BLR would due wonders, especially because the paper cites specific equations form the BLR paper. To make the paper more readable, a small section in the appendix with details relevant to the paper and its experiments would do wonders.\n\nWhile I understand the theory of the method, I have a slight practical disconnect between the experiments from this paper and the BLR. Since Adam naturally arises from the BLR, the posterior variances can be constructed easily and, crucially, the posterior covariance would be diagonal, sidestepping messy matrix inversions. But, from reading the appendix section, it seems like the authors are doing the opposite and are instead getting the Hessian of the loss function (in the appendix the authors write: \"Variance computation requires inversion of matrices. For large problems, we use a Kronecker factored Laplace variance approximation as implemented in the laplace package[5]\"). If this is indeed the case, then I do find this slightly concerning and is a major limitation of the method as computing Hessians are expensive and practically infeasible for modern architectures. Moreover, it doesn't fall in line of the mantra of the paper which is to leverage the BLR to get these sensitivity measure for practically free.  \n\nPlease let me know I have misunderstood what is going on.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nA limitation section is missing from the paper, which I think is important to have.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the sensitivity of machine learning models to training data. Previously, sensitivity was often studied through empirical investigations which was costly and do not always generalize across models. In this paper, the authors present the memory perturbation equation (MPE), based on the Bayesian learning rule, as a unifying equation to understand sensitivity of generic ML algorithms to training data. The MPE has two features: Sensitivities to examples is estimated by using natural gradients of those examples alone, and examples with larger natural gradients contributes more to the sensitivity. The authors show that the MPE when specialized to Gaussian posteriors, recovers influence functions in linear models and deep learning, and examples with high prediction error and predictions variances are the most influential and the sensitivity is obtained by multiplying the two; The sensitivity can be estimated cheaply whenever natural gradients are cheap to compute; Finally, the authors show empirically that the MPE can be used to accurately estimate generalization performance on image classification tasks.\n\nSTRENGTHS:\n- The paper is well written and easy to follow for non-experts.\n- The proposed MPE formulation is well supported by theoretical foundations based on bayesian learning rule. And according to the authors, the proposed MPE has several nice and intuitive theoretical properties for Gaussian posteriors.\n- The experiments are well designed to show various aspects of the the properties of the proposed sensitivity measure, which is interesting and clear.\n\nWEAKNESSES:\n- I don’t see much discussions on related work or empirical comparisons to state-of-the-art methods\n- The example datasets used in the empirical evaluations seem to be mostly smaller datasets like MNIST and CIFAR.\n\nQUESTIONS:\nOut of curiosity, can the proposed MPE or part of the proposed perturbation-based formulation be used to perform machine learning data valuation, i.e., to evaluate the importance of groups of data samples towards a machine learning task (e.g., https://bair.berkeley.edu/blog/2019/12/16/data-worth/)\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations in their discussions section. I don’t see any potential negative impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a generic method, relying on some Bayesian analogy, to perform instance-wise sensitivity analysis of learned models. A generic updating rule and derivative is presented, as well as its different variations on popular models (Neural networks and Gaussian models). One notable use of the proposed sensitivity indices is to estimate the genralisation capabilities of a given model. \n\nExperiments are performed on classical MNIST/FMNIST data sets to confirm that the behaviour of the method is the expected one.\n\nSTRENGTHS:\n+: a versatile method to perform sensitivity analysis\n\n+: a strong technical proposal, at least from what I can assess\n\nWEAKNESSES:\nSee questions for more detailed comment\n\n-: link with Bayesian approach remains to some extent unclear in the general case. \n\n-: the approach mainly consider instance-wise sensitivity indices, and it is unclear whether this is sufficient in general?  \n\n(minor) -: the paper contains a number of typos, and a final read could be useful to correct them (e.g., L41 \"can estimated\", L63 \"the expressions shows\", L59, \"data as large\", L73, \"such topics as rare\", ...)\n\nQUESTIONS:\n* Bayesian approximation validity: In L92, I found it very strange that one can assume that there is always a proportional link between a Bayesian posterior and the exponential of a Loss function. Sure, the exponential allow to turn the additivity of the loss function into a product form mimicking Bayesian updating, but is it guaranteed that any loss function can be related to a legitimate (proper) prior/likelihood pair in a standard Bayesian framework? Similarly, except for the Gaussian case, what is the quality one can expect from an appriximation using the exponential family? This is a bit hard to figure out, as all examples given in the rest of the paper rest on the (specific) Gaussian case? A good summary of this whole question would be \"How Bayesian (in the sense of satiisfying the axioms and coherence of Bayesian approaches, as expressed by, e.g., De Finetti) actually is the BLR rule?\"\n\n* In echo to my previous question, do we have an idea of how often the premisses of Theorem 2 ($q_\\lambda=p(\\theta|\\mathcal{D})$) are true? Verifying it seems highly non-trivial to me. \n\n* Thanks to the underlying assumptions made, it makes perfect sense to consider that perturbation effects are additive and that samples have independent effects on the model. However, it would seem more reasonable to consider that, in practice, samples do have interactions between them and that the impact on the learned model are not merely additive, hence the need to include some possible dependencies/interactions between the samples. Maybe the authors could clarify a bit the situations where the additivity assumption makes sense? \n\n* While the removal of an observation is perfectly interpretable from the data perspective, I would appreciate some clarification about the $\\epsilon_i$ perturbation. From what I get, this would correspond to weighting the data  in the learning scheme, and see what happens in this case. Is this correct, or would it be more accurate to consider that the data can move within a given neighboorhood within the input space (a perturbation that would probably make more sense from a physical viewpoint, even if I am fine with the weighting interpretation). \n\n* Suggestion: I do not really see the advantage of repeating figures that can be found in some other places in Figure 1, especially as a reader will not have the necessay elements to properly interpret them at this stage. Maybe remove them to better discuss some other points?\n\nLIMITATIONS:\nSee questions about some possible limitations, not especially discussed/mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}