Laplace Sample Information:  Data Informativeness Through a Bayesian Lens
Authors: Johannes Kaiser
Kristian Schwethelm
Daniel Rueckert
Georgios Kaissis
Venue: ICLR 2025 Poster (2025)
Keywords: Sample informativeness
Sample Information
Sample  Difficulty
Long-tailed distribution
Leave-one-out retraining
KL Divergence
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: This paper establishes a novel measure of sample informativeness based on the KL divergence applied on a quasi Bayesian approximaiton of the model parameters.

Abstract: Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. 
We propose $\text{\textit{Laplace Sample Information}}$ ($\mathsf{LSI}$) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings.
$\mathsf{LSI}$ leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a sample of interest from the dataset.
We experimentally show that $\mathsf{LSI}$ is effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty.
We demonstrate these capabilities of $\mathsf{LSI}$ on image and text data in supervised and unsupervised settings.
Moreover, we show that $\mathsf{LSI}$ can be computed efficiently through probes and transfers well to the training of large models.

Published as a conference paper at ICLR 2025
LAPLACE SAMPLE INFORMATION: DATA INFORMA- TIVENESS THROUGH A BAYESIAN LENS
Johannes Kaiser & Kristian Schwethelm & Daniel Rückert & Georgios Kaissis AI in Healthcare and Medicine; Munich Center for Machine Learning (MCML) Technical University of Munich, Germany ( (johannes.. ka taiser, g ka ssi s @t de
ABSTRACT
Accurately estimating the informativeness of individual samples in a dataset is an important objective in deep learning, as it can guide sample selection, which can improve model efficiency and accuracy by removing redundant or potentially harmful samples. We propose Laplace Sample Information (LSI) measure of sample informativeness grounded in information theory widely applicable across model architectures and learning settings. LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by sample of interest from the dataset. We experimentally show that LSI effective in ordering the data with respect to typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty. We demonstrate these capabilities of LSI on image and text data in supervised and unsupervised settings. Moreover, we show that LSI can be computed efficiently through probes and transfers well to the training of large models.
INTRODUCTION
Deep learning (DL) relies fundamentally on pattern extraction and knowledge acquisition from underlying data to determine model's parameters. Throughout the training of neural network, each sample in the training dataset imparts information on these parameters. However, the amount of information varies between samples. Beyond providing insights on learning dynamics, measuring per-sample information content allows selecting the most beneficial subsets of a given dataset for model training (i.e. dataset condensation or distillation (Chen et al., 2022: Tan et al., 2023)). This can help to reduce the amount of required computational resources, rendering model training more (Strubell et al., 2019). However, achieving these efficiency goals
economical and energy-efficiee without decreasing the learni model's 's accuracy (or other beneficial properties such as sub-group fairness (Ghosh et al., 2023)) is of particular importance and represents a challenge which -so far- has not been sufficiently addressed (Asi & Duchi, 2019). Additionally, evaluating the contribution of individual samples to the final model is of importance across many domains of machine learning (ML) research such privacy preservation (Unsal & Önen, 2023), memorization analysis (Bansal et al., 2022), sample difficulty evaluation (Harutyunyan et al., 2021), machine unlearning (Xu et al., 2023), and equitable reimbursement for data contributors (Cummings al., 2015). All of the aforementioned workflows thus stand to gain from a well-defined notion of sample informativeness.
We contend that a useful measure of sample informativeness should (1) be applicable most datasets and learning settings without imposing simplifications on the model or constraints on the training and (2) be grounded in information theory to truly measure information and not a related but different- quantity like sample difficulty. While some previous methods (Agarwal et al., 2022: Jiang et al., 2020) are applicable broad range of training settings, they do not measure information flow but rather related quantities (e.g. sample difficulty) which merely approximate informativeness. To compute an information- information-thhorettiaall theoretically grounded measure of sample informativeness. the Kullback Leibler (KL) divergence can be employed to approximate the information flow (more formally, the conditional point-wise mutual information) from a single sample to the parameters of the network. However, the KL divergence is defined probability distributions, whereas most trained neural networks (excluding, e.g., Bayesian neural networks) comprise a single set of parameters, for which the KL divergence is undefined. To enable computing the KI. divergence in this setting, prior works have used

---

Published as a conference paper at ICLR 2025
techniques like linearization (Harutyunyan et al., 2021). However, this approach leads performance degradation (Chizat et al., 2019) and scales poorly to larger realdworld datasets. Alternativel,, noisy iterative learning algorithms, e.g. Langevin dynamics (Negrea et al., 2019) Gaussian processes (Ye et al., 2023), result a distribution of parameters but come with computational cost that increases proportionally with dataset size (Brosse al., 2018).
To address the aforementioned challenge, we propose Laplace Sample Information (LSI) (Definition 2) as measure of the sample informativeness. LSI estimates the KL divergence-based information via a Laplace approximation, a post-hoc method to construct a quasi-Bayesian learner from a trained neural network, and to thus obtain distribution over model parameters. LSI is applicable to various model architectures, learning settings, and can be computed through automatic differentiation, which facilitates the use of LSI to analyze sample informativeness in diverse DI. tasks.
Similar to the aforementioned KI. divergence-based techniques, LSI relies on leave-one-out (LOO) retraining of the model to provide a direct measure of informativeness from omitting single sample. This requires repetitive training, which can become exceedingly costly for larger models. To mitigate this cost, we show that LSI can be computed using a (very) small model as probe. The sample order obtained from this probe generalizes well to the training of larger models. This approach allows us to combine the fidelity of a true 1DO-based informativeness metric while remaining economical and time-efficient We provide the code well as the pre-computed values of LSI for common datasets under
Our contributions can be summarized as follows:
Our primary contribution is the introduction of LSI, an approximation of the unique infor- mation contributed by an individual sample to the parameters of neural network computed via the KI--divergence of I Laplace approximations to the posterior of neural network's parameters;
We demonstrate the ability of LSI to identify spectrum of typical/atypical samples, detect mislabeling, and assess informativeness on the level of individual samples, dataset classes and the entire dataset in supervised and unsupervised tasks across various data modalities (images and text);
We show that LSI can be efficiently computed by probing the model features and generalizes effectively to large architectures.
2 RELATED WORK
Sample Informativeness The notion of sample informativeness is defined somewhat inconsistently in literature. For instance, some studies focus on the information contributed by individual samples to learner, defining informativeness the reduction in parameter uncertainty upon the addition of a sample (Dwork et al., 2015; Rogers et al., 2016). A more recent body of work describes sample informativeness by extending Shannon information theory to consider computational constraints (Xu et al., 2020; Ethayarajh et al., 2022). Yet another research direction employs influence functions to estimate the informativeness of individual samples. This method examines changes in model parameters due to the inclusion of specific samples. based on the model's final parameters (Koh & Liang, 2017: Pruthi et al., 2020; Schioppa et al., 2022). However, this approach does not account for the so-called "butterfly effect" in neural networks (Basu et al., 2020; Ferrara, 2024), where omitting a single sample at the beginning of training can alter the gradient trajectory and lead to significant misestimation of sample's true influence; recent work has shown that influence functions are poor substitutes for true leave-one-out (LOO) retraining (Schioppa et al., 2024; Bac et al., 2022).
Our work's 's use of the KL-divergence is closely related to the intommation-thhorree notion of al- gorithmic stability, which investigates the change in a model due to single sample differences in underlying datasets (Bassily et al., 2016; Raginsky et al., 2016; Steinke & Zakynthinou, 2020). For instance, Bassily et al. (2016); Steinke & Zakynthinou (2020); Feldman & Steinke (2018) define max KL stability and the average leave one out KL stability as supp,p- <L D) pa(0 D-))
and supp n Em (L(pA(0 D) 1l pA(O | D-1)) respectively, with (A | D) being the output distribution over the parameters O of an algorithm A trained on the dataset D {.1,... 2n and D = D {.]. While max KL stability is defined across all possible datasets, average KL stability is defined a fixed underlying dataset. LSI is thus closely related to average per-sample level KL stability: this definition is also used by Harutyunyan et al. 12.21). Moreover, a connection between LSI and Differential Privacy exists, which we investigate in Appendix H.
2

---

Published as a conference paper at ICLR 2025
Estimating Parameter Distributions Computing the KL divergence on the constant sets of pa- rameters (i.e. degenerate random variables) outputted by most monn-aaesian learning algorithms is impossible. Negrea et al. (2019); Ye et al. (2023) circumvent this issue by leveraging a Bayesian learning setting, which generates a distribution of parameters, while Rammal et al. (2022) add noise to the final parameters (smoothing) generate distribution. Another line of work aims approximate the training of a neural network by employing continuous stochastic differential equations (Li et al., 2017) through neural tangent kernels using linear approximation of KL divergence- baseed informativeness notation (Harutyunyan et al., 2021). Our approach mirrors the Bayesian strategy, but we establish a distribution on the network parameters post-hoc by employing a Laplace approximation to the posterior parameter distribution (Daxberger et al., 2021; MacKay, 1992).
Sample Difficulty demonstrated by Ethayarajh et al. (2022), sample difficulty is related sample informativeness, as it (informally) measures the probability that a neural network can learn sample. Previous studies approximate sample difficulty by examining the impact of model compression on individual samples (Hooker et al., 2020b;a) or by assessing the effect of individual samples the loss of a held-out dataset (Mindermann et al., 2022; Wu et al., 2020). Our work related to methods that infer sample difficulty from loss landscape geometry (Zielinski et al., 2020; Chatterjee, 2020; Agarwal et al., 2022; Katharopoulos & Fleuret, 2018), as the Laplace approximation leverages curvature information. However, we note that -while sample difficulty can be proxy for informativeness- the two concepts are distinct, and, while the aforementioned methods use sample difficulty to infer informativeness (Agarwal et al., 2022). we directly measure sample informativeness, which allows us to draw conclusions about sample difficulty.
3 LAPLACE SAMPLE INFORMATION
Next, we formally introduce LSI and its theoretical underpinnings.
Sample Information As diseussed above, LSI is per-sample informativeness notion based on the KL divergence, the arguably most established information theoretic divergence. Consider a (probabilistic) training algorithm A, a training dataset D {21.... 2n an input-label pair of interest 2; = (æ,, yi), and the training dataset with 2; removed D D' { The distribution of parameters induced by is denoted as PA(O (0 D), where A E TRKK is a specific realization of the random variable of the parameters θ To approximate the (per-sample) information flow of the sample of interest 2, to the neural network's s parameters, we compute the KL divergence of the distribution of parameters of model trained on D to the distribution of parameters of model trained on D- This information measure is called the Sample Information (Sl) of 24:
Definition (Sample Information). The Sample Information of 2; is defined as:
SI(A, D, 23) = KL (pa (0 D) (pa (0 ()D)))
SI is justified information ceemmatton-theoorticalll theoretically, as it represents an upper bound on the point- wiit conditional mutual information between the parameters of the trained neural network and the datapoint 2;. For this reason, SI has already been used in previous literature (Harutyunyan et al., 2021).
Alternatively, SI can be understood through the lens of information-theoreti hypothesis testing: In information theory, (L(Pa(0) D) I| pA D-)) can be interpreted as the expected discrimination information or the expected weight of evidence of D vs. D through observing 0. In other words, SI can be interpreted as the weight of the discriminatory evidence induced by 2;. Intuitively, samples that contribute a lot of information to allow for improved insight about the underlying dataset compared to samples that contribute little information, which the KL divergence can measure.
Laplace Approximation The definition of above implies that the training algorithm is proba- bilistic; however, most neural networks are trained with Maximum Likelihood Estimation (MLE). This does not result in a distribution over trained parameters which is required to estimate the KI. divergence. To remedy this, previous works have leveraged probabilisticBayesian neural network training (Negrea et al., 2019; Ye el al., 2023) which can be computationally expensive, or added noise directly the final parameters (Rammal et al., 2022), which reduces model utility. contrast, we propose to construct quasi-Bayenn Bayesian learner from the trained model parameters by fitting posterior probability distribution to the parameters using a Laplace approximation, classical technique in Bayesian inference, whose application in neural networks has recently witnessed increased interest.
3

---

Published as a conference paper at ICLR 2025
We use the "classical" Laplace approximation (Daxberger et al., 2021) throughout and discuss the recently proposed Riemann Laplace approximation (Bergamin et al., 2024) in Appendix I. The Laplace approximation leverages second-order Taylor expansion of the loss landscape around the final parameters of the model Ag. It thus assumes that the loss function is convex around the local minima and can be approximated by quadratic function, which for the analysis of the training behavior of neural networks (Wen et al., 2020; Zhu et al., 2018; He al., 2019). The Laplace approximation models the parameters of the neural network as multivariate Gaussian distributions whose covariance is based on the local curvature of the loss landscape. This normality assumption is made by invoking the central limit theorem on the distribution of the pa- rameters after many training steps. Despite these assumptions, the aforementioned works show the Laplace approximation to efficient and competitive with fully Bayesian training deep networks, which motivates its use in our work.
common assumption made
Formally, suppose that a neural network is trained on dataset by minimizing the loss function C(D, 0) with I (Ins Mni () as the empirical loss term and regularizer ((()) on the parameters 0 IR resulting in:
((() + E (I Un: (((n))))))))
0 = arg min (DD arg min BERK
BERK
Considering the empirical loss term as an i.i.d. log-likelihodd (Zi. Mn: Mn: = - log p(un (zx))
and the regularizer as a log-prior r(0) = log p(0) establishes i as maximum a-posteriori estimate (MAP) of the parameters. other words, the MLE of the regularized learner is interpreted MAP solution whose prior is defined by the regularizer. The Laplace approximation then uses a
second-order Taylor expansion of L around 0 to construct Gaussian approximation of p D):
((D,0) x L(D, 0) + (VoL(D,
0) + 0) 10- ()) 0).
(2)
Considering a converged model, the first order term vanishes as VoL(D, و x 0 yielding the Laplace approximation the posterior distribution as:
p(0 D) = ( Σ) with E= (V)
(3)
Remark that while the derivation of the Laplace approximation requires the model to converged, show that the sample order established by LSI remains largely consistent across training. Thus, according to our empirical findings in shown Appendix G, employing LSI to gain insights does not necessitate the convergence of the model.
Laplace Sample Information (LSI) After introducing the key theoretical insights, we are now ready to define LSI. Concretely, combine the Laplace approximation of the posterior parameter distribution (Equation (3)) and the definition of sample informativeness (Definition 1), which yields definition of Laplace Sample Information:
Definition (Laplace Sample Information). Let A be (non- -ony-aaaasi learning algorithm with parameters 0, a loss function L. a dataset D) and D- = D \ {z,): ) moreover, let OMLE be the maximum likelihood estimate of 0. Then, the LSI of the investigated sample z, with respect to A and D is defined as:
LSI( A, ' & KL(N(0, E)I
(4)
Above, Ó A O (AM)(A(()) and 0-1 OM (A(D ")).
Moreover, E A (V2C(D, 0)) and Wー 1 (VGL(D-10-1))-
The Laplace approximation results in Gaussian distribution over K neural network parameters, enabling the computation of the KL divergence and, consequently, the LSI as follows:
(det(E-1) )))
LSI (2, -, (n((E-))- - K -)
LSI has a notable benefit: since the Laplace approximation is computed after model training, the computation of LSI is agnostic to model architecture, the dataset, the loss function, and the learning
4

---

Published as a conference paper at ICLR 2025
setting. In Appendix A, show the benefits of LSI compared two other information measures from recent literature (Harutyunyan et al., 2021; Wongso et al., 2023).
Efficient Computation of LSI In terms of computational considerations, the Laplace approximation of a neural network requires the computation of the Hessian of the loss function with respect to the parameters. Unfortunately, for K total model parameters, the Hessian requires O(K2) memory, which is intractable even for moderate networks. Additionally, the inversion of the Hessian for computing the covariance matrix has time complexity of O(K3). To address this issue in practice, we leverage the established technique of using Hessian approximation. We note that LSI is agnostic to the exact approximation. For example, block diagonal approximations such as K-FAC or their eigenvalue- -aasel counterparts (Grosse & Martens, 2016; Bae et al., 2018) can be used. However, in our experiments, we find that even a diagonal approximation works very well, i.e. discarding all but the diagonal Hessian entries (Farquhar et al., 2020; Kirkpatrick et al., 2017), resulting in O (') in memory and a time complexity of only ((K) for the inversion. We show that the LSI of this diagonal approximation (as well as the KFAC approximation) is well-correlated with the LSI using the full Hessian in Appendix C, indicating the applicability of the diagonal approximation.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model and Datasets We demonstrate the utility of LSI with experiments supervised image tasks using CIFAR-10, CIFAR-100 (Krizhevsky & Hinton, 2009), medical imaging dataset (pediatric pneumonia, i.e. lung infection in children), (Kermany al., 2018) and two ten-class subsets of the ImageNet dataset (Deng et al., 2009), Imagewoof and Imagenette (Howard, 2019). We select CIFAR-10 and CIFAR- 100 as easy and challenging benchmarks, and the two ImageNet subsets as easy and challenging representatives of large image datasets. The pneumonia dataset represents challenging real-works use case, as two of the classes (bacterial and viral pneumonia) are hard to distinguish, even for human experts.
Recall that LOO-estimation of LSI for any given sample necessitates retraining model. To circum- vent this requirement and lessen the computational burden, we employ probe, whereby we estimate LSI on a small model trained on image features computed by a frozen feature extractor. For the image tasks, we employ ResNet-18 (He al., 2016) up to the fully connected layer pre- pre-trained on ImageNet as the frozen feature extractor and append a single hidden layer with a ReLu activation function followed by classification layer as task-specific classification head/ probe model. The fact that the sample informativeness order captured by LSI on the probe transfers excellently to larger models, which is discussed in detail in Section 4.4 and Appendix D, is one of LSI's key benefits. We find that employing a probe allows the computation of the sample information with a speedup of at least three orders of magnitude. Interestingly, despite the fact that the Laplace approximation requires a converged model, empirically LSI is applicable even far prior to model convergence (after brief warm-up period), as shown in Appendix G.
Beyond image classification, we show the applicability of LSI in text sentiment analysis task on the IMDb dataset (Maas et al., 2011). As a model, we employ a pre-trained BERT (Devlin et al., 2019) as feature extractor and the probe model described above.
Moreover, we compute LSI in unsupervised contrastive learning using CLIP (Radford et al., 2021) between image and caption pairs of the COCO dataset (Lin et al., 2015). To decrease the compu- tational complexity, we reduce the dataset to images carrying segmentation masks of bananas or clephants. We use frozen vision and language transformers (BERT (Devlin et al., 2019) and ViT-B/16 (Dosovitskiy et al., 2020)) as a backbone and compute LSI with respect to the embedding layer (acting as the probe).
To evaluate LSI independently of the sample order, we used full-batch gradient descent for model training. Note that LSI makes assumptions about the training process and remains applicable in batched training setting, either combined with multiple re- trainining to average out the effect of batch sample ordering or sampling from the entire dataset, e.g. using Poisson sampling (Abadi et al., 2016).
Training Parameters and Hardware All training and description of the hardware used and resources required is provided in Appendix O.
are provided in Appendix N
5

---

Published as a conference paper at ICLR 2025
4.2 ESTIMATING DATA TYPICALITY, INFORMATIVENESS AND DIFFICULTY USING LSI
LSI Orders Samples According to Typicality We begin by showing that LSI provides an inter- pretable notion of individual sample informativeness. Recall that, shown by Feldman (2020), data-generating distributions tend to exhibit a long tail concerning sample typicality (i.e. view angle, object size, etc.). LSI unveils this long-tailed distribution, as the LSI distribution on the examined datasets also exhibits long tail (Figure 1). As most samples have low LSI and only a few samples have high LSI, only a small subset of the data is highly informative, while most samples contribute little information to the parameters. Note that, as the model has no prior "knowledge" about typicality or about the data distribution, this notion of sample typicality arises during training and is encoded through the amount of information individual samples contribute to the model. Thus, this notion of typicality does not necessarily coincide with its human interpretation.
CIFAR- 10
CIFAR-100
Imagenette
Imagewoof
100
10
10
Density
100
50
50
5
5
0.00
0.02
0.04 LSI
0.06
0.00
0.02
0.04 LSI
0.06
0.0
0.1 0.2
0.3
0.4
0.0
0.1
0.2
0.3
0.4
LSI
LSI
Figure 1: LSI distribution across individual samples of the investigated datasets.
Investigating samples with low and high LSI, we find that samples that are strongly representative of their class provide little unique information to the model parameters. Consequently, representative samples have low LSI, while samples with high LSI often are cropped, mislabeled, or show the object from an uncommon perspective. This indicates that the model "encodes" the long-tailed sub-population distribution described by Feldman (2020).
For example, while the least informative samples in the Australian terrier class in ImageNet are well-framed and highly typical representatives of the breed, high LSI samples also contain a variety of different dog breeds, i.c. mislabeled samples. Similarly, high LSI samples of the radio class contain image of a car. Low LSI samples in the pneumonia dataset contain chest x-rays that have similar exposure and framing, while high LSI samples contain cropped, over/ underexposed images as well a chest-x-ray of an adult woman, which is clearly wrongly included in a pediatric (children's s) dataset (Figure 2).
an
larger selection of images beyond the aforementioned representative findings can be found in Ap- pendix K (ImageNet), Appendix L (CIFAR-10), and Appendix M (pneumonia dataset). Exemplarily, in CIFAR-10, the lowest LSI within the bird and airplane categories feature a blue sky background, while the samples carrying the highest LSI have the sun as the background for airplanes or are frontal full-frame views of ostriches, a rather unrepresentative member of the bird class.
Low LSI
High LSI
ImageNet
Australian Terrier
ImageNet
Radio
Pneumonia
Figure 2: Selected images with low/high LSI in ImageNet and the medical dataset. Samples with low LSI are representative of their underlying class, whereas high LSI samples are often mislabeled/ out-of-distribution (red dots) or atypical with respect exposure, viewing angle, etc.

---

Published as a conference paper at ICLR 2025
In the text domain, unambiguous texts aligning with their label carry low sample informativeness and thus are assigned low LSI. Conversely, high LSI indicates mislabeled, ambiguous, or label- contradicting texts, as shown exemplarily in Figure 3.
Low LSI sample with positive label:
High sample negative label:
This film brilliaat! It touches everyone who
Earty, heavy war-time propagenda short urging people be cureful with their spending praciices [..]] Uving scare. guilt and war-time production and therefore incresree and subrespent spending practices [...] serions problems during and after the war. truly window into the past, historically and culturalls.
extraordiaary way. It really takes you back to your youth and puts a perspective how you view your childhood memories, There so many layers to this film. It innovative und absolutely fabulous!
Legend: Amhiguuty
Lahel conmudicting
Label confirming
Figure 3: LSI supervised text classification on the IMDb dataset using BERT
Even in multi-modal tasks, we find that typical samples with a prominent portrayal of class represen- tatives (e.g. bunches of bananas) have low LSI, as they carry little unique information. Images in which the subject is not prominently shown, not corresponding to the banana label, or do not contain any bananas (e.g. box of cucumbers) have high LSI (shown in Figure 4).
Low LSI
High LSI
ditigg ras Aach other
oren hasanas randhe fooss zarana
with drsan
banunes
cackmbee..
Figure 4: Selected images with low/high LSI of contrastive learning on COCO
Focusing on Mislabeled Samples To further show the effect mislabeled data on sample in- formativeness and demonstrate the capabilities of LSI in detecting mislabeled samples, we apply deliberate label noise (random label flips to any other class) to 10% of CIFAR-10 and pneumonia dataset samples and recompute LSI for all samples. Figure 5 shows that mislabeled samples, on average, have substantially higher LSI and thus provide more unique sample information. Notably, portion of mislabeled samples has higher LSI than even the most informative correctly labeled samples, indicating the disproportionate detrimental effect poor quality data and highlighting the importance of meticulous dataset curation (Zha al., 2023). Further experiments human mislabeled data are in Appendix E.
Paeumonia
CIFAR-10
1.5
Corrupted Labels
Deasity
1.5
Uncorrupted Labels
1.0
1.0
0.5
0.5
0.0
0.0
10
10-a
10-3
10-1
10a
10
10-3
10-a
10-1
Figure 5: LSI distribution on data with corrupted labels (mislabeled) vs. uncorrupted labels
LSI can furthermore effectively distinguish label corruption (mislabeling, Figure left) from data comuption (by including non-informative lorem ipsum text with random labels, Figure 6 right). Since lorem ipsum self-similar and provides little information (neither label confirming nor contradicting). the LSI is concentrated around an intermediate value, whereas the mislabeled text has higher LSI, as it is more informative to the model (strong contradictory information).
Sample Informativeness Increases with Smaller Datasets Interestingly, the LSI for the ImageNet subsets is about one order of magnitude larger than for CIFAR- 10 when using the same experimental setup (same training parameters and model), seen in Figure 1. Due to the smaller dataset size (9 469 compared to 50 000), individual samples of the ImageNet subsets provide more information to the neural network than individual samples of CIFAR-10. This result is corroborated by picking a subset of the same dataset (see Appendix F). Therefore, with smaller datasets, the presence or absence of individual samples becomes more influential on the model parameters.
7

---

Published as a conference paper at ICLR 2025
IMDb 20% corrupted
IMDD 20% Larem Impsum
Corrupted (Random) Uncorrapted Labels
Corrupted
Donsity
Density
(Lorem)
0.50
Uncorrapted Data
0.25
10
10-a
10-1
10-3
10
10-1
LS1
LSI
Figure 6: LSI in supervised text classification on the IMDb dataset using BERT
Establishing Class-wise Informativeness Ordering using LSI We next investigate the class-wise distribution of LSI, which allows for reasoning about which class imparts the most information to the model parameters during model training. Figure 7 (left) shows the class-specific LSI for the three different classes of the pediatric pneumonia dataset. We consulted a diagnostic radiologist who confirmed that this class ranking corresponds to the difficulty humans have in classifying these pathologies in children. In particular, while classifying radiograph abnormal is easy, distinguishing between bucterial and viral pneumonia is difficult, which mirrors the relationship between the distributions in the figure. To further emphasize the capabilities of LSI to assess class informativeness, we investigate the change in class-level distributions LSI on the pncumonia dataset when the feature embeddings of one class are corrupted with additive Gaussian noise (variance as data noise in Figure 7). Note that adding noise to the embeddings increases variance and makes the samples seem "less similar". This is different from Figure 6 right, where the corrupted data is highly self-similar and has little effect on LSI. Here, since the model is required to exploit more of the information contained in every sample, the individual samples become more informative; correspondingly, with higher added noise, the LSI distribution of the corrupted class shifts towards higher values.
Data Noise
Data Noise 0.5
Data Noise 1.0
Data Noise 5.0
1.0
Bacterial
Normal
Density
0.5
0.0
10 10 10" 10?
10-2 10" 102 LSI
10-2 10a 10a LSI
10"410-2 10-2 10" 10a
LS1
LSI
Figure 7: LSI distributions across three classes of the pneumonia dataset. By increasing the additive Gaussian noise standard deviation (scaled the initial standard deviation of the embedding) the normal class, their dissimilarity increases, requiring the model to extract more of their unique information its parameters. This increase informativeness leads higher class-wise LSI.
Measuring Dataset Difficulty using LSI Comparing the LSI distribution between datasets allows us to reason about the overall dataset's difficulty. For example, samples of CIFAR- 100 have, on average, higher LSI than the samples of CIFAR-10 10 when using the same experimental setup (see Figure 1). This indicates that CIFAR-100 contains more samples that carry higher unique information than CIFAR-I 10 which aligns well with CIFAR-100 having more classes and thus fewer samples per class than CIFAR-10 and being generally considered more difficult. The same phenomenon arises when comparing the distributions of the ImageNet subsets with the easier Imagenette having a lower average LSI and slimmer tail than the more difficult Imagewoof.
4.3 LSI AS PROXY FOR SAMPLE LEARNABILITY AND OUT-OP-SAMPLE ACCURACY
Sample Learnability As discussed above, sample difficulty has been previously used as a proxy for sample informativeness (Agarwal al., 2022). We next show that this relation is indeed justified by
8

---

Published as a conference paper at ICLR 2025
demonstrating that the "reverse direction" also holds, i.e. that sample informativeness also predicts sample difficulty, which we define as the sample-wer wise training accuracy. To this end. we partition each dataset into label-stratified subsets of 1/3 of the full dataset size. Note that, since choosing the subsets based on LSI alone may introduce class imbalances, as classes are not uniformly distributed with respect to their LSI, construct stratified subsets by combining data with respect to their class-specific LSI orderings. For instance, the subset containing samples with the highest LSI contains the 1/3 of samples of each class carrying the highest LSI. While show the results on CIFAR-I0 here, these results generalize to all other investigated datasets, which are shown in Appendix J.
Following the hypothesis that more informative samples are harder learn (i.e., classify accurately). we define sample learnability as the probability that model correctly assigns the label during training (equivalently to sop-laacurrcc). We observe that the learnability follows the ordering of the subsets with respect to their LSI (Figure 8 left): models trained on subset of low LSI samples achieve substantially higher training accuracy than models trained on subsets with medium or high LSI. Thus, LSI effectively serves as means of ordering samples with respect to their learnability for the model.
Out-of-Sample Accuracy While learnability assesses training accuracy, that is, the ability of model to fir, it is also (or perhaps more) important to assess how valuable the information gleaned from training samples for predicting unseen samples, i.c. out-of-sample (test) accuracy.
Regarding test accuracy (Figure 8 right), the models trained intermediate LSI data exhibit the best test accuracy and smallest train/test gap. Samples with low LSI have (ncar-)perfec train accuracy but reduced test accuracy, while high LSI samples not fitted well (low train accuracy) but tend to generalize better than their training performance. This indicates that intermediate and high LSI samples form the dataset partitions generally associated with learning generalizable representations and benign memorization (as defined by Feldman (2020)). In contrast, low LSI samples seem to be disposable or even harmful generalization (i.e., the model overfits on of low LSI samples).
1.0
1.0
High LSI
0.8
Tost Accuracy
0.8
Mid LSI
1rain Accuracy
Low LSI
0.6
0.6
Dummy Baseline
0.4
0.4
0.2
0.2
0.0
0.0
100
200
300
400
500
0
100
200
300
400
500
Epochs
Epochs
Figure 8: Training accuracy (left) and test accuracy (right) of models trained on subsets of CIFAR-10 containing 1/3 of the complete dataset with the highest, intermediate, and lowest LSI values compared to the dummy baseline of a model predicting the majority class.
4.4 TRANSLATING LSI BETWEEN PROBE AND FULL MODEL
So far, considered the LSI computed probe consisting of fully connected layer acting the features computed offline by frozen R8sNet-18. This allows for the computation of the LSI for each sample in the dataset in a short amount of time while keeping the respective Hessian tractable. LSI allows the establishment of an ordering of samples with respect to the information they impart in the model parameters. As shown previously, this orders the data from the most typical samples of a respective class (on which the probe model overfits) to the most atypical samples (which the probe incapable of fitting well). As LSI a sample-specific property, we expect an (approximately) consistent ordering of samples when training model directly on the underlying data (rather than on the feature embeddings). To investigate this hypothesis, we repeat the experiment described in Section 4.3 CIFAR-I0 by training ResNet-9 the dataset partitions established by the probe. As shown in Figure 9, the same pattern emerges as when the probe is used concerning the sample learnability and generalization capabilities: the low LSI subset is easy fit but generalizes just slightly worse than the mid LSI subset, and the high LSI subset is hard to fit and the worst in terms of test accuracy. However, the ResNet-9 model has more learnable parameters compared the probe

---

Published as a conference paper at ICLR 2025
model, it is capable of overfitting on the high LSI data. Interestingly, overfitting on high LSI samples occurs later in training than overfitting on lowinternediate LSI samples. The sample informativeness ordering derived by the probe generalizes other datasets and other model architectures and is consistent. Further results are shown in Appendix J. Additionally, this ordering emerges very early during training and does, in fact, not require the model to be fully converged (Appendix G).
1.0
0.8
0.8
Train Accuracy
Test Accuracy
0.6
High LSI
0.4
Mid
0.4
Low LSI
0.2
Dummy Baseline
0.0
0.0
0
100
200
300
400
100
200
300
400
Epochs
Epachs
Figure 9: Training accuracy (left) and test accuracy (right) of a ResNet-9 trained (from scratch) on subsets of CIFAR-10 containing 1/3 of the complete dataset with the highest, intermediate and lowest LSI values and compared to the dummy baseline of model always predicting the majority class.
4.5 ADDITIONAL FINDINGS AND RESULTS
refer to the Appendix for a comparison of LSI with other methods (Appendix A, Appendix B) and additional experiments investigating Hessian approximations (Appendix C); the correlation between LSI from probing vs. full models (Appendix D); effect humanly mislabeled data on LSI (Appendix E); effects of dataset size LSI (Appendix F); the robustness of LSI throughout training (Appendix G); its relationship with Differential Privacy (DP) (Appendix H); its computation via the Riemann Laplace approximation (Bergamin al. (2024), Appendix I), transferring LSI from the probe to other architectures (Appendix J), as well as for a collection of samples with low and high LSI for selected datasets (Appendices K to M).
DISCUSSION AND CONCLUSION
We introduced Laplace Sample Information, measure of information flow from individual samples to the parameters of a neural network, i.e. of the information a training sample contributes to the final model. We demonstrated the capabilities of LSI a tool predict typical/atypical data, with atypical samples carrying more unique information, and for detecting out-of out-of-distribution and mislabeled samples. Furthermore, we established LSI as a class-wise informativeness measure, which correlates with human perception and underscores the utility of LSI for reasoning about dataset difficulty. Therefore, we expect LSI to useful measure sample- wise informativeness in various areas of ML research. Compared to recently proposed informativeness measures, LSI does not require linearization and is much more scalable than Harutyunyan et al. (2021). Moreover, applies to arbitrary architectures and task-agnostic (e.g. to self-super learning with CLIP) contrary to Wongso et al. (2023). For a detailed comparison to these techniques, see Appendix A.
Due to the high computational cost of LOO retraining, less, the ordering properties defined by LSI on the probe translate effectively to larger models. We note that using probe proxy remains design choice favouring efficiency; the LSI formalism is model-agnostic and can be used with any Hessian approximation. As shown in Appendix C, LSI can be used with memory-efficient (K-FAC); thus anticipate that it can transfer LLM-scale workflows (Grosse et al., 2023) which we intend to investigate in the future.
employ probe to compute LSI. Nonethe-
In conclusion, LSI combines model- and data-centric approaches to estimate sample informativeness. We foresee an important role in combining metrics like LSI with characterizations determined through human reasoning to facilitate model and dataset introspection, for example, in topics like algorithmic fairness or AI safety.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, llya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp. 308-318, 2016.
Chirag Agarwal, Daniel D' Dosuuza, and Sara Hooker. Estimating example difficulty using variance of gradients. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Hilal Asi and John C Duchi. The importance of better models in stochastic optimization. Proceedings of the National Academy of Sciences, 166/46)22992-222990, 2019.
Juhan Bae, Guodong Zhang. and Roger Grosse. Eigenvalue corrected noisy natural gradient. arXiv preprint 1rw.v:1811.1255555111111 2018
Juhan Bae, Nathan Ng. Alston Lo, Marzyeh Ghassemi, and Roger B Grosse, influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953- 2022.
Rachit Bansal, Danish Pruthi, and Yonatan Belinkov. Measures of information reflect memorization patterns, Advances in Neural Information Processing Systems, 35:18472-18486, 2022.
Raef Bassily, Kobbi Nissim, Adam Smith, Thomas Steinke, Uri Stemmer, and Jonathan Ullman. Algorithmic stability for adaptive data analysis. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. 1046-1059, 2016.
Samyadeep Basu, Philip Pope, and Soheil Feizi. Influence functions in deep learning are fragile. arXiv preprint arXiv:2006.14653 14651, 2020.
Federico Bergamin, Pablo Moreno-Muñoz, Søren Hauberg, and Georgios Arvanitidis. Riemannian laplace approximations for bayesian neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Nicolas Brosse, Alain Durmus, and Eric Moulines. The promises and pitfalls of stochastic gradient langevin dynamics. Advances in Neural Information Processing Systems, 2018.
Zhiqi Bu, Hua Wang, and Qi Long. On the convergence and calibration of deep learning with differential privacy. arXiv preprint arXiv:2106.07830, 2021.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization gradient descent-based optimization. arXiv preprint arXiv:2002.10657, 2020.
Dingfan Chen, Raouf Kerkouche, and Mario Fritz. Private generation with discriminative information. Advances in Neural Information Processing Systems, 35: 16678-1600,,, 2022.
Lenaic Chizat, Edouard Oyallon, and Francis Bach. On lazy training in differentiable programming. Advances in neural information processing systems, 32, 2019.
Rachel Cummings, Katrina Ligett, Aaron Roth, Zhiwei Steven Wu, and Juba Ziani. Accuracy for sale: Aggregating data with variance constraint. In Proceedings the 2015 conference on innovations in theoretical computer science, pp. 317-324, 2015.
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace refux-effortless bayesian deep learning. Advances in Neural Information Processing Systems, 34:20089-20100 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Jacob Devlin, Ming-Wa Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: bidirectional transformers for language understanding, 2019, URL https: // /rrxiv.rgg abs/1810. 04805.
Pre-training of deep
11

---

Published as a conference paper at ICLR 2025
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. CoRR, abs/2010. 1929, 2020. URL htwps:ttp///.. / arxiv, https////wwwww../////a//b////// 2010 11....9999 1192..
Cynthia Dwork, Vitaly Feldman, Moritz Hardt, Toni Pitassi, Omer Reingold, and Aaron Roth. Generalization in adaptive data analysis and holdout reuse. Advances in Neural Information Processing Systems, 28, 2015.
Kawin Ethayarajh, Yejin Choi, and Swabha Swayamdipta. Understanding dataset difficulty with V-usable information. In International Conference on Machine Learning, pp. 5988-6008. PMLR, 2022.
Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: Deep bayesian neural nets do not need complex weight posterior approximations. Advances in Neural Information Processing Systems, 33:4346-4357, 2020.
Vitaly Feldman. Does learning require memorization? a short tale about a long tail. In Proceedings of the 52nd Annual ACM SIGACT Symposium on Theory of Computing, pp. 954-959, 2020.
Vitaly Feldman and Thomas Steinke. Calibrating noise variance in adaptive data analysis. In Conference On Learning Theory, pp. 535-544. PMLR, 2018
Emilio Ferrara. The butterfly effect in artificial intelligence systems: Implications for ai bias and fairness. Machine Learning with Applications, 15:100525, 2024.
Bishwamittra Ghosh, Debabrota Basu, and Kuldeep Meel. "how biased are your features?":
Computing faimess influence functions with global sensitivity analysis. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, pp. 138-148, 2023.
Ziv Goldfeld and Kristjan Greenewald. Sliced mutual information: A scalable measure of statistical dependence. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 17567-17578. Cur- Associates, Inc., 2021. URL https: http::s://///cceeeeeeeeeoe ptpp paper/2021 /ft.flllle 92 4 661 (0385f6681110 ff
neur ps ww/ppppr.pppppccct.ppt.
files/
Roger Grosse and James Martens. A factooee approximate fisher matrix for convolution layers. International Conference on Machine Learning, pp. 573-582, PMLR, 2016.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with influence functions. arXiv preprint (rX.i::330..0296 2023.
Hrayr Harutyunyan, Alessandro Achille, Giovanni Paolini, Orchid Majumder, Avinash Ravichandran, Rahul Bhotika, and Stefano Soatto. Estimating informativeness of samples with smooth unique 2021.
information. arXiv preprint arXiv:2101.0666
Fengxiang He, Tongliang Liu, and Dacheng Tao. Control batch size and learning rate to generalize well: Theoretical and empirical evidence. Advances in neural information processing systems, 32, 2019.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
S Hooker, A Courville, G Clark, Y Dauphin, and A Frome. What do compressed deep neural networks forget? arxiv. Learning, 2020a.
Sara Hooker, Nyalleng Moorosi, Gregory Clark, Samy Bengio, and Emily Denton. Characterising bias in compressed models. arXiv preprint arXiv:2010.03058, 2020b.
Jeremy Howard. Imagenette, 2019. URL h http:///gggggabccc///aaaaaaaaaaa
https: //ththhhttth//tt/////// htt.uu...oo.aatttteeeeeeee wtwm/Etstaa//mmmgeeeetttttttttte
12

---

Published as a conference paper at ICLR 2025
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural regularities of labeled data in overparameterized models. arXiv preprint arXiv:2002.03206 2020.
Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pp. 2525-2534. PMLR, 2018.
Daniel S. Kermany, Michael Goldbaum, Wenjia Cai, Carolina C.S. Valentim, Huiying Liang, Sally L. Baxter, Alex McKeown, Ge Yang. Xiaokang Wu, Fangbing Yan, Justin Dong, Made K. Prasadha, Jacqueline Pei, Magdalene Y.L. Ting, Jie Zhu, Christina Li, Sierra Hewett, Jason Dong, Ian Ziyar, Alexander Shi, Runze Zhang, Lianghong Zheng, Rui Hou, William Shi, Xin Fu, Yaou Duan, Viet A.N. Huu, Cindy Wen, Edward D. Zhang, Charlotte L. Zhang, Oulan Li, Xiaobo Wang, Michael A. Singer, Xiaodong Sun, Jie Xu, Ali Tafreshi, M. Anthony Lewis, Huimin Xia, and Kang Zhang. Identifying medical diagnoses and treatable diseases by image-based deep learning. Cell, 122-5)111221133..1 February 2018 ISSN 0092-86744 doi: 10. (011//.cell.2018.0222 http:/ /xxp doi. org/10 101011100 6/ cell. 2018. 02. 010.
URL
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barnskaakaaiikka, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3520 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report O, University of Toronto, Toronto, Ontario, 2009. URL https:://ww/ cs. toronto.. wdw/-kriz/Leaminnge. features-eeeeee-eeeeeeeeeeeeeeee -oseeess--0------- pdf.
Qianxiao Li, Cheng Tai, and E Weinan. Stochastic modified equations and adaptive stochastic gradient algorithms. International Conference on Machine Learning, pp. 2101-2110. PMLR, 2017.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015. URL hwwpthtttt///ett /ttttss///aiivviii org/ abs / /1405. 0312.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisties: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www, wwww.aclle. web. www.//taggooogoooooooooooooooooo /tgpp//////......111111111111111
David J. C. MacKay. Bayesian Interpolation. Neural Computation, 4(3): 447, 05 1992. ISSN 0899-7667. doi: 10.1162/neco.1992444155 URL httpps://det wtt.:/..rrroo..1 100 wttpo////...........e.e.e 1992.4.3.20 415..
Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In International Conference on Machine Learning, pp. 15630-15649, PMLR, 2022.
Ilya Mironov. Rényi differential privacy. In 2017 IEEE 30th computer ccurity symposium (CSF), pp. 263 275. IEEE, 2017.
Jeffrey Negrea, Mahdi Haghifam, Gintare Karolina Dziugaite, Ashish Khisti, and Daniel M Roy. Information-thheoreiee generalization bounds for sgld via data-dependent estimates. Advances in Neural Information Processing Systems, 32, 2019.
Kobbi Nissim and Uri Stemmer. On the generalization properties of differential privacy. CoRR, abs/l 2015.
13

---

Published as a conference paper at ICLR 2025
Guillermo Ortiz-Jimenez, Seyed-Mohsen Moosavi- and Pascal Frossard. What can linearized neural networks actually say about generalization?
In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neu- ral Information Processing Systems, volume 34, pp. 8998-9010. Curran Associates, Inc., 2021. URL https::/pproeeeeeeeee
neurips www.ips.cc/pppp...epeeeeee httpppee_ffllspppprrrreee
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior scale. arXiv preprint arXiv:2303.14186, 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL https:
//arxiv. otg/ abs 2aabs/2100 221..21...000 00020.
Maxim Raginsky, Alexander Rakhlin, Matthew Tsao, Yihong Wu, and Aolin Xu. Information- theoretic analysis stability and bias of learning algorithms. 2016 IEEE Information Theory Workshop (ITW), pp. 26 30. IEEE, 2016.
Mohamad Rida Rammal, Alessandro Achille, Aditya Golatkar, Suhas Diggavi, and Stefano Soatto. On leave-one-out conditional mutual information for generalization. Advances in Neural Information Processing Systems, 15:10179-10199, 2022.
Ryan Rogers, Aaron Roth, Adam Smith, and Om Thakkar. Max-information, differential privacy. and post-selection hypothesis testing. In 2016 IEEE 57th Annual Symposium on Foundations of Computer Science (FOCS), pp. 487-494 IEEE, 2016
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Inielligence, volume 36, pp. 8179-8186 2022.
Andrea Schioppa, Katja Filippova, Ivan Titov, and Polina Zablotskaia. Theoretical and practical perspectives on what influence functions do. Advances Neural Information Processing Systems, 36, 2024.
Thomas Steinke and Lydia Zakynthinou. Reasoning about generalization via conditional mutual information. In Conference on Learning Theory, pp. 3437-3452. PMLR, 2020.
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint /rXiv:/966.02220 2019.
Haoru Tan, Sitong Wu, Fei Du, Yukang Chen, Zhibin Wang, Fan Wang, and Xiaojuan Qi. Data pruning via moving-one--amplee-ut. arXiv preprint arXiv:2310. 14664, 2023.
Ryan Therrien and Scott Doyle Role of training data variability on classifier performance and generalizability. In Medical Imaging 2018: Digital Pathology, volume 10581, pp. 58-70. SPIE, 2018.
Ayge Ünsal and Melek Önen. Information cooomation-thhoretiie approaches to differential privacy. ACM Comput. Surv. 56(3), oct 2023. ISSN 0360-0300. doi: 10. 10.1145/3604904 URL httpt:://ti..oooeteeetottt 1145/360490 3044904.
Yu- Yu-Xiang Wang. Per-instance diffferential privacy. arXiv preprint arXiv: 2017.
Jiaheng Wei, Zhaowei Zhu, Hao Cheng, Tongliang Liu, Gang Niu, and Yang Liu. Learning with noisy labels revisited: A study using real-world human annotations. arXiv preprint 1Xi:02110.11088. 2021.
Yeming Wen, Kevin Luk, Maxime Gazeau, Guodong Zhang, Harris Chan, and Jimmy Ba. An empirical study of stochastic gradient descent with structured covariance noise. In International Conference Artificial Intelligence and Statistics, pp. 3621-3631 PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Shelvia Wongso, Rohan Ghosh, and M. Motani. Pointwise sliced mutual information for neural net- work explainability. 2023 IEEE International Symposium on Information Theory (ISIT), pp. 1776- 1781, 2023. URL https::///// http://.......nnssssnasieieeeeee semanticscholar org/ CorpusID 2210811111
Yinjun Wu, Edgar Dobriban, and Susan Davidson. Deltagrad: Rapid retraining of machine learning models. In International Conference on Machine Learning, pp. 10355-10366, PMLR, 2020.
Heng Xu, Tianqing Zhu, Lefeng Zhang, Wanlei Zhou, and Philip S Yu. Machine unlearning: A survey. ACM Computing Surveys, 56(1):1-36, 2023.
Yilun Xu, Shengjia Zhao, Jiaming Song, Russell Stewart, and Stefano Ermon. A theory of usable information under computational constraints. arXiv preprint arXiv:2002.10689, 2020.
Jiayuan Ye, Anastasia Borovykh, Soufiane Hayou, and Reza Shokri. Leave-one-out distinguishability in machine learning. arXiv preprint arXiv:2309.17310, 2023.
Da Yu. Gautam Kamath, Janardhan Kulkarni, Tie- Yan Liu, Jian Yin, and Huishuai Zhang. Indi- vidual privacy accounting for differentially private stochastic gradient descent. arXiv preprint 12Ni::2206602611, 2022.
Daochen Zha, Zaid Pervaiz Bhat, Kwei-Hemg Lai, Fan Yang. Zhimeng Jiang, Shaochen Zhong, and Xia Hu. Data-centric artificial intelligence: A survey. arXiv preprint arXiv:2303.10158 2023.
Zhanxing Zhu, Jingfeng Wu, Bing Yu, Lei Wu, and Jinwen Ma. The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects. arXiv preprint arXiv: 00195, 2018.
Piotr Zielinski, Shankar Krishnan, and Satrajit Chatterjee. Weak and strong gradient directions: Explaining memorization, generalization, and hardness of examples at scale, arXiv preprint arXiv:2003.07422, 2020.
15

---

Published as a conference paper at ICLR 2025
A COMPARISON TO OTHER INFORMATIVENESS METRICS
As our work is an approximation of point-wise mutual information, we compare LSI with point-wise mutual information measures and bounds thereof. Since mutual information hard to compute in high-diimensionl settings such as deep learning, "slicing" has gained traction as a scalable and efficient method for computing it, which retains some of the properties of the original MI (Goldfeld & Greenewald, 2021). We thus compare LSI to point-wise sliced mutual information (PSMI) (Wongso et al., 2023) and also to smooth unique information (SUT) (Harutyunyan et al., 2021) Figure 10. To our knowledge, these are the only methods performing noint-wise informativeness estimation based on mutual information, that claim to scale beyond toy datasets. We thus believe that these are only salient comparisons to other techniques from current literature.
Our comparison shows that, while SUI and PSMI are nearly uncorrelated, likely due to compounding approximation error, LSI is strongly correlated with PSMI and weakly with SUI, indicating that while both SUI and PSMI measure some aspect of sample informativeness neither of them seems to capture the full picture. LSI is positively correlated with PSMI, which is result of the relationship between the KI. divergence and point-wise mutual information mentioned in Section 3. Interestingly, the fact that sliced MI is computational approximation of mutual information and lacks some of the inequality)
guarantees inherent to the "true" MI (notably, the absence of the information-processe seems somewhat diminish PSMI's expressiveness. LSI, on the other hand, seems suffer less from the approximation of the Hessian (see Appendix C). Thus, LSI and PSMI measure distinct but related quantities, and both detect the same underlying structure in the data. In terms of SUI, the metric relies on a global linearization of the entire network, drastic intervention that can make the model fragile (Ortiz-Jimenez et al., 2021). Moreover, SUI extremely memory-inefficient and does not scale to larger datasets (the SUI paper limited its evaluation to binary classification datasets with ca. 1000 samples). LSI is much more flexible regarding the choice of Hessian approximation and Laplace approximation, does not intervene the model weights, and does not have the same memory inefficiency problem. Our comparison is thus based on a 1000 sample three class subset classification task of CIFAR-10.
11-9
82
-22
-8.
-2R
01
2.0
0.5
1.0
2.5
24
02
6.
1.2
1.5
2.0
2.5
3.0
35
-2.
-0.6
-24
-0.3
6.0
02
Beg)
Bog!
PSN
Spearman's FC 0.60
Speammrna 0.29
Sparrnat' 0.1.1
Figure 10: Correlation plots between LSI, Smooth Unique Information (SUI) and Point-wise Sliced Mutual Information (PSMI) on a CIFAR-I0 subset. While LSI and PSMI are strongly correlated, SUI and LSI exhibit weaker correlation, likely due the limitations of SUI. SUI and PSMI are nearly uncorrelated, likely due to compounding approximation errors.
16

---

Published as a conference paper at ICLR 2025
B COMPARISON OF LSI WITH TRAK
To show the differences between leave-one-out retraining-based data valuation metrics to influence estimation, compare LSI with TRAK (Park et al., 2023) and discuss the results. Note that while both methods perform data attribution, they differ in their fundamentals. LSI measures the informativeness of individual samples to the weights of the trained neural network. At the same time, TRAK aims to measure the influence of individual samples on correctly predicting sample of interest. Therefore, LSI provides singular value for each sample the train set, while TRAK provides N x M tensor of values with N and M being the sample count of the train and test set, respectively. To compare these two measures we assign the datapoinnts the train set the average of their TRAK values across all test datapoints.
0.025
0.015
0.010
0.005
0.000
.0.010
0.25
1.25
1.75
Figure 11: Correlation with confidence interval of 0.95 of LSI with TRAK
Figure 11 shows weak correlation (Spearman's Rank correlation of 0.15) between LSI and TRAK scores assigned to each datapoint in the Imagenette dataset.
To further compare and TRAK, we perform a qualitative analysis of samples assigned the highest and lowest values of these measures. Figure 12 shows that LSI separates the dataset into samples of little unique information (less typical) carrying low LSI and samples of large unique information (visually dominated by people. mislabeled out of distribution) of high LSI. Opposingly, TRAK fails at this, with out of distribution images like the car carrying average TRAK scores. Moreover, the images of low and high TRAK scores do not substantially differ, both show clear images of radios, however with white background for high TRAK scores. Therefore, we conclude that while TRAK may be meaningful in assessing sample-wise cross-influence, fails data attribution measure specific to a sample.
17

---

Published as a conference paper at ICLR 2025
Low LSI 6.%%
High LSI
LOS COL
Lo
1MA AAT
18A6 24.2%
TME ZL-W
ANN JELOO
ANG ALLA
JANK Saa
TMC
1646: 22.39
7.4%
243%
1100%
AAN LIVTT
IN 2V%
TM 7VR
IN WOUT
153
TRAK
High TRAK scores
LGS
SAUS
Lis 27.4%
30.% TRUR
LA EL.%%
LGS 32.3%
DA
LGI TO
12.30.% "ENE
LG 9G.
TRMM 235
TIME
99.0%
TEAE 90.9
90.99
Figure 12: Exemplary images of the highest and lowest LSI and TRAK for the Radio class in Imagenette. The images are labeled with corresponding percentiles of them in the measure-ordered dataset.
18

---

Published as a conference paper at ICLR 2025
C APPROXIMATION OF THE HESSIAN
Due to computational constraints, we refrain from using the full Hessian of the parameters of the neural network compute the LSI and use diagonal approximation. Figure 13 shows that LSI computed on the diagonal approximation of the Hessian, LSI computed on the KFAC approximation of the Hessian and LSI computed on the full Hessian correlate strongly with Spearman's is R 0.9.
6.000
8842 Hodian
2,015
3.015
13 LSie
0,000
1.000
1.00
6.000
0.025
2,000
0.315
1.000
0,325
1.000
0.333
3.000
6-155
0.20
1,25
3.000
0.035
0.032
2,15
0.33
Speamion 5 0.92
Stcoman's R.
Sp-smarrts 0.99
Figure 13: Correlation between ISSI on the full Hessian, on the diagonal approximation and the KFAC approximation of the Hessian. All approximations correlate strongly with LSI computed on the full Hessian. LSI is computed using the last-layer probe on CIFAR-10
D PROBE MODEL APPROXIMATION OF LSI
While LSI does not put any constraints on the model is computed on, training large models until convergence in leave-on-out setting computationally very expensive. Thus, employ probe model, that acts upon the features probed from a pre-trained feature extractor. To indicate the applicability of this probing approach, show the correlation of scores computed on 400 samples of CIFAR-10 for LSI computed on all parameters of a CNN (3 convolutional layers with one fully connected layer head), last-layer parameters of the CNN and the probe model. Note, all except the last require the retraining of the whole model for each data point. Computing LSI on the probe solely requires the retraining of the probe, which can be done in a small fraction of the time. Table shows that LSI computed using the probe model is an excellent approximation to LSI computed on full CNN with a Spearman's s R > 0.9.
Probe-Model Parameters
Last-Layer Parameters
Full CNN Parameters
0.93
0.98
Last- Layer Parameters
0.90
Table 1: Spearman's 's R between LSI computed on all parameters of CNN, the last layer parameters and probe model.
19

---

Published as a conference paper at ICLR 2025
E HUMANLY MISLABELED EXAMPLES
To extend our experiment on the effect of mislabeled data on LSI, we conduct further experiments on CIFAR-ION with the aggregated labels setting, which assigns the label datapoint according to a majority vote across several human labellers (Wei et al., 2021). CIFAR-100 introduces set of human labels the original CIFAR-10 data. With this, it also introduces human mislabeling. Other than the previously considered setting of label flipping, these mislabels should be closer to the correct label. Exemplarily, while with label flipping. an image showing a dog on grass and thus initially labeled as dog is equally likely to be labeled an airplane or deer. With CIFAR-1ON, the image of the dog will more likely to be mislabeled as deer than as an airplane.
Figure 14 (left) shows the distribution of LSI across correctly labeled samples with human labeling error. As with label flipping (Figure 14 (right)), the individual information mislabeled samples contribute to the weights of the model is higher than for correctly labeled samples measured with LSI.
However, this increase in LSI due to mislabeling is smaller than for label flipping. Consequently, mislabeling due to human misinterpretation results in data-label pairs that provide less unique information than label flipping. This follows intuition that, exemplarily, brown figure dog) on a green landscape labeled as deer is less unique than it were labeled as an airplane.
CIPAR-1ON
Label
Corrupted Uncaerupted Labels
1.5
Corrupted Data
3.0
(Human Label Error)
1.0
Uncorrupted
0.5
0.0
0.0
10-1
10-3
10-2
10-1
10-4
10r1
LSI
1SI
Figure 14: LSI distribution on data with corrupted labels (mislabeled) vs. uncorrupted labels with human mislabeling (left) and label flipping (right)
F LSI INCREASES WITH SMALLER DATASETS
To substantiate the findings of Section 4.2, we compare the distribution of LSI on CIFARIO and on subset of CIFARIO of 1/5 the original size (Figure 15). As discussed previously, with smaller sample count in the individual datasets, the information individual samples contribute to the parameters increases (in this case, by around one order of magnitude).
CIFAR10
CIFAR10 Subset
100
Density
5
0.00
0.04
0.06
0.08
0.0
0.1
0.4
LSI
LSI
Figure 15: LSI distributions of CIFARI0 and subset of 10. The LSI computed on the subset about one magnitude larger, with fewer samples, individual samples are required be more informative to the model.
20

---

Published as a conference paper at ICLR 2025
G APPLICABILITY OF LSI BEFORE MODEL CONVERGENCE
To bound conditional point-wise mutual information, LSI requires the underlying model to be converged. However, in leave-one-out retraining setting, this can be computationally expensive. We therefore investigate the robustness of sample order as imposed by LSI throughout the training. Figure 16 shows, that after an initial warm-up phase, the sample ordering remains largely consistent. This indicates, that the early stage models approximate the ordering well.
244
Spearman Rank Cornllatton opoch
2.0
0.6
0.4
500
Epochs
Figure 16: Correlation of the LSI sample ordering between the last epoch (1000) and during training (blue curve). After a warm-up period, the ordering becomes and remains consistent.
H LSI UNDER DIFFERENTIAL PRIVACY
LSI quantifies the information flow from an individual sample to the neural network throughout the training process. DP applied to model training through DP stochastic gradient descent (DP-SGD) (Abadi et al., 2016) offers means of limiting the influence exerted by any single sample on the parameters of the neural network by clipping and applying noise the per-sample gradients. LSI lends itself natural choice for assessing the impact of individual samples on differentially private training as well as the effect of DP on the contribution each sample to the neural network's 's training process, which we investigate here. Note that, as discussed in Section 2, the KL divergence plays an important role in assessing algorithmic stability, and DP implies a strong stability condition, reinforcing this link.
Differential Privacy Background To show the strong connection between LSI and DP, we recall the definition of Rényi differential privacy (RDP) (Mironov, 2017)
Definition 3 ((o, <)-RDP). A randomized mechanism f : 'D - R for R as the co-domain of f holds that
satisfies (a, RDP for all adjacent D,
Da (f (D) (D') S.
Above, the adjacency between D and is defined as the two databases differing by single entry, which be through adding, removing, replacing single sample. RDP is based the Rényí divergence, given as:
Definition (Rényi divergence). For two probability distributions and Q the Rényi divergence of order a >> 1 is
((x))
a
D (P Q)
A 1
log E B
The sensitivity of function is an important measure of DP. It refers to the maximal change the output of function can experience between adjacent databases and determines the magnitude the additive noise.
Definition 5 (Global Sensitivity). Given function f D + R for R as the codomain of f and all adjacent databases as defined previously D, D' the global sensitivity with respect the p-norm is:
GS(f) sup [lf f(D' D,D
21

---

Published as a conference paper at ICLR 2025
Definition 6 (Local Sensitivity). Given function f: - R for R as the codomain of f and specifie databases D E 'D and its adjacent databases D' E D the local sensitivity is with respect to the p-norm is:
LS D)
(
D
DP operationalized in DL through DP-SGD (Abadi et al., 2016). essence, DP-SGD different from SGD as it privatizes the minibatch (called lot in DP) gradients by:
1. Bounding the global sensitivity of SGD by clipping the per-sample gradients gi to a pre- determined by the clipping norm C' before aggregating them across the B samples contained in the lot to clipped gradient Ielip
gi/ max (I, |ealle
Jelip
M i=0
(5)
2. Adding zero-centered Gaussian noise with a noise multiplier a to the minibatch gradient
Spriv = Ielip + E, E ~ ~N(, 62)
(6)
Relation between LSI and DP With a + 1, D. (P continuity. Intuitively. LSI thus effectively measures (ler instance RDP post hoc, where the privacy of single sample (instance) expressed with respect et al., 2022). Note that the DP guarantee of DP-SGD is diven with respect to the released gradients, but since the KI. (and the Rényi) divergence both satisfy the data processing inequality (equivalently, DP is closed under post-processing). the effect translates to the model parameters. To thoroughly investigate the impact of DP on the LSI and thus on the information flow the individual samples to the trained neural network, we examine the effects of clipping and noising separately. As the additive noise in DP-SGD introduces variance in our training setup, we average the results DP-SGD training on CIFAR-10 10 across five different seeds.
(L Q). which is justified by
fixed dataset (Wang, 2017: Yu
Results With an increasing clipping norm, the LSI and, thus, the contribution of individual samples the final parameters of the neural network decreases (Figure 17). Consequently, analyzing the LSI offers empirical support for the mathematical description that clipping gradients in DP-SGD restrict the global sensitivity and, thus, the per-sample gradients, thereby bounding the maximal information a sample can contribute to the final parameters of the model.
Notably, we show that reducing the clipping threshold results in the LSI becoming more similar across samples, with the LSI across all samples becoming less variant (Figure 17). Thus, with a decreased clipping threshold, the samples in the training data more evenly contribute their information the neural network parameters. This aligns with previous findings showing the importance of dataset variability on generalization (Therrien & Doyle, 2018) and the beneficial role of DP for generalization (Nissim & Stemmer, 2015).
Our experiments empirically show that adding zero-centered Gaussian noise scaled with respect to the maximum gradient norm of the gradients has no observable influence the distribution of LSI the dataset (Figure 17). While the clipping of individual gradients only influences those samples whose gradients are larger than the clipping threshold, thereby changing the amount of information individual samples contribute to the training, the noise gets added to all samples. The noise increases the variance in the per-sample gradients. However, their average contribution to the neural network parameters remains unchanged. This aligns with the findings of previous research, showing adding noise solely influences the privacy risks of training the model, but not the convergence or calibration, while the gradient clipping solely influences the convergence and calibration of the neural network (Bu et al., 2021).
22

---

Published as a conference paper at ICLR 2025
Clip: 0.01
Clip: 0.1
Clip: Clip: 5
Density
Density
5
2
1.5
1.0
0.5
0.0
10
10 9
10
10
10 1
Figure 17: The left plot shows the distribution the LSI of CIFAR-10 10 across varying per-sample gradient clipping norms with additive noise. The right shows the distribution of the LSI with clipping norm 0.1 and varying additive noise to achieve a (c, 10-P)-DP. With diminishing clipping norm, LSI decreases in magnitude and variance, while with increasing c, the distribution of LSI remains unchanged in magnitude and variance.
23

---

Published as a conference paper at ICLR 2025
I GENERALIZATION TO RIEMANN LAPLACE APPROXIMATION
Recently, a novel approximation of the posterior model distribution has been proposed in Bergamin al. (2024), which allows for sampling models while considering the shape the underlying loss landscape. While computing the LSI on this Riemannian Laplace approximation (RLA) is possible, requires fitting a distribution via Kernel Density Estimation the samples produced by the RLA. These distributions are then used to compute the LSI. We show, using the same sampling and fitting approach on the standard Laplace approximation, that while this process produces the correct LSI values for "intermediately" spaced parameter distributions (Figure 18), it breaks down when the distributions are very similar or very distant (Figure 19). Very similar distributions (which the setting in leave-onc-out retraining) require many samples to be drawn to capture the slight differences in distributions. Similarly, very distant distributions require inordinately many samples to capture the similarities in distributions. RLA additionally requires solving differential equation for each sample of model parameters. Generating sufficiently many samples to adequately describe the distributions is thus computationally infeasible, especially as the sampling would have to be performed each retraining cycle. Due to these reasons, we use the "traditional" Laplace approximation in our work.
10 KDE samples
50 KDE samples
100 KDE samples
500 KDE samples
0.25
0.45
KOE KDE- Laplare 1St
0.40
KDE-Laplace LSt
KDF-L Laplace
1.95
0.20
KDE-Lap KDE-Laplac aplace
0.35
KDB-1 Laplace
0.15
1.90
0.15
0.30
1.85
0.10
1.80
0.10
0.20
0.05
2.1
0.14
0.055
KDB RLA LSI
KDE-RLA REE-AA LST
0.30
KDE-RU CDE-RLALL LSI
0.050
KDE RLA
0.25
0.040
0.10
0.035
1.8
0.20
0.030
0.08
0.000
0.000
0,050
LSI
LSI
LSI
LSI
Figure 18: Correlation between LSI computed on the kernel density estimate drawn from Riemann Laplace approximation, the kernel density estimate drawn from Laplace approximation and LSI computed on the gaussian distributions estimated by the Laplace approximation without kernel density estimation. The figure shows LSI computed on distributions with intermediate LOO distribution shift.
24

---

Published as a conference paper at ICLR 2025
10 KDE samples
50 KDE samples
100 KDE samples
500 KDE samples
0.0850
0.0600
XDE: XDE-Laplace Lapl-apeee LSI
2.10
KDE-Laplace LSt
0.0825
(DP-laplaca 1.S
0.0575
Laplace
2.05
0.0800
0.0550
0.20
0.0525
2,00
0.0750
0.0725
0.19
0.0475
1.95
0.84
0.19
0.030
KDE REARRAA LST LSI
KDE-RLA LST
0.070
KDE-RI ADE-RLA LSS
0.82
0.38
KDE-RLA
0.028
0.80
0.17
0.026
0.060
0.024
0.78
0.16
0.0
2.5
5.0
0.0
2.5
5.0
0.0
2.5
5.0
0.0
2.5
5.0
16-6
16-6
16-6
16-6
Figure 19: Correlation between LSI computed the kernel density estimate drawn from Riemann Laplace approximation, the kernel density estimate drawn from Laplace approximation and LSI computed on the gaussian distributions estimated by the Laplace approximation without kernel density estimation. The figure shows LSI computed on distributions with small LOO distribution shift.
ADDITIONAL RESULTS ON TRANSFERRING LSI FROM LINEAR PROBE TO FULL MODEL
In this section, we provide additional results on transferring the LSI ordering computed on the probe (called proxy in the figures) to other architectures. We are considering the following models: the probe, an MIP, three-layer CNN, ResNet-9 and a ResNet-18 18 for CIFAR10 and CIFARIOG, and ResNet- 18 for the ImageNet subsets and the pneumonia dataset.
25

---

Published as a conference paper at ICLR 2025
CIFAR1O Proxy
1.0
1.0
Tost Accuracy
0.8
Train Accuracy
0.6
0.4
High LSI
Mid-LSt
0.2
Low LS1
Dummy Baseline
0.0
200
400
Epochs
Epochs
CIPARIO MLP
1.0
1.0
Tost Accuracy
Train Accuracy
0.6
0.4
High
LSI
0.2
Low LSI
0.0
Dummy Baseline
0.0
100
200
300
100
500
0
100
200
300
400
500
Epochs
Epochs
0.8
0.8
17am Acnuracy
Test Accurscy
0.6
High-
0.4
Mid LSI
0.2
Low LSI
0.2
Dummy Baseline
100
200
300
400
500
100
200
300
400
500
Epochs
Epechs
CIRAR1O ResNet-9
1.0
1.0
Iran Accuracy
0.6
0.6
High LSI
Test
Mid LS1
0.2
0.2
Dummy Baseline
0.0
Epochs
Epochs
CIPAR10 ResNet-18 18
1.0
1.0
0.8
17ain Accurscy
Accuracy
0.6
0.4
High LSI
0.4
Mid LSI
Low LSI
0.2
Dumny Baseline
0.0
100
200
300
400
100
200
300
400
Epochs
Epochs
Figure 20: Accuracy of LSI based subsets of CIFARIO across an MLP (flattened input), CNN, ResNet-9, R8sNet-18
26

---

Published as a conference paper at ICLR 2025
CIFAR100 ProxY
1.0
1.0
Train Accuracy
Jest Accuracy
0.6
0.6
High LS1
LS1
0.2
Lowr LS!
0.2
0.0
200
500
200
Epochs
Epachs
CIBARI00 MLP
1.0
1.0
0.8
Train Accuracy
0.6
0.4
High
Test
0.4
LSI
Low LSI
0.0
0.0
100
200
300
100
500
100
200
300
100
500
Epochs
Epochs
CIFAR100
0.8
0.8
17am Acnuracy
Test Accurscy
0.6
High LS1
0.4
Mid LSI
0.2
Low LSI
0.2
Dompy Paseline
100
280
300
400
500
100
200
300
400
500
Epochs
Epechs
CIEAR1 00 ResNet-9
1.0
1.0
Tost Accuracy
0.8
Iran Accuraccy
0.6
0.4
High LSI
Mid LS1
0.2
Low LSI
Dumayy Baseling
0.0
Epochs
Epochs
CIPARI00 D0 ResNet-18
1.0
0.8
1rain Accuracy
Accuracy
0.6
0.4
High LS1
0.4
Mid LSI
Low LSI
0.2
Dummy Baseline
0.0
100
200
300
400
100
200
300
400
Epochs
Epochs
Figure 21: Accuracy of LSI based subsets of CIFAR100 across MLP (flattened input), CNN, ResNet-9, R8sNet-18
27

---

Published as a conference paper at ICLR 2025
Imagenette Proxy
1.0
Train Accuracy
Jest Accuracy
0.6
0.6
High LS1
Mid L.S1
0.2
LS1
0.2
Dummy Baseline
0.0
500
Epochs
Epachs
Imagenette ResNet-18
1.0
1.0
0.8
Train Accuracy
Accuracy
0.6
0.4
High LS1
0.4
LSI
Low LS1
Dummy Baseline
0.0
0.0
100
200 Epochs
300
100
100
200
300
100
Epochs
Figure 22: Accuracy of LSI based subsets of Imagenette across ResNet- 18
Imagewonf Proxy
1.0
0.8
1ram Accuracy
Test Accuracy
0.6
High LS1
Mid LSI
0.2
Low LS1
0.2
Dummy Baseline
200
300
400
500
100
200
300
400
500
Epochs
Epechs
Imagewoof ResNet-18
1.0
1.0
0.8
Train Accurscy
Accurcyy
0.6
0.4
High
0.4
LS1
Dumny Baseline
0.0
0.0
Epochs
Epochs
Figure 23: Accuracy of LSI based subsets of Imagewoof across ResNet-1 18
28

---

Published as a conference paper at ICLR 2025
Pneumonia Proxy
1.0
Train Accuracy
Jest Accuracy
0.6
0.6
High LS1
Mid L.11
0.2
Low LSI
0.2
Dummy Baseline
0.0
500
500
Epochs
Epachs
Pneumonia ResNet-18
1.0
1.0
0.8
Train Accuracy
Accuracy
0.6
0.4
High LS1
0.4
LSI
Low LSI
Dummy Baseline
0.0
0.0
100
200 Epochs
300
400
100
200
300
100
Epochs
Figure 24: Accuracy of LSI based subsets of the Pneumonia dataset across ResNet-18 18
K EXEMPLARY IMAGES OF IMAGENET WITH Low AND HIGH LSI
We show the applicability LSI on ImageNet (Deng et al., 2009) using two popular subsets, Ima- genette and Imagewoof (Howard, 2019). While Imagenette contains subset of easily distinguishable classes, Imagewoof solely contains classes of dogs and is considered more difficult. Computing the LSI allows the detection of mislabeled examples (e.g. the image of the car in the class radio). The following figures show samples of high and low LSI, with mislabeled examples indicated by a red dot (to the best of our knowledge) and with a yellow dot if an instance of another class is present in the image.
Low LSI
High LSI
Figure 25: Exemplary images of the highest and lowest LSI of the Australian terrier class. Note the presence of images of dogs with different breeds and images of non-dogs (e.g. the mushroom) in the high LSI parition.
29

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 26: Exemplary images of the highest and lowest LSI of the Border terrier class. Note again the presence of different breeds and atypical images (e.g. the shirt with small dog logo) in the high LSI parition.
Low LSI
High LSI
Figure 27: Exemplary images of the highest and lowest LSI of the church class. Note that low LSI images mostly show outside views of churches, while high LSI images show churches from uncommon perspectives and indoor views, which could be confused with other indoor spaces).
30

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 28: Exemplary images of the highest and lowest LSI of the radio class. Note that the the high LSI partition includes several uncommon-looking radios (e.g. the children's radio in the bottom row or the antique radio in the third row, and at least one image of a different class (the red car).
L EXEMPLARY IMAGES OF CIFAR-10 WITH Low AND HIGH LSI
Low LSI
High LSI
Figure 29: Exemplary images of the highest and lowest LSI of the airplane class. Note the differences in perspective and background color.
31

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 30: Exemplary images of the highest and lowest LSI of the automobile class. Note the differences in background and subject size in the frame.
Low LSI
High LSI
Figure 31: Exemplary images of the highest and lowest LSI of the bird class. Note the overrepresen-
tation of close-crop ("portrait") images of bird heads in the high LSI partition compared the mostly whole-bird images in the low LSI partition.
Low LSI
High LSI
Figure 32: Exemplary images of the highest and lowest LSI of the cat class. Note the heterogeneity of the high LSI partition, including atypical backgrounds and different cropping.
32

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 33: Exemplary images the highest and lowest LSI of the deer class. Note the heterogeneity of the high LSI partition, including atypical backgrounds and different cropping.
High LSI
Figure 34: Exemplary images of the highest and lowest LSI of the dog class. Note the presence of least one non-dog (the fox in the top row) and the inconsistent framing and subject size in the high LSI paritition compared to the high homogeneity of the low LSI paritition.
Low LSI
High LSI
Figure 35: Exemplary images of the highest and lowest LSI of the frog class. Note the heterogeneity of the high LSI partition, including atypical backgrounds and different cropping, the presence of an albino frog and at least two images where the subject is camouflaged (bottom row).
33

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 36: Exemplary images of the highest and lowest LSI the horse class. Note the heterogeneity of the high LSI partition, including different framing, cropping and backgrounds, as well as white horses, whereas the low LSI partition contains mostly brown horses.
Low LSI
High LSI
SAMPLE
Figure 37: Exemplary images of the highest and lowest LSI of the ship class. Note the heterogeneity of the high LSI partition and the different backgrounds compared to the standard backgrounds in the low LSI partition.
34

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 38: Exemplary images the highest and lowest LSI of the truck class. Note the atypical perspectives in the high LSI partition, as well as the inclusion of least one ambiguous image (the trailer in the third row).
M EXEMPLARY IMAGES OF MEDICAL IMAGING DATA (PNEUMONIA) WITH Low AND HIGH LSI
Examples carrying low LSI are similar to each other and are captured with very symmetrical view. Samples carrying high LSI are angled, have a different exposure, or contain samples that are not supposed to be in the dataset (images of adult females in a dataset of child pneumonia x-ray images).
Low LSI
llgh LSI
Figure 39: Exemplary images of the highest and lowest LSI of the bacterial class.
35

---

Published as a conference paper at ICLR 2025
Low LSI
High LSI
Figure 40: Exemplary images the highest and lowest LSI of the normal class.
High LSI
Figure 41: Exemplary images of the highest and lowest LSI of the viral class.
N TRAINING PARAMETERS FOR THE EXPERIMENTS
Table 2: Training parameters for each of the experiments
Sample Difficulty/ Generalization
LSI under
Parameter
LSi-Distribution
Differential Privacy
Learning rate
0.04
0.04
0.04
Weight decay (L2)
0.01
0.01
0.01
Nesterov Momentum
0.9
0.9
0.9
Dataset
Full
1/3 Subsets
1/5 Subsets 700
Epochs/ Steps
1000
1000
Averaged across Seeds
5
O HARDWARE SETUP AND COMPUTATIONAL RESSOURCES
All experiments described in this paper and its appendix are performed on NVidia H100 GPU (80GB VRAM) with 2 AMD EPYC 9354 32- Porr CPUs. While we employ an 80GE GPU, the required VRAM is far smaller, such that LSI can easily be computed GPUs of solely 24GB of VRAM. Table shows the time took to compute the LSI for each sample in each dataset used in
36

---

Published as a conference paper at ICLR 2025
this publication. Note, that the retraining requires the repeated transfer of data from the CPU Memory to the GPU Memory. Thus, computing the LSI on the GPU rather than the CPU provides solely speedup of around 2, which potentially provides room for improvement.
Table 3:
Training parameters for each the experiments
CIFAR-10
CIFAR-100
Pneumonia
Imagenette
Imagewoof
Compute (h)
-18
-18
-0.5
-1.7
-1.7
The computation LSI under differential privacy was performed on subsets of 10000 samples CIFAR-10 and required -12h of compute. All other experiments required <Ih of compute. Preliminary experiments required a total compute of about 5x the computational time that yielded the results that are shown in this manuscript.
P FUNDING ACKNOWLEDGEMENTS
GK received support from the German Ministry of Education and Research and the Medical Infor- matics Initiative as part of the PrivateAIM Project, from the Bavarian Collaborative Research Project PRIPREKI of the Free State of Bavaria Funding Programme Artificial Intelligence Data Science. This work was partially funded by the Konrad Zuse School of Excellence in Reliable AI (relAl).
This work is co-funded by the European Union under Grant Agreement 101 100633 (EUCAIM). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Commission. Neither the European Union nor the granting authority can be held responsible for them.
37


Reviews:
[{"review_id": "SuZvtA2BhY", "reviewer": "Reviewer_j9mc", "summary": "This paper proposes a measure named Laplace Sample Information (LSI) to estimate the informativeness of individual samples in a dataset for deep learning methods. Specifically, LSI leverages a Bayesian approximation to the weight posterior and the KL divergence to measure the change in the parameter distribution induced by a leave-one-out sample from the dataset. Experimental results show that LSI is effective in ordering the data w.r.t. typicality, detecting mislabeled samples, measuring class-wise informativeness, and assessing dataset difficulty.", "questions": "None", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Overall, this paper is well-written and easy to follow.\n2. This paper considers the important task of estimating the sample information.\n3. Experimental results illustrate the effectiveness of the proposed measure LSI.", "weaknesses": "1. The novelty of the proposed method might be limited as the Laplacian approximation is widely used in the Bayesian machine learning community.\n2. Theoretically, formal results are lacking to support the effectiveness of the proposed method.\n3. Large experiments involving large datasets and models are lacking, which results in that this work might have little influence in practice.\n\nMinor issues:\n1. Typos exist in Lines 194-195: shownAppendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mDQKLfRyuv", "reviewer": "Reviewer_6Bs2", "summary": "This paper presents Laplace Sample Information (LSI), a metric to quantify how much individual data points contribute to a neural network’s learning process. By approximating a Bayesian model through the Laplace method, LSI uses KL divergence to capture each sample’s impact on the model’s parameters, which can be particularly beneficial for detecting mislabeled data, understanding dataset difficulty and informativeness. Empirically, the authors validate LSI across a range of datasets and demonstrate that it transfers well from smaller “probe” models to larger architectures.", "questions": "Pleaser refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* **Originality and Quality**\nThe approach is novel because it uses a post-hoc Bayesian approximation to evaluate sample informativeness in standard neural networks. The authors include experiment results on multiple datasets, from CIFAR-10 to IMDb, showing that LSI can detect mislabeled samples and assess informativeness across classes. \n\n* **Clarity and Significance**\nThe paper is well-organized and easy to follow. As claimed in the paper, LSI is beneficial for identifying potentially mislabeled data, as well as evaluating dataset difficulty and informativeness. The proposed method could have a substantial impact, particularly for data-centric AI and applications where dataset quality is key.", "weaknesses": "**1. Computational Cost**\n\nCalculating LSI requires Hessians, which may be expensive on large datasets. While the probe model helps, a more detailed breakdown of computation time, evaluation quality, and scalability would be helpful for readers to understand the effectiveness of LSI.\n\n**2. Limitations of Probe Model** \n\nWhile using a smaller probe model makes computation faster, it could introduce discrepancies in sample rankings when applied to larger models. Could the authors provide more insight into how well LSI rankings from the probe model align with those from larger models?\n\n**3. Understanding the Effectiveness of LSI**\n\nDuring the investigation of low/high LSI samples, is it possible to have statistical measurements about the comparisons/differences between high and low LSI samples. It is not clear if the observation of the difference among a few samples can generalized to the larger scale. Besides, there are no baseline performances reported in this paper. (i.e., for the detection of mislabeled samples, I believe there are certain methods already working on it.)\n\n**4. Experiments on Real-world Mislabeled Samples**\n\nIn the section ```Focusing on Mislabeled Samples```, it would be more beneficial if authors could test the effectiveness of LSI on real-world noise (CIFAR-10N [R1], or CIFAR-10H [R2]), instead of synthetic ones (random label flips). \n\n**References:**\n\nR1: Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations. ICLR 2022.\n\nR2: Human uncertainty makes classification more robust. ICCV 2019.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBa0pBny48", "reviewer": "Reviewer_1zef", "summary": "The paper describes a novel approach to measuring the informativeness of individual samples in machine learning datasets, named the Laplace Sample Information (LSI). LSI is designed to quantify how much information each sample contributes to the model's parameters using a Bayesian approximation. The method applies the Laplace approximation to generate a distribution over model parameters and subsequently employs Kullback-Leibler divergence to measure each sample's unique contribution to this distribution. \n\nExperiments on CIFAR-10, CIFAR-100, ImageNet subsets (Imagewoof and Imagenette), a pediatric pneumonia dataset, and the IMDb text sentiment dataset. LSI revealed that only a small subset of samples contributes significant information to the model, while most samples have low informativeness. On CIFAR-10 with 10% label noise, mislabeled samples generally had higher LSI values than correctly labeled samples, showing that LSI can effectively highlight mislabeled data. Other experiments collectively validated LSI’s utility across various applications: ranking samples by informativeness, detecting mislabeled data, assessing dataset and class difficulty, and improving model generalization.", "questions": "Please see weaknessnes", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- LSI offers a new perspective on sample informativeness, focusing directly on sample information flow rather than related metrics like sample difficulty. This approach addresses a gap in the current literature, where sample informativeness has often been approximated indirectly.\n\n- LSI is adaptable across different model architectures, learning tasks (supervised and unsupervised), and data modalities (e.g., images, text). LSI has valuable applications in identifying mislabeled samples, estimating dataset difficulty, and detecting out-of-distribution data.\n\n- By using small probe models, the authors make LSI computationally feasible for larger models and datasets. This approach yields a substantial efficiency gain without significantly compromising the integrity of the informativeness rankings.\n\n- The authors evaluate LSI on various datasets, including CIFAR-10, CIFAR-100, ImageNet subsets, and text datasets, with extensive experiments validating LSI’s effectiveness in distinguishing sample informativeness. They also investigate LSI’s behavior under label noise.", "weaknesses": "- The experiments mainly evaluate the proposed LSI measures. How does the proposed measure compare to other existing related measures, such as gradient similarity, influence functions, and TRAK (Park et al., ICML'23)? \n\n- Although LSI uses a probe model for efficiency, computing LSI still requires repeated training sessions, especially in large datasets. It would be better to describe its computation complexity and compare it with existing measures. \n\n- The reliance on the Laplace approximation assumes that the model parameters follow a multivariate Gaussian distribution. While the authors justify this through the central limit theorem, it would be better to provide empirical justifications, especially for highly complex or non-convex loss landscapes. \n\n- The method relies on approximations of the Hessian, particularly when using diagonal approximations for large models. It would be better to analyze how this approximation affects LSI in different neural architectures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MX4rH5CLje", "reviewer": "Reviewer_qsLf", "summary": "This paper considered an information theoretical framework in understanding the data sample importance. Specifically, the intuition is further formulated as the difference of parameter distributions. Experiments on various and extensive experiments demonstrate the effectiveness of the proposed methods.", "questions": "See the questions in weakness sections.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I think this paper raised an important problem in explaining the important data sample. \n- The proposed method is theoretically principled with Bayes principle. \n- Extensive experimental results are presented with CV and NLP datasets (since this method is model agnostic). \n- The paper is clearly written and accessible.", "weaknesses": "There are two major weak points within the paper. Therefore I currently provide a borderline score. \n\n1. About the novelty. How this method is different from the influence function based method. Both methods require some calculation on the higher-order derivative. Based on the given context, I find it a bit hard to differentiate the differences. \n2. About the calculation LSI in line 212, I could not understand how exactly \\hat_{theta}^{-i} and \\Sigma^{-i} are estimated. Here the technical detail is very limited, therefore I could not understand why estimating these can be done after training. Please give clear details how LSI is calculated. \n3. (Minor) The paper is a bit large by itself, 32Mb ! I could not properly read the results due to its large file.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)