Measures of Information Reflect Memorization Patterns
Authors: Rachit Bansal
Danish Pruthi
Yonatan Belinkov
Venue: NeurIPS 2022 Accept (2022)
Keywords: OOD generalization
memorization
spurious correlations
challenge sets
evaluation
model selection
information

Abstract: Neural networks are known to exploit spurious artifacts (or shortcuts) that co-occur with a target label, exhibiting heuristic memorization. On the other hand, networks have been shown to memorize training examples, resulting in example-level memorization. These kinds of memorization impede generalization of networks beyond their training distributions. Detecting such memorization could be challenging, often requiring researchers to curate tailored test sets. In this work, we hypothesize—and subsequently show—that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization. We quantify the diversity in the neural activations through information-theoretic measures and find support for our hypothesis in experiments spanning several natural language and vision tasks. Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed on unlabeled in-distribution examples. Lastly, we demonstrate the utility of our findings for the problem of model selection.

Measures of Information Reflect Memorization Patterns
Rachit Bansal +
Danish Pruthi Amazon Web Services danish@hey. com
Delhi Technological University
racbausa@gmail.cm com
Yonatan Belinkov
Technion Israel Institute of Technology
belinnoo@@tecmion..ee ac.
Abstract
Neural networks are known to exploit spurious artifacts (or shortcuts) that co-oceur with a target label, exhibiting hewristic memorization. On the other hand, networks have been shown to memorize training examples, resulting in example-level memo- rization. These kinds of memorization impede generalization of networks beyond their training distributions. Detecting such memorization could be challenging, of- ten requiring researchers curate tailored test sets. In this work, hypothesize and subsequently show-that the diversity in the activation patterns of different neurons is reflective of model generalization and memorization. We quantify the diversity in the neural activations through information-theoretie measures and find support for our hypothesis in experiments spanning several natural language and vision tasks. Importantly, we discover that information organization points to the two forms of memorization, even for neural activations computed unlabeled in-distribution examples. Lastly, we demonstrate the utility of our findings for the problem of model selection. The associated code and other resources for this work are available at https: //iff ormatioooooooo moaammusssessss cs. techni on. ac. il.
Introduction
Current day deep learning networks are limited their ability to generalize across different domains and settings. Prior studies found that these networks rely spurious artifacts that are correlated with a target label (Schölkopf et al., 2012; Lapuschkin et al., 2019; Geirhos et al., 2019, 2020, inter alia). We refer to learning of such artifacts (also known as heuristics shorteuts) as heuristic memorization. Further, neural networks can also memorize individual training examples and their labels; for instance, when subset of the examples are incorrectly labeled (Zhang et al., 2017; Arpit et al., 2017; Tänzer et al., 2021). We refer to this behavior as example-lene memorization. A large body past work has established that these facets of memorization pose threat to generalization, especially in out-of- distribution (OOD) scenarios where the memorized input features and corresponding target mappings do not hold -avid et al., 2010; Wang et al., 202 Hendrycks et al., 202 la; Shen et al., 2021). To simulate such OOD distributions, however, researchers are required to laboriously collect specialized and labeled datasets to measure the extent of suspected fallacies in models. While these sets make it possible to assess model behavior over a chosen set of features, the larger remaining features remain
Work done during a visit at the Technion, Israel. The author is now at Google Research India. h Work done while Carnegie Mellon University, prior to joining Amazon. Supported by the Viterbi Fellowship in the Center for Computer Engineering the Technion.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

(a)
(b)
(c)
Induced heuristic
Shuffld labels
Task
(d)
Neuron activations for the base task
(e) Low meuron diversity
(f)
Aigh inter-nouron and intra diversity
500
500
1200
1900
1500
200%
2000
Neuron Index
Neuron Index
Neuron Index
Entropy proxy intra-neuron diversity
proxy
civersity
(g)
(h)
Houristic Memoriztiio
Example-ievel Mamorization
Neuron Entropy
2.
Demoky of Neuron Pairs
Heurietic Memorization
Base
Base
0.6
Examplelevel
Memorizaion
-22
0.0
Mutual Information
Figure 1: (a) A toy setup of separating concentric circles; (b) An additional feature spuriously simplifies the task, inciting heuristic memorization; (c) Shuffled target labels induce example-leve memorization; (d) Neuron activations for two-layered weed-forward network trained for the base task in (a); (e) Activation patterns for the network reflect low intra-neuron and inter-neuron diversity when trained on (b); (1) High intra-neuron and inter-neuron diversity is seen when the network trained on (c); (g) Entropy acts as proxy to intra-neuron diversity; (h) Mutual Information acts as proxy inter-neuron diversity. Distinguishable patterns for the three networks seen in (g) and (h).
hard to identify and study. Moreover, these sets are truly extrinsic in nature, necessitating the use of performance measures, which in turn lack interpretability and not indicative of internal workings that manifest certain model behaviors. These considerations motivate evaluation strategies that are intrinsic network and indicate model generalization while not posing practical bottlenecks in terms of specialized labeled sets. Here, we study information organization as one such potential strategy.
In this work, posit that organization of information across internal activations of network could be indicative of memorization. Consider a sample task of separating concentrie circles, illustrated in Figure la. A two-layered feed-forward network can learn the circular decision boundary for this task. However, if the nature of this learning task is changed, the network may resort to memorization. When spurious feature is introduced in this dataset such that its value (+/-) correlates to the label (0/1) (Figure 1b), the network memorizes the feature-to-label mapping. reflected a uniform activation pattern across neurons (Figure le). In contrast, when labels for the original set shuffled (Figure lc), the same network memorizes individual examples during training and shows a high amount of diversity in its activation patterns (Figure 1f). This example demonstrates how memorizing behavior observed through diversity in neuron activations.
We formalize the notion of diversity across neuron activations through two measures: (i) intra-neuron diversity: the variation of activations for a neuron across a set of examples, and (ii) inter-neuron diversity: the dissimilarity between pairwise neuron activations the same set of examples. We hypothesize that the nature of these quantities for two networks could point to underlying differences in their generalizing behavior. In order to quantify intra-neuron and inter-neuron diversity, adopt the information-theone
measures of entropy and mutual information (MI), respectively.
Throughout this work, we investigate if diversity across neural activations ($2) reflects model generalizability. We compare networks with varying levels of heuristic ($3) or example-level ($44 memorization across variety of settings: synthetic setups based the IMDb (Maas et al., 2011)

---

and MNIST (Lecun et al., 1998) datasets for both memorization types, as well as naturally occurring scenarios gender bias on Bias- -inss-in-aae An-Bios (De-Arteaga et al., 2019) and OOD image classification on NICO (Zhang et al., 2022). We find that the information measures consistently capture differences among networks with varying degrees of memorization: Low entropy and high MI are characteristic of networks that show heuristic memorization, while high entropy and low MI are indicative of example- level memorization. Lastly, we evaluate these measures from the viewpoint of model selection and note strong correlations to rankings from domain-specific evaluation metrics ($5).
2 Methods
As per the data processing inequality (Beaudry & Renner, 2012), a part of the neural network (referred as the encoder) compresses the most relevant information of given input X, into a representation H. This compressed information is processed by a classification head (or, a decoder) to produce an output Y corresponding to the given input. We hypothesize that the organization of information across neurons of the encoder is indicative of model generalization. We study two complementary properties that capture this information organization for given network:
(i) Intra-neuron diversity: How do activations of given neuron vary across different input examples. We measure the entropy of neural activations (across examples) as a proxy.
(ii) Inter-neuron diversity: How unique the activation of a neuron compared to other neurons. We quantify this via mutual information between activations of pairwise neurons.
Below, we discuss the information measures formally.
2.1 Information Measures
For any given encoder (consisting of N neurons) that maps the input to dense hidden representation, we denote the activation of the tth neuron as a random variable, A, E .a...... a where each measurement is an activation over an example from set of size S. The probability over this continuous activation space is computed by binning into discrete ranges (Darbellay & Vajda, 1999), and we denote each discretized activation value as à. Importantly, the set of examples which the activations are computed come from distribution that similar to the underlying training set itself.
Entropy We measure the Shannon entropy for each neuron in the concerned network, proxy of intra-neuron diversity. Following the definition of Shannon entropy, this is given as:
Naaa
HAA) = E [((a))] = aGEA:
> p(a)) log(
(I)
p(a!)
Mutual Information We compute the mutual information (MI) between underlying neurons as a proxy to inter-neuron diversity. Specifically, we compute the MI between all neuron pairs in the network. Thus, the set of MI values I (AA) for a particular neuron A, is given as:
I(A.) (I(A;
An))
(2)
where, T(X; Y) depicts the MI between variables X and Y. Unless stated otherwise, this T(A;) is computed V i [I....,N] NN resulting into a square matrix of size (N x N).
This process of computing the information measures for network on a given set of examples is summarized in Algorithm 1. Further details on the computation are given in appendix A.
2.2 Toy Setup: Concentric Circles
Here, we briefly discuss the information-theereiicc metrics for the example of concentric circles from the introduction (Figure l). To recap, we consider setup to compare networks showing the two forms of memorization and observe discernible differences in their activation patterns: heuristic
In principle, we would compute MI across neuron sets; approximate this through individual neuron pairs.
3

---

Algorithm 1 Computation of information measures. A Algorithmic procedures ENTROPY and MI are
specified by algorithms 2 and 3 in appendix A.
1: AI....,AN t (f(x))
D Computing activations for all neurons Initiating computations for Entropy and MI >> Iterating over the set of neurons Initiating MI for a particular neuron > Following Equation and Algorithm 2
2: H 0]: 0
3: for i [I,...,NN N do
4:
1- f
5:
H, < ENTROPY(A))
6:
for j E fI..... N do
Inner loop over the set of neurons
7:
1; + I, + (I,A,,A) Aj)
> Following Equation and Algorithm 3
8:
end for
9:
H H + H:
T -+1
Following Equation
11: end for
memorization corresponds to low intra-neuron and inter-neuron diversity, while example-leved memorization corresponds to high diversity (Figures le and If). We expect that this difference in diversity would be captured through the above defined information measures.
Figure 1g presents the distribution of entropy values for each of the three networks with varying generalization behaviors. Throughout this work, we visualize this distribution of entropy using similar box-plots, where black marker within the boxes depicts the median of the distribution and a notch neighboring this marker depicts the 95% confidence interval around the median. We observe that entropy for the network exhibiting heuristic memorization is distributed around lower point than the others, whereas entropy for the network with example-level memorization is higher.
Furthermore, Figure 1h shows the distribution of MI for the three networks. To interpret the distribution of MI (an N x N square matrix), we fit Gaussian mixture model over all values and visualize through a density plot, where the density (y-axis) at each point corresponds to the number of neurons pairs that exhibit that MI value (x-axis). Larger peaks in these density plots suggest large number of neurons pairs are concentrated in that region. Interestingly, we see such peaks for the three networks at distinct values of MI. For the network showing example-level memorization (high inter-neuron diversity), most of the neuron pairs show low values of MI. In contrast, heuristic memorization (low inter-neuron diversity) has high neuron pair density for higher MI values.
Based on these findings, we formulate two hypotheses, summarized in Table 1:
Table 1: Summarizing our hypotheses.
H1 Networks exhibiting heuristic memorization would show low inter- and intra-neuron diversity, reflected through low entropy and high MI values.
Memorization
Diversity
Intra-neuron
Inter-neuron
H2 Networks exhibiting example-level memorization would show high inter- and intra-neuron diversity, reflected through high entropy and low MI values.
(x Entropy)
(x MI-1)
Heuristic Example-level
f
f
3 Heuristic Memorization
Here, study different networks with varying degrees of heuristic memorization, and examine if the information measures measures-aimed aimed to capture neuron diversity- diversity-indicate indicate the extent of memorization.
3.1 Semi-synthetic Setups
We synthetically introduce spurious artifacts in the training examples such that they co-occur with target labels. Networks trained on such a set are prone to memorizing these artifacts. The same correlations with an artifact do not hold in the validation sets. To obtain set of networks with varying
"This difference in activation patterns for the two memorizing could be caused by several factors, including functional complexity (Lee et al., 2020): Functions that encode individual data points (as example- level memorization) need to much more complex than functions that learn shortcuts (heuristic memorization). We make comparison with standard complexity measures in appendix C.4 and observe that our information measures correlate more strongly with generalization rerformanee-especiillly for heuristic memorization.

---

3-layered MLP on Colored MNIST
DistilBERT on Sentiment Adjectives
3.5
4.0
Layer-5 Layer-6
Neuron Entropy
3.0
0.62
Validation Validation Accuracy
Neuron Entropy
3.5
0.90
Validation Accuracy
0.60
3.0
0.58
2.5
0.56
2.0
0.80
Layer-1
0.75
Layer-2
0.5
Layer-3
00.50
a
Figure 2: The relation between entropy of neural activations and heuristic memorization. For both the setups, networks trained higher show higher heuristic memorization (as depicted by the dipping model accuracy line), accompanied with lower entropy values.
3-fayered Colored
DistilBERT Sentiment Adjectives
Density of Neuran Pairs
3.0
Density of Neuron Paits
0.0
0.0
2.5
0.25
0.5
0.25
2.0
0.5
0.75
0.4
0.5
0.75
1.5
0.3
1.0
0.1
0.0
0.0
0.1
0.2
0.3
0.4
0.5
0.2
0.0
0.2
0.4
0.6
0.8
1.0
1.2
Mutual Information
Mutual Information
Figure 3: Distribution of mutual information (MI) of pairs of neurons for networks with varying heuristic memorization. For both settings, networks trained on training sets with larger amounts of spurious correlations (f a)) exhibit higher mutual information across their neuron pairs.
degrees of this heuristic memorization, we consider parameter a that controls the fraction of the training examples for which the spurious correlation holds true. We consider the following setups:
Colored MNIST In this setting, the MNIST dataset (Lecun et al., 1998) is configured such that a network trained on this set simply learns to identify the color of images and not the digits themselves (Arjovsky et al., 2019). Particularly, digits 0-4 are grouped as label while 5-9 the other, and images for these labels are colored green and red, respectively. For this setup, we train multi-layer perceptron (MLP) networks for varying values of a, which corresponds the fraction of training instances that abide to the color-to-labes correlation. The considered values of a and other details for this setup are given in appendix B.I.
Sentiment Adjectives In this setup, sub-sample examples from the IMDb dataset (Maas et al., 2011) 1) that contain at least one adjective from a list of positive and negative adjectives. Then, examples that contain any of the positive adjectives ("good"", "great", etc.) are marked with the positive label, whereas ones that contain any negative adjectives ("bad", "awful", etc.) are labeled as negative. We exclude examples that contain adjectives from both lists. The motivation to use this setup is to introduce heuristics in the form of adjectives in the training set. We fine-tune DistilBERT-bar models (Sanh al., 2019) on this task for different values of a (fraction of examples that obey the heuristic). The full set of adjectives considered and further details are outlined in appendix B.2.
Through these experiments, we first note that low entropy across neural activations indicates heuristic memorization networks. This evident from Figure where we see that (1) as we increase the validation performance decreases, indicating heuristic memorization (see the solid line in the plots); and (2) with an increase in this heuristic memorization, we see lower entropy across neural activations. We show the entropy values of neural activations for the 3 layers of an MLP
Results:
5

---

1.0
RoBERTa-baze on Bias Blos
RoBERTo-base on Bias
2.5
1.
Density Neuron Pairs
2.00
oversampled
original
subsagoled
2.0
1.75
ongiina
subsampled
overstmpled
scrubbed
Suffiiienny Gap
0.6
Compress
1.0
1.00
3.4
52
0.2
Training Sets
5.0
0.0
0.20
0.25
0.30
Mutual Information
0.35
9.40
0.45
0.50
0.55
0.0
Figure 4: Distributions of entropy and MI across final layer activations of RoBERTa base differentiate networks fine-tuned on original and de-biasing sets for Bias-in in-Bios.. Color of boxes and Gaussian plots corresponds to extractability of gender information in model representations as estimated through MDL probing (Voita Titov, 2020) lighter colors indicate lower extractability (less bias).
trained on Colored MNIST (left sub-plot) and for the last two layers of DistilBERT on Sentiment Adjectives (right sub-plot). In both these two scenarios, with increasing values of a, with a particularly sharp decline when a 1.0.
consistent drop in the entropy
Furthermore, we observe that networks with higher heuristic memorization exhibit higher mutual information across pairs of neurons. In Figure 3, networks with higher memorization (? a), have larger density of neurons in the high mutual information region. While this trend is consistent across the two settings, we see some qualitative differences: The memorizing (a 1.0) MLP network on Colored MNIST (left) has uniform distribution across the entire scale of MI values, while DistilBERT on Sentiment Adjectives (right) largely has a high-density peak for an MI ~ 0.9.
3.2 Natural Setups
Next, we investigate setups where spurious correlations are not synthetically induced, but occur naturally in the datasets. Below, describe two such scenarios:
Occupation Prediction We first study the task of predicting occupations from biographies on the Llas-nn-Bio dataset (De-Arteaga et al., 2019). Given the skewed distribution of genders across occupations, models pick up cues that reveal the biographee's gender. For instance, most biographies corresponding to the "professor" occupation are of males. Models trained on this dataset can learn such spurious associations. To evaluate how much the trained networks encode gender, we measure compression values by training a gender classifier on the internal representations of the network and computing its minimum description length (MDL). These compression values act proxy the ease of extracting gender information from representations (Voita & Titov, 2020; Orgad et al., 2022).
We consider a variety of training sets by sub-sampin sampling and over-sampling examples for each profession the dataset: This is done to balance the number examples across each gender. We do this for both the original inputs in the dataset (raw) and scrubbed examples, wherein gender-specific information (such as pronouns) is removed (similar to setups in De-Arteaga et al. (2019)). We perform our analysis on RoBERTa-bart (Liu et al., 2019) fine-tuned for these training sets."
Results: In Figure 4, we observe the distribution of the two information measures for the last layer of networks trained on the different training sets. This variation is shown in conjunction with compression values across the network using the MDL. probe. Following our initial hypothesis (Table I; HI), we expect that networks with higher representation of bias will have lower entropy. Indeed, in Figure (left), the network trained on the original training set (i.e., raw original) shows the lowest entropy. This finding is in line with our hypothesis, since the other networks are trained on either gender-ballee balanced or scrubbed sets. However, we do not observe consistent trends among
1 Considerable changes in entropy values are not seen for initial DistilBERT layers, suggesting that spurious correlations are largely captured by later layers. Detailed results covering other layers are given in appendix C.I.
'We use trained checkpoints released by Orgad et al. (2022). More details are given in appendix B.3.
'The difference in compression values across training sets is more prominent in higher layers, yet the correlation between compression and MI remains high throughout the network. We discuss this in appendix C.3.

---

ResNet-18 NICO*
ResNet-18 NICO*
3.4
Training Sit (w Valication Accuracy)
Density of Neuron Pairs
Set Type
Unbalanced (35.03%)
Balanced (50.08%)
Unbalanced
3.2
Balanced
intropy
3.0
w 2.8
2.6
Layer Block-1 Layer Block-2
Layer Block:3 Layer Block-4
Mutual Information
Figure 5: Entropy and MI for ResNet-18 18 on the +ICO++ dataset. The two training sets balanced and unbalanced seccedd-esutttt into models that vary in their generalization to contextual features beyond on what they were trained on. This distinction reflected in the information measurements.
networks trained on these de-biasing sets. On the other hand, we do see clear patterns in MI that distinguish networks in line with their compression values (Figure 4, right). As go from lower higher values of MI (left to right), the density plots get darker, corresponding to higher compression values (higher bias). prominent distinction seen between the raw and scrubbed sets, which separated on two sides of the plot.
Image Classification with Contexts Next, we consider scenario from computer vision, where the task is to identify the presented object in a particular context. We use a subset of the dICa*ddttaset (Zhang et al., 2022), which consists of images of animals in variety of contexts. For each animal class, there exist two types of contexts: individual, those that are specific to only that animal and are not present for all classes (such as roaring bear), and common, contexts that exist across all classes (such as images taken in dark).
For our analysis, design training sets unbalanced and balanced varying in the distribu- tion common contexts across examples. Each animal in the unbal anced set occurs in a particular common context that is chosen for that animal. In contrast, the balanced set contains images from all common contexts, for each animal. Thus, network trained on the unbal anced set is likely to pick the context-to-animal mapping (i.e., case of heuristic memorization).
Results: We train R8sNet-18 (He et al., 2015) networks for the two sets and evaluate them on the common NICO+ evaluation set, balanced across all common contexts. We consider the hidden representation from each of the 4 blocks of layers in the network to compute the information measures reported in Figure 5. From the left sub-figure,
observe that the entropy for networks trained on the balanced set is consistently greater than the unbal anced set across all layer blocks. Furthermore, observe that distribution of MI (right) across pairwise neurons also reflects the difference between the networks, corroborating our hypothesis. Neuron pairs for the network that memorizes the correlation with image contexts (unbalanced) more densely concentrated at higher MI values.
4 Example-level Memorization
We now examine how the distribution of information measures across networks change when they memorize individual examples. Following original hypotheses (Table 1; H2), expect such networks to display high nntronneuron and inter-neuron diversity, and thus high entropy and low MI.
We perform the analysis for example-level memorization the standard datasets of MNIST (Lecun et al., 1998) and IMDb (Maas al., 2011) on a 3-layered MI.P and DistilBERT--oa order to study how the diversity of neurons changes with increasing example-level memorization, induce varying levels of label noise by randomly shuffling a fraction of training examples' target labels (denoted by parameter 5). We then analyze these trained networks the original validation set.
respectively. In
First, we note that model performance on the validation set decreases with increased label shuffling, validating increase in example-level memorization (Figure 6). Interestingly, this dip in validation accuracy is accompanied with a consistent rise entropy across the neurons. For MI.P networks trained on MNIST (left), a distinct rise in entropy even with a small amount of label
Results:

---

3-layered MLP on Shuffled MNIST
DistilBERT on Shuffled IMDb
1.0
Layer-1
Validation Accuracy
2.5
Layer-2
0.90
Entrapy
0.8
Neuron Entropy
Accurac
0.6
0.75
0.4
1.0
0.70
Layer-1
0.65
Layer-2
0.2
0.5
Layer-3
0.0
B
B
Figure 6: Entropy across neuron activations increases with greater example-level memorization (1 ()).
3-layered MLP on Shuffied MNIST
DistilBERT on Shuffied IMDb
Density of Neuran Pairs
Density Neuron Pairs
1.2
B
0.25
1.0
0.2
0.75
0.8
0.6
0.6
4
0.4
2-
0.2
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Mutual Information
Mutual Information
Figure 7: Networks that show higher example-level memorization (1 3) have high density of neuron pairs for lower MI values. Here, MI computed across the first layer for both the networks.
shuffling (3 0.25), followed by a steady increase (layers and 3) or no change (layer 1) in entropy. A dissimilar trend is seen for DistilBERT' fine-tuned on IMDb (right): a distinct rise for high values of and consistent value for low no label shuffling. While our hypothesis holds true in both settings, we speculate the difference between them is due to the pre-trained initialization of DistilBERT, which has been shown to act as an implicit regularization during fine-tuning (Tu et al., 2020). That is, here, DistiBBERT might be learning task-relevant information despite some amount of label noise (note that this not evident through validation performance alone).
Our hypothesis for the relation between example-level memorization and MI is supported by Figure 7. In both settings, networks trained higher 3 values consist neuron pairs that show low values of MI (left side of the plots). In line with the previous observations, we find that MLPs trained on some amount of label noise (any 3 > 0.00) on MNIST (left sub-plot) have higher density neuron pairs concentrated at low values of MI. Meanwhile, for DistilBERT on IMDb (right sub-plot), we observe that neuron pair density gradually shifts towards lower values of MI with increasing 33.
5
Model Selection
In the previous sections, we have seen that studying information organization through the presented measures allow us to qualitatively distinguish networks with different generalizing behaviors. A natural application of our findings is the problem of model selection: given a list of models, rank them based on their generalizability. To demonstrate the utility our insights, we compare the correlations between rankings obtained through our information-therree measures (which do no require labeled data) and the generalization ability
model on a labeled held-out set.
We consider the same tasks and networks as discussed in the prior sections, and compute the rankings using (i) extrinsic evaluation metrics defined for the task (such validation accuracy for Colored MNIST and compression for Bias-ing -nn-in---in Bios). (ii) the mean of entropy values, and (iii) the mean of
Although MI values remain non non-negative throughout, the x-axis in our density plots might show negative values artifact of fitting Gaussian mixture model.

---

Table 2: We measure the correlation (Kendall's 7) between model rankings based on their generaliza- tion as estimated through extrinsic metrics on labeled test sets and those obtained via information measures. Note that r can range from -1.0 (perfect disagreement) to 1.0 (perfect agreement).
Sentiment
Colored
Shuffled
Shuffled
Bias-in-Bioo
Adjectives
MNIST
IMDb
MNIST
Validation Accuracy
Validation Accuracy
Comp- ression
TPR
Suff. Gap
Validation
Validation Accuracy
Gap
Accuracy
Mcan Entropy
0.80
1.00
0.47
0.20
0.20
0.60
1.00
Mean MI
0.80
1.00
0.60
0.07
0.33
0.80
1.00
MI values computed for the same networks. We then compute the Kendall rank correlation coefficient, T, between these rankings (between (i) & (ii), and & (iii)) to evaluate the agreement amongst them.
We observe high correlation values for all the comparisons (Table 2). Particularly high correlations are observed for setups with synthetically induced spurious correlations ($3.1) and shufiled labels ($4), with rankings on Colored MNIST being perfectly correlated. Correlations on Bias -ina-innioss are positive but lower, likely due the more nuanced setup, where the memorization is less pronounced and extrinsic metrics are weakly correlated even among themselves (appendix D.I; Orgad et al., 2022). These positive correlations are important because- correlations are computed- the information measures are purely intrinsic to the model and do not assume access any OOD data. We perform additional comparative discussion with standard conventional methods for model selection in appendix D.2.
the other metrics across which the
6
Related Work
large body of work aims at measuring and quantifying generalization, especially in out-of- distribution (OOD) scenarios (Ben-David et al., 2010; Hendrycks et al., 2021a; Wang et al., 2021b). curate and label a set of examples to evaluate if networks exploit certain heuristics or shortcuts (Lapuschkin et al., 2019; Zhao et al., 2018). Several past studies create such sets spanning different domains and tasks to shed light on common failure modes in both the trained models, and the datasets used to train them. A body of such work exists for several tasks across vision (Russakovsky et al., 2015; Hendrycks & Dietterich, 2019; Hendrycks et al., 2021a,b) and NLP (McCoy al., 2019; Naik et al., 2018; Ravichander al., 2021; Kim & Linzen, 2020).
The most common approach
Closely related to the motivations for our work, past work has attempted to evaluate models using techniques that go beyond extrinsic evaluation. Training dynamics have been explored to assess the role of individual examples in training sets (Swayamdipta et al., 2020) and how specific knowledge features are temporally picked during training (Saphra & Lopez, 2019). Recent work has noted that frequent spurious artifacts are learnt prior to general patterns during training (Tänzer et al., 2021), I), in turn followed by memorization of individual examples (Arpit et al., 2017). Closely sharing motivations of using information- theooretic viewpoints for intrinsic evaluation over network activations, past work has investigated probing or diagnostic classifiers (Ettinger et al., 2016; Adi al., 2017; Belinkov et al., 2017: Hupkes et al., 2018). Researchers have further extended this paradigm to analyze the role of individual neurons (Dalvi et al., 2019; Durrani et al., 2020; Bau et al., 2017, 2020), although this approach may fail to identify causal roles (Antverg & Belinkov, 2022; Belinkov, 2022). Other work using information theory to study neural networks has focused through the lens of the information bottleneck principle (Tishby et al., 1999), categorizing learning into distinct phases (Shwartz-Ziv & Tishby, 2017; Saxe al., 2018) and obtaining generalization bounds (Tishby& & Zaslavsky, 2015). Follow-up work has made use of such measures to regularize training for robustness (Wang al., 2021a) and low-resource learning (Mahabadi al., 2021).
their learning process
7 Limitations and Future Directions
Below, we describe some of the limitations of our work and discuss future research directions.

---

Comparative Nature of Observations The current findings and insights derived from the informa- tion measures are comparative in nature that could be limitation when being applied for practical use cases. In order to assess some given models using the information measures described herein, we must a-priori know at least one of two things: (i) one of the given models that generalizes well, so that the rest could be mench-marked against it, or (ii) the kind of memorization that the models trained on the dataset are expected to possess, so that we could make a comparison between the given models. Further, one may want to compare models that do not belong to the same model family, ar- chitecture and hyperparameter set. In such cases, values from our information measures might not be directly comparable across these different models. We design simple experiment study this in appendix C.2 where we compute our measures for models with varying capacity. We note that values for networks with different capacities lie on different scales and hence not directly comparable.
Scaling to Larger Models While the analysis presented in this work is performed for small to moderately large networks like MLPs, RoBERTa, and ResNet-for whom our hypothesized trend holds consistently- more research is needed study the scaling behavior of these information measures function of data and model size (Rosenfeld et al., 2020; Kaplan et al., 2020).
Practical Applications. In this work, we show the utility of observations for the preliminary use case of model selection. More research is required to investigate the usefulness of our observations in other scenarios. One viable direction to explore is the problem of OOD detection (Arora et al., 2021>-deciding whether a specific data point is OOD-by computing point-wise versions of the information measures. Another case where the proposed information measures could also be useful is regularizing models, where regularizing the MI/entropy values to certain a band of values might yield more generalizing models. Such regularization can also be coupled with our understanding of training dynamics from prior work (Tänzer et al., 2021) that has identified training stages where particular forms of memorization is seen to exist. Our understanding of training dynamics, itself, could be enhanced by studying the progression of neuron diversity across training steps and hence noting the generalization patterns that emerge.
8 Conclusion
In this work, we have taken a step towards identifying generalization behavior of neural network models based on their intrinsic activation patterss. We presented information information-theoreiic theoretic measures that allow us to distinguish between models that show two kinds of memorization: those that pick up surface-level spurious correlations (heuristic memorization) and those that overfit on individual training instances (example-level memorization). Through investigations spanning multiple natural language and vision tasks, we corroborated our hypothesis that such memorization reflected in diversity across neural activations, and hence the defined information measures that quantify them. Finally, we demonstrated potential application of this framework for model selection.
Acknowledgments
We are grateful to the Technion CS NLP group and others at the Technion-paaricularyy Mor Ventura, Michael Toker, Hadas Orgad, Reda Igbaria, Zach Bamberger, Adir Rahamim, Anja Reusch, and Gail Weiss-for the insightful discussions that shaped this work. RB would like to extend his gratitude to his dorm-mates and friends-Atulya Josh, Cornelius, David, Kristóf, Pratibha, Ajay, Navdeep-forr being constant source of home during his time at the Technion. RB would also like to thank the support staff at Delhi Technological University and the Technion for their administrative support. We also thank the anonymous reviewers and area chairs during the review process at NeurIPS 2022 for their careful analysis of our work. This research was supported by the Israel Science Foundation (grant No. 448/20) and by an Azrieli Foundation Early Career Faculty Fellowship.
References
Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. Fine- fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In 5th International Conference on Learning Repre- sentations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Truck Proceedings. Open- Review.net, 2017. URL htttp::/openreeeeee hptp///wppeeeeeeee////e/e.
10

---

Antverg, O. and Belinkov, Y. On the pitfalls of analyzing individual neurons in language models. In International Conference on Learning Representations, 2022. URL https: net/ Forum/1A--Bu0OEMPQII.
Arjovsky, M., Bottou, L.. Guirajani. I., and Lopez-Paz, D. Invariant risk minimization. CoRR, abs/1907.02 020993 2019, URL http:://axxvvvooo/oeae
.28933.
Arora, U., Huang, W., and He, H. Types of out-of-diitriiution texts and how to detect them. In Moens, M., Huang, X., Specia, L., and Yih, S. W. (eds.), Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 10687-10701. Association for Computational Linguistics, 2021. doi: 10. 10018833/112222.cmmg wwt./22.eemnlpmmmiiii.. main. 835. URL https: //tpi./diiiioo htttps://doi.org////// 10 1866311166/v12211 emnlp mannp-main.ce www..miiin....... 835.
Arpit, D., Jastrzebski, S., Ballas, N., Krueger, D., Bengio, E.,, Kanwal, M. S., Maharaj, T., Fischer, A., Courville, A. C., Bengio, Y., and Lacoste-Julien, S. A closer look at memorization in deep networks. In Precup. D. and Teh, Y. W. (eds.), Proceedings the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017, volume 70 of Proceedings of Machine Learning Research, pp. 233-242. PMLR, 2017. URL press/vil v70/ http://////ttt/////// awpaaaaa..aatttt.aa html.
mlr.
Bau, D.. Zhou, B., Khosla, A., Oliva, A., and Torralba, A. Network dissection: Quantifying interpretability of deep visual representations. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 25199-3227, 2017.
Bau, D., Zhu, J.-Y., Strobelt, H., Lapedriza, A., Zhou, B., and Torralba, A. Understanding the role of individual units in a deep neural network. Proceedings of the National Academy of Sciences, 2020. URL https: //np:://www //www. .nas. org/ cetg/ttttt/////////
ISSN 0027-8424. doi: 10.
Beaudry, N. J. and Renner, Comput., (2(5-6):332-441) -411, 2012.
intuitive proof of the data processing inequality. Quantum Inf. doi: 10.26421/Q1C11.55-64444 URL https://doi. org/10.
26421/QIC12.5 5-6-4.
Belghazi, M. I., Baratin, A., Rajeswar, S., Ozair, S., Bengio, Y., Hjelm, R. D., and Courville, C. Mutual information neural estimation. In Dy, J. G. and Krause, A. (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 530-539. PMLR, 2018. URL http: httttp:p/prrcceeinggs mlr. www.ppressvveeeee /v00/ 2/800/bblli htsp/bblllgaaaa gwwzilaaaaa.. htmml.....
Belinkov, Y. Probing classifiers: Promises, shortcomings, and advances. (1):207-219, 2022. doi: 00.1162/coli/_aa000000 URL https: //doi. www.l.orgg1... 11111000.11/11.1.11111 00422.
Comput. Linguistics, 48
Belinkov, Y., Durrani, N., Dalvi, F., Saijad, H., and Glass, J. R. What do neural machine translation models learn about morphology? In Barzilay, R. and Kan, M. (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACI. 2017, Vancouver, Canada, July 30 August 4, Volume 1: Long Papers, pp. 861-872. Association for Computational Linguistics, 2017. doi: 18.18653/v 1/P17-1080 URL https: / //ooi//oi//// org/ 10. 1w111//////1/-/11/
Ben-David, S., Blitzer, J., Crammer, K., Kulesza, A., Pereira, F., and Vaughan. J. W. A the- ory of learning from different domains. Mach. Learn., 19(1-2):1511175, 2010. doi: 10.1007/ $10994-009-5152--1 URL https: /ttp:::////ooooooo org 10. 1007 000000//110994400-5555510
Cheng, P., Hao, W. Dai, S., Liu, J., Gan, Z.. and Carin, L. CLUB: A contrastive log-ratio upper bound of mutual information. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13- July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1779-1788 PMLR, 2020. URL http: http:////ooceeeeeeeeee mww. alr.press/ che//ch html.
Dalvi, F., Durrani, N., Sajjad, H., Belinkov, Y., Bau, A., and Glass, J. R. What is one grain of sand in the desert? analyzing individual neurons in deep NI. models. The Thirty- AAAI Conference Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications
11

---

of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019. Honolulu, Hawaii, USA, January 27 February 1, 2019, pp. 6309-6317. AAAI Press, 2019. doi: 10. (1609//aaivv3311133331111 //doi.
URL. https:
www.oo./////....... //doi.org/10. 1609 aaai. v33101 330163099
Darbellay, G. and Vajda, I. Estimation of the information by adaptive partitioning of the observation space. IEEE Transactions on Information Theory, 45(4):1315-13211 1999. doi: 10.1109/18. 761290.
De-Arteaga, M., Romanov, A., Wallach, H. M., Chayes, J. T., Borgs, C., Chouldechova, A., Geyik, S. C., Kenthapadi, K., and Kalai, A. T. Bias in bios: A case study of semantic representation bias in a high-stakes setting. danah boyd and Morgenstern, J. H. (eds.). Proceedings of the Conference Fairness, Accountabiliry, and Transparency, FAT 2019, Atlanta, GA, USA, January 29-31, 2019, pp. 120-128. ACM, 2019, doi: 10. 10.1145/3287556032887522 URL https: //doi.org// 10. 1014112285/760 3287572.
Durrani, N., Sajjad, H., Dalvi, F., and Belinkov, Y. Analyzing individual neurons in pre-trained language models. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 4865 085-4480.44 Association for Computational Linguistics, 2020. doi: 10. 18653/v1/2020 1/06:////20000.emmpmmaaa.a 395.
URL https: / wttp://doi.oooooo orw.oi.org/o org 10rg//1..0 18653/v1//2022
mall-mmain..
Ettinger, A., Elgohary, A., and Resnik, P. Probing for semantic evidence of composition by means of simple classification tasks. In Proceedings of the Ist Workshop on Evaluating Sector-pacce Repre- sentations for NLP, pp. 134-139, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1118133/1//11-2220
URL httppts://claaattololoo .rg. org//ogg// 2rg/N16-2524.
Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. Imagenet- trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL https://openreviiet http::///pppnrrreeeeeteffaraaaaa net/:/rmmffii Bygh9j09KX.
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., and Wichmann, F. A. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665 673, 2020.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. CoRR, his//552200385,, 2015. URL http://arxiv.orrt http://axxvv..rraaaaaaa wwwp/////////////e//o///eeeeeee/ hta.b///11....0 03385.
Hendrycks, D. and Dietterich, T. G. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representationss ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL /tttps://ppnreee .eteteeeeeeeeeeeeeeeeeeeeeeeeeee Forumiidd+HJ28ti10qqi
Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., Song, D., Steinhardt, J., and Gilmer, J. The many faces of robustness: A critical analysis of out-of-distributtinn generalization. In 202/ IEEE/CVF International Conference Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021, pp. 1320-88329 IEEE, 2021a. doi: 10. (0.1109//CCV4992222221 2021.00823
URL https:///dooo oogiii.rr//ae 11..0.1119////889220000
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversar- ial examples. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pp. 15262-15271. Computer Vision Foundation / IEEE, 2021b. URL attps://openeet theeee Brrraarraaaa_aaaaaes____r___....
thecvf com/
.ttll.
Hupkes, D., Veldhoen, S., and Zuidema, W. H. Visualisation and 'diagnostic classifiers' reveal how recurrent and recursive neural networks process hierarchical structure. J. Artif. Intell. Res., 61: 907-926, 2018.
12

---

Jiang, Y., Neyshabur, B., Mobahi, H., Krishnan, D., and Bengio, S. Fantastic generalization measures and where to find them, 2019. URL attps://arxis http//aaxi..orraaara//a/aa hts//bb//1999 htabs///99....0 02178.
Kaplan, J., McCandlish, S., Henighan, T. J., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling laws for neural language models. ArXiv, abs/2001 0bs/2001.08361,, 2020.
Kim, N. and Linzen, T. COGS: A compositional generalization challenge based on semantic interpre- tation. In Webber, B.. Cohn, T., He, Y., and Liu, Y. (cds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 9087-9105. Association for Computational Linguistics, 2020. doi: (0.18653/v1/22 1/2020. emnlp-main.735 731. URL htthtt//p//oo http:///ooooogg///// n0rg/10.10 1ttg/10.18663//12222222 emnlp-main.755 731.
Kraskov, A., Stögbauer, H., and Grassberger, P. Estimating mutual information. Phys. Rev. E, 69: 066138, Jun 2004. doi: 10. 100.PPhy.Rc...6.0666.... URL https://link aps org/doi/19 /oi//00011110 1108/PhysRevE. 69.66999800 066138.
Lapuschkin, S., Wäldchen, S., Binder, A., Montavon, G., Samek, W., and Müller, K. Unmasking clever hans predictors and assessing what machines really learn. CoRR, abs/1902.10178, 2019.
URL http:://axxxii/ http://aaavvvro////aa//s httg/abs/19002. 10178.
Lecun, Y., Bottou, L., Bengio, Y., and Haffner, Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 86(11):228-22324 1998. doi: 10.1 109/5.726791
Lee, Y., Lee, J., Hwang, S. J., Yang. E., and Choi, S. Neural complexity measures. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information
Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL httttss:pproceeeedeee ingrouuringss..uu......... neurips. Abw.24//777AcA--raatttc.... .tt..rrttttttaa.ll..............
cc/ cc/paper/ / 2/////2220/00////
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L.,, and Stoyanov, V. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692, 2019. URL http: / //arxiv./.ii/////a org/ abs/a/ss//1// 11692.
Maas, L., Daly, R. E., Pham, P. T., Huang. D.,, Ng. A. Y., and Potts, C. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL http: //www aclweb. org/ anthology/ P11-1015.
Mahabadi, R. K., Belinkov, Y., and Henderson, J. Variational information bottleneck for effective low- resource fine-tuning, In 9th International Conference on Learning Representations, ICLR 1021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021. URL htRp::/t::/ //pprr/ee/e/eeeeeeeeeeeeeee net/ wwwodiddxvhZ---a111 _DD.
McCoy, T., Pavlick, E., and Linzen, T. Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3428-3448, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10. N0118853/v1/P99 1334. URL https: / aclhoooloyo org/P19- 91334.
Naik, A., Ravichander, A.. Sadeh, N. M., Rosé, C. P.,, and Neubig, G. Stress test evaluation for natural language inference. In Bender, E. M., Derczynski, I.., and Isabelle, P. (eds.), Proceedings of the 27th International Conference Computational Linguistics, COLING 2018, Santa Fe, New Mexico, USA, August 20-26, 2018, pp. org/ 2018. URL
Association for Computational Linguistics, Corg/C18-11 C18 1198/.
Orgad, H., Goldfarb-Tarren S., and Belinkov, Y. How gender debiasing affects internal model representations, and why it matters. CoRR, wttp/222440682777 2022. doi: 10. 10448500/rrxi.22044068777 URL https: / /toi. org/10 10. 48550/ arriv..iiiiv..ii. 2204. 06827.
Ravichander, A., Dalmia, S., Ryskina, M., Metze, F., Hovy, E.,, and Black, A. W. NoiseQA: Challenge
set evaluation for user-centric question answering. Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2976-2992, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/2011 -eal--main. 259. URL https: : / /awaaaacnnnoooyy....omoo org/2021. htt2222212221111111
13

---

Rosenfeld, J. S., Rosenfeld. A., Belinkov, Y. and Shavit, N. A constructive prediction of the generalization enror across scales. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL. https: //apenreview.o formm??rrm? id=ryenvpEKDr.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A.. Bernstein, M., Berg, A. C., and Fei-Fei. L. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211 2015. doi: 10.1007/ s1 1263--055-
Sanh, V., Debut, L.,, Chaumond, J., and Wolf, T. Distilbert, distilled version of BERT: smaller, faster, cheaper and lighter. CoRR, abs/1910.01 108, 2019. htl.httt//aaaee httpp://aaaii...oaaaa/////// .01008.1..0000111000010001000000
Saphra, N. and Lopez, A. Understanding learning dynamics of language models with SVCCA. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long and Short Papers), pp. 3257-3267, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 11185/61//N1911133 URL. https: a/ aalaaaaaaantooooyyyoooooo org/ N19- 1329.
Saxe, A. M., Bansal, Y., Dapello. J., Advani, M., Kolchinsky, A., Tracey, B. D., and Cox, D. D. On the information bottleneck theory of deep learning. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https: //pp/////ee/e/eeeeeeeeeee net/ forum?dd=ry_.P--- A-
Schölkopf, B., Janzing, D., Peters, J., Sgouritsa, E., Zhang, K., and Mooij, J. M. On causal and anticausal learning. In Proceedings of the 29th International Conference Machine Learning, ICML 2012, Edinburgh, Scotland, UK, June 26 July 1, 2012. icml.ce/ Omnipress, 2012. URL http:// /tttp://icml. cc/2022/ ccc/////pappers///////// pappps///222 /625. pdf.
Shen, Z. Liu, J., He, Y., Zhang, X., Xu, R., Yu, H., and Cui, P. Towards oft-o--distribution generalization: A survey. CoRR, abs/2108. 13624, 2021. URL httppp:s//axxii../ot 2208. 13624.
Shwartz-Ziv, R. and Tishby, N. Opening the black box of deep neural networks via information. CoRR, 0bss/1703.00/// 2017. URL http: //arxiv/////i//eeaaaeeeeeeeema abt///3./
Swayamdipta, S., Schwartz, R., Lourie, N., Wang, Y., Hajishirzi, H., Smith, N. A., and Choi, Y. Dataset cartography: Mapping and diagnosing datasets with training dynamics. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 746. URL Association for Computational Linguistics, 2020. doi: 10011865//1/2220.emmll htttss://oii. org/ 1/////0.186655 00/0655/v1/2200 emn.p- mwin. 746.
Tänzer, M., Ruder, S., and Rei, M. BERT memorisation and pitfalls in low-resource scenarios. CoRR, faas/105500822 2021. URL https://arxiv org/ att// 00828.
Tishby, N. and Zaslavsky, N. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop, ITW 2015, Jerusalem, Israel, April 26 May 1, 2015, pp. 1-5. IEEE, 2015. doi: 10.1 10.10000222155.33311660 URL https: http://oo..oo/to org/ 10. 1109/ ITt 2015. 7133169.
Tishby, N., Pereira, F. C.,, and Bialek, W. The information bottleneck method. In Proc. of the 37-th Annual Allerton Conference on Communication, Control and Computing, pp. 368 368-377, 1999. URL https:/ / /arxivvvvv..///////ae///aaail/a/ org/ abs/ https//hhsicsssoot/se
Tu, L., Lalwani, G., Gella, S., and He, An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models. Transactions of the Association for Computational Linguistics, 8:621-633, 10 2020. ISSN 2307-387X. doi: 10.1162/tacl a_ 00335. URL https: //doi org/ 10. 11111///aaca_0000000 a_00355.
Voita, E. and Titov, I. information-theorete probing with minimum description length. In Webber, B., Cohn, T., He, Y., and Liu, Y. (eds.). Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pp. 183-196.
14

---

Association for Computational Linguistics, 2020. doi: 10.18663//1/2022..emm www.:/2ww..mmmmmmm..a.aaa main. URL
https://doi. org/ 10. 18653/v1/2000 emmlp main. 14.
Wang, B., Wang. S., Cheng, Y., Gan, Z., Jia, R., Li, B., and Liu, J. Infobert: Improving robustness of language models from an information theoretic perspective. In 9th International Conference Learning Representations. ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, net/ htt./formmmddd hhhrrniiddhptttnnnssuu
2021a. URL httptt://opeerrrrreie
Wang, J., Lan, C., Liu, C.,, Ouyang, Y., and Qin, T. Generalizing unseen domains: A survey on domain generalization. In Zhou, Z. (ed.), Proceedings of the Thirlieth International Joint Conference Artificial Intelligence, IJCAI 2021, Virtual Event Monireal, Canada, 19-27 August 2021, pp. 4627-4635. ijcai.org, 2021b. doi: 10.24963/ijcai 2t1p63//ccai222216222222 2021/628. URL https://doi:i org/ 10. 24963/ijcaiiii 2021/628.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. Understanding deep learning requires rethinking generalization. arXiv: 1611.033300 [cs], February 2017. URL http: org/ abs/1611. 03530. arXiv: 1611.03530.
Zhang. X., He, Y., Xu, R., Yu, H., Shen, Z., and Cui, P. Nico++: Towards better benchmarking for domain generalization, 2022. URL https://arxiv.
Zhao, Z., Dua, D., and Singh, S. Generating natural adversarial examples. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL stttps://penrre .net// net/ forum? Berum?id=H1BB.g220
15


Reviews:
[{"review_id": "lJbiKem7dB4", "reviewer": "Reviewer_hktY", "summary": "The paper investigate the detection of memorization. The author categorize the memorization into 2 kinds: heuristic based and example based. The main hypothesis is, compared with the base case, the neural activations of heuristic based memorization has lower entropy and low MI, while the other one has higher entropy and MI. This is empirically and qualitatively validated on both language and vision tasks, and the author provide a quantitative analysis of this metric as a way for model selection.", "questions": "1. Is entropy and MI strongly correlated? If so can we just use one of these metric?\n2. Can you provide some insights on how this can be useful in designing a regularizer for training?", "limitations": "Not that I can see.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\nThe paper is easy to follow and well presented. The idea of observing neural activities for memorization is interesting, and I think proposition of MI and inner neurons entropy could be useful for researchers interested this topic. The experiments are solid to show the soundness of the hypothesis.\n\nWeakness:\nThe main critique for me is that, while the model selection application is given, I am not sure how good that number is. There are some other baselines on ranking models based on generalization metric, e.g., [1], [2]. I would like to see more comparison with some baselines from these work to see the value of model selection.\n\n\n[1] https://arxiv.org/abs/1912.02178\n[2] https://arxiv.org/abs/2012.07976", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "fhXTXae1xY", "reviewer": "Reviewer_LLo7", "summary": "This paper proposes to analyse DNNs' generalization behaviours by looking at activation patterns, in particular the entropy of a neuron, and the mutual information between neurons. Experiments show that these measures rank different models similarly to other extrinsic measures (which require labels). This is shown by perturbing labels and training data distributions, leading to two extremes (1) spurious correlations and (2) example memorization.\n\nAn interesting finding of this paper is that a \"normal\" neural network's entropy and MI seems to be in a sweet spot. Too much MI and the network is probably having spurious correlations, too little, and it's probably memorizing individual examples.", "questions": "Concerns:\n\nI'm not sure I understand what entropy is being computed, or if it even is an entropy. Shouldn't you want to compute the entropy of the empirical categorical distribution of $A_i$ over the dataset, when $A_i$ is discretized into bins? The sum in the entropy calculation would then be over the number of bins, not the number of examples $S$. $H(A_i)$ should be\n$$H(A_i) = \\sum_{j=1}^{N_{bins}} p(\\hat{a} _ {i}^j) \\log p(\\hat{a} _ {i}^j)$$\nwhere \n\n$$p(\\hat{a}^j_i) = \\frac{1}{S}\\sum_k^S 1[\\mathrm{bin}(a_{(i,k)}) ==  j]$$\n\nWould it be possible to reproduce the plots in the paper with the above computation? -- I'd really like to understand this before accepting this paper.\n\n> [L98] we fit a Gaussian mixture model over all values\n\nWhy not just plot the histogram of values? Seems like this could hide some artefacts.\n\nAs mentioned above, what models are Table 2's values computed on? Would a practitioner looking to apply this model selection see this by, say, just changing the learning rate or capacity of a model?\n", "limitations": "The authors address social impacts, but limitations are limited to future work rather than fundamental weaknesses of the proposed approach.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality: This paper is not the first to analyse entropies and mutual information within neural networks. The specific instantiation and what's being tested are novel, and hopefully informative to the community, but not wholly surprising.\n\nQuality/Clarity: The experiments in the paper are fairly well explained and executed (although I have some concerns), and the thesis is very clear.\n\nSignificance: These findings are certainly interesting, but I'm not sure they are groundbreaking either. The authors suggest that the proposed measures can be used to rank models and thus perform model selection, but this result feels a bit forced, since the authors appear to compare models which they specifically train with various algorithmic changes/perturbations that might cause this.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dGo3RZbFcaT", "reviewer": "Reviewer_yJeZ", "summary": "This paper studies two types of memorization: i) heuristic memorization and ii) example-level memorization. The authors study the diversity of activations units of the neural networks either across a set of examples (intra-neuron) or for the same example (inter-neuron). They observe, through various experiments on both synthetic and real datasets, that models experiencing heuristic memorization have lower diversity in their activations compared to models experiencing example-level memorization. They also suggest using their two metrics for model selection.", "questions": "1- In Figure 1, how would you compare the generalization ability of the three networks that are trained on the three tasks? Would you assume that the first model (trained on the base task) is the most \"generalizable\"? If so, then how would you explain Figure 1 (g) and (h) in terms of capturing generalization? Ranking the models based on these two figures gives us: model (b) > model (a) > model (c). Hence, if Neuron Entropy and Density of Neuron Pairs are reflective of generalization, this result would suggest that the model with \"heuristic memorization\" is the most \"generalizable\", which is counterintuitive.\n\n2- The paper tries to distinguish heuristic memorization from example-level memorization. But what is not clear is how to distinguish memorization (either type) from no memorization? If the method is not able to do so, then I would suggest combining this approach with a method that detects memorization. That is, first, detect with that method whether the model is experiencing any memorization or not. Next, if the model is memorizing, then distinguish its type using your approach (low/high entropy/MI). This would be a much more interesting tool.\n\n3- One concern raised in the introduction is that it's difficult to generate OOD datasets. How is this solved in your paper?\n\n4- Why are the activation units of only the encoder considered? Why not use both the encoder and the decoder? What is even meant here by encoder and decoder? (lines 58-63)\n\n5- How do you make sure that the models trained on synthetic datasets with heuristic memorization do actually memorize? This is important to double-check. Because it could be that the network chooses to classify based on the more complex features (for example the circles in Figure 1) rather than based on the simple induced feature. If that's the case, then studying that model is not the same as studying heuristic memorization. The validation accuracy shows this to some extent, but it would be better to see some examples as well. For example, for a new green point in the outer circle but high in the induced feature, what is the label?", "limitations": "The authors have not mentioned the limitations of their method in the main paper. But in the appendix, they mention potential societal impacts. ", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1- The paper is well-written and the figures look nice.\n\n2- To the best of my knowledge, the proposed metrics to distinguish between two types of memorization are novel.\n\n3- The empirical results include both vision and language tasks, and also both synthetic and real datasets.\n\nWeaknesses:\n\n1- The application for the method is not clear. It is also not clear what are the particular insights that this study brings to the table; is it that \"models experiencing example-level memorization are more complicated than models experiencing heuristic memorization\", and that \"models experiencing heuristic memorization are less complicated than models not experiencing that\"? Aren't these conclusions expected from our understandings of neural networks? What is the new outcome of this study? \n\n2- Comparison with other complexity measures is missing. If the activation diversity is simply capturing the complexity of the learned model, then how does it compare to other neural network complexity measures? What would be the specific advantage of these new metrics? \n\n3- Lack of comparison to other methods also applies to the model selection section. How does the model compare to other generalization measures? Also, the obtained Kendall tau is computed for which models? What is the scope of this model selection? If it's just computed over a few networks then the generalizability would be questionable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3EKpKVWU3x", "reviewer": "Reviewer_V8qo", "summary": "\nThis paper studied two types of memorization that impedes the generalization, the heuristic memorization and the example-level memorization, respectively.  This paper proposed that diversity in the activation pattern is linked to indicate these memorizations through information-theoretic measures.\n\nTo study the cause of the two types of memorizations, this paper establish the analysis from information theory side, it introduced two information theory measurements: the cross-sample diversity (i.e. entropy) and the cross-neuron diversity (MI). It then showed that heuristic memorization is linked to low cross-sample and cross-neuron diversity, while the example level memorization is linked to the both high value for these two diversities. The authors then demonstrated a potential application of this framework for model selection.\n", "questions": "The applicability of these two metrics across heterogeneous architectures are under-discussed. Based on my understanding, the table 3 in the supple line 565 is only describing architectures within one model setting. If one trained two models with different architectures, could these metrics stil be comparable? \n\nMinors:\n\nline 153:  \"This is done\" capital T.", "limitations": "There is no obvious negative societal impact given the theoretical nature of this work.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThis paper is overall well-written in clarity. The attacked problem, model memorization quatification, is a theoretically important task. This paper proposed one way to understand the neural network behavior with grounded tools.\n\n\nAs one limitation, the author claimed in line 264 that this model could be used point-wise to characterize model behavior over individual examples, potentially useful for OOD detection; indeed there is a practical gap before pointwise application, as when people build a single model (drawn one point from the model architecture space) with arbitrary shape, it is easy to measure the entropy and MI but difficult to tell whether these number indicates a high memorization ratio. When people drawn multiple samples from model architecture space, it is also hard to tell whether these meaturements remains descriptive across heterogeneous architectures.\n\nAs the metric values are relative references not direct pointer, the authors are encoraged to use a subsection to discuss about the applicabilicable range and the possible issue in practical run.\n\nOn the other hand, observed from the results in Fig 2/3, the argued relationship can be outweighted than the variance except when the handcraft ratio alpha is extremely large. Similar observation can be found in Fig 6/7, where the pattenr of the mean of entropy and MI is observable, but the variance remains big. Given this, an independent subsection in the paper for more discussions and adequate claim is recommended.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept